A high-speed motion capture method based on sparse IMU

Abstract

The application discloses a high-speed motion capture method based on sparse IMU, which comprises data acquisition, using an interpolation algorithm to improve the sampling frequency of the AMASS data set, and then using optical marker data to be equivalent to IMU data; data preprocessing to obtain corrected magnetometer readings; predicting the accelerometer and gyroscope of the super range through the IMU prediction algorithm and correcting; implementing the regression compensation algorithm on the predicted data; inferring the position of other joints of each joint of the human body relative to the root node, and estimating the rotation attitude of each joint to the root node and the root node speed, to obtain accurate human body motion. The application is based on sparse IMU to capture high-speed motion, provides more portable and simple wearable sensors for professional athletes, and provides motion data, effectively improving the efficiency and accuracy of human motion capture.

Description

Technical Field

[0001] This invention belongs to the field of human dynamics technology, and specifically relates to a high-speed motion capture method based on sparse IMU. Background Technology

[0002] Human motion capture technology uses sensors, cameras, or other devices to capture human movements and convert them into digital data. This technology has been widely used in various fields, including film and animation production, virtual reality and augmented reality (VR / AR), sports training and analysis, medicine and rehabilitation, robotics and artificial intelligence, and sports and entertainment games.

[0003] Currently, mainstream motion capture technology remains primarily optical motion capture. This technology (such as the Vicon system) requires placing optical markers on the joints of the subject and deploying multiple high-speed cameras (typically using infrared light sources) within the capture area. To ensure accurate capture, the optical markers need to be captured simultaneously by at least three cameras, placing extremely high demands on the camera placement and number. A complete motion capture operation typically requires a significant amount of time to adjust the camera positions. Furthermore, optical motion capture has other drawbacks, such as high equipment cost, sensitivity to lighting conditions and obstructions, complex pre-production processes, and the intrusive interference of optical markers, which may limit the natural movement of the subject. Due to the limited capture area, the application of optical motion capture in large-scale scenarios is also restricted, making it difficult to apply to consumer-grade applications.

[0004] Another technology is motion capture systems based on inertial measurement units (IMUs), such as Xsens. These systems capture and record motion data by deploying IMU sensors at the joints of the subject. An IMU typically consists of an accelerometer, gyroscope, and magnetometer, which can measure acceleration, angular velocity, and magnetic field strength. Although commercially available inertial motion capture systems (such as Xsens) can capture human movements relatively accurately, some limitations remain. For example, the Xsens system requires 17 IMU sensors to be deployed on the human body. While this dense layout provides accurate data, it increases system cost and may interfere with the subject's natural movements.

[0005] To address these issues, researchers have developed motion capture technology based on sparse IMUs. This technology reduces the number of IMUs required by combining human kinematic correlation with artificial intelligence algorithms while still reconstructing human posture. For example, Sony's Mocopi system uses six IMU sensors to capture the rotation of the head, waist, both forearms, and both lower legs, and then infers the rotation of other joints without deployed IMUs. However, the Mocopi system does not completely solve the IMU drift problem, thus requiring frequent motion calibration. Furthermore, the system has poor accuracy in predicting the rotation of joints without deployed IMUs, and there is a significant delay in motion reconstruction, all of which affect the user experience.

[0006] However, current sparse IMU motion capture technology does not support the capture of high-speed movements, such as tennis hitting and baseball pitching. One reason is that the update frequency of neural networks is usually low (less than 60 Hz), and according to the Nyquist-Shannon sampling theorem, it is difficult to accurately reconstruct high-speed movements. Another reason is that the range of low-cost MEMSMUs (Micro-Electro-Mechanical Inertial measurement units) is usually insufficient to acquire high-speed acceleration and angular velocity (accelerometer range ±16g, gyroscope range ±2000 dps, while the shoulder internal rotation angular velocity during baseball pitching reaches 5000-8000 dps, and the hand acceleration reaches 140g). Such high-speed movements beyond the range often prevent the IMU from obtaining accurate posture and movement data.

[0007] Therefore, existing attitude capture technology still needs to be improved. Summary of the Invention

[0008] Purpose of the invention: In order to overcome the above shortcomings, the purpose of this invention is to provide a high-speed motion capture method based on sparse IMU. This method captures high-speed motion based on sparse IMU, provides professional athletes with a more lightweight and simple wearable sensor, and provides motion data, effectively improving the efficiency and accuracy of human motion capture, so as to better meet the needs of users.

[0009] Technical Solution: To achieve the above objectives, this invention provides a high-speed motion capture method based on sparse IMUs, comprising the following steps:

[0010] S1): The IMU and magnetometer in the measurement unit detect data of the corresponding parts of each joint in the human body and send the obtained data to the microcontroller unit;

[0011] S2): In the microcontroller unit, the original magnetometer reading and the original IMU reading are corrected to obtain several correction parameters, including the corrected magnetometer reading, the corrected acceleration, and the corrected angular velocity.

[0012] S3): Predict and correct for over-range readings of the IMU accelerometer and gyroscope using the IMU prediction algorithm;

[0013] S4): Data fusion, which is to fuse the corrected magnetometer readings and the predicted over-range readings of the IMU accelerometer and gyroscope through the AHRS system to obtain the rotational attitude and geographic acceleration of the joints corresponding to the deployment positions of each measurement module.

[0014] S5): By analyzing and comparing the rotational attitude and geographic acceleration of the joints corresponding to the deployment positions of each measurement module obtained in S4) with the initial calibration data of the T action through the SMPL coordinate system, the rotational attitude of the SMPL coordinate system and the acceleration of each leaf node relative to the root node are obtained.

[0015] S6): Using a set of LSTM neural networks and an inertial navigation system, the position of each joint relative to the root node is inferred from the rotational attitude of the SMPL coordinate system obtained in S5) and the acceleration of the leaf node relative to the root node. The rotational attitude of each joint and the velocity of the root node are estimated, thereby obtaining accurate human motion.

[0016] The high-speed motion capture method based on sparse IMU described in this invention, in step S2), corrects the original magnetometer readings and the original IMU readings in the microcontroller unit to obtain several correction parameters, specifically:

[0017] The zero bias, scaling factor, and non-orthogonal error of the magnetometer were observed in real time using UKF, and the original magnetometer readings were corrected to obtain the corrected magnetometer.

[0018] The raw IMU readings are fitted to the IMU's zero bias, scale factor, and small rotation matrix using the least squares method through the microcontroller units in each measurement module / glove. Then, the corrected IMU readings, namely the corrected acceleration and corrected angular velocity, are obtained through correction.

[0019] In this invention, UKF is used to compensate for the zero bias, scaling factor, and non-orthogonality error of the magnetometer in real time, and the original magnetometer reading is corrected to obtain the corrected magnetometer reading. The specific correction process is as follows:

[0020] The measurement of a magnetometer can be modeled as follows:

[0021] B k =(I+D) -1 (A k H k +b+ε k ), k=1,...,N

[0022] Among them, B kH is the magnetic field measurement value of the magnetometer at time k. k It is the corresponding value of the geomagnetic field related to the Earth's fixed coordinate system, A k Let I be the unknown attitude matrix of the magnetometer relative to the geographic coordinate system, where I is a 3×3 identity matrix, D is the matrix of the magnetometer's scale factor error and non-orthogonal error, b is the magnetometer's zero bias vector, and ε is the matrix of the unknown attitude matrix of the magnetometer relative to the geographic coordinate system. k It is a measurement noise vector, assuming the measurement noise vector has a mean of zero and a covariance of ∑ k Gaussian process;

[0023] z k The measurement equation is:

[0024] z k =||B k || 2 -||H k || 2

[0025] Expand and organize z k Measurement equation:

[0026]

[0027] Where, v is defined k Error term for the noise component:

[0028] v k =2((I+D)B k -b) T ε k -||ε k || 2

[0029] v k It is approximately Gaussian noise, so its mean μ k and variance for:

[0030] μ k =E{v k}=E{2((I+D)B k -b) T ε k -||ε k || 2}=-Tr(Σ k )

[0031]

[0032] Where E{} is the expectation of the variable, Tr() is the trace of the variable, and σ kIt is a function of the magnetometer's scaling factor error and non-orthogonal row error matrix D and the magnetometer's zero bias vector b. Then, the variables are iteratively optimized using UKF. The prediction and correction steps follow the standard unscented Kalman filter method to obtain more accurate magnetometer readings.

[0033] In the high-speed motion capture method based on sparse IMU described in this invention, in step S3), the over-range readings of the IMU accelerometer and gyroscope are predicted by the IMU prediction algorithm and corrected. Since the IMU accelerometer and gyroscope contain data in three axes, an LSTM neural network is designed for each axis of the IMU accelerometer and gyroscope to ensure synchronous prediction and compensation of multi-dimensional data.

[0034] Taking angular velocity measured by a gyroscope as an example, the specific process is as follows:

[0035] S31): Data caching and saturation detection:

[0036] First, the corrected IMU reading in S2) is stored in the PC's cache for subsequent analysis and prediction. When the gyroscope reading reaches the upper or lower limit of its range, the IMU prediction algorithm identifies that the IMU has entered a saturation state.

[0037] When the gyroscope data is saturated, the saturation counter tracks the duration of the IMU reading in the saturated state. When the data tracked by the saturation counter value exceeds the set threshold N, the IMU prediction algorithm determines that there are too many over-range frames and fails to predict. This state is not normal high-speed motion, but rather an IMU malfunction. At this time, the prediction algorithm will stop to prevent the further propagation of erroneous data. When the data tracked by the saturation counter value is less than the set threshold N, the following prediction steps will be performed.

[0038] S32): Prediction and compensation in LSTM neural networks:

[0039] First, a prediction model based on an LSTM neural network is constructed. During the operation, high-range IMU data is used as the training reference for the LSTM neural network. The LSTM neural network is trained through supervised learning. The LSTM neural network predicts the overrange data based on the cached IMU data.

[0040] When the gyroscope reading returns to the range, the saturation counter will automatically reset to zero, indicating that the IMU has returned to normal.

[0041] S33): Regression Compensation:

[0042] When users prioritize the real-time performance of high-speed motion capture, the IMU prediction algorithm can optionally disable regression compensation. When users prioritize the accuracy of motion capture, the IMU prediction algorithm can add regression compensation to further optimize the accuracy of the prediction.

[0043] The high-speed motion capture method based on sparse IMU described in this invention includes the following specific formula for the regression compensation algorithm:

[0044]

[0045] Where i represents the frame number after entering the saturation state, and i is an integer less than the maximum value m of the saturation counter. To correct the predicted angular velocity of the i-th frame, For the predicted angular velocity of the i-th frame before correction, m gyr Angular velocity was measured in the first frame after the system returned to normal. The predicted angular velocity data for the m-th frame before correction is the predicted angular velocity of the last frame in the saturation state.

[0046] The high-speed motion capture method based on sparse IMU described in this invention includes the following specific regression compensation method:

[0047] When the angular velocity exceeds the gyroscope's range (between 10.4 and 10.6 s), it enters saturation, and the gyroscope reading stops at 2000 dps. At this point, the saturation counter increments, and the IMU prediction algorithm infers the next 256 frames of data based on the previous 512 frames. Then, the saturation counter increments to 17, and the gyroscope reading is less than 2000, returning to the range. Frames 1 to 17 after entering saturation are selected, and regression compensation is performed on them to obtain the corrected predicted gyroscope readings.

[0048] If the customer requires highly real-time data, the algorithm outputs predicted gyr; if the customer requires even higher accuracy, the algorithm outputs compensated gyr.

[0049] The high-speed motion capture method based on sparse IMU described in this invention uses an LSTM neural network-based prediction model with a five-layer structure: one dropout layer, one linear input layer, two LSTM hidden layers, and one linear output layer. Each LSTM hidden layer contains 64 neurons to capture the complex temporal relationships of motion data. The input to the dropout layer is the IMU data from the past K frames in the buffer, and the output of the linear output layer is the predicted data from L frames. During operation, the LSTM neural network models and learns the motion data in the IMU's out-of-range range, training the LSTM neural network to predict the waveform of motion data within the IMU's out-of-range range. High-range IMU data is used as a training reference, and the LSTM neural network is trained through supervised learning. The five-layer structure of the LSTM neural network-based prediction model operates as follows:

[0050] The input K-frame IMU data first passes through a dropout layer, where some features are randomly zeroed while maintaining the shape of the input. This helps prevent the model from overfitting and enhances the model's generalization ability.

[0051] The output of the dropout layer is fed into the linear input layer, which transforms the dimension of the input features into a vector of length 64 through a linear mapping to match the input of the subsequent LSTM hidden layer. This can compress features and perform feature space transformation.

[0052] The output of the linear input layer is fed into the first LSTM hidden layer, which captures the short-term temporal relationships of motion data, i.e., models the temporal dependence of the input sequence; the second LSTM hidden layer further extracts higher-order temporal features, enhancing the model's understanding of complex temporal relationships.

[0053] The outputs of the two LSTM hidden layers eventually enter the linear output layer. The linear output layer maps the output of the LSTM hidden state to the target prediction sequence. It transforms the high-dimensional hidden state into a prediction data sequence of length L frames, providing the motion prediction result for each frame.

[0054] The high-speed motion capture method based on sparse IMU described in this invention employs cubic spline interpolation to better fit the curves between data points. Especially when capturing high-speed motion, it provides a smoother transition and reduces oscillations. Cubic spline interpolation constructs piecewise cubic polynomials to ensure that the curves in each interval have continuous first and second derivatives, thereby improving the fitting accuracy and making it more suitable for interpolation processing of high-speed motion data.

[0055] By using cubic spline interpolation, we are able to increase the frame rate to 500Hz while preserving the original motion characteristics, and ensure that the generated interpolated data can more accurately reflect motion details.

[0056] The LSTM neural network described in this invention uses the AMASS dataset as training data. Interpolation techniques are employed to uniformly increase the frame rate of the AMASS dataset to 500Hz. The optically labeled data used in the AMASS dataset is equivalently converted to IMU data, and then the equivalently converted IMU data is used as input for supervised training. The specific process of equivalently converting the optically labeled data used in the AMASS dataset to IMU data is as follows:

[0057] The rotation matrices of each leaf node joint in the AMASS dataset are selected, specifically the rotation matrices corresponding to the rotation postures of the left and right ankles, left and right wrists, pelvis, and head. Then, the rotation matrices in the SMPL coordinate system are obtained through forward kinematics.

[0058] In addition, the AMASS dataset also includes spatial location data of the human body mesh. The acceleration information of the leaf node positions is obtained by taking the second derivative of the spatial location data of the human body mesh.

[0059] The high-speed motion capture method based on sparse IMU described in this invention, the specific process of data fusion in step S4) is as follows: the northeast coordinate system is E, and the IMU coordinate system is S;

[0060] The data returned by the IMU's three-axis gyroscope is the angular velocity of the system around the x, y, and z axes. These three angular velocities are represented by ωx, ωy, and ωz, respectively. x ω y ω z In other words, the data returned by the IMU's three-axis gyroscope can be viewed as an attitude quaternion with a real part of zero, using the corrected angular velocity ω. S To indicate:

[0061]

[0062] attitude quaternion change rate With the current attitude quaternion and angular velocity ω S The details are as follows:

[0063]

[0064] Given the attitude quaternion at time t-1 and angular velocity ω S t-1 and the angular velocity ω at time t s t Given a system sampling interval of Δt, calculate the attitude quaternion at time t.

[0065]

[0066] Wherein, K1 is a preliminary estimate of the attitude quaternion change, which represents the rate of change of the attitude quaternion caused by the gyroscope angular velocity information;

[0067] K2 is the attitude quaternion update calculated by adding the time step correction to the attitude quaternion at time t-1.

[0068] The approximate rate of change of the object's attitude quaternion between time t-1 and time t is: In practice, however, the exact attitude quaternion at time t-1 is often not obtained; instead, an optimal estimate is obtained. Therefore, the attitude quaternion updated at time t based on gyroscope data from the northeast-to-east coordinate system to the sensor coordinate system. as follows:

[0069]

[0070] Assuming the gyroscope accurately calculates the attitude, then the E-frame gravity vector... The acceleration vector g in the attitude solution in the S-frame S t It should be:

[0071]

[0072] The acceleration vector g of attitude calculation S t And the actual accelerometer readings The deviation is e s a,t =cross(a s t ,g s t )

[0073] Assuming the corrected magnetometer reading in the S-frame is... After rotating to the E series:

[0074]

[0075] Since the E-frame uses a northeast coordinate system, the geomagnetic field lines only have components along the x and z axes. Therefore, the theoretical magnetometer output value b in the E-frame is... E t for:

[0076]

[0077] Theoretically h y ≈0, but the actual attitude calculation has an error h. y ≠0, therefore:

[0078]

[0079] The theoretical magnetometer output value b of the E-series E t Rotate again to the S system to obtain W. S t :

[0080]

[0081] Then calculate W. S t The corrected magnetometer measured the value m in the S-system. S t Deviation:

[0082] eS m，t =cross(m S t W S t )

[0083] Therefore, the total error of the magnetometer and IMU is:

[0084] e S t =e S a，t +e S m，t The gyroscope is compensated for by a PI controller:

[0085]

[0086] Therefore, the compensated attitude quaternion is updated as follows:

[0087]

[0088] Normalized pose quaternion, final pose update:

[0089]

[0090] Preferably, in step S6), a set of LSTM neural networks and an inertial navigation system infer the position of each joint relative to the root node based on the rotational attitude of the SMPL coordinate system obtained in step S5 and the acceleration of the leaf node relative to the root node, and estimate the rotational attitude of each joint and the velocity of the root node.

[0091] The LSTM neural network has five branches, three of which are used to estimate the pose of human joints, and the other two are used to estimate the velocity of the human body in the SMPL coordinate system. The specific allocation is as follows:

[0092] One of the components, an LSTM neural network, obtains the acceleration and rotational attitude of the IMU from the measurement module to estimate the position of each joint of the human body relative to the root node.

[0093] Then, using the positions of each joint of the human body relative to the root node and IMU information as input, an LSTM neural network is used to infer the positions of other joints of the human body relative to the root node, except for the finger joints.

[0094] Finally, using the positions of all the joints in the human body and the measurement information from the IMU as input, an LSTM neural network is used to estimate the rotational orientation of the other joints relative to the root node.

[0095] The rotation attitude of the root node is the IMU solution attitude transformed into the SMPL coordinate system.

[0096] The root node velocity described in this invention is constrained by physical motion during processing, specifically as follows:

[0097] The acceleration and rotation of the SMPL coordinate system, obtained from the IMU measurements of the six measurement modules, are used as inputs. An LSTM neural network is used to estimate the probability of whether the left and right sides will touch the ground. If the probability of touching the ground is high, the foot will not slide relative to the ground, and the velocity of the root node is calculated using forward dynamics. If the probability of touching the ground is low, an LSTM neural network is used to directly predict the velocity of the root node from the estimated joint velocities (excluding the fingers) and the acceleration information of the six IMUs.

[0098] The prediction process for the root node's speed is as follows:

[0099] The effective range of the landing probability p is 0.5 to 0.9, and the side with the higher probability is selected.

[0100] If p is less than 0.5, then the root node velocity v2 predicted by the RNN is used entirely;

[0101] If p is greater than 0.9, then the root node velocity v1 calculated using forward dynamics is used entirely;

[0102] If p is between 0.5 and 0.9, then use

[0103] v=v1*(p-0.5) / (0.9-0.5)+v2*(0.9-p) / (0.9-0.5);

[0104] The speed based on forward dynamics calculation and speed based on LSTM neural network estimation are combined to form the final root node geographic system speed.

[0105] As can be seen from the above technical solution, the present invention has the following beneficial effects:

[0106] 1. The present invention discloses a high-speed motion capture method based on sparse IMU, which captures high-speed motion based on sparse IMU, providing professional athletes with a more lightweight and simple wearable sensor and providing motion data, effectively improving the efficiency and accuracy of human motion capture, and better meeting the needs of users.

[0107] 2. This invention effectively solves the problem that motion acceleration and angular velocity exceed the range of existing MEMSIMUs by using an IMU prediction algorithm, thus compensating for the range limitations of low-cost IMUs. A saturation counter is introduced into the IMU prediction algorithm to track the duration of sensor readings in a saturated state. When the counter value exceeds a set threshold N, the algorithm determines that this state may not be normal high-speed motion, but rather a sensor malfunction. At this point, the prediction algorithm will stop to prevent the further propagation of erroneous data. Furthermore, LSTM neural network prediction and compensation are used to predict accelerometer and gyroscope data during the over-range period, further improving the accuracy of the prediction.

[0108] 3. Regression compensation algorithm settings: When users prioritize the real-time performance of high-speed motion capture, our algorithm can optionally disable regression compensation; when users prioritize the accuracy of motion capture, regression compensation is used to obtain corrected predictive gyroscope readings, further improving the accuracy of prediction.

[0109] 4. This invention uses the AMASS dataset as training data, which contains more than 40 hours of motion capture data, covering more than 11,000 different actions, effectively increasing the frame rate of motion capture. At the same time, by using interpolation technology, the frame rate of the dataset is uniformly increased to 500Hz to ensure that the training data can more accurately reflect the characteristics of high-speed motion. This effectively solves the problem that the sampling frequency of most data in the original dataset is 59, 60, 100 and 120 frames per second (fps), which may be insufficient for capturing details of high-speed motion.

[0110] 5. In this invention, the optically labeled data used in the AMASS dataset is equivalently converted into IMU data for supervised learning during the training of the new neural network. In this way, the new neural network can better adapt to the characteristics of high-speed motion data and provide higher reconstruction accuracy when processing high-speed motion. Attached Figure Description

[0111] Figure 1 This is a schematic diagram of the high-speed motion capture method based on sparse IMU described in this invention.

[0112] Figure 2 This is a flowchart illustrating the IMU prediction algorithm for predicting overrange IMU data in this invention.

[0113] Figure 3 This refers to the angular velocity data of the upper limb joints of an amateur baseball player during pitching in this invention;

[0114] Figure 4This is a schematic diagram of the prediction and compensation of the overrange gyroscope reading in this invention. In the figure, Figure A is a schematic diagram of the original x-axis reading of the gyroscope, Figure B is a schematic diagram of the LSTM neural network predicted reading before correction, and Figure C is a schematic diagram of the predicted reading after correction.

[0115] Figure 5 This is a schematic diagram of the electrical connections of the measurement module in this invention;

[0116] Figure 6 This is a schematic diagram of the deployment of the measurement module in this invention;

[0117] Figure 7 This is a schematic diagram of the router network configuration in this invention;

[0118] Figure 8 This is a flowchart of the data synchronization algorithm in this invention;

[0119] Figure 9 Here is a list of 22 joints in the human body model of this invention;

[0120] Figure 10 This is a flowchart of the automated calibration method for whole-body posture prediction in this invention. Detailed Implementation

[0121] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0122] Example 1

[0123] The high-speed motion capture method based on sparse IMU described in this embodiment uses a motion capture system including a measurement module, a router, and a PC host computer. The measurement module includes a measurement unit, a WiFi module, a microcontroller unit, and a battery module. The measurement unit and the WiFi module are serially connected to the microcontroller unit, and the measurement unit, WiFi module, and microcontroller unit are all connected to the battery module. Each measurement module includes a measurement unit, and the measurement unit includes at least one IMU and at least one magnetometer.

[0124] The router is connected to the measurement module and the PC host computer. The router is used to network all measurement modules and the PC host computer. The router records the physical address of the Wi-Fi of all measurement modules, assigns a specified IP address, and then sends the data to the PC host computer through the network communication protocol.

[0125] In this embodiment, by improving the structure of the motion capture system, it is possible to reconstruct the motion posture of the main joints of the human body in real time using a small number of IMUs, thereby improving the accuracy of human motion capture and expanding the application scenarios of IMU-based motion capture. At the same time, the hardware cost of IMUs is lower than that of motion capture cameras, which effectively solves the problems existing in the prior art.

[0126] like Figure 6 The measurement module shown is located on the left and right forearms near the wrist joint, on the left and right lower legs near the knee joint, behind the head, and behind the waist. It should be noted that the position of the measurement module can be adaptively adjusted according to the actual needs of motion capture.

[0127] like Figure 7 The human motion signal captured by the measurement unit shown is sent to the corresponding microcontroller unit. The inertial signal is preprocessed by the filter in the microcontroller unit. The rotational attitude (such as Euler angles, attitude quaternions, or rotation matrix) is obtained by fusing the inertial signal through the attitude reference system (AHRS). Then, the IMU preprocessed data and rotational attitude are sent to the WiFi module. Finally, the data is sent to the router through a network transmission protocol (such as TCP protocol), and then sent to the PC host computer through a network communication protocol.

[0128] like Figure 1 The high-speed motion capture method based on sparse IMUs, as shown, includes the following steps:

[0129] S1): Data acquisition, that is, the IMU and magnetometer in the measurement unit detect the data of the corresponding parts of each joint, obtain the raw IMU reading and the raw magnetometer reading, and send the obtained data to the microcontroller unit;

[0130] S2): Data preprocessing, which involves correcting the original IMU readings and original magnetometer readings in the microcontroller unit. This is achieved by using UKF to compensate for the magnetometer's zero bias, scale factor, and non-orthogonal error in real time, and then correcting the original magnetometer readings to obtain the corrected magnetometer readings. The original IMU readings are fitted to the IMU's zero bias, scale factor, and small rotation matrix using the least squares method by the microcontroller units of each measurement module, and then corrected to obtain the corrected IMU readings, namely, corrected acceleration and corrected angular velocity.

[0131] S3): Predict and correct the over-range readings of the IMU accelerometer and gyroscope using the IMU prediction algorithm;

[0132] S4): Data fusion, which is to fuse the corrected magnetometer readings and the predicted overrange readings of the IMU accelerometer and gyroscope through the AHRS system to obtain the rotational attitude and geographic acceleration of the joints corresponding to the deployment positions of each measurement module.

[0133] S5): The rotational attitude and geographic acceleration obtained in S4) are analyzed and compared with the initial calibration data of the T action using the SMPL coordinate system to obtain the rotational attitude and the acceleration of the leaf node relative to the root node in the SMPL coordinate system.

[0134] S6): Using a set of LSTM neural networks and an inertial navigation system, the position of each joint relative to the root node is inferred from the rotational attitude of the SMPL coordinate system obtained in S5) and the acceleration of the leaf node relative to the root node. The rotational attitude of each joint and the velocity of the root node are estimated, thereby obtaining accurate human motion.

[0135] It should be noted that the node at the back of the waist is the root node, and the nodes at other locations are leaf nodes.

[0136] It should be noted that during high-speed motion, the IMU reading may exceed the IMU's range. Therefore, a predictive algorithm is used to predict the reading that exceeds the range.

[0137] The current IMU range is limited. This situation is particularly pronounced during high-speed movements involving critical joints such as the shoulder. Due to the characteristics of movement and muscles, exceeding the IMU range typically occurs within a very short timeframe, manifesting as a momentary high-intensity movement rather than a sustained state of over-range measurement, such as... Figure 3 As shown. The data comes from "IMU Sensor Module for the Measurement of High-speed Motion in the Analysis of Human Skills".

[0138] In the high-speed motion capture method based on sparse IMU described in this embodiment, in step S3), the over-range readings of the IMU accelerometer and gyroscope are predicted by the IMU prediction algorithm and corrected. Since the IMU accelerometer and gyroscope contain data in three axes, an LSTM neural network is designed for each axis of the IMU accelerometer and gyroscope to ensure synchronous prediction and compensation of multi-dimensional data.

[0139] Taking angular velocity measured by a gyroscope as an example, the specific process is as follows:

[0140] S31): Data caching and saturation detection:

[0141] First, the corrected IMU reading in S2) is stored in the PC's cache for subsequent analysis and prediction. When the gyroscope reading reaches the upper or lower limit of its range, the IMU prediction algorithm identifies that the IMU has entered a saturation state.

[0142] When the gyroscope data is saturated, the saturation counter tracks the duration of the IMU reading in the saturated state. When the data tracked by the saturation counter value exceeds the set threshold N, the IMU prediction algorithm determines that there are too many over-range frames and fails to predict. This state is not normal high-speed motion, but rather an IMU malfunction. At this time, the prediction algorithm will stop to prevent the further propagation of erroneous data. When the data tracked by the saturation counter value is less than the set threshold N, the following prediction steps will be performed.

[0143] S32): Prediction and compensation in LSTM neural networks:

[0144] First, a prediction model based on an LSTM neural network is constructed. During the operation, high-range IMU data is used as the training reference (training set and test set) for the LSTM neural network. The LSTM neural network is trained through supervised learning. The LSTM neural network predicts the out-of-range data based on the cached IMU data.

[0145] When the gyroscope reading returns to the range, the saturation counter will automatically reset to zero, indicating that the IMU has returned to normal.

[0146] S33): Regression Compensation:

[0147] When users prioritize the real-time performance of high-speed motion capture, the IMU prediction algorithm can optionally disable regression compensation. When users prioritize the accuracy of motion capture, the IMU prediction algorithm can add regression compensation to further optimize the accuracy of the prediction.

[0148] The high-speed motion capture method based on sparse IMU described in this embodiment has the following specific formula for the regression compensation algorithm:

[0149]

[0150] Where i represents the frame number after entering the saturation state, and i is an integer less than the maximum value m of the saturation counter. To correct the predicted angular velocity of the i-th frame, For the predicted angular velocity of the i-th frame before correction, m gyr Angular velocity was measured in the first frame after the system returned to normal. The predicted angular velocity data for the m-th frame before correction is the predicted angular velocity of the last frame in the saturation state.

[0151] The high-speed motion capture method based on sparse IMU described in this embodiment includes the following specific regression compensation method: (e.g.) Figure 4 As shown, the horizontal axis represents time, and the vertical axis represents angular velocity;

[0152] When the angular velocity exceeds the gyroscope's range (between 10.4 and 10.6 s), it enters saturation, and the gyroscope reading stops at 2000 dps. At this point, the saturation counter increments, and the IMU prediction algorithm infers the next 256 frames of data based on the previous 512 frames. Then, the saturation counter increments to 17, and the gyroscope reading is less than 2000, returning to the range. Frames 1 to 17 after entering saturation are selected, and regression compensation is performed on them to obtain the corrected predicted gyroscope readings.

[0153] If the customer requires highly real-time data, the algorithm outputs predicted gyr; if the customer requires even higher accuracy, the algorithm outputs compensated gyr.

[0154] The high-speed motion capture method based on sparse IMU described in this embodiment uses an LSTM neural network-based prediction model with a five-layer structure: one dropout layer, one linear input layer, two LSTM hidden layers, and one linear output layer. Each LSTM hidden layer contains 64 neurons to capture the complex temporal relationships of motion data. The input to the dropout layer is the IMU data of the past K frames in the buffer, and the output of the linear output layer is the predicted data of L frames. During operation, the LSTM neural network is used to model and learn the motion data of the IMU's out-of-range range. The LSTM neural network is trained to predict the waveform of motion data within the IMU's out-of-range range. High-range IMU data is used as a training reference, and the LSTM neural network is trained through supervised learning.

[0155] The five-layer structure of the prediction model based on the LSTM neural network operates as follows:

[0156] The input K-frame IMU data first passes through a dropout layer, where some features are randomly zeroed while maintaining the shape of the input. This helps prevent the model from overfitting and enhances the model's generalization ability.

[0157] The output of the dropout layer is fed into the linear input layer, which transforms the dimension of the input features into a vector of length 64 through a linear mapping to match the input of the subsequent LSTM hidden layer. This can compress features and perform feature space transformation.

[0158] The output of the linear input layer is fed into the first LSTM hidden layer, which captures the short-term temporal relationships of motion data, i.e., models the temporal dependence of the input sequence; the second LSTM hidden layer further extracts higher-order temporal features, enhancing the model's understanding of complex temporal relationships.

[0159] The outputs of the two LSTM hidden layers eventually enter the linear output layer. The linear output layer maps the output of the LSTM hidden state to the target prediction sequence. It transforms the high-dimensional hidden state into a prediction data sequence of length L frames, providing the motion prediction result for each frame.

[0160] High-range IMU data was used as a reference when training the network weights. These high-range IMUs can comprehensively capture data waveforms during high-speed motion, but their high cost makes them unsuitable for large-scale application in low-cost solutions. Therefore, we modeled and learned the motion data beyond the range using a neural network, training an LSTM neural network to predict the waveforms of motion data within the beyond-range range. Ultimately, this method allows inexpensive low-range IMUs to achieve similar results to high-range IMUs through neural network prediction, balancing cost and performance. Since accelerometers and gyroscopes typically contain data along three axes, a separate LSTM neural network was designed for each axis to ensure synchronous prediction and compensation of multi-dimensional data.

[0161] The high-speed motion capture method based on sparse IMU described in this embodiment uses cubic spline interpolation to better fit the curves between data points. Especially when capturing high-speed motion, it can provide a smoother transition and reduce oscillations. Cubic spline interpolation constructs piecewise cubic polynomials to ensure that the curves in each interval have continuous first and second derivatives, thereby improving the fitting accuracy and making it more suitable for interpolation processing of high-speed motion data.

[0162] By using cubic spline interpolation, we are able to increase the frame rate to 500Hz while preserving the original motion characteristics, and ensure that the generated interpolated data can more accurately reflect motion details.

[0163] The high-speed motion capture method based on sparse IMU described in this embodiment uses the AMASS dataset as training data for the LSTM neural network. Interpolation techniques are used to uniformly increase the frame rate of the AMASS dataset to 500Hz. The optically labeled data used in the AMASS dataset is equivalently converted into IMU data, and then the equivalently converted IMU data is used as input for supervised training. The specific process of equivalently converting the optically labeled data used in the AMASS dataset into IMU data is as follows:

[0164] The rotation matrices of each leaf node joint in the AMASS dataset are selected, specifically the rotation matrices corresponding to the rotation postures of the left and right ankles, left and right wrists, pelvis, and head. Then, the rotation matrices in the SMPL coordinate system are obtained through forward kinematics.

[0165] In addition, the AMASS dataset also includes spatial location data of the human body mesh. The acceleration information of the leaf node positions is obtained by taking the second derivative of the spatial location data of the human body mesh.

[0166] The interpolation technique employs cubic spline interpolation, which allows motion capture to increase the frame rate to 500Hz while preserving the original motion characteristics, and ensures that the generated interpolated data can more accurately reflect motion details and is compatible with high-sampling-frequency IMUs.

[0167] The accelerometer and gyroscope contain data for three axes. An LSTM neural network was designed for each axis of the accelerometer and gyroscope to ensure synchronous prediction and compensation of multi-dimensional data.

[0168] Example 2

[0169] The high-speed motion capture method based on sparse IMU described in this embodiment is the same as the method in Embodiment 1. Furthermore, in the data preprocessing, UKF is used to compensate for the magnetometer's zero bias, scaling factor, and non-orthogonal error in real time, and the original magnetometer readings are corrected to obtain the corrected magnetometer reading scaling factor. The specific correction process is as follows:

[0170] The measurement of a magnetometer can be modeled as follows:

[0171] B k =(I+D) -1 (A k H k +b+ε k ), k=1,...,N

[0172] Among them, B k H is the magnetic field measurement value of the magnetometer at time k. k It is the corresponding value of the geomagnetic field related to the Earth's fixed coordinate system, A k Let I be the unknown attitude matrix of the magnetometer relative to the geographic coordinate system, where I is a 3×3 identity matrix, D is the matrix of the magnetometer's scale factor error and non-orthogonal alignment error (inter-axis coupling), b is the magnetometer's zero bias vector, and ε is the matrix of the unknown attitude matrix of the magnetometer relative to the geographic coordinate system. k It is a measurement noise vector, assuming the measurement noise vector has a mean of zero and a covariance of ∑ k Gaussian process;

[0173] Z k The measurement equation is:

[0174] Z k =||B k || 2 -||H k || 2

[0175] Expand and organize Z k Measurement equation:

[0176]

[0177] Where, v is defined k Error term for the noise component:

[0178] v k =2((I+D)B k -b) T ε k -||ε k || 2

[0179] v k It is approximately Gaussian noise, so its mean μ k and variance for:

[0180] μ k =E{v k}=E{2((I+D)B k -b) T ε k -||e k || 2}=-Tr(Σ k )

[0181]

[0182] Where E{} is the expectation of the variable, Tr() is the trace of the variable, and σ k It is a function of the matrix D of the magnetometer's scale factor error and non-orthogonal row error, and the magnetometer's zero bias vector b. To estimate D and b, the following variables are defined:

[0183] F2D+D 2

[0184] f = [F 11 ,F 22 ,F 33 ,F 12 ,F 13 ,F 23 ] T

[0185] c = (I + D)b

[0186]

[0187] Where F is a symmetric matrix, F ij Let represent the element in the i-th row and j-th column of the symmetric matrix F; ij = 11, 22, 33, 12, 13, 23;

[0188] f is a 6-row, 1-column column vector composed of the elements of the symmetric matrix F;

[0189] c is a 3x1 column vector, calculated from I, D, and b;

[0190] B i，k B k The vector in the i-th row of the matrix is ​​a vector with 1 row and 3 columns; B k The matrix is ​​the product of the vectors in the i-th row of the matrix; it is a 3x3 matrix. B k The matrix is ​​the product of the i-th row vector and the j-th row vector. It is a 3x3 matrix. The permutations of i and j are i=1 and j=2, i=1 and j=3, and i=2 and j=3.

[0191] L k Indicates by and -S k A matrix formed by splicing together elements;

[0192] θ is a 9x1 column vector consisting of c and f;

[0193] So z was reorganized k have:

[0194]

[0195] Where b(c,f) is a function of b expressed by c and f, and b(θ) is a function of b expressed by θ;

[0196] Once the symmetric matrices F and c are determined, D and b can be derived. The specific derivation process is as follows:

[0197] The eigenvalue decomposition of the symmetric matrix F is as follows:

[0198] F = UVU T

[0199] Where U is an orthogonal matrix, that is, the eigenvector matrix of F;

[0200] V = diag(V) 11 V 22 V 33 F is a diagonal matrix, and its diagonal elements are the eigenvalues ​​of F.

[0201] Eigenvector definition Fu i as follows:

[0202] Fu i =UVU T u i=Udiag(0,…,V ii ,…,0)=V ii u i

[0203] Among them, u i V is the i-th column vector of the orthogonal matrix U. ii It is the i-th diagonal element of matrix V; then define a new diagonal matrix W = diag(W 11 W 22 W 33 Its diagonal elements are obtained by the following formula:

[0204] Where j = 1, 2, 3;

[0205] Therefore, D and b are obtained as follows:

[0206] D=UWU T

[0207] b = (I + D) -1 c

[0208] Then we define θ for the k-th frame, i.e., θ k For system state; z k For the measurement equation, the following Kalman filter equation can be constructed:

[0209]

[0210] Then, UKF is used to iteratively optimize the variables, and the prediction and correction steps follow the standard unscented Kalman filter method.

[0211] This allows for real-time estimation of the values ​​of D and b, resulting in more accurate magnetometer readings.

[0212] Because the system is a discrete system, therefore θ k The variable value at time K is represented by the UKF variable value at the next discrete time k+1 derived from the previous discrete time k. The system state is equivalent to the state variables in modern control theory, which is the smallest set of variables that can completely determine the system's operating state.

[0213] It should be noted that UKF optimizes θ k This makes each step of the observation Z k Minimize the covariance of the residuals between the UKF model predictions and the UKF model predictions.

[0214] In the high-speed motion capture method based on sparse IMU described in this embodiment, the specific process of data fusion in step S4) is as follows: the northeast coordinate system is E, and the IMU coordinate system is S;

[0215] The data returned by the IMU's three-axis gyroscope is the angular velocity of the system around the x, y, and z axes. These three angular velocities are represented by ωx, ωy, and ωz, respectively. x ω y ω z In other words, the data returned by the IMU's three-axis gyroscope can be viewed as an attitude quaternion with a real part of zero, using the corrected angular velocity ω. S To indicate:

[0216]

[0217] attitude quaternion change rate With the current attitude quaternion and angular velocity ω S The details are as follows:

[0218]

[0219] Given the attitude quaternion at time t-1 and angular velocity ω S t-1 and the angular velocity ω at time t S t Given a system sampling interval of Δt, calculate the attitude quaternion at time t.

[0220]

[0221] Wherein, K1 is a preliminary estimate of the attitude quaternion change, which represents the rate of change of the attitude quaternion caused by the gyroscope angular velocity information;

[0222] K2 is the attitude quaternion update calculated by adding the time step correction to the attitude quaternion at time t-1.

[0223] The approximate rate of change of the object's attitude quaternion between time t-1 and time t is: In practice, however, the exact attitude quaternion at time t-1 is often not obtained; instead, an optimal estimate is obtained. Therefore, the attitude quaternion updated at time t based on gyroscope data from the northeast-to-east coordinate system to the sensor coordinate system. as follows:

[0224]

[0225] Assuming the gyroscope accurately calculates the attitude, then the E-frame gravity vector... The acceleration vector g in the attitude solution in the S-frame s t It should be:

[0226]

[0227] The acceleration vector g of attitude calculation S t And the actual accelerometer readings The deviation is e S a，t =cross(a S t g S t )

[0228] Assuming the corrected magnetometer reading in the S-frame is... After rotating to the E series:

[0229]

[0230] Since the E-frame uses a northeast coordinate system, the geomagnetic field lines only have components along the x and z axes. Therefore, the theoretical magnetometer output value b in the E-frame is... E t for:

[0231]

[0232] Theoretically h y ≈0, but the actual attitude calculation has an error h. y ≠0, therefore:

[0233]

[0234] The theoretical magnetometer output value b of the E-series E t Rotate again to the S system to obtain W. s t :

[0235]

[0236] Then calculate W. s t The corrected magnetometer measured the value m in the S-system. S t Deviation:

[0237] e S m，t =cross(m S t W S t )

[0238] Therefore, the total error of the magnetometer and IMU is:

[0239] e S t =eS a，t +e S m，t

[0240] The gyroscope is compensated for by a PI controller:

[0241]

[0242] Therefore, the compensated attitude quaternion is updated as follows:

[0243]

[0244] Normalized pose quaternion, final pose update:

[0245]

[0246] In this embodiment, in S6), a set of LSTM neural networks and an inertial navigation system infer the position of each joint relative to the root node based on the rotational attitude of the SMPL coordinate system and the acceleration of the leaf node relative to the root node obtained in S5), and estimate the rotational attitude of each joint and the velocity of the root node. The LSTM neural network has 5 branches, of which 3 are used to estimate the attitude of the human joints and the other 2 are used to estimate the velocity of the human body in the SMPL coordinate system. The specific allocation is as follows:

[0247] One of the components, an LSTM neural network, obtains the acceleration and rotational attitude of the IMU from the measurement module to estimate the position of each joint relative to the root node.

[0248] Then, using the position of each joint relative to the root node and IMU information as input, an LSTM neural network is used to infer the position of other joints in the human body relative to the root node, except for the finger joints.

[0249] Finally, using the positions of all the joints in the human body and the measurement information from the IMU as input, an LSTM neural network is used to estimate the rotational orientation of the other joints relative to the root node.

[0250] The rotational attitude of the root node is the IMU-calculated attitude transformed into the SMPL coordinate system. The root node velocity is constrained by physical motion during processing, specifically as follows:

[0251] The acceleration and rotation of the SMPL coordinate system obtained from the IMU measurements of each measurement module are used as inputs. An LSTM neural network is used to estimate the probability of whether the left and right sides will touch the ground. If the probability of touching the ground is high, the foot will not slide relative to the ground, and the velocity of the root node is calculated using forward dynamics. If the probability of touching the ground is low, an LSTM neural network is used to directly predict the velocity of the root node from the estimated joint velocities (excluding the fingers) and the acceleration information of each IMU.

[0252] The prediction process for the root node's velocity is as follows:

[0253] The effective range of the landing probability p is 0.5 to 0.9, and the side with the higher probability is selected.

[0254] If p is less than 0.5, then the root node velocity v2 predicted by the RNN is used entirely;

[0255] If p is greater than 0.9, then the root node velocity v1 calculated using forward dynamics is used entirely;

[0256] If p is between 0.5 and 0.9, then use

[0257] v=v1*(p-0.5) / (0.9-0.5)+v2*(0.9-p) / (0.9-0.5);

[0258] The speed based on forward dynamics calculation and speed based on LSTM neural network estimation are combined to form the final root node geographic system speed.

[0259] It should be noted that each LSTM neural network requires different types of data and is trained independently. Relevant datasets include: 1) DIP-IMU [Huang et al. 2018], which contains IMU measurement data and pose parameters from 10 subjects wearing 17 IMUs performing approximately 90 minutes of movement; 2) TotalCapture [Trumble et al. 2017], which contains IMU measurement data, pose parameters, and global translation data from 5 subjects wearing 13 IMUs performing approximately 50 minutes of movement; and 3) AMASS [Mahmood et al. 2019], a dataset composed of existing motion capture (mocap) datasets, containing pose parameters and global translation data from over 300 subjects performing over 40 hours of movement. The training, validation, and test sets are divided proportionally, with the training set comprising 70%-80% of the total data, the validation set 10%-15%, and the test set 10%-15%. Further details are omitted here.

[0260] Example 3

[0261] The high-speed motion capture method based on sparse IMU described in this embodiment is the same as that in embodiments 1 and 3. Furthermore, the "T-pose" action, in motion capture or human posture recognition, typically refers to a posture resembling the letter "T," with arms extended horizontally to the ground and feet together. This posture can be used to initialize the sensor calibration of the motion capture system or device (the IMU's deployment position may be misaligned with muscles, skin, or the wearing method and corresponding skeletal joints). The T-pose action primarily eliminates sensor-to-skeleton errors. Under the T-pose action, the rotation matrix of all joints is an identity matrix, but the rotation matrix calculated by the corresponding IMU... Various differences exist, which can be addressed through a compensation matrix. Align them into an identity matrix.

[0262] like Figure 10 The T-action shown is used to initialize the sensor calibration of the motion capture system or device.

[0263] The method is as follows:

[0264] S51): First, after the user puts on all the measurement modules / measuring gloves, he faces due north and assumes a Tpose pose. After the motion capture system or device is powered on, the distributed MCU uses UKF to observe the magnetometer's zero bias, scale factor, and non-orthogonal error in real time. The AHRS system is used to fuse the corrected IMU readings and the corrected magnetometer readings. The implementation method is the same as S2) and S4) in Example 1.

[0265] S52): AHRS calculates and obtains the attitude of each measurement unit in the measurement module / measuring glove in the NED coordinate system during the Tpose motion. Specifically, the distributed MCUs calculate the attitude of each measurement unit in the NED coordinate system, and then transmit the obtained attitudes of each measurement unit in the NED coordinate system to the PC via a router. The attitude of the measurement unit in the NED coordinate system calculated in the i-frame is recorded as...

[0266] S53): When the user makes a T-pose facing due north, the user is on a horizontal surface facing due north, and the relative orientation of the three axes of the human body (upper left and front) and geographic north-east is fixed. The rotation matrix of the SMPL coordinate system and the NED geographic coordinate system is automatically obtained by the PC host computer. for It also possesses the property of an orthogonal matrix;

[0267] S54): Data storage, i.e., the host PC stores the attitude quaternions of each IMU and the inertial acceleration measured by the sensors. Measuring angular velocity All are stored in the cache array;

[0268] S55): User Tpose calibration action judgment. If the change amplitude of the Euler angle detected in the IMU exceeds the preset value (e.g., 10°) during the user Tpose calibration action, it indicates that the user Tpose calibration action has been stopped.

[0269] If significant jitter occurs, the automated calibration is considered a failure.

[0270] S56): When the user performs a Tpose calibration action, if the change amplitude of the Euler angle detected in the IMU is less than the preset value and the set calibration time has not been reached, then return to S54); if the change amplitude of the Euler angle detected in the IMU is less than the preset value and the set calibration time (e.g., 5 seconds) has been reached, then the average attitude quaternion, acceleration and angular velocity are calculated from the cache array, and the average attitude quaternion is converted into a rotation matrix after standardization.

[0271] S57): Obtain the calibration parameters of the accelerometer and gyroscope, that is, calculate the error of acceleration and angular velocity from the cached data group in S54).

[0272] It should be noted that the maximum and minimum Euler angles of the user in the Tpose action differ by less than 10°.

[0273] The T-action described in this embodiment is used to initialize the sensor calibration method of the motion capture system or device, reducing the user's calibration actions, simplifying the calibration process before motion capture by sparse IMU, reducing the probability of calibration errors, and improving the reliability of calibration data and user experience.

[0274] In Example 1, S5) analyzes and compares the rotational attitude and geographic acceleration of the joints corresponding to the deployment positions of each measurement module obtained in S4) with the initial calibration data of the T-motion using the SMPL coordinate system. Specifically, it compares the data obtained in S56).

[0275] The distributed MCU in S52 calculates the attitude of each measurement module within the measurement module / measuring glove in the NED coordinate system. It is necessary to calibrate the error of the IMU relative to each joint. The specific calibration process is as follows:

[0276] The deployment position of the IMU may be misaligned due to muscles, skin, or wearing method and corresponding joints. The error of the IMU relative to the human skeleton is usually calibrated using Tpose / Apose / attention posture.

[0277] The specific errors are as follows:

[0278]

[0279] in, The representation of the i-th frame pose of the skeleton for deploying the measurement unit in the SMPL coordinate system. The representation of the geographic coordinate system in the SMPL coordinate system is as described in S43). It has the property of being an orthogonal matrix, therefore This is the error matrix from the human skeleton to the corresponding IMU.

[0280] During the calibration of the IMU relative to each joint, the poses of each joint in all SMPL coordinate systems are affected when performing the Tpose motion. Both are identity matrices I 3×3 ,

[0281] Right now

[0282] in, t0 is the effective rotation matrix of the IMU body coordinate system in the geographic coordinate system during the TTpose action, t1 is the start time of the TTpose action, and t1 is the end time of the TTpose action.

[0283] This is the error matrix from the human skeleton to the IMU, which is an orthogonal identity matrix. The specific solution process is as follows:

[0284]

[0285] This invention uses a mathematical model with reduced degrees of freedom and a distributed computing architecture. Measurement modules 1-6 are deployed on the left and right forearms near the wrist joints, the left and right lower legs near the knee joints, the back of the head, and the back of the waist, respectively. Their respective MCUs perform attitude calculations for these joints and package the geographic attitude, acceleration, and angular velocity information to send to the router. Two measurement gloves are worn on the left and right hands, respectively. Measurement glove 1 calculates the geographic attitude, acceleration, and angular velocity information of the back of the hand, the proximal phalanx of the thumb, the index, middle, ring, and little fingers, and sends it to the router. The data preprocessing and fusion are all completed in the corresponding microcontroller units. The MCU performs a lot of preprocessing work, reducing the computing load on the PC host computer.

[0286] In this embodiment, S53) involves obtaining the rotation matrix. After the process, the IMU attitude in the geographic coordinate system needs to be transformed to the SMPL coordinate system. The specific calibration process is as follows:

[0287] The SMPL coordinate system (human coordinate system) has xyz axes corresponding to the left-hand sagittal plane normal vector, the upward cross-sectional normal vector, and the forward coronal plane normal vector, which has a rotational deviation from the Northeast Earth (NED) geographic coordinate system.

[0288] The specific deviations are as follows:

[0289]

[0290] Among them, v e For the representation of any vector in a geographic coordinate system, v s This represents any vector in the SMPL coordinate system.

[0291] In this embodiment, due to manufacturing errors, the output acceleration and angular velocity of the IMU deviate from the actual values. For example, in a stationary state, the gyroscope should output a very small reading (containing only the Earth's rotation angular velocity reading, 7.2722×10^-5 rads / s), but the actual output may deviate to 0.05 rads / s. This zero bias will introduce significant errors during long-term integration, resulting in attitude drift.

[0292] In step S57), the calculation of acceleration and angular velocity requires calibration of the IMU and magnetometer errors. This is achieved by using an error model composed of inertial acceleration error, angular velocity error, and magnetometer error to calibrate the IMU and magnetometer. The error model is as follows:

[0293] in, For real inertial acceleration, T is the inertial acceleration measured by the IMU. a K is a tiny rotation matrix for accelerometer correction. a b′ is the accelerometer scale factor. a For accelerometer zero bias, ε a For accelerometer white noise, assume ε a It has zero mean and variance σ a Gaussian process;

[0294] For true angular velocity, T is the angular velocity measured by the IMU. ω K is a tiny rotation matrix for gyroscope correction. ω b′ is the gyroscope scale factor. ω For zero bias of the gyroscope, ε ω This is white noise from the gyroscope.

[0295] In this embodiment, since all IMUs are assumed to be relatively stationary during the Tpose action, the least squares method is used to fit the IMU's zero bias, scale factor, and small rotation matrix. Then, the corrected IMU readings are obtained through correction, resulting in corrected acceleration and corrected angular velocity. The error parameters for IMU acceleration and IMU angular velocity are then calculated.

[0296] The objective function for acceleration is:

[0297] Where g is the local acceleration, obtained by minimizing J a Fit T a K a b' a The T a K a b′ a These are the minute rotation matrix for accelerometer correction, the accelerometer scale factor, and the accelerometer zero bias, respectively.

[0298] The objective function for angular velocity is:

[0299] By minimizing J ω Fit T ω K ω b′ ω To obtain more accurate motion acceleration, the host PC then sends these correction parameters to each MCU through the router to obtain more accurate inertial information;

[0300] Among them, T ω K ω b′ ω These are the micro-rotation matrix for gyroscope correction, the gyroscope scale factor, and the gyroscope zero bias, respectively.

[0301] In this embodiment, the process of calculating the average attitude quaternion of the IMU in S56), and converting the average attitude quaternion into a rotation matrix after standardization, is as follows:

[0302]

[0303] in, For the i-th frame attitude quaternion of the IMU deployed at the specified joint position in the cache array, the attitude quaternion is the unit attitude quaternion rotated from the geographic coordinate system to the IMU body coordinate system;

[0304] The average attitude quaternion of the IMUs deployed at the specified joint positions in the cache array;

[0305] Let q0 be the normalized average attitude quaternion, with its real part being q0 and its three imaginary parts being q1, q2, and q3.

[0306] The effective rotation matrix of the IMU body coordinate system in the geographic coordinate system during the T-action is as described above.

[0307] Example 4

[0308] In this embodiment, as shown Figure 8 As shown, the high-speed motion capture method based on sparse IMU is the same as the high-speed motion capture method based on sparse IMU in Example 1. Furthermore, during the data acquisition process in S1), a data synchronization algorithm is used to ensure that the data from the measurement units in each measurement module of the LSTM neural network are acquired at the same time, improving the accuracy of motion reconstruction. The specific data synchronization algorithm is as follows:

[0309] Configure the PC host computer with specific IP and DNS addresses to avoid the latency caused by dynamic IP allocation;

[0310] Then configure quality of service, disable bandwidth control, and close unnecessary background applications to minimize network latency on the PC and router.

[0311] The PC will allocate a dedicated thread to run the clock script. The clock script will increment the synchronization sequence number sync_id by one every 5 milliseconds. The clock script is configured to have high precision, real-time performance, and stability.

[0312] The clock script runs on a dedicated thread on the PC, independent of other tasks, thus reducing the possibility of resource contention or latency, and ensuring that the synchronization sequence number sync_id is incremented every 5 milliseconds;

[0313] The PC host computer will wait for the measurement module / glove to connect to the router. When the PC host computer detects that a measurement module / glove has connected to the network, the PC host computer sends a sync_id to the measurement module / glove. The measurement module / glove will record this sync_id and localize it. At this time, the localized sync_id is recorded as sync_id1. The measurement module / glove increments the localized sync_id1 by one every 5 milliseconds.

[0314] Once the PC detects that all measurement modules / gloves are connected to the network, it stores the communication data of all measurement modules / gloves in a buffer. Data with consistent sync_id1 to sync_id8 are sent to the LSTM neural network for subsequent algorithm processing. It should be noted that here, the corrected acceleration, angular velocity, magnetometer, and attitude calculation data are sent to the neural network for inference.

[0315] In this embodiment, the use of a data synchronization algorithm, with the PC configured with a specific IP and DNS address, effectively avoids the latency caused by dynamic IP allocation. Configuring Quality of Service (QoS), disabling bandwidth control, and closing unnecessary background applications minimizes network latency between the PC and the router, effectively ensuring that the data from the six measurement modules of the neural network are collected at the same time, which helps improve the accuracy of motion reconstruction.

[0316] The PC host computer is configured with specific IP and DNS addresses. Specific means that the PC host computer is configured with statically predefined IP addresses and DNS server addresses, and Dynamic Host Configuration Protocol (DHCP) is disabled to eliminate IP address negotiation delays. The IP address is configured to a private address range of the local area network (e.g., 192.168.0.0 / 16), and the DNS server address is set to the local router's internal network address or 127.0.0.1.

[0317] Configuring a PC with a specific IP and DNS address can effectively avoid the latency caused by dynamic IP allocation. Configuring Quality of Service (QoS), disabling bandwidth control, and closing unnecessary background applications can minimize network latency for both the PC and the router. This effectively ensures that the data from each measurement module of the neural network processing are collected at the same time, which helps improve the accuracy of motion reconstruction.

[0318] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of the present invention, and these improvements should also be considered within the scope of protection of the present invention.

Claims

1. A high-speed motion capture method based on sparse IMU, characterized in that: Includes the following steps: S1): The IMU and magnetometer in the measurement unit detect data of the corresponding parts of each joint in the human body and send the obtained data to the microcontroller unit; S2): In the microcontroller unit, the original magnetometer reading and the original IMU reading are corrected to obtain several correction parameters, including the corrected magnetometer reading, the corrected acceleration, and the corrected angular velocity; S3): Predict and correct for over-range readings of the IMU accelerometer and gyroscope using an IMU prediction algorithm; S4): Data fusion, which is to fuse the corrected magnetometer readings and the predicted over-range readings of the IMU accelerometer and gyroscope through the AHRS system to obtain the rotational attitude and geographic acceleration of the joints corresponding to the deployment positions of each measurement module. S5): By analyzing and comparing the rotational attitude and geographic acceleration of the joints corresponding to the deployment positions of each measurement module obtained in S4) with the initial calibration data of the T action through the SMPL coordinate system, the rotational attitude of the SMPL coordinate system and the acceleration of each leaf node relative to the root node are obtained. S6): Using a set of LSTM neural networks and an inertial navigation system, the position of each joint relative to the root node is inferred from the rotational attitude of the SMPL coordinate system obtained in S5) and the acceleration of the leaf node relative to the root node. The rotational attitude of each joint and the velocity of the root node are estimated, thereby obtaining accurate human motion.

2. The high-speed motion capture method based on sparse IMU according to claim 1, characterized in that: In S3), the over-range readings of the IMU accelerometer and gyroscope are predicted and corrected using the IMU prediction algorithm. Since the IMU accelerometer and gyroscope contain data in three axes, an LSTM neural network is designed for each axis of the IMU accelerometer and gyroscope to ensure synchronous prediction and compensation of multi-dimensional data. Taking the angular velocity measured by a gyroscope as an example, the specific process is as follows: S31): Data buffering and saturation detection: First, the corrected IMU readings in S2) are stored in the PC's cache for subsequent analysis and prediction. When the gyroscope readings reach the upper and lower limits of its range, the IMU prediction algorithm identifies that the IMU has entered a saturation state. When the gyroscope data is saturated, the saturation counter tracks the duration of the IMU reading in the saturated state. When the data tracked by the saturation counter value exceeds the set threshold N, the IMU prediction algorithm determines that there are too many over-range frames and fails to predict. This state is not normal high-speed motion, but rather an IMU malfunction. At this time, the prediction algorithm will stop to prevent the further propagation of erroneous data. When the data tracked by the saturation counter value is less than the set threshold N, the following prediction steps will be performed. S32): Prediction and compensation using LSTM neural networks: First, a prediction model based on an LSTM neural network is constructed. During the operation, high-range IMU data is used as the training reference for the LSTM neural network. The LSTM neural network is trained through supervised learning, and the LSTM neural network predicts the out-of-range data based on the cached IMU data. When the gyroscope reading returns to the range, the saturation counter will automatically reset to zero, indicating that the IMU has returned to normal. S33): Regression Compensation: If the user prioritizes the real-time performance of high-speed motion capture, the IMU prediction algorithm will cancel regression compensation; if the user prioritizes the accuracy of motion capture, the IMU prediction algorithm will add a regression compensation algorithm to further optimize the accuracy of the prediction.

3. The high-speed motion capture method based on sparse IMU according to claim 2, characterized in that: The specific formula for the regression compensation algorithm is as follows: in, This refers to the frame number after the system enters saturation. The integer is less than the maximum value m of the saturation counter. For the revised first Frame prediction angular velocity, For the first time before the correction Frame prediction angular velocity, Angular velocity was measured in the first frame after the system returned to normal. The predicted angular velocity data for the m-th frame before correction is the predicted angular velocity of the last frame in the saturation state.

4. The high-speed motion capture method based on sparse IMU according to claim 3, characterized in that: The prediction model based on the LSTM neural network includes a five-layer structure, consisting of one dropout layer, one linear input layer, two LSTM hidden layers, and one linear output layer. Each LSTM hidden layer contains 64 neurons to capture the complex temporal relationships of motion data; the input to the dropout layer is the IMU data of the past K frames in the buffer, and the output of the linear output layer is the prediction data of L frames. During the work process, the motion data of the IMU over-range portion is modeled and learned by the LSTM neural network. The LSTM neural network is trained to predict the waveform of motion data in the IMU over-range range. High-range IMU data is used as training reference, and the LSTM neural network is trained through supervised learning. The five-layer structure of the prediction model based on the LSTM neural network operates as follows: The input K-frame IMU data first passes through a dropout layer, where some features are randomly zeroed while maintaining the shape of the input. This helps prevent the model from overfitting and enhances the model's generalization ability. The output of the dropout layer is fed into the linear input layer, which transforms the dimension of the input features into a vector of length 64 through a linear mapping to match the input of the subsequent LSTM hidden layer. This can compress the input features and perform feature space transformation. The output of the linear input layer is fed into the first LSTM hidden layer, which captures the short-term temporal relationships of the motion data, i.e., models the temporal dependence of the input sequence; the second LSTM hidden layer further extracts higher-order temporal features, enhancing the model's understanding of complex temporal relationships. The outputs of the two LSTM hidden layers eventually enter the linear output layer. The linear output layer maps the output of the LSTM hidden state to the target prediction sequence. It transforms the high-dimensional hidden state into a prediction data sequence of length L frames, providing motion prediction results for each frame.

5. The high-speed motion capture method based on sparse IMU according to claim 1, characterized in that: S2) In the microcontroller unit, the original magnetometer reading and the original IMU reading are corrected to obtain several correction parameters, specifically: The zero bias, scale factor, and non-orthogonal error of the magnetometer were observed in real time using UKF, and the original magnetometer readings were corrected to obtain the corrected magnetometer. The raw IMU readings are fitted to the IMU's zero bias, scale factor, and small rotation matrix using the least squares method through the microcontroller units in each measurement module / glove. Then, the corrected IMU readings, namely the corrected acceleration and corrected angular velocity, are obtained through correction.

6. The high-speed motion capture method based on sparse IMU according to claim 2, characterized in that: The UKF real-time compensation method is used to compensate for the magnetometer's zero bias, scaling factor, and non-orthogonality errors, and the original magnetometer reading is corrected to obtain the corrected magnetometer reading. The specific correction process is as follows: The magnetometer measurement can be modeled as follows: in, It is the measurement value of the magnetic field by the magnetometer at time k. These are the corresponding values ​​of the geomagnetic field in relation to the Earth's fixed coordinate system. It is the unknown attitude matrix of the magnetometer relative to the geographic coordinate system. Let D be a 3×3 identity matrix, and let D be the matrix of the magnetometer's scale factor error and non-orthogonal row error. It is the zero bias vector of the magnetometer. It is a measurement noise vector, assuming the measurement noise vector has a mean of zero and a covariance of... Gaussian process; The measurement equation is: Expand and organize Measurement equation: Among them, the definition Error term for the noise component: It is approximately Gaussian noise, so its mean is... and variance for: Where E{} is the expectation of the variable, and Tr() is the trace of the variable. It is a function of the magnetometer's scaling factor error and non-orthogonal row error matrix D and the magnetometer's zero bias vector b. Then, the variables are iteratively optimized using UKF. The prediction and correction steps follow the standard unscented Kalman filter method to obtain more accurate magnetometer readings.

7. The high-speed motion capture method based on sparse IMU according to claim 1, characterized in that: The LSTM neural network uses the AMASS dataset as training data. By using interpolation technology, the frame rate of the AMASS dataset is uniformly increased to 500Hz. The optical label data used in the AMASS dataset is equivalently converted into IMU data, and then the equivalently converted IMU data is used as input for supervised training. The specific process of converting the optically labeled data used in the AMASS dataset into IMU data is as follows: The rotation matrices of each leaf node joint in the AMASS dataset are selected, specifically the rotation matrices corresponding to the rotation postures of the left and right ankles, left and right wrists, pelvis, and head. Then, the rotation matrices in the SMPL coordinate system are obtained through forward kinematics. In addition, the AMASS dataset also includes spatial location data of the human body mesh. The acceleration information of the leaf node positions is obtained by taking the second derivative of the spatial location data of the human body mesh.

8. The high-speed motion capture method based on sparse IMU according to claim 1, characterized in that: In S6), a set of LSTM neural networks and an inertial navigation system infer the position of each joint relative to the root node based on the rotation attitude of the SMPL coordinate system obtained in S5) and the acceleration of the leaf node relative to the root node, and estimate the rotation attitude of each joint and the velocity of the root node. The LSTM neural network has five branches, three of which are used to estimate the pose of human joints, and the other two are used to estimate the velocity of the human body in the SMPL coordinate system. The specific allocation is as follows: One of the components, an LSTM neural network, obtains the acceleration and rotational attitude of the IMU from the measurement module to estimate the position of each joint of the human body relative to the root node. Then, using the positions of each joint of the human body relative to the root node and IMU information as input, an LSTM neural network is used to infer the positions of other joints of the human body relative to the root node, except for the finger joints. Finally, using the positions of all the joints in the human body and the measurement information from the IMU as input, an LSTM neural network is used to estimate the rotational orientation of the other joints relative to the root node. The rotation attitude of the root node is the IMU solution attitude transformed into the SMPL coordinate system.

9. A high-speed motion capture method based on sparse IMU according to claim 8, characterized in that: The root node velocity is constrained by physical motion during processing, specifically as follows: Using the acceleration and rotation of the SMPL coordinate system obtained from the IMU measurements of the six measurement modules as input, an LSTM neural network is used to estimate the probability of whether the left and right sides will touch the ground. If the probability of touching the ground is high, the foot will not slide relative to the ground, and the velocity of the root node is calculated using forward dynamics. If the probability of touching the ground is low, an LSTM neural network is used to directly predict the velocity of the root node from the estimated joint velocities (excluding the fingers) and the acceleration information of the six IMUs.

Images