Construction method of human motion digital twinning based on inertial motion capture technology

Abstract

The application discloses a kind of human motion digital twin construction methods based on inertial action capture technology.Firstly, action capture device is worn on human body, and the original sensing data of each joint corresponding sensor is collected, the original sensing data is processed using adaptive extended Kalman filter algorithm, the attitude of each joint corresponding sensor is obtained, and the original sensing data and sensor attitude data are sent to host computer;Host computer constructs human motion digital twin model according to human skeleton;The attitude calibration matrix of each joint is calculated using human posture calibration method, and the real-time sensor attitude data is calibrated to obtain the attitude of each joint of human body, drive human motion digital twin model;While driving the model, the joint parameters calculated are compared with human joint parameter threshold, and the evolution model is dynamically updated.The application can realize that human motion digital twin model is synchronously mapped real human motion, and model is updated in real time dynamically.

Description

Technical Field

[0001] This invention relates to a method for constructing human motion digital twins in the fields of electronic information engineering and computer science, and in particular, it designs a method for constructing human motion digital twins based on inertial motion capture technology. Background Technology

[0002] In the industrial sector, digital twins construct virtual models of physical entities in cyberspace, using real-time updated data and historical state data to simulate the behavior of physical entities in real space. This allows for the monitoring, diagnosis, prediction, and control of the working processes and states of physical entities in real-world environments. With the advancement and development of key enabling technologies, digital twins are demonstrating enormous potential in the continuous interaction and connectivity between physical and network systems, and are gradually penetrating other fields.

[0003] Humans are not only the most proactive and dynamic key factor in manufacturing, production, and service applications, but also the ultimate service target of all activities. "Human-centered" manufacturing, production, and service applications are gradually attracting widespread attention from academia and industry. Currently, traditional research on digital twin systems mainly focuses on mechanical and physical systems, lacking attention and research on the "human" element.

[0004] Unlike traditional digital twin systems, human digital twins aim to construct a digital twin of the human body from multiple dimensions, including genes, physiology, and behavior, dynamically tracking changes in the human body in the physical world. This provides guidance for monitoring and analyzing the human body and realizing human-computer interaction. Current research on human digital twins largely focuses on genetics and biology, simulating organs or microstructures within the human body, such as cardiovascular and cardiac digital twins, to provide personalized medical services. However, research on the physiological and behavioral states of the human body is lacking. Human motion digital twins, on the other hand, characterize the human body from the perspectives of physiological behavior and human factors engineering. They aim to acquire the human body's behavioral and motion states in physical space through data acquisition devices, mapping and tracking the human body's motion in the information space, analyzing joint parameters, and enabling real-time dynamic evolution and updates of the digital twin model. Summary of the Invention

[0005] To address the problems and needs existing in the background technology, this invention provides a method for constructing a human motion digital twin based on inertial motion capture technology, and the technical solution adopted is as follows:

[0006] 1) Wear a multi-node inertial motion capture device on the human body and use the multi-node inertial motion capture device to collect the raw sensing data of the sensors corresponding to each joint of the human body. After processing the raw sensing data of the sensors corresponding to each joint using the adaptive extended Kalman filter algorithm, obtain the sensor posture of the sensors corresponding to each joint of the human body. Use the raw sensing data of the sensors corresponding to each joint and the corresponding sensor posture as the sensing data of each joint and wirelessly send them to the host computer.

[0007] 2) Based on the sensor data of each joint, the installation position of the multi-node inertial motion capture device, and the skeletal structure of the human body, the host computer constructs a digital twin model of human motion; based on the sensor data of each joint, the posture calibration matrix of each joint is calculated using the human posture calibration method, and then the posture calibration matrix of each joint is used to perform joint calibration on the sensor posture of the corresponding joint to obtain the posture of each joint. The posture of each joint is composed of the posture of the human body posture, and finally the human motion digital twin model in the host computer is driven according to the human body posture.

[0008] 3) The multi-node inertial motion capture device collects and sends real-time sensing data of each joint to the host computer in real time. Based on the real-time posture calibration matrix of each joint and the real-time sensing data, the human joint parameters are calculated. The joint parameters are compared with the corresponding parameter thresholds. Based on the comparison results, the human motion digital twin model is iteratively evolved to realize the dynamic update of the human motion digital twin model.

[0009] In step 1), the sensors of each sub-node in the multi-node inertial motion capture device collect the angular velocity, acceleration and magnetic field strength of the sensor corresponding to the human joint, and the angular velocity, acceleration and magnetic field strength constitute the original sensing data of the sensor corresponding to the current joint.

[0010] In step 1), the raw sensor data of each joint's corresponding sensor is processed using the adaptive extended Kalman filter algorithm to obtain the sensor posture of each joint of the human body, specifically:

[0011] S1: Determine the acceleration-magnetic field strength parameters based on the original sensing data of the sensors corresponding to each joint. Construct the error function and cost function between the geographic coordinate system and the sensor coordinate system of the current joint based on the acceleration-magnetic field strength parameters of the current joint. Solve the error function and cost function of the current joint using the Gauss-Newton algorithm to obtain the attitude quaternion between the geographic coordinate system and the sensor coordinate system of the current joint. Use the attitude quaternion between the geographic coordinate system and the sensor coordinate system corresponding to the current joint as the initial attitude of the sensor corresponding to the current joint.

[0012] S2: Use the initial sensor attitude and angular velocity of the current joint as the state variables of the adaptive extended Kalman filter and solve them to obtain the sensor filtered attitude of the current joint.

[0013] S3: Calculate the distance between the sensor-filtered posture of the current joint at the current moment and the sensor-filtered posture at the previous moment, and the distance between the negative of the sensor-filtered posture of the current joint at the current moment and the sensor-filtered posture at the previous moment. Take the sensor-filtered posture corresponding to the smallest of the two distances as the sensor posture of the current joint at the current moment.

[0014] S4: Repeat S1-S3 to process the raw sensor data of the remaining joints and obtain the sensor posture of each joint of the human body.

[0015] The adaptive extended Kalman filter is obtained by improving the measurement noise mean and measurement noise variance matrix in the extended Kalman filter algorithm. A forgetting factor and a forgetting factor weighting coefficient are introduced to dynamically update the measurement noise. The formulas for the measurement noise mean and measurement noise variance matrix are as follows:

[0016]

[0017]

[0018]

[0019]

[0020] Where I represents the identity matrix, This represents the estimated mean value of the measurement noise at the k-th discrete time. This represents the forgetting factor weighting coefficient of the mean noise of the k-th discrete-time measurement. This represents the forgetting factor weighting coefficient of the mean noise of the (k-1)th discrete-time measurement. The initial value of the forgetting factor weighting coefficient representing the mean of measurement noise. Z represents the mean estimate of the measurement noise at the (k-1)th discrete time. k This represents the discretized extended Kalman filter observations at the k-th discrete time. H represents the state prediction value in the extended Kalman filter, and H represents the measurement array in the extended Kalman filter. Let represent the estimated value of the measurement noise variance matrix at the k-th discrete time. This represents the estimated value of the measurement noise variance matrix at the (k-1)th discrete time. This represents the forgetting factor weighting coefficient of the variance matrix of the k-th discrete-time measurement noise. This represents the forgetting factor weighting coefficient of the (k-1)th discrete-time measurement noise variance matrix. e represents the initial value of the forgetting factor weighting coefficient of the measurement noise variance matrix. k-i and e k P represents the measurement estimation error at the (k-1)th and kth discrete times in the extended Kalman filter, respectively. k / k-1 This represents the mean square error of the one-step prediction at the k-th discrete time, calculated from the mean square error of the estimated mean square error at the (k-1)-th discrete time in the extended Kalman filter. b represents the forgetting factor, N represents the discrete time length, T represents the matrix transpose, e represents the natural constant, Tr() represents the trace of the matrix, c1, c2, and c3 represent the first, second, and third coefficients, and || represents the absolute value operation.

[0021] In step 2), the posture calibration matrix of each joint is calculated using a human posture calibration method, specifically as follows:

[0022] First, the multi-node inertial motion capture device collects sensor data of the human body in three calibration postures. Then, it calculates the initial posture calibration matrix corresponding to each joint based on the sensor data in the three calibration postures. Next, it uses the initial posture calibration matrix of each joint to perform joint calibration on the sensor posture of the corresponding joint to obtain the posture of each joint. Finally, based on the attributes of the three calibration postures and the posture of each joint, it corrects the initial posture calibration matrix corresponding to each joint to obtain the posture calibration matrix corresponding to each joint.

[0023] The initial posture calibration matrix corresponding to each joint is composed of the X-axis vector, Y-axis vector, and Z-axis vector of the body coordinate system. The formula for calculating the X-axis vector is as follows:

[0024] X' BCS =[acc' x acc' y acc' z ]

[0025] The formula for calculating the Y-axis measurement is as follows:

[0026]

[0027] Y” BCS =PCA([gyr') x ,gyr' y ,gyr' z ])

[0028]

[0029] The formula for calculating the Z-axis dimension is as follows:

[0030]

[0031]

[0032] Z”' BCS =PCA([gyr”) x ,gyr” y ,gyr” z ])

[0033] Where X' BCS Y' represents the first X-axis vector of the body coordinate system. BCS 、Y” BCS 、Y”' BCS These represent the first, second, and third Y-axis vectors of the body coordinate system, respectively, and Z' BCS Z” BCS Z”' BCS Let acc' represent the first, second, and third Z-axis vectors of the body coordinate system, respectively. x ,acc' y ,acc' z These represent the components of the acceleration measured by the sensors at each joint of the human body in the first calibration posture, along the XYZ axes of its own sensor coordinate system; acc” x ,acc” y ,acc” z These represent the components of the acceleration measured by the sensors at each joint of the human body in the second calibration posture, along the X, Y, and Z axes of its own sensor coordinate system; norm() represents normalization; |||| represents Euclidean norm calculation; PCA() represents principal component analysis, gyr' x ,gyr' y ,gyr' z These represent the components of the angular velocity measured by the sensors at each joint of the human body in the second calibration posture, along the XYZ axes of its own sensor coordinate system; gyr” x ,gyr” y ,gyr” z These represent the components of the angular velocity measured by the sensors at each joint of the human body in the third calibration posture, along the XYZ axes of its own sensor coordinate system.

[0034] The posture calibration matrices for the hip joint, spine, head, and left and right thighs are calculated in the same way, consisting of the first X-axis vector, first Y-axis vector, and first Z-axis vector of the body coordinate system; the posture calibration matrices for the left and right lower legs and left and right feet are calculated in the same way, consisting of the first X-axis vector, second Y-axis vector, and second Z-axis vector of the body coordinate system; the posture calibration matrices for the left and right upper arms, left and right forearms, and left and right hands are calculated in the same way, consisting of the first X-axis vector, third Y-axis vector, and third Z-axis vector of the body coordinate system.

[0035] The initial posture calibration matrix for each joint is corrected based on the attributes of the three calibration postures and the initial posture of each joint to obtain the posture calibration matrix for each joint. Specifically:

[0036] Based on sensor data of the human body in the first calibration posture, the joint angles of the left and right elbow joints, spine, and left and right knee joints are calculated. Based on the calculated joint angles of the left and right elbow joints, spine, and left and right knee joints, the initial posture calibration matrices of the left and right upper arms and left and right forearms, head and hip joints, and left and right thighs and left and right calves are corrected according to the attributes of the first calibration posture. The specific formulas are as follows:

[0037] proximalCor = vec2mat(acc proximal (gravityVec)

[0038] proximalRotM + =proximalCor*proximalRotM -

[0039] Here, proximalCor is the relative rotation matrix between the acceleration vector and the gravity vector of each joint, vec2mat() converts the vector into matrix form, and acc proximal Represents the acceleration vector of each joint, gravityVec represents the gravity vector of each joint; proximalRotM - The initial pose calibration matrix for each joint, proximalRotM + The corrected posture calibration matrix for each joint;

[0040] Next, the degrees of freedom of the knee and shoulder joints are calculated based on the sensor data of the human body in the second and third calibration postures. Based on the degrees of freedom of the knee and shoulder joints, and according to the attributes of the second and third calibration postures and the initial postures of the left and right upper arms, left and right forearms, left and right thighs and left and right lower legs, the corrected posture calibration matrices of the left and right upper arms, left and right forearms, left and right thighs and left and right lower legs are compensated to obtain the final posture calibration matrix of the corresponding joints. The specific calculation formula is as follows:

[0041] angleOffset=(computeAngle(proximalQua,distalQua)) / 2,

[0042] proximalRotM +' =ang2mat(-angleoffset)*proximalRotM +

[0043] Here, `angleOffset` represents the error in the joint angles of the shoulder joint's flexion / extension, horizontal adduction / abduction, or internal / external rotation; `proximalQua` represents the initial posture of the left and right upper arms or left and right thighs; `distalQua` represents the initial posture of the left and right forearms or left and right lower legs; `computeAngle()` converts the posture to angle form; `ang2mat()` converts the angle to matrix form; and `proximalRotM`... +' The final attitude calibration matrix for each joint.

[0044] The three calibration postures are the first calibration posture, the second calibration posture, and the third calibration posture, respectively.

[0045] First calibration posture: Stand upright with your torso straight, arms hanging naturally at your sides with palms facing your body, feet shoulder-width apart with the soles of your feet parallel and facing forward.

[0046] Second calibration posture: Maintain a seated posture, lean your upper limbs and torso forward, place your elbows and forearms flat on the table, with the distance between your elbows the same as your shoulders, keep your forearms parallel, place your thighs flat on the chair, keep them parallel to the ground, and swing your left and right lower legs back and forth around your knees at the same frequency.

[0047] Third calibration posture: Maintain a seated posture, raise both arms vertically downwards to shoulder height on both sides of the body, palms facing the ground, upper limbs in a "T" shape, thighs flat on the chair, parallel to the ground, lower legs at a 90° angle to the ground, feet shoulder-width apart, soles parallel to each other and facing forward.

[0048] In step 3), while driving the human motion digital twin model in real time, the host computer calculates the human joint parameters and compares the joint parameters with the corresponding parameter threshold range. If the comparison result exceeds the corresponding threshold range, the host computer prompts the human to stop moving and reminds the user to check whether the current motion state of the real human body is consistent with the normal physiological motion state. If the current motion state of the real human body is normal, the human motion digital twin model is dynamically updated, prompting the human body to execute three calibration postures, updating the posture calibration matrix of each joint of the human body, and re-driving the human motion digital twin model, thereby realizing the dynamic evolution of the human motion digital twin model.

[0049] The beneficial effects of this invention are as follows:

[0050] This invention provides a method for constructing a human motion digital twin. It employs an adaptive extended Kalman filter algorithm to dynamically update measurement noise parameters and fuses inertial sensor data to calculate sensor attitude, improving attitude calculation accuracy. By combining attitude calibration posture and joint angle attributes, a human attitude calibration matrix is ​​obtained, reducing errors caused by inaccurate calibration posture or sensor wear position offset. The calculated human joint attitude synchronously drives and dynamically updates the human digital model, enabling the simultaneous evolution and updating of the human motion digital twin model while mapping human motion to the digital model. This ensures both the stable and correct operation of the digital model and its dynamic updating. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the algorithm flow of the present invention;

[0052] Figure 2 This is a schematic diagram of the hardware framework of the present invention;

[0053] Figure 3 This is a flowchart illustrating the iterative evolution of the human motion digital twin model of the present invention.

[0054] Figure 4 This invention refers to a digital twin model of a person wearing an inertial motion capture device and their human motion. Detailed Implementation

[0055] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to facilitate understanding of the invention by those skilled in the art. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various modifications within the spirit and scope of the invention as defined and determined by the appended claims are obvious, and all inventions utilizing the concept of the present invention are within the scope of protection. In the detailed description of the embodiments of the present invention, the schematic diagrams are merely examples for ease of explanation and should not be construed as limiting the scope of protection of the present invention.

[0056] This invention discloses a human motion digital twin system, primarily comprising hardware and software. An inertial motion capture device collects and processes sensor data via inertial sensor sub-nodes fixed to the head, upper arms, forearms, spine, hip joints, thighs, calves, and feet, and sends this data to a host computer. Before receiving the sensor data, the host computer first constructs a human digital model based on the human skeletal structure using a computer rendering engine. During the human posture calibration phase, the host computer calculates the posture calibration matrix of the human joints based on the motion data of the calibration posture and stores it on the host computer. Then, it uses the posture calibration matrix to calibrate the sensor posture matrix received from the sub-nodes, calculates the real-time posture matrix of the human joints, and drives the human motion digital twin model in real time. Finally, while driving the model in real time, the host computer calculates human joint parameters based on the real-time posture matrix and sensor data, compares and judges whether the current state of the human motion digital twin model is normal by considering the threshold range of the parameters, iteratively evolves the human digital model, and dynamically updates the human digital model.

[0057] Figure 1 The flowchart for the implementation of the digital twin system construction method for human motion provided in the embodiments of the present invention is as follows: Figure 1 As shown, the present invention includes the following steps:

[0058] Multi-node inertial motion capture equipment is used to collect sensor data corresponding to human joints. This data is then transmitted wirelessly to a host computer. The host computer constructs a human digital model and uses the calculated human joint posture data to drive the model. Specifically:

[0059] 1) Wear a multi-node inertial motion capture device on the human body and use the multi-node inertial motion capture device to collect the raw sensing data of the sensors corresponding to each joint of the human body. After processing the raw sensing data of the sensors corresponding to each joint using the adaptive extended Kalman filter algorithm, obtain the sensor posture of the sensors corresponding to each joint of the human body. Use the raw sensing data of the sensors corresponding to each joint and the corresponding sensor posture as the sensing data of each joint and wirelessly send them to the host computer via TCP protocol.

[0060] In practice, the multi-node inertial motion capture device includes a master node and multiple sub-nodes. The master node is set at the hip joint, and each sub-node corresponds to a sensor of each human joint. The master node issues commands to the sub-nodes to realize functions such as data on / off reception, node restart, and specific data reception. The sensor data of each joint corresponding to the sensor is encapsulated into sensor data frames and sent to the master node. The master node then transmits the aggregated sensor data of all sub-nodes to the host computer via wireless communication.

[0061] In step 1), the gyroscope, accelerometer and magnetometer in the sensors of each sub-node of the multi-node inertial motion capture device collect the angular velocity, acceleration and magnetic field strength of the sensor corresponding to the human joint, and the angular velocity, acceleration and magnetic field strength constitute the original sensing data of the sensor corresponding to the current joint.

[0062] In step 1), the raw sensor data of each joint's corresponding sensor is processed using the adaptive extended Kalman filter algorithm to obtain the sensor pose of each joint of the human body, specifically:

[0063] S1: Determine the acceleration-magnetic field strength parameter based on the raw sensing data of the corresponding sensor for each joint. The acceleration-magnetic field strength parameter is denoted as z. mea , satisfying z mea =[acc x acc y acc z mag x mag y mag z ], where acc x acc y acc z These are the components of the acceleration measured by the sensors corresponding to each joint along the XYZ axes of its own sensor coordinate system, mag. x mag y mag z These represent the XYZ axes of the magnetic field strength measured by the sensors corresponding to each joint, along their own sensor coordinate system. Based on the acceleration-magnetic field strength parameters of the current joint, an error function and a cost function are constructed between the geographic coordinate system and the sensor coordinate system of the current joint. The Gauss-Newton algorithm is used to solve the error function and cost function of the current joint to obtain the attitude quaternion between the geographic coordinate system and the sensor coordinate system of the current joint. This attitude quaternion is then used as the initial attitude of the sensor corresponding to the current joint.

[0064] In this invention, quaternions are used to represent the sensor attitude of each joint, where g represents gravitational acceleration and m... ex m ey m ez Let z represent the components of Earth's magnetic field strength along the X, Y, Z axes of the geographic coordinate system. Then, the Earth's gravitational acceleration-magnetic field strength parameter z in the geographic coordinate system... earth Represented as:

[0065] z earth =[0 0g m ex m ey m ez ]

[0066] The components of acceleration and magnetic field strength of each joint measured by accelerometers and magnetometers along the XYZ axes of its own sensor coordinate system are respectively acc x acc y acc z and mag x mag y mag z The acceleration-magnetic field strength parameter z corresponding to the current joint mea The formula is as follows:

[0067] z mea =[acc x acc y acc z mag x mag y mag z ]

[0068] Based on the current acceleration-magnetic field strength parameter z of the joint mea The error function ε(q) and the cost function f(q) are constructed as follows:

[0069]

[0070]

[0071] Where q is the attitude quaternion between the geographic coordinate system and the sensor coordinate system of the current joint, and R(q) represents the attitude rotation matrix corresponding to the attitude quaternion q. This represents quaternion multiplication, and T represents matrix transpose.

[0072] The Gauss-Newton algorithm is then used to calculate and obtain the attitude quaternion between the geographic coordinate system and the sensor coordinate system.

[0073] S2: Use the initial sensor attitude and angular velocity of the current joint as the state variables of the adaptive extended Kalman filter and solve them to obtain the sensor filtered attitude of the current joint.

[0074] The adaptive extended Kalman filter is obtained by improving the measurement noise mean and measurement noise variance matrix in the extended Kalman filter algorithm. It introduces a forgetting factor and a forgetting factor weighting coefficient to dynamically update the measurement noise. The formulas for the measurement noise mean and measurement noise variance matrix are as follows:

[0075]

[0076]

[0077]

[0078]

[0079] Where I represents the identity matrix, This represents the estimated mean value of the measurement noise at the k-th discrete time. This represents the forgetting factor weighting coefficient of the mean noise of the k-th discrete-time measurement. This represents the forgetting factor weighting coefficient of the mean noise of the (k-1)th discrete-time measurement. The initial value of the forgetting factor weighting coefficient representing the mean of measurement noise. Z represents the mean estimate of the measurement noise at the (k-1)th discrete time. k This represents the discretized extended Kalman filter observations at the k-th discrete time. H represents the state prediction value in the extended Kalman filter, and H represents the measurement array in the extended Kalman filter. Let represent the estimated value of the measurement noise variance matrix at the k-th discrete time. This represents the estimated value of the measurement noise variance matrix at the (k-1)th discrete time. This represents the forgetting factor weighting coefficient of the variance matrix of the k-th discrete-time measurement noise. This represents the forgetting factor weighting coefficient of the (k-1)th discrete-time measurement noise variance matrix. e represents the initial value of the forgetting factor weighting coefficient of the measurement noise variance matrix. k-i and e k P represents the measurement estimation error at the (k-1)th and kth discrete times in the extended Kalman filter, respectively. k / k-1 This represents the mean square error of the one-step prediction at the k-th discrete time, calculated from the mean square error of the estimated mean square error at the (k-1)-th discrete time in the extended Kalman filter. b represents the forgetting factor, N represents the discrete time length, T represents the matrix transpose, e represents the natural constant, Tr() represents the trace of the matrix, c1, c2, and c3 represent the first, second, and third coefficients, which are constants, and || represents the absolute value operation.

[0080] Specifically:

[0081] Based on acceleration and magnetic field strength, this invention estimates the initial sensor attitude between the geographic coordinate system and the sensor coordinate system. Combining the measured angular velocity, the initial sensor attitude and angular velocity are used as state variables to construct an adaptive extended Kalman filter.

[0082] System state equations: The system's state vector X(t) and its derivative are constructed using the angular velocity measured at time t and the attitude quaternion calculated based on the Gauss-Newton algorithm.

[0083]

[0084] Where, ω=[ω x ω y ω z ], q=[q0 q1 q2 q3], w, τ=[τ x τ y τ z The numbers ] represent angular velocity, quaternion, Gaussian white noise, and model parameters, respectively. This represents quaternion multiplication. In this invention, the initial value of the angular velocity in the state vector is [0.001 0.001 0.001], and the initial value of the quaternion is the initial value of the quaternion after iterative update by the Gauss-Newton algorithm. The model parameter τ = [0.04 0.04 0.04].

[0085] To linearize and discretize the aforementioned nonlinear system, the state vector is first linearized around its optimal state estimate. Therefore, the true trajectory X(t) and the nominal trajectory... The deviation δX(t) is:

[0086]

[0087] The state equation of a nonlinear physical system is estimated at its optimal state. Expanding the surrounding area into a Taylor series yields the corresponding deviation function. for:

[0088]

[0089]

[0090] Δt represents the discretized sampling time interval. Indicates the nominal trajectory angular velocity at the bottom, Indicates the nominal trajectory The quaternion under the given name, where e represents the natural constant,

[0091] The above equation is then discretized.

[0092] Observation equation: Using angular velocity and quaternions as observations, construct the observation Z(t) of the extended Kalman filter, and discretize it into Zt. k ,

[0093] Z k =HX k +v k

[0094] Where ω(t) is the angular velocity at time t, q(t) is the quaternion at time t, and in the specific implementation, H is the identity matrix, Xk Let v be the state variable at discrete time k. k The measurement noise is at discrete time k.

[0095] State parameter initialization and estimation: Based on the determined system state equations and observation equations, obtain the one-step prediction mean square error P of the filter estimate. k / k-1 and its estimated mean square error P k for,

[0096]

[0097] Where, Φ k,k-1 Let Q be the state transition matrix from the (k-1)th discrete time to the kth discrete time. k-1 Let I be the variance matrix of the system noise at the (k-1)th discrete time, and let H be the identity matrix. k Let K be the measurement array at the k-th discrete time. k R is the filter gain at the k-th discrete time. k Let P0 be the variance matrix of the measurement noise at the k-th discrete time, and T denote the transpose. The initial value P0 is a symmetric and positive definite matrix, and its initial value is set as follows:

[0098] P0=diag(0.0001 0.0001 0.0001 0.001 0.001 0.001 0.001)

[0099] Extended Kalman filter gain K k for:

[0100]

[0101] The system noise parameters of each sensor sub-node are relatively stable. These parameters are determined through modeling.

[0102]

[0103] Where M is the sampling period, A2 is a 3×4 zero matrix, A3 is a 4×3 zero matrix, A4 is a 4×4 zero matrix, and system noise is... The mean parameter q is initialized as a 7×1 zero matrix.

[0104] Measurement noise is mainly caused by the external environment and is prone to change. Therefore, an improved adaptive algorithm is used to process it, and the initial value of the measurement noise covariance is set.

[0105]

[0106] The mean parameter r is initialized as a 7×1 zero matrix.

[0107] Since model errors in stochastic systems can affect the output, noise parameters are estimated in real time through measurement output while performing state estimation. Because system noise depends on the sensing system itself and its parameters are relatively stable, these parameters are determined through modeling. Measurement noise, however, is mainly caused by the external environment and is prone to change. Therefore, this invention employs an improved adaptive algorithm to process measurement noise. Based on the aforementioned filtering, the state quantity is predicted in one step at the k-th discrete time. Replace the real state quantity X at the k-th discrete time. k This causes an estimation error e in the measurement. k for:

[0108]

[0109] Among them, Z k For the system quantity measurement at the k-th discrete time, The measurement is an estimate of the k-th discrete time based on the predicted state variables at the (k-1)-th discrete time.

[0110] Considering that the actual noise parameters are in a state of slow change, an exponential decaying memory weighted average algorithm is adopted for the noise parameters, introducing a forgetting factor b and a forgetting factor weighting coefficient, and targeting the mean r of the measured noise. k The variance matrix R of the measurement noise k Using a specific forgetting factor, r k The corresponding weighting coefficient for the forgetting factor is R k The corresponding weighting coefficient for the forgetting factor is

[0111]

[0112] in, This represents the mean estimate of the measurement noise at discrete times k and k-1.

[0113] because With Z k It is closely related to measurement information and is easily affected by it; therefore, this algorithm uses smoothing estimation for it.

[0114]

[0115] in, These are the variance matrix estimates of the measurement noise at discrete times k and k-1.

[0116] The value of b determines the different weight allocations. The smaller the value of b, the greater the weight of recent data and the smaller the weight of older data. To capture the attitude of the inertial sensing unit under different motion states (high-speed and low-speed motion), this algorithm samples an adaptive forgetting factor to estimate measurement noise under different dynamic environments. The covariance obtained through the innovation is compared with the theoretical covariance to determine the occurrence of outliers and the stability of noise estimation.

[0117]

[0118] To ensure the stability of the filter, the forgetting factor b should have a larger value when the measurement noise is stable, and a smaller value when the measurement noise is unstable. c3 is a constant. Therefore, this invention sets the adaptive forgetting factor as follows:

[0119]

[0120] The values ​​of c1 and c2 can be dynamically adjusted.

[0121] S3: Quaternions and their opposites represent the same attitude information; that is, [q0 q1 q2 q3] and [-q0 -q1 -q2-q3] represent the same attitude. To smooth the transformation trajectory between quaternions at any two adjacent time points, calculate the distance between the sensor-filtered attitude of the current joint at the current time point and its sensor-filtered attitude at the previous time point, and the distance between the opposite of the sensor-filtered attitude of the current joint at the current time point and its sensor-filtered attitude at the previous time point. The sensor-filtered attitude corresponding to the smallest of the two distances is taken as the sensor attitude of the current joint at the current time point.

[0122] S4: Repeat S1-S3 to process the raw sensor data of the remaining joints and obtain the sensor posture of each joint of the human body.

[0123] 2) The host computer parses the sensor data according to the sensor data frame format to obtain the sensor data of each joint; based on the sensor data of each joint, the installation position of the multi-node inertial motion capture device and the structure of the human body, the host computer constructs a human motion digital twin model; based on the sensor data of each joint, the human posture calibration method is used to calculate and obtain the posture calibration matrix of each joint, and store it locally; then, the posture calibration matrix of each joint is used to perform joint calibration on the sensor posture of the corresponding joint to obtain the posture of each joint (i.e., real-time posture matrix); the posture calibration matrix of each joint is multiplied by the real-time sensor posture of the corresponding joint to obtain the real-time posture of the joint; the postures of each joint form the human posture; finally, the human motion digital twin model in the host computer is driven according to the human posture.

[0124] In step 2), the posture calibration matrix for each joint is calculated using a human posture calibration method, specifically as follows:

[0125] The human body's joints include the head, left and right upper arms, left and right forearms, left and right hands, spine, hip joints, left and right thighs, left and right calves, and left and right feet.

[0126] How to establish the human body's coordinate system:

[0127] With the hip joint as the origin, when the human body maintains the first calibration posture, the X-axis is on the sagittal plane of the human body, coincides with the vertical axis, and points vertically upward; the Y-axis is on the coronal plane of the human body, coincides with the coronal axis, and points to the right side of the human body; the Z-axis is on the horizontal plane of the human body, coincides with the sagittal axis, and points to the front of the human body. The XYZ axes of the body coordinate system will change dynamically with the movement of the human body.

[0128] First, the multi-node inertial motion capture device collects sensor data of the human body in three calibration postures. Then, based on the sensor data in the three calibration postures, the initial posture calibration matrix corresponding to each joint is calculated. The three calibration postures are the first calibration posture, the second calibration posture, and the third calibration posture.

[0129] First calibration posture: Stand upright with your torso straight, arms hanging naturally at your sides with palms facing your body, feet shoulder-width apart with the soles of your feet parallel and facing forward.

[0130] Second calibration posture: Maintain a seated posture, lean your upper limbs and torso forward, place your elbows and forearms flat on the table, with the distance between your elbows the same as your shoulders, keep your forearms parallel, place your thighs flat on the chair, keep them parallel to the ground, and swing your left and right lower legs back and forth around your knees at the same frequency.

[0131] Third calibration posture: Maintain a seated posture, raise both arms vertically downwards to shoulder height on both sides of the body, palms facing the ground, upper limbs in a "T" shape, thighs flat on the chair, parallel to the ground, lower legs at a 90° angle to the ground, feet shoulder-width apart, soles parallel to each other and facing forward.

[0132] The initial posture calibration matrix for each joint consists of the X-axis vector, Y-axis vector, and Z-axis vector of the body coordinate system. The formula for calculating the X-axis vector is as follows:

[0133] X' BCS =[acc' x acc' y acc' z ]

[0134] The formula for calculating the Y-axis measurement is as follows:

[0135]

[0136] Y” BCS =PCA([gyr') x ,gyr' y ,gyr' z ])

[0137]

[0138] The formula for calculating the Z-axis dimension is as follows:

[0139]

[0140]

[0141] Z”' BCS =PCA([gyr”) x ,gyr” y ,gyr” z ])

[0142] Where X' BCS Y' represents the first X-axis vector of the body coordinate system. BCS 、Y” BCS 、Y”' BCS These represent the first, second, and third Y-axis vectors of the body coordinate system, respectively, and Z' BCS Z” BCS Z”' BCS Let acc' represent the first, second, and third Z-axis vectors of the body coordinate system, respectively. x ,acc' y ,acc' z These represent the components of the acceleration measured by the sensors at each joint of the human body in the first calibration posture, along the XYZ axes of its own sensor coordinate system; acc” x ,acc” y ,acc” z These represent the components of the acceleration measured by the sensors at each joint of the human body in the second calibration posture, along the X, Y, and Z axes of its own sensor coordinate system; norm() represents normalization; |||| represents Euclidean norm calculation; PCA() represents principal component analysis, gyr' x ,gyr' y ,gyr' z These represent the components of the angular velocity measured by the sensors at each joint of the human body in the second calibration posture, along the XYZ axes of its own sensor coordinate system; gyr” x ,gyr” y ,gyr” zThese represent the components of the angular velocity measured by the sensors at each joint of the human body in the third calibration posture, along the XYZ axes of its own sensor coordinate system.

[0143] The posture calibration matrices for the hip joint, spine, head, and left and right thighs are calculated in the same way, consisting of the first X-axis vector, first Y-axis vector, and first Z-axis vector of the body coordinate system; the posture calibration matrices for the left and right lower legs and left and right feet are calculated in the same way, consisting of the first X-axis vector, second Y-axis vector, and second Z-axis vector of the body coordinate system; the posture calibration matrices for the left and right upper arms, left and right forearms, and left and right hands are calculated in the same way, consisting of the first X-axis vector, third Y-axis vector, and third Z-axis vector of the body coordinate system.

[0144] Next, the sensor poses of the corresponding joints are calibrated using the initial pose calibration matrices of each joint to obtain the initial poses of each joint. Finally, based on the attributes of the three calibration poses and the initial poses of each joint, the initial pose calibration matrices corresponding to each joint are corrected to obtain the pose calibration matrices for each joint, specifically as follows:

[0145] Based on sensor data from the human body in the first calibration posture, the joint angles of the left and right elbows, spine, and left and right knees are calculated. Based on these calculated joint angles, the initial posture calibration matrices for the left and right upper arms and forearms, head and hip joints, and left and right thighs and lower legs are corrected according to the attributes of the first calibration posture. The elbow joint angle is composed of the upper arm and forearm, the spine joint angle is composed of the head and hip joints, and the knee joint angle is composed of the thigh and lower leg. The specific formulas are as follows:

[0146] proximalCor = vec2mat(acc proximal (gravityVec)

[0147] proximalRotM + =proximalCor*proximalRotM -

[0148] Here, proximalCor is the relative rotation matrix between the acceleration vector and the gravity vector of each joint, vec2mat() converts the vector into matrix form, and acc proximal The acceleration vector for each joint is represented by [acc]. x acc y acc z ], gravityVec represents the gravity vector of each joint; proximalRotM -The initial pose calibration matrix for each joint, proximalRotM + The corrected posture calibration matrix for each joint;

[0149] The first calibration posture attribute is that when the human body is in an upright position, the theoretical value of the joint angles at the elbow, spine, and knee is 0°.

[0150] Next, the degrees of freedom of the knee and shoulder joints are calculated based on the sensor data of the human body in the second and third calibration postures. Based on the degrees of freedom of the knee and shoulder joints, and according to the attributes of the second and third calibration postures and the initial postures of the left and right upper arms, left and right forearms, left and right thighs and left and right lower legs, the corrected posture calibration matrices of the left and right upper arms, left and right forearms, left and right thighs and left and right lower legs are compensated to obtain the final posture calibration matrix of the corresponding joints. The specific calculation formula is as follows:

[0151] angleOffset=(computeAngle(proximalQua,distalQua)) / 2,

[0152] proximalRotM +' =ang2mat(-angleoffset)*proximalRotM +

[0153] Here, `angleOffset` represents the error in the joint angles of the shoulder joint's flexion / extension, horizontal adduction / abduction, or internal / external rotation; `proximalQua` represents the initial posture of the left and right upper arms or left and right thighs; `distalQua` represents the initial posture of the left and right forearms or left and right lower legs; `computeAngle()` converts the posture to angle form; `ang2mat()` converts the angle to matrix form; and `proximalRotM`... +' The final attitude calibration matrix for each joint.

[0154] In the second and third calibration postures, the rotational movements of the arm around the shoulder and the rotational movements of the lower leg around the knee are simplified to single degrees of freedom: knee flexion and extension, and shoulder abduction and adduction. The theoretical angle values ​​for knee flexion and extension and other degrees of freedom of the shoulder are 0°. Other degrees of freedom of the knee include internal / external rotation, and other degrees of freedom of the shoulder include flexion / extension and horizontal adduction / abduction.

[0155] 3) The multi-node inertial motion capture device collects and sends real-time sensor data of each joint to the host computer. Based on the real-time posture calibration matrix of the human joints and the sensor data, it calculates the human joint parameters, compares these parameters with corresponding parameter thresholds, and iteratively evolves the human motion digital twin model based on the comparison results, achieving dynamic updates of the human motion digital twin model. The main human joint parameters include angle, displacement, angular displacement, velocity, angular velocity, acceleration, and angular acceleration.

[0156] In step 3), while driving the human motion digital twin model in real time, the host computer calculates the human joint parameters and compares the joint parameters with the corresponding parameter threshold range. If the comparison result exceeds the corresponding threshold range, it may be due to improper human motion posture or external force interference to the joints. The host computer prompts the human to stop moving and reminds the user to check whether the current motion state of the real human body matches the normal physiological motion state. If the current motion state of the real human body is normal, that is, the feedback result excludes abnormalities caused by human or external interference, it means that the current human digital model cannot correctly reflect the real human motion state. In this case, the human motion digital twin model is dynamically updated, prompting the human body to perform three calibration postures, updating the posture calibration matrix of each joint of the human body, and re-driving the human motion digital twin model, thereby realizing the dynamic evolution of the human motion digital twin model. The above steps continue to be executed until the human motion digital twin system is exited.

[0157] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0158] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a human motion digital twin based on inertial motion capture technology, characterized in that, Includes the following steps: 1) Wear a multi-node inertial motion capture device on the human body and use the multi-node inertial motion capture device to collect the raw sensing data of the sensors corresponding to each joint of the human body. After processing the raw sensing data of the sensors corresponding to each joint using the adaptive extended Kalman filter algorithm, obtain the sensor posture of the sensors corresponding to each joint of the human body. Use the raw sensing data of the sensors corresponding to each joint and the corresponding sensor posture as the sensing data of each joint and wirelessly send them to the host computer. In step 1), the raw sensor data of each joint's corresponding sensor is processed using the adaptive extended Kalman filter algorithm to obtain the sensor posture of each joint of the human body, specifically: S1: Determine the acceleration-magnetic field strength parameters based on the original sensing data of the sensors corresponding to each joint. Construct the error function and cost function between the geographic coordinate system and the sensor coordinate system of the current joint based on the acceleration-magnetic field strength parameters of the current joint. Solve the error function and cost function of the current joint using the Gauss-Newton algorithm to obtain the attitude quaternion between the geographic coordinate system and the sensor coordinate system of the current joint. Use the attitude quaternion between the geographic coordinate system and the sensor coordinate system corresponding to the current joint as the initial attitude of the sensor corresponding to the current joint. S2: Use the initial sensor attitude and angular velocity of the current joint as the state variables of the adaptive extended Kalman filter and solve them to obtain the sensor filtered attitude of the current joint. S3: Calculate the distance between the sensor filter posture of the current joint at the current moment and the sensor filter posture at the previous moment, and the distance between the negative of the sensor filter posture of the current joint at the current moment and the sensor filter posture at the previous moment. Take the sensor filter posture corresponding to the smallest distance of the two distances as the sensor posture of the current joint at the current moment. S4: Repeat S1-S3 to process the raw sensor data of the remaining joints and obtain the sensor posture of each joint of the human body. The adaptive extended Kalman filter is obtained by improving the measurement noise mean and measurement noise variance matrix in the extended Kalman filter algorithm. A forgetting factor and a forgetting factor weighting coefficient are introduced to dynamically update the measurement noise. The formulas for the measurement noise mean and measurement noise variance matrix are as follows: Where I represents the identity matrix, This represents the estimated mean value of the measurement noise at the k-th discrete time. This represents the forgetting factor weighting coefficient of the mean noise of the k-th discrete-time measurement. This represents the forgetting factor weighting coefficient of the mean noise of the (k-1)th discrete-time measurement. The initial value of the forgetting factor weighting coefficient representing the mean of measurement noise. This represents the estimated mean value of the measurement noise at the (k-1)th discrete time. This represents the discretized extended Kalman filter observations at the k-th discrete time. H represents the state prediction value in the extended Kalman filter, and H represents the measurement array in the extended Kalman filter. Let represent the estimated value of the measurement noise variance matrix at the k-th discrete time. This represents the estimated value of the measurement noise variance matrix at the (k-1)th discrete time. This represents the forgetting factor weighting coefficient of the variance matrix of the k-th discrete-time measurement noise. This represents the forgetting factor weighting coefficient of the (k-1)th discrete-time measurement noise variance matrix. This represents the initial value of the forgetting factor weighting coefficient of the measurement noise variance matrix. and Let represent the measurement estimation errors at the (k-1)th and kth discrete times in the extended Kalman filter, respectively. Let represent the mean square error of the one-step prediction at the k-th discrete time, calculated from the mean square error of the estimate at the (k-1)-th discrete time in the extended Kalman filter. Let b represent the forgetting factor, N represent the discrete time length, T represent the matrix transpose, and e represent the natural constant. The trace of the matrix is ​​represented by c1, c2, and c3, which represent the first, second, and third coefficients, respectively. This represents the absolute value operation; 2) Based on the sensor data of each joint, the installation position of the multi-node inertial motion capture device, and the skeletal structure of the human body, the host computer constructs a digital twin model of human motion; based on the sensor data of each joint, the posture calibration matrix of each joint is calculated using the human posture calibration method, and then the posture calibration matrix of each joint is used to perform joint calibration on the sensor posture of the corresponding joint to obtain the posture of each joint. The posture of each joint is composed of the posture of the human body posture, and finally the human motion digital twin model in the host computer is driven according to the human body posture. In step 2), the posture calibration matrix of each joint is calculated using a human posture calibration method, specifically as follows: First, the multi-node inertial motion capture device collects sensor data of the human body in three calibration postures. Then, it calculates the initial posture calibration matrix corresponding to each joint based on the sensor data of the three calibration postures. Next, it uses the initial posture calibration matrix of each joint to perform joint calibration on the sensor posture of the corresponding joint to obtain the posture of each joint. Finally, based on the attributes of the three calibration postures and the posture of each joint, it corrects the initial posture calibration matrix corresponding to each joint to obtain the posture calibration matrix corresponding to each joint. 3) The multi-node inertial motion capture device collects and sends real-time sensing data of each joint to the host computer in real time. Based on the real-time posture calibration matrix of each joint and the real-time sensing data, the human joint parameters are calculated. The joint parameters are compared with the corresponding parameter thresholds. Based on the comparison results, the human motion digital twin model is iteratively evolved to realize the dynamic update of the human motion digital twin model.

2. The method for constructing a human motion digital twin based on inertial motion capture technology according to claim 1, characterized in that, In step 1), the sensors of each sub-node in the multi-node inertial motion capture device collect the angular velocity, acceleration and magnetic field strength of the sensor corresponding to the human joint, and the angular velocity, acceleration and magnetic field strength constitute the original sensing data of the sensor corresponding to the current joint.

3. The method for constructing a human motion digital twin based on inertial motion capture technology according to claim 1, characterized in that, The initial posture calibration matrix corresponding to each joint is composed of the X-axis vector, Y-axis vector, and Z-axis vector of the body coordinate system. The formula for calculating the X-axis vector is as follows: The formula for calculating the Y-axis measurement is as follows: The formula for calculating the Z-axis dimension is as follows: in, This represents the first X-axis vector of the body coordinate system. , , These represent the first, second, and third Y-axis vectors of the body coordinate system, respectively. , , These represent the first, second, and third Z-axis vectors of the body coordinate system, respectively. These represent the components of the acceleration measured by the sensors at each joint of the human body in the first calibration posture, along the XYZ axes of its own sensor coordinate system. These represent the components of the acceleration measured by the sensors at each joint of the human body in the second calibration posture, along the XYZ axes of its own sensor coordinate system. This indicates normalization processing; Represents the calculation of the Euclidean norm; Principal component analysis is used. These represent the components of the angular velocity measured by the sensors at each joint of the human body in the second calibration posture, along the XYZ axes of its own sensor coordinate system. These represent the components of the angular velocity measured by the sensors at each joint of the human body in the third calibration posture, along the XYZ axes of its own sensor coordinate system. The posture calibration matrices for the hip joint, spine, head, and left and right thighs are calculated in the same way, consisting of the first X-axis vector, first Y-axis vector, and first Z-axis vector of the body coordinate system; the posture calibration matrices for the left and right lower legs and left and right feet are calculated in the same way, consisting of the first X-axis vector, second Y-axis vector, and second Z-axis vector of the body coordinate system; the posture calibration matrices for the left and right upper arms, left and right forearms, and left and right hands are calculated in the same way, consisting of the first X-axis vector, third Y-axis vector, and third Z-axis vector of the body coordinate system.

4. The method for constructing a human motion digital twin based on inertial motion capture technology according to claim 1, characterized in that, The initial posture calibration matrix for each joint is corrected based on the attributes of the three calibration postures and the initial posture of each joint to obtain the posture calibration matrix for each joint. Specifically: Based on sensor data of the human body in the first calibration posture, the joint angles of the left and right elbow joints, spine, and left and right knee joints are calculated. Based on the calculated joint angles of the left and right elbow joints, spine, and left and right knee joints, the initial posture calibration matrices of the left and right upper arms and left and right forearms, head and hip joints, and left and right thighs and left and right calves are corrected according to the attributes of the first calibration posture. The specific formulas are as follows: in, The relative rotation matrix between the acceleration vector and the gravity vector for each joint. It converts a vector into a matrix form. This represents the acceleration vector of each joint. This represents the gravity vector at each joint; The initial attitude calibration matrix for each joint, The corrected posture calibration matrix for each joint; Next, the degrees of freedom of the knee and shoulder joints are calculated based on the sensor data of the human body in the second and third calibration postures. Based on the degrees of freedom of the knee and shoulder joints, and according to the attributes of the second and third calibration postures and the initial postures of the left and right upper arms, left and right forearms, left and right thighs and left and right lower legs, the corrected posture calibration matrices of the left and right upper arms, left and right forearms, left and right thighs and left and right lower legs are compensated to obtain the final posture calibration matrix of the corresponding joints. The specific calculation formula is as follows: ， in, The error in the joint angles representing the degrees of freedom of shoulder flexion / extension, shoulder horizontal adduction / abduction, or knee internal / external rotation is indicated by the following: This indicates the initial posture of the left and right upper arms or left and right thighs. This indicates the initial posture of the left and right forearms or the left and right lower legs. This indicates that the posture is converted into an angular form. This represents converting angles into matrix form. The final attitude calibration matrix for each joint.

5. The method for constructing a human motion digital twin based on inertial motion capture technology according to claim 1, characterized in that, The three calibration postures are the first calibration posture, the second calibration posture, and the third calibration posture, respectively. First calibration posture: Stand upright with your torso straight, arms hanging naturally at your sides with palms facing your body, feet shoulder-width apart with the soles of your feet parallel and facing forward. Second calibration posture: Maintain a seated posture, lean your upper limbs and torso forward, place your elbows and forearms flat on the table, with the distance between your elbows the same as your shoulders, keep your forearms parallel, place your thighs flat on the chair, keep them parallel to the ground, and swing your left and right lower legs back and forth around your knees at the same frequency. Third calibration posture: Maintain a seated posture, raise both arms vertically downwards to shoulder height on both sides of the body, palms facing the ground, upper limbs in a "T" shape, thighs flat on the chair, parallel to the ground, lower legs at a 90° angle to the ground, feet shoulder-width apart, soles parallel to each other and facing forward.

6. The method for constructing a human motion digital twin based on inertial motion capture technology according to claim 1, characterized in that, In step 3), while driving the human motion digital twin model in real time, the host computer calculates the human joint parameters and compares the joint parameters with the corresponding parameter threshold range. If the comparison result exceeds the corresponding threshold range, the host computer prompts the human to stop moving and reminds the user to check whether the current motion state of the real human body is consistent with the normal physiological motion state. If the current motion state of the real human body is normal, the human motion digital twin model is dynamically updated, prompting the human body to execute three calibration postures, updating the posture calibration matrix of each joint of the human body, and re-driving the human motion digital twin model, thereby realizing the dynamic evolution of the human motion digital twin model.

Images