A human joint angle data processing method based on multi-sensor fusion

Abstract

The present invention provides a method for processing human joint angle data based on multi-sensor fusion. The method uses an IMU inertial measurement unit and an RGB-D depth camera as sensors, obtains data information from different sensors, and uses the self-constraint of human joints to calculate the human joint angle data measured by each sensor, and then fuses the measurement information of the IMU inertial measurement unit and the RGB-D depth camera through a Kalman filter algorithm to obtain the final joint angle information. The present invention not only solves the problems of large jitter of depth camera data and large drift of the inertial measurement unit over time, but also improves the stability of its own movements and solves the problem of partial self-occlusion, while reducing the overall system cost, improving the accuracy of motion estimation and the portability of the overall system. The calculated joint angle data can be used as input instructions for human-computer interaction, and is used for human motion estimation, virtual human driving, and robot control.

Description

Technical Field

[0001] The present invention relates to the technical fields of motion estimation, virtual human driving and robot motion control, and in particular to a human joint angle data processing method based on multi-sensor fusion. Background Art

[0002] There are two main methods for human motion estimation. One method uses a camera array in a unified coordinate system as input, while the other method uses a full-body wearable device as input. In order to achieve the goal of high-precision human motion estimation, these methods often require high accuracy of sensors, which results in high cost of overall sensor configuration.

[0003] At the same time, these methods have many problems in actual scenarios. For example, in some complex environments, especially in scenes with strong or weak light, the camera array will lose the action information captured by the camera. At the same time, the camera array requires a long time of preliminary preparation, and the accuracy and complexity of the coordinate system of multiple sensors will also cause errors in the measurement data. If the RGB-D depth camera is used alone, the calculated skeleton points will have problems such as large data jitter and joint self-occlusion.

[0004] However, full-body wearable devices will experience large data drift over time, resulting in posture measurement deviations after a period of exercise. At the same time, full-body wearable devices also have problems such as high deployment costs and long wearing times. In addition, full-body wearable devices may slip during exercise, resulting in data deviations. Summary of the invention

[0005] In order to solve the technical problems existing in the above existing human motion estimation methods, the present invention provides a human joint angle data processing method based on multi-sensor fusion.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A method for processing human joint angle data based on multi-sensor fusion, comprising the following steps:

[0008] a) Use the inertial measurement unit to collect the three-dimensional acceleration, angular velocity and magnetic field strength data (a, ω, m) of the target human body parts, and use the RGB-D depth camera to collect the three-dimensional spatial position data (x, y, z) of the target human body bone points;

[0009] b) The data (a, ω, m) collected by the inertial measurement unit are solved by the Madgwick algorithm to obtain the quaternion information of each inertial measurement unit, and the self-constraint Q of the human joint is used to calculate the quaternion information of each inertial measurement unit. trans Q zero =Q t, get the quaternion Q of the human body corresponding to the joint rotation trans (q0, q1, q2, q3), where Q trans Q is the quaternion representation of the corresponding joint rotation of the human body; zero Q is the quaternion representation of the corresponding joint relative to the initial coordinate system during the initial zero posture calibration; t is the quaternion representation of the corresponding joint relative to the initial coordinate system at time t; the Euler angle information of the corresponding joint of the human body is calculated through the quaternion and Euler angle conversion formula (Roller IMU , Pitch IMU , Yaw IMU );

[0010] c) establishing a uniform velocity model for the data (x, y, z) collected by the RGB-D depth camera. The state space is taken as the observation space, and a Kalman filter is performed to obtain the smoothed spatial position data of the skeleton points (x′, y′, z′). The three types of joint angles with different calculation methods are calculated by using the self-constraint of the human joints, and then the Euler angle information of the corresponding joints of the human body is obtained (Roller angle information). RGB-D , Pitch RGB-D , Yaw RGB-D ), where Roll RGB-D Pitch is the corresponding joint roll angle finally calculated by the depth camera; RGB-D Yaw is the pitch angle of the corresponding joint finally calculated by the depth camera; RGB-D The corresponding joint yaw angle finally calculated by the depth camera;

[0011] d) The Euler angles calculated by the inertial measurement unit and the RGB-D depth camera are fused by Kalman filtering.

[0012]

[0013] Substitute into the following equation and iterate:

[0014]

[0015]

[0016] K t =P t|t-1 H T (HP t|t-1 H T +R) -1

[0017]

[0018] P t|t=(IK t H)P t|t-1 Among them, F t is the state transformation matrix at time t; is the estimate of the state at time t-1; P t|t-1 is the covariance matrix of the posterior estimation error at time t-1 for time t; P t-1|t-1 is the posterior estimation error covariance matrix at time t-1; Q is the process noise covariance matrix; K t is the optimal Kalman gain matrix; H is the observation model matrix; R is the observation noise covariance matrix; is the estimate of the state at time t; P t|t is the posterior estimation error covariance matrix at time t;

[0019] e) Output the corresponding joint angle vector To the virtual human or robot end to reproduce the action, where Roll' is the corresponding joint roll angle after Kalman filter fusion; Pitch' is the corresponding joint pitch angle after Kalman filter fusion; Yaw' is the corresponding joint yaw angle after Kalman filter fusion.

[0020] In the present invention, the inertial measurement unit and the depth camera do not need to have a unified coordinate system, but they need to have a unified clock. Since multiple sensors jointly observe and collect human limb movement information, the constraints of the human joints themselves can be used to convert the commonly used spatial position information into spatial rotation information, and the changes in the human joint angles observed by the multiple sensors can be calculated separately, which can reduce the errors caused by the unified spatial position coordinate system among multiple sensors in practical applications. At the same time, the clock is unified to ensure that the initial and end times of the multi-sensor data are consistent and the data acquisition frequency is kept consistent.

[0021] Furthermore, the present invention reduces the jitter of the depth camera data and improves the situation where the inertial measurement unit data drifts greatly over time through the improved Kalman filter algorithm. The Kalman filter algorithm used in step d) of the present invention refers to updating the F in the Kalman filter algorithm in a data-driven manner. t The matrix enables the prediction process of the Kalman equation to include multi-sensor measurement information and complement the advantages of multiple sensors.

[0022] Furthermore, the inertial measurement unit of the present invention needs to be calibrated for initial posture, which is to obtain data of fixed posture of human body in the initial stage, and use these data as a reference to calculate the rotation angle corresponding to each joint measured by each inertial measurement unit.

[0023] Furthermore, the inertial measurement units are respectively bound to the upper arms, forearms and backs of hands of the left and right upper limbs of the human body, and the thighs, calves and insteps of the left and right lower limbs of the human body, and each inertial measurement unit includes: an accelerometer, a gyroscope and a magnetometer.

[0024] The advantages of the present invention are: using relatively common and low-cost inertial measurement units and cameras as sensors, and by utilizing the constraints of the human body's own joints, the corresponding joint angle information can be collected and output through different sensors without unifying the spatial position coordinate system of the two sensors, and then the joint angle data measured by each sensor are fused through an improved extended Kalman filter algorithm, so that the robot can obtain more accurate joint angle data. The present invention not only solves the problem of large jitter of depth camera data and large drift of the inertial measurement unit over time, but also improves the stability of its own movements and solves the problem of partial self-occlusion. This method reduces the overall system cost while improving the accuracy of motion estimation and the portability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of the method of the present invention;

[0026] Figure 2 It is a schematic diagram of the placement of the inertial measurement unit of the present invention;

[0027] Figure 3 It is a posture schematic diagram of the initial zero posture calibration of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. The described specific embodiments are only used to explain the present invention and are not used to limit the present invention.

[0029] like Figure 1 As shown, this embodiment provides a method for processing human joint angle data based on multi-sensor fusion, comprising the following steps:

[0030] a) Place an inertial measurement unit at the corresponding body part of the human body, such as Figure 2 As shown, there are twelve inertial measurement units in this embodiment, which are respectively bound to the upper arms, lower arms and backs of the hands of the left and right upper limbs of the human body, and the thighs, calves and insteps of the left and right lower limbs of the human body. The IMU inertial measurement unit includes: an accelerometer, a gyroscope and a magnetometer. Its data jitter is small and can better describe the limb posture, but it will produce data drift. Place a Kinect camera (as an RGB-D depth camera) directly in front of the person. Its data jitter is large, but it will not produce data drift, which can better describe the overall posture. Before officially starting, perform an initial zero posture calibration, such as Figure 3Measure and record the human body posture as shown. Figure 3 In the posture shown, the acceleration a of each inertial measurement unit zero , angular velocity ω zero , and magnetic field strength m zero . And use the quaternion solution algorithm to solve it into quaternion Q zero .

[0031] b) Collect the target human motion information through the IMU inertial measurement unit and Kinect camera. Since the IMU inertial measurement unit is composed of an accelerometer, a gyroscope and a magnetometer, the corresponding hardware chips complete the calculation tasks and connect to the host receiver through the Bluetooth protocol to achieve remote data transmission; while the Kinect camera uses the official interface components to communicate with the host, and the software is written to achieve synchronous triggering of the two sensors to achieve the effect of unified clock.

[0032] The raw data of the inertial measurement unit is acceleration a t , angular velocity ω t and magnetic field strength m t The original data of the Kinect camera is an RGB image plus depth information. The skeleton point extraction algorithm is used to extract the three-dimensional spatial coordinates (x, y, z) of each skeleton point of the human body.

[0033] c) The IMU data (a, ω, m) collected in the above steps are solved using the quaternion solution algorithm to convert the raw data of the inertial measurement unit into the quaternion Q t , the order is WXYZ. According to the self-restraint of the corresponding joints of the robot and the human body, taking the right arm of the robot as an example, its joint angle is calculated as follows:

[0034]

[0035]

[0036]

[0037] RShoulderPitch t =asin(2(q0q2-q3q1))

[0038]

[0039]

[0040]

[0041]

[0042] in, The quaternion representation of the right shoulder joint rotation; It is the quaternion representation of the right shoulder joint relative to the initial coordinate system during the initial zero posture calibration; is the quaternion representation of the right shoulder joint relative to the initial coordinate system at time t; (q0, q1, q2, q3) is the specific representation of the corresponding quaternion; RShoulderRoll t is the right shoulder joint roll angle at time t; RShoulderPitch t is the pitch angle of the right shoulder joint at time t; The quaternion representation of the right elbow joint rotation; It is the quaternion representation of the right elbow joint relative to the initial coordinate system during the initial zero posture calibration; is the quaternion representation of the right elbow joint relative to the initial coordinate system at time t; RElbowRoll t is the right elbow roll angle at time t; RElbowYaw t is the right elbow joint deflection angle at time t.

[0043] By integrating the above calculation results, we can get the joint angle vector (Roll) corresponding to each joint. IMU , Pitch IMU , Yaw IMU ).

[0044] d) The spatial position information (x, y, z) of the human skeleton points is obtained through the Kinect camera skeleton point algorithm. Since the Kinect camera skeleton point algorithm has a large jitter problem, a uniform speed model is established for its movement. As the state space, (x, y, z) is used as the observation quantity. After a conventional Kalman filter, relatively smooth data (x′, y′, z′) is obtained. Then, according to the self-restraint of the corresponding joints of the robot and the human body, the corresponding joint angles of the robot are calculated, which are divided into the following three cases:

[0045] Category 1 (shoulder pitch angle, shoulder roll angle, hip pitch angle, hip roll angle)

[0046] Taking the shoulder joint pitch angle as an example, this angle is the angle between the projection of the upper arm on the XZ plane and the X-axis, which can be calculated based on the angle formula between a vector and a plane.

[0047] Category 2 (elbow roll angle, knee pitch angle)

[0048] This type of joint angle involves the movement of two limbs. The elbow joint roll angle is the angle between the robot's upper arm vector and the forearm vector, which can be calculated based on the angle formula between vectors.

[0049] Category 3 (elbow joint deflection angle)

[0050] To calculate the elbow joint deflection angle, you need to calculate the angle formed by the plane where the forearm is located (established by the three joint points of the elbow joint, shoulder joint, and wrist joint) and the plane where the upper arm is located (established by the three joint points of the elbow joint, right shoulder joint, and left shoulder joint). The angle can be calculated according to the angle formula between planes.

[0051] By integrating the above calculation results, we can get the joint angle vector (Roll) corresponding to each joint. RGB-D , Pitch RGB-D , Yaw RGB-D ).

[0052] e) The Euler angles calculated by the IMU inertial measurement unit and the Kinect camera are fused using an extended Kalman filter:

[0053] The prediction equation of the improved Kalman filter equation of the present invention is as follows:

[0054]

[0055]

[0056] Among them, F t is the state transformation matrix at time t; is the estimate of the state at time t-1; P t|t-1 is the covariance matrix of the posterior estimation error at time t-1 for time t; P t-1|t-1 is the posterior estimation error covariance matrix at time t-1; Q is the process noise covariance matrix.

[0057] In the above prediction equation, and are all known quantities. The present invention calculates F by data-driven method. t Matrix, since the human motion model is assumed to be a uniform speed model, let F t The matrix is Among them, A, B, and C are three unknown parameters, and F can be solved by three linear equations. t The value of .

[0058] The update equation of the improved Kalman filter equation of the present invention is as follows:

[0059] K t =P t|t-1 H T (HP t|t-1 H T +R) -1

[0060]

[0061] P t|t =(IK t H)P t|t-1

[0062] Among them, K t is the optimal Kalman gain matrix; H is the observation model matrix; R is the observation noise covariance matrix; is the estimate of the state at time t; P t|t is the posterior estimation error covariance matrix at time t.

[0063] Kalman filter output The measurement information of the inertial measurement unit (IMU) and the Kinect depth camera are integrated, and the large jitter of the Kinect camera data and the large drift of the inertial measurement unit (IMU) over time are improved through the multi-sensor fusion algorithm.

[0064] f) Output corresponding joint angle Go to the virtual human or robot to reproduce the actions.

Claims

1. A method for processing human joint angle data based on multi-sensor fusion, characterized in that: The following steps are involved: a) Use the inertial measurement unit to collect the three-dimensional acceleration, angular velocity and magnetic field strength data (a, ω, m) of the target human body parts, and use the RGB-D depth camera to collect the three-dimensional spatial position data (x, y, z) of the target human body bone points; b) The data (a, ω, m) collected by the inertial measurement unit are solved by the Madgwick algorithm to obtain the quaternion information of each inertial measurement unit, and the self-constraint Q of the human joint is used to calculate the quaternion information of each inertial measurement unit. trans Q zero =Q t , get the quaternion Q of the human body corresponding to the joint rotation trans (q0,q1,q2,q3), where Q trans It is the quaternion representation of the corresponding joint rotation of the human body; Q zero Q is the quaternion representation of the corresponding joint relative to the initial coordinate system during the initial zero posture calibration; t is the quaternion representation of the corresponding joint relative to the initial coordinate system at time t; the Euler angle information of the corresponding joint of the human body is calculated through the quaternion and Euler angle conversion formula (Roller IMU ,Pitch IMU ,Yaw IMU ); c) establishing a uniform velocity model for the data (x, y, z) collected by the RGB-D depth camera. As the state space, a Kalman filter is performed with (x, y, z) as the observation quantity to obtain the smoothed spatial position data of the skeleton point (x ′ ,y ′ ,z ′ ), the self-constraint of human joints is used to calculate the three types of joint angles with different calculation methods, and then the Euler angle information of the corresponding joints of the human body is obtained (Roll RGB-D ,Pitch RGB-D ,Yaw RGB-D ), where Roll RGB-D Pitch is the corresponding joint roll angle finally calculated by the depth camera; RGB-D The corresponding joint pitch angle finally calculated by the depth camera; Yaw RGB-D The corresponding joint yaw angle finally calculated by the depth camera; d) The Euler angles calculated by the inertial measurement unit and the RGB-D depth camera are fused by Kalman filtering. Substitute into the following equation and iterate: K t =P t|t-1 H T (HP t|t-1 H T +R) -1 P t|t =(I-K t H)P t|t-1 Among them, F t is the state transformation matrix at time t; is the estimate of the state at time t-1; P t|t-1 is the covariance matrix of the posterior estimation error at time t-1 for time t; P t-1|t-1 is the posterior estimation error covariance matrix at time t-1; Q is the process noise covariance matrix; K t is the optimal Kalman gain matrix; H is the observation model matrix; R is the observation noise covariance matrix; is the estimate of the state at time t; P t|t is the posterior estimation error covariance matrix at time t; e) Output the corresponding joint angle vector To the virtual human or robot to reproduce the action, Roll ′ Pitch is the corresponding joint roll angle after Kalman filter fusion; ′ is the pitch angle of the corresponding joint after Kalman filter fusion; Yaw ′ is the corresponding joint yaw angle after Kalman filter fusion.

2. A method for processing human joint angle data based on multi-sensor fusion according to claim 1, characterized in that: In step a), the inertial measurement unit and the RGB-D depth camera do not need to have a unified coordinate system, but need to have a unified clock to ensure that the initial and end times of the data collected by the two are consistent and the data collection frequency is kept consistent.

3. The method for processing human joint angle data based on multi-sensor fusion according to claim 1, characterized in that: In the step a), the inertial measurement unit is first calibrated for initial posture, that is, data of the fixed posture of the human body is acquired in the initial stage, and the rotation angle corresponding to each joint measured by each inertial measurement unit is calculated based on these data.

4. A method for processing human joint angle data based on multi-sensor fusion according to any one of claims 1 to 3, characterized in that: The inertial measurement units are respectively bound to the upper arms, lower arms and backs of hands of the left and right upper limbs of the human body, and the thighs, lower legs and backs of feet of the left and right lower limbs of the human body. Each inertial measurement unit includes: an accelerometer, a gyroscope and a magnetometer.

Images