A human posture estimation method based on the fusion of visual and inertial information

Abstract

A human pose estimation method based on the fusion of visual and inertial information addresses the limitation of 3D vision sensor-based human pose estimation methods in providing three-degree-of-freedom rotational information. Leveraging the complementarity of visual and inertial information, a nonlinear optimization method adaptively fuses visual, inertial, and prior human pose information to obtain the rotation angle of human skeletal nodes and the global position of the root bone node at each moment, thus achieving real-time human pose estimation. This invention effectively improves the accuracy and robustness of human pose estimation, compensating for the susceptibility of visual sensors to occlusion and the accumulation of errors in inertial data over time.

Description

Technical Field

[0001] The invention belongs to the field of human body posture estimation, in particular to a human body posture estimation method based on the fusion of vision and inertial information. Background Art

[0002] Human posture estimation technology has important application value. With the development of technologies such as visual sensors, inertial measurement units and artificial intelligence, human posture estimation has gradually been applied to fields such as human-machine collaboration, video surveillance, film and television production, and industrial and agricultural production. For example, it is used to ensure the safety of workers in the process of human-machine collaboration, and to record and analyze people's behavior in monitoring images.

[0003] 3D human posture estimation technology has become mature. With the development of behavior recognition, auxiliary training and human-machine collaboration, people need 6D human posture estimation information to develop applications. For example, in dance auxiliary training, 6D human posture estimation includes joint rotation information, which captures richer dance movement details and achieves better training results for students. In daily production and life, the human posture estimation method based on 3D vision is the most common and practical. This method can accurately extract the joint points of the human skeleton and obtain 3D human posture information, but when the human body is self-occluded or the camera is partially occluded, the data credibility is reduced. The inertial measurement unit can provide spatial rotation information, and its output is stable, but the error of the rotation information will accumulate over time. The complementarity of visual and inertial information can be used to estimate the 6D posture of the human body. The three-degree-of-freedom displacement information output by the vision and the three-degree-of-freedom rotation information output by the inertia are simply combined. At this time, the human posture estimation system obtained has poor robustness and low accuracy. At present, there is no technology that can robustly and real-timely fuse visual and inertial information to solve the problem of 6D human posture estimation. Summary of the invention

[0004] In order to overcome the shortcoming that the human body posture estimation method based on 3D vision sensor cannot provide three-degree-of-freedom rotation information, the present invention provides a human body posture estimation method based on vision and inertial information fusion, which utilizes the complementarity of vision and inertial information, nonlinearly fuses vision, inertia and human body posture prior knowledge, obtains the rotation angle of the human body skeleton node and the global position of the root skeleton node at each moment, completes real-time human body posture estimation, and effectively improves the accuracy and robustness of human body posture estimation.

[0005] The technical solution adopted by the present invention comprises the following steps:

[0006] A method for estimating human posture based on the fusion of visual and inertial information, comprising the following steps:

[0007] Step 1) Establish the kinematic model of each bone node of the human body, determine the optimization variable θ, and determine the homogeneous transformation matrix between the camera coordinate system c and the global coordinate system g The rotation matrix between the inertial coordinate system n and the global coordinate system g Inertial sensor i and corresponding bone coordinate system b i The displacement matrix between and the rotation matrix

[0008] Step 2) Set the output frequency of vision and inertia to be consistent, with the rotation constraint E of the inertial sensor R (θ), acceleration constraint E A (θ), visual sensor position constraint E P (θ) and the human posture prior constraint E prior (θ) constructs an optimization problem for the optimization items and sets the weights of each optimization item;

[0009] Step 3) Read the position measurement value of the vision sensor at each moment and the rotation measurement value R of the inertial sensor i and acceleration measurement a i , calculate the sensor measurement values ​​of each optimization item in the unified coordinate system With estimated value

[0010] Step 4) Solve the nonlinear least squares optimization problem. The optimal solution θ at each moment is the optimal rotation angle of each human skeleton node and the optimal global position of the root skeleton node n1 at the current moment. Then, according to the established human skeleton node kinematic model, the human body posture estimation at the current moment is obtained;

[0011] Repeat steps 3) and 4) to complete the state estimation of each joint of the human body at each moment, and obtain real-time human posture estimation based on the fusion of visual and inertial information.

[0012] Furthermore, in the step 1), the camera coordinate system c represents the coordinate system of the depth camera, the inertial coordinate system n represents the unified coordinate system after all inertial sensors are calibrated, and the global coordinate system g is aligned with the initial coordinate system of the skeleton node.

[0013] In step 2), the rotation constraint E R (θ) is established by the difference between the measured and estimated values ​​of each IMU rotation matrix; the acceleration constraint E A (θ) is established by minimizing the difference between the measured and estimated values ​​of each IMU acceleration; the position constraint E P(θ) is established by minimizing the difference between the measured value and the estimated value of the global position of each bone node; the human body posture prior constraint E prior (θ) is established through existing human pose estimation datasets.

[0014] In the step 3), the is the position information of each joint of the human body read from the visual sensor, R i is the rotation information read from the inertial gyroscope, a i It is the acceleration information read from the inertial accelerometer.

[0015] In the step 4), the root skeleton node n1 is located at the pelvic joint of the human body.

[0016] The beneficial effects of the present invention are mainly manifested in: providing a human body posture estimation method based on the fusion of visual and inertial information, aiming at the shortcoming that the human body posture estimation method based on 3D visual sensor lacks three-degree-of-freedom rotation information output, adopting a nonlinear optimization method to adaptively fuse visual information, inertial information and human body posture prior information to obtain 6D human body posture estimation, while improving the accuracy and robustness of human body posture estimation, making up for the shortcomings of visual sensors being susceptible to occlusion and inertial data accumulating errors over time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the human posture estimation method based on the fusion of visual and inertial information.

[0018] Figure 2 This is a schematic diagram of the upper body bone nodes and the IMU wearing position.

[0019] Figure 3 This is a schematic diagram of the placement of visual sensors.

[0020] Figure 4 It is a flow chart of the human posture estimation algorithm that integrates visual and inertial information. DETAILED DESCRIPTION

[0021] In order to make the technical solution and design idea of ​​the present invention clearer, the upper body of the human body is selected as the posture estimation object, and two visual sensors and five inertial sensors are used. The present invention is further described in conjunction with the accompanying drawings.

[0022] Reference Figure 1 , Figure 2 , Figure 3 and Figure 4 , a human body posture estimation method based on the fusion of visual and inertial information, the method comprising the following steps:

[0023] Step 1) Establish the kinematic model of each bone node of the human body, determine the optimization variable θ, and determine the homogeneous transformation matrix between the camera coordinate system c and the global coordinate system g The rotation matrix between the inertial coordinate system n and the global coordinate system g Inertial sensor i and corresponding bone coordinate system b i The displacement matrix between and the rotation matrix The process is as follows:

[0024] 1.1) The human skeleton is defined as a rigid body connected to each other. The initial coordinate system B of the skeleton nodes is aligned with the global coordinate system g. The number of upper body bones n is defined. b =13, such as Figure 2 As shown in the figure, the bone nodes are left hand, right hand, left forearm, right forearm, left upper arm, right upper arm, left shoulder, right shoulder, spine 1 to 4 and pelvis, where the pelvis is regarded as the root bone node n1 and the child bone node n b (b≥2) have a relative rotation matrix R with its parent node b and the relatively constant displacement t b Each bone has 3 rotational degrees of freedom, the root node has a global displacement (x1, y1, z1), and 42 (d = 3 + 3 × n b =42) degrees of freedom to describe the movement of the entire upper body, and record the 42 variables as a 42-dimensional vector θ. Vector θ is used as the optimization variable of the optimization problem. The homogeneous transformation matrix of each rigid bone in the global coordinate system is derived from the forward kinematics formula.

[0025]

[0026] Where P(b) is the set of all bones;

[0027] 1.2) If Figure 3 As shown, two visual sensors are placed in front of the tester, and the distance between the visual sensors and the tester is L = 2 meters. The translation matrix of the two cameras with respect to the global coordinate system g is obtained using the "Zhang Zhengyou camera calibration method" and the rotation matrix Then determine the homogeneous transformation matrix of the camera coordinate system c and the global coordinate system g

[0028] 1.3) Place the inertial sensor IMU at the global coordinate system g, so that the inertial sensor coordinate system n is aligned with the global coordinate system g, and obtain the inertial sensor output value at this time, which is the rotation matrix between the inertial sensor coordinate system n and the global coordinate system g Repeat the above operation to obtain the i-th (i=1,2,3,4,5) inertial sensor coordinate system n iThe rotation matrix between the global coordinate system g

[0029] 1.4) If Figure 2 As shown, the IMU is worn at the corresponding bone points of the left hand, right hand, left forearm, right forearm, and pelvis. i And the corresponding bone coordinate system b i There is no displacement between At the initial moment, the tester performs the "T-pose" calibration posture, at which time the IMU is defined i The measured value is R i_initial , then IMU i And the corresponding bone coordinate system b i The rotation matrix between It is expressed as:

[0030]

[0031] Step 2) Set the output frequency of vision and inertia to 30HZ, with the rotation constraint E of the inertial sensor R (θ), acceleration constraint E A (θ), visual sensor position constraint E P (θ) and the human posture prior constraint E prior (θ) constructs an optimization problem for the optimization item and sets the weight of each optimization item; the process is as follows:

[0032] 2.1) IMU i The difference between the measured value and the estimated value of the rotation matrix of the corresponding bone node in the global coordinate system is used as the rotation term constraint of the IMU. The measured value of the rotation matrix of the corresponding bone node is expressed as:

[0033]

[0034] Among them, R i For IMU i The rotation measurement value of the corresponding bone node is expressed as:

[0035]

[0036] Among them, P(b i ) is bone b i The collection of all parent bones.

[0037] In summary, the energy function of the rotation term is defined as:

[0038]

[0039] where ψ(·) extracts the vector part of the quaternion representation of the rotation matrix, and λ Ris the weight of the rotation term energy function, ρ R (·) represents a loss function.

[0040] 2.2) Minimize IMU i Acceleration measurement value a i The error between the estimated value and the acceleration constraint of the IMU. It is expressed as:

[0041]

[0042] in, The left side (t-1) of equation (6) indicates that the current moment uses the acceleration constraint of the previous moment. Acceleration measurement value in the global coordinate system Calculated from the rotation information and acceleration measurement of the previous frame. Acceleration measurement It is expressed as:

[0043]

[0044] Among them, a g is the acceleration due to gravity.

[0045] In summary, the energy function of the acceleration term is defined as:

[0046]

[0047] Among them, λ A is the weight of the acceleration term energy function, ρ A (·) represents a loss function.

[0048] 2.3) Get the global coordinates (x, y, z) of the human skeleton nodes from the depth camera of the visual sensor, and add the constraint term that minimizes the measured value and estimated value of the global position of the skeleton nodes. Define the number of skeleton nodes used for position constraint terms as n p , the position of the bone node in the camera c coordinate system is The number of cameras is n c . Estimated values ​​of the bone node positions It is expressed as:

[0049]

[0050] In summary, the energy function of the position constraint is defined as:

[0051]

[0052] Among them, λ P is the weight of the energy function of the position constraint, ρ P (·) represents a loss function.

[0053] 2.4) Considering the actual limitation of the freedom of motion of the skeleton, the posture prior term E is used prior (θ) to limit the unreasonable movement of the joint. prior (θ) is established through the existing human pose estimation dataset “TotalCapture(2017)”, which contains 126,000 frames of human motion pose data.

[0054] First, perform k-means clustering on all the data in the dataset, and select the cluster type k = 126000 / 100 = 1260. Then take the mean of all cluster centers to get the mean μ of the posture. Finally, perform statistical analysis on the original data to get the standard deviation σ of the posture and the upper and lower limits of the degrees of freedom of each bone node θ max and θ min . Therefore, the posture prior is defined as:

[0055]

[0056] in, The dimension is 36, and there is no restriction on the displacement and rotation of the root node. prior is the weight of the energy function of the posture prior, ρ prior (·) represents a loss function.

[0057] 2.5) In summary, construct the optimization problem:

[0058]

[0059] Among them, E A 、E P 、E prior The loss function in is set to ρ(x) = log(1 + x), and the impact of outliers is limited by increasing the penalty for outliers according to the set ratio. The weight of each optimization item is set to λ R =0.1,λ P =10,λ A =0.005,λ prior =0.0001.

[0060] Step 3) Read the position measurement value of the vision sensor at each moment and the rotation measurement value R of the inertial sensor i and acceleration measurement a i , calculate the sensor measurement values ​​of each optimization item in the unified coordinate system With estimated value

[0061] Step 4) Solve the nonlinear least squares optimization problem. The optimal solution θ at each moment is the optimal rotation angle of each human skeleton node and the optimal global position of the root skeleton node n1 at the current moment. Then, according to the established human skeleton node kinematic model, the human body posture estimation at the current moment is obtained;

[0062] like Figure 1 As shown, steps 3) and 4) are repeated to complete the optimal estimation of the position and rotation of the human body joints at each moment, and obtain real-time human body posture estimation based on the fusion of visual and inertial information.

[0063] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A method for estimating human posture based on the fusion of visual and inertial information, characterized in that the method comprises the following steps: Step 1) Establish the kinematic model of each bone node of the human body, determine the optimization variable θ, and determine the homogeneous transformation matrix between the camera coordinate system c and the global coordinate system g The rotation matrix between the inertial coordinate system n and the global coordinate system g Inertial sensor i and corresponding bone coordinate system b i The displacement matrix between and the rotation matrix Step 2) Set the output frequency of vision and inertia to be consistent, with the rotation constraint E of the inertial sensor R (θ), acceleration constraint E A (θ), visual sensor position constraint E P (θ) and the human posture prior constraint E prior (θ) constructs an optimization problem for the optimization items and sets the weights of each optimization item; Step 3) Read the position measurement value of the vision sensor at each moment and the rotation measurement value R of the inertial sensor i and acceleration measurement a i , calculate the sensor measurement values ​​of each optimization item in the unified coordinate system With estimated value Step 4) Solve the nonlinear least squares optimization problem to obtain the optimal solution θ at each moment, that is, the optimal rotation angle of each human skeletal node and the optimal global position of the root skeletal node n1 at the current moment, and obtain the human body posture estimation at the current moment based on the established human skeletal node kinematic model; Step 5) Repeat steps 3) and 4) to complete the state estimation of each joint point of the human body at each moment, and obtain real-time human posture estimation based on the fusion of visual and inertial information; The process of step 1) is: 1.1) The human skeleton is defined as a rigid body connected to each other. The initial coordinate system B of the skeleton nodes is aligned with the global coordinate system g. The number of upper body bones n is defined. b = 13, the bone nodes are left hand, right hand, left forearm, right forearm, left upper arm, right upper arm, left shoulder, right shoulder, spine 1 to 4 and pelvis, of which the pelvis is regarded as the root bone node n1, and the child bone node n b They all have a relative rotation matrix R with their parent node b and the relatively constant displacement t b , b≥2, each bone has 3 rotational degrees of freedom, the root node has a global displacement (x1, y1, z1), d=3+3×n b=42, using 42 degrees of freedom to describe the movement of the entire upper body, and recording the 42 variables as a 42-dimensional vector θ. Vector θ is used as the optimization variable of the optimization problem, and the homogeneous transformation matrix of each rigid bone in the global coordinate system is derived from the forward kinematics formula. Where P(b) is the set of all bones; 1.2) Place two visual sensors in front of the tester, with a distance of L = 2 meters between the visual sensors and the tester. Use the "Zhang Zhengyou camera calibration method" to obtain the translation matrix of the two cameras with respect to the global coordinate system g and the rotation matrix Then determine the homogeneous transformation matrix of the camera coordinate system c and the global coordinate system g 1.3) Place the inertial sensor IMU at the global coordinate system g, so that the inertial sensor coordinate system n is aligned with the global coordinate system g, and obtain the inertial sensor output value at this time, which is the rotation matrix between the inertial sensor coordinate system n and the global coordinate system g Repeat the above operation to obtain the i-th (i=1,2,3,4,5) inertial sensor coordinate system n i The rotation matrix between the global coordinate system g 1.4) IMU is worn on the corresponding bone points of the left hand, right hand, left forearm, right forearm and pelvis. i And the corresponding bone coordinate system b i There is no displacement between At the initial moment, the tester performs the "T-pose" calibration posture, at which time the IMU is defined i The measured value is R i_initial , then IMU i And the corresponding bone coordinate system b i The rotation matrix between It is expressed as: The process of step 2) is: 2.1) IMU i The difference between the measured value and the estimated value of the rotation matrix of the corresponding bone node in the global coordinate system is used as the rotation term constraint of the IMU. The measured value of the rotation matrix of the corresponding bone node is expressed as: in, R i For IMU i The rotation measurement value of the corresponding bone node is expressed as: Among them, P(b i ) is bone b i The collection of all parent bones; In summary, the energy function of the rotation term is defined as: where ψ(·) extracts the vector part of the quaternion representation of the rotation matrix, and λ R is the weight of the rotation term energy function, ρ R (·) represents a loss function; 2.2) Minimize IMU i Acceleration measurement value a i The error between the estimated value and the acceleration constraint is used as the IMU acceleration constraint. It is expressed as: in, The left side of equation (6) (t-1) indicates that the acceleration constraint of the previous moment is used at the current moment, and the acceleration measurement value in the global coordinate system is The acceleration measurement value is calculated by the rotation information and acceleration measurement value of the previous frame. It is expressed as: Among them, a g is the acceleration due to gravity; In summary, the energy function of the acceleration term is defined as: Among them, λ A is the weight of the acceleration term energy function, ρ A (·) represents a loss function; 2.3) Get the global coordinates (x, y, z) of the human skeleton nodes from the depth camera of the visual sensor, add the constraint item that minimizes the measured value and the estimated value of the global position of the skeleton nodes, and define the number of skeleton nodes used for the position constraint item as n p , the position of the bone node in the camera c coordinate system is The number of cameras is n c , the estimated value of the bone node position It is expressed as: In summary, the energy function of the position constraint is defined as: Among them, λ P is the weight of the energy function of the position constraint, ρ P (·) represents a loss function; 2.4) Considering the actual limitation of the freedom of motion of the skeleton, the posture prior term E is used prior (θ) to limit the unreasonable movement of the joint, E prior (θ) is established through the existing human posture estimation dataset "TotalCapture (2017)", which contains 126,000 frames of human motion posture data; First, perform k-means clustering on all the data in the dataset, select cluster type k = 126000 / 100 = 1260, then take the mean of all cluster centers to get the mean μ of the posture, and finally perform statistical analysis on the original data to get the standard deviation σ of the posture and the upper and lower limits θ of the degrees of freedom of each bone node max and θ min , thus, the posture prior is defined as: in, The dimension is 36, and there is no restriction on the displacement and rotation of the root node. prior is the weight of the energy function of the posture prior, ρ prior (·) represents a loss function; 2.5) In summary, construct the optimization problem: Among them, E A 、E P 、E prior The loss function in is set to ρ(x) = log(1 + x). The impact of outliers is limited by increasing the penalty for outliers according to the set ratio. The weight of each optimization item is set to λ R =0.1,λ P =10,λ A =0.005,λ prior =0.0001.

2. The method for estimating human posture based on the fusion of visual and inertial information as claimed in claim 1, characterized in that: In the step 1), the camera coordinate system c represents the coordinate system of the depth camera, the inertial coordinate system n represents the unified coordinate system of all inertial sensors after calibration, and the global coordinate system g is aligned with the initial coordinate system of the skeleton node.

3. The method for estimating human posture based on the fusion of visual and inertial information as claimed in claim 1 or 2, characterized in that: In step 2), the rotation constraint E R (θ) is established by the difference between the measured and estimated values ​​of each IMU rotation matrix; the acceleration constraint E A (θ) is established by minimizing the difference between the measured and estimated values ​​of each IMU acceleration; the position constraint E P (θ) is established by minimizing the difference between the measured value and the estimated value of the global position of each bone node; the human body posture prior constraint E prior (θ) is established through existing human pose estimation datasets.

4. The method for estimating human posture based on the fusion of visual and inertial information as claimed in claim 1 or 2, characterized in that: In the step 3), the is the position information of each joint of the human body read from the visual sensor, R i is the rotation information read from the inertial gyroscope, a i It is the acceleration information read from the inertial accelerometer.

5. The method for estimating human posture based on the fusion of visual and inertial information as claimed in claim 1 or 2, characterized in that: In the step 4), the root skeleton node n1 is located at the pelvic joint of the human body.

Images