A Hand Pose Measurement Method Incorporating Constraints

Abstract

The present invention discloses a hand gesture measurement method integrating constraints. Raw data is obtained through inertial sensors in a data glove, and a Kalman filter is used for attitude calculation of the sensors. Attitude information of joints is obtained from the attitude results of sensors on adjacent bone segments. Constraints are integrated into the Kalman filter model through a projection method, and the joint constraint information of the hand in physiological knowledge is converted into an inequality constraint equation set for joint attitude. The active set method is used for constraint solving to obtain the hand gesture information under physiological constraints. The hand gesture calculated by this method is more in line with the real hand structure. Integrating physiological constraints as a known inequality constraint information into the gesture measurement system also reduces the error between the calculated value of the system attitude and the real hand gesture.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and particularly relates to a method for measuring hand postures. Background Art

[0002] In recent years, the human-computer interaction technology has developed rapidly. It mainly identifies various human behaviors, body postures, etc. through relevant devices to interact with computer devices, providing users with natural, intuitive, and convenient interaction experiences. The data glove is an important field in human-computer interaction. It captures the movements of the human hand through optical fibers, flexure resistors, or inertial devices, and then provides data support for research topics and application fields such as augmented reality, intelligent driving, remote control operation, or medical rehabilitation.

[0003] The hand posture calculation method based on inertial sensors has the advantages of being portable and easy to use, with gesture interaction not being restricted by space; strong environmental adaptability, and sensor data not being affected by the surrounding environment; and relatively low power consumption. Some researchers have traditionally obtained the posture information of inertial sensors through a Kalman filter system or a complementary filter system, and assembled them into a data glove to calculate hand postures. Due to many reasons such as the inertial sensors not completely fitting the finger joints or the system errors accumulating over time, the hand postures measured by these methods sometimes cannot reflect the true hand posture information. For example, the angles of the metacarpophalangeal joints or proximal interphalangeal joints are too large or too small, resulting in obvious abnormalities in the hand postures. However, some of the errors can be reduced or avoided through the currently known hand constraints in physiology. Summary of the Invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a hand posture measurement method that integrates constraints. Raw data is obtained through inertial sensors in the data glove, and a Kalman filter is used to calculate the posture of the sensors. The posture information of the joints is obtained from the posture results of the sensors on adjacent bone segments. The constraints are integrated into the Kalman filter model through the projection method, and the joint constraint information of the hand in physiological knowledge is converted into an inequality constraint equation set for the joint postures. The active set method is used to solve the constraints to obtain the hand posture information under physiological constraints. The hand postures calculated by this method are more in line with the true hand structure. By integrating the physiological constraints as a known inequality constraint information into the posture measurement system, the error between the calculated value of the system posture and the true hand posture is also reduced.

[0005] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0006] Step 1: Use sensors to collect and calculate the posture information of hand bone segments. Through quaternion multiplication operations on adjacent bone segments, six hand joint posture information are obtained. Each joint posture is represented by a set of quaternions, denoted as:

[0007] x = [T ip T mcp I pip I mcp M pip M mcp (1)

[0008] Wherein, T ip represents the attitude of the thumb interphalangeal joint, and T mcp represents the attitude of the thumb metacarpophalangeal joint, I pip represents the attitude of the proximal interphalangeal joint of the index finger, and I mcp represents the attitude of the index finger metacarpophalangeal joint, M pip represents the attitude of the proximal interphalangeal joint of the middle finger, and M mcp represents the attitude of the middle finger metacarpophalangeal joint;

[0009] Convert the physiological joint constraints of the hand into the general form of inequalities based on quaternions: q i represents the component value in the quaternion of the joint attitude, i = 1, 2, 3, 4, θ1 represents the joint bending angle, and θ2 represents the joint abduction angle. Thus, a set of equations and inequality equations related to the constraints is obtained

[0010] Step 2: Adopt the projection method to fuse the physiological joint constraints of the hand into the derivation of the Kalman filter system. Use the physiological joint constraints as prior knowledge to solve the system state, and fuse them into the state equation of the Kalman filter system to obtain the prior attitude value that conforms to the hand constraints, which is described by the following formula:

[0011]

[0012] Wherein, represents the prior estimate value of the hand attitude at time k, A represents the state transition matrix obtained by integrating the angular velocity, ω represents the system process noise that conforms to the Gaussian distribution, represents the hand attitude value that satisfies the hand constraints at time k - 1, and is also the system state that conforms to the physiological constraints at the previous moment;

[0013] When the system obtains the optimal solution of the system in the unconstrained state, use the constraint solving method to solve the optimal solution of the system state within the constraint range Ignore the time identifier k of the system state, project the unconstrained state estimate value onto the constraint plane, and find the optimal solution of the system state in the constraint plane, which is described as the following inequality constraint optimization problem:

[0014]

[0015] In the formula, represents the optimal value obtained on the spatial plane represented by the constraint equation system, that is, the system state that satisfies the constraint relationship, x represents the system state in the unconstrained state, W is the identity matrix, and the solution with the smallest Euclidean distance from the current system state in the constraint space is obtained. represents the set of equality constraints satisfied by the hand posture. represents the set of inequality constraints that the hand posture needs to satisfy.

[0016] Step 3: Use the active set method to perform programming and solution for Equation (3);

[0017] Set the initial feasible solution x0 and the initial working set W0, perform iterative update on Equation (3), and obtain the stage feasible solution x t and the current stage effective constraint working set W t ;

[0018] If x t satisfies the KKT conditions of the inequality constraint optimization problem Equation (3), and the Lagrange multiplier λ corresponding to each constraint in the working set W t is non - negative, then x t is the optimal solution;

[0019] If it does not satisfy the KKT conditions of the inequality constraint optimization problem Equation (3) and x t makes the objective function decrease within the step size α along the constraint condition direction in W t , then take x t+1 =x t +α×f(x), where f(x) is the differential function with respect to x t and represents the direction vector that can make the objective function decrease;

[0020] If x t makes the objective function have a minimum value within the step size α along the constraint condition direction in W t , then α takes the step size when the objective function takes the minimum value;

[0021] If x t reaches the boundary value of the constraint, then in the next iteration, the algorithm will update the working set W t , remove the ineffective constraints and add new constraints, and the judgment condition for removing the constraints is to calculate whether the Lagrange multiplier λ corresponding to each constraint in W t is less than 0, delete the element corresponding to the smallest negative multiplier, and start a new iteration.

[0022] Preferably, the sensor is an inertial sensor.

[0023] The beneficial effects of the present invention are as follows:

[0024] The present invention is a method for measuring hand postures that can utilize physiological constraints, which can correct the calculation results that do not conform to the actual postures obtained by a hand posture measurement system during posture calculation; this method can also use the hand structure constraints with physiological significance as a priori knowledge to reduce the error between the calculated posture and the true hand posture. Description of the Drawings

[0025] Figure 1 It is a flow chart of the method of the present invention.

[0026] Figure 2 It is a schematic diagram of hand joint names. Detailed Embodiment

[0027] The present invention will be further described below in conjunction with the drawings and embodiments.

[0028] The technical problem to be solved by the present invention is that the hand postures measured by the method of posture calculation often cannot reflect the true hand posture information, and there are still avoidable errors between the posture calculation system and the true hand posture. In view of the above problems, the present invention proposes a hand posture measurement method integrating constraints, which can integrate the joint constraint information in physiology into the Kalman posture calculation model to ensure that the results of the multi-sensor data of the data glove obtained are more in line with the true hand structure after posture solution.

[0029] A hand posture measurement method integrating constraints mainly includes the following steps:

[0030] a) First, calculate and obtain the hand bone joint postures through an inertial sensor, calculate the joint posture information through quaternion multiplication, and convert the physiological joint constraints into a numerical constraint form for quaternions; secondly, fuse the constraint information as a priori knowledge into the state equation of the Kalman system, and use the projection method to project the unconstrained system state onto the constraint plane to convert it into the general form of a quadratic programming problem.

[0031] b) For the quadratic programming problem with inequality constraints, use the active set method to solve the constraints. In each iteration, start from the current feasible solution, regard the constraints that are active at this point as equality constraints, remove the constraints that are not active, minimize the objective function under this equality constraint, and repeat the operations of adding constraints and removing constraints after obtaining a new and better feasible solution, and finally calculate and solve to obtain the system state that conforms to the physiological constraints.

[0032] In the first step, by attaching the inertial sensor to the bone joints, collecting and calculating the bone joint posture information, each bone joint posture is represented in the form of a quaternion, and the difference in adjacent bone joint postures can be calculated through quaternion multiplication to obtain the hand joint posture information, denoted as:

[0033] x = [T ip T mcp I pip I mcp M pip M mcp

[0034] In Equation 1, T ip represents the attitude of the thumb interphalangeal joint, T mcp represents the attitude of the thumb metacarpophalangeal joint, I pip represents the attitude of the proximal interphalangeal joint of the index finger, I mcp represents the attitude of the metacarpophalangeal joint of the index finger, M pip represents the attitude of the proximal interphalangeal joint of the middle finger, M mcp represents the attitude of the metacarpophalangeal joint of the middle finger. Due to the joint restriction of multiple muscles and bones in the hand, the hand movement has a certain range of limitations. These limitations cause the joint to have a maximum movement angle and a minimum movement angle when bending and stretching, which are the physiological constraints of the human hand. Through the mapping relationship, the physiological joint constraints can be transformed into the general form of inequalities based on quaternions: q i represents the component value in the quaternion of the joint attitude, i = 1, 2, 3, 4, θ1 represents the joint bending angle, θ2 represents the joint abduction angle, and thus a set of equations and inequality equations related to the constraints is obtained By solving the joint attitude in the constraint space, the hand attitude information that conforms to the physiological joint constraints can be obtained.

[0035] In the first step, the constraints are integrated into the Kalman filter model through the projection method. The physiological constraints are used as prior knowledge and placed in the state equation of the Kalman filter system to obtain the prior attitude value of the hand that conforms to the hand constraints, which can be described by the following formula:

[0036]

[0037] In the formula, represents the prior estimated value of the hand attitude at time k, A represents the state transition matrix obtained by integrating the angular velocity, ω represents the system process noise that conforms to the Gaussian distribution, represents the hand attitude value that satisfies the hand constraints at time k - 1, which is also the system state that conforms to the physiological constraints at the previous time. When the system obtains the optimal solution of the system in the unconstrained state, a suitable constraint solving method needs to be adopted to solve the optimal solution of the system state within the constraint range. The main idea of the projection method is to establish a space plane for the constraints and project the unconstrained state estimate value onto the constraint surface, thereby solving the system state, which is described as the following inequality constraint optimization problem:

[0038] ​

[0039] In the formula, represents the optimal value obtained on the spatial plane represented by the constraint equation system, that is, the system state satisfying the constraint relationship, x represents the system state under the unconstrained state, W is the identity matrix, and the solution with the minimum Euclidean distance from the current system state in the constraint space is obtained. represents the set of equality constraint equations satisfied by the joint attitude quaternion. represents the set of inequality constraint equations that the joint attitude needs to satisfy; a mapping relationship is established between the structured model of the hand constraint and the constraints of the quadratic programming solution problem. Based on this mapping relationship, the hand constraint can be fused into the quadratic programming solution problem, so as to ensure that the obtained hand posture conforms to the constraint relationship of the hand joints, and then the hand posture tracking algorithm with fused constraints is realized.

[0040] For the above-mentioned constraint solution by transforming it into a quadratic programming problem, the method adopted in the present invention is the active set method for planning and solving. According to the above description, the inequality constraint equation system in the quadratic programming solution problem after transforming the physiological constraint can be obtained, the initial feasible solution x0 and the initial working set W0 are set, and the phased feasible solution x t and the constraint working set W t effective in the current stage are obtained; if x t satisfies the KKT conditions of the inequality constraint optimization problem and the Lagrange multiplier λ corresponding to each constraint in the working set W t is non-negative, then x t is the optimal solution; if it does not satisfy the KKT conditions of the inequality constraint optimization problem and x t makes the objective function decrease within the step length α along the constraint condition direction in W t , then take x t+1 =x t +α×f(x), where f(x) is the differential function with respect to x t , representing the direction vector that can make the objective function decrease; if x t makes the objective function have a minimum value within the step length α along the constraint condition direction in W t , then α takes the step length when the objective function takes the minimum value; if x t reaches the boundary value of the constraint, then in the next iteration, the algorithm will update the working set W t , remove the ineffective constraints and add new constraints, and the judgment condition for removing the constraints is to calculate whether the Lagrange multiplier λ corresponding to each constraint in W t is less than 0, delete the element corresponding to the smallest negative multiplier, and start a new iteration. Specific embodiment:

[0042] In view of some problems existing in the existing hand gesture calculation methods, the present invention proposes a gesture calculation method that integrates physiological hand joint constraints. By transforming the physiological constraints into a quadratic programming solution problem and integrating it into the Kalman filter, not only can the error between the gesture obtained by the system at special joint angles and the true gesture be reduced, but also the error accumulated by the system due to long-term operation can be reduced.

[0043] As Figure 1 shown, the specific implementation process of a gesture measurement method integrating hand constraints proposed by the present invention is as follows:

[0044] First, raw data is obtained through an inertial sensor, and the estimated value of the system gesture in the unconstrained state is calculated using a traditional Kalman filter model. Secondly, the constraint is integrated into the Kalman filter model through the projection method. The unconstrained state estimate is projected onto the constraint plane formed by the constraint equations for solution, and the physiological constraint is transformed into the general form of a quadratic programming problem. The active set method is used to solve the constraint. For quadratic programming with inequality constraints, in each iteration, starting from the current feasible solution, the constraints that are active at this point are used as equality constraints, and the inactive constraints are removed. Under this equality constraint, the objective function is minimized. After obtaining a new and better feasible solution, the operations of adding constraints and removing constraints are repeated, and finally the system state that conforms to the physiological constraints is calculated and solved.

[0045] The names of each hand joint are as Figure 2 shown. In the hand gesture calculation model of the present invention, since the inertial sensor is attached to the bone joint, the calculation of the sensor gesture can be regarded as the calculation of the gesture information of the hand bone joint to which it is attached. The gesture of each bone joint is represented in the form of a quaternion. By calculating the difference in the gestures of adjacent bone joints through quaternion multiplication, the hand joint gesture information can be obtained, denoted as:

[0046] x = [T ip T mcp I pip I mcp M pip M mcp

[0047] In Equation 1, T ip represents the gesture of the thumb interphalangeal joint, T mcp represents the gesture of the thumb metacarpophalangeal joint, I pip represents the gesture of the proximal interphalangeal joint of the index finger, I mcp represents the gesture of the metacarpophalangeal joint of the index finger, M pip represents the gesture of the proximal interphalangeal joint of the middle finger, M mcp represents the gesture of the metacarpophalangeal joint of the middle finger.

[0048] ​For example, when calculating the hand posture, the posture of the metacarpophalangeal joint of the index finger can be calculated through the postures of two adjacent phalanges of the back of the hand and the proximal phalanx of the index finger. This joint can perform horizontal abduction and adduction, as well as flexion and extension perpendicular to the palm, but cannot perform rotation around the index finger. Therefore, this metacarpophalangeal joint has two degrees of freedom, and the posture of the fingertip joint can be solved through the constraint information between joints. When calculating the postures of other joints, the posture information of each joint of the three fingers can be calculated by the above method.

[0049] Due to the joint constraints of various muscles and bones in the hand, the hand movement has certain range limitations. These limitations cause the joints to have a maximum and a minimum angle of movement when bending and stretching. This is the physiological constraint of the human hand. Through the mapping relationship, the physiological joint constraint can be transformed into the general form of an inequality based on quaternions: q i represents the component value in the quaternion of the joint posture, i = 1, 2, 3, 4, θ1 represents the joint bending angle, and θ2 represents the joint abduction angle. Thus, a set of equations and inequality equations related to the constraints is obtained. By solving the joint posture in the constraint space, the hand posture information that conforms to the physiological joint constraint can be obtained. The hand joint constraint information is shown in Table 1:

[0050] Table 1 Hand joint constraint information

[0051]

[0052] In the above table, flexion refers to the behavior of the finger naturally stretching and bending towards the palm, and abduction refers to the behavior of the finger swinging around the palm as the axis. The hand joint constraints are transformed into numerical equations and inequality constraints as shown in the following table:

[0053] Table 2 General form of the transformation of hand joint constraints into numerical constraints

[0054]

[0055] In the above table, q i in the proximal joint of the index finger represents the i-th value of the quaternion of the proximal joint posture of the index finger. Since this joint can only perform flexion, it is transformed into a quaternion representation that can only perform pitch movements, and the pitch range is between [0°, 110°], and it cannot perform movements of other degrees of freedom.

[0056] For example, when calculating the hand posture of a user, the calculated bending angle of the metacarpophalangeal joint of the index finger is 130°. However, generally speaking, a normal person's finger will not form such a large bending angle. Therefore, this calculation result is not realistic. More or less, it is because the model has received interference and the system error has increased. It can be found in the joint constraint information that the bending angle range of the metacarpophalangeal joint is [0°, 110°]. Placing the calculated system state in the constraint space for solution, the posture information obtained after constraint solution satisfies this joint constraint and also conforms to the posture of an actual human hand.

[0057] It can be found from the physiological joint constraint relationship that the proximal interphalangeal joints of the thumb, index finger and middle finger have only one degree of freedom and can only perform bending movements, and need to satisfy the inequality angle constraint. The metacarpal joints of the thumb, index finger and middle finger have two degrees of freedom and need to satisfy two kinds of inequality angle constraints. Based on the characteristics of quaternions, when the posture lacks degrees of freedom, certain equality constraints need to be imposed on the quaternions.

[0058] After that, the constraints are fused into the Kalman filter model through the projection method. Taking the physiological constraints as prior knowledge and placing them in the state equation of the Kalman filter system, the prior posture value of the hand that conforms to the hand constraints can be obtained, which can be described by the following formula:

[0059]

[0060] In the formula, represents the prior estimated value of the hand posture at time k, A represents the state transition matrix obtained by integrating the angular velocity, ω represents the system process noise that conforms to the Gaussian distribution, represents the hand posture value that satisfies the hand constraints at time k - 1, which is also the system state that conforms to the physiological constraints at the previous moment. When the system obtains the optimal solution of the system in the unconstrained state, a suitable constraint solution method needs to be adopted to solve the optimal solution of the system state within the constraint range. The main idea of the projection method is to establish a spatial plane for the constraints and project the unconstrained state estimate value onto the constraint plane, so as to solve the system state, which is described as the following inequality constraint optimization problem:

[0061]

[0062] In the formula, represents the optimal value obtained on the spatial plane represented by the constraint equation set, that is, the system state that satisfies the constraint relationship. x represents the system state in the unconstrained state, W is the identity matrix, and the solution with the minimum Euclidean distance from the current system state in the constraint space is obtained. represents the set of equality constraints satisfied by the joint posture quaternion. It represents the set of inequality constraints that the joint posture needs to satisfy; a mapping relationship is established between the structured model of the hand constraint and the constraints of the quadratic programming solution problem. Based on this mapping relationship, the hand constraint can be incorporated into the quadratic programming solution problem, thereby ensuring that the obtained hand posture conforms to the constraint relationship of the hand joints, and further realizing the hand posture tracking algorithm with fused constraints.

[0063] For the above transformation into a quadratic programming problem for constraint solving, the method adopted by the present invention is the active set method for planning and solving. According to the above description, the inequality constraint equation system in the quadratic programming solution problem after transforming the physiological constraint can be obtained. Set the initial feasible solution x0 and the initial working set W0 to obtain the stage feasible solution x t and the constraint working set W that takes effect in the current stage t ; if x t satisfies the KKT conditions of the inequality constraint optimization problem, and the Lagrange multiplier λ corresponding to each constraint in the working set W t is non - negative, then x t is the optimal solution; if it does not satisfy the KKT conditions of the inequality constraint optimization problem and x t decreases the objective function within the step size α along the direction of the constraint conditions in W t , then take x t+1 =x t +α×f(x), where f(x) is the differential function with respect to x t , representing the direction vector that can make the objective function decrease; if x t has a minimum value of the objective function within the step size α along the direction of the constraint conditions in W t , then α takes the step size when the objective function takes the minimum value; if x t reaches the boundary value of the constraint, then in the next iteration, the algorithm will update the working set W t , remove the invalid constraints and add new constraints, and the judgment condition for removing the constraints is to calculate whether the Lagrange multiplier λ corresponding to each constraint in W t is less than 0, delete the element corresponding to the smallest negative multiplier, and start a new iteration.

Claims

1. A hand gesture measurement method integrating constraints, characterized in that, It includes the following steps: Step 1: Use a sensor to collect and calculate the hand joint posture information. Through quaternion multiplication operations on adjacent joints, six hand joint posture information are obtained. Each joint posture is represented by a set of quaternions, denoted as: x = [T ip T mcp I pip I mcp M pip M mcp (1) where, T ip represents the attitude of the thumb interphalangeal joint, T mcp represents the attitude of the thumb metacarpophalangeal joint, I pip represents the attitude of the proximal interphalangeal joint of the index finger, I mcp represents the attitude of the metacarpophalangeal joint of the index finger, M pip represents the attitude of the proximal interphalangeal joint of the middle finger, M mcp represents the attitude of the metacarpophalangeal joint of the middle finger; Convert the hand physiological joint constraints into the general form of inequalities based on quaternions: q4 = 0, q i represents the component values in the joint attitude quaternion, i = 1, 2, 3, 4, θ1 represents the joint bending angle, and θ2 represents the joint abduction angle, thus obtaining a set of equations and inequality equation systems related to the constraints Step 2: Adopt the projection method to fuse the hand physiological joint constraints into the derivation of the Kalman filter system. Use the physiological joint constraints as prior knowledge to solve the system state, and fuse them into the state equation of the Kalman filter system to obtain the prior posture value that conforms to the hand constraints, described by the following formula: In the formula, represents the prior estimated value of the hand posture at time k, A represents the state transition matrix obtained by integrating the angular velocity, and ω represents the system process noise that conforms to the Gaussian distribution. represents the hand posture value that satisfies the hand constraint at time k-1, and is also the system state that conforms to the physiological constraint at the previous time. When the system obtains the system optimal solution in the unconstrained state, a constraint solving method is adopted to solve the optimal solution of the system state within the constraints. Ignoring the time identifier k of the system state, project the unconstrained state estimate onto the constraint plane, and find the optimal solution of the system state in the constraint plane, which is described as the following inequality constraint optimization problem: In the formula, represents the optimal value obtained on the spatial plane represented by the constraint equations, that is, the system state satisfying the constraint relationship, x represents the system state in the unconstrained state, W is the identity matrix, and the solution with the minimum Euclidean distance from the current system state in the constraint space is obtained. represents the set of equality constraints satisfied by the hand pose, represents the set of inequality constraints that the hand pose needs to satisfy; Step 3: Use the active set method to perform planning and solution on Equation (3); Set the initial feasible solution \(x_0\) and the initial working set \(W_0\), and perform iterative updates on Equation (3) to obtain the stage feasible solution \(x\) t and the constraint working set \(W\) that is effective in the current stage t ; If x at the current stage t satisfies the KKT conditions of the inequality constraint optimization problem in Equation (3), and the Lagrange multiplier λ corresponding to each constraint in the working set W t is non - negative, then x t is the optimal solution; If the KKT conditions of the inequality constraint optimization problem in Equation (3) are not satisfied and x t along W t makes the objective function decrease within the step size α along the direction of the constraint condition in t+1 then take x t = x t + α × f(x), where f(x) is the differential function with respect to x If x t along W t makes the objective function have a minimum value within the step size α along the direction of the constraint condition in If x t reaches the boundary value of the constraint, then in the next iteration, the algorithm will update the working set W t , remove the invalid constraints and add new constraints, and the judgment condition for removing the constraints is to calculate the Lagrange multiplier λ corresponding to each constraint in W t to see if it is less than 0, delete the element corresponding to the smallest negative multiplier, and start a new iteration.

2. The hand gesture measurement method with integrated constraints according to claim 1, characterized in that The sensor is an inertial sensor.

Images