Hand posture recognition method and system based on RGB image

Abstract

The invention provides a hand posture recognition method and system based on an RGB image, and the method comprises the steps: obtaining an RGB image containing a target hand, and obtaining a two-dimensional hand joint point position based on the RGB image; training under the three-dimensional hand parameterized model based on the two-dimensional hand joint point position to obtain a shape parameter and a posture parameter, and obtaining the three-dimensional hand joint point position based on the shape parameter and the posture parameter; performing conversion processing based on the attitude parameters to obtain a first rotation matrix; performing prediction processing under a preset attention network based on the RGB image and the two-dimensional hand joint point position to obtain a second rotation matrix; obtaining a visible confidence coefficient corresponding to the two-dimensional hand joint point position, and obtaining a hand rotation matrix based on the visible confidence coefficient, the first rotation matrix and the second rotation matrix; and obtaining a gesture recognition result based on the three-dimensional hand joint point position and the hand rotation matrix. According to the invention, the accuracy of hand posture recognition can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of gesture recognition, and particularly relates to a hand posture recognition method and system based on an RGB image. BACKGROUND

[0002] In the field of robot teleoperation, the key technical link is to accurately capture the hand action and convert the hand action into the operation instruction of the robot. The traditional hand posture recognition method recognizes the hand posture based on a monocular RGB image, which detects the hand in the image through a pre-trained model and outputs the two-dimensional coordinate positions of 21 joint nodes of the hand to realize the recognition of the hand posture. However, the traditional method is limited by the recognition accuracy and stability, and the accuracy of hand posture recognition is not high.

[0003] To solve the above problems, the prior art obtains the position of the target hand in the image through a detection algorithm, extracts hand features from the image by using a deep learning reconstruction model, and regresses hand parameters based on a Mano model. Then, a loss function including three-dimensional posture, two-dimensional projection, shape and position loss is constructed to optimize training, and a discriminator is introduced for adversarial training to improve the accuracy of parameter estimation. Finally, the three-dimensional joint node coordinates and posture parameters of the hand are output to reconstruct the hand model. However, in the posture parameter regression process, the prior art mainly relies on a single frame of static image and a forward regression of a neural network, lacks time sequence constraints, and is difficult to guarantee the posture smoothness and consistency under multiple frames of continuous data. Moreover, the prior art only relies on the robustness of the neural network itself to extract hand features, so in complex scenes such as occlusion and motion blur, the final hand posture tracking may produce hand posture jumps or abnormalities, greatly reducing the accuracy of hand gesture recognition. SUMMARY

[0004] The application aims to provide a hand posture recognition method and system based on an RGB image to solve the above technical problems and improve the accuracy of hand gesture recognition.

[0005] To solve the above technical problems, the application provides a hand posture recognition method based on an RGB image, which includes the following steps:

[0006] An RGB image containing a target hand is obtained, and based on the RGB image, two-dimensional hand joint node positions are obtained;

[0007] Based on the two-dimensional hand joint node positions, a three-dimensional hand parameterized model is trained to obtain shape parameters and posture parameters, and based on the shape parameters and the posture parameters, three-dimensional hand joint node positions are obtained;

[0008] Based on the posture parameters, conversion processing is performed to obtain a first rotation matrix;

[0009] Based on the two-dimensional hand joint position, a prediction process is performed under a preset attention network to obtain a second rotation matrix.

[0010] A visible confidence corresponding to the two-dimensional hand joint position is obtained, and based on the visible confidence, the first rotation matrix and the second rotation matrix, a hand rotation matrix is obtained.

[0011] Based on the three-dimensional hand joint position and the hand rotation matrix, a gesture recognition result is obtained.

[0012] The above scheme re-substitutes the shape parameter and the pose parameter into the three-dimensional hand parameterized model, which can reconstruct the three-dimensional hand model and the corresponding joint coordinates in the camera coordinate system to obtain the three-dimensional hand joint position; the first rotation matrix obtained based on the pose parameter can maintain geometric consistency with the overall pose of the actual hand, but in the scene where the hand joint is blocked, the pose parameter obtained by the three-dimensional hand parameterized model may deviate to cause rotation error; therefore, the second rotation matrix estimated by the preset attention network is obtained, and then the first rotation matrix and the second rotation matrix are fused based on the visible confidence to obtain the hand rotation matrix, so that the stability and accuracy of the obtained hand rotation matrix are higher, and even in a complex environment such as blocking and uneven illumination, the hand pose jumping or abnormality can be reduced, so that the hand rotation matrix obtained has good stability and accuracy, and the accuracy of the hand gesture recognition is improved.

[0013] Further, based on the two-dimensional hand joint position, the three-dimensional hand parameterized model is trained to obtain the shape parameter and the pose parameter, and based on the shape parameter and the pose parameter, the three-dimensional hand joint position is obtained, including: based on the two-dimensional hand joint position, a two-dimensional joint node relative position is obtained; based on the two-dimensional joint node relative position, a regression prediction process is performed under the three-dimensional hand parameterized model to obtain an initial shape parameter and an initial pose parameter; based on the initial shape parameter and the initial pose parameter, an optimization process is performed under a preset smoothing constraint to obtain the shape parameter and the pose parameter; and based on the shape parameter and the pose parameter, the three-dimensional hand joint position is obtained.

[0014] In the scheme, the two-dimensional hand joint position includes positions of multiple hand joints, one of the hand joints is selected as a reference point, and the two-dimensional hand joint position is processed based on the reference point to obtain a two-dimensional joint relative position; and the scheme maps the two-dimensional joint relative position to a three-dimensional hand parameterized model for regression prediction processing to obtain initial posture parameters for controlling a hand posture and initial shape parameters for controlling inherent shape features such as fatness and finger length of the hand shape, but projecting information of a two-dimensional plane to a three-dimensional plane may have certain deviation, and there may be a sudden change in the hand posture, so the obtained initial shape parameters and initial posture parameters need to be subjected to smooth constraint optimization processing, which can greatly improve the accuracy and stability of the obtained three-dimensional hand joint position, and in continuous hand posture tracking and identification, there is no sudden change or jitter of the hand posture between adjacent frames.

[0015] Further, the optimization processing based on the initial shape parameters and the initial posture parameters under a preset smooth constraint to obtain shape parameters and posture parameters includes: obtaining initial three-dimensional hand joint positions based on the initial shape parameters and the initial posture parameters, and obtaining a two-dimensional projection constraint based on the initial three-dimensional hand joint positions and the two-dimensional hand joint positions; obtaining a kinematics constraint based on the initial posture parameters under a preset joint limit; obtaining historical shape parameters and historical posture parameters, and obtaining a time sequence smooth constraint based on the historical shape parameters, the historical posture parameters, the initial shape parameters, and the initial posture parameters; constructing an energy loss function based on the two-dimensional projection constraint, the kinematics constraint, and the time sequence smooth constraint; and obtaining shape parameters and posture parameters based on the energy loss function.

[0016] Further, the obtaining of the two-dimensional projection constraint based on the initial three-dimensional hand joint positions and the two-dimensional hand joint positions includes: mapping the initial three-dimensional hand joint positions to two-dimensional mapping joint positions under a preset projection function; performing deviation calculation based on the two-dimensional mapping joint positions and the two-dimensional hand joint positions to obtain a re-projection error, and taking the re-projection error as the two-dimensional projection constraint.

[0017] Further, the obtaining the historical shape parameters and the historical pose parameters, and obtaining the time sequence smoothing constraint based on the historical shape parameters, the historical pose parameters, the initial shape parameters and the initial pose parameters comprises: obtaining the historical shape parameters and the historical pose parameters, and obtaining the predicted shape parameters and the predicted pose parameters based on the historical shape parameters, the historical pose parameters, the initial shape parameters and the initial pose parameters under a preset recurrent neural network; performing bias calculation based on the predicted shape parameters and the initial shape parameters to obtain shape parameter estimation bias; performing bias calculation based on the predicted pose parameters and the initial pose parameters to obtain pose parameter estimation bias; performing bias calculation based on the initial pose parameters and the historical pose parameters to obtain pose parameter historical bias; and obtaining the time sequence smoothing constraint based on the shape parameter estimation bias, the pose parameter estimation bias and the pose parameter historical bias.

[0018] In the above scheme, the two-dimensional projection constraint is constructed based on the reprojection error obtained from the three-dimensional hand joint position and the two-dimensional hand joint position, and the two-dimensional projection constraint ensures that the obtained three-dimensional hand joint position is as close as possible to the two-dimensional hand joint position when projected back to the two-dimensional plane by minimizing the reprojection error, thereby improving the accuracy of the obtained three-dimensional hand joint position; the kinematics constraint of the above scheme constrains the initial pose parameters through joint limits, so as to ensure that the obtained hand pose is more natural and avoid the situation that the hand joints are excessively bent; and the time sequence smoothing constraint utilizes the continuity of the hand pose change in the process of continuous hand pose tracking and recognition, thereby ensuring the stability and continuity of the obtained three-dimensional hand joint position.

[0019] Further, after obtaining the visible confidence corresponding to the two-dimensional hand joint position and obtaining the hand rotation matrix based on the visible confidence, the first rotation matrix and the second rotation matrix, the method further comprises: obtaining a historical reprojection error, and performing normalization processing based on the historical reprojection error and the reprojection error to obtain a two-dimensional projection residual confidence; obtaining a historical three-dimensional hand joint position, and performing forward difference calculation based on the historical three-dimensional hand joint position and the three-dimensional hand joint position to obtain a historical joint velocity and a current joint velocity, and obtaining a time sequence consistency confidence based on the historical joint velocity and the current joint velocity; obtaining a comprehensive confidence based on the two-dimensional projection residual confidence and the time sequence consistency confidence; and updating the three-dimensional hand joint position under a preset multi-source trajectory fusion optimization strategy based on the comprehensive confidence.

[0020] Further, the three-dimensional hand joint position is updated under a preset multi-source trajectory fusion optimization strategy based on the comprehensive confidence, including: if the comprehensive confidence does not satisfy a preset threshold, a predicted three-dimensional hand joint position is obtained by bidirectional interpolation prediction based on the historical three-dimensional hand joint position and the three-dimensional hand joint position; and the three-dimensional hand joint position is updated by fusion processing based on the predicted three-dimensional hand joint position and the historical three-dimensional hand joint position.

[0021] In the above scheme, the two-dimensional projection residual confidence quantifies the consistency of the three-dimensional hand joint position and the two-dimensional hand joint position, and the higher the value of the two-dimensional projection residual confidence, the better the consistency of the three-dimensional hand joint position and the two-dimensional hand joint position, and when tracking and identifying the target hand posture, if the current frame is suddenly disturbed by factors such as occlusion, the value of the two-dimensional projection residual confidence will decrease; and in the above scheme, the time consistency confidence is obtained by forward difference calculation based on the historical three-dimensional hand joint position and the three-dimensional hand joint position, which can quantify the motion corresponding to the three-dimensional hand joint position obtained in the current frame, and if the motion corresponding to the three-dimensional hand joint position obtained in the current frame is rapid and unnatural, the time consistency confidence will decrease significantly; the comprehensive confidence obtained based on the two-dimensional projection residual confidence and the time consistency confidence can quantify the reliability of the obtained three-dimensional hand joint position; when the comprehensive confidence is low, the three-dimensional hand joint position is updated under a preset multi-source trajectory fusion optimization strategy to improve the accuracy of the obtained three-dimensional hand joint position; thus, the scheme can improve the stability of hand posture recognition in complex environments such as occlusion and uneven illumination, and further improve the accuracy of hand posture recognition.

[0022] Further, the second rotation matrix is obtained by prediction processing under a preset attention network based on the RGB image and the two-dimensional hand joint position, including: the hand region of interest is obtained by cropping processing based on the RGB image and the two-dimensional hand joint position; the multi-scale feature is obtained by feature extraction under a preset lightweight backbone network based on the hand region of interest; the predicted rotation axis angle is obtained by prediction processing under a preset regional attention network based on the multi-scale feature; and the initial rotation matrix is obtained under the Rodrigues formula based on the predicted rotation axis angle; and the second rotation matrix is obtained by Schmidt orthogonalization correction processing based on the initial rotation matrix.

[0023] Further, the obtaining the visible confidence corresponding to the two-dimensional hand joint position and obtaining the hand rotation matrix based on the visible confidence, the first rotation matrix and the second rotation matrix comprises: obtaining the visible confidence corresponding to the two-dimensional hand joint position and obtaining the visibility score based on the visible confidence; performing calculation processing under a Sigmoid function based on the visibility score to obtain a predicted fusion weight; performing fusion processing under a preset rotation interpolation algorithm based on the predicted fusion weight, the first rotation matrix and the second rotation matrix to obtain the hand rotation matrix.

[0024] In the above scheme, the visible confidence can quantify the occlusion of each joint in the two-dimensional hand joint position, and the predicted fusion weight obtained based on the visibility score will affect the fusion weight size of the predicted second rotation matrix in the finally obtained hand rotation matrix, that is, the higher the visibility score, the smaller the corresponding fusion weight of the second rotation matrix. It can be seen that the present scheme can dynamically regulate the size of the predicted fusion weight based on the visible confidence, so that the hand rotation matrix obtained still has high stability and accuracy in complex environments such as occlusion and uneven illumination, and the accuracy of hand posture recognition is improved.

[0025] The application further provides a hand posture recognition system based on an RGB image, comprising: a two-dimensional hand joint position acquisition module, configured to acquire an RGB image containing a target hand and obtain a two-dimensional hand joint position based on the RGB image; a three-dimensional hand joint position acquisition module, configured to train under a three-dimensional hand parameterization model based on the two-dimensional hand joint position obtained by the two-dimensional hand joint position acquisition module to obtain shape parameters and posture parameters, and obtain a three-dimensional hand joint position based on the shape parameters and the posture parameters; a first rotation matrix conversion module, configured to perform conversion processing based on the posture parameters obtained by the three-dimensional hand joint position acquisition module to obtain a first rotation matrix; a second rotation matrix prediction module, configured to perform prediction processing under a preset attention network based on the RGB image and the two-dimensional hand joint position obtained by the two-dimensional hand joint position acquisition module to obtain a second rotation matrix; a hand rotation matrix fusion module, configured to obtain a visible confidence corresponding to the two-dimensional hand joint position and obtain a hand rotation matrix based on the visible confidence, the first rotation matrix obtained by the first rotation matrix conversion module and the second rotation matrix obtained by the second rotation matrix prediction module; and a gesture recognition module, configured to obtain a gesture recognition result based on the three-dimensional hand joint position obtained by the three-dimensional hand joint position acquisition module and the hand rotation matrix obtained by the hand rotation matrix fusion module.

[0026] The scheme is based on a preset attention network to obtain a predicted estimated second rotation matrix, and then based on the obtained visible confidence, the first rotation matrix and the second rotation matrix are fused to obtain a hand rotation matrix. The hand rotation matrix obtained in this way can maintain high stability and accuracy in complex environments such as occlusion and uneven illumination, reduce the abnormal jump of the hand posture, and thus improve the accuracy of hand posture recognition. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A technical implementation flowchart of a hand posture recognition method based on an RGB image is provided for an embodiment of the present application.

[0028] Figure 2 A hand posture recognition system architecture schematic diagram is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0030] Please refer to Figure 1 The present embodiment provides a hand posture recognition method based on an RGB image, comprising the following steps:

[0031] Step S1: An RGB image containing a target hand is obtained, and based on the RGB image, two-dimensional hand joint positions are obtained.

[0032] Step S2: Based on the two-dimensional hand joint positions, a three-dimensional hand parameterized model is trained to obtain shape parameters and posture parameters, and based on the shape parameters and the posture parameters, three-dimensional hand joint positions are obtained.

[0033] Step S3: Conversion processing is performed based on the posture parameters to obtain a first rotation matrix.

[0034] Step S4: Based on the two-dimensional hand joint positions, a preset attention network is used for prediction processing to obtain a second rotation matrix.

[0035] Step S5: The visible confidence corresponding to the two-dimensional hand joint positions is obtained, and based on the visible confidence, the first rotation matrix and the second rotation matrix, a hand rotation matrix is obtained.

[0036] Step S6: obtaining a gesture recognition result based on the three-dimensional hand joint position and the hand rotation matrix.

[0037] The above embodiment re-substitutes the shape parameters and the pose parameters into the three-dimensional hand parameterized model, and can reconstruct the three-dimensional hand model in the camera coordinate system and the corresponding joint coordinates to obtain the three-dimensional hand joint position. The first rotation matrix obtained based on the pose parameters in the embodiment can be geometrically consistent with the overall pose of the actual hand, but in the scenario where the hand joints are blocked, the pose parameters obtained by the three-dimensional hand parameterized model may deviate and cause rotation errors. Therefore, the embodiment obtains a second rotation matrix estimated by a preset attention network, and then fuses the first rotation matrix and the second rotation matrix based on the visible confidence to obtain the hand rotation matrix. The stability and accuracy of the hand rotation matrix obtained in this way are higher, and the hand pose jumping or abnormality can be reduced even in a complex environment such as occlusion and uneven illumination, so that the obtained hand rotation matrix has better stability and accuracy, and the accuracy of the hand pose recognition is improved.

[0038] In an embodiment, a human pose estimation model represented by VitPose or other convolutional neural networks is used to process the RGB image to detect the hand in the RGB image and extract the two-dimensional image coordinates of 21 key points of each hand as the two-dimensional hand joint position. The embodiment uses the human pose estimation model represented by VitPose or other convolutional neural networks to process the RGB image, which can improve the calculation efficiency and ensure the detection accuracy, and improve the accuracy of the obtained two-dimensional hand joint position.

[0039] Further, the training based on the two-dimensional hand joint position in the three-dimensional hand parameterized model to obtain the shape parameters and the pose parameters, and obtaining the three-dimensional hand joint position based on the shape parameters and the pose parameters, comprises: obtaining a two-dimensional joint relative position based on the two-dimensional hand joint position; performing regression prediction processing in the three-dimensional hand parameterized model based on the two-dimensional joint relative position to obtain initial shape parameters and initial pose parameters; performing optimization processing under a preset smoothing constraint based on the initial shape parameters and the initial pose parameters to obtain the shape parameters and the pose parameters; and obtaining the three-dimensional hand joint position based on the shape parameters and the pose parameters.

[0040] In the above embodiments, the two-dimensional hand joint position includes positions of multiple hand joints, one of the hand joints is selected as a reference point, and the two-dimensional hand joint position is processed based on the reference point to obtain a two-dimensional joint relative position; and the two-dimensional joint relative position is mapped to a three-dimensional hand parameterized model for regression prediction processing to obtain initial posture parameters for controlling a hand posture and initial shape parameters for controlling inherent shape features such as fatness and finger length of the hand shape, but projecting information of a two-dimensional plane to a three-dimensional plane may have certain deviation, and there may be a sudden change in the hand posture, so the obtained initial shape parameters and initial posture parameters need to be subjected to smooth constraint optimization processing, which can greatly improve the accuracy and stability of the obtained three-dimensional hand joint position, and there is no sudden change or jitter in the hand posture between adjacent frames in continuous hand posture tracking and identification.

[0041] In an embodiment, the three-dimensional hand parameterized model adopts a Mano model, and the reference point is a wrist point position in the two-dimensional hand joint position.

[0042] Further, the optimization processing based on the initial shape parameters and the initial posture parameters under a preset smooth constraint to obtain shape parameters and posture parameters includes: obtaining initial three-dimensional hand joint positions based on the initial shape parameters and the initial posture parameters, and obtaining a two-dimensional projection constraint based on the initial three-dimensional hand joint positions and the two-dimensional hand joint position; obtaining a kinematics constraint under a preset joint limit based on the initial posture parameters; obtaining historical shape parameters and historical posture parameters, and obtaining a time sequence smooth constraint based on the historical shape parameters, the historical posture parameters, the initial shape parameters and the initial posture parameters; constructing an energy loss function based on the two-dimensional projection constraint, the kinematics constraint and the time sequence smooth constraint; and obtaining the shape parameters and the posture parameters based on the energy loss function.

[0043] Further, the obtaining of the two-dimensional projection constraint based on the initial three-dimensional hand joint positions and the two-dimensional hand joint position includes: mapping the initial three-dimensional hand joint positions to two-dimensional mapping joint positions under a preset projection function; performing deviation calculation based on the two-dimensional mapping joint positions and the two-dimensional hand joint position to obtain a re-projection error, and taking the re-projection error as the two-dimensional projection constraint.

[0044] In an embodiment, the initial shape parameters are the initial posture parameters are the three-dimensional hand joint positions are J = {J i (β,θ)|i=1,2,...,21}, wherein, J iis the i-th three-dimensional joint position in the initial three-dimensional hand joint positions, and J i i,x i,y i,z , where J i,x is the x-axis position component of the i-th three-dimensional joint position, J i,y is the y-axis position component of the i-th three-dimensional joint position, and J i,z is the z-axis position component of the i-th three-dimensional joint position; and in this embodiment, the preset projection function is

[0045]

[0046] , where K is a preset camera intrinsic matrix, and Π(·) is a two-dimensional mapped joint position of the initial three-dimensional hand joint position mapped under the preset projection function; and in this embodiment, the two-dimensional hand joint position is {k i i i , i = 1, 2, …, 21}, where k i is the i-th two-dimensional joint position of the two-dimensional hand joint position, x i is the x-axis position component of the i-th two-dimensional joint position, and y i is the y-axis position component of the i-th two-dimensional joint position; and in this embodiment, the obtained reprojection error L proi is

[0047]

[0048] In an embodiment, the pose parameter θ is the local rotation angle corresponding to the j-th joint θ j (j = 1, 2, …, 15), where j = 1, 2, …, 15 corresponds to 15 joints with degrees of freedom. The physiological rotation range [θ j,min , θ j,max ] is set as a preset joint limit, where θ j,min is the minimum value of the local rotation angle corresponding to the j-th joint, and θ j,max is the maximum value of the local rotation angle corresponding to the j-th joint, for example, the range of the index metacarpal joint is 0°-90°; thus, the obtained kinematic constraint L kin of this embodiment is

[0049]

[0050] ​​​​​Further, the obtaining the historical shape parameters and the historical pose parameters, and obtaining the time sequence smoothing constraint based on the historical shape parameters, the historical pose parameters, the initial shape parameters and the initial pose parameters comprises: obtaining the historical shape parameters and the historical pose parameters, and obtaining the predicted shape parameters and the predicted pose parameters under a preset recurrent neural network based on the historical shape parameters, the historical pose parameters, the initial shape parameters and the initial pose parameters; performing deviation calculation based on the predicted shape parameters and the initial shape parameters to obtain a shape parameter estimation deviation; performing deviation calculation based on the predicted pose parameters and the initial pose parameters to obtain a pose parameter estimation deviation; performing deviation calculation based on the initial pose parameters and the historical pose parameters to obtain a pose parameter historical deviation; and obtaining the time sequence smoothing constraint based on the shape parameter estimation deviation, the pose parameter estimation deviation and the pose parameter historical deviation.

[0051] In the above embodiment, the two-dimensional projection constraint is constructed based on the reprojection error obtained from the three-dimensional hand joint position and the two-dimensional hand joint position, and the two-dimensional projection constraint ensures that the obtained three-dimensional hand joint position is as coincident as possible with the two-dimensional hand joint position when projected back to the two-dimensional plane by minimizing the reprojection error, thereby improving the accuracy of the obtained three-dimensional hand joint position; the kinematics constraint of the above embodiment constrains the initial pose parameters by joint limits, so as to ensure that the obtained hand pose is more natural and avoid the situation that the hand joints are excessively bent; and the time sequence smoothing constraint of the above embodiment utilizes the continuity of the hand pose change in the process of continuous hand pose tracking and recognition, thereby ensuring the stability and continuity of the obtained three-dimensional hand joint position.

[0052] In an embodiment, the shape parameters of the current frame are β t , the historical pose parameters are θ t , the historical shape parameters are the historical pose parameters are The predicted shape parameters obtained by the GRU prediction in the embodiment are β t,pred , the predicted pose parameters are θ t,pred , and the time sequence smoothing constraint L temp is:

[0053]

[0054] wherein λ is a time continuity weight, and in the embodiment, the time sequence smoothing constraint is obtained by penalizing the deviation of the current parameters from the predicted values and the historical values, thereby ensuring the time sequence smoothness of the hand pose recognition.

[0055] In an embodiment, the constructed energy loss function E(β, θ) is:

[0056] E(β, θ) = w1·Lproj + w2 L kin + w3 L temp ;

[0057] wherein, w1 is a weight corresponding to the two-dimensional projection constraint, w2 is a weight corresponding to the kinematics constraint, w3 is a weight corresponding to the time sequence smoothing constraint, and w1+w2+w3=1. In this embodiment, the Levenberg-Marquardt algorithm is used to iteratively optimize the energy function E(β, θ), and the parameters are constantly updated by calculating the gradient of the loss function with respect to the parameters until the iteration limit (usually 30 times) is reached or the loss change is less than a threshold (e.g., 1x10 -5 ), to obtain the optimal shape parameter β and pose parameter θ.

[0058] Further, after obtaining the hand rotation matrix based on the visible confidence, the first rotation matrix and the second rotation matrix, the method further includes: obtaining a historical reprojection error, and performing normalization processing on the historical reprojection error and the reprojection error to obtain a two-dimensional projection residual confidence; obtaining a historical three-dimensional hand joint position, and performing forward difference calculation on the historical three-dimensional hand joint position and the three-dimensional hand joint position to obtain a historical joint speed and a current joint speed, and obtaining a time sequence consistency confidence based on the historical joint speed and the current joint speed; obtaining a comprehensive confidence based on the two-dimensional projection residual confidence and the time sequence consistency confidence; and updating the three-dimensional hand joint position under a preset multi-source trajectory fusion optimization strategy based on the comprehensive confidence.

[0059] In an embodiment, the historical reprojection error contains a current time t reprojection error , and is converted into a two-dimensional projection residual confidence through normalization. The normalization conversion formula is wherein, is the maximum reprojection error value in the historical reprojection error

[0060] In an embodiment, the historical joint speed and the current joint speed are obtained by performing forward difference calculation on the historical three-dimensional hand joint position and the three-dimensional hand joint position, and the time sequence consistency confidence is obtained based on the historical joint speed and the current joint speed under the formula wherein, is the speed of joint i at time frame t-1, and λ is a proportional coefficient. for and The modulus of the difference.

[0061] Further, updating the three-dimensional hand joint position based on the comprehensive confidence level under a preset multi-source trajectory fusion optimization strategy includes: if the comprehensive confidence level does not meet a preset threshold, performing bidirectional interpolation prediction based on the historical three-dimensional hand joint position and the current three-dimensional hand joint position to obtain a predicted three-dimensional hand joint position; and performing fusion processing based on the predicted three-dimensional hand joint position and the historical three-dimensional hand joint position to update the three-dimensional hand joint position.

[0062] In the above embodiments, the 2D projection residual confidence quantifies the consistency between the 3D hand joint positions and the 2D hand joint positions. The higher the 2D projection residual confidence value, the better the consistency between the 3D and 2D hand joint positions. Furthermore, when tracking and recognizing the target hand posture, if the current frame is suddenly interfered with due to occlusion or other factors, the 2D projection residual confidence value will decrease. In addition, in the above embodiments, the temporal consistency confidence is obtained by forward differencing based on historical 3D hand joint positions and the current 3D hand joint positions, which can quantify the motion corresponding to the 3D hand joint positions obtained in the current frame. If the motion corresponding to the 3D hand joint position obtained in the current frame is a rapid, unnatural motion, the temporal consistency confidence will be significantly reduced. This embodiment uses a comprehensive confidence score obtained based on the 2D projection residual confidence score and the temporal consistency confidence score to quantify the reliability of the obtained 3D hand joint position. When the comprehensive confidence score is low, the 3D hand joint position will be updated under a preset multi-source trajectory fusion optimization strategy to improve the accuracy of the obtained 3D hand joint position. Thus, this embodiment can improve the stability of hand pose recognition in complex environments such as occlusion and uneven lighting, thereby improving the accuracy of hand pose recognition.

[0063] In one embodiment, the confidence level of the two-dimensional projection residual and time series consistency confidence In the formula The weighted fusion process is then performed to obtain the overall confidence level. Where ω1 is the confidence level of the two-dimensional projection residual. The corresponding preset weights, where ω2 is the time-series consistency confidence level. The corresponding preset weights. In this embodiment, when the overall confidence level... (At low confidence) bidirectional interpolation is performed to predict the predicted three-dimensional hand joint positions J. t+1 Based on the predicted three-dimensional hand joint positions J t+1 and historical key point location J t-1 In the formula fusion processing is performed to update the three-dimensional hand joint position wherein a is an interpolation coefficient, and is dynamically adjusted based on the comprehensive confidence, and has a higher value at high confidence.

[0064] Further, the prediction processing based on the RGB image and the two-dimensional hand joint position under the preset attention network to obtain the second rotation matrix includes: performing cropping processing based on the RGB image and the two-dimensional hand joint position to obtain a hand region of interest; performing feature extraction based on the hand region of interest under a preset lightweight backbone network to obtain multi-scale features; performing prediction processing based on the multi-scale features under a preset regional attention network to obtain a predicted rotation axis angle; obtaining an initial rotation matrix under the Rodrigues formula based on the predicted rotation axis angle; and performing Schmidt orthogonalization correction processing based on the initial rotation matrix to obtain the second rotation matrix.

[0065] In an embodiment, after the Schmidt orthogonalization correction processing based on the initial rotation matrix to obtain the second rotation matrix, the method further includes: obtaining a real rotation matrix, and constructing a geodesic distance loss function based on the real rotation matrix and the second rotation matrix; training the preset regional attention network based on the geodesic distance loss function to update the preset regional attention network. In this embodiment, the geodesic distance loss function can directly measure the angle difference between the real rotation matrix and the second rotation matrix obtained by prediction in the three-dimensional space, and the preset regional attention network is trained based on the angle difference, which can improve the accuracy of the predicted rotation matrix of the preset regional attention network, and further improve the accuracy of hand pose recognition in continuous hand pose tracking and recognition.

[0066] In an embodiment, the real rotation matrix R tg is obtained based on the real rotation matrix R tg and the second rotation matrix R ran to construct a geodesic distance loss function wherein tr(·) is a matrix trace operation.

[0067] Further, the acquiring the visible confidence corresponding to the two-dimensional hand joint position and obtaining a hand rotation matrix based on the visible confidence, the first rotation matrix and the second rotation matrix comprises: acquiring the visible confidence corresponding to the two-dimensional hand joint position and obtaining a visibility score based on the visible confidence; performing calculation processing under a Sigmoid function based on the visibility score to obtain a predicted fusion weight; performing fusion processing under a preset rotation interpolation algorithm based on the predicted fusion weight, the first rotation matrix and the second rotation matrix to obtain a hand rotation matrix.

[0068] In the above embodiment, the visible confidence can quantify the occlusion of each joint in the two-dimensional hand joint position, and the predicted fusion weight obtained based on the visibility score will affect the fusion weight size of the predicted second rotation matrix in the finally obtained hand rotation matrix, that is, the higher the visibility score, the smaller the corresponding fusion weight of the second rotation matrix. It can be seen that the embodiment can dynamically regulate the size of the predicted fusion weight based on the visible confidence, so that the hand rotation matrix obtained still has high stability and accuracy in complex environments such as occlusion and uneven illumination, and the accuracy of hand pose recognition is improved.

[0069] In an embodiment, the pose parameter θ includes three global rotation parameters ω x , ω y and ω z represented in the form of axis angle, and the global pose parameter and the rotation angle ω = || θ global ||, and the rotation axis u = (ω x / ω, ω y / ω, ω z / ω); in the embodiment, the rotation angle ω and the rotation axis u obtained based on the pose parameter are converted into the first rotation matrix R mano under the Rodrigues formula, and the formula is as follows:

[0070]

[0071] wherein I is a 3×3 unit matrix, [u] × is the skew-symmetric matrix of the rotation axis u, and u x = ω x / ω, u y = ω y / ω, y z = ω z / ω; the first rotation matrix obtained based on the global rotation parameters included in the pose parameter in the embodiment can maintain geometric consistency with the overall pose (joint rotation, shape) of the hand.

[0072] In one embodiment, a region of interest (ROI) is obtained by cropping the RGB image and the two-dimensional hand joint positions. This ROI is 256×256 pixels in size and encompasses the entire area from the wrist to the fingertips. Based on this ROI, multi-scale features are extracted using the lightweight MobileViT algorithm with approximately 8M parameters. This embodiment further enhances the feature weights of rotation-sensitive areas such as the center of the palm and the web of the hand using a spatial attention module based on these multi-scale features. Finally, prediction processing is performed using a region attention network based on these multi-scale features to obtain the predicted rotation axis angle. The predicted rotation axis angles are used to obtain the initial rotation matrix under the Rodrigues formula. This initial rotation matrix is ​​then subjected to Gram-Schmidt orthogonalization to ensure that the resulting second rotation matrix R... ran Satisfying orthogonality To avoid non-physical rotation.

[0073] In one embodiment, the visibility confidence level {c} corresponding to 21 key points of the two-dimensional hand joint position is obtained. i ∈[0,1]|i=1,...,21},c i Let c be the visibility confidence score corresponding to the i-th keypoint in the two-dimensional hand joint position, and c i =1 indicates fully visible, c i =0 indicates complete occlusion. Based on this visibility confidence level, in the formula... Below, the visibility score v is obtained, where d i The normalized value of the distance between the i-th key point and the RGB image boundary is used to prevent edge key points from being misjudged as occlusions due to truncation. In this embodiment, v∈[0,1], and the higher the value of the visibility score v, the better the complete visibility of the hand.

[0074] In one embodiment, let the first rotation matrix R be... mano The weight is ω mano The second rotation matrix R ran The weight is ω ran Furthermore, this weight is used as the prediction fusion weight, and ω is obtained based on the visibility score v. mano =σ(5(v-0.5)), ω ran =1-ω mano Where σ(·) is the Sigmoid function, to ensure that when v≥0.7 (high visibility): ω ran ≈0.2, ω mano ≈0.8 (based on the first rotation matrix); when v≤0.3 (low visibility): ωran ≈0.8, ω mano ≈0.2, (mainly the second rotation matrix); when it is the intermediate pose: the weight is smoothly transitioned; in this embodiment, the prediction fusion weight ω ran , the first rotation matrix R mano and the second rotation matrix R ran , the fusion processing is performed under a preset rotation interpolation algorithm to obtain a hand rotation matrix R fusion , and the calculation formula of the preset rotation interpolation algorithm is wherein, log(·) is a logarithmic mapping of a rotation matrix, exp(·) is an exponential mapping of a rotation matrix, wherein the logarithmic mapping and the exponential mapping are obtained based on Lie group theory, and it is ensured that the fused R fusion is still an orthogonal matrix.

[0075] In an embodiment, after obtaining the gesture recognition result based on the three-dimensional hand joint position and the hand rotation matrix, the gesture recognition result is applied to a robot, specifically: the user makes specific actions such as hand opening according to the prompt, the coordinate mapping parameters are fine-tuned by detecting the position changes of the key points in these actions, so as to improve the accuracy of coordinate mapping and ensure the consistency of coordinate mapping when used by different users; subsequently, the three-dimensional hand joint position and the hand rotation matrix are converted to the OpenXR standard coordinate system according to the left-hand system-right-hand system conversion rule, at the same time, based on the hand representation model of 25 key points in the OpenXR coordinate system, the mapping of 21 hand key points is completed, and the remaining 4 points can be determined based on linear difference; the hand representation model in the OpenXR coordinate system after mapping is used to control the robot action.

[0076] Please refer to Figure 2The application further provides a hand gesture recognition system based on an RGB image, comprising: a two-dimensional hand joint position acquisition module, configured to acquire an RGB image containing a target hand, and obtain two-dimensional hand joint positions based on the RGB image; a three-dimensional hand joint position acquisition module, configured to obtain three-dimensional hand joint positions based on the two-dimensional hand joint positions obtained by the two-dimensional hand joint position acquisition module, train under a three-dimensional hand parameterized model to obtain shape parameters and posture parameters, and obtain the three-dimensional hand joint positions based on the shape parameters and the posture parameters; a first rotation matrix conversion module, configured to convert based on the posture parameters obtained by the three-dimensional hand joint position acquisition module to obtain a first rotation matrix; a second rotation matrix prediction module, configured to predict under a preset attention network based on the RGB image and the two-dimensional hand joint positions obtained by the two-dimensional hand joint position acquisition module to obtain a second rotation matrix; a hand rotation matrix fusion module, configured to obtain a visible confidence corresponding to the two-dimensional hand joint positions, and obtain a hand rotation matrix based on the visible confidence, the first rotation matrix obtained by the first rotation matrix conversion module, and the second rotation matrix obtained by the second rotation matrix prediction module; and a gesture recognition module, configured to obtain a gesture recognition result based on the three-dimensional hand joint positions obtained by the three-dimensional hand joint position acquisition module and the hand rotation matrix obtained by the hand rotation matrix fusion module.

[0077] The above embodiment obtains a predicted estimated second rotation matrix based on a preset attention network, and then fuses the first rotation matrix and the second rotation matrix based on the obtained visible confidence to obtain a hand rotation matrix. The hand rotation matrix obtained in this way can maintain high stability and accuracy in complex environments such as occlusion and uneven illumination, reduces the situation of abnormal jumping of hand postures, and thus improves the accuracy of hand gesture recognition.

[0078] In an embodiment, a body data acquisition system is also included, which synchronously acquires RGB images, two-dimensional hand joint point coordinates and posture information, robot control signals and feedback data (such as the position and force of an end effector) of the robot during the operation of the above-mentioned hand posture recognition system based on RGB images. The multi-thread technology is used to realize the parallel acquisition of various types of data, ensuring the time synchronization of various types of data. The body data acquisition system also stores the data in a structured manner, which is convenient for subsequent query, analysis and use. After the data is acquired and stored, the acquired various types of data are analyzed under the quality evaluation mechanism based on time stability and target response to determine whether the hand posture trajectory is stable and whether the response of the robot to the control signal meets the expectation, and the failed data segments that do not meet the quality requirements are automatically removed. After the acquired various types of data are removed, the corresponding task action labels such as "grabbing", "placing" and "rotating" are automatically generated according to the task type and hand action characteristics. At the same time, the automatically generated labels can be verified by manual sampling to ensure the accuracy of the labels.

[0079] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A hand pose recognition method based on RGB images, characterized in that, Includes the following steps: Obtain an RGB image containing the target hand, and based on the RGB image, obtain the two-dimensional hand joint positions; Based on the two-dimensional hand joint positions, training is performed under a three-dimensional hand parametric model to obtain shape parameters and posture parameters, and based on the shape parameters and posture parameters, the three-dimensional hand joint positions are obtained. Based on the attitude parameters, a transformation process is performed to obtain the first rotation matrix; Based on the RGB image and the two-dimensional hand joint positions, a prediction process is performed under a preset attention network to obtain a second rotation matrix; Obtain the visibility confidence level corresponding to the position of the two-dimensional hand joint, and obtain the hand rotation matrix based on the visibility confidence level, the first rotation matrix, and the second rotation matrix; Based on the three-dimensional hand joint positions and the hand rotation matrix, the gesture recognition result is obtained.

2. The hand pose recognition method based on RGB images according to claim 1, characterized in that, The process of training a three-dimensional parametric hand model based on the two-dimensional hand joint positions to obtain shape and pose parameters, and then obtaining the three-dimensional hand joint positions based on the shape and pose parameters, includes: Based on the positions of the two-dimensional hand joints, the relative positions of the two-dimensional joints are obtained; Based on the relative positions of the two-dimensional joints, regression prediction is performed under the three-dimensional hand parametric model to obtain the initial shape parameters and initial posture parameters. Based on the initial shape parameters and the initial attitude parameters, optimization processing is performed under a preset smoothing constraint to obtain the shape parameters and attitude parameters. Based on the shape parameters and the posture parameters, the positions of the three-dimensional hand joints are obtained.

3. The hand pose recognition method based on RGB images according to claim 2, characterized in that, The optimization process based on the initial shape parameters and the initial pose parameters, under a preset smoothing constraint, yields shape parameters and pose parameters, including: Based on the initial shape parameters and the initial pose parameters, the initial three-dimensional hand joint positions are obtained, and based on the initial three-dimensional hand joint positions and the two-dimensional hand joint positions, the two-dimensional projection constraints are obtained. Based on the initial attitude parameters, kinematic constraints are obtained under preset joint limitations; Obtain historical shape parameters and historical attitude parameters, and obtain temporal smoothing constraints based on the historical shape parameters, historical attitude parameters, initial shape parameters, and initial attitude parameters; An energy loss function is constructed based on the two-dimensional projection constraint, the kinematic constraint, and the temporal smoothing constraint. Based on the energy loss function, the shape parameters and attitude parameters are obtained.

4. The hand pose recognition method based on RGB images according to claim 3, characterized in that, The step of obtaining two-dimensional projection constraints based on the initial three-dimensional hand joint positions and the two-dimensional hand joint positions includes: The initial three-dimensional hand joint positions are mapped to two-dimensional mapped joint positions under a preset projection function; The reprojection error is calculated based on the deviation between the two-dimensional mapped joint position and the two-dimensional hand joint position, and the reprojection error is used as a two-dimensional projection constraint.

5. The hand pose recognition method based on RGB images according to claim 3, characterized in that, The step of obtaining historical shape parameters and historical attitude parameters, and obtaining temporal smoothing constraints based on the historical shape parameters, historical attitude parameters, initial shape parameters, and initial attitude parameters, includes: Historical shape parameters and historical pose parameters are obtained, and based on the historical shape parameters, the historical pose parameters, the initial shape parameters, and the initial pose parameters, predicted shape parameters and predicted pose parameters are obtained under a preset recurrent neural network. Based on the predicted shape parameters and the initial shape parameters, the deviation is calculated to obtain the shape parameter estimation deviation; The deviation is calculated based on the predicted attitude parameters and the initial attitude parameters to obtain the attitude parameter estimation deviation; Based on the initial attitude parameters and the historical attitude parameters, the deviation is calculated to obtain the historical deviation of the attitude parameters; Based on the shape parameter estimation bias, the attitude parameter estimation bias, and the attitude parameter historical bias, a temporal smoothing constraint is obtained.

6. The hand pose recognition method based on RGB images according to claim 4, characterized in that, After obtaining the visibility confidence level corresponding to the two-dimensional hand joint position, and obtaining the hand rotation matrix based on the visibility confidence level, the first rotation matrix, and the second rotation matrix, the method further includes: The historical reprojection error is obtained, and the reprojection error is normalized based on the historical reprojection error to obtain the confidence level of the two-dimensional projection residual. The historical 3D hand joint positions are obtained, and forward difference calculation is performed based on the historical 3D hand joint positions and the current 3D hand joint positions to obtain the historical joint velocity and the current joint velocity. Based on the historical joint velocity and the current joint velocity, the temporal consistency confidence is obtained. Based on the confidence level of the two-dimensional projection residual and the confidence level of the temporal consistency, a comprehensive confidence level is obtained; Based on the comprehensive confidence level, the positions of the three-dimensional hand joints are updated under a preset multi-source trajectory fusion optimization strategy.

7. The hand pose recognition method based on RGB images according to claim 6, characterized in that, The step of updating the three-dimensional hand joint positions based on the comprehensive confidence level, under a preset multi-source trajectory fusion optimization strategy, includes: If the overall confidence level does not meet the preset threshold, bidirectional interpolation prediction is performed based on the historical three-dimensional hand joint position and the three-dimensional hand joint position to obtain the predicted three-dimensional hand joint position. The predicted 3D hand joint positions and the historical 3D hand joint positions are fused together to update the 3D hand joint positions.

8. The hand pose recognition method based on RGB images according to claim 1, characterized in that, The second rotation matrix is ​​obtained by performing prediction processing based on the RGB image and the two-dimensional hand joint positions under a preset attention network, including: The region of interest for the hand is obtained by cropping based on the RGB image and the position of the two-dimensional hand joints. Based on the aforementioned hand region of interest, feature extraction is performed on a preset lightweight backbone network to obtain multi-scale features. Based on the multi-scale features, prediction processing is performed under a preset region attention network to obtain the predicted rotation axis angle. Based on the predicted rotation axis angle, the initial rotation matrix is ​​obtained using the Rodriguez formula; Based on the initial rotation matrix, a Schmitt orthogonalization correction process is performed to obtain the second rotation matrix.

9. The hand pose recognition method based on RGB images according to claim 1, characterized in that, The step of obtaining the visibility confidence level corresponding to the two-dimensional hand joint position, and obtaining the hand rotation matrix based on the visibility confidence level, the first rotation matrix, and the second rotation matrix, includes: Obtain the visibility confidence level corresponding to the position of the two-dimensional hand joint, and obtain the visibility score based on the visibility confidence level; Based on the visibility score, the prediction fusion weights are obtained by calculation under the Sigmoid function. Based on the predicted fusion weights, the first rotation matrix, and the second rotation matrix, a fusion process is performed under a preset rotation interpolation algorithm to obtain the hand rotation matrix.

10. A hand pose recognition system based on RGB images, characterized in that, A method for implementing a hand pose recognition method based on RGB images as described in any one of claims 1 to 9 includes: A two-dimensional hand joint position acquisition module is used to acquire an RGB image containing the target hand and obtain the two-dimensional hand joint position based on the RGB image; A three-dimensional hand joint position acquisition module is used to train a three-dimensional hand parametric model based on the two-dimensional hand joint position obtained by the two-dimensional hand joint position acquisition module to obtain shape parameters and posture parameters, and to obtain the three-dimensional hand joint position based on the shape parameters and posture parameters. The first rotation matrix conversion module is used to convert the posture parameters obtained by the three-dimensional hand joint position acquisition module to obtain the first rotation matrix. The second rotation matrix prediction module is used to perform prediction processing under a preset attention network based on the RGB image and the two-dimensional hand joint position obtained by the two-dimensional hand joint position acquisition module to obtain the second rotation matrix. The hand rotation matrix fusion module is used to obtain the visibility confidence corresponding to the position of the two-dimensional hand joint, and to obtain the hand rotation matrix based on the visibility confidence, the first rotation matrix obtained by the first rotation matrix conversion module and the second rotation matrix obtained by the second rotation matrix prediction module. The gesture recognition module is used to obtain gesture recognition results based on the three-dimensional hand joint position obtained by the three-dimensional hand joint position acquisition module and the hand rotation matrix obtained by the hand rotation matrix fusion module.

Images