Dexterous hand control method and device, electronic equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of computers, and provides a dexterous hand control method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: obtaining a to-be-processed color image; the color image to be processed is associated with at least one dexterous hand; if the to-be-processed color image comprises a hand and the confidence of the hand in the to-be-processed color image is lower than a preset confidence threshold, generating a hand image based on the to-be-processed color image; performing hand posture estimation based on the hand image to obtain three-dimensional joint coordinate information of a hand in the hand image; determining bending angle information of fingers of the hand based on the three-dimensional joint coordinate information; and controlling the associated dexterous hand based on the bending angle information. According to the invention, the adaptability of the dexterous hand control can be improved while the dexterous hand control is deployed at low cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a dexterous hand control method and device, electronic equipment, storage medium and computer program product. BACKGROUND

[0002] With the continuous development of robot technology, dexterous hands are increasingly widely used in remote operations. Dexterous hands can perform complex tasks in various scenarios, such as industrial production, service industry and dangerous operations, by simulating the actions of human hands. In order to improve the operation performance and flexibility of dexterous hands, existing technical solutions mainly focus on accurate capture and reproduction of hand movements. These solutions use different sensing technologies and data processing methods to achieve high-precision hand pose estimation and control. Different sensing technologies (such as wearable devices) and data processing methods can be used to achieve accurate capture and reproduction of hand movements, and some introduce force feedback mechanisms to estimate object properties or optimize the gripping process.

[0003] However, current dexterous hand control technologies either require complex data collection devices such as force control gloves, data gloves, etc., which are not suitable for low-cost deployment, or directly determine the three-dimensional key point coordinates of the hand by recognizing images, which has poor adaptability when there is hand occlusion. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, the present application proposes a dexterous hand control method, device, electronic equipment, storage medium and computer program product to solve the problem of traditional inadaptability to low-cost deployment or poor adaptability, and to achieve low-cost deployment while improving adaptability.

[0005] According to the dexterous hand control method of the first aspect of the present application, it comprises: obtaining a color image to be processed; the color image to be processed is associated with at least one dexterous hand; if the color image to be processed includes a hand, and the confidence of the hand in the color image to be processed is lower than a preset confidence threshold, generating a hand image based on the color image to be processed; performing hand pose estimation based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image; determining the bending angle information of the fingers of the hand based on the three-dimensional joint coordinate information; controlling the associated dexterous hand based on the bending angle information.

[0006] According to an embodiment of the present application, after obtaining the color image to be processed, it further comprises: inputting the color image to be processed into the hand recognition model to obtain a hand recognition result output by the hand recognition model; the hand recognition result includes whether a hand exists in the color image to be processed and a confidence level of the hand when the hand exists; the hand recognition model is trained based on sample color images, labels of whether a hand exists in the sample color images, and labeled images including hand bounding boxes when a hand exists in the sample color images.

[0007] According to an embodiment of the present application, the hand recognition result further includes a bounding box of the hand in the color image to be processed; and the generating of the hand image based on the color image to be processed includes: cutting a hand image from the color image to be processed based on the bounding box.

[0008] According to an embodiment of the present application, after obtaining the hand recognition result output by the hand recognition model, the method further includes: if the color image to be processed includes a hand and the confidence level of the hand in the color image to be processed is greater than or equal to a preset confidence threshold, obtaining a depth image corresponding to the color image to be processed; generating a point cloud image based on the depth image and the color image to be processed; determining three-dimensional joint coordinate information of the hand in the color image to be processed based on the point cloud image; determining bending angle information of fingers of the hand based on the three-dimensional joint coordinate information; controlling an associated dexterous hand based on the bending angle information.

[0009] According to an embodiment of the present application, the determining of the bending angle information of the fingers of the hand based on the three-dimensional joint coordinate information includes: for each finger of the hand, respectively performing angle calculation based on three-dimensional coordinates of the joint nodes of the corresponding finger in the three-dimensional joint coordinate information to obtain the bending angle of the corresponding finger.

[0010] According to an embodiment of the present application, the controlling of the associated dexterous hand based on the bending angle information includes: sending the bending angle information to a controller of the associated dexterous hand to instruct the controller to control the associated dexterous hand based on the bending angle information.

[0011] The dexterous hand control device according to the second aspect of the present application includes: an obtaining module configured to obtain a color image to be processed; the color image to be processed is associated with at least one dexterous hand; The generating module is configured to generate a hand image based on the color image to be processed if the color image to be processed includes a hand and confidence of the hand in the color image to be processed is lower than a preset confidence threshold. The evaluating module is configured to perform hand pose estimation based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image. The determining module is configured to determine bending angle information of a finger of the hand based on the three-dimensional joint coordinate information. The control module is configured to control the associated dexterous hand based on the bending angle information.

[0012] The electronic device according to the third aspect of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the dexterous hand control method according to any one of the above when executing the computer program.

[0013] The storage medium according to the fourth aspect of the present application is a non-transitory computer readable storage medium, and a computer program is stored on the storage medium. The computer program is executed by a processor to implement the dexterous hand control method according to any one of the above.

[0014] The computer program product according to the fifth aspect of the present application includes a computer program, and the computer program is executed by a processor to implement the dexterous hand control method according to any one of the above.

[0015] The one or more technical solutions described above in the embodiments of the present application have at least the following technical effects: By performing hand recognition after obtaining the color image to be processed, and then determining that the color image to be processed includes a hand and that confidence of the hand in the color image to be processed is lower than a preset confidence threshold, a hand image can be generated based on the color image to be processed and hand pose estimation can be performed based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image. The bending angle information of the finger of the hand can be accurately determined based on the three-dimensional joint coordinate information, and the associated dexterous hand can be controlled based on the bending angle information. Since the three-dimensional joint coordinates of the hand can be determined based on the color image for dexterous hand control, there is no need to wear complex data acquisition equipment, which is suitable for low-cost deployment. Even if the confidence of the hand in the color image to be processed is lower than the preset confidence threshold due to occlusion or other reasons, the hand pose estimation can be performed based on the hand image generated from the color image, and the three-dimensional joint coordinate information of the hand in the hand image can be estimated. Furthermore, the dexterous hand can be controlled based on the three-dimensional joint coordinates, and therefore the adaptability can be improved.

[0016] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative effort.

[0018] Figure 1 is a flowchart of the dexterous hand control method provided by the embodiments of the present application.

[0019] Figure 2 is a schematic diagram of three-dimensional joint coordinate information in the dexterous hand control method provided by the embodiments of the present application.

[0020] Figure 3 is a schematic diagram of the overall flow of the dexterous hand control method provided by the embodiments of the present application.

[0021] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present application will be further described in detail below in conjunction with the drawings and examples. The following examples are used to illustrate the present application, but cannot be used to limit the scope of the present application.

[0023] In the description of the embodiments of the present application, it should be noted that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the embodiments of the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0024] In the description of the embodiments of the present application, it should be noted that unless specifically defined and limited otherwise, the terms "connected", "connected to", "connection" should be interpreted broadly, for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0025] In the embodiments of the present application, unless specifically defined and limited otherwise, the first feature is "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature "above", "over" and "on" the second feature can be that the first feature is directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "under" and "under" the second feature can be that the first feature is directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0026] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

[0027] The present application provides a dexterous hand control method, device, electronic equipment, storage medium and computer program product.

[0028] Figure 1 The flowchart of the dexterous hand control method provided by the embodiments of the present application is shown in FIG. Figure 1 As shown in the figure, the dexterous hand control method comprises: Step 110, acquiring a to-be-processed color image; the to-be-processed color image is associated with at least one dexterous hand.

[0029] Step 120, if the to-be-processed color image includes a hand, and the confidence of the hand in the to-be-processed color image is lower than a preset confidence threshold, generating a hand image based on the to-be-processed color image.

[0030] In step 130, hand pose estimation is performed based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image.

[0031] In step 140, the bending angle information of the fingers of the hand is determined based on the three-dimensional joint coordinate information.

[0032] In step 150, the associated dexterous hand is controlled based on the bending angle information.

[0033] It should be noted that the execution subject of the dexterous hand control method provided in the embodiments of the present application can be a server, a computer device, etc. The computer device can be, for example, a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an Ultra-mobile Personal Computer (UMPC), a netbook, or a Personal Digital Assistant (PDA), etc. It should be noted that the data (including image data specifically) required to be obtained in the present application is obtained through a regular channel after authorization by a relevant user.

[0034] The server or computer device of the present application can be provided with or connected to a dexterous hand control device, so as to control the dexterous hand control device to execute the dexterous hand control method of the present application.

[0035] The server or computer device in the present application can be connected to an image acquisition device, and can further interact with the image acquisition device, for example, to control the image acquisition device and obtain image data from the image acquisition device.

[0036] The image acquisition device in the present application can be, for example, an RGB (Red, Green, Blue) camera or an RGBD camera. The RGB camera captures color images based on the three primary colors of red, green, and blue. The RGBD camera is a camera that combines RGB color images and depth images. That is, the RGB camera can obtain color images and depth images corresponding to the color images.

[0037] It should be noted that the server or computer device in the present application can also be connected to a controller of the dexterous hand, so as to control the dexterous hand.

[0038] In the present application, the image acquisition device can be associated with at least one dexterous hand of at least one robot, so that the images acquired by the image acquisition device are also associated with the corresponding dexterous hand, so as to realize remote operation of the dexterous hand based on the images.

[0039] Therefore, after the image acquisition device is arranged, the user can show the hand action in the collectable area of the image acquisition device, so that the image acquisition device can acquire the image of the collectable area.

[0040] Further, the color image acquired from the image acquisition device can be used as the color image to be processed.

[0041] Further, the color image to be processed can be recognized, and after the recognition is completed, a hand recognition result including information such as whether the hand of the user exists in the color image to be processed, the confidence of the hand when the hand exists in the color image to be processed, and a labeled image including a hand label frame is obtained.

[0042] The confidence of the hand can be used to represent the confidence degree of the hand recognition. For example, the non-user hand, the environmental influence, and the hand occlusion will affect the confidence of the hand.

[0043] In the present application, a confidence value can be set in advance based on actual demand or scene index as a threshold for distinguishing the high and low confidence, that is, if the confidence of the hand is greater than or equal to the pre-set confidence threshold, it is determined that the confidence is high, and if the confidence of the hand is lower than the pre-set confidence threshold, it is determined that the confidence is low.

[0044] It should be noted that before determining the high and low confidence, the confidence of the hand obtained can be compared with a minimum confidence first. The minimum confidence can be used to represent whether the hand in the color image to be processed is a real hand. Therefore, if the confidence of the hand is greater than or equal to the minimum confidence, it is determined that the hand is a real hand, and if the confidence of the hand is lower than the minimum confidence, it is determined that the hand is not a real hand. The determination of the high and low confidence is performed only when the hand confidence is greater than or equal to the minimum confidence.

[0045] Based on this, when it is determined that the color image to be processed includes the hand, and the confidence of the hand in the color image to be processed is lower than the pre-set confidence threshold, the hand image can be acquired from the color image to be processed.

[0046] It should be further noted that a certain number of images containing hand and joint label information of the hand can be acquired as training data, and the pre-constructed model is trained through the training data. After the training is completed, a hand pose estimation model that can estimate the hand pose according to the input hand image and output the three-dimensional joint coordinate information of the hand in the hand image is obtained.

[0047] In the constructed model, a feature extraction model and a hand model with articulated and non-rigid deformations (MANO) can be included. The feature extraction model can be a VIT model. The VIT model, i.e., Vision Transformer model, is an image classification model based on a Transformer architecture. The Transformer architecture is a deep learning model based on a self-attention mechanism.

[0048] Specifically, the VIT model, as the backbone of the feature extraction network, first divides the image into image blocks, then uses multiple transformer encoder layers combined with a fully connected layer as a feature extraction module, and finally outputs 61 model parameters. The first 3 parameters are camera-related parameters denoted by Π, the middle 10 parameters are shape parameters denoted by β, and the last 48 parameters are position parameters denoted by θ.

[0049] MANO contains 778 vertices and 1538 faces, and a complete forward kinematic tree is constructed according to 16 key points and 5 points obtained from the vertices.

[0050] The input of MANO is the shape parameter β, the position parameter θ, and the camera-related parameter Π obtained in the above process. Deformation is performed using the following formula: ; Linear blend skinning is performed using the following formula, where W is the skinning weight and J(θ) is the position of the node: ; Finally, the actual hand joint positions are obtained, specifically the three-dimensional joint coordinate information formed by the three-dimensional coordinates of the key points corresponding to the hand joint.

[0051] Therefore, after obtaining the hand image, the hand image can be input into the hand pose estimation model, and the hand pose estimation model is used to estimate the hand pose of the hand image, and finally the three-dimensional joint coordinate information of the hand in the hand image output by the hand pose estimation model is obtained.

[0052] MANO is introduced as prior information in the pose estimation process, which enhances the accuracy of hand pose prediction in occluded environments.

[0053] After obtaining the three-dimensional joint coordinate information of the hand, the bending angle of each finger can be approximately determined by calculating the included angle between the joints of the fingers, and the bending angle information of the fingers is obtained.

[0054] Further, the bending angle information of the fingers can be used to control the associated dexterous hand, control the dexterous hand to output the same hand posture, and realize remote operation of the dexterous hand.

[0055] According to the dexterous hand control method, after obtaining the color image to be processed, hand recognition is performed, and when it is determined that the hand is included in the color image to be processed and the confidence of the hand in the color image to be processed is lower than the preset confidence threshold due to occlusion or the like, a hand image can be generated based on the color image to be processed, and hand posture estimation can be performed based on the hand image to obtain the three-dimensional joint coordinate information of the hand in the hand image. Further, the bending angle information of the fingers of the hand can be accurately determined based on the three-dimensional joint coordinate information, and the associated dexterous hand can be controlled based on the bending angle information. Since the three-dimensional joint coordinates of the hand can be determined based on the color image to control the dexterous hand, there is no need to wear complex data acquisition equipment, which is suitable for low-cost deployment. Even if the confidence of the hand in the color image to be processed is lower than the preset confidence threshold due to occlusion or the like, the hand posture estimation can be performed based on the hand image generated from the color image, the three-dimensional joint coordinate information of the hand in the hand image can be estimated, and the dexterous hand can be controlled based on the three-dimensional joint coordinates, thereby improving the adaptability.

[0056] Based on the above embodiments, after obtaining the color image to be processed, the following steps are further included: The color image to be processed is input into a hand recognition model to obtain a hand recognition result output by the hand recognition model; wherein the hand recognition result includes whether a hand exists in the color image to be processed and the confidence of the hand when the hand exists; and the hand recognition model is trained based on sample color images, labels of whether a hand exists in the sample color images, and annotated images including hand annotation boxes when a hand exists in the sample color images.

[0057] Specifically, a certain number of color images can be pre-acquired as sample color images, wherein part of the color images include a hand of a user, and another part of the color images do not include the hand of the user. Moreover, for each sample color image, a label of whether a hand exists or does not exist can be set; and for the color image in which a hand exists, an annotation box can be added to the hand in the image to form an annotated image.

[0058] Therefore, based on the above images and related information, a target detection model is trained, and after the training is completed, a hand recognition model that can recognize whether a hand exists in an input color image, the confidence of the hand when the hand exists, and the annotation box of the hand when the hand exists is obtained.

[0059] The target detection model may be, for example, YOLO, VITdet, etc. YOLO and ViTDet are two different target detection models. YOLO solves the target detection task as a regression problem, directly from image pixels to bounding box coordinates and class probabilities. ViTDet divides the image into a series of image blocks, then inputs these image blocks as a sequence into a Transformer model, and uses a self-attention mechanism to learn image features and perform target detection. The Transformer model is a deep learning architecture based on a self-attention mechanism.

[0060] Therefore, after obtaining the to-be-processed color image, the to-be-processed color image can be input into the hand recognition model to obtain the hand recognition result output by the hand recognition model.

[0061] The hand recognition model is used to identify whether a hand exists in the to-be-processed color image, so that when it is determined that the to-be-processed color image includes a hand and the confidence of the hand in the to-be-processed color image is lower than a preset confidence threshold due to occlusion or the like, a hand image can be generated based on the to-be-processed color image and hand pose estimation can be performed based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image. Further, the three-dimensional joint coordinate information can be used to accurately determine the bending angle information of the fingers of the hand, and the associated dexterous hand can be controlled based on the bending angle information. Since the three-dimensional joint coordinates of the hand can be determined based on the color image for dexterous hand control, there is no need to wear complex data acquisition equipment, which is suitable for low-cost deployment. Moreover, even if the confidence of the hand in the to-be-processed color image is lower than the preset confidence threshold due to occlusion or the like, the hand pose estimation can be performed based on the hand image generated from the color image, so that the three-dimensional joint coordinate information of the hand in the hand image can be estimated, and further, the dexterous hand can be controlled based on the three-dimensional joint coordinates, thereby improving the adaptability.

[0062] Based on the above embodiments, the hand image is generated based on the to-be-processed color image, which includes: The hand image is cropped from the to-be-processed color image based on the bounding box.

[0063] Specifically, since the hand recognition result further includes a bounding box of the hand in the to-be-processed color image, the hand image can be cropped from the to-be-processed color image based on the bounding box.

[0064] The present application accurately cuts the hand image from the color image to be processed, so that the hand posture estimation can be performed based on the hand image, and the three-dimensional joint coordinate information of the hand in the hand image is obtained; further, the bending angle information of the fingers of the hand can be accurately determined based on the three-dimensional joint coordinate information, and the associated dexterous hand can be controlled based on the bending angle information. Since the three-dimensional joint coordinates of the hand can be determined through the color image for dexterous hand control, it is not necessary to wear complex data acquisition equipment, which is suitable for low-cost deployment; and even if the confidence of the hand in the color image to be processed is lower than the pre-set confidence threshold due to occlusion or other reasons, the hand posture estimation can be performed through the hand image generated from the color image, so that the three-dimensional joint coordinate information of the hand in the hand image is estimated, and further the dexterous hand control can be performed according to the three-dimensional joint coordinates, so that the adaptability can be improved.

[0065] Based on the above embodiment, after obtaining the hand recognition result output by the hand recognition model, the following steps are further included: If the hand is included in the color image to be processed, and the confidence of the hand in the color image to be processed is greater than or equal to the pre-set confidence threshold, the depth image corresponding to the color image to be processed is obtained; The point cloud image is generated based on the depth image and the color image to be processed; The three-dimensional joint coordinate information of the hand in the color image to be processed is determined based on the point cloud image; The bending angle information of the fingers of the hand is determined based on the three-dimensional joint coordinate information; The associated dexterous hand is controlled based on the bending angle information.

[0066] Specifically, after obtaining the hand recognition result, if it is determined that the hand is included in the color image to be processed, and the confidence of the hand in the color image to be processed is greater than or equal to the pre-set confidence threshold, the depth image corresponding to the color image to be processed can be obtained.

[0067] It can be understood that if the depth image corresponding to the color image to be processed cannot be obtained, the above steps of generating the hand image based on the color image to be processed, performing the hand posture estimation based on the hand image to obtain the three-dimensional joint coordinate information of the hand in the hand image, determining the bending angle information of the fingers of the hand based on the three-dimensional joint coordinate information, and controlling the associated dexterous hand based on the bending angle information can be performed.

[0068] If the depth image corresponding to the color image to be processed is successfully obtained, the point cloud image can be generated based on the depth image and the color image to be processed.

[0069] In one embodiment, the present application can generate the point cloud image by the following steps: Obtaining intrinsic and extrinsic parameters: First, the intrinsic matrix of the camera (including focal length and principal point coordinates) and the extrinsic parameters between the two cameras (including rotation matrix and translation vector) corresponding to the two images need to be obtained. These parameters are usually obtained through the calibration process.

[0070] Depth map to three-dimensional point conversion: For each pixel point in the depth image, use the depth value and the intrinsic matrix to convert it from image coordinates to a three-dimensional point in the world coordinate system.

[0071] Coordinate system conversion: Convert the space point in the depth camera coordinate system to the RGB camera coordinate system, which can be obtained by multiplying the rotation matrix and adding the translation vector to get Prgb.

[0072] Three-dimensional point to RGB image coordinate conversion: Multiply the intrinsic matrix of the RGB camera by Prgb to get the pixel coordinates prgb in the RGB image.

[0073] Color assignment: According to the coordinates of prgb, the corresponding color value in the RGB image is taken out and assigned to the corresponding pixel point in the depth image, so as to obtain the depth image with color information, that is, the point cloud image.

[0074] Traversal and processing: Repeat the above steps for each pixel point in the depth image to finally obtain the entire aligned point cloud image.

[0075] Further, the present application can extract the three-dimensional coordinates of the joints of each finger in the hand of the color image to be processed from the point cloud image, and form three-dimensional joint coordinate information from the joint coordinate set of each finger. This process can be achieved by traditional technology or newly created technology, which is not limited in the present application.

[0076] After obtaining the three-dimensional joint coordinate information of the hand, the present application can approximate the bending angle of each finger by calculating the included angle between the joints of the finger to obtain the bending angle information of the finger.

[0077] Further, based on the bending angle information of the finger, the associated dexterous hand can be controlled to output the same hand posture, realizing the remote operation of the dexterous hand.

[0078] When the hand is included in the color image to be processed and the confidence of the hand in the color image to be processed is greater than or equal to the preset confidence threshold, the present application can determine the three-dimensional joint coordinate information of the hand in the color image to be processed according to the point cloud image generated from the depth image and the color image to be processed, accurately determine the bending angle information of the fingers of the hand based on the three-dimensional joint coordinate information, and control the associated dexterous hand based on the bending angle information. Therefore, the acquisition speed of the dexterous hand control information can be improved without reducing the accuracy.

[0079] Based on the above embodiments, the bending angle information of the fingers of the hand is determined based on the three-dimensional joint coordinate information, comprising: For each finger of the hand, the included angle is calculated according to the three-dimensional coordinates of the joint nodes of the corresponding finger in the three-dimensional joint coordinate information, to obtain the bending angle of the corresponding finger.

[0080] Specifically, when determining the bending angle information of the fingers of the hand based on the three-dimensional joint coordinate information, for each finger, the included angle can be calculated according to the three-dimensional coordinates of the joint nodes corresponding to the finger in the three-dimensional joint coordinate information.

[0081] Figure 2 is a schematic diagram of three-dimensional joint coordinate information in the dexterous hand control method provided by the present application, as Figure 2 shown, the three-dimensional joint coordinate information of the present application can include the joint nodes of all fingers in the hand (from joint node 0 to joint node 20).

[0082] For the little finger, index finger, middle finger and ring finger, there is only one degree of freedom on the 6-DOF dexterous hand, so the bending angle of each finger can be approximated by calculating the angle of the second joint of each finger.

[0083] Taking the little finger as an example, define the vector obtained by subtracting the three-dimensional coordinates of joint node 18 from the three-dimensional coordinates of joint node 17, the vector obtained by subtracting the three-dimensional coordinates of joint node 19 from the three-dimensional coordinates of joint node 18. The included angle between is the bending angle θ of the little finger joint. It can be calculated by the following formula: .

[0084] For the thumb, there are two degrees of freedom on the 6-DOF dexterous hand. Respectively represent the rotation angle of the thumb joint node 1 around the x axis , and the rotation angle of the thumb joint node 1 around the z axis. The calculation process is: define the vector obtained by subtracting the three-dimensional coordinates of joint node 0 from the three-dimensional coordinates of joint node 4, the included angle between the projection of the camera coordinate system yoz plane and the y axis of the camera coordinate system is equal to , the included angle between the projection of the camera coordinate system xoy plane and the y axis of the camera coordinate system is equal to

[0085] The present application can accurately determine the bending angle information of the fingers of the hand through the three-dimensional joint coordinate information, so that the associated dexterous hand can be controlled based on the bending angle information, and the accuracy of the dexterous hand control is improved.

[0086] Based on the above embodiments, the associated dexterous hand is controlled based on the bending angle information, which includes: The bending angle information is sent to the controller of the associated dexterous hand to instruct the controller to control the associated dexterous hand based on the bending angle information.

[0087] Specifically, after obtaining the bending angle information, the bending angle information can be sent to the controller of the associated dexterous hand.

[0088] After receiving the bending angle information, the controller of the corresponding dexterous hand can control each finger in the dexterous hand to move according to the bending angle of each finger in the bending angle information.

[0089] The application can control the associated dexterous hand based on the bending angle information, thereby improving the accuracy of dexterous hand control.

[0090] Figure 3 is the overall flowchart of the dexterous hand control method provided by the embodiments of the application, as Figure 3 shown, the dexterous hand control method of the application can include the following steps: Step S1.1: Obtain an RGB image; Step S1.2: Obtain a depth image; Step S2.1: Hand position positioning, specifically, a hand recognition algorithm can be used to recognize the hand in the RGB image, and when it is determined that there is a hand in the RGB image, the hand image can be cut out from the RGB image; Step S2.2: Point cloud map generation, align the depth image and the RGB image to obtain a point cloud map; Step S3.1: Hand feature extraction: for the RGB image with a confidence lower than the pre-set confidence threshold, the hand image is input into the hand recognition model, and the VIT model in the hand recognition model is used as the backbone of the feature extraction network. First, the image is cut into image blocks, then a combination of multiple transformer encoder layers and a full connection layer is used as a feature extraction module, and finally 61 model parameters are output. The first three parameters are camera-related parameters represented by Π, the middle 10 parameters are shape parameters represented by β, and the last 48 parameters are position parameters represented by θ; Step S3.2: Point cloud map mapping: for the image with a confidence higher than the pre-set confidence threshold, the three-dimensional joint coordinate information formed by the three-dimensional joint coordinates with the palm center as the origin is obtained through the mapping of the point cloud map; Step S4: Standard model generation, wherein the standard model is MANO in the hand recognition model; Step S5: For the image with low confidence, three-dimensional joint coordinate information obtained by hand pose estimation according to the hand image is utilized to realize key point coordinate generation by MANO output; Step S6: The bending angle of each finger is calculated by using the three-dimensional joint coordinate information, the joint angle mapping is completed, and the bending angle information is formed; Step S7: The bending angle information is sent to the controller of the dexterous hand, and the controller controls the movement of the dexterous hand according to the bending angle information.

[0091] The present application can realize accurate estimation of hand pose and control of dexterous hand by only relying on camera and image processing algorithm, which reduces the complexity and cost of teleoperation control system, and also helps to improve the portability and application range of the system.

[0092] The dexterous hand control device provided by the present application is described below, and the dexterous hand control device described below can be correspondingly referred to the dexterous hand control method described above.

[0093] Further, the present application also provides a dexterous hand control device.

[0094] The dexterous hand control device comprises: An acquisition module is configured to acquire a color image to be processed; the color image to be processed is associated with at least one dexterous hand; A generation module is configured to generate a hand image based on the color image to be processed if the color image to be processed includes a hand, and the confidence of the hand in the color image to be processed is lower than a preset confidence threshold; An evaluation module is configured to perform hand pose estimation based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image; A determination module is configured to determine bending angle information of the fingers of the hand based on the three-dimensional joint coordinate information; A control module is configured to control the associated dexterous hand based on the bending angle information.

[0095] The dexterous hand control device of the present application can generate a hand image based on the to-be-processed color image and perform hand posture estimation based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image when the hand is determined to be included in the to-be-processed color image and the confidence of the hand in the to-be-processed color image is lower than a preset confidence threshold due to occlusion or the like. The bending angle information of the fingers of the hand can be accurately determined based on the three-dimensional joint coordinate information, and the associated dexterous hand can be controlled based on the bending angle information. Since the three-dimensional joint coordinates of the hand can be determined based on the color image for dexterous hand control, there is no need to wear complex data acquisition equipment, which is suitable for low-cost deployment. Even if the confidence of the hand in the to-be-processed color image is lower than the preset confidence threshold due to occlusion or the like, the hand posture estimation can be performed based on the hand image generated from the color image, the three-dimensional joint coordinate information of the hand in the hand image can be estimated, and the dexterous hand control can be further performed based on the three-dimensional joint coordinates, thereby improving the adaptability.

[0096] In one embodiment, the obtaining module is further configured to: input the to-be-processed color image into a hand recognition model to obtain a hand recognition result output by the hand recognition model; wherein the hand recognition result includes whether a hand exists in the to-be-processed color image and a confidence of the hand when the hand exists; and the hand recognition model is trained based on sample color images, labels of whether a hand exists in the sample color images, and annotated images including a hand annotation frame when a hand exists in the sample color images.

[0097] In one embodiment, the generating module is specifically configured to: cut a hand image from the to-be-processed color image based on the annotation frame.

[0098] In one embodiment, the obtaining module is further configured to: if the to-be-processed color image includes a hand and the confidence of the hand in the to-be-processed color image is greater than or equal to a preset confidence threshold, obtain a depth image corresponding to the to-be-processed color image; generate a point cloud image based on the depth image and the to-be-processed color image; determine three-dimensional joint coordinate information of the hand in the to-be-processed color image based on the point cloud image; determine bending angle information of the fingers of the hand based on the three-dimensional joint coordinate information; control the associated dexterous hand based on the bending angle information.

[0099] In one embodiment, the obtaining module is further configured to: For each finger of the hand, the included angle is calculated based on the three-dimensional coordinates of the joint points of the corresponding finger in the three-dimensional joint coordinate information to obtain the bending angle of the corresponding finger.

[0100] In one embodiment, the acquisition module is further configured to: The bending angle information is sent to the controller of the associated dexterous hand to instruct the controller to control the associated dexterous hand based on the bending angle information.

[0101] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the following method: acquiring a color image to be processed; the color image to be processed is associated with at least one dexterous hand; If the color image to be processed includes a hand, and the confidence level of the hand in the color image to be processed is lower than a preset confidence threshold, a hand image is generated based on the color image to be processed. Based on the hand image, hand pose estimation is performed to obtain the three-dimensional joint coordinate information of the hand in the hand image; The bending angle information of the fingers of the hand is determined based on the three-dimensional joint coordinate information; The associated dexterous hand is controlled based on the bending angle information.

[0102] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] In another aspect, the embodiments of the present application also provide a non-transitory computer-readable storage medium, which has stored thereon a computer program, and the computer program is executed by a processor to implement the method provided by the above embodiments, for example, including: obtaining a color image to be processed; the color image to be processed is associated with at least one dexterous hand; if the color image to be processed includes a hand, and a confidence of the hand in the color image to be processed is lower than a preset confidence threshold, generating a hand image based on the color image to be processed; performing hand pose estimation based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image; determining bending angle information of fingers of the hand based on the three-dimensional joint coordinate information; controlling the associated dexterous hand based on the bending angle information.

[0104] In another aspect, the embodiments of the present application also provide a computer program product, which has stored thereon a computer program, and the computer program is executed by a processor to implement the method provided by the above embodiments, for example, including: obtaining a color image to be processed; the color image to be processed is associated with at least one dexterous hand; if the color image to be processed includes a hand, and a confidence of the hand in the color image to be processed is lower than a preset confidence threshold, generating a hand image based on the color image to be processed; performing hand pose estimation based on the hand image to obtain three-dimensional joint coordinate information of the hand in the hand image; determining bending angle information of fingers of the hand based on the three-dimensional joint coordinate information; controlling the associated dexterous hand based on the bending angle information.

[0105] The above-mentioned apparatus embodiments are only illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments. Those skilled in the art can understand and implement it without creative labor.

[0106] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the description of the above embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the present application, and are not limiting to the present application. Although the present application is described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications or equivalent replacements of the technical solutions of the present application do not deviate from the spirit and scope of the present application.

Claims

1. A method for controlling a dexterous hand, characterized in that, include: Acquire a color image to be processed; the color image to be processed is associated with at least one dexterous hand; If the color image to be processed includes a hand, and the confidence level of the hand in the color image to be processed is lower than a preset confidence threshold, a hand image is generated based on the color image to be processed. Based on the hand image, hand pose estimation is performed to obtain the three-dimensional joint coordinate information of the hand in the hand image; The bending angle information of the fingers of the hand is determined based on the three-dimensional joint coordinate information; The associated dexterous hand is controlled based on the bending angle information.

2. The dexterous hand control method according to claim 1, characterized in that, After acquiring the color image to be processed, the process also includes: The color image to be processed is input into the hand recognition model to obtain the hand recognition result output by the hand recognition model; wherein, the hand recognition result includes whether a hand exists in the color image to be processed and the confidence level of the hand when the hand exists; the hand recognition model is trained based on sample color images, labels of whether a hand exists in the sample color images, and labeled images including hand bounding boxes when a hand exists in the sample color images.

3. The dexterous hand control method according to claim 2, characterized in that, The hand recognition result also includes bounding boxes for the hand in the color image to be processed; the step of generating a hand image based on the color image to be processed includes: Based on the labeled box, a hand image is extracted from the color image to be processed.

4. The dexterous hand control method according to claim 2, characterized in that, After obtaining the hand recognition result output by the hand recognition model, the process also includes: If the color image to be processed includes a hand, and the confidence level of the hand in the color image to be processed is greater than or equal to a preset confidence threshold, then obtain the depth image corresponding to the color image to be processed. A point cloud image is generated based on the depth image and the color image to be processed; Based on the point cloud image, determine the three-dimensional joint coordinate information of the hand in the color image to be processed; The bending angle information of the fingers of the hand is determined based on the three-dimensional joint coordinate information; The associated dexterous hand is controlled based on the bending angle information.

5. The dexterous hand control method according to claim 4, characterized in that, The process of determining the finger bending angle information of the hand based on the three-dimensional joint coordinate information includes: For each finger of the hand, the included angle is calculated based on the three-dimensional coordinates of the joint points of the corresponding finger in the three-dimensional joint coordinate information to obtain the bending angle of the corresponding finger.

6. The dexterous hand control method according to claim 4, characterized in that, The control of the associated dexterous hand based on the bending angle information includes: The bending angle information is sent to the controller of the associated dexterous hand to instruct the controller to control the associated dexterous hand based on the bending angle information.

7. A dexterous hand control device, characterized in that, include: An acquisition module is used to acquire a color image to be processed; the color image to be processed is associated with at least one dexterous hand; The generation module is used to generate a hand image based on the color image to be processed if the color image to be processed includes a hand and the confidence level of the hand in the color image to be processed is lower than a preset confidence threshold. An evaluation module is used to estimate the hand pose based on the hand image to obtain the three-dimensional joint coordinate information of the hand in the hand image; The determination module is used to determine the bending angle information of the fingers of the hand based on the three-dimensional joint coordinate information; A control module is used to control the associated dexterous hand based on the bending angle information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the dexterous hand control method as described in any one of claims 1-6.

9. A storage medium, said storage medium being a non-transitory computer-readable storage medium, wherein a computer program is stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dexterous hand control method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the dexterous hand control method according to any one of claims 1-6.

Images