A system, method and apparatus for constructing a dexterous hand training sample
By acquiring training sample data of dexterity hand using image and tactile sensors, the problem of insufficient training sample accuracy in existing methods is solved, and high-precision training of dexterity hand in functional movements is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PASSINI PERCEPTION TECH (SHENZHEN) CO LTD
- Filing Date
- 2025-01-10
- Publication Date
- 2026-07-10
AI Technical Summary
Existing methods for constructing training samples for dexterous hands cannot meet the requirements for dexterous hands to perform functional movements. Training sample data collected in simulation environments cannot fully simulate real environments, and direct data mapping has significant errors.
The system, consisting of an instruction generation device, an image sensor, a tactile sensor, and a controller, estimates the pose of the target object and the dexterous hand through image data, calculates the relative pose, and acquires the bending angle and tactile information of the mechanical finger, which is then saved as training samples.
It improves the accuracy of training samples, which can meet the requirements of dexterous hands to perform functional movements, reduces data conversion errors, and improves the accuracy of relative pose.
Smart Images
Figure CN122353666A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robotic arm technology, and in particular to a system, method and apparatus for constructing training samples for dexterous hands. Background Technology
[0002] A dexterous hand is a humanoid robotic hand with a palm and fingers. It can not only grasp objects but also perform complex functional actions after grasping different objects, such as scanning barcodes with a moving barcode scanner, unscrewing a water bottle cap, operating a mouse, and operating a remote control. Understandably, to enable a dexterous hand to perform these functional actions, it is necessary to accurately construct training samples to ensure that the relative pose, tactile information, and motion information of the dexterous hand when grasping objects correspond to the required functional actions.
[0003] Most existing methods for constructing dexterous hand training samples generate training sample data using reinforcement learning in a simulation environment, such as UniDexGrasp and MultiDex. However, since training sample data collected in a simulation environment cannot fully simulate the real environment (e.g., it cannot collect tactile information), it cannot meet the requirements for dexterous hands to perform functional movements. Other methods construct dexterous hand training samples by directly mapping data of the human hand grasping objects onto the dexterous hand. However, training samples constructed in this way are prone to significant errors due to the data transformation process, and still cannot meet the requirements for dexterous hands to perform functional movements.
[0004] It is evident that existing methods for constructing training samples for dexterous hands cannot meet the requirements for dexterous hands to perform functional movements. Summary of the Invention
[0005] The purpose of this application is to provide a system, method, and apparatus for constructing dexterity training samples, so as to solve the technical problem that existing methods for constructing dexterity training samples cannot meet the requirements for dexterity to achieve functional movements.
[0006] In a first aspect, embodiments of this application provide a dexterous hand grasping data acquisition system, the grasping data acquisition system comprising: an instruction generation device, an image sensor, a dexterous hand, and a controller, the dexterous hand being provided with a tactile sensor and flexible mechanical fingers, the instruction generation device, the image sensor, and the tactile sensor being communicatively connected to the controller, and the controller being communicatively connected to the dexterous hand;
[0007] The instruction generation device is used to generate a first motion instruction and a second motion instruction;
[0008] The image sensor is used to acquire image data of the target object and image data of the dexterous hand;
[0009] The controller is used to acquire the target object image data and estimate the target object pose based on the target object image data;
[0010] Obtain the first motion command to instruct the dexterous hand to move to a grasping position close to the target object;
[0011] Acquire the dexterous hand image data, and estimate the dexterous hand pose at the grasping position based on the dexterous hand image data;
[0012] The relative pose of the dexterous hand to the target object is calculated based on the pose of the target object and the pose of the dexterous hand when the dexterous hand is in the grasping position.
[0013] The second motion command is obtained to instruct the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object;
[0014] Calculate the bending angle value of the mechanical finger when it bends to the target grasping position, and obtain the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position;
[0015] The relative pose, the bending angle value, and the tactile information are saved as training samples for the dexterous hand.
[0016] Optionally, it also includes a drive device, the dexterous hand is mounted on the drive device, and the controller is communicatively connected to the drive device. The controller instructs the drive device to move the dexterous hand to a grasping position close to the target object through the first motion command.
[0017] And / or, the system for constructing dexterous hand training samples further includes a scanning device for scanning a target object to obtain a 3D model of the target object, and for scanning the dexterous hand to obtain a 3D model of the dexterous hand, the scanning device being communicatively connected to the controller;
[0018] And / or, the instruction generating device is a remotely operated device.
[0019] Secondly, embodiments of this application provide a method for constructing training samples for a dexterous hand, applied to a dexterous hand grasping data acquisition system. The grasping data acquisition system includes: an instruction generation device, an image sensor, a dexterous hand, and a controller. The dexterous hand is equipped with a tactile sensor and flexible mechanical fingers. The method includes:
[0020] Acquire target object image data, and estimate the target object pose based on the target object image data;
[0021] Obtain a first motion command to instruct the dexterous hand to move to a grasping position close to the target object;
[0022] Acquire dexterity hand image data, and estimate the dexterity hand pose at the grasping position based on the dexterity hand image data;
[0023] The relative pose of the dexterous hand to the target object is calculated based on the pose of the target object and the pose of the dexterous hand when the dexterous hand is in the grasping position.
[0024] Obtain a second motion command to instruct the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object;
[0025] Calculate the bending angle value of the mechanical finger when it bends to the target grasping position, and obtain the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position;
[0026] The relative pose, the bending angle value, and the tactile information are saved as training samples for the dexterous hand.
[0027] Optionally, estimating the pose of the target object based on the target object image data specifically includes:
[0028] The pose of the target object is estimated based on the target object image data using a preset pose estimation model.
[0029] The step of estimating the dexterity hand pose at the grasping position based on the dexterity hand image data specifically includes:
[0030] The pre-defined pose estimation model is used to estimate the pose of the dexterous hand at the grasping position based on the dexterous hand image data.
[0031] Optionally, acquiring the target object image data and estimating the target object pose based on the target object image data includes the following steps:
[0032] The target object image data is acquired multiple times, and the initial object pose of multiple frames is estimated based on the acquired target object image data.
[0033] The target object pose is obtained based on the initial object pose in multiple frames.
[0034] And / or, acquiring dexterous hand image data and estimating the dexterous hand pose at the grasping position based on the dexterous hand image data includes the following steps:
[0035] The dexterous hand image data is acquired multiple times, and the initial hand pose of multiple frames is estimated based on the acquired dexterous hand image data.
[0036] The dexterous hand pose is obtained based on the initial hand pose in multiple frames.
[0037] Optionally, the step of determining the target object pose based on the initial object pose across multiple frames includes the following steps:
[0038] Remove outliers from the initial object pose in multiple frames;
[0039] The average value of the initial object pose in multiple frames after removing outliers is calculated to obtain the average object pose, and the target object pose is obtained based on the average object pose.
[0040] The process of obtaining the dexterous hand pose based on the initial hand pose in multiple frames includes the following steps:
[0041] Remove outliers from the initial hand pose in multiple frames;
[0042] The average value of the initial hand pose is calculated for multiple frames after outliers are removed to obtain the average hand pose, and the dexterous hand pose is obtained based on the average hand pose.
[0043] Optionally, the step of obtaining the target object pose based on the average pose of the object includes the following steps:
[0044] Obtain a 3D model of the target object;
[0045] A first pixel plane is generated based on the target object image data, and the first pixel plane contains the segmentation result of the target object;
[0046] Based on the average pose of the object, the 3D model of the target object is projected onto the first pixel plane, and the first intersection-union ratio of the 3D model of the target object and the segmentation result of the target object is calculated;
[0047] Determine whether the first intersection-to-union ratio reaches a first preset value; if the first intersection-to-union ratio reaches the first preset value, use the average pose of the object as the pose of the target object.
[0048] The process of determining the dexterity hand position based on the average value of the hand position includes the following steps:
[0049] Obtain a 3D model of a dexterous hand;
[0050] A second pixel plane is generated based on the dexterous hand image data, the second pixel plane containing the segmentation result of the dexterous hand;
[0051] Based on the average pose of the hand, the 3D model of the dexterous hand is projected onto the second pixel plane, and the second intersection-union ratio of the 3D model of the dexterous hand and the segmentation result of the dexterous hand is calculated;
[0052] Determine whether the second crossover ratio reaches the second preset value. If the second crossover ratio reaches the second preset value, the average value of the hand pose is taken as the dexterous hand pose.
[0053] Optionally, before instructing the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object via the second motion command, the method further includes the following steps:
[0054] The initial value is set to the bending angle of the mechanical finger when the dexterous hand is located at the grasping position.
[0055] Optionally, the system for constructing dexterous hand training samples further includes a driving device, the dexterous hand is mounted on the driving device, and the controller is communicatively connected to the driving device. The controller instructs the driving device to move the dexterous hand to a grasping position close to the target object through the first motion command.
[0056] And / or, the system for constructing dexterous hand training samples further includes a scanning device for scanning a target object to obtain a 3D model of the target object, and for scanning the dexterous hand to obtain a 3D model of the dexterous hand, the scanning device being communicatively connected to the controller;
[0057] And / or, the instruction generating device is a remotely operated device.
[0058] Thirdly, embodiments of this application provide an apparatus for constructing dexterous hand training samples, the apparatus comprising:
[0059] The object pose estimation module is used to acquire target object image data and estimate the target object pose based on the target object image data.
[0060] The first motion instruction module is used to acquire a first motion instruction, so as to instruct the dexterous hand to move to a grasping position close to the target object through the first motion instruction;
[0061] The hand pose estimation module is used to acquire dexterous hand image data and estimate the dexterous hand pose of the dexterous hand at the grasping position based on the dexterous hand image data.
[0062] The pose calculation module is used to calculate the relative pose of the dexterous hand relative to the target object when it is located at the grasping position, based on the pose of the target object and the pose of the dexterous hand.
[0063] The second motion instruction module is used to acquire a second motion instruction, so as to instruct the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform a functional action on the target object.
[0064] The information acquisition module is used to calculate the bending angle value of the mechanical finger when it bends to the target grasping position, and to acquire the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position;
[0065] The data storage module is used to save the relative pose, the bending angle value, and the tactile information as training samples for the dexterous hand.
[0066] Compared with the prior art, the embodiments of this application have the following main advantages:
[0067] This application embodiment estimates the target object pose based on the target object image data, estimates the dexterity hand pose at the grasping position based on the dexterity hand image data, and calculates the relative pose of the dexterity hand relative to the target object when it is at the grasping position based on the target object pose and the dexterity hand pose. Since the target object pose and the dexterity hand pose are only converted by the image sensor, and the relative pose is directly calculated from the two, the error of the relative pose is small and the accuracy is higher. Furthermore, by instructing the mechanical finger to bend to the target grasping position that enables the dexterity hand to perform functional actions on the target object through the second motion command, the bending angle value of the mechanical finger and the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position can be obtained. Thus, the above-mentioned relative pose, bending angle value and tactile information can be saved as training samples of the dexterity hand, so that the training samples of the dexterity hand meet the requirements of the dexterity hand to perform functional actions, solving the technical problem that the method of constructing training samples of the dexterity hand cannot meet the requirements of the dexterity hand to perform functional actions. Attached Figure Description
[0068] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0069] Figure 1 This is a schematic diagram of a structure that uses a dexterous hand to grasp a barcode scanner for scanning.
[0070] Figure 2 This is a structural block diagram of the system for constructing dexterous hand training samples provided in the embodiments of this application;
[0071] Figure 3This is a flowchart illustrating the method for constructing dexterous hand training samples provided in an embodiment of this application;
[0072] Figure 4 This is a schematic diagram of the structure of the apparatus for constructing dexterous hand training samples provided in an embodiment of this application;
[0073] Figure 5 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application.
[0074] Explanation of reference numerals in the attached figures:
[0075] 100. Instruction generation device; 200. Image sensor; 300. Dexterous hand; 310. Mechanical finger; 400. Controller; 500. Tactile sensor; 600. Device for constructing training samples for dexterous hand; 610. Object pose estimation module; 620. First motion instruction module; 630. Hand pose estimation module; 640. Pose calculation module; 650. Second motion instruction module; 660. Information acquisition module; 670. Data storage module; 700. Computer equipment; 710. Memory; 720. Processor; 730. Network interface. Detailed Implementation
[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.
[0077] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0078] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0079] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0080] Please see Figure 1-2 The first part of the embodiments of this application provides a system for constructing training samples of a dexterous hand, the system including: instruction generation device 100, image sensor 200, dexterous hand 300 and controller 400.
[0081] Among them, the dexterous hand 300 refers to an end effector typically mounted on a robot's robotic arm, used to grasp target objects and perform functional actions on them after grasping them. For example, the dexterous hand can be used to perform the functional action of moving the barcode scanner to scan codes by pressing a button after grasping the barcode scanner (see...). Figure 1 This device can be used to perform functional actions such as unscrewing the cap of a water bottle after grasping it, operating a mouse after grasping it, and operating a remote control after grasping it. However, the dexterous hand 300 in this application is not limited to being mounted on the robotic arm of a robot; it can also be mounted on, for example, the drive mechanism of a mobile robot platform or experimental equipment, as long as the dexterous hand 300 can move when subjected to a driving force.
[0082] In this embodiment, the dexterous hand 300 is provided with a flexible mechanical finger 310 to realize the function of grasping objects, and is provided with a tactile sensor 500 to collect tactile information when grasping objects.
[0083] The instruction generation device 100 is used to generate a first motion instruction to send the first motion instruction to the controller 400, and to generate a second motion instruction to send the second motion instruction to the controller 400.
[0084] It should be noted that the first motion command and the second motion command generated by the command generation device 100 can be commands that the controller 400 can directly execute after acquiring them, or they can be data information that the controller 400 needs to calculate or convert before it can execute the corresponding commands. For example, the command generation device 100 can be a computing terminal device that pre-stores the first motion command and the second motion command, so that it can directly send the first motion command and the second motion command that can be executed by the controller 400; the command generation device 100 can also be a remote operation device, which generates the first motion data and the second motion data through the actual operation of the operator. The first motion data and the second motion data here are the first motion command and the second motion command mentioned above. After acquiring the first motion data and the second motion data, the controller 400 also needs to convert them into corresponding commands that it can execute.
[0085] Image sensor 200 is used to acquire image data of a target object and send the target object image data to controller 400. The target object image data refers to image information containing the target object, and may include color image information and / or depth image information of the target object.
[0086] The image sensor 200 is also used to acquire dexterous hand image data to send the dexterous hand image data to the controller 400. The dexterous hand image data refers to image information including the dexterous hand 300 located at the grasping position described below. The dexterous hand image data may also include color image information and / or depth image information of the dexterous hand 300.
[0087] The instruction generation device 100, image sensor 200, and tactile sensor 500 of the dexterous hand 300 are all communicatively connected to the controller 400, enabling the controller 400 to receive information sent by the instruction generation device 100, image sensor 200, and tactile sensor 500 respectively. This allows the controller 400 to acquire the first and second motion commands, target object image data, dexterous hand image data, and tactile information. The controller 400 is configured to communicate with the dexterous hand 300, enabling the controller 400 to control the dexterous hand 300 to perform movements. The controller 400 is used to implement the method for constructing dexterous hand training samples provided in the second part of the embodiments of this application.
[0088] It should be noted that the controller 400 in this application embodiment can be integrated entirely or partially on the dexterous hand 300, or it can be integrated entirely or partially on other devices, such as the instruction generation device 100, the image sensor 200, or the driving device or scanning device described below. This application does not specifically limit it, as long as the controller 400 can realize the various communication connections described in the embodiments of this application.
[0089] The controller 400 in this application embodiment may be, but is not limited to: a computer terminal (PC); an industrial personal computer terminal (IPC); a mobile terminal; a server; a system including a terminal and a server, and implemented through the interaction between the terminal and the server; a programmable logic controller (PLC); a field-programmable gate array (FPGA); a digital signal processor (DSP) or a microcontroller unit (controller unit), etc.
[0090] In one embodiment, the system for constructing dexterous hand training samples further includes a driving device, on which the dexterous hand 300 is mounted, and a controller 400 is communicatively connected to the driving device. The controller 400 instructs the driving device to move via a first motion command, thereby causing the driving device to move the dexterous hand 300 to a position close to the target object for grasping.
[0091] In this embodiment of the application, by directly mounting the dexterous hand 300 onto the driving device, the driving device can drive the dexterous hand 300 to move, without limiting the dexterous hand 300 to be mounted on the robot's robotic arm. This allows the process of using the system described in this embodiment of the application to construct dexterous hand training samples to be realized independently of the robotic arm and the robot. It also allows the subsequent training process of the dexterous hand 300 using training samples and the process of performing corresponding functional actions on the target object after training to be realized independently of the robotic arm and the robot.
[0092] For example, the driving device can be a driving device for a robotic arm or a mobile robot platform.
[0093] In one embodiment, the system for constructing dexterity hand training samples further includes a scanning device for scanning a target object to obtain a 3D model of the target object, and for scanning the dexterity hand 300 to obtain a 3D model of the dexterity hand. The scanning device is communicatively connected to a controller 400 to send the 3D model of the target object and the 3D model of the dexterity hand to the controller 400, thereby enabling the controller 400 to determine the pose of the target object based on the target object image data and the target object's 3D model after acquiring the target object image data; and enabling the controller 400 to determine the dexterity hand pose of the dexterity hand 300 at the grasping position based on the dexterity hand image data and the dexterity hand's 3D model after acquiring the dexterity hand image data.
[0094] In one embodiment, the instruction generation device 100 is a remotely operated device.
[0095] Specifically, the teleoperation device includes an operating device and an information interaction device. The operating device can be, for example, a wearable device (such as an exoskeleton or data glove), a motion-sensing device (such as a motion-sensing camera), a touch interface, and a control handle. The user (i.e., the operator) of the teleoperation device performs various actions using the operating device. The information interaction device is used to collect motion information of the operator's various actions in real time and send this motion information to the control device used to control the movements of the dexterous hand 300, so that the dexterous hand 300 can follow the operator's movements based on the motion information. The information interaction device may include, for example, a detector for detecting motion information and a transmitter for sending motion information. In this embodiment, the controller 400 is the control device described herein, and the first motion command and the second motion command are essentially the motion information described herein.
[0096] Understandably, by using the remote operation device as the instruction generation device 100, the dexterous hand 300 can follow the operator's actions in real time, enabling the dexterous hand 300 to successfully complete various functional actions for various target objects, thereby ensuring that the training samples of the dexterous hand 300 constructed according to the embodiments of this application can meet the requirements for the dexterous hand 300 to achieve functional actions.
[0097] Please see Figure 3 The second part of the embodiments of this application provides a method for constructing dexterity training samples, which is applied to the system for constructing dexterity training samples described above. The method can be executed by the controller 400 of the system for constructing dexterity training samples described above.
[0098] The method for constructing dexterous hand training samples includes:
[0099] Step S100: Obtain target object image data, and estimate the target object pose based on the target object image data;
[0100] Step S200: Obtain a first motion command to instruct the dexterous hand to move to a grasping position close to the target object;
[0101] Step S300: Obtain dexterous hand image data, and estimate the dexterous hand pose at the grasping position based on the dexterous hand image data;
[0102] Step S400: Calculate the relative pose of the dexterous hand to the target object when it is in the grasping position based on the pose of the target object and the pose of the dexterous hand;
[0103] Step S500: Obtain a second motion command to instruct the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object.
[0104] Step S600: Calculate the bending angle value of the mechanical finger when it bends to the target grasping position, and obtain the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position;
[0105] Step S700: Save the relative pose, bending angle value and tactile information as training samples for the dexterous hand.
[0106] It should be noted that the target object image data in step S100 and the dexterous hand image data in step S300 can both be acquired by the image sensor 200 of the system for constructing dexterous hand training samples described above. The grasping position in step S300 refers to the position of the dexterous hand itself within a preset range near the target object. Once the dexterous hand is in the grasping position, it can stably grasp the target object and perform functional actions on it simply by bending the mechanical fingers of the dexterous hand. The target grasping position in steps S500 and S600 refers to the bending position of the mechanical fingers of the dexterous hand when it stably grasps the target object and can perform functional actions on it. In other words, the grasping position is the position of the dexterous hand itself that allows its mechanical fingers to bend and move to the target grasping position. In this embodiment, the grasping position can be a single, fixed position or a series of uncertain positions; correspondingly, the target grasping position can also be a single, fixed position or a series of uncertain positions. For example, when the target object is a circular symmetrical object, the grasping position can be a series of positions within a preset radius near the target object, and this series of grasping positions corresponds to a series of target grasping positions. The bending angle value obtained in step S600 refers to the magnitude of the bending angle generated by each joint of the mechanical finger during the bending movement from the grasping position to the target grasping position when the dexterous hand is in the grasping position. The specific method for obtaining the bending angle value can be achieved by measuring it using an encoder installed on the motor of the corresponding joint inside the mechanical finger. Understandably, by obtaining the bending angle value of the mechanical finger, the bending movement process of the mechanical finger can be completely recorded, thereby enabling control of the bending movement process of the mechanical finger by adjusting the bending angle value.
[0107] This application embodiment estimates the target object pose based on the target object image data, estimates the dexterity hand pose at the grasping position based on the dexterity hand image data, and calculates the relative pose of the dexterity hand relative to the target object when it is at the grasping position based on the target object pose and the dexterity hand pose. Since the target object pose and the dexterity hand pose are only converted by the image sensor, and the relative pose is directly calculated from the two, the error of the relative pose is small and the accuracy is higher. Furthermore, by instructing the mechanical finger to bend to the target grasping position that enables the dexterity hand to perform functional actions on the target object through the second motion command, the bending angle value of the mechanical finger and the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position can be obtained. Thus, the above-mentioned relative pose, bending angle value and tactile information can be saved as training samples of the dexterity hand, so that the training samples of the dexterity hand meet the requirements of the dexterity hand to perform functional actions, solving the technical problem that the method of constructing training samples for the dexterity hand cannot meet the requirements of the dexterity hand to perform functional actions.
[0108] In one embodiment, before step S500, instructing the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object via a second motion command, the following steps are further included:
[0109] The initial value is set to the bending angle of the mechanical finger when the dexterous hand is in the grasping position.
[0110] Understandably, by first setting the bending angle of the mechanical finger at the position to be grasped by the dexterous hand as an initial value, and then executing steps S500 and S600 to obtain the bending angle value of the mechanical finger, since the bending angle value of the mechanical finger has been set to the initial value before executing steps S500 and S600, the bending angle value of the mechanical finger can be obtained more quickly, and computational resources are saved.
[0111] For example, the initial value can be set to zero.
[0112] In one embodiment, before step S200 instructs the dexterous hand to move to a grasping position close to the target object via a first motion command, the following steps are further included:
[0113] The target object type is determined based on the target object image data, and the position to be captured is set according to the target object type.
[0114] Understandably, by first determining the target object type based on its image data and then setting the grasping position accordingly, the determined grasping position better aligns with the functional movements required by the dexterous hand. This means the mechanical fingers of the dexterous hand at the grasping position can more easily bend and move to the target grasping position, resulting in a training sample that better meets the needs of the dexterous hand for performing functional movements. For example, the shape and size of the target object can be obtained from its image data, and the grasping position can be set based on this. For instance, if the target object's shape and size are similar to a water bottle, the grasping position can be set to a distance of 1 cm between the dexterous hand's palm and the target object.
[0115] In one embodiment, step S100, which estimates the pose of the target object based on the target object image data, specifically includes: estimating the pose of the target object based on the target object image data using a preset pose estimation model; step S300, which estimates the pose of the dexterous hand 300 at the grasping position based on the dexterous hand image data, specifically includes: estimating the pose of the dexterous hand 300 at the grasping position using a preset pose estimation model based on the dexterous hand image data.
[0116] Understandably, in this embodiment, because the same pose estimation model (i.e., the preset pose estimation model) is used to estimate the target object pose and the dexterous hand pose, the adverse effects caused by the difference in errors between different pose estimation models are reduced. This results in higher accuracy of the final calculated relative pose and also saves, to some extent, the computational resources required for both object pose estimation and dexterous hand pose estimation, thus improving computational efficiency. For example, the preset pose estimation model may be either the ZebraPose algorithm or the FoundationPose algorithm.
[0117] It should be noted that when using the ZebraPose algorithm model to estimate the pose of a target object, it is necessary to first obtain the 3D model of the target object, the color image dataset, and the pose of the target object in the color image. The ZebraPose algorithm model can only be used after pre-training, and the target object image data must only include the color image information of the target object. Similarly, when using the ZebraPose algorithm model to estimate the pose of a dexterous hand, it is necessary to first obtain the 3D model of the dexterous hand, the color image dataset, and the pose of the dexterous hand in the color image. The ZebraPose algorithm model can only be used after pre-training, and the dexterous hand image data must only include the color image information of the dexterous hand.
[0118] When using the FoundationPose algorithm model to estimate the pose of a target object, a 3D model of the target object must be obtained first, and the target object image data must include both color image information and depth image information. Similarly, when using the FoundationPose algorithm model to estimate the pose of a dexterous hand, a 3D model of the dexterous hand must be obtained first, and the dexterous hand image data must include both color image information and depth image information of the dexterous hand.
[0119] In one embodiment, step S100 includes the following steps:
[0120] S110. Acquire target object image data multiple times, and estimate the initial object pose of multiple frames based on the acquired target object image data.
[0121] S120. Obtain the target object pose based on the initial object pose of multiple frames.
[0122] Step S300 includes the following steps:
[0123] S310. Acquire dexterous hand image data multiple times, and estimate the initial hand pose of multiple frames based on the acquired dexterous hand image data;
[0124] S320, Dexterity hand pose is obtained based on multiple initial hand poses.
[0125] In step S110, acquiring multiple target object image data can be achieved by having the image sensor 200 acquire target object image data multiple times within a first preset time period. Similarly, in step S310, acquiring multiple dexterous hand image data can be achieved by having the image sensor 200 acquire dexterous hand image data multiple times within a second preset time period. The first and second preset time periods can be the same or different; the number of times the target object image data is acquired and the number of times the dexterous hand image data is acquired can be the same or different.
[0126] Understandably, in this embodiment, estimating the target object pose based on multiple acquired target object image data and estimating the dexterous hand pose based on multiple acquired dexterous hand image data can achieve higher accuracy and smaller error for the target object pose and dexterous hand pose respectively, compared to estimating based on a single acquired target object image data and a single acquired dexterous hand image data.
[0127] In one embodiment, step S120 includes the following steps:
[0128] S121. Remove outliers from the initial object poses of multiple frames;
[0129] S122. Calculate the average value of the initial object pose in multiple frames after removing outliers to obtain the average object pose, and obtain the target object pose based on the average object pose.
[0130] Step S320 includes the following steps:
[0131] S321. Remove outliers from the initial hand poses in multiple frames;
[0132] S322. Calculate the average value of the initial hand pose in multiple frames after removing outliers to obtain the average hand pose, and obtain the dexterous hand pose based on the average hand pose.
[0133] It should be noted that the outlier removal in the initial object pose of multiple frames in step S121 refers to the removal of the initial object pose in multiple frames where the displacement or rotation has extremely large (or small) values; the outlier removal in the initial hand pose of multiple frames in step S321 refers to the removal of the initial hand pose in multiple frames where the displacement or rotation has extremely large (or small) values.
[0134] Understandably, in this embodiment, by calculating the average object pose of multiple initial object poses and obtaining the target object pose based on the average object pose, the accuracy of the target object pose can be improved; by calculating the average hand pose of multiple initial hand poses and obtaining the dexterous hand pose based on the average hand pose, the accuracy of the dexterous hand pose can be improved.
[0135] In one embodiment, step S122 includes the following steps:
[0136] S1221. Obtain the 3D model of the target object;
[0137] S1222. Generate a first pixel plane based on the target object image data. The first pixel plane contains the segmentation result of the target object.
[0138] S1223. Project the 3D model of the target object onto the first pixel plane based on the average pose of the object, and calculate the first intersection-union ratio between the 3D model of the target object and the segmentation result of the target object.
[0139] S1224. Determine whether the first crossover ratio reaches the first preset value. When the first crossover ratio reaches the first preset value, use the average pose of the object as the pose of the target object.
[0140] Step S322 includes the following steps:
[0141] S3221. Obtain a 3D model of the dexterous hand;
[0142] S3222. Generate a second pixel plane based on the dexterous hand image data. The second pixel plane contains the segmentation results of the dexterous hand.
[0143] S3223. Based on the average pose of the hand parts, project the 3D model of the dexterous hand onto the second pixel plane, and calculate the second intersection-union ratio between the 3D model of the dexterous hand and the segmentation result of the dexterous hand.
[0144] S3224. Determine whether the second crossover ratio reaches the second preset value. When the second crossover ratio reaches the second preset value, take the average hand position as the dexterous hand position.
[0145] It should be noted that the 3D model of the target object in step S1221 and the 3D model of the dexterous hand in step S3221 can both be obtained through the scanning device of the system for constructing dexterous hand training samples described above. The first intersection-union ratio (IU) represents the degree of overlap between the 3D model of the target object projected onto the first pixel plane based on the average object pose and the segmentation result of the target object. By judging whether the first IU reaches the first preset value, the average object pose is used as the target object pose only when the first IU reaches the first preset value, thus making the reliability of the target object pose higher. The second IU represents the degree of overlap between the 3D model of the dexterous hand projected onto the second pixel plane based on the average hand pose and the segmentation result of the dexterous hand. By judging whether the second IU reaches the second preset value, the average hand pose is used as the dexterous hand pose only when the second IU reaches the second preset value, thus making the reliability of the dexterous hand pose higher.
[0146] Please see Figure 3As a response to the above Figure 2 In addition to the implementation of the method shown, the third part of this application embodiment also provides an apparatus 600 for constructing dexterous hand training samples. The apparatus 600 is generally disposed in the controller 400 of the system for constructing dexterous hand training samples described above. The apparatus 600 includes:
[0147] The object pose estimation module 610 is used to acquire target object image data and estimate the target object pose based on the target object image data.
[0148] The first motion instruction module 620 is used to acquire a first motion instruction, so as to instruct the dexterous hand to move to the grasping position close to the target object through the first motion instruction;
[0149] The hand pose estimation module 630 is used to acquire dexterous hand image data and estimate the dexterous hand pose at the grasping position based on the dexterous hand image data.
[0150] The pose calculation module 640 is used to calculate the relative pose of the dexterous hand 300 to the target object when it is in the grasping position based on the pose of the target object and the pose of the dexterous hand.
[0151] The second motion instruction module 650 is used to acquire a second motion instruction, so as to instruct the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object.
[0152] The information acquisition module 660 is used to calculate the bending angle value of the mechanical finger bending movement to the target grasping position, and to acquire the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position;
[0153] The data storage module 670 is used to save relative pose, bending angle values and tactile information as training samples for the dexterous hand.
[0154] In one embodiment, the device 600 further includes:
[0155] The initial value setting module is used to set the bending angle of the mechanical finger at the position to be grasped by the dexterous hand as the initial value before instructing the mechanical finger to bend to the target grasping position through the second motion command to enable the dexterous hand to perform functional actions on the target object.
[0156] In one embodiment, the device 600 further includes:
[0157] The grasping position setting module is used to determine the type of the target object based on the target object image data and set the grasping position according to the target object type before instructing the dexterous hand to move to the grasping position close to the target object through the first motion command.
[0158] In one embodiment, the object pose estimation module 610 includes:
[0159] The initial object pose module is used to acquire target object image data multiple times and estimate the initial object pose of multiple frames based on the acquired target object image data.
[0160] The object pose estimation module is used to obtain the pose of the target object based on the initial object poses of multiple frames.
[0161] In one embodiment, the hand pose estimation module 630 includes:
[0162] The initial hand pose module is used to acquire dexterity hand image data multiple times and estimate the initial hand pose of multiple frames based on the acquired dexterity hand image data.
[0163] The hand pose acquisition module is used to obtain the dexterous hand pose based on the initial hand poses of multiple frames.
[0164] In one embodiment, the object pose determination module includes:
[0165] The first elimination module is used to eliminate outliers in the initial object poses of multiple frames.
[0166] The first calculation module is used to calculate the average value of the initial object pose in multiple frames after removing outliers to obtain the average object pose, and to obtain the target object pose based on the average object pose.
[0167] In one embodiment, the hand pose determination module includes:
[0168] The second elimination module is used to eliminate outliers in the initial hand pose of multiple frames;
[0169] The second calculation module is used to calculate the average value of the initial hand pose in multiple frames after removing outliers to obtain the average hand pose, and to obtain the dexterous hand pose based on the average hand pose.
[0170] In one embodiment, the first computing module includes:
[0171] The first model acquisition module is used to acquire the 3D model of the target object;
[0172] The first pixel plane module is used to generate a first pixel plane based on the target object image data. The first pixel plane contains the segmentation result of the target object.
[0173] The first intersection-union module is used to project the 3D model of the target object onto the first pixel plane based on the average pose of the object, and calculate the first intersection-union ratio between the 3D model of the target object and the segmentation result of the target object.
[0174] The first judgment module is used to determine whether the first intersection-union ratio reaches the first preset value. When the first intersection-union ratio reaches the first preset value, the average pose of the object is used as the pose of the target object.
[0175] In one embodiment, the second computing module includes:
[0176] The second model acquisition module is used to acquire the 3D model of the dexterous hand;
[0177] The second pixel plane module is used to generate a second pixel plane based on the dexterous hand image data. The second pixel plane contains the segmentation results of the dexterous hand.
[0178] The second cross-union module is used to project the 3D model of the dexterous hand onto the second pixel plane based on the average pose of the hand parts, and calculate the second cross-union ratio between the 3D model of the dexterous hand and the segmentation result of the dexterous hand.
[0179] The second judgment module is used to determine whether the second cross-union ratio reaches the second preset value. When the second cross-union ratio reaches the second preset value, the average hand pose is taken as the dexterous hand pose.
[0180] Please see Figure 5 The fourth part of this application also provides a computer device 700.
[0181] The computer device 700 can be a terminal or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.
[0182] The computer device 700 includes a memory 710, a processor 720, and a network interface 730 that are interconnected via a system bus. It should be noted that only the computer device 700 with components is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device 700 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0183] The memory 710 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 710 may be an internal storage unit of the computer device 700, such as the hard disk or memory of the computer device 700. In other embodiments, the memory 710 may also be an external storage device of the computer device 700, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 700. Of course, the memory 710 may also include both internal storage units and external storage devices of the computer device 700. In this embodiment, the memory 710 is typically used to store the operating system and various application software installed on the computer device 700, such as program code for methods of constructing dexterous hand training samples. Furthermore, the memory 710 can also be used to temporarily store various types of data that have been output or will be output.
[0184] In some embodiments, the processor 720 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 720 is typically used to control the overall operation of the computer device 700. In this embodiment, the processor 720 is used to run program code stored in the memory 710 or process data, for example, to run program code for a method of constructing dexterity hand training samples.
[0185] The network interface 730 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 700 and other electronic devices.
[0186] The fifth part of this application also provides a computer-readable storage medium storing a computer program for constructing dexterity training samples. The computer program for constructing dexterity training samples can be executed by at least one processor to cause the at least one processor to perform the steps of the method for constructing dexterity training samples as described above.
[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software and its necessary general-purpose hardware platform, and of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0188] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.
Claims
1. A system for constructing training samples for dexterous hands, characterized in that, The system includes: an instruction generation device, an image sensor, a dexterous hand, and a controller. The dexterous hand is equipped with a tactile sensor and flexible mechanical fingers. The instruction generation device, the image sensor, and the tactile sensor are all communicatively connected to the controller, and the controller is communicatively connected to the dexterous hand. The instruction generation device is used to generate a first motion instruction and a second motion instruction; The image sensor is used to acquire image data of the target object and image data of the dexterous hand; The controller is used to acquire the target object image data and estimate the target object pose based on the target object image data; Obtain the first motion command to instruct the dexterous hand to move to a grasping position close to the target object; Acquire the dexterous hand image data, and estimate the dexterous hand pose at the grasping position based on the dexterous hand image data; The relative pose of the dexterous hand to the target object is calculated based on the pose of the target object and the pose of the dexterous hand when the dexterous hand is in the grasping position. The second motion command is obtained to instruct the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object; Calculate the bending angle value of the mechanical finger when it bends to the target grasping position, and obtain the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position; The relative pose, the bending angle value, and the tactile information are saved as training samples for the dexterous hand.
2. The system for constructing dexterous hand training samples according to claim 1, characterized in that, It also includes a drive device, the dexterous hand is mounted on the drive device, and the controller is communicatively connected to the drive device. The controller instructs the drive device to move the dexterous hand to a position close to the target object for grasping via the first motion command. And / or, the system for constructing dexterous hand training samples further includes a scanning device for scanning a target object to obtain a 3D model of the target object, and for scanning the dexterous hand to obtain a 3D model of the dexterous hand, the scanning device being communicatively connected to the controller; And / or, the instruction generating device is a remotely operated device.
3. A method for constructing dexterous hand training samples, applied to a system for constructing dexterous hand training samples, the system comprising: The instruction generation device, image sensor, dexterous hand, and controller, wherein the dexterous hand is equipped with a tactile sensor and flexible mechanical fingers, characterized in that the method comprises: Acquire target object image data, and estimate the target object pose based on the target object image data; Obtain a first motion command to instruct the dexterous hand to move to a grasping position close to the target object; Acquire dexterity hand image data, and estimate the dexterity hand pose at the grasping position based on the dexterity hand image data; The relative pose of the dexterous hand to the target object is calculated based on the pose of the target object and the pose of the dexterous hand when the dexterous hand is in the grasping position. Obtain a second motion command to instruct the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object; Calculate the bending angle value of the mechanical finger when it bends to the target grasping position, and obtain the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position; The relative pose, the bending angle value, and the tactile information are saved as training samples for the dexterous hand.
4. The method for constructing dexterous hand training samples according to claim 3, characterized in that, The estimation of the target object pose based on the target object image data specifically includes: The pose of the target object is estimated based on the target object image data using a preset pose estimation model. The step of estimating the dexterity hand pose at the grasping position based on the dexterity hand image data specifically includes: The pre-defined pose estimation model is used to estimate the pose of the dexterous hand at the grasping position based on the dexterous hand image data.
5. The method for constructing dexterous hand training samples according to claim 3 or 4, characterized in that, The process of acquiring target object image data and estimating the target object pose based on the target object image data includes the following steps: The target object image data is acquired multiple times, and the initial object pose of multiple frames is estimated based on the acquired target object image data. The target object pose is obtained based on the initial object pose in multiple frames. And / or, acquiring dexterous hand image data and estimating the dexterous hand pose at the grasping position based on the dexterous hand image data includes the following steps: The dexterous hand image data is acquired multiple times, and the initial hand pose of multiple frames is estimated based on the acquired dexterous hand image data. The dexterous hand pose is obtained based on the initial hand pose in multiple frames.
6. The method for constructing dexterous hand training samples according to claim 5, characterized in that, The process of obtaining the target object pose based on the initial object pose across multiple frames includes the following steps: Remove outliers from the initial object pose in multiple frames; The average value of the initial object pose in multiple frames after removing outliers is calculated to obtain the average object pose, and the target object pose is obtained based on the average object pose. The process of obtaining the dexterous hand pose based on the initial hand pose in multiple frames includes the following steps: Remove outliers from the initial hand pose in multiple frames; The average value of the initial hand pose is calculated for multiple frames after outliers are removed to obtain the average hand pose, and the dexterous hand pose is obtained based on the average hand pose.
7. The method for constructing dexterous hand training samples according to claim 6, characterized in that, The step of determining the pose of the target object based on the average pose of the object includes the following steps: Obtain a 3D model of the target object; A first pixel plane is generated based on the target object image data, and the first pixel plane contains the segmentation result of the target object; Based on the average pose of the object, the 3D model of the target object is projected onto the first pixel plane, and the first intersection-union ratio of the 3D model of the target object and the segmentation result of the target object is calculated; Determine whether the first intersection-to-union ratio reaches a first preset value; if the first intersection-to-union ratio reaches the first preset value, use the average pose of the object as the pose of the target object. The process of determining the dexterity hand position based on the average value of the hand position includes the following steps: Obtain a 3D model of a dexterous hand; A second pixel plane is generated based on the dexterous hand image data, the second pixel plane containing the segmentation result of the dexterous hand; Based on the average pose of the hand, the 3D model of the dexterous hand is projected onto the second pixel plane, and the second intersection-union ratio of the 3D model of the dexterous hand and the segmentation result of the dexterous hand is calculated; Determine whether the second crossover ratio reaches the second preset value. If the second crossover ratio reaches the second preset value, the average value of the hand pose is taken as the dexterous hand pose.
8. The method for constructing dexterous hand training samples according to claim 3, characterized in that, Before instructing the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform functional actions on the target object via the second motion command, the following steps are also included: The initial value is set to the bending angle of the mechanical finger when the dexterous hand is located at the grasping position.
9. The method for constructing dexterous hand training samples according to claim 3, characterized in that, The system for constructing dexterity hand training samples also includes a driving device, the dexterity hand is mounted on the driving device, and the controller is communicatively connected to the driving device. The controller instructs the driving device to move the dexterity hand to a grasping position close to the target object through the first motion command. And / or, the system for constructing dexterous hand training samples further includes a scanning device for scanning a target object to obtain a 3D model of the target object, and for scanning the dexterous hand to obtain a 3D model of the dexterous hand, the scanning device being communicatively connected to the controller; And / or, the instruction generating device is a remotely operated device.
10. An apparatus for constructing training samples for dexterous hands, characterized in that, The device includes: The object pose estimation module is used to acquire target object image data and estimate the target object pose based on the target object image data. The first motion instruction module is used to acquire a first motion instruction, so as to instruct the dexterous hand to move to a grasping position close to the target object through the first motion instruction; The hand pose estimation module is used to acquire dexterous hand image data and estimate the dexterous hand pose of the dexterous hand at the grasping position based on the dexterous hand image data. The pose calculation module is used to calculate the relative pose of the dexterous hand relative to the target object when it is located at the grasping position, based on the pose of the target object and the pose of the dexterous hand. The second motion instruction module is used to acquire a second motion instruction, so as to instruct the mechanical finger to bend to a target grasping position that enables the dexterous hand to perform a functional action on the target object. The information acquisition module is used to calculate the bending angle value of the mechanical finger when it bends to the target grasping position, and to acquire the tactile information collected by the tactile sensor when the mechanical finger bends to the target grasping position; The data storage module is used to save the relative pose, the bending angle value, and the tactile information as training samples for the dexterous hand.