Automatic grabbing method based on 3D hand-eye system and five-finger dexterous hand

Abstract

The invention discloses an automatic grabbing method based on a 3D hand-eye system and a five-finger dexterous hand. The automatic grabbing method comprises the following steps that S100, automatic hand-eye calibration is conducted on the 3D hand-eye system; s200, collecting a video stream by a camera of the RGB camera, detecting a to-be-grabbed object, and calculating a central point coordinate; s300, performing visual angle full conversion on a depth image output by the IR structured light depth camera, and calculating a space coordinate and an estimated size under an object camera coordinate system in combination with an RGB detection center point coordinate; s400, the space coordinate position under the camera coordinate system is firstly converted to be under the five-finger dexterous hand grabbing coordinate system and then converted to be under the mechanical arm base coordinate system; and S500, the detected object is grabbed through the five-finger dexterous hand. The method can be widely applied to multiple fields of intelligent warehousing, automatic production lines, household robots, unmanned distribution and the like, the operation efficiency is greatly improved, and the labor cost is reduced.

Description

Technical Field

[0001] The present disclosure belongs to the fields of computer vision, intelligent robots, and image processing technologies, and particularly relates to an automatic grasping method based on a 3D hand-eye system and a five-finger dexterous hand. Background Art

[0002] With the rapid development of robotics and artificial intelligence, automated grasping technology has become an indispensable core application across multiple industries. Traditional grasping methods often rely on complex robotic arm designs or external sensors, but these methods are often limited by factors such as the grasping environment, object shape, and placement. The accuracy and flexibility of automated grasping are particularly challenging in dynamic environments, such as those in logistics, manufacturing, healthcare, and home services.

[0003] How to develop a more intelligent and efficient solution based on the collaborative work of the 3D hand-eye system and the five-finger dexterous hand to cope with various complex environments and changing task requirements has become an important topic in current research and industrial applications. Summary of the Invention

[0004] In order to solve the above problems, the present disclosure provides an automatic grasping method based on a 3D hand-eye system and a five-finger dexterous hand, which includes the following steps:

[0005] S100: Automatic hand-eye calibration of the 3D hand-eye system;

[0006] S200: The RGB camera collects video streams, detects objects to be grasped, and calculates the center coordinates.

[0007] S300: Perform a full perspective conversion on the depth map output by the IR structured light depth camera and calculate the spatial coordinates and estimated size of the object in the camera coordinate system by combining the RGB detection center point coordinates;

[0008] S400: first converting the spatial coordinate position in the camera coordinate system to the five-finger dexterous hand grasping coordinate system, and then converting it to the robot arm base coordinate system;

[0009] S500: Grasping detected objects with a five-finger dexterous hand.

[0010] In addition, the present invention also discloses an automatic grasping device based on a 3D hand-eye system and a five-finger dexterous hand, comprising:

[0011] Device for automatic hand-eye calibration of a 3D hand-eye system;

[0012] A device used for collecting video streams from RGB cameras, detecting objects to be grasped, and calculating center point coordinates;

[0013] A device for performing a full perspective conversion on the depth map output by the IR structured light depth camera and calculating the spatial coordinates and estimated size of the object in the camera coordinate system by combining the RGB detection center point coordinates;

[0014] A device for converting the position in the camera coordinate system into the five-finger dexterous hand grasping coordinate system and then into the robot arm base coordinate system;

[0015] Apparatus for grasping detected objects using a five-fingered dexterous hand.

[0016] In addition, the present invention also discloses a computer storage medium, wherein the storage medium includes computer instructions, and when the computer instructions are run on the computer, the computer executes the method.

[0017] In addition, the present invention also discloses an electronic device, wherein the electronic device includes:

[0018] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein:

[0019] When the processor executes the program, the method described is implemented.

[0020] Through the above technical solution, the method, based on a high-precision 3D hand-eye system, achieves automatic grasping of objects under inspection. It is capable of achieving high-precision, high-accuracy, and full-space automatic grasping of a variety of complex-shaped objects (including cylindrical and circular objects, which are difficult to grasp, as well as small objects). This method also features a simple process, simple grasping operations, a lightweight algorithm, and low grasping latency, making it suitable for most robotic platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of an automatic grasping method based on a 3D hand-eye system and a five-finger dexterous hand, provided in an example of the present disclosure;

[0022] Figure 2 This is a schematic diagram of the results of a method for detecting the position and size of an object provided by an example of the present disclosure;

[0023] Figure 3 This is a schematic diagram of the results of a method for pixel-level alignment of depth data and RGB data provided by an example of the present disclosure. DETAILED DESCRIPTION

[0024] The present disclosure will be further described in detail below with reference to Figures 1 to 3 and implementation examples.

[0025] In one embodiment, Figure 1As shown, the present disclosure provides an automatic grasping method based on a 3D hand-eye system and a five-finger dexterous hand, which includes the following steps:

[0026] S100: Automatic hand-eye calibration of the 3D hand-eye system;

[0027] S200: The RGB camera collects video streams, detects objects to be grasped, and calculates the center coordinates.

[0028] S300: Perform a full perspective conversion on the depth map output by the IR structured light depth camera and calculate the spatial coordinates and estimated size of the object in the camera coordinate system by combining the RGB detection center point coordinates;

[0029] S400: first converting the spatial coordinate position in the camera coordinate system to the five-finger dexterous hand grasping coordinate system, and then converting it to the robot arm base coordinate system;

[0030] S500: Grasping detected objects with a five-finger dexterous hand.

[0031] In this embodiment, first, after sending the automatic grasping instruction, the object to be grasped is continuously detected. If detected, it jumps to the next step, otherwise it maintains the original state; secondly, the depth map perspective is fully converted, and the three-dimensional coordinates and size of the center point of the object in the RGB camera coordinate system are clustered and calculated; thirdly, the position in the camera coordinate system is first converted to the five-finger dexterous hand grasping coordinate system, and then converted to the robotic arm base coordinate system; finally, the detected object is grasped by the five-finger dexterous hand. Since this method is based on the 3D hand-eye system and performs a full conversion of the depth map perspective, it can grasp a variety of objects with complex shapes (including cylindrical, circular objects and small objects that are not easy to grasp), making the method disclosed by the present invention a high-precision, easy-to-operate automatic grasping solution. After sending the automatic grasping instruction, the object to be grasped is continuously detected. If detected, it jumps to the next step, otherwise the state is maintained; the depth map perspective is fully converted, and the three-dimensional coordinates and size of the center point of the object in the RGB camera coordinate system are clustered and calculated; the position in the camera coordinate system is first converted to the five-finger dexterous hand grasping coordinate system, and then converted to the robotic arm base coordinate system; finally, the detected object is grasped by the five-finger dexterous hand.

[0032] According to the actual verification results, this method realizes the automatic grasping of various objects, and the most important effect is the robust grasping of objects of various shapes and sizes.

[0033] In S300, the depth map is converted to an angle using internal and external camera parameters to achieve alignment with the detection frame, facilitating subsequent accurate grasping. In S400, multiple coordinate system transformations are performed to determine the object's spatial coordinates within the robot's base coordinate system. In S500, an API is called to plan a path from the initial point to the spatial coordinate system obtained in S400. In addition to automatic grasping, this system also includes functions such as manual grasping. The dimensions in S300 primarily provide a rough reference for manual grasping; a secondary function is to roughly determine whether an object exceeds the grasping capacity of the five-finger dexterous hand.

[0034] In another embodiment, step S100 further includes the following steps:

[0035] S101: First, keep the robot arm in the power-on reset state, and then place the provided calibration plate completely in the prompt box of the automatic calibration program. The larger the area of the prompt box occupied by the calibration plate, the higher the accuracy;

[0036] S102: After placement is complete, data collection begins (the robotic arm base and calibration plate must not move during the collection process). The robotic arm will reach 15 preset positions covering the entire space and record the RGB camera image, quaternion and 3D coordinates in the base coordinate system at different positions.

[0037] S103: Start automatic calibration. The program will calibrate the collected RGB camera image to obtain the rotation and translation matrix from the calibration plate coordinate system to the camera coordinate system. ; According to the quaternion and three-dimensional coordinates in the base coordinate system, the rotation and translation matrices of the manipulator end coordinate system to the manipulator base coordinate system are obtained ;Depend on and The rotation and translation matrix from the camera coordinate system to the robotic arm end coordinate system can be solved , the calibration is completed.

[0038] In another embodiment, step S200 further includes the following steps:

[0039] S201: Process the RGB input stream within the hand, and obtain N result pairs for each frame detection. Each result pair contains 1 normalized confidence, 1 normalized classification probability, and 4 normalized anchor box offsets;

[0040] S202: Each result pair corresponds to a predefined initial anchor box, and the result pairs with confidence greater than the confidence threshold threshold_S202 are retained;

[0041] S203: Calculate the intersection-over-union ratio between the result pairs, and for all result pairs whose intersection-over-union ratio is greater than the suppression threshold threshold_S203, only retain the result pair with the highest confidence.

[0042] In this way, the detection of the object to be grasped is achieved. For example, threshold_S202 is selected as 0.6 and threshold_S203 is selected as 0.5 to obtain the comprehensive optimal effect under multiple distances.

[0043] The in-hand RGB input stream refers to the video stream from the in-hand RGB camera. 3D hand-eye systems are primarily categorized into two types: in-hand and out-of-hand systems. The video stream from the RGB camera in an in-hand system is referred to as the "in-hand RGB input stream." In-hand systems, the RGB camera is fixed and moves with the robotic arm, while out-of-hand systems are fixed and do not move with the robotic arm.

[0044] Further, for example, the detected object is The 320 image is represented and used for the subsequent calculation of the center point coordinates.

[0045] In another embodiment, the object detection input in step S200 is 320 320 images, the output is the normalized offset of the center point coordinates, the normalized offset of the width and height, the confidence probability, and the classification probability. The center point coordinates of the object can be obtained through decoding. The calculation process of the center point coordinates is as follows:

[0046] S2011: RGB image of size 320×320 (that is, the detected object is converted to 320 320 images are represented and used for subsequent calculation of center point coordinates), and multi-scale feature extraction is performed through a network composed of stacked convolutional layers (backbone layer);

[0047] S2012: After multi-scale features are fused through FPN (neck layer) at multiple levels, the output is a multi-scale fused feature map with sizes of 1×64×40×40, 1×128×20×20, and 1×256×10×10 respectively.

[0048] S2013: The multi-scale fused feature maps (i.e., multi-level feature maps) obtained in S2012 are fed into the detection head (head layer) to obtain outputs at three scales, with sizes of 1×4800×10, 1×1200×10, and 1×300×10, respectively. The 4800, 1200, and 300 correspond to the number of predefined anchor boxes at different scales (three different anchor boxes are initialized for each point in each feature map layer), and the 10 represents x, y, w, h, confidence, and the probabilities of five different classes.

[0049] In another embodiment, step S300 further includes the following steps:

[0050] S301: Align depth data and RGB data at the pixel level;

[0051] S302: The depth data after the perspective conversion is aligned with the RGB pixel level, and the depth data in the detection box is clustered into three categories using the RGB detection results;

[0052] S303: Calculate the three-dimensional coordinates of the object's center point in the camera coordinate system using the center point coordinates, center point depth data, and RGB camera intrinsic parameters. Once the center point coordinates are obtained, the corresponding center point depth data is obtained through S301 and S302. The RGB camera intrinsic parameters are known data from the camera.

[0053] S304: Estimate the three-dimensional size of the object using the width and height of the detection frame, the mean depth, and the RGB camera intrinsic parameters.

[0054] For this example, see Figure 2 The depth data within the detection box is clustered into three categories: category 0 is noise, which has been set to 0 by the algorithm; category 1 is the depth data corresponding to the object; and category 2 is the depth data corresponding to the background. The center point depth data is the average of the median 10 data points in category 1.

[0055] In another embodiment, step S301 further includes the following steps:

[0056] S3011: Convert the depth data from the depth camera's perspective into the depth camera coordinate system using the depth camera's intrinsic parameters to obtain point cloud data.

[0057] S3012: Convert the point cloud data to RGB perspective using the external parameters of the RGB camera and the depth camera;

[0058] S3013: Convert point cloud data into RGB depth map using RGB camera intrinsic parameters to achieve full pixel-level alignment of depth images.

[0059] For this example, see Figure 3 . Use IR camera internal reference in S301 , external parameters between RGB and IR cameras , RGB camera internal parameters , align the depth under the IR perspective to the RGB perspective. The specific formula for the above steps is as follows,

[0060] In S3011:

[0061]

[0062]

[0063]

[0064] in, is the depth map obtained by the IR camera, and is the coordinate of the principal point of the IR camera, and is the depth map pixel coordinate, and The focal length of the IR camera is divided by the pixel length of the camera in the x and y axes, respectively (in mm).

[0065] In S3012:

[0066]

[0067] in, and They are the rotation and translation matrices from the IR coordinate system to the RGB coordinate system, It is the point cloud data obtained in S3011.

[0068] In S3013:

[0069]

[0070]

[0071] in, 、 、 Corresponding to the results obtained in S3012 Chinese data, and The focal length of the IR camera is divided by the pixel length of the camera in the x and y axes, respectively (in mm).

[0072] In another embodiment, step S400 further includes the following steps:

[0073] S401: Automatically calibrate the eye-in-hand 3D hand-eye system to obtain the rotation and translation matrices from the RGB camera coordinate system to the robotic arm end coordinate system;

[0074] S402: The transformation relationship between the five-finger dexterous hand coordinate system and the RGB camera coordinate system can be measured with high precision, and a translation matrix from the five-finger dexterous hand coordinate system to the RGB camera coordinate system is obtained;

[0075] S403: Calculate the three-dimensional coordinates of the object to be grasped in the robotic arm base coordinate system through the rotation and translation matrices from the RGB camera coordinate system to the robotic arm end coordinate system, the translation matrix from the five-finger dexterous hand coordinate system to the RGB camera coordinate system, and the rotation and translation matrices from the robotic arm end coordinate system to the robotic arm base coordinate system.

[0076] In this example, because the hardware system's installation accuracy and repeatability are guaranteed, the deviation between the five-fingered dexterous hand coordinate system and the RGB camera coordinate system is solely a translational deviation. Therefore, the transformation relationship between the five-fingered dexterous hand coordinate system and the RGB camera coordinate system can be measured with high precision. Using these two transformation matrices and the rotation and translation matrices used to transform the manipulator's end-of-arm coordinate system to the manipulator's base coordinate system, the precise 3D coordinates of the object to be grasped in the manipulator's base coordinate system can be calculated, enabling fast and robust one-shot grasping.

[0077] The specific measurement method in S402 is as follows: 3D modeling of the manipulator is performed at different distances using a proprietary macro structured light camera. The first and last feature points are marked in the RGB image and mapped to a point cloud to obtain two 3D coordinates. The distance in space is calculated to obtain the true size between the first and last feature points. The average of the true sizes at different distances is then calculated to obtain the final size, which is the translation distance along a specific axis (depending on the key points). Selecting three different sets of feature points can determine the translation distances along the X, Y, and Z axes.

[0078] In another embodiment, step S401 further includes the following steps:

[0079] S4011: Place the checkerboard calibration plate in the field of view of the RGB camera. During the calibration process, the relative position of the manipulator base and the calibration plate remains unchanged.

[0080] S4012: The end of the manipulator automatically reaches 15 positions. At each position, it captures the calibration plate image and records the position and quaternion operation from the manipulator end coordinate system to the manipulator base coordinate system in ROS.

[0081] S4013: Calculate the camera intrinsic parameters based on the calibration image. Based on the camera intrinsic parameters and the recorded position and quaternion, the rotation and translation matrices for converting the RGB camera coordinate system to the robot arm end coordinate system can be calculated.

[0082] In this embodiment, ROS (Robot Operating System) is an open source framework suitable for robots.

[0083] In another embodiment, the placement distance in step S4011 is within 0.3 to 0.4 m.

[0084] In another embodiment, an automatic grasping device based on a 3D hand-eye system and a five-finger dexterous hand is composed of a 3D hand-eye system, a five-finger dexterous hand, a robotic arm, and a main control AI module, wherein:

[0085] The 3D hand-eye system is used to obtain the spatial position and estimated size of the object to be grasped from the acquired RGBD video stream information;

[0086] The five-finger dexterous hand is used to automatically grasp objects of different shapes, can be manipulated like a human hand, and has five independently controllable fingers;

[0087] The robotic arm locates the object using the spatial coordinates of the object obtained by the 3D hand-eye system, and has a certain arm length so that the object falls within the graspable range of the robotic arm;

[0088] The master AI module is used for RGBD video stream processing, AI object recognition, coordinate conversion, and driving and controlling the movement of the five-finger dexterous hand and robotic arm of the 3D hand-eye system.

[0089] In another embodiment, an automatic grasping device based on a 3D hand-eye system and a five-finger dexterous hand includes:

[0090] Device for automatic hand-eye calibration of a 3D hand-eye system;

[0091] A device used for collecting video streams with RGB cameras, detecting objects to be grasped, and calculating center point coordinates;

[0092] A device for performing a full perspective conversion on the depth map output by the IR structured light depth camera and calculating the spatial coordinates and estimated size of the object in the camera coordinate system by combining the RGB detection center point coordinates;

[0093] A device for converting the position in the camera coordinate system into the five-finger dexterous hand grasping coordinate system and then into the robot arm base coordinate system;

[0094] Apparatus for grasping detected objects using a five-fingered dexterous hand.

[0095] In addition, the present invention also discloses a computer storage medium, wherein the storage medium includes computer instructions, and when the computer is run on the computer, the computer executes any of the methods described above.

[0096] In addition, the present invention also discloses an electronic device, wherein the electronic device includes:

[0097] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein:

[0098] When the processor executes the program, any of the above methods is implemented.

[0099] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. An automatic grasping method based on a 3D hand-eye system and a five-finger dexterous hand, comprising the following steps: S100: Automatic hand-eye calibration of the 3D hand-eye system; S200: The RGB camera collects video streams, detects objects to be grasped, and calculates the center coordinates. S300: Perform a full perspective conversion on the depth map output by the IR structured light depth camera and calculate the spatial coordinates and estimated size of the object in the camera coordinate system by combining the RGB detection center point coordinates; S400: first converting the spatial coordinate position in the camera coordinate system to the five-finger dexterous hand grasping coordinate system, and then converting it to the robot arm base coordinate system; S500: Grasping detected objects with a five-finger dexterous hand.

2. The method according to claim 1, step S200 further comprises the following steps: preferably, S201: Process the RGB input stream within the hand, and obtain N result pairs for each frame detection. Each result pair contains 1 normalized confidence, 1 normalized classification probability, and 4 normalized anchor box offsets; S202: Each result pair corresponds to a predefined initial anchor box, and the result pairs with confidence greater than the confidence threshold threshold_S202 are retained; S203: Calculate the intersection-over-union ratio between the result pairs, and for all result pairs whose intersection-over-union ratio is greater than the suppression threshold threshold_S203, only retain the result pair with the highest confidence.

3. The method according to claim 1, step S300 further comprising the following steps: S301: Align depth data and RGB data at the pixel level; S302: The depth data after the perspective conversion is aligned with the RGB pixel level, and the depth data in the detection box is clustered into three categories using the RGB detection results; S303: Calculate the three-dimensional coordinates of the object center point in the camera coordinate system using the center point coordinates, center point depth data, and RGB camera intrinsic parameters; S304: Estimate the three-dimensional size of the object using the width and height of the detection frame, the mean depth, and the RGB camera intrinsic parameters.

4. The method according to claim 3, wherein step S301 further comprises the following steps: S3011: Convert the depth data from the depth camera's perspective into the depth camera coordinate system using the depth camera's intrinsic parameters to obtain point cloud data. S3012: Convert the point cloud data to RGB perspective using the external parameters of the RGB camera and the depth camera; S3013: Convert point cloud data into RGB depth map using RGB camera intrinsic parameters to achieve full pixel-level alignment of depth images.

5. The method according to claim 1, step S400 further comprising the following steps: S401: Automatically calibrate the eye-in-hand 3D hand-eye system to obtain the rotation and translation matrices from the RGB camera coordinate system to the robotic arm end coordinate system; S402: The transformation relationship between the five-finger dexterous hand coordinate system and the RGB camera coordinate system can be measured with high precision, and a translation matrix from the five-finger dexterous hand coordinate system to the RGB camera coordinate system is obtained; S403: Calculate the three-dimensional coordinates of the object to be grasped in the robotic arm base coordinate system through the rotation and translation matrices from the RGB camera coordinate system to the robotic arm end coordinate system, the translation matrix from the five-finger dexterous hand coordinate system to the RGB camera coordinate system, and the rotation and translation matrices from the robotic arm end coordinate system to the robotic arm base coordinate system.

6. The method according to claim 5, wherein step S401 further comprises the following steps: S4011: Place the checkerboard calibration plate in the field of view of the RGB camera. During the calibration process, the relative position of the manipulator base and the calibration plate remains unchanged. S4012: The end of the manipulator automatically reaches 15 positions. At each position, it captures the calibration plate image and records the position and quaternion operation from the manipulator end coordinate system to the manipulator base coordinate system in ROS. S4013: Calculate the camera intrinsic parameters based on the calibration image. Based on the camera intrinsic parameters and the recorded position and quaternion, the rotation and translation matrices for converting the RGB camera coordinate system to the robot arm end coordinate system can be calculated.

7. The method according to claim 6, wherein the placement distance in step S4011 is within 0.3 to 0.4 m.

8. An automatic grasping device based on a 3D hand-eye system and a five-finger dexterous hand, comprising: Device for automatic hand-eye calibration of a 3D hand-eye system; A device used for collecting video streams from RGB cameras, detecting objects to be grasped, and calculating center point coordinates; A device for performing a full perspective conversion on the depth map output by the IR structured light depth camera and calculating the spatial coordinates and estimated size of the object in the camera coordinate system by combining the RGB detection center point coordinates; A device for converting the position in the camera coordinate system into the five-finger dexterous hand grasping coordinate system and then into the robot arm base coordinate system; Apparatus for grasping detected objects using a five-fingered dexterous hand.

9. A computer storage medium, wherein: The storage medium includes computer instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 7.

10. An electronic device, wherein: The electronic device comprises: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images