A mechanical arm target grabbing method based on open semantic segmentation

Abstract

The application discloses a mechanical arm target grabbing method based on open semantic segmentation, and belongs to the technical field of robot vision and intelligent control. The method acquires an RGB color image and a depth image of a scene to be grabbed through a depth camera, performs open semantic recognition on a target object according to a text semantic instruction input by a user, obtains a target candidate region, and generates a target instance mask. The target instance mask is used to perform spatial constraint on the depth image, and a local point cloud corresponding to the target object is extracted in combination with a camera internal parameter. The local point cloud is input into a grabbing pose prediction network to generate a plurality of candidate 6D grabbing poses, and an optimal grabbing pose is determined. Finally, a coordinate conversion relationship obtained based on hand-eye calibration is used to solve inverse kinematics and plan motion, so that a mechanical arm is controlled to complete grabbing. The application can reduce the interference of a complex background, adjacent objects and occlusion on grabbing pose estimation, and improve the accuracy and reliability of target grabbing.

Description

Technical Field

[0001] This invention belongs to the field of robot vision and intelligent control technology, specifically relating to a target grasping method for robotic arms based on open semantic segmentation. Background Technology

[0002] With the development of artificial intelligence, machine vision, 3D perception, and robot control technologies, robotic arms are increasingly being used in industrial sorting, intelligent warehousing, flexible manufacturing, and service robots. Target grasping is a key step for robotic arms to complete tasks such as handling, sorting, and assembly. Its core lies in enabling the robotic arm to accurately identify the target object to be grasped in complex scenarios and generate a stable and reliable grasping posture.

[0003] Existing robotic arm grasping methods typically rely on preset target categories, fixed grasping points, taught trajectories, or standardized working environments. When the grasping scene contains multiple adjacent objects, cluttered stacking, partial occlusion, background interference, or changes in target category, traditional grasping methods are prone to problems such as inaccurate target localization, incomplete target point cloud extraction, background point cloud infiltration, and grasping pose estimation deviations, thereby reducing the robotic arm's grasping success rate and scene adaptability.

[0004] In recent years, significant progress has been made in deep learning-based methods for object detection, instance segmentation, 3D point cloud processing, and grasping pose estimation. Depth cameras can simultaneously acquire RGB color images and depth images of the scene to be grasped, providing robotic arms with 2D semantic information and 3D spatial information. Semantic segmentation models can obtain pixel-level regions of target objects, and grasping pose prediction networks can generate candidate 6D grasping poses based on point cloud data. However, in complex or cluttered scenes or open target scenarios, existing methods still suffer from insufficient target region constraints, infiltration of background or adjacent object point clouds, low accuracy in extracting local target point clouds, and inadequate selection of candidate grasping poses. These issues mean that the accuracy and robustness of robotic arms in grasping specified target objects still need improvement.

[0005] Therefore, there is an urgent need for a robotic arm grasping method that can achieve open target recognition based on text semantic instructions and extract local target point clouds and estimate 6D grasping pose by constraining target instance masks, so as to improve the positioning accuracy of specified target objects, the reliability of grasping pose estimation and the stability of robotic arm grasping in complex scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a robotic arm target grasping method based on open semantic segmentation, so as to solve the problems that existing robotic arm grasping methods are easily affected by background objects, adjacent objects, cluttered stacking and occlusion in complex scenes, resulting in inaccurate target area positioning, low target point cloud extraction accuracy, insufficient reliability of grasping pose estimation and low grasping success rate.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] Step S1: Build a target grasping application scenario for the robotic arm. Set up a depth camera, a robotic arm, and a gripper installed at the end of the robotic arm in the workspace of the robotic arm. Obtain the coordinate transformation relationship between the camera coordinate system of the depth camera and the coordinate system of the robotic arm base through hand-eye calibration.

[0009] Step S2: Acquire RGB-D images of the scene to be captured using a depth camera. The RGB-D images include RGB color images and depth images corresponding to the RGB color images.

[0010] Step S3: Receive text semantic instructions for the target object to be captured, input the RGB color image and text semantic instructions into the open semantic segmentation model, perform open semantic recognition and instance segmentation on the target object to be captured, and obtain the target instance mask corresponding to the target object to be captured.

[0011] Step S4: Spatial constraints are applied to the depth image based on the target instance mask, and a local target point cloud corresponding to the target object to be grasped is generated by combining the depth camera intrinsic parameters. The local target point cloud is input into the grasping pose prediction network to generate multiple candidate 6D grasping poses. Based on the grasping confidence, the constraint relationship of the target instance mask, and the collision detection results, the multiple candidate 6D grasping poses are filtered to obtain the optimal grasping pose in the camera coordinate system.

[0012] Step S5: Based on the coordinate transformation relationship, the optimal grasping pose in the camera coordinate system is converted into the target grasping pose in the robot arm base coordinate system. Through inverse kinematics solution and motion planning of the robot arm, the robot arm is controlled to drive the gripper to move to the target grasping pose, and the gripper is controlled to close, so as to complete the grasping of the target object.

[0013] Compared with the prior art, the present invention has at least the following advantages:

[0014] First, this invention achieves open-vocabulary target recognition by combining textual semantic commands with an open semantic segmentation model. It can locate the specified target object to be grasped based on the semantic description input by the user, improving the robotic arm's adaptability to different types of target objects and open scenes. Second, this invention utilizes target instance masks to spatially constrain depth images and extracts local target point clouds corresponding to the target object to be grasped based on the target instance masks. This effectively reduces the interference of complex backgrounds, adjacent objects, cluttered stacking, and occluded areas on point cloud extraction and 6D grasping pose estimation, improving the accuracy of target point cloud extraction. Third, this invention inputs the local target point cloud into a grasping pose prediction network to generate candidate 6D grasping poses. It then combines grasping confidence, target instance mask constraint relationships, and collision detection results for comprehensive screening, improving the reliability of the optimal grasping pose, thereby increasing the robotic arm's grasping success rate and grasping stability. Attached Figure Description

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0016] Figure 1 This is a schematic diagram of the overall process of a robotic arm target grasping method based on open semantic segmentation according to the present invention;

[0017] Figure 2 This is a schematic diagram illustrating the acquisition of RGB color images and depth images in the target grasping application scenario of the robotic arm of the present invention;

[0018] Figure 3 This is a schematic diagram illustrating the present invention of performing open semantic recognition on the target object to be grasped based on text semantic instructions, obtaining the target candidate region, and generating a target instance mask;

[0019] Figure 4 This is a schematic diagram illustrating the present invention of extracting local target point clouds based on target instance mask constraints and generating candidate 6D grasping poses;

[0020] Figure 5 This is an experimental schematic diagram of the robotic arm of the present invention controlling the end effector gripper to grasp the target object according to the optimal grasping posture. Detailed Implementation

[0021] To make the objectives, technical solutions, and beneficial effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] like Figure 1As shown, this embodiment provides a robotic arm target grasping method based on open semantic segmentation. The method mainly includes robotic arm grasping scene construction and hand-eye calibration, RGB-D image acquisition, open semantic target recognition and target instance mask generation, local target point cloud extraction and 6D grasping pose estimation based on target instance mask, as well as coordinate transformation, motion planning and robotic arm grasping execution.

[0023] Step S1: Set up a target grasping application scenario for the robotic arm and perform hand-eye calibration.

[0024] Step S1-1: Set up a target grasping experimental scenario within the robotic arm's workspace, including the robotic arm, gripper, and depth camera. The gripper is mounted at the end of the robotic arm to perform the grasping operation of the target object. The depth camera is fixed outside the robotic arm's workspace, with its field of view covering the grasping area, forming a robotic arm target grasping application scenario where the eye is outside the hand.

[0025] Step S1-2: In this embodiment, the depth camera used is an Azure Kinect depth camera. To enable the target pose acquired by the depth camera to be used for robotic arm control, hand-eye calibration of the depth camera and the robotic arm is required to obtain the extrinsic parameter relationship between the camera coordinate system of the depth camera and the base coordinate system of the robotic arm. This extrinsic parameter relationship can be represented as a rigid body transformation matrix:

[0026]

[0027] in, This represents the homogeneous transformation matrix from the camera coordinate system to the robot arm base coordinate system. Represents the rotation matrix. This represents the translation vector. After calibration, this coordinate transformation matrix can transform the grasping pose in the camera coordinate system to the robot arm base coordinate system.

[0028] Step S2: Acquire an RGB-D image of the scene to be captured.

[0029] Step S2-1: As Figure 2 As shown, an Azure Kinect depth camera is used to acquire images of the scene to be captured, obtaining RGB color images and depth images at the same time. The RGB color image provides color, texture, and semantic information of the target object, while the depth image provides distance information corresponding to each pixel position in the scene. In this embodiment, the resolution of the RGB color image can be 1280×720 pixels, and the depth value of each pixel in the depth image can be in millimeters.

[0030] Step S2-2: Subsequently, the RGB color image and the depth image are spatially aligned to establish a mapping relationship between pixels in the RGB color image and corresponding depth pixels in the depth image, resulting in registered RGB-D image data. This RGB-D image data is used for subsequent open semantic recognition, target instance mask generation, local target point cloud extraction, and 6D grasping pose estimation.

[0031] Step S3: Obtain the target mask map of the target object based on the text semantic instructions.

[0032] Step S3-1: The system receives text semantic instructions input by the user for the target object to be grasped, such as "banana," "orange," or other target names. These text semantic instructions specify the target object the robotic arm needs to grasp. The system inputs the text semantic instructions along with the RGB color image obtained in step S2 into the open semantic segmentation model, enabling the model to search for the corresponding target in the image based on the text semantic information.

[0033] Step S3-2: As Figure 3 As shown, in this embodiment, the open semantic segmentation model can adopt the Grounded SAM2 text semantic segmentation model. This model first performs open-vocabulary target recognition in the RGB color image based on the text semantic instructions, and outputs the target candidate regions corresponding to the text semantic instructions. Its recognition process can be abstractly represented as follows:

[0034]

[0035] in, This represents the semantic text instructions input by the user. Represents an RGB color image. This indicates the process of target localization for open vocabulary. This represents the target candidate region output by the model.

[0036] Step S3-3: After obtaining the target candidate region, the model further performs instance segmentation on the candidate region to generate a target instance mask corresponding to the target object to be captured. The target instance mask can be represented as a binary matrix:

[0037]

[0038] in, This represents the pixel region containing the target object to be captured. When... When the pixel is in the specified position, it indicates that the pixel belongs to the target object to be captured. When the value is set to 0, it indicates that the pixel belongs to the background or a non-target region. This target instance mask serves as spatial constraint information for subsequent depth image filtering and local point cloud extraction.

[0039] Step S4: Extract local point cloud based on target mask constraints and generate optimal grasping pose.

[0040] Step S4-1: Spatial constraints are applied to the depth image obtained in step S2 based on the target instance mask obtained in step S3. The depth information within the corresponding pixel area of ​​the target instance mask is retained, while the background depth information outside the corresponding pixel area of ​​the target instance mask is removed, thereby reducing the interference of background objects, adjacent objects, and irrelevant areas on subsequent 6D grasping pose estimation.

[0041] Step S4-2: Based on the effective depth information within the corresponding pixel region of the target instance mask, and combined with the depth camera intrinsic parameters, convert the two-dimensional pixel points corresponding to the target object to be captured into three-dimensional point cloud data in the camera coordinate system to obtain the local target point cloud corresponding to the target object to be captured.

[0042] Step S4-3: As Figure 4 As shown, the obtained local target point cloud is input into the GraspNet grasping pose prediction network to generate multiple candidate 6D grasping poses. Further, these candidate 6D grasping poses are scored and filtered. Scoring factors include target instance mask constraints, grasping confidence, and gripper collision detection results. Candidate 6D grasping poses that do not meet target instance mask constraints or have collision risks are eliminated. For candidate 6D grasping poses that meet the constraints, they are ranked according to their comprehensive scores, and the candidate 6D grasping pose with the highest score is selected as the optimal grasping pose in the camera coordinate system.

[0043] Step S5: Coordinate transformation, motion planning and robot grasping execution.

[0044] Step S5-1: Based on the coordinate transformation relationship between the camera coordinate system and the robotic arm base coordinate system obtained in step S1, convert the optimal grasping pose in the camera coordinate system obtained in step S4 into the target grasping pose in the robotic arm base coordinate system.

[0045] Step S5-2: Let the optimal grasping pose in the camera coordinate system be denoted as... The coordinate transformation matrix between the camera coordinate system and the robot arm base coordinate system is denoted as... The target grasping pose in the robot arm base coordinate system. It can be represented as

[0046]

[0047] in, This represents the homogeneous transformation matrix of the capture pose relative to the camera coordinate system. This represents the homogeneous transformation matrix of the camera coordinate system relative to the robot arm base coordinate system. This represents the homogeneous transformation matrix of the target grasping pose relative to the coordinate system of the robotic arm base.

[0048] Step S5-3: Based on the target grasping pose in the robot arm's base coordinate system, perform inverse kinematics calculations on the robot arm to obtain the target joint angles of each joint, and determine whether the target grasping pose satisfies the robot arm's motion range and reachability constraints. If the target grasping pose satisfies the robot arm's reachability constraints, perform motion planning based on the target joint angles to generate the robot arm's motion trajectory from its current position to the target grasping pose.

[0049] Step S5-4: Finally, as Figure 5 As shown, the robotic arm is controlled to move the gripper to the target grasping position according to the planned motion trajectory, and the gripper is controlled to perform a closing action, thereby completing the grasping operation of the specified target object.

[0050] Through the above implementation methods, this invention can utilize textual semantic instructions to perform open-vocabulary recognition and instance segmentation of specified target objects in complex scenes, and use target instance masks to spatially constrain depth images and point cloud data, thereby extracting local target point clouds corresponding to the target object to be grasped. Compared with methods that directly generate grasping poses based on complete scene point clouds, this invention can effectively reduce the interference of complex backgrounds, adjacent objects, and occlusion areas on 6D grasping pose estimation, improve the accuracy of target grasping pose estimation, and enhance the stability of robotic arm grasping operations.

Claims

1. A method for target grasping using a robotic arm based on open semantic segmentation, characterized in that, Includes the following steps: Step S1: Build a target grasping application scenario for the robotic arm. Set up a depth camera, a robotic arm, and a gripper installed at the end of the robotic arm in the workspace of the robotic arm. Obtain the coordinate transformation relationship between the camera coordinate system of the depth camera and the coordinate system of the robotic arm base through hand-eye calibration. Step S2: Acquire RGB-D images of the scene to be captured using a depth camera. The RGB-D images include RGB color images and depth images corresponding to the RGB color images. Step S3: Receive text semantic instructions for the target object to be captured, input the RGB color image and text semantic instructions into the open semantic segmentation model, perform open semantic recognition and instance segmentation on the target object to be captured, and obtain the target instance mask corresponding to the target object to be captured. Step S4: Spatial constraints are applied to the depth image based on the target instance mask, and a local target point cloud corresponding to the target object to be grasped is generated by combining the camera intrinsic parameters. The local target point cloud is input into the grasping pose prediction network to generate multiple candidate 6D grasping poses. The multiple candidate 6D grasping poses are then filtered based on the grasping confidence, the constraint relationship of the target instance mask, and the collision detection results to obtain the optimal grasping pose in the camera coordinate system. Step S5: Based on the coordinate transformation relationship, the optimal grasping pose in the camera coordinate system is converted into the target grasping pose in the robot arm base coordinate system. Through inverse kinematics solution and motion planning of the robot arm, the robot arm is controlled to drive the gripper to move to the target grasping pose, and the gripper is controlled to close, so as to complete the grasping of the target object.

2. The robotic arm target grasping method based on open semantic segmentation according to claim 1, characterized in that, Step S1 specifically includes: Step S1-1: Build a target grasping application scenario for a robotic arm. Set up a robotic arm, gripper, and depth camera in the robotic arm's workspace. The gripper is installed at the end of the robotic arm, and the depth camera is fixed outside the robotic arm's workspace. Make sure the depth camera's field of view covers the area to be grasped, so as to form a target grasping application scenario for a robotic arm where the eye is outside the hand. Step S1-2: Perform hand-eye calibration for the target grasping application scenario of the robotic arm with eyes outside the hands, and obtain the coordinate transformation matrix between the camera coordinate system of the depth camera and the coordinate system of the robotic arm base.

3. The robotic arm target grasping method based on open semantic segmentation according to claim 1, characterized in that, Step S2 specifically includes the following steps: Step S2-1: Use the Azure Kinect depth camera to capture images of the scene to be captured, and obtain RGB color images and depth images at the same time. The resolution of the RGB color image is 1280×720 pixels, and the depth value of each pixel in the depth image is in millimeters. Step S2-2: Spatially align the RGB color image and the depth image to establish a mapping relationship between pixels in the RGB color image and corresponding depth pixels in the depth image, thereby obtaining RGB-D image data for subsequent open semantic recognition, target instance mask generation, local target point cloud extraction, and 6D grasping pose estimation.

4. The robotic arm target grasping method based on open semantic segmentation according to claim 1, characterized in that, Step S3 specifically includes the following steps: Step S3-1: Receive text semantic instructions for the target object to be captured, and acquire the RGB color image obtained in step S2. Use the text semantic instructions and the RGB color image as input to the open semantic segmentation model. Step S3-2: Open vocabulary target recognition is performed on the RGB color image using the Grounded SAM2 text semantic segmentation model. The corresponding target object to be grasped is located in the RGB color image according to the text semantic instructions, and the target candidate region of the target object to be grasped is obtained. Step S3-3: Generate a target instance mask of the target object to be grasped based on the target candidate region. The target instance mask is used to represent the pixel-level region of the target object to be grasped in the RGB color image and serves as the input constraint information for subsequent depth image constraints, local target point cloud extraction and 6D grasping pose estimation.

5. The robotic arm target grasping method based on open semantic segmentation according to claim 1, characterized in that, Step S4 specifically includes the following steps: Step S4-1: Spatial constraints are applied to the depth image obtained in step S2 based on the target instance mask obtained in step S3, retaining the depth information within the corresponding pixel area of ​​the target instance mask and removing the background depth information outside the corresponding pixel area of ​​the target instance mask. Step S4-2: Based on the effective depth information within the pixel region corresponding to the target instance mask, and combined with the depth camera intrinsic parameters, convert the two-dimensional pixel points corresponding to the target object to be captured into three-dimensional point cloud data in the camera coordinate system to obtain the local target point cloud corresponding to the target object to be captured. Step S4-3: Input the local point cloud data into the GraspPose Generation Network (GraspNet) to generate multiple candidate 6D grasp poses. Step S4-4: Based on the target instance mask constraint relationship, grasping confidence and gripper collision detection results, a comprehensive score and ranking of multiple candidate 6D grasping poses are performed, and the optimal grasping pose in the camera coordinate system is selected.

6. The robotic arm target grasping method based on open semantic segmentation according to claim 1, characterized in that, Step S5 specifically includes the following steps: Step S5-1: Based on the coordinate transformation relationship between the camera coordinate system and the robotic arm base coordinate system obtained in step S1, the optimal grasping pose in the camera coordinate system obtained in step S4 is converted into the target grasping pose in the robotic arm base coordinate system. Step S5-2: Based on the target grasping pose in the robot arm base coordinate system, perform inverse kinematics solution on the robot arm to obtain the target joint angles of each joint of the robot arm, and determine whether the target grasping pose satisfies the robot arm's motion range constraints and reachability constraints. Step S5-3: Under the condition that the target grasping pose satisfies the robotic arm's motion range constraint and reachability constraint, motion planning is performed according to the target joint angle to generate the motion trajectory of the robotic arm from the current position to the target grasping pose. Step S5-4: Control the robotic arm to move the gripper to the target grasping position according to the motion trajectory, and control the gripper to perform a closing action to complete the grasping operation of the target object.

Images