An object grasping method and system based on image segmentation and object pose estimation

Abstract

The present invention provides an object grasping method and system based on image segmentation and object pose estimation, which relate to the field of computer vision technology. The method comprises: acquiring an original image of an object to be grasped; preprocessing the original image to obtain a target image; generating an object mask of the object to be grasped according to the target image; determining a 2D bounding box of the object to be grasped in the target image according to the object mask; performing a preliminary estimation of the pose of the object to be grasped by using a rotation and translation sampling technique to obtain a preliminary pose; performing a refined estimation of the pose of the object to be grasped by using a pose refinement network to obtain a refined pose; converting the refined pose from a camera coordinate system to a robotic arm coordinate system; calculating a target grasping pose in the robotic arm coordinate system based on a predefined fixed transformation; converting the target grasping pose into a joint angle of the robotic arm according to an inverse kinematics algorithm, and controlling the robotic arm to grasp the object to be grasped according to the joint angle.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an object grasping method and system based on image segmentation and object pose estimation. Background Art

[0002] With the rapid development of robotics, robots are increasingly being used in industries such as industry, logistics, and healthcare. Accurately estimating the pose of objects and rapidly grasping them has become a key technology, particularly in grasping tasks. Pose estimation accuracy and grasping speed directly impact a robot's efficiency and operational stability.

[0003] Existing techniques for object grasping typically rely on relatively simple pose estimation methods, such as traditional algorithms based on feature matching or template matching. While these methods work well in some simple environments, they often lack sufficient estimation accuracy in complex environments, especially when objects are partially occluded or when multiple objects are present in the scene. This makes it difficult to meet the requirements for high-precision grasping.

[0004] In addition, these traditional methods have slow computation speeds and cannot meet the requirements of real-time control. Summary of the Invention

[0005] In order to solve the technical problems in the existing pose estimation methods that the estimation accuracy is often insufficient when the object is partially occluded or there are multiple objects in the scene, making it difficult to meet the needs of high-precision grasping, and the calculation speed is slow and cannot meet the requirements of real-time control, the present invention provides an object grasping method and system based on image segmentation and object pose estimation.

[0006] The technical solutions provided by the embodiments of the present invention are as follows:

[0007] First aspect:

[0008] An embodiment of the present invention provides an object grasping method based on image segmentation and object pose estimation, comprising:

[0009] S1: Obtain the original image of the object to be grasped;

[0010] S2: Preprocess the original image to obtain a target image;

[0011] S3: Generate an object mask of the object to be grasped through the SAM model and the DINOv2 model according to the target image;

[0012] S4: Determine a 2D bounding box of the object to be grasped in the target image according to the object mask;

[0013] S5: Based on the median depth of the 2D bounding box, preliminarily estimate the pose of the object to be grasped by using a rotation and translation sampling technique to obtain multiple preliminary poses;

[0014] S6: According to each of the preliminary poses and the target image, a pose refinement network is used to refine and estimate the pose of the object to be grasped to obtain a refined pose;

[0015] S7: converting the refined pose from the camera coordinate system to the robot arm coordinate system;

[0016] S8: Calculating a target grasping pose in the manipulator coordinate system based on a predefined fixed transformation between the end effector and the object to be grasped;

[0017] S9: converting the target grasping posture into the joint angle of the robotic arm through an inverse kinematics algorithm;

[0018] S10: Controlling the robot arm to grasp the object to be grasped according to the joint angle.

[0019] Second aspect:

[0020] An embodiment of the present invention provides an object grasping system based on image segmentation and object pose estimation, comprising:

[0021] processor;

[0022] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the object grasping method based on image segmentation and object pose estimation as described in the first aspect is implemented.

[0023] The third aspect:

[0024] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the object grasping method based on image segmentation and object pose estimation as described in the first aspect is implemented.

[0025] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0026] In an embodiment of the present invention, an object mask of an object to be grasped is generated by using a SAM model and a DINOv2 model. When the object is partially occluded or there are multiple objects in the scene, the accuracy of pose estimation is improved, which can meet the needs of high-precision grasping. According to the object mask, the 2D bounding box of the object to be grasped in the target image is determined. Based on the median depth of the 2D bounding box, a rotation and translation sampling technique is used to preliminarily estimate the pose of the object to be grasped, and multiple preliminary poses are obtained. According to each preliminary pose and the target image, a pose refinement network is used to refine the pose of the object to be grasped, and a refined pose is obtained. This improves the calculation speed and can meet the requirements of real-time control. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic flow chart of an object grasping method based on image segmentation and object pose estimation provided by an embodiment of the present invention;

[0029] Figure 2 A schematic structural diagram of an object grasping system based on image segmentation and object pose estimation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0031] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0032] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0033] In the embodiment of the present invention, sometimes a subscript such as W1 may be written as a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0034] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0035] Reference Manual Figure 1 , shows a flow chart of an object grasping method based on image segmentation and object pose estimation provided by an embodiment of the present invention.

[0036] An embodiment of the present invention provides an object grasping method based on image segmentation and object pose estimation. This method can be implemented by an object grasping device based on image segmentation and object pose estimation, which can be a terminal or a server. The processing flow of the object grasping method based on image segmentation and object pose estimation can include the following steps:

[0037] S1: Obtain the original image of the object to be grasped.

[0038] Specifically, an industrial computer controls an RGBD camera to obtain the original image of the object to be grasped.

[0039] It should be noted that the original image includes a color image and a depth image of the object to be captured.

[0040] It's important to note that an RGBD camera combines a traditional RGB camera with a depth sensor. It not only captures color images (RGB images) of objects, but also obtains depth information (D) for each pixel, representing the distance between the object and the camera. This enables RGBD cameras to provide depth information in three-dimensional space, helping computer vision systems perceive and understand the shape, position, and spatial relationships of objects in a scene. Common RGBD cameras include Microsoft's Kinect and Intel's RealSense series, which are widely used in robotics, virtual reality, augmented reality, and 3D modeling.

[0041] In this invention, by capturing the original image of the object to be grasped using an RGBD camera, both the color image and depth information of the object can be obtained, providing the object's accurate position and shape in three-dimensional space. This enables the system to better understand the spatial relationships of objects, providing an accurate data foundation for subsequent object recognition, pose estimation, and grasping operations, improving grasping accuracy and stability, especially in complex environments or with partial occlusion.

[0042] S2: Preprocess the original image to obtain the target image.

[0043] Specifically, the industrial computer performs an alignment operation on the acquired color image and depth image to obtain a target image.

[0044] Optionally, the color image and the depth image are aligned according to the following formula:

[0045]

[0046] Among them, I RGB (x,y) represents the pixel value at the coordinate (x,y) in the original image, Represents the coordinates in the depth image The pixel value at .

[0047] It should be noted that alignment involves spatially aligning the pixels in the color image and depth image, ensuring that their pixel positions match one-to-one, resulting in a target image that combines color and depth information. This operation typically uses the camera's intrinsic parameter matrix and coordinate transformation methods to map the three-dimensional spatial information in the depth image to the two-dimensional pixel coordinate system of the color image. This ensures that depth values ​​accurately correspond to color information, providing an accurate data foundation for subsequent image analysis and processing.

[0048] In the present invention, by performing an alignment operation on the color image and the depth image, the pixel positions of the two are matched one by one, thereby obtaining a target image that combines color and depth information. This alignment operation uses the camera's intrinsic parameter matrix and coordinate transformation method to map the three-dimensional spatial information in the depth image to the two-dimensional pixel coordinate system of the color image, so that the depth value can accurately correspond to the color information. In this way, the visual features and spatial features of the object can be obtained simultaneously, providing an accurate data basis for subsequent tasks such as object recognition, pose estimation, and grasping, reducing noise and errors, and improving the accuracy and robustness of subsequent processing, especially in complex environments and when objects are occluded, ensuring the accurate positioning and grasping of objects.

[0049] S3: Based on the target image, the object mask of the object to be grasped is generated through the SAM model and the DINOv2 model.

[0050] It's important to note that the SAM (Segment Anything Model) is an image segmentation model designed to automatically generate object masks from images. Using deep learning technology, the SAM model accurately segments objects in images. It's applicable not only to known object types but also to unknown objects.

[0051] The Distilled Knowledge Network v2 (DINOv2) model is a self-supervised learning model that focuses on image feature extraction and representation learning. Pre-trained on large-scale unlabeled data, it extracts fine-grained visual features from images, including semantic, appearance, and geometric information. DINOv2 demonstrates exceptional performance in tasks such as image segmentation and object detection, particularly in zero-shot settings. It effectively associates visual information with context, enhancing the model's ability to process unseen objects.

[0052] Optionally, the SAM model uses 42 view templates rendered in Blender.

[0053] It's important to note that Blender is open-source 3D creation software widely used in a variety of fields, including modeling, rendering, animation, sculpting, texturing, simulation, and video editing. It provides a comprehensive suite of tools for artists and developers creating 3D content. Blender supports a wide range of 3D production workflows, from modeling, sculpting, and animation to rendering and physics simulation. Its rendering engines, such as Eevee and Cycles, produce high-quality images and animations, and are widely used in film, game development, and virtual reality production.

[0054] In this paper, by combining the SAM model, the DINOv2 model, and 42 viewpoint templates rendered by Blender, the accuracy and robustness of object segmentation and pose estimation can be significantly improved. Using these 42 viewpoint templates, the system can generate object images from multiple angles, increasing the diversity of visual information. This helps the system accurately identify and segment objects even when they are partially occluded or in complex environments, thereby achieving efficient object grasping and pose estimation.

[0055] In a possible implementation, S3 specifically includes sub-steps S301 to S303:

[0056] S301: Segment the target image using the SAM model to generate multiple candidate segmentation regions.

[0057] Optionally, the target image is segmented according to the following formula to generate multiple candidate segmentation regions.

[0058]

[0059] Among them, M represents the candidate segmentation area, C represents the confidence score corresponding to the candidate segmentation area, Represents the mask decoder of the SAM model, which is used to generate candidate segmentation regions and confidence scores. Represents the image encoder of the SAM model, which is used to extract image features. I represents the input target image. Represents the prompt encoder, which is used to input prompt information, and Pr represents the prompt information.

[0060] S302: Extracting image features of each candidate segmentation region through the DINOv2 model.

[0061] Optionally, image features of each candidate segmentation region are extracted according to the following formula:

[0062]

[0063] in, F m represents the image features of the mth candidate segmentation region extracted, Represents the image feature extraction process of the DINOv2 model, Preprocess() represents the preprocessing of the candidate segmentation area (such as resizing or normalization), M m Indicates the m candidate segmentation regions.

[0064] Optionally, the image features include: semantic features, appearance features, and geometric features.

[0065] S303: Generate an object mask according to the image features of each candidate segmentation region through a visibility ratio weight adjustment mechanism.

[0066] It should be noted that the visibility ratio weight adjustment mechanism is a strategy used in image processing or computer vision tasks to deal with partial occlusion or incomplete visibility of objects. In image segmentation or object detection, objects may partially disappear due to perspective or background occlusion. This mechanism analyzes the ratio of the visible area to the occluded area of ​​the object in the image and dynamically adjusts the weights of each part to ensure that the object mask is generated as accurately as possible. When segmenting objects, the visibility ratio weight adjustment mechanism can give higher weights based on the visible part of the object, thereby improving the accuracy of the segmentation results, especially in complex or partially occluded scenes.

[0067] In this invention, the SAM model uses deep learning technology to accurately segment objects in images and adapt to the detection of known and unknown objects, especially in complex backgrounds and partial occlusion. The DINOv2 model, as a self-supervised learning model, can extract fine-grained visual features from unlabeled data, enhance the model's ability to process new objects, and improve the accuracy of image segmentation and object detection. Combined with the visibility ratio weight adjustment mechanism, the system can adjust the weight according to the visible part of the object, especially in the case of object occlusion or complex background, to ensure the accuracy of the segmentation results.

[0068] In a possible implementation, S303 specifically includes sub-steps S3031 to S3035:

[0069] S3031: Perform matching scores on each candidate segmentation region according to the semantic features to obtain a semantic matching score for each candidate segmentation region.

[0070] Optionally, a matching score is performed on each candidate segmentation region according to the following formula to obtain a semantic matching score of each candidate segmentation region:

[0071]

[0072] Among them, S sem Represents the semantic matching score of the candidate segmentation area, topK represents the selection of the top K candidate segmentation areas with the strongest similarity, Represents the mth candidate segmentation region CLS embedded, Represents the kth template image CLS embedded.

[0073] It should be noted that CLS embedding is a feature representation in the DINOv2 model, which extracts high-level semantic features from images through self-supervised learning. Each candidate segmentation region and template image generates a CLS embedding to represent its semantic features. Suppose we have an image containing multiple objects, and the goal is to identify a specific cup from it. Multiple candidate segmentation regions are generated by the SAM model, and then the CLS embedding of each region is extracted using the DINOv2 model. The cosine similarity between the CLS embedding of each candidate region and the CLS embedding of the cup template is calculated, and finally the semantic matching score of each candidate region is obtained. The candidate region with the highest score will be selected as the object mask of the cup.

[0074] S3032: Perform matching scores on each candidate segmentation region based on the appearance features to obtain an appearance matching score for each candidate segmentation region.

[0075] Optionally, a matching score is performed on each candidate segmentation region according to the following formula to obtain an appearance matching score of each candidate segmentation region:

[0076]

[0077] Among them, S appe represents the appearance matching score of the candidate segmentation region, Represents the mth candidate segmentation region The number of block embeddings, max means taking the maximum value, represents the j-th block embedding of the m-th candidate segmentation region, Represents the best matching template image The i-th block embedding of .

[0078] It's important to note that the appearance match score evaluates the match between a candidate segmentation region and the target object by analyzing its appearance features. Suppose we have an image containing multiple objects, and the goal is to identify a specific cup. The SAM model generates multiple candidate segmentation regions, and the DINOv2 model extracts the block embeddings for each region. The cosine similarity between the block embedding of each candidate region and the block embedding of the cup template is calculated, ultimately yielding an appearance match score for each candidate region. The candidate region with the highest score is selected as the object mask for the cup.

[0079] S3033: Perform matching scores on each candidate segmentation region based on the geometric features to obtain a geometric matching score for each candidate segmentation region.

[0080] Optionally, a matching score is performed on each candidate segmentation region according to the following formula to obtain a geometric matching score of each candidate segmentation region:

[0081]

[0082] Among them, S geo represents the geometric matching score of the candidate segmentation region, represents the 2D bounding box of the mth candidate segmentation region, Represents the 2D bounding box of the object to be grasped after projection from the initial pose.

[0083] It should be noted that in the object grasping method based on image segmentation and object pose estimation, the geometric matching score is to evaluate the degree of match between the candidate segmentation region and the target object by analyzing the geometric features of the candidate segmentation region. Geometric features usually include information such as the bounding box, shape, and position of the object. The degree of geometric matching is evaluated by calculating the intersection and union between the bounding box of the candidate segmentation region and the bounding box of the target object. Suppose we have an image containing multiple objects, and the goal is to identify a specific cup from it. Multiple candidate segmentation regions are generated through the SAM model, and then the IoU between the bounding box of each candidate region and the bounding box of the cup template is calculated to finally obtain the geometric matching score of each candidate region. The candidate region with the highest score will be selected as the object mask of the cup.

[0084] S3034: Using the visibility ratio weight adjustment mechanism, a weighted sum is performed on the semantic matching score, the appearance matching score, and the geometric matching score of each candidate segmentation region to obtain a comprehensive matching score for each candidate segmentation region.

[0085] Optionally, according to the following formula, the semantic matching score, appearance matching score, and geometric matching score of each candidate segmentation region are weighted and summed through the visibility ratio weight adjustment mechanism to obtain a comprehensive matching score for each candidate segmentation region:

[0086]

[0087] Among them, s m represents the comprehensive matching score of the candidate segmentation region, r vis Represents the visibility ratio weight.

[0088] S3035: Select the candidate segmentation region with the highest comprehensive matching score as the object mask.

[0089] In the present invention, by weighting the matching scores of semantic features, appearance features and geometric features and combining them with the visibility ratio weight adjustment mechanism, the accuracy and robustness of object segmentation can be effectively improved. Semantic features, appearance features and geometric features evaluate the segmented area of ​​the object from different angles, respectively, so that the score of each candidate segmentation area is more comprehensive and accurate. The visibility ratio weight adjustment mechanism can better cope with the situation of object occlusion and partial visibility by adjusting the weight of each candidate segmentation area. By giving priority to the scores of those visible parts, the mis-segmentation caused by occlusion or incomplete object information can be reduced, thereby improving the accuracy of the mask.

[0090] Specifically, the industrial computer sends the target image to the server, and the server generates an object mask.

[0091] Furthermore, the server performs pose estimation based on the generated object mask to obtain a refined pose.

[0092] S4: Determine the 2D bounding box of the object to be grasped in the target image based on the object mask.

[0093] In this paper, a 2D bounding box is generated from an object mask to clearly determine the object's position and size in the image, providing accurate spatial information for subsequent pose estimation. The bounding box helps the algorithm focus on the valid area of ​​the object, reducing interference from background and noise on pose estimation, thereby improving estimation accuracy.

[0094] S5: Based on the median depth of the 2D bounding box, the pose of the object to be grasped is preliminarily estimated through the rotation and translation sampling technique to obtain multiple preliminary poses.

[0095] It's important to note that median depth refers to the depth value located in the middle of the depth data for an object or scene in 3D space. In a depth image, each pixel represents the distance from the object's surface to the camera. Median depth is calculated by sorting all depth values ​​and selecting the one in the middle.

[0096] It should be noted that rotation and translation sampling technology is a method for object pose estimation. It determines the position and orientation of an object by sampling the pose of the object in three-dimensional space. Specifically, rotation and translation sampling technology generates multiple sampling points uniformly around the object. These points include the object's rotation angle and position offset, thereby generating multiple possible object poses. During the rotation phase, the sampling will rotate around the object at different angles. During the translation phase, the sampling involves different position offsets of the object in space. Through this technology, a set of possible object poses can be obtained for subsequent precise estimation and optimization, thereby improving the accuracy of pose estimation.

[0097] In this paper, median depth provides stable depth information by eliminating noise and extreme values ​​in the depth image, providing a reliable starting point for pose estimation. Rotational translation sampling generates multiple sampling points evenly around the object, covering a wide range of possible poses and positions. This ensures that the system can fully explore the object's pose, especially in complex and occluded environments.

[0098] In a possible implementation, S5 specifically includes sub-steps S501 to S503:

[0099] S501: The position of the 3D point corresponding to the median depth of the 2D bounding box is used as the center position of the object to be grasped.

[0100] In the present invention, by using the 3D point corresponding to the median depth of the 2D bounding box as the center position of the object to be grasped, it helps to ensure that the pose estimation starts from the true center of the object, avoids deviations caused by errors, and makes subsequent pose estimation more accurate.

[0101] S502: Sampling the poses of the object to be grasped under multiple viewing angles on the icosahedron at the center of the object in a uniform sampling manner.

[0102] In the present invention, by uniformly sampling multiple perspectives on the icosahedron at the center of the object, comprehensive sampling can be performed from multiple angles, which means that the algorithm can consider all possible rotation angles of the object, thereby avoiding errors or omissions that may be caused by relying solely on a single perspective, and enhancing the reliability of pose estimation.

[0103] S503: Perform in-plane rotation sampling on the postures at each viewing angle to obtain multiple preliminary postures.

[0104] In this paper, by performing in-plane rotation sampling on the pose at each sampling viewpoint, the object's rotation information is further optimized, generating multiple different preliminary poses and providing a diverse set of candidate poses for subsequent precise optimization. This approach is particularly well-suited for handling complex environments and occluded scenes, ensuring more accurate and robust pose estimation in practical applications.

[0105] S6: Based on each preliminary pose and the target image, the pose of the object to be grasped is refined and estimated through the pose refinement network to obtain a refined pose.

[0106] Optionally, the pose refinement network is specifically a convolutional neural network.

[0107] It's important to note that convolutional neural networks (CNNs) are deep learning models widely used in fields such as image recognition, video analysis, and natural language processing. CNNs process data by mimicking the workings of biological visual systems. They consist of multiple convolutional layers, pooling layers, and fully connected layers. In the convolutional layers, the network convolves the input image with filters (also called convolution kernels) to extract local features such as edges and textures. Pooling layers downsample the image to reduce the dimensionality of the data while retaining important information. Through multiple layers of convolution and pooling operations, CNNs are able to gradually learn higher-level features, ultimately performing tasks such as classification or regression through fully connected layers.

[0108] In this paper, CNN gradually extracts local and global features from the image through convolutional and pooling layers, enabling the model to learn deep image information, such as object edges, textures, and shapes. This feature extraction capability enables the pose refinement process to extract more fine-grained features from the target image, effectively determining the matching degree between different poses.

[0109] In a possible implementation, S6 specifically includes sub-steps S601 to S604:

[0110] S601: Generate multiple renderings of the object to be grasped according to each preliminary pose.

[0111] S602: Extracting a feature map of the target image through a convolutional neural network.

[0112] S603: Compare the matching degree between each rendering image and the feature map to obtain a score of the posture in each rendering image.

[0113] Optionally, the matching degree between each rendering image and the feature map is compared according to the following formula to obtain the score of the pose in each rendering image:

[0114]

[0115] Among them, s hyp represents the score of the pose in the rendered image, Represents the sparse point coordinates of the y-th rendering image, Represents the sparse point set of the y-th rendering image, min represents the minimum value, Represents the sparse point coordinates of the z-th feature map, represents the sparse point set of the z-th feature map, represents the assumed rotation matrix, represents the transpose operation, represents the hypothetical translation vector, Represents a sparse point set points.

[0116] It's important to note that both renderings and feature maps are represented as sparse sets of points, with each point representing a key feature in the image. Suppose we have an image containing multiple objects, and our goal is to identify a specific cup. We generate multiple renderings using preliminary poses, then use a CNN to extract the feature map of the target image. We calculate the degree of match between the sparse point set of each rendering and the sparse point set of the feature map, ultimately yielding a pose score for each rendering. The pose represented by the rendering with the highest score is selected as the refined pose.

[0117] S604: Select the pose with the highest score as the refined pose.

[0118] In the present invention, by generating multiple renderings and using CNN to extract the feature map of the target image, the matching degree of different poses can be accurately compared. When faced with image data in different environments, CNN can effectively handle noise, occlusion, and images under different lighting conditions. This enables the pose refinement network to maintain high robustness and adaptability in various dynamic scenes, and can still provide accurate pose estimates even in the case of unseen objects or complex visual environments. The CNN deep learning framework can process large amounts of image data in a relatively short time through efficient convolution operations and feature map extraction, thereby accelerating the calculation process of pose refinement.

[0119] Furthermore, the server returns the generated refined pose to the industrial computer, and the industrial computer determines the joint angles of the robotic arm based on the refined pose.

[0120] S7: Convert the refined pose from the camera coordinate system to the robot coordinate system.

[0121] Optionally, convert the refined pose from the camera coordinate system to the robot coordinate system according to the following formula:

[0122]

[0123] in, represents the refined pose in the robotic arm coordinate system, represents the transformation matrix from the camera coordinate system to the robotic arm coordinate system (which can be obtained through the calibration process), Represents the refined pose in the camera coordinate system.

[0124] In this invention, the refined pose is converted from the camera coordinate system to the robotic arm coordinate system, ensuring that the object's pose information can be accurately applied to the robotic arm's control system. Because the camera and robotic arm are in different coordinate systems, this conversion process eliminates the coordinate discrepancy between them, allowing the robotic arm to perform precise operations based on the object's position information acquired by the camera. This improves the robot's accuracy and stability in grasping tasks, ensuring that the robotic arm can correctly position and execute grasping actions.

[0125] S8: Calculate the target grasping pose based on the predefined fixed transformation between the end effector and the object to be grasped in the robot arm coordinate system.

[0126] It should be noted that a predefined fixed transformation between the end effector and the object to be grasped refers to a predefined and fixed spatial relationship between the end effector (such as a robotic gripper or fixture) and the object to be grasped in the robotic system. This transformation typically includes information such as the relative position and orientation (rotation) between the object and the end effector. This fixed transformation ensures that the end effector can accurately contact the object when the robotic arm performs a grasping task, regardless of its position or posture. This fixed transformation is usually predefined in the robotic control system and ensures that the end effector can correctly grasp the object through precise coordinate system transformation.

[0127] Furthermore, in robotic arm operations, the end effector is the final component of the arm, used for direct contact with external objects, such as a gripper, suction cup, or manipulator. The object to be grasped is the target object that the robotic arm needs to manipulate. A fixed transformation refers to a predefined, fixed geometric relationship between the end effector and the object to be grasped. A fixed transformation consists of two components: rotation and translation, and is typically represented by a 4x4 homogeneous transformation matrix.

[0128] In a possible implementation, S8 specifically includes:

[0129] The target grasping pose is calculated according to the following formula:

[0130]

[0131] in, represents the target grasping pose in the robot arm coordinate system, represents the refined pose in the robotic arm coordinate system, Represents the inverse of a predefined fixed transformation between the end effector and the object to be grasped.

[0132] In this invention, by calculating the target grasping pose based on a predefined fixed transformation between the end effector and the object to be grasped, it is possible to ensure that the robotic arm's end effector always contacts the object in the correct posture and position. The fixed transformation provides a stable spatial relationship between the object and the end effector, allowing the robotic arm to accurately interact with the object during grasping tasks, regardless of how the object's specific position or pose changes. In this way, the system can achieve more efficient and stable grasping operations, avoid misgrasping due to posture changes, and improve the success rate of grasping tasks and the accuracy of robot operations.

[0133] S9: The target grasping posture is converted into the joint angles of the robotic arm through the inverse kinematics algorithm.

[0134] It should be noted that the inverse kinematics algorithm is a mathematical method used to calculate the angles or positions of robot joints. Its purpose is to reversely solve the angles or displacements that each joint needs to achieve based on the target position and posture of the end effector of the robotic arm. Typically, the robot control system requires the end effector (such as a robotic gripper) to reach a specific spatial position and orientation, and the inverse kinematics algorithm calculates how to adjust the angles or positions of each joint of the robotic arm to achieve this goal based on the position and posture information of the end effector. The inverse kinematics problem is very important in robotics because it converts targets in three-dimensional space into instructions in joint space, ensuring that the robotic arm can accurately perform tasks such as grasping and moving.

[0135] In this invention, the target grasping posture is converted into the joint angles of the robotic arm through an inverse kinematics algorithm, ensuring that the robotic arm can accurately perform grasping tasks according to the predetermined target position and posture. The inverse kinematics algorithm reversely calculates the required angles or displacements of each joint based on the target position and posture of the end effector, allowing the robotic arm to accurately control the movement of each joint, thereby achieving high-precision grasping operations. This process can effectively convert the target in three-dimensional space into joint space instructions for the robotic arm, ensuring that the robotic arm can accurately and stably perform grasping tasks in complex environments, thereby improving the efficiency and accuracy of robot operations.

[0136] S10: Control the robot arm to grasp the object according to the joint angle.

[0137] Specifically, the industrial computer controls the robotic arm to grasp the object according to the calculated joint angle.

[0138] In this invention, an industrial computer controls the robotic arm to perform grasping based on calculated joint angles, ensuring the arm's precise grasping. This control method enables the robotic arm to move precisely according to the calculated joint angles, avoiding grasping failures caused by errors or instability, ensuring the accuracy and reliability of the grasping action. This significantly improves the efficiency and success rate of robotic grasping, especially in complex environments or when objects are in irregular postures.

[0139] Reference Manual Figure 2 , which shows a structural diagram of an object grasping system based on image segmentation and object pose estimation provided by the present invention.

[0140] The present invention further provides an object grasping system 20 based on image segmentation and object pose estimation, which is applied to the above-mentioned object grasping method based on image segmentation and object pose estimation, comprising:

[0141] Processor 201;

[0142] The memory 202 stores computer-readable instructions, which, when executed by the processor 201 , implement the object grasping method based on image segmentation and object pose estimation as described in the method embodiment.

[0143] The object grasping system 20 based on image segmentation and object pose estimation provided by the present invention can execute the above-mentioned object grasping method based on image segmentation and object pose estimation, and achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0144] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0145] In an embodiment of the present invention, an object mask of an object to be grasped is generated by using a SAM model and a DINOv2 model. When the object is partially occluded or there are multiple objects in the scene, the accuracy of pose estimation is improved, which can meet the needs of high-precision grasping. According to the object mask, the 2D bounding box of the object to be grasped in the target image is determined. Based on the median depth of the 2D bounding box, a rotation and translation sampling technique is used to preliminarily estimate the pose of the object to be grasped, and multiple preliminary poses are obtained. According to each preliminary pose and the target image, a pose refinement network is used to refine the pose of the object to be grasped, and a refined pose is obtained. This improves the calculation speed and can meet the requirements of real-time control.

[0146] It should be understood that the processor in the embodiments of the present invention may be a central processing unit (CPU), but may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0147] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0148] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0149] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0150] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0151] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0152] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0153] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0157] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0158] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the object grasping method based on image segmentation and object pose estimation as described in the method embodiment is implemented.

[0159] The computer-readable storage medium provided by the present invention can implement the steps and effects of the object grasping method based on image segmentation and object pose estimation of the above-mentioned method embodiment. To avoid repetition, the present invention will not go into details.

[0160] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0161] In an embodiment of the present invention, an object mask of an object to be grasped is generated by using a SAM model and a DINOv2 model. When the object is partially occluded or there are multiple objects in the scene, the accuracy of pose estimation is improved, which can meet the needs of high-precision grasping. According to the object mask, the 2D bounding box of the object to be grasped in the target image is determined. Based on the median depth of the 2D bounding box, a rotation and translation sampling technique is used to preliminarily estimate the pose of the object to be grasped, and multiple preliminary poses are obtained. According to each preliminary pose and the target image, a pose refinement network is used to refine the pose of the object to be grasped, and a refined pose is obtained. This improves the calculation speed and can meet the requirements of real-time control.

[0162] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

[0163] There are a few points to note:

[0164] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0165] (2) For the sake of clarity, the thickness of layers or regions in the drawings used to describe the embodiments of the present invention are exaggerated or reduced, that is, these drawings are not drawn to scale. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly on" or "under" the other element or intervening elements may be present.

[0166] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to form new embodiments.

[0167] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An object grasping method based on image segmentation and object pose estimation, characterized in that: include: S1: Obtain the original image of the object to be grasped; S2: Preprocess the original image to obtain a target image; S3: Generate an object mask of the object to be grasped through the SAM model and the DINOv2 model according to the target image; S4: Determine a 2D bounding box of the object to be grasped in the target image according to the object mask; S5: Based on the median depth of the 2D bounding box, preliminarily estimate the pose of the object to be grasped by using a rotation and translation sampling technique to obtain multiple preliminary poses; S6: According to each of the preliminary poses and the target image, a pose refinement network is used to refine and estimate the pose of the object to be grasped to obtain a refined pose; S7: converting the refined pose from the camera coordinate system to the robot arm coordinate system; S8: Calculating a target grasping pose in the manipulator coordinate system based on a predefined fixed transformation between the end effector and the object to be grasped; S9: converting the target grasping posture into the joint angle of the robotic arm through an inverse kinematics algorithm; S10: Controlling the robot arm to grasp the object to be grasped according to the joint angle; Wherein, the S3 specifically includes: S301: Segment the target image using the SAM model to generate multiple candidate segmentation regions; S302: Extracting image features of each candidate segmentation region using the DINOv2 model; S303: Generate the object mask according to the image features of each candidate segmentation region through a visibility ratio weight adjustment mechanism; The image features include semantic features, appearance features and geometric features; The S303 specifically includes: S3031: performing matching scores on each of the candidate segmentation regions according to the semantic features to obtain semantic matching scores for each of the candidate segmentation regions; S3032: performing a matching score on each of the candidate segmented regions based on the appearance features to obtain an appearance matching score for each of the candidate segmented regions; S3033: performing matching scores on each of the candidate segmentation regions according to the geometric features to obtain a geometric matching score for each of the candidate segmentation regions; S3034: performing weighted summation of the semantic matching score, the appearance matching score, and the geometric matching score of each candidate segmentation region through the visibility ratio weight adjustment mechanism to obtain a comprehensive matching score of each candidate segmentation region; S3035: Selecting the candidate segmentation region with the highest comprehensive matching score as the object mask; Wherein, the posture refinement network is specifically a convolutional neural network; Wherein, the S6 specifically includes: S601: generating multiple renderings of the object to be grasped according to each of the preliminary poses; S602: Extracting a feature map of the target image through the convolutional neural network; S603: Compare the matching degree between each of the renderings and the feature map to obtain a score for the posture in each of the renderings: ； in, represents the score of the pose in the rendered image, Indicates the The sparse point coordinates of the rendered image, Indicates the A sparse point set of a rendering image, min means taking the minimum value, Indicates the The sparse point coordinates of the feature map, Indicates the A sparse point set of feature maps, represents the assumed rotation matrix, represents the transpose operation, represents the hypothetical translation vector, Represents a sparse point set The number of points; S604: Select the posture with the highest score as the refined posture.

2. The object grasping method based on image segmentation and object pose estimation according to claim 1, characterized in that: The S5 specifically includes: S501: taking the position of the 3D point corresponding to the median depth of the 2D bounding box as the center position of the object to be grasped; S502: Sampling the poses of the object to be grasped under multiple viewing angles on an icosahedron at the center of the object in a uniform sampling manner; S503: Perform in-plane rotation sampling on the postures under each of the viewing angles to obtain a plurality of preliminary postures.

3. The object grasping method based on image segmentation and object pose estimation according to claim 1, characterized in that: The S8 is specifically: The target grasping pose is calculated according to the following formula: ； in, represents the target grasping pose in the robot arm coordinate system, represents the refined pose in the robotic arm coordinate system, represents the inverse operation of the predefined fixed transformation between the end effector and the object to be grasped.

4. An object grasping system based on image segmentation and object pose estimation, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the object grasping method based on image segmentation and object pose estimation according to any one of claims 1 to 3 is implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the object grasping method based on image segmentation and object pose estimation as claimed in any one of claims 1 to 3 is implemented.

Images