Transparent object mechanical arm grabbing method and system based on semantic geometry fusion perception

Abstract

The invention discloses a transparent object mechanical arm grabbing method and system based on semantic geometry fusion perception, and the method comprises the steps: obtaining an RGB image and an original depth image of a to-be-grabbed object, carrying out the fusion prediction based on a pre-trained monocular perception network model, and outputting a segmentation mask and a restoration depth value of the to-be-grabbed transparent object; performing restoration processing on the original depth image through the restoration depth value, and outputting a complete three-dimensional point cloud of the to-be-grabbed object; object grabbing detection is carried out based on the grabbing detection network model, and a plurality of candidate 6-DoF grabbing postures and confidence degree scores and disturbance tolerance degrees corresponding to the grabbing postures are obtained; and non-maximum suppression is conducted through a collision test, selection is conducted, and the optimal grabbing posture is determined to achieve grabbing of the transparent object. According to the method, the object grabbing accuracy in a real scene can be improved. The transparent object mechanical arm grabbing method and system based on semantic and geometric fusion perception can be widely applied to the technical field of robot grabbing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot grasping, and in particular to a transparent object robot arm grasping method and system based on semantic-geometric fusion perception. BACKGROUND

[0002] Robot grasping is a core task in the field of automation, and is widely used in industrial manufacturing, logistics sorting, service robots and other scenarios. Traditional grasping systems rely on depth cameras (such as Kinect, RealSense) to perceive the three-dimensional geometric information of objects, and then plan the grasping pose. However, when facing transparent objects (such as glass cups, plastic bottles, etc.), these systems will encounter great challenges. Due to the optical properties (refraction, reflection) of transparent objects, depth cameras based on structured light or binocular vision cannot obtain accurate depth information, and there are a large number of noises and holes in the generated point cloud, so accurately segmenting (distinguishing objects from background) and depth estimation (judging distance) of transparent objects is still a major challenge, making subsequent grasping planning algorithms unable to work normally.

[0003] Further, to solve the problem of grasping transparent objects, related technologies have proposed various solutions, such as using a laser radar or a high-frequency structured light scanner to obtain the depth information of transparent objects, but this solution has high hardware costs and slow scanning speed, making it difficult to meet the real-time grasping requirements, or a method based on multi-view geometry. By taking pictures of transparent objects from multiple angles, a three-dimensional model is reconstructed using the contour information, but this method requires a complex multi-camera system or multiple movements of the robot arm for shooting, resulting in high system complexity and computational overhead, or a method based on pure RGB image learning. Directly learning the grasping pose from the RGB image completely avoids the depth perception problem, but this type of method has poor generalization ability and is not suitable for objects and complex scenes that have not been seen before, or a step-by-step processing method. First, segment the transparent object, then perform independent depth completion, and finally use traditional grasping planning algorithms. However, the steps are independent of each other, and the errors will accumulate, and the complementary nature of semantic and geometric information is not fully utilized. SUMMARY

[0004] To solve the above technical problems, the purpose of the present application is to provide a transparent object robot arm grasping method and system based on semantic-geometric fusion perception, which can directly learn from the original point cloud input and generate reliable 6-DoF grasping poses, improving the grasping accuracy of objects in real scenes.

[0005] The first technical solution adopted by the present application is a transparent object robot arm grasping method based on semantic-geometric fusion perception, comprising the following steps: An RGB image and an original depth image of the object to be grasped are acquired, and a pre-trained monocular perception network model is used to perform fusion prediction on the RGB image of the object to be grasped, to output a segmentation mask and a repaired depth value of the transparent object to be grasped. Based on the segmentation mask of the transparent object to be grasped, a region is determined, the original depth image is repaired by using the repaired depth value, and a complete three-dimensional point cloud of the object to be grasped is output in combination with the camera intrinsic parameters. Based on the grasping detection network model, the complete three-dimensional point cloud of the object to be grasped is subjected to object grasping detection, to obtain a plurality of candidate 6-DoF grasping poses, and a confidence score and a perturbation tolerance corresponding to each grasping pose. Through a collision test, non-maximum suppression is performed on the plurality of candidate 6-DoF grasping poses, and each grasping pose is selected in combination with the confidence score and the perturbation tolerance corresponding to the grasping pose, to determine an optimal grasping pose and realize transparent object grasping.

[0006] Further, the step of acquiring an RGB image and an original depth image of the object to be grasped, and performing fusion prediction on the RGB image of the object to be grasped based on a pre-trained monocular perception network model to output a segmentation mask and a repaired depth value of the transparent object to be grasped specifically comprises: An RGB image and an original depth image of the object to be grasped in a target scene are acquired by using a camera; The RGB image of the object to be grasped is input into a pre-trained monocular perception network model, and the pre-trained monocular perception network model comprises a Transformer-based visual encoder, a reorganization module, and an iterative fusion decoder; Based on the visual encoder of the pre-trained monocular perception network model, the RGB image of the object to be grasped is subjected to cutting processing, linear projection, and position information adding processing in sequence, to obtain visual tokens of the object to be grasped; Based on the reorganization module of the pre-trained monocular perception network model, the visual tokens of the object to be grasped are converted into corresponding feature maps and subjected to reorganization processing, to obtain multi-scale feature maps of the object to be grasped; Based on the iterative fusion decoder of the pre-trained monocular perception network model, the multi-scale feature maps of the object to be grasped are subjected to semantic geometric fusion prediction, to output a segmentation mask and a repaired depth value of the transparent object to be grasped.

[0007] Further, the step of performing semantic geometric fusion prediction on the multi-scale feature maps of the object to be grasped based on the iterative fusion decoder of the pre-trained monocular perception network model to output a segmentation mask and a repaired depth value of the transparent object to be grasped specifically comprises: The multi-scale feature maps of the object to be grasped are input into the iterative fusion decoder of the pre-trained monocular perception network model; The channel attention module and the spatial attention module based on the iterative fusion decoder extract information of depth features and segmentation features in a multi-scale feature map of the object to be grabbed, to obtain semantic and geometric information. The semantic and geometric information are fused through symmetric multiplication interaction, and a segmentation mask and a repaired depth value of the transparent object to be grabbed are output.

[0008] Further, the segmentation mask of the transparent object to be grabbed is used for region judgment, the original depth image is repaired through the repaired depth value, and the camera intrinsic parameter is combined to output the complete three-dimensional point cloud of the object to be grabbed, which specifically includes: The segmentation mask of the transparent object to be grabbed is used for region judgment to obtain a transparent object region and a non-transparent object region. The transparent object region is repaired through the repaired depth value output by the pre-trained monocular perception network model, and the non-transparent object region is output through the original depth image to obtain a repaired depth map. The repaired depth map and the camera intrinsic parameter are combined to output the complete three-dimensional point cloud of the object to be grabbed.

[0009] Further, the complete three-dimensional point cloud of the object to be grabbed is subjected to object grabbing detection based on the grabbing detection network model to obtain a plurality of candidate 6-DoF grabbing poses, a confidence score corresponding to each grabbing pose, and a perturbation tolerance. The complete three-dimensional point cloud of the object to be grabbed is input into the grabbing detection network model. The point cloud feature extractor of the grabbing detection network model is used for down-sampling processing of the complete three-dimensional point cloud of the object to be grabbed to obtain an original feature dimension. The proximity network of the grabbing detection network model is used for grabbing point prediction of the complete three-dimensional point cloud of the object to be grabbed to output a plurality of candidate grabbing points and proximity vectors. The plurality of candidate grabbing points, the proximity vectors, and the original feature dimension are combined to obtain a point cloud cluster after grouping alignment. The OperationNet network of the grabbing detection network model is used for prediction of the point cloud cluster after grouping alignment to obtain in-plane rotation of the gripper, proximity distance, gripper width, and grabbing confidence score. The ToleranceNet network of the grabbing detection network model is used for prediction of the tolerance of each grabbing pose to perturbation according to the in-plane rotation of the gripper, the proximity distance, the gripper width, and the grabbing confidence score to output a plurality of candidate 6-DoF grabbing poses, a confidence score corresponding to each grabbing pose, and a perturbation tolerance.

[0010] Further, the step of combining the plurality of candidate grasping points, the approaching vector and the original feature dimension to obtain the grouped and aligned point cloud cluster, specifically comprises: For the plurality of candidate grasping points and the approaching vector, sampling points in the cylindrical region around the vector to obtain a plurality of sampling points; Converting the plurality of sampling points into a new local coordinate system with the grasping point as the origin and the Z-axis aligned with the approaching vector, and combining the original feature dimension for grouping and alignment to obtain the grouped and aligned point cloud cluster.

[0011] Further, the step of performing non-maximum suppression on the plurality of candidate 6-DoF grasping poses through collision test, and selecting according to the confidence score and the perturbation tolerance corresponding to each grasping pose to determine the optimal grasping pose to achieve transparent object grasping, specifically comprises: Performing non-maximum suppression on the plurality of candidate 6-DoF grasping poses through collision test to obtain a filtered grasping pose list; Layering the filtered grasping pose list according to the grasping confidence score, and sorting in each layer according to the perturbation tolerance to obtain the optimal grasping pose; Converting the optimal grasping pose into a motion trajectory of the robot arm and controlling the execution of grasping to achieve transparent object grasping.

[0012] The second technical solution adopted by the application is a transparent object robot arm grasping system based on semantic geometry fusion perception, comprising: A first module for acquiring an RGB image and an original depth image of the object to be grasped, and performing fusion prediction on the RGB image of the object to be grasped based on a pre-trained monocular perception network model, and outputting a segmentation mask and a repaired depth value of the transparent object to be grasped; A second module for performing region judgment based on the segmentation mask of the transparent object to be grasped, performing repair processing on the original depth image through the repaired depth value, and combining the camera intrinsic parameter to output a complete three-dimensional point cloud of the object to be grasped; A third module for performing object grasping detection on the complete three-dimensional point cloud of the object to be grasped based on a grasping detection network model to obtain a plurality of candidate 6-DoF grasping poses and a confidence score and a perturbation tolerance corresponding to each grasping pose; A fourth module for performing non-maximum suppression on the plurality of candidate 6-DoF grasping poses through collision test, and selecting according to the confidence score and the perturbation tolerance corresponding to each grasping pose to determine the optimal grasping pose to achieve transparent object grasping.

[0013] The method and system have the advantages that the method and system acquire an RGB image and an original depth image of a to-be-grabbed object, perform fusion prediction on the RGB image of the to-be-grabbed object based on a pre-trained monocular perception network model, output a segmentation mask and a repaired depth value of the to-be-grabbed transparent object, use the powerful depth learning capability of the monocular perception network (IFMP-Net), and simultaneously output an accurate transparent object segmentation mask and a repaired high-precision depth map only by using monocular RGB input, further perform region judgment based on the segmentation mask of the to-be-grabbed transparent object, perform repair processing on the original depth image by using the repaired depth value, combine the camera intrinsic parameters, and output complete three-dimensional point cloud of the to-be-grabbed object, further perform object grabbing detection on the complete three-dimensional point cloud of the to-be-grabbed object based on a grabbing detection network model, obtain a plurality of candidate 6-DoF grabbing poses, a confidence score corresponding to each grabbing pose, and a perturbation tolerance, learn and generate reliable 6-DoF grabbing poses directly from original point cloud input, introduce a grabbing affinity field to predict the perturbation tolerance, further improve the robustness of grabbing, enable the mechanical arm to select a stable grabbing pose that has a high success rate and is not sensitive to actual execution deviation, finally, perform non-maximum suppression on the plurality of candidate 6-DoF grabbing poses through a collision test, select the plurality of candidate 6-DoF grabbing poses based on the confidence score corresponding to each grabbing pose and the perturbation tolerance, and determine an optimal grabbing pose to realize transparent object grabbing, thereby improving the grabbing accuracy of objects in a real scene. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a step flowchart of the transparent object mechanical arm grabbing method based on semantic-geometric fusion perception of the application; Figure 2 is a structural block diagram of the transparent object mechanical arm grabbing system based on semantic-geometric fusion perception of the application; Figure 3 is a flowchart of the transparent object mechanical arm grabbing provided by the embodiment of the application; Figure 4 is an iterative fusion monocular perception network architecture diagram provided by the embodiment of the application; Figure 5 is a semantic-geometric fusion module architecture diagram provided by the embodiment of the application; Figure 6 is an end-to-end grabbing pose detection network model architecture diagram provided by the embodiment of the application. DETAILED DESCRIPTION

[0015] The application will be further described in detail below in combination with the drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0016] Firstly, the technical terms of the embodiments of the application are explained: 1) IFMP-Net (Iterative Fusion Monocular Perception Network): represents an end-to-end deep learning model proposed in the embodiments of the application. The network takes a single RGB image as the only input, and outputs high-precision transparent object depth estimation map and semantic segmentation mask in parallel through the internally integrated Transformer-based visual encoder, reorganization module and iterative fusion decoder. The core innovation lies in the "iterative fusion" mechanism, that is, through a lightweight gating unit cycle optimization of a semantic-geometric fusion module based on double attention (channel and spatial attention), the prediction result is gradually refined, thereby effectively solving the perception problem of transparent objects due to the lack of texture and complex optical properties.

[0017] 2) GraspNet-E2E (End-to-End Grasp Pose Detection Network): an end-to-end grasp pose detection network model. The network takes scene three-dimensional point cloud data as input, directly learns and predicts the executable, high-confidence and robust 6-DoF (six degrees of freedom) grasp pose of the object without human intervention or intermediate independent modules.

[0018] 3) End-to-End: represents a design paradigm of a deep learning model. In the embodiments of the application, it specifically refers to the two models of IFMP-Net and GraspNet-E2E. That is, the model starts from the most original input data (for IFMP-Net, it is RGB pixels, and for GraspNet-E2E, it is point cloud), through multiple layers of nonlinear transformation, automatically learns and extracts the required useful features, and outputs the final target result (for IFMP-Net, it is depth map and segmentation mask, and for GraspNet-E2E, it is grasp pose) in one step, eliminating all intermediate steps that require human intervention, feature engineering or multiple independent modules in series, reducing error accumulation, and improving system efficiency and performance.

[0019] 4) 6-DoF (Six Degrees of Freedom): A parameter used to completely describe the pose of a rigid body in three-dimensional space. It includes three degrees of freedom of position (translation along the X, Y, Z axes) and three degrees of freedom of orientation (rotation around the X, Y, Z axes). In the embodiments of the present application, the grasp pose of the robot gripper predicted by the GraspNet-E2E network is a 6-DoF pose.

[0020] 5) Point Cloud: A data set composed of a set of discrete points in three-dimensional space. Each point usually contains its spatial coordinates (X, Y, Z), and can be attached with color (R, G, B), normal, etc. In the embodiments of the present application, the point cloud generation module uses the repaired depth map and RGB image back projection to generate a point cloud that accurately describes the three-dimensional geometry of the surface of all objects (including transparent objects) in the scene, which is the input of the GraspNet-E2E network.

[0021] 6) Segmentation Mask: A two-dimensional image that corresponds one-to-one with the pixels of the input image. In the embodiments of the present application, it specifically refers to the transparent object segmentation mask output by the IFMP-Net. The value of each pixel identifies whether the pixel belongs to a transparent object (for example, the pixel value of the transparent object region is 1, and the pixel value of the background region is 0).

[0022] 7) Depth Inpainting Module: A key processing module in the system of the present application. Its function is to receive the original sensor depth map and the depth map and segmentation mask predicted by the IFMP-Net, and according to the guidance of the segmentation mask, to perform a hybrid data source strategy: for the region identified by the mask as a transparent object, use the more accurate depth value predicted by the IFMP-Net; for the non-transparent object background region, preferentially use the reliable depth value collected by the original depth camera. Finally, output a complete and high-precision fused depth map.

[0023] 8) Perturbation Tolerance: A quantitative indicator predicted by the ToleranceNet subnetwork in the GraspNet-E2E network. It is used to measure the maximum perturbation range that a grasp pose can tolerate in terms of pose (position and orientation). The higher the tolerance, the more robust the grasp pose, and the higher the probability of successful grasping even if there is some positioning error or object movement during actual execution. This indicator is one of the key bases for the grasp planning module to perform grasp selection and sorting.

[0024] 9) Non-Maximum Suppression (NMS): a post-processing algorithm in computer vision. In this invention, it is applied in the Grasp Planning Module. It is used to filter the large amount of spatially highly overlapping or similar redundant grasp poses generated by GraspNet-E2E, and only keep the one with the highest confidence in each local region, thus obtaining a sparse and representative list of grasp candidates, greatly improving the efficiency of subsequent processing (such as collision detection).

[0025] 10) Grasp Pose: refers to the spatial position and orientation that the robot's end-effector (such as a two-finger gripper) needs to reach in order to successfully and stably grasp the target object. It is a 6-DoF pose that defines the state of the gripper at the instant before the grasp action occurs, including the position of the gripper center point, the approach direction of the gripper, and the orientation of the gripper opening.

[0026] 11) Semantic-Geometric Fusion Module: the core submodule in the iterative fusion decoder of the IFMP-Net in this invention. This module uses channel attention mechanism (CAM) and spatial attention mechanism (SAM) to achieve bidirectional and adaptive feature fusion between depth prediction task (geometric information) and segmentation prediction task (semantic information) in a symmetric multiplication interaction manner. The purpose is to let the network automatically learn and utilize the complementarity between the two tasks: segmentation information provides object boundary constraints for depth prediction, and depth information provides spatial structure context for segmentation, thereby jointly improving the accuracy of transparent object perception. The specific calculation method is shown in Equations (1) to (4) in the specification.

[0027] 12) Grasp Affinity Field: a concept learned by the ToleranceNet submodule in the GraspNet-E2E network. It is not a direct output value, but an internal representation for measuring the robustness of grasp poses. This field quantifies how the stability of grasp poses changes in the space around the grasp point. The network learns and predicts this field, thereby indirectly outputs a specific perturbation tolerance value that represents the range of stability of the grasp pose in its parameter space.

[0028] 13) Channel Attention Mechanism (CAM) & Spatial Attention Mechanism (SAM): Two attention mechanisms used to enhance the expressiveness of deep learning models. Channel attention focuses on "what features are more important", it learns a weight for each channel of the feature map to emphasize informative feature channels and suppress unimportant ones. Spatial attention focuses on "where is more important", it learns a weight map for each spatial location of the feature map to highlight key regions. In the semantic-geometric fusion module of the invention, both are used in series to efficiently extract and fuse complementary information from another task.

[0029] 14) Visual Encoder: The first stage of the IFMP-Net network, responsible for extracting multi-level feature representations from input RGB images. The invention employs a Vision Transformer (ViT) as the encoder backbone network, which captures global contextual dependencies in images by dividing the image into patches and using self-attention mechanisms, which is crucial for understanding transparent objects that lack local texture.

[0030] 15) Token: In deep learning models based on Transformer architecture (such as the visual encoder used in the IFMP-Net of the invention), Token is the basic unit of input information. When processing images, the input image is first divided into multiple fixed-size, non-overlapping image patches. Each image patch is then flattened and mapped to a high-dimensional vector space through a linear projection layer, and this high-dimensional vector is called a Token. Each Token not only contains the visual information of its corresponding image patch, but also has a position encoding attached to it to inject the spatial position information of the image patch in the original image. These Tokens are then input as a sequence into the Transformer encoder for global context modeling through self-attention mechanisms. In short, Token is an intermediate carrier that converts images from pixel space to feature representation space that can be processed by the model.

[0031] 16) Reassembly Module: A module in the IFMP-Net between the visual encoder and the iterative fusion decoder. Its function is to reassemble and upsample the serialized tokens output by the ViT encoder to build a multi-scale pyramid feature map suitable for dense prediction tasks (depth estimation, segmentation), providing rich information for subsequent decoding and fusion.

[0032] Reference Figure 1 The application provides a transparent object mechanical arm grabbing method based on semantic geometry fusion perception, which comprises the following steps: S100, acquire the RGB image and the original depth image of the object to be grabbed, and perform fusion prediction on the RGB image of the object to be grabbed based on a pre-trained monocular perception network model, to output the segmentation mask and the repaired depth value of the transparent object to be grabbed; Specifically, the RGB image and the original depth image of the object to be grabbed in the target scene are acquired by a camera; the RGB image of the object to be grabbed is input into a pre-trained monocular perception network model, which comprises a vision encoder based on a Transformer, a reorganization module and an iterative fusion decoder; based on the vision encoder of the pre-trained monocular perception network model, the RGB image of the object to be grabbed is sequentially subjected to cropping processing, linear projection and position information adding processing, to obtain the vision token of the object to be grabbed; based on the reorganization module of the pre-trained monocular perception network model, the vision token of the object to be grabbed is converted into a corresponding feature map and subjected to reorganization processing, to obtain the multi-scale feature map of the object to be grabbed; based on the iterative fusion decoder of the pre-trained monocular perception network model, the multi-scale feature map of the object to be grabbed is subjected to semantic geometry fusion prediction, to output the segmentation mask and the repaired depth value of the transparent object to be grabbed.

[0033] In this embodiment, as shown in Figure 4 the proposed monocular perception network architecture diagram (IFMP-Net) is shown. It includes a vision encoder based on a Transformer, a reorganization module, and an iterative fusion decoder. Given an RGB input, our model can predict depth and segmentation mask through encoding, reorganization, and iterative fusion decoding.

[0034] The vision encoder uses a vision transformer (ViT) as our backbone network to extract multi-layer features. First, the input RGB image is cropped into non-overlapping image blocks, then these image blocks are embedded into tokens through linear projection, and the tokens are given position information and processed through multiple Transformer blocks with multi-head self-attention. The encoder consists of 12 Transformer blocks, from which 4 layers of tokens are selected, which are uniformly distributed from shallow to deep, for the subsequent modules.

[0035] In the reconstruction module, since ViT encodes image features into tokens with the same spatial resolution, we need to convert them back into feature maps for subsequent fusion and prediction. Visual tokens are reshaped into corresponding feature maps through stitching and projection. To fully utilize features at different levels, we represent them in a multi-scale manner, where deeper features correspond to smaller resolutions. The result of the reconstruction module is two four-layer pyramids, one for depth and one for segmentation.

[0036] A semantic and geometric fusion module is designed in the iterative fusion decoder to better utilize the complementary information from the two tasks. The weighted decoder is iteratively updated through a lightweight gating unit to progressively optimize the initial results. The final prediction result is obtained from the two prediction heads after the last iteration.

[0037] Furthermore, such as Figure 5 As shown, due to the optical properties of transparent objects, predicting depth and segmentation independently from a single RGB image is particularly difficult. To improve the performance of these two tasks, a novel attention-based fusion module is designed to fully utilize the complementary information from the two branches. Utilizing two feature pyramids for depth and segmentation from the previous module, we apply semantic and geometric fusion at each layer to integrate multi-scale features. Figure 5 In this example, we'll take a single layer of features. Given a depth feature at a certain scale and... Segmentation features First, a channel attention module (CAM) and a spatial attention module (SAM) are applied sequentially to extract meaningful cues. Leveraging this powerful representational capability, the module can automatically learn important semantic and geometric information implicit in depth and segmentation features. Then, these extracted information are fused through symmetric multiplication. Specifically, the channel attention and spatial attention are computed as follows:

[0038]

[0039] in represent or . and These represent average pooling (Avg pooling) and max pooling, respectively. This indicates a fully connected layer (FC). express The convolution operation. It is the sigmoid function. Taking deep features as an example, the entire semantic and geometric fusion module is defined as follows:

[0040]

[0041] wherein represents element-wise multiplication.

[0042] In the above formula, and is an intermediate representation of the depth and segmentation features, represents the fused depth feature. The segmentation feature is processed in the same way.

[0043] S200, based on the segmentation mask of the transparent object to be grabbed, the region is judged, the original depth image is repaired by repairing the depth value, and combined with the camera internal parameter, the complete three-dimensional point cloud of the object to be grabbed is output; Specifically, based on the segmentation mask of the transparent object to be grabbed, the region is judged, and the transparent object region and the non-transparent object region are obtained; for the transparent object region, the repaired depth value output by the pre-trained monocular perception network model is used for repair, and for the non-transparent object region, the original depth image is used for output, to obtain the repaired depth map; combined with the repaired depth map and the camera internal parameter, the complete three-dimensional point cloud of the object to be grabbed is output.

[0044] S300, based on the object grabbing detection network model, the complete three-dimensional point cloud of the object to be grabbed is detected, and a plurality of candidate 6-DoF grabbing poses, confidence scores and perturbation tolerances corresponding to each grabbing pose are obtained; Specifically, the complete three-dimensional point cloud of the object to be grabbed is input into the grabbing detection network model; based on the point cloud feature extractor of the grabbing detection network model, the complete three-dimensional point cloud of the object to be grabbed is down-sampled to obtain the original feature dimension; based on the proximity network of the grabbing detection network model, the complete three-dimensional point cloud of the object to be grabbed is predicted to output a plurality of candidate grabbing points and proximity vectors; the plurality of candidate grabbing points, proximity vectors and original feature dimensions are combined to obtain grouped and aligned point cloud clusters; based on the OperationNet network of the grabbing detection network model, the grouped and aligned point cloud clusters are predicted to obtain the in-plane rotation of the gripper, the proximity distance, the gripper width and the grabbing confidence score; based on the ToleranceNet network of the grabbing detection network model, the tolerance of each grabbing pose to perturbation is predicted according to the in-plane rotation of the gripper, the proximity distance, the gripper width and the grabbing confidence score, and a plurality of candidate 6-DoF grabbing poses, confidence scores and perturbation tolerances corresponding to each grabbing pose are output.

[0045] In this embodiment, as Figure 6As shown, a end-to-end network architecture for grasp pose detection (GraspNet-E2E) is demonstrated, which is able to predict robust grasp poses from point cloud input.

[0046] Firstly, the input point cloud is down-sampled by the backbone network of PointNet++ to obtain M points with C-dimensional features. Here, M is the number of points after down-sampling, and C is the feature dimension of each point. Each point contains a three-dimensional coordinate and a C-dimensional feature, represented as M x (3+C).

[0047] The result of the previous step is input into the approach network, which outputs a set of candidate grasp points. Each grasp point corresponds to one or more feasible approach vectors and a confidence score of the approach direction.

[0048] Then, the output (graspable points and approach vectors) from ApproachNet is combined with the original extracted features (M x (3+C)) to generate a unified representation for each grasp candidate. For each grasp candidate point and its predicted approach vector, the algorithm samples points within a cylindrical region around the vector. Then, these sampled points are transformed into a new local coordinate system with the grasp point as the origin and the Z-axis aligned with the approach vector, thereby achieving an "alignment" operation. This process generates K grouped point cloud clusters, each point having 3+C-dimensional features . K represents the number of potential grasp candidates.

[0049] According to the grouped and aligned point cloud clusters, the OperationNet network predicts specific grasp operation parameters. These parameters include in-plane rotation (R) of the gripper, approach distance (D), gripper width (W), and grasp confidence score (S). For K grasp candidates, each candidate predicts 3 operation parameters (rotation, distance, width) for R rotations, represented as K x R x 3. The ToleranceNet network introduces a grasp affinity field, aiming to predict the tolerance of each grasp pose to perturbations, thereby improving grasp robustness. It quantifies the maximum amount of perturbation that a grasp pose can withstand without resulting in a failed grasp. For K grasp candidates, each candidate predicts 1 perturbation tolerance value for R rotations, represented as K x R x 1.

[0050] S400, through collision test, non-maximum suppression is performed on a plurality of candidate 6-DoF grasp poses, and each grasp pose is selected according to the corresponding confidence score and perturbation tolerance, to determine an optimal grasp pose to realize transparent object grasping.

[0051] Specifically, non-maximum suppression is performed on a plurality of candidate 6-DoF grasp poses through a collision test to obtain a filtered grasp pose list; the filtered grasp pose list is layered according to a grasp confidence score, and each layer is sorted according to a perturbation tolerance to obtain an optimal grasp pose; the optimal grasp pose is converted into a motion trajectory of the robotic arm and is used to control the execution of grasping, thereby achieving transparent object grasping.

[0052] In this embodiment, after obtaining all potential grasp poses and their operation parameters and robustness information, a filtering step is performed to select a final grasp, and the filtering method includes pose non-maximum suppression: eliminating highly overlapping or very similar redundant grasp poses in space and retaining the optimal representative in each independent grasp intention; layering and sorting: layering the remaining grasp poses according to a grasp confidence score, and then sorting each layer according to a perturbation tolerance. Select the optimal: select the grasp pose with the highest confidence and the highest tolerance from the sorted list, and the final output is K refined grasp poses, which are considered to be optimal and robust.

[0053] In summary, the embodiment of the present application proposes an end-to-end iterative fusion monocular perception network (IFMP-Net), which can perform transparent object segmentation and depth estimation tasks well only through a single RGB image. The core idea of the framework is to fully utilize the interaction between semantic and geometric information, and to gradually improve the prediction quality through an iterative refinement process, thereby meeting the requirement of improving transparent object perception accuracy. Then, through deep fusion with an end-to-end grasp pose detection network model (GraspNet-E2E), reliable grasp poses of the transparent object are generated, error accumulation is effectively avoided, the synergistic effect of semantic and geometric information is fully utilized, and finally the task of accurate identification and grasping of transparent objects in a cluttered scene is realized.

[0054] As Figure 3As shown, first, the RGB-D camera is used to obtain the RGB image and depth image of the current working scene. Due to the optical properties (refraction, reflection) of the transparent object, the depth map in the transparent object area is invalid. The RGB image is input into the pre-trained IFMP-Net network model, and the segmentation mask and depth map of the transparent object are output. Based on the original depth map and the depth map and segmentation mask generated by the IFMP-Net module. According to the segmentation mask, judge which area in the scene is transparent / high-reflective object. For these areas, use the repaired depth value output by the IFMP-Net; other areas use the reliable data of the original depth camera, and the output is the repaired depth map. Based on the repaired depth map and the camera internal parameter, a complete three-dimensional point cloud is generated. The three-dimensional point cloud is input into the pre-trained GraspNet-E2E model, and a large number of candidate 6-DoF grasp poses, confidence scores and perturbation tolerances of each pose are output. Collision detection is verified, and then non-maximum suppression is performed on all generated grasp poses to remove redundancy, and a more concise and representative grasp pose list is obtained. Finally, the optimal grasp pose is selected according to the grasp quality score and the perturbation tolerance. The selected optimal grasp pose is input into the motion planner (such as MoveIt!, OMPL). The planner will comprehensively consider the current state of the manipulator, joint limits, environmental obstacles (point cloud), etc. to plan a collision-free and smooth motion trajectory from the current pose to the grasp pre-pose. Then the trajectory information is input into the manipulator execution grasp module to execute the grasp.

[0055] Therefore, the embodiments of the present application have the following distinguishing technical feature points compared with the prior art: 1) High-precision transparent object geometry perception is achieved: traditional depth cameras will produce a large amount of noise and holes on the surface of transparent objects, resulting in the inability to obtain effective geometric information. The embodiments of the present application utilize the powerful depth learning capability of the monocular perception network (IFMP-Net), which can output accurate transparent object segmentation mask and repaired high-precision depth map simultaneously only with monocular RGB input. This algorithm can infer the transparent object surface shape that conforms to the physical law through iterative semantic and geometric information fusion, thereby laying a reliable data foundation for subsequent grasp planning. The repaired depth map is almost comparable in quality to that collected on non-transparent objects.

[0056] 2) High-reliability grasp pose is generated: the proposed end-to-end grasp pose detection network (GraspNet-E2E) can directly learn and generate reliable 6-DoF grasp poses from the original point cloud input, which is an advantage over traditional and early deep learning grasp networks. In addition, the grasp affinity field prediction perturbation tolerance is introduced, which further improves the robustness of the grasp, enabling the manipulator to select stable grasps that are not only highly successful but also insensitive to actual execution deviations.

[0057] 3) Constructed end-to-end system-level synergistic advantage: the embodiment of the application is not only an improvement of a single algorithm, but also a seamless connection of IFMP-Net and GraspNet-E2E into an efficient system. This series connection architecture forms a strong synergistic effect: IFMP-Net model "repairs the world" for GraspNet-E2E, and GraspNet-E2E "realizes value" for IFMP-Net. The whole process is integrated from perception to planning, the structure is clear, and the error accumulation problem existing in the traditional step-by-step system is avoided, which significantly improves the overall success rate and robustness of the final grasping task.

[0058] 4) Greatly reduces system cost: the embodiment of the application only relies on a common monocular camera as the core sensor, completely avoiding the dependence on special depth sensors (such as laser radar and high-frequency structured light) which are expensive and susceptible to interference. This greatly reduces the hardware cost and simplifies the complexity of system calibration and maintenance. Promotes the popularization and application landing of transparent object grasping technology.

[0059] Further, the embodiment of the application protects the specific architecture of the Transformer-based visual encoder, the reorganization module and the iterative fusion decoder implemented in the IFMP-Net network. In particular, the specific calculation method of the semantic and geometric fusion module based on double attention (channel attention and spatial attention) in the iterative fusion decoder is protected. Through the symmetric multiplication interaction, the module realizes the full fusion of complementary information between the depth and segmentation tasks, and through the iterative update of the lightweight gating unit to gradually optimize the prediction result. This is the core algorithm innovation to realize high-precision transparent object perception.

[0060] Therefore, the embodiment of the application has the following advantages compared with the prior art: 1) The IFMP-Net and GraspNet-E2E adopted are both carefully designed end-to-end deep learning algorithms. IFMP-Net significantly improves the transparent object perception accuracy through the iterative fusion mechanism, and GraspNet-E2E predicts the grasping robustness through the affinity field mechanism. More importantly, through the specific fusion method, IFMP-Net provides high-quality input for GraspNet-E2E, and GraspNet-E2E provides value for the implementation of IFMP-Net, avoiding error accumulation between independent modules, forming a "1+1>2" synergistic effect.

[0061] 2) The grasp tolerance predicted by GraspNet-E2E provides an unprecedented robustness indicator for grasp selection. At the same time, the generalization ability of the deep learning model itself to unknown objects and scenes is much stronger than traditional geometric methods. The general perception ability of IFMP-Net to transparent objects also ensures that the system has good results for various transparent objects.

[0062] 3) The application innovatively uses a hybrid depth repair strategy while maintaining high precision, making full use of reliable background information in the original depth map, which may reduce the burden of IFMP-Net and improve overall efficiency. The system only needs an RGB-D camera, which is low in cost and high in integration. The generated grasp poses are physically verified and semantically optimized, and are very suitable for real robot application scenarios.

[0063] Reference Figure 2 , the transparent object robot grasping system based on semantic geometry fusion perception comprises: A first module 201 is configured to acquire an RGB image and an original depth image of an object to be grasped, and perform fusion prediction on the RGB image of the object to be grasped based on a pre-trained monocular perception network model, to output a segmentation mask and a repaired depth value of the transparent object to be grasped. A second module 202 is configured to perform region judgment based on the segmentation mask of the transparent object to be grasped, perform repair processing on the original depth image through the repaired depth value, and combine camera intrinsic parameters to output a complete three-dimensional point cloud of the object to be grasped. A third module 203 is configured to perform object grasping detection on the complete three-dimensional point cloud of the object to be grasped based on a grasp detection network model, to obtain a plurality of candidate 6-DoF grasp poses, and a confidence score and a grasp tolerance corresponding to each grasp pose. A fourth module 204 is configured to perform non-maximum suppression on the plurality of candidate 6-DoF grasp poses through collision test, and select an optimal grasp pose by combining the confidence score and the grasp tolerance corresponding to each grasp pose, to realize transparent object grasping.

[0064] Finally, the embodiment of the present application can be summarized as the following module composition, including image acquisition module, monocular perception network module (IFMP-Net), depth repair module, point cloud generation module, grasp detection network module (GraspNet-E2E), grasp planning module and execution module. The image acquisition module (such as RGB-D camera) acquires the RGB image and the original depth map (the depth map is invalid in the transparent object area) of the scene. The monocular perception network model (IFMP-Net) receives the RGB image and outputs the accurate transparent object segmentation mask and depth map. The depth repair module receives the original depth map from the initial camera and the generated depth map and segmentation mask from the IFMP-Net module. According to the segmentation mask, it is judged which area in the scene is transparent / high-reflective object. For these areas, the repaired depth value output by the IFMP-Net is used; for other areas, the reliable data of the original depth camera is used, and the output is the repaired depth map. The point cloud generation module uses the camera internal parameter to back-project the RGB image and the depth map to generate a three-dimensional point cloud containing accurate transparent object geometric information. The grasp prediction module loads the pre-trained GraspNet-E2E end-to-end grasp prediction model, receives the three-dimensional point cloud, and outputs a large number of candidate 6-DoF grasp poses and confidence scores for each pose. The grasp planning module performs collision detection verification on the candidate grasp poses, then performs non-maximum suppression on all generated grasp poses to remove redundancy, obtains a more concise and representative grasp pose list, and filters out the optimal grasp pose according to the confidence score and the perturbation tolerance. The control execution module converts the optimal grasp pose into the motion trajectory of the robot arm and controls it to execute the grasp.

[0065] The contents in the above method embodiments are all applicable to the present system embodiment, the present system embodiment specifically realizes the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0066] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A transparent object mechanical arm grasping method based on semantic geometry fusion perception, characterized in that, The method comprises the following steps: obtaining an RGB image and an original depth image of the object to be grasped, and performing fusion prediction on the RGB image of the object to be grasped based on a pre-trained monocular perception network model to output a segmentation mask and a repaired depth value of the transparent object to be grasped; based on the segmentation mask of the transparent object to be grasped, judging the region, repairing the original depth image through the repaired depth value, and combining the camera intrinsic parameter to output a complete three-dimensional point cloud of the object to be grasped; based on the grasping detection network model, performing object grasping detection on the complete three-dimensional point cloud of the object to be grasped to obtain a plurality of candidate 6-DoF grasping poses, a confidence score corresponding to each grasping pose, and a perturbation tolerance; through collision test, non-maximum suppression is performed on the plurality of candidate 6-DoF grasping poses, and each grasping pose is selected in combination with the confidence score corresponding to each grasping pose and the perturbation tolerance to determine an optimal grasping pose to realize transparent object grasping.

2. The transparent object mechanical arm grasping method based on semantic geometry fusion perception according to claim 1, characterized in that, The step of obtaining an RGB image and an original depth image of the object to be grasped, and performing fusion prediction on the RGB image of the object to be grasped based on a pre-trained monocular perception network model to output a segmentation mask and a repaired depth value of the transparent object to be grasped specifically comprises: obtaining an RGB image and an original depth image of the object to be grasped through a camera; inputting the RGB image of the object to be grasped into a pre-trained monocular perception network model, wherein the pre-trained monocular perception network model comprises a Transformer-based visual encoder, a reorganization module and an iterative fusion decoder; based on the visual encoder of the pre-trained monocular perception network model, performing cutting processing, linear projection and adding position information processing on the RGB image of the object to be grasped in sequence to obtain visual tokens of the object to be grasped; based on the reorganization module of the pre-trained monocular perception network model, converting the visual tokens of the object to be grasped into corresponding feature maps and performing reorganization processing to obtain multi-scale feature maps of the object to be grasped; based on the iterative fusion decoder of the pre-trained monocular perception network model, performing semantic and geometric fusion prediction on the multi-scale feature maps of the object to be grasped to output a segmentation mask and a repaired depth value of the transparent object to be grasped.

3. The method according to claim 2, wherein, The step of based on the iterative fusion decoder of the pre-trained monocular perception network model, performing semantic and geometric fusion prediction on the multi-scale feature maps of the object to be grasped to output a segmentation mask and a repaired depth value of the transparent object to be grasped specifically comprises: inputting the multi-scale feature maps of the object to be grasped into the iterative fusion decoder of the pre-trained monocular perception network model; based on the channel attention module and the spatial attention module of the iterative fusion decoder, extracting information of depth features and segmentation features in the multi-scale feature maps of the object to be grasped to obtain semantic and geometric information; through symmetric multiplication interaction, the semantic and geometric information are fused to output a segmentation mask and a repaired depth value of the transparent object to be grasped.

4. The transparent object mechanical arm grabbing method based on semantic geometry fusion perception according to claim 3, characterized in that, The region judgment is performed based on the segmentation mask of the transparent object to be grabbed, the original depth image is repaired by using the repaired depth value, and the complete three-dimensional point cloud of the object to be grabbed is output by combining the camera intrinsic parameter. The region judgment is performed based on the segmentation mask of the transparent object to be grabbed, and the transparent object region and the non-transparent object region are obtained. The transparent object region is repaired by using the repaired depth value output by the pre-trained monocular perception network model, and the non-transparent object region is output by using the original depth image, so as to obtain the repaired depth map. The complete three-dimensional point cloud of the object to be grabbed is output by combining the repaired depth map and the camera intrinsic parameter.

5. The transparent object mechanical arm grasping method based on semantic geometry fusion perception according to claim 4, characterized in that, The object grabbing detection is performed on the complete three-dimensional point cloud of the object to be grabbed based on the grabbing detection network model, and a plurality of candidate 6-DoF grabbing poses, a confidence score corresponding to each grabbing pose and a perturbation tolerance are obtained. The complete three-dimensional point cloud of the object to be grabbed is input into the grabbing detection network model. The complete three-dimensional point cloud of the object to be grabbed is down-sampled by using the point cloud feature extractor of the grabbing detection network model, and the original feature dimension is obtained. The complete three-dimensional point cloud of the object to be grabbed is grabbed by using the proximity network of the grabbing detection network model, and a plurality of candidate grabbing points and proximity vectors are output. The plurality of candidate grabbing points, the proximity vectors and the original feature dimension are combined to obtain the grouped and aligned point cloud cluster. The grouped and aligned point cloud cluster is predicted by using the OperationNet network of the grabbing detection network model, and the in-plane rotation of the gripper, the proximity distance, the gripper width and the grabbing confidence score are obtained. The tolerance of each grabbing pose to perturbation is predicted based on the in-plane rotation of the gripper, the proximity distance, the gripper width and the grabbing confidence score by using the ToleranceNet network of the grabbing detection network model, and a plurality of candidate 6-DoF grabbing poses, a confidence score corresponding to each grabbing pose and a perturbation tolerance are output.

6. The transparent object mechanical arm grasping method based on semantic geometry fusion perception according to claim 5, characterized in that, The plurality of candidate grabbing points, the proximity vectors and the original feature dimension are combined to obtain the grouped and aligned point cloud cluster. A plurality of sampling points are obtained by sampling points in the cylindrical region around the vector. The plurality of sampling points are converted into a new local coordinate system, the origin of the coordinate system is the grabbing point, the Z-axis is aligned with the proximity vector, and the original feature dimension is combined for grouping and alignment to obtain the grouped and aligned point cloud cluster.

7. The method according to claim 6, wherein, The plurality of candidate 6-DoF grabbing poses are subjected to non-maximum suppression by using the collision test, and the optimal grabbing pose is determined by combining the confidence score corresponding to each grabbing pose and the perturbation tolerance to realize transparent object grabbing. ​ ​ The optimal grasping pose is converted into a motion trajectory of the robot arm, and the robot arm is controlled to perform grasping, so that the transparent object is grasped.

8. A transparent object robot grasping system based on semantic geometry fusion perception, characterized in that, The method comprises the following modules: A first module is configured to acquire an RGB image and an original depth image of an object to be grasped, and perform fusion prediction on the RGB image of the object to be grasped based on a pre-trained monocular perception network model, and output a segmentation mask and a repaired depth value of the transparent object to be grasped. A second module is configured to perform region judgment based on the segmentation mask of the transparent object to be grasped, perform repair processing on the original depth image by using the repaired depth value, and combine a camera intrinsic parameter to output a complete three-dimensional point cloud of the object to be grasped. A third module is configured to perform object grasping detection on the complete three-dimensional point cloud of the object to be grasped based on a grasping detection network model, and obtain a plurality of candidate 6-DoF grasping poses, a confidence score corresponding to each grasping pose, and a perturbation tolerance. A fourth module is configured to perform non-maximum suppression on the plurality of candidate 6-DoF grasping poses by using a collision test, and select the grasping poses in combination with the confidence score corresponding to each grasping pose and the perturbation tolerance, so as to determine an optimal grasping pose and realize grasping of the transparent object.

Images