Robot six-degree-of-freedom grasp detection method and apparatus

Abstract

This invention provides a six-degree-of-freedom (DOF) grasping detection method and apparatus for robots, relating to the field of robot operation technology. The method includes: acquiring a color-depth image of the current scene, inputting it into a grasping detection network, and obtaining the six-DOF grasping detection result output by the grasping detection network; performing feature encoding on the color-depth image to obtain an image feature map, predicting Gaussian kernel parameters, and generating a continuous grasping quality heatmap and grasping attribute map; processing the grasping quality heatmap to obtain a spatial pose feature tensor, and combining it with the grasping attribute map to generate the six-DOF grasping detection result. The method and apparatus provided by this invention generate a continuous grasping quality heatmap and grasping attribute map by performing feature encoding on the color-depth image and Gaussian density rendering, processing the grasping quality heatmap to obtain a spatial pose feature tensor, and then generating a six-DOF grasping detection result. This method can generate a grasping quality heatmap with continuity and integrity, improving the accuracy of grasping detection.

Description

Technical Field

[0001] This invention relates to the field of robot operation technology, and in particular to a robot six-degree-of-freedom grasping and detection method and device. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence and robotics, performing hand-eye coordination operations such as grasping and assembly has become an important prerequisite for robots to carry out autonomous operations. Because real-world physical scenarios are often complex and varied, and the shapes and arrangements of target objects differ, traditional two-dimensional planar grasping and detection methods are insufficient to meet the diverse operational needs of complex three-dimensional scenarios. Therefore, how to accurately calculate the spatial position and orientation of the object to be grasped by the robot actuator directly in three-dimensional space based on sensor observation data under a given environment—that is, accurate six-degrees-of-freedom (6-DoF) grasping and detection—has become a core overall requirement urgently needing to be addressed in the field of robot manipulation technology.

[0003] To meet the aforementioned high-precision grasping and detection requirements, existing technologies are gradually shifting towards cross-modal grasping and detection schemes that combine high-level semantic cues from 2D images with precise spatial geometry from 3D point clouds. While this approach narrows the search range in 3D space, it has significant limitations in practical applications. For example, the generated grasping quality heatmap suffers from blurred boundaries and discontinuities. Furthermore, it fails to fully utilize the complementary information between different modal features when fusing 2D image features and 3D point cloud features, ultimately resulting in the accuracy and robustness of robot 6-DoF grasping and detection failing to meet the requirements of practical industrial applications.

[0004] Therefore, how to improve the accuracy of 6-DoF grasping and detection by robots has become a technical problem that the industry urgently needs to solve. Summary of the Invention

[0005] This invention provides a robot six-degree-of-freedom grasping detection method and apparatus to solve the defects of blurred and discontinuous boundaries of grasping quality heatmaps and insufficient cross-modal feature fusion in the prior art, thereby improving the accuracy of robot 6-DoF grasping detection.

[0006] This invention provides a six-degree-of-freedom grasping and detection method for robots, comprising: Obtain a color-depth image of the current scene; The color-depth image is input into the grasping detection network to obtain the six-degree-of-freedom grasping detection result output by the grasping detection network; The grasping and detection network includes a Gaussian heatmap generation module and a cross-modal grasping and detection module; The Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, and predicts the Gaussian kernel parameters corresponding to each anchor point position in the image feature map. It then performs a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous crawling quality heatmap and crawling attribute map. The cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the module generates the six-degree-of-freedom grasping detection result.

[0007] In some embodiments, the cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the module generates the six-degree-of-freedom grasping detection result, including: Based on the response values ​​of each pixel in the captured quality heatmap, a preset number of pixels are selected, and a two-dimensional capture candidate coordinate set is constructed. By combining depth information, the pixels in the set of two-dimensional grabbing candidate coordinates are projected onto three-dimensional space to obtain the coordinate tensor of the three-dimensional grabbing center candidate point. The color-depth image is back-projected to obtain a three-dimensional point cloud; Using each 3D grabbing center candidate point as the query center, a ball query operation is performed in the 3D point cloud to obtain the local point cloud corresponding to each 3D grabbing center candidate point. Feature extraction is performed on the local point cloud to obtain the regional point cloud features. The local point cloud is reprojected onto the two-dimensional image plane where the image feature map is located, and the region image features corresponding to the local point cloud are retrieved from the image feature map by interpolation sampling. The regional point cloud features and the regional image features are concatenated to form a fusion feature, and cross-modal attention calculation is performed on the fusion feature to obtain the spatial pose feature tensor of the fusion image and the deep structure information of the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the six-degree-of-freedom grasping detection result is generated.

[0008] In some embodiments, the Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, including: The color-depth image is initially downsampled using an initial convolutional layer and a max-pooling layer. The processed image is input into multiple sequentially connected residual feature blocks for feature extraction, resulting in an image feature map with reduced size and expanded channel count.

[0009] In some embodiments, predicting the Gaussian kernel parameters corresponding to each anchor point position in the image feature map involves performing a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous capture quality heatmap and capture attribute map, including: Parallel convolution processing is performed on the image feature maps to predict the actual offset map, horizontal standard deviation map, vertical standard deviation map, correlation coefficient map, and magnitude weight map, which serve as the Gaussian kernel parameters corresponding to each anchor point position. Based on the horizontal standard deviation map and the vertical standard deviation map, the corresponding local influence radius is calculated and generated; In the constructed two-dimensional grid coordinate system, the Gaussian kernel parameters of the anchor points in the image feature map covered by the local influence radius are retrieved at each grid pixel coordinate, and a two-dimensional Gaussian kernel weighted superposition operation is performed to obtain the superimposed feature tensor. The superimposed feature tensor is segmented along the feature channel dimension to obtain the crawling quality heatmap and the crawling attribute map, respectively.

[0010] In some embodiments, the step of projecting pixels from the two-dimensional grabbing candidate coordinate set onto three-dimensional space using depth information to obtain a three-dimensional grabbing center candidate point coordinate tensor includes: The two-dimensional grid coordinates of each pixel in the two-dimensional crawling candidate coordinate set on the crawling quality heatmap are mapped to the resolution-aligned original depth map space to obtain the corresponding depth value. By combining the camera intrinsic parameter matrix, the two-dimensional grid coordinates and the depth values ​​are converted into three-dimensional coordinates to obtain the coordinate tensor of the three-dimensional grabbing center candidate point.

[0011] In some embodiments, the step of using each 3D grabbing center candidate point as the query center, performing a sphere query operation in the 3D point cloud to obtain the local point cloud corresponding to each 3D grabbing center candidate point, and extracting features from the local point cloud to obtain region point cloud features includes: Using a preset query radius, a preset number of nearest neighbor points closest to each of the candidate points of the three-dimensional grasping center are obtained in the three-dimensional point cloud, forming a local point cloud cluster tensor as the local point cloud; The local point cloud cluster tensor is input into a small point cloud processing network containing a first one-dimensional convolutional layer and a second one-dimensional convolutional layer connected in sequence for processing to extract the regional point cloud features.

[0012] In some embodiments, performing cross-modal attention computation on the fused features to obtain the spatial pose feature tensor of the fused image and the deep structure information of the point cloud includes: The fused features are input into three fully connected layers in parallel to generate query tensors, key tensors, and value tensors, respectively. The query tensor and the transpose of the key tensor are subjected to batch matrix multiplication, and after processing with a scaling factor, a Softmax operation is performed to obtain the cross-modal attention weight matrix. The cross-modal attention weight matrix is ​​multiplied with the value tensor to obtain context-weighted features; The context-weighted features are reduced in dimensionality by a feature mapping layer, and max pooling is performed along the point dimension to remove the spatial dimension, thus obtaining the spatial pose feature tensor.

[0013] In some embodiments, the grabbing detection network is trained based on the following steps: Obtain training samples and their corresponding heatmap labels representing the actual crawling quality. The horizontal and vertical edge extraction operators are used to perform convolution calculations on the real crawled quality heatmap labels to extract first-order gradient change features and calculate the synthetic gradient magnitude. The synthesized gradient magnitude is normalized to generate a gradient magnitude attention mask; The error between the predicted crawling quality heatmap and the actual crawling quality heatmap label is weighted and calculated based on the gradient magnitude attention mask to obtain the heatmap quality loss. A total loss function is constructed based on the heatmap quality loss, crawling attribute loss, and crawling detection loss, and the parameters of the crawling detection network are updated based on the total loss function.

[0014] In some embodiments, the normalization of the synthesized gradient magnitude to generate a gradient magnitude attention mask includes: Obtain the global minimum and global maximum values ​​of the synthesized gradient magnitudes; Calculate the difference between the synthesized gradient magnitude and the global minimum, and divide it by the difference between the global maximum and the global minimum to obtain the normalization ratio. The normalized ratio is weighted based on a preset gradient weighting coefficient, and a basic weight constant is added to generate the gradient magnitude attention mask.

[0015] This invention provides a robot six-degree-of-freedom grasping and detection device, comprising: The image acquisition module is used to acquire a color-depth image of the current scene; The model inference module is used to input the color-depth image into the grasping detection network and obtain the six-degree-of-freedom grasping detection result output by the grasping detection network. The grasping and detection network includes a Gaussian heatmap generation module and a cross-modal grasping and detection module; The Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, and predicts the Gaussian kernel parameters corresponding to each anchor point position in the image feature map. It then performs a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous crawling quality heatmap and crawling attribute map. The cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the module generates the six-degree-of-freedom grasping detection result.

[0016] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the robot six-degree-of-freedom grasping and detection method.

[0017] The present invention provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the robot six-degree-of-freedom grasping and detection method.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot six-degree-of-freedom grasping and detection method.

[0019] The robot six-degree-of-freedom grasping detection method and apparatus provided by this invention encodes features of color-depth images and predicts Gaussian kernel parameters through a Gaussian heatmap generation module. Then, it performs two-dimensional Gaussian kernel weighted superposition operation through Gaussian density rendering to generate continuous grasping quality heatmaps and grasping attribute maps. Simultaneously, a cross-modal grasping detection module performs three-dimensional candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmaps to obtain spatial posture feature tensors. Based on the grasping attribute map and spatial posture feature tensors, a six-degree-of-freedom grasping detection result is generated. This method can generate grasping quality heatmaps with continuity and integrity and achieve deep fusion of image features and point cloud features, thereby improving the accuracy of grasping detection. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the six-degree-of-freedom grasping and detection method for robots provided by this invention.

[0023] Figure 2 This is a schematic diagram of the structure of the robot six-degree-of-freedom grasping and detection device provided by the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps, units, or modules is not necessarily limited to those explicitly listed, but may include other steps, units, or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0027] To facilitate a full understanding of the technical solution of this application, the following content is hereby introduced: Since traditional 2D grasping detection is difficult to adapt to the diverse operational needs in complex 3D scenes, 6-DoF grasping detection methods that directly predict the robot gripper pose in 3D space are gradually becoming mainstream. Among them, cross-modal robot 6-DoF grasping detection methods that combine high-level semantic cues from 2D images with precise spatial geometry from 3D point clouds are beginning to attract attention. It takes scene color-depth (Red, Green, Blue-D) data acquired by the robot's vision system as input, firstly uses a neural network to predict a grasping quality heatmap on the 2D image plane as a visual prior cue; then, given high-response regions in the grasping quality heatmap, it inversely projects them into 3D space to generate grasping candidate points, and crops the corresponding local neighborhood point clouds from the 3D point cloud observation; finally, it fuses the local point cloud features with the retrieved 2D image features to regress the complete 3D grasping pose.

[0028] While this type of method narrows the search range in three-dimensional space, it has many problems. For example, since the two-dimensional capture quality heatmap is mostly generated by simple pixel-by-pixel discrete regression, it is prone to problems such as blurred boundaries, discontinuity or local incompleteness, which affects the rationality of the capture distribution. In addition, the image and point cloud feature fusion method is relatively rudimentary, usually relying only on channel stitching, and failing to fully explore the inherent dependency between the high-level semantics of the image and the local spatial geometry of the point cloud.

[0029] To address the issues of poor grasping quality heatmap quality and insufficient cross-modal feature fusion in existing cross-modal robot 6-DoF grasping detection methods, which reduce grasping detection accuracy, this invention proposes a robot six-DOF grasping detection method. By generating Gaussian heatmaps and performing cross-modal grasping detection, the accuracy of robot 6-DoF grasping detection is effectively improved.

[0030] Figure 1 This is a flowchart illustrating the six-degree-of-freedom grasping and detection method for robots provided by this invention, as shown below. Figure 1 As shown, the method includes steps 110 and 120.

[0031] Step 110: Obtain the color-depth image of the current scene.

[0032] Specifically, the robot six-degree-of-freedom grasping detection method provided in this embodiment of the invention is executed by a robot six-degree-of-freedom grasping detection device. This device can be implemented in software, such as a robot six-degree-of-freedom grasping detection program running in a computer; or it can be implemented in hardware, such as a computer or server that executes the robot six-degree-of-freedom grasping detection method.

[0033] A color-depth image is image data that simultaneously contains both color and depth information of a scene. Color information records the color and texture features of the surfaces of objects in the scene, while depth information records the distance from the surfaces of objects to the camera. Color-depth images can be acquired using 3D vision sensors, such as depth cameras. A depth camera can simultaneously output a color image and an aligned depth image; each pixel value in the depth image represents the distance between the corresponding spatial point and the camera.

[0034] When performing grasping and detection, the first step is to acquire a color-depth image of the current scene. This is done using a 3D vision sensor mounted on the robot to obtain an RGB-D image of the current scene.

[0035] In this embodiment of the invention, the robot acquires RGB-D images using a depth camera (preferably Kinect V2), which contains 3D RGB color channels and 1D depth channels.

[0036] Step 120: Input the color-depth image into the grasping detection network and obtain the six-degree-of-freedom grasping detection result output by the grasping detection network; The grasping and detection network includes a Gaussian heatmap generation module and a cross-modal grasping and detection module; The Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, and predicts the Gaussian kernel parameters corresponding to each anchor point position in the image feature map. It then performs a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous crawling quality heatmap and crawling attribute map. The cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the module generates the six-degree-of-freedom grasping detection result.

[0037] Specifically, the six-degree-of-freedom grasping detection results include the three-dimensional coordinates of the grasping position and the three rotation angles of the grasping posture, for a total of six degrees of freedom, which are used to guide the robot gripper to grasp the target object at the appropriate angle and position.

[0038] The grasping detection network is a pre-trained deep learning network capable of extracting features from an input color-depth image and outputting six-DOF grasping detection results for a robot gripper. In this embodiment of the invention, the grasping detection network is a pre-constructed robot 6-DoF grasping detection network, including a Gaussian heatmap generation module and a cross-modal grasping detection module. Based on pre-constructed training samples and a pre-designed loss function, the grasping detection network is trained to obtain the parameters of each module, resulting in the final grasping detection network for robot 6-DoF grasping detection.

[0039] The RGB-D image is input into the trained robot 6-DoF grasping detection network, and the robot 6-DoF grasping detection result is output.

[0040] Feature encoding refers to the process of converting a raw color-depth image into a smaller image feature map with a larger number of channels through a series of convolution and pooling operations.

[0041] Each spatial location in the image feature map corresponds to a local region in the original image, and each channel encodes different types of visual information.

[0042] The RGB-D image is input into the Gaussian heatmap generation module of the trained robot 6-DoF grasping detection network to extract image feature maps, grasping quality heatmaps, and three grasping attribute maps. The Gaussian heatmap generation module first performs feature encoding on the color-depth image. After obtaining the image feature map, the Gaussian heatmap generation module sets an anchor point at the center of each spatial grid in the image feature map and predicts a set of Gaussian kernel parameters for each anchor point. The Gaussian kernel parameters describe the shape, size, and orientation of the two-dimensional Gaussian distribution centered at that anchor point. The Gaussian kernel parameters include center offset, horizontal standard deviation, vertical standard deviation, correlation coefficient, and magnitude weight. The center offset is used to correct the positional deviation of the anchor point; the horizontal and vertical standard deviations determine the diffusion range of the Gaussian distribution in the horizontal and vertical directions; the correlation coefficient describes the correlation of the Gaussian distribution in the horizontal and vertical directions; and the magnitude weight determines the contribution of the Gaussian kernel during superposition.

[0043] After obtaining the Gaussian kernel parameters for all anchor points, the Gaussian heatmap generation module uses Gaussian density rendering to weight and superimpose the two-dimensional Gaussian kernels corresponding to all anchor points within their local influence range according to their respective amplitude weights, generating a continuous grabbing quality heatmap and several grabbing attribute maps. The grabbing quality heatmap reflects the probability distribution of each location in the scene being suitable for grabbing. The grabbing attribute maps provide auxiliary information such as grabbing angle, grabbing width, and grabbing depth.

[0044] Unlike traditional pixel-by-pixel discrete regression, this method, which uses Gaussian kernel parameter prediction and density rendering, generates high-quality heatmaps with better continuity and integrity.

[0045] The RGB-D image, image feature map, grasping quality heatmap, and three grasping attribute maps are input into the cross-modal grasping detection module of the trained robot 6-DoF grasping detection network for processing, outputting the robot 6-DoF grasping detection results. The cross-modal grasping detection module is responsible for deep fusion of 2D image information and 3D point cloud information. This module first reads the grasping quality heatmap output by the Gaussian heatmap generation module, selects pixels with high response values ​​as candidate points, and then projects these candidate points into 3D space to obtain a series of 3D grasping candidate points. Next, with each 3D grasping candidate point as the center, it searches for surrounding local point cloud regions in the 3D point cloud of the scene and extracts point cloud features from these local point clouds. Simultaneously, this module reprojects the local point clouds onto the image feature map and obtains the image features corresponding to the local point clouds through interpolation sampling. Finally, the point cloud features and image features are subjected to cross-modal attention fusion processing to obtain a spatial pose feature tensor that integrates image information and point cloud information. Based on the grasping attribute map and the spatial pose feature tensor, a six-degree-of-freedom grasping detection result is finally generated.

[0046] The robot six-DOF grasping detection method provided in this invention uses a Gaussian heatmap generation module to encode features in a color-depth image and predict Gaussian kernel parameters. Then, it uses Gaussian density rendering to perform a two-dimensional Gaussian kernel weighted superposition operation to generate a continuous grasping quality heatmap and a grasping attribute map. Simultaneously, a cross-modal grasping detection module performs three-dimensional candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor. Based on the grasping attribute map and the spatial pose feature tensor, a six-DOF grasping detection result is generated. This method can generate a grasping quality heatmap with continuity and integrity, and achieve deep fusion of image features and point cloud features, thereby improving the accuracy of grasping detection.

[0047] In some embodiments, the cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the module generates the six-degree-of-freedom grasping detection result, including: Based on the response values ​​of each pixel in the captured quality heatmap, a preset number of pixels are selected, and a two-dimensional capture candidate coordinate set is constructed. By combining depth information, the pixels in the set of two-dimensional grabbing candidate coordinates are projected onto three-dimensional space to obtain the coordinate tensor of the three-dimensional grabbing center candidate point. The color-depth image is back-projected to obtain a three-dimensional point cloud; Using each 3D grabbing center candidate point as the query center, a ball query operation is performed in the 3D point cloud to obtain the local point cloud corresponding to each 3D grabbing center candidate point. Feature extraction is performed on the local point cloud to obtain the regional point cloud features. The local point cloud is reprojected onto the two-dimensional image plane where the image feature map is located, and the region image features corresponding to the local point cloud are retrieved from the image feature map by interpolation sampling. The regional point cloud features and the regional image features are concatenated to form a fusion feature, and cross-modal attention calculation is performed on the fusion feature to obtain the spatial pose feature tensor of the fusion image and the deep structure information of the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the six-degree-of-freedom grasping detection result is generated.

[0048] Specifically, in this embodiment of the invention, the cross-modal grasping detection module is used to fuse image and point cloud features to generate high-quality grasping detection results, including a 2D-3D projector, a local point cloud feature extractor, a feature retrieval and stitching unit, a cross-modal attention unit, and a multi-grasping generator.

[0049] 2D-3D projector receiving and capturing quality heatmap First, select and capture quality heatmaps element by element. The highest response value in the middle In this embodiment of the invention, preferably, each pixel is used. And record this Each sampling point captures a quality heatmap. The two-dimensional grid coordinates on the grid form a set of two-dimensional crawling candidate coordinates. A higher response value indicates that the location is more likely to contain a suitable area for crawling.

[0050] By combining depth information, each pixel in this two-dimensional coordinate set is projected into three-dimensional space to obtain the three-dimensional spatial coordinates corresponding to each pixel. These three-dimensional coordinates constitute the coordinate tensor of the three-dimensional grabbing center candidate point. The three-dimensional grabbing center candidate point is the basis for subsequent processing and represents the potential grabbing position.

[0051] When extracting local point cloud features, the cross-modal grasping and detection module first back-projects the color-depth image to obtain the 3D point cloud of the scene. The 3D point cloud is a collection of numerous 3D spatial points, each containing 3D coordinate information and reflecting the geometry of object surfaces in the scene. A sphere query operation is performed on each 3D grasping center candidate point within the 3D point cloud. A sphere query involves drawing a spherical region with a radius equal to a given point in 3D space and filtering out all points within that region. Through the sphere query operation, the point cloud within the local region surrounding each 3D grasping center candidate point is obtained, resulting in a local point cloud cluster tensor. This refers to a local point cloud; this local point cloud contains geometric details of the surface of objects near the candidate point. Feature extraction is performed on each local point cloud to obtain a feature vector that describes the geometric features of that local region, i.e., the region point cloud feature. .

[0052] Next, image feature retrieval is performed. In this embodiment of the invention, the feature retrieval and stitching unit converts the local point cloud cluster tensor... (size is) The 3D coordinates of all points are obtained using the camera intrinsic parameter matrix. Reprojecting the image onto a 2D image plane yields a normalized pixel coordinate tensor. Using this normalized pixel coordinate tensor, the `torch.nn.functional.grid_sample` function in PyTorch can be called to sample pixels of size [size missing]. Image feature map Bilinear interpolation is used for sampling to obtain regional image features. Its size is Complete the regional point cloud features Corresponding regional image features Feature retrieval was performed. Subsequently, the `torch.cat` function in PyTorch was used to extract the region point cloud features. and regional image features By stitching along the last channel dimension, the fused features are obtained. Its size is .

[0053] Cross-modal attention computation is performed on the fused feature to obtain the spatial pose feature tensor of the fused image and the deep structural information of the point cloud.

[0054] Next, crawl detection results are generated based on the multi-crawler generator. The multi-crawler generator receives three crawl attribute graphs. Spatial attitude feature tensor The final 6-DoF grasping detection result of the robot is generated. This multi-grasp generator multiplies the spatial pose feature tensor element-wise with the three grasping attribute maps to obtain three weighted features of grasping attributes. The three weighted features of grasping attributes are concatenated and input into a fully connected layer to regress the three-dimensional coordinates of the grasping position and the three rotation angles of the grasping posture, which are used as the six-DOF grasping detection result. The grasping attribute map provides information such as grasping angle, grasping width, and grasping depth, while the spatial pose feature tensor provides comprehensive features of grasping position and posture. The combination of the two can output a complete six-DOF grasping pose.

[0055] The robot six-degree-of-freedom grasping detection method provided in this invention can achieve accurate mapping from two-dimensional heatmaps to three-dimensional candidate points and stepwise fusion of image features and point cloud features by selecting candidate points based on grasping quality heatmaps, projecting them into three-dimensional space, obtaining local point clouds through ball query, extracting point cloud features, retrieving image features, stitching and fusing, and performing cross-modal attention calculation. This further improves the effect of cross-modal feature fusion.

[0056] In some embodiments, the Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, including: The color-depth image is initially downsampled using an initial convolutional layer and a max-pooling layer. The processed image is input into multiple sequentially connected residual feature blocks for feature extraction, resulting in an image feature map with reduced size and expanded channel count.

[0057] Specifically, during the initial downsampling process, the Gaussian heatmap generation module first uses an initial convolutional layer to convolve the color-depth image. Convolution extracts local features from the image, such as edges, textures, and color variations. Then, a max-pooling layer downsamples the convolutional result. Max-pooling is a downsampling operation that divides the image into several small regions and selects the maximum value within each region as its representative value, thereby reducing the image size. Through the processing of the initial convolutional layer and the max-pooling layer, the size of the color-depth image is reduced while retaining key feature information useful for subsequent processing.

[0058] The image obtained after initial downsampling is input into multiple sequentially connected residual feature blocks. Residual feature blocks are a commonly used deep learning network structure, with each residual feature block containing multiple convolutional layers and a skip connection. These residual feature blocks are sequentially connected, with each block extracting more abstract and higher-level features based on the previous one. After processing by multiple residual feature blocks, the image size is further reduced, while the number of feature channels gradually increases. The final result is a smaller image feature map with a larger number of channels, which serves as the input for subsequent Gaussian kernel parameter prediction.

[0059] In this embodiment of the invention, the Gaussian heatmap generation module is used to establish a Gaussian mixture model in the image to mine the crawling quality distribution, including an image feature encoder, a Gaussian kernel regressor, and a Gaussian density renderer. The image feature encoder outputs an image feature map, while the Gaussian kernel regressor and Gaussian density renderer output a crawling quality heatmap and three crawling attribute maps.

[0060] In this embodiment of the invention, the image feature encoder consists of an initial convolutional layer Conv0, a max-pooling layer Max_Pool_0, and four sequentially connected residual feature blocks Layer1, Layer2, Layer3, and Layer4. The initial convolutional layer Conv0 uses a 7×7×4×64 convolutional kernel with parameters in the format of height×width×input channel×output channel; the stride is (2, 2); the padding size is (3, 3), using zero padding, and has learnable bias parameters. The max-pooling layer Max_Pool_0 is implemented using the nn.MaxPool2d function from the torch library, with a 3×3 kernel size, a stride of (2, 2), and a padding size of (1, 1). The four residual feature blocks Layer1, Layer2, Layer3, and Layer4 adopt four stages composed of residual feature blocks in the standard ResNet-50 backbone network structure. For a size of... Input RGB-D image (height × width × channels, including 3D RGB color channels and 1D depth channel) After being processed sequentially through Conv0, Max_Pool_0, Layer1, Layer2, Layer3, and Layer4, the image feature map is obtained. Its size is reduced to 1 / 8 of the input image, and the number of feature channels is expanded to 256, i.e., the image feature map. The dimensions are H / 8×W / 8×256.

[0061] The robot six-degree-of-freedom grasping and detection method provided in this invention can effectively downsample and extract features from color-depth images through an initial convolutional layer, a max pooling layer, and multiple sequentially connected residual feature blocks, thereby providing high-quality image feature maps for subsequent Gaussian kernel parameter prediction.

[0062] In some embodiments, predicting the Gaussian kernel parameters corresponding to each anchor point position in the image feature map involves performing a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous capture quality heatmap and capture attribute map, including: Parallel convolution processing is performed on the image feature maps to predict the actual offset map, horizontal standard deviation map, vertical standard deviation map, correlation coefficient map, and magnitude weight map, which serve as the Gaussian kernel parameters corresponding to each anchor point position. Based on the horizontal standard deviation map and the vertical standard deviation map, the corresponding local influence radius is calculated and generated; In the constructed two-dimensional grid coordinate system, the Gaussian kernel parameters of the anchor points in the image feature map covered by the local influence radius are retrieved at each grid pixel coordinate, and a two-dimensional Gaussian kernel weighted superposition operation is performed to obtain the superimposed feature tensor. The superimposed feature tensor is segmented along the feature channel dimension to obtain the crawling quality heatmap and the crawling attribute map, respectively.

[0063] Specifically, after obtaining the image feature map, the Gaussian heatmap generation module predicts the Gaussian kernel parameters corresponding to each anchor point position based on the image feature map, and generates continuous crawling quality heatmaps and crawling attribute maps through Gaussian density rendering.

[0064] In this embodiment of the invention, the Gaussian kernel regressor comprises four parallel convolutional layers: Conv_mu, Conv_sigma, Conv_rho, and Conv_A. Each convolutional layer has a stride of (1, 1), a padding size of (1, 1), zero padding, and a learnable bias. All layers are based on image feature maps. For input.

[0065] Among them, Conv_mu uses a size of high convolution kernel (high) Width Input Channel Output channel), output center offset feature map, size is The center offset feature map is activated by the hyperbolic tangent function torch.nn.Tanh in PyTorch and multiplied by the maximum offset constant. (Preferred in this embodiment of the invention is 4.0) to obtain the actual offset map .

[0066] Conv_sigma adopts The convolution kernel outputs a standard deviation feature map, which is then processed by the `torch.nn.Softplus` function in PyTorch and a minimal constant is added. (Preferred in the embodiments of the present invention) ), to obtain the horizontal and vertical standard deviation plots of the two-dimensional Gaussian kernel. , The size is .

[0067] Conv_rho adopts The convolution kernel outputs a correlation feature map, which is then activated by the hyperbolic tangent function `torch.nn.Tanh` in PyTorch to obtain the correlation coefficient map. The size is .

[0068] Conv_A adopts The convolution kernel outputs an amplitude feature map, which is then activated by `torch.nn.Sigmoid` in PyTorch to obtain an amplitude weight map that includes grasping quality and three grasping attributes. The size is .

[0069] Gaussian kernel regressor received image feature map Generate actual offset map Horizontal and vertical standard deviation charts , Correlation coefficient plot Amplitude weighting diagram and in image feature map An anchor point is generated at the center of each spatial grid, resulting in a total of [number] anchor points. Anchor points.

[0070] The Gaussian density renderer is based on the actual offset map of the above output. Horizontal and vertical standard deviation charts , Correlation coefficient plot Amplitude weighting diagram The specific steps for reconstructing a continuous distribution heatmap are as follows: First, based on the input level standard deviation chart Vertical standard deviation chart Calculate and generate the corresponding local influence radius pixel by pixel. Its calculation formula is ; Subsequently, a two-dimensional grid coordinate system is established in space, with dimensions of... ; for each pixel coordinate in the grid coordinate system At this location, retrieve the local influence radius. Covered image feature map The Gaussian kernel parameters corresponding to the middle anchor point correspond to the actual offset map. Horizontal and vertical standard deviation charts , Correlation coefficient plot Amplitude weighting diagram pixel coordinates The value at the specified location is calculated, and a two-dimensional Gaussian kernel weighted superposition operation is performed. The two-dimensional Gaussian kernel weighted superposition operation is as follows: In the two-dimensional grid coordinate system, for each grid pixel coordinate, traverse each anchor point covered by its local influence radius, calculate the Gaussian response value of the two-dimensional Gaussian kernel at that grid pixel coordinate corresponding to the anchor point, and multiply all Gaussian response values ​​by the corresponding magnitude weights and sum them to obtain the superposition value at that grid pixel coordinate; after traversing all grid pixel coordinates, the superposition feature tensor is obtained. The calculated superposition feature tensor has a size of [missing information]. Then, the data is split along the channel dimension of the superimposed feature tensor. This can be done using the feature channel-level splitting function `torch.split` in PyTorch. After splitting, the capture quality heatmaps are obtained sequentially. and three grab attribute graphs The dimensions of these four are all .

[0071] The robot six-degree-of-freedom grasping detection method provided in this invention generates a grasping quality heatmap and grasping attribute map with continuity and integrity by predicting the actual offset map, horizontal standard deviation map, vertical standard deviation map, correlation coefficient map and amplitude weight map, and calculating the local influence radius based on the standard deviation and then performing a two-dimensional Gaussian kernel weighted superposition operation. This solves the problem of blurred and discontinuous heatmap boundaries caused by traditional discrete regression methods.

[0072] In some embodiments, the step of projecting pixels from the two-dimensional grabbing candidate coordinate set onto three-dimensional space using depth information to obtain a three-dimensional grabbing center candidate point coordinate tensor includes: The two-dimensional grid coordinates of each pixel in the two-dimensional crawling candidate coordinate set on the crawling quality heatmap are mapped to the resolution-aligned original depth map space to obtain the corresponding depth value. By combining the camera intrinsic parameter matrix, the two-dimensional grid coordinates and the depth values ​​are converted into three-dimensional coordinates to obtain the coordinate tensor of the three-dimensional grabbing center candidate point.

[0073] Specifically, in this embodiment of the invention, a set of candidate coordinates for two-dimensional grasping is obtained. , convert two-dimensional grid coordinates Mapped to a resolution of In the original aligned depth map space, query to obtain the depth value of the corresponding pixel position. Then, combined with the camera intrinsic parameter matrix Following the inverse transformation of the pinhole camera projection model, the two-dimensional grid coordinates and corresponding depth values ​​are converted into three-dimensional coordinates in the camera coordinate system, and the dimensions are calculated. 3D grabbing center candidate point coordinate tensor The inverse transformation is to multiply the two-dimensional grid coordinates by the depth value and then multiply the inverse of the camera intrinsic parameter matrix by the left.

[0074] The robot six-degree-of-freedom grasping and detection method provided in this invention obtains depth values ​​by mapping the two-dimensional grid coordinates on the heat map to the original depth map space, and converts them into three-dimensional coordinates by combining the camera intrinsic parameter matrix. This enables accurate mapping from low-resolution heat maps to high-precision three-dimensional candidate points, thereby ensuring the positioning accuracy of the three-dimensional candidate points.

[0075] In some embodiments, the step of using each 3D grasping center candidate point as the query center, performing a sphere query operation in the 3D point cloud to obtain the local point cloud corresponding to each 3D grasping center candidate point, and extracting features from the local point cloud to obtain region point cloud features includes: Using a preset query radius, a preset number of nearest neighbor points closest to each of the candidate points of the three-dimensional grasping center are obtained in the three-dimensional point cloud, forming a local point cloud cluster tensor as the local point cloud; The local point cloud cluster tensor is input into a small point cloud processing network containing a first one-dimensional convolutional layer and a second one-dimensional convolutional layer connected in sequence for processing to extract the regional point cloud features.

[0076] Specifically, in this embodiment of the invention, the local point cloud feature extractor uses the coordinate tensor of the three-dimensional capture center candidate point. Each point in the graph serves as the query center, obtained by inputting an RGB-D image. Perform a sphere query operation in the 3D point cloud of the scene obtained by back projection: The query radius is meters. For each query center, its position in the 3D point cloud of the scene is obtained sequentially. Nearest neighbor points (preferred) The three-dimensional coordinates of the point cloud are used to form a local point cloud cluster tensor. The size is Local point cloud cluster tensor The data is fed into a small point cloud processing network composed of multilayer perceptrons. This network contains a first one-dimensional convolutional layer Conv1D_1 and a second one-dimensional convolutional layer Conv1D_2 connected sequentially. The weight size of Conv1D_1 is... (Input Channel) Output channels), with batch normalization and ReLU activation; the weight size of Conv1D_2 is... Also with batch normalization and ReLU activation. Local point cloud cluster tensor After processing by a small point cloud processing network, the regional point cloud features are obtained. The size is .

[0077] The robot six-degree-of-freedom grasping and detection method provided in this invention obtains nearest neighbor points to form local point cloud clusters with a preset query radius, and performs feature extraction through a two-layer one-dimensional convolutional network. This effectively extracts local geometric features from the three-dimensional point cloud, thereby providing high-quality point cloud features for subsequent cross-modal fusion.

[0078] In some embodiments, performing cross-modal attention computation on the fused features to obtain the spatial pose feature tensor of the fused image and the deep structure information of the point cloud includes: The fused features are input into three fully connected layers in parallel to generate query tensors, key tensors, and value tensors, respectively. The query tensor and the transpose of the key tensor are subjected to batch matrix multiplication, and after processing with a scaling factor, a Softmax operation is performed to obtain the cross-modal attention weight matrix. The cross-modal attention weight matrix is ​​multiplied with the value tensor to obtain context-weighted features; The context-weighted features are reduced in dimensionality by a feature mapping layer, and max pooling is performed along the point dimension to remove the spatial dimension, thus obtaining the spatial pose feature tensor.

[0079] Specifically, in this embodiment of the invention, the cross-modal attention unit receives fused features. This allows it to pass through three fully connected layers, Linear_Q, Linear_K, and Linear_V, in parallel. The weights of these three fully connected layers are all [size missing]. None of them have bias. Fusion features After processing by Linear_Q, Linear_K, and Linear_V, query tensors are generated respectively. Key tensors Sum tensor All sizes Then, the query tensor will be used. With key tensor transpose (size Perform batch matrix multiplication (using the torch.bmm function in PyTorch), and divide by the scaling factor. Then, perform the `torch.nn.Softmax` operation in PyTorch along the last dimension to obtain a size of... The cross-modal attention weight matrix. This cross-modal attention weight matrix is ​​then compared with the value tensor. Perform matrix multiplication to obtain a matrix of size. The context-weighted features are then processed by a feature mapping layer MLP_attn (a fully connected layer with a size of [size missing]). Dimensionality reduction to (with ReLU) Then, a max pooling operation Max_Pool_1D (implemented by calling the torch.nn.MaxPool1d function in PyTorch) is performed along the point dimension (i.e., the second dimension, with a size of 256). After pooling, the dimension becomes By using the `torch.squeeze` function in PyTorch to remove spatial dimensions of size 1, a spatial pose feature tensor that aggregates image and point cloud deep structure information is finally obtained. Its size is .

[0080] The robot six-degree-of-freedom grasping and detection method provided in this invention generates query tensors, key tensors, and value tensors through three fully connected layers, and performs batch matrix multiplication, Softmax normalization, weighted summation, dimensionality reduction, and pooling operations. It can automatically learn the dependency relationship between image features and point cloud features, thereby achieving more comprehensive cross-modal feature fusion than simple channel stitching.

[0081] In some embodiments, the grabbing detection network is trained based on the following steps: Obtain training samples and their corresponding heatmap labels representing the actual crawling quality. The horizontal and vertical edge extraction operators are used to perform convolution calculations on the real crawled quality heatmap labels to extract first-order gradient change features and calculate the synthetic gradient magnitude. The synthesized gradient magnitude is normalized to generate a gradient magnitude attention mask; The error between the predicted crawling quality heatmap and the actual crawling quality heatmap label is weighted and calculated based on the gradient magnitude attention mask to obtain the heatmap quality loss. A total loss function is constructed based on the heatmap quality loss, crawling attribute loss, and crawling detection loss, and the parameters of the crawling detection network are updated based on the total loss function.

[0082] Specifically, the grab detection network needs to be trained before use. The purpose of training is to enable the network to accurately predict six-DOF grab detection results from color-depth images. The training process requires a large number of training samples, each of which includes a sample color-depth image and a corresponding true grab quality heatmap label. The true grab quality heatmap label is an ideal heatmap that is manually labeled or generated by other means, reflecting the true grab quality distribution at various locations in the scene.

[0083] During training, sample color-depth images are first input into the crawling detection network to be trained, and the network generates a predicted crawling quality heatmap. There is a difference between the predicted crawling quality heatmap and the actual crawling quality heatmap labels; the training objective is to reduce this difference.

[0084] In this embodiment of the invention, the network is trained on the GraspNet-1 Billion dataset and employs an Adaptive Moment Estimation (Adam) optimizer. To enable the network to learn the physical edge distribution of the real grasping region, this invention designs a gradient magnitude attention mask. For a given real crawl quality heatmap label Using the horizontal and vertical Sobel operators respectively ( Perform convolution calculations on it:

[0085]

[0086] The horizontal Sobel operator and the vertical Sobel operator Defined as a fixed convolution kernel matrix: , In the formula This represents a two-dimensional discrete pixel-level convolution operation, used to perform convolution at image pixel coordinates. First-order gradient change features are extracted. The synthetic gradient magnitude is then further calculated. The composite gradient magnitude reflects the degree of change in the heatmap value at that pixel location. At the edge of the capture area, the heatmap value changes more significantly, and the composite gradient magnitude is also larger; inside the capture area or in the background area, the heatmap value changes less significantly, and the composite gradient magnitude is also smaller.

[0087] The synthesized gradient magnitude is normalized to generate a gradient magnitude attention mask. Each value in the gradient magnitude attention mask represents the importance of the corresponding pixel location in the loss calculation. At the edges of the grasping region, the synthesized gradient magnitude is large, and therefore the normalized value is also large, resulting in larger values ​​for the gradient magnitude attention mask at these locations. In non-edge regions, the gradient magnitude attention mask values ​​are smaller.

[0088] In this embodiment of the invention, the total loss function of the network Includes heatmap quality loss Attribute capture loss and capture detection loss : .in, Employing gradient magnitude attention mask The weighted mean square error is calculated by multiplying the squared error between the predicted and actual crawl quality heatmap labels at each pixel position by the gradient magnitude attention mask value at the corresponding pixel position, and then summing the weighted squared errors of all pixel positions in the heatmap and dividing by the product of the height and width of the heatmap to obtain the heatmap quality loss. This forces the network to focus on optimizing the boundary edge between high response areas and the background. SmoothL1 loss is used; The standard point cloud 6-DoF grasping pose chamfering distance loss is adopted. That is, for each real grasping pose in the real grasping pose set, the predicted grasping pose with the closest Euclidean distance in the predicted grasping pose set is found, the square of the minimum distance is calculated, and the sum of the squares of the minimum distances of all real grasping poses is divided by the number of poses in the real grasping pose set. Similarly, for each predicted grasping pose in the predicted grasping pose set, the real grasping pose with the closest Euclidean distance in the real grasping pose set is found, the square of the minimum distance is calculated, and the sum of the squares of the minimum distances of all predicted grasping poses is divided by the number of poses in the predicted grasping pose set. The above two results are added together to obtain the grasping detection loss. and The multi-task loss weighting coefficients are preferably set to 1.0 and 10.0.

[0089] The parameters of the capture and detection network are updated based on the total loss function. The parameter update method typically uses a gradient descent optimizer to calculate the gradient of the total loss function with respect to the parameters of each layer in the network. Then, the parameter values ​​are adjusted in the opposite direction of the gradient to gradually reduce the total loss function. By iterating through this training process, the prediction accuracy of the network gradually improves until the training termination condition is met.

[0090] The robot six-degree-of-freedom grasping detection method provided in this invention calculates the gradient magnitude of the real heatmap by using an edge extraction operator and generates a gradient magnitude attention mask. It assigns higher weights to the edge regions of the heatmap for loss weighting and combines grasping attribute loss and grasping detection loss to construct a total loss function. This enables the network to focus on the edge positions of the grasping region during training, thereby improving the edge clarity and overall quality of the grasping heatmap.

[0091] In some embodiments, the normalization of the synthesized gradient magnitude to generate a gradient magnitude attention mask includes: Obtain the global minimum and global maximum values ​​of the synthesized gradient magnitudes; Calculate the difference between the synthesized gradient magnitude and the global minimum, and divide it by the difference between the global maximum and the global minimum to obtain the normalization ratio. The normalized ratio is weighted based on a preset gradient weighting coefficient, and a basic weight constant is added to generate the gradient magnitude attention mask.

[0092] Specifically, after obtaining the synthesized gradient magnitude, it is necessary to normalize the synthesized gradient magnitude to generate a gradient magnitude attention mask.

[0093] In this embodiment of the invention, the global minimum and global maximum values ​​of the synthesized gradient magnitudes are first obtained. The global minimum value refers to the minimum value of the synthesized gradient magnitudes at all pixel positions in the entire real-world captured quality heatmap label, and the global maximum value refers to the maximum value of the synthesized gradient magnitudes at all pixel positions.

[0094] Gradient magnitude attention mask is generated by normalizing the maximum and minimum values:

[0095] in, The gradient weighting coefficient is used to control the relative importance of edge regions in the loss calculation. The value of the gradient weighting coefficient can be set according to actual needs, and it is preferred to set it to 4.0. and These represent the labels for the entire real-world captured heatmap of the current input. The global minimum and global maximum values ​​of the synthesized gradient magnitude calculated from all pixels.

[0096] The robot six-degree-of-freedom grasping and detection method provided in this invention can generate an attention mask with larger values ​​in edge regions and smaller values ​​in non-edge regions by obtaining the global minimum and maximum values ​​of gradient magnitude, calculating the normalization ratio, weighting by gradient weighting coefficients and adding a basic weight constant. This enables targeted weighted training of heatmap edge regions and improves the accuracy of predicting heatmap edge positions.

[0097] The apparatus provided in the embodiments of the present invention will be described below. The apparatus described below can be referred to in correspondence with the method described above.

[0098] Figure 2 This is a schematic diagram of the structure of the robot six-degree-of-freedom grasping and detection device provided by the present invention, as shown below. Figure 2 As shown, the device includes an image acquisition module 210 and a model inference module 220 connected in sequence.

[0099] Image acquisition module 210 is used to acquire a color-depth image of the current scene; The model inference module 220 is used to input the color-depth image into the grasping detection network and obtain the six-degree-of-freedom grasping detection result output by the grasping detection network; The grasping and detection network includes a Gaussian heatmap generation module and a cross-modal grasping and detection module; The Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, and predicts the Gaussian kernel parameters corresponding to each anchor point position in the image feature map. It then performs a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous crawling quality heatmap and crawling attribute map. The cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the module generates the six-degree-of-freedom grasping detection result.

[0100] The robot six-DOF grasping detection device provided in this invention uses a Gaussian heatmap generation module to encode features in a color-depth image and predict Gaussian kernel parameters. Then, it uses Gaussian density rendering to perform two-dimensional Gaussian kernel weighted superposition operations to generate continuous grasping quality heatmaps and grasping attribute maps. Simultaneously, a cross-modal grasping detection module performs three-dimensional candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor. Based on the grasping attribute map and the spatial pose feature tensor, a six-DOF grasping detection result is generated. This device can generate grasping quality heatmaps with continuity and integrity, and achieve deep fusion of image features and point cloud features, thereby improving the accuracy of grasping detection.

[0101] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communications bus 340. The processor 310 can call logical commands stored in the memory 330 to execute the methods described in the above embodiments, for example: A color-depth image of the current scene is acquired; the color-depth image is input into a grasping detection network to obtain a six-degree-of-freedom grasping detection result output by the grasping detection network; wherein, the grasping detection network includes a Gaussian heatmap generation module and a cross-modal grasping detection module; the Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, and predicts the Gaussian kernel parameters corresponding to each anchor point position in the image feature map, and performs a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous grasping quality heatmap and a grasping attribute map; the cross-modal grasping detection module performs three-dimensional candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the image and the deep structure information of the point cloud, and generates the six-degree-of-freedom grasping detection result based on the grasping attribute map and the spatial pose feature tensor.

[0102] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] The processor in the electronic device provided in this embodiment of the invention can call logical instructions in the memory to implement the above method. Its specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, which will not be repeated here.

[0104] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0105] The specific implementation method is the same as the aforementioned method implementation method and can achieve the same beneficial effects, so it will not be repeated here.

[0106] This invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0107] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot six-degree-of-freedom grasping and detection method, characterized in that, include: Obtain a color-depth image of the current scene; The color-depth image is input into the grasping detection network to obtain the six-degree-of-freedom grasping detection result output by the grasping detection network; The grasping and detection network includes a Gaussian heatmap generation module and a cross-modal grasping and detection module; The Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, and predicts the Gaussian kernel parameters corresponding to each anchor point position in the image feature map. It then performs a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous crawling quality heatmap and crawling attribute map. The cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the module generates the six-degree-of-freedom grasping detection result.

2. The robot six-degree-of-freedom grasping and detection method according to claim 1, characterized in that, The cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, it generates the six-degree-of-freedom grasping detection result, including: Based on the response values ​​of each pixel in the captured quality heatmap, a preset number of pixels are selected, and a two-dimensional capture candidate coordinate set is constructed. By combining depth information, the pixels in the set of two-dimensional grabbing candidate coordinates are projected onto three-dimensional space to obtain the coordinate tensor of the three-dimensional grabbing center candidate point. The color-depth image is back-projected to obtain a three-dimensional point cloud; Using each 3D grabbing center candidate point as the query center, a ball query operation is performed in the 3D point cloud to obtain the local point cloud corresponding to each 3D grabbing center candidate point. Feature extraction is performed on the local point cloud to obtain the regional point cloud features. The local point cloud is reprojected onto the two-dimensional image plane where the image feature map is located, and the region image features corresponding to the local point cloud are retrieved from the image feature map by interpolation sampling. The regional point cloud features and the regional image features are concatenated to form a fusion feature, and cross-modal attention calculation is performed on the fusion feature to obtain the spatial pose feature tensor of the fusion image and the deep structure information of the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the six-degree-of-freedom grasping detection result is generated.

3. The robot six-degree-of-freedom grasping and detection method according to claim 1, characterized in that, The Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, including: The color-depth image is initially downsampled using an initial convolutional layer and a max-pooling layer. The processed image is input into multiple sequentially connected residual feature blocks for feature extraction, resulting in an image feature map with reduced size and expanded channel count.

4. The robot six-degree-of-freedom grasping and detection method according to claim 1, characterized in that, The method of predicting the Gaussian kernel parameters corresponding to each anchor point position in the image feature map involves performing a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous crawling quality heatmap and crawling attribute map, including: Parallel convolution processing is performed on the image feature maps to predict the actual offset map, horizontal standard deviation map, vertical standard deviation map, correlation coefficient map, and magnitude weight map, which serve as the Gaussian kernel parameters corresponding to each anchor point position. Based on the horizontal standard deviation map and the vertical standard deviation map, the corresponding local influence radius is calculated and generated; In the constructed two-dimensional grid coordinate system, the Gaussian kernel parameters of the anchor points in the image feature map covered by the local influence radius are retrieved at each grid pixel coordinate, and a two-dimensional Gaussian kernel weighted superposition operation is performed to obtain the superimposed feature tensor. The superimposed feature tensor is segmented along the feature channel dimension to obtain the crawling quality heatmap and the crawling attribute map, respectively.

5. The robot six-degree-of-freedom grasping and detection method according to claim 2, characterized in that, The step of combining depth information to project the pixels in the two-dimensional grabbing candidate coordinate set onto three-dimensional space to obtain the three-dimensional grabbing center candidate point coordinate tensor includes: The two-dimensional grid coordinates of each pixel in the two-dimensional crawling candidate coordinate set on the crawling quality heatmap are mapped to the resolution-aligned original depth map space to obtain the corresponding depth value. By combining the camera intrinsic parameter matrix, the two-dimensional grid coordinates and the depth values ​​are converted into three-dimensional coordinates to obtain the coordinate tensor of the three-dimensional grabbing center candidate point.

6. The robot six-degree-of-freedom grasping and detection method according to claim 2, characterized in that, The process involves using each 3D grabbing center candidate point as the query center, performing a sphere query operation in the 3D point cloud to obtain the local point cloud corresponding to each 3D grabbing center candidate point, and extracting features from the local point cloud to obtain region point cloud features, including: Using a preset query radius, a preset number of nearest neighbor points closest to each of the candidate points of the three-dimensional grasping center are obtained in the three-dimensional point cloud, forming a local point cloud cluster tensor as the local point cloud; The local point cloud cluster tensor is input into a small point cloud processing network containing a first one-dimensional convolutional layer and a second one-dimensional convolutional layer connected in sequence for processing to extract the regional point cloud features.

7. The robot six-degree-of-freedom grasping and detection method according to claim 2, characterized in that, The step of performing cross-modal attention calculation on the fused features to obtain the spatial pose feature tensor of the fused image and the deep structure information of the point cloud includes: The fused features are input into three fully connected layers in parallel to generate query tensors, key tensors, and value tensors, respectively. The query tensor and the transpose of the key tensor are subjected to batch matrix multiplication, and after processing with a scaling factor, a Softmax operation is performed to obtain the cross-modal attention weight matrix. The cross-modal attention weight matrix is ​​multiplied with the value tensor to obtain context-weighted features; The context-weighted features are reduced in dimensionality by a feature mapping layer, and max pooling is performed along the point dimension to remove the spatial dimension, thus obtaining the spatial pose feature tensor.

8. The robot six-degree-of-freedom grasping and detection method according to claim 1, characterized in that, The crawling detection network was trained based on the following steps: Obtain training samples and their corresponding heatmap labels representing the actual crawling quality. The horizontal and vertical edge extraction operators are used to perform convolution calculations on the real crawled quality heatmap labels to extract first-order gradient change features and calculate the synthetic gradient magnitude. The synthesized gradient magnitude is normalized to generate a gradient magnitude attention mask; The error between the predicted crawling quality heatmap and the actual crawling quality heatmap label is weighted and calculated based on the gradient magnitude attention mask to obtain the heatmap quality loss. A total loss function is constructed based on the heatmap quality loss, crawling attribute loss, and crawling detection loss, and the parameters of the crawling detection network are updated based on the total loss function.

9. The robot six-degree-of-freedom grasping and detection method according to claim 8, characterized in that, The step of normalizing the synthesized gradient magnitude to generate a gradient magnitude attention mask includes: Obtain the global minimum and global maximum values ​​of the synthesized gradient magnitudes; Calculate the difference between the synthesized gradient magnitude and the global minimum, and divide it by the difference between the global maximum and the global minimum to obtain the normalization ratio. The normalized ratio is weighted based on a preset gradient weighting coefficient, and a basic weight constant is added to generate the gradient magnitude attention mask.

10. A six-degree-of-freedom robotic grasping and detection device, characterized in that, include: The image acquisition module is used to acquire a color-depth image of the current scene; The model inference module is used to input the color-depth image into the grasping detection network and obtain the six-degree-of-freedom grasping detection result output by the grasping detection network. The grasping and detection network includes a Gaussian heatmap generation module and a cross-modal grasping and detection module; The Gaussian heatmap generation module performs feature encoding on the color-depth image to obtain an image feature map, and predicts the Gaussian kernel parameters corresponding to each anchor point position in the image feature map. It then performs a two-dimensional Gaussian kernel weighted superposition operation on each Gaussian kernel parameter through Gaussian density rendering to generate a continuous crawling quality heatmap and crawling attribute map. The cross-modal grasping detection module performs 3D candidate point projection, local point cloud feature extraction, image feature retrieval, and cross-modal attention fusion processing on the grasping quality heatmap to obtain a spatial pose feature tensor that fuses the deep structure information of the image and the point cloud. Based on the grasping attribute map and the spatial pose feature tensor, the module generates the six-degree-of-freedom grasping detection result.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the robot six-degree-of-freedom grasping and detection method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot six-degree-of-freedom grasping and detection method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot six-degree-of-freedom grasping and detection method as described in any one of claims 1 to 9.

Images