A Transformer-based region estimation and multi-level feature fusion grasping and detection method

Abstract

The present invention discloses a Transformer-based region estimation and multi-level feature fusion grasping and detection method, which relates to the field of robotics. The steps of the present invention are as follows: the RGB image is detected by a 2D target detector to select the target area that needs to be grasped and detected, and is combined with the depth image of the same object into a four-channel picture as input; the four-channel input is divided into multiple non-overlapping patches by the Patch Partition module, and each patch is regarded as a token, representing a cascade of original input pixels. The present invention adopts the Transformer architecture and uses the encoder and decoder structure of the Transformer architecture to encode and decode the grasping and detection image, so that the model has excellent global feature association and modeling capabilities, and has stronger feature expression and generalization capabilities; and in order to enhance the model's migration and generalization capabilities and enrich and retain more fine-grained features, regional estimation and multi-level feature fusion are used to enhance the model's grasping performance.

Description

Technical Field

[0001] The present invention belongs to the field of robotics technology, and in particular relates to a Transformer-based region estimation and multi-level feature fusion grasping and detection method. Background Art

[0002] Advances in science and technology are driving the development of the robotics industry. Robotic applications in healthcare, education, public services, and other fields are expanding year by year. Implementing gripping operations with robotic arms in unknown and unstructured environments is a fundamental yet crucial part of daily robotics operations. Grasping position detection involves using algorithms to find a feasible grasping position on a given object that meets the requirements of the grasping task.

[0003] Before the 21st century, analytical and empirical methods were the mainstream approaches to grasp position detection. Analytical methods primarily utilized geometric modeling, kinematics, and dynamics knowledge, and generated a suitable grasping configuration through complex modeling calculations. Empirical methods focused on summarizing experience from previous successful grasps, omitting the process of establishing mathematical and physical models. Instead, grasp positions were ranked based on certain criteria to ultimately determine the grasp position.

[0004] Currently, most people use data-driven machine learning methods to detect the target grasping position. Machine learning grasping detection methods vary depending on the actual application scenario. Common grasping scenarios can be divided into two categories: 2D plane grasping and 6DoF spatial grasping. 2D plane grasping methods are limited in grasping direction and can only obtain relevant information about the object to be grasped from one direction and grasp it. The input information commonly used is RGB images and RGB-D images.

[0005] It is mainly divided into two-stage grasp detection and single-stage grasp detection; the two-stage grasp detection, the first stage is the candidate grasp generation stage, in this stage, the model generates multiple grasp positions from the image as candidate grasps; the second stage is the candidate grasp evaluation stage, in this stage, by designing the evaluation criteria in the model, the areas corresponding to the candidate grasps sampled in the previous stage are scored, so as to obtain the grasp success rate corresponding to each area; the single-stage grasp detection mainly utilizes the sliding window effect and parameter sharing characteristics of the convolutional neural network to realize the calculation sharing of different image areas, that is, feature calculation is performed on the complete original image, and target detection is performed on all unknown objects in the image, thereby integrating the previously complicated candidate area sampling and feature calculation together, and obtaining the target result after only a single detection;

[0006] With the development of sensors, six-degree-of-freedom grasping for point clouds and depth images has gradually become a hot topic in the field of robotics. 6Dof spatial grasping methods allow grippers to grasp objects from any direction in three-dimensional physical space. In 6Dof spatial grasping methods, the common research scenario is grasping known objects, so such problems are often converted into 6D object pose estimation problems.

[0007] Traditional analytical methods for predicting grasping positions must make certain assumptions, such as accurately obtaining model parameters of the object to be grasped, environmental parameters, and reasonably simplifying the fixture model. However, these assumptions are often difficult to hold true in real-world grasping scenarios. Analytical methods also require 3D modeling of known objects, but it is often difficult to completely and accurately reconstruct actual objects seen for the first time through visual perception. Empirical methods rely heavily on previous grasping experience, and using them to detect grasping positions requires the establishment of a comprehensive grasping experience database. However, establishing such a database is impractical, as it is impossible to collect all grasping positions.

[0008] Furthermore, due to the increasing complexity of unstructured environments and the diversification of grasping tasks, empirical methods have become increasingly computationally complex, making it difficult to meet the demands of real-world tasks. Among the current machine learning-based robotic grasping position detection methods, Lenz's two-stage grasping detection method is one of the earliest methods to apply machine learning to robotic grasping detection, opening up new avenues for robotic grasping research.

[0009] However, this two-stage detection method requires traversing the entire image, which is very time-consuming and has poor real-time performance in actual application. Mingshuai Dong proposed a model designed to solve the problem of target object grasping detection in complex scenes. The model first uses a network to segment the corresponding object, and then uses another network to detect the grasping position on the segmented object, which has a good detection effect. However, the model is completely dependent on the segmentation efficiency of the first network and is limited by the segmentation recall rate of the target segmentation network. Kumra uses two parallel network structures to perform multimodal fusion of RGB images and three-channel depth images to form a grasping detection model, and finally uses a shallow convolutional neural network to predict the grasping position of a given object. However, Kumra's multimodal fusion operation is simply splicing the final feature output of the two networks, without fully utilizing the feature information of each granularity in the network, which may lead to the loss of some feature information of the detection target, which is not conducive to the grasping detection of multiple targets. The 6Dof grasping detection method is usually based on 3D point cloud data for detection, and has high requirements for the integrity of point cloud data. Therefore, the efficiency of this type of algorithm is relatively low.

[0010] Current machine learning-based grasping and detection methods are mainly built based on convolutional neural networks (CNNs). Since convolutional networks have poor ability to associate global information, the grasping quality of the network will drop significantly when there are multiple targets or the scene changes in the real environment. Summary of the Invention

[0011] The purpose of the present invention is to provide a Transformer-based region estimation and multi-level feature fusion capture detection method to solve the technical problems raised in the above background technology.

[0012] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0013] The present invention is a Transformer-based region estimation and multi-level feature fusion capture detection method, the steps are as follows:

[0014] The RGB image is detected by the 2D object detector to select the target area that needs to be grasped and detected, and is combined with the depth image of the same object to form a four-channel image as input;

[0015] The four-channel input is split into multiple non-overlapping patches by the Patch Partition module. Each patch is considered a "token" and represents the concatenation of the original input pixels.

[0016] The fixed position code is embedded into each token and then passed to the encoder module;

[0017] In the encoder, the Linear Embedding layer converts each token into a C-dimensional embedding vector, and then the number of tokens is changed through the patch merging + Transformer Block module;

[0018] The decoder performs the opposite operation of patch merging and recovers the changed number of tokens through the patch expanding + transformer block module.

[0019] The output features of each level collected by the Encoder are fused through a multi-level feature fusion network;

[0020] Generate pixel heatmaps of the same size as the detection image to predict grasp locations.

[0021] Furthermore, the Encoder module collects the feature maps processed by each Encoder sub-block, fuses the feature maps through downsampling and convolution operations, and meets the input specifications of the Decoder module.

[0022] Furthermore, the fused feature map that meets the input specifications of the Decoder module is processed by the Decoder module, and the width of the grabbing box related to the grab position prediction, the sine and cosine components of the rotation angle of the grabbing box, and the grab confidence score are obtained in the grab prediction module.

[0023] Furthermore, the grasp prediction module obtains the width of the grasp box predicted for the grasp position, the sine and cosine components of the grasp box's rotation angle, and the grasp confidence score, which is expressed as follows:

[0024]

[0025]

[0026]

[0027]

[0028] Among them, g represents the grasping position predicted by the grasping detection model, g Score 、g Sin 、g Cos 、g width Represents the confidence score of the predicted grasp g, the sin and cos components of the rotation angle, and the width of the grasp prediction box; x n Represents the data input from the grasp detection network to the grasp prediction module; These are the four convolution operations that the crawl detection network performs on the input data.

[0029] Furthermore, the RGB image is detected by the 2D object detector to select the target area that needs to be captured and detected. The steps are as follows:

[0030] The RGB image is detected by Faster-RCNN, the position of the grasped object is marked with a rectangular box, and the other backgrounds in the rectangular area are removed and filled with 0.

[0031] Furthermore, before the data enters a single Transformerblock, it is normalized using LN and then enters the W-MSA module through the linear mapping matrix W QKV Project the input sequence t to obtain Q, K, and V, which are expressed as follows:

[0032] [Q,K,V]=tW QKV

[0033] Where t represents the input sequence, Q represents the query vector, K represents the key vector, and R represents the value vector;

[0034] The similarity between Q and K is calculated by matrix dot product and weighted with V. The calculation expression of self-attention is as follows:

[0035]

[0036] Where Attention(Q,K,V) represents the self-attention score of the input sequence t, SoftMax represents the normalized exponential function, d is the dimension of the vector Q, B represents the learned relative position encoding, and T represents the transposition symbol, which is used to transpose the vector K.

[0037] After passing through the W-MSA module, the residual connection is used to fuse with the initial input and then normalized. After passing through two linear connection layers (MLP) with GELU, the residual connection is used again to fuse the previous input information before passing it to the Transformer block with SW-MSA. The expression is as follows:

[0038]

[0039]

[0040]

[0041]

[0042] in, and t l They represent the outputs of the lth module (S) W-MSA module and MLP module, t l-1 Represents the output features of the previous layer.

[0043] Furthermore, each pixel in the heat map represents the captured position of the pixel;

[0044] The output resolution of the decoder is converted to the same size as the image through linear layers and reshape operations. Four 1x1 convolutions are used to generate four single-channel pixel heat maps corresponding to S, W, sin 2θ, and cos 2θ of the captured detection. The grasping angle θ is determined, and the final grasp is determined by the highest-scoring grasp in the grasping score graph S.

[0045] Furthermore, the predicted grasping position is verified as follows:

[0046] When the grasp position G and the ground truth G' both meet the condition |G θ -G' θ|<30° and conditions Recognize that the grasping position G is correct;

[0047] Conditions|G θ -G' θ |<30° means the angle difference between the predicted grab box and the annotation rectangle is less than 30°;

[0048] condition Indicates that the intersection-over-union ratio of the predicted grab box and the annotated rectangle is greater than 25%.

[0049] The present invention has the following beneficial effects:

[0050] 1. This paper adopts the Transformer architecture, using its encoder and decoder structures to encode and decode the grasp detection images, giving the model excellent global feature association and modeling capabilities. Compared with traditional grasp detection models based on convolutional neural networks, it has stronger feature expression and generalization capabilities. To enhance the model's transfer and generalization capabilities and enrich and retain more fine-grained features, it also uses regional estimation and multi-level feature fusion to enhance the model's grasp performance. Extensive experiments have shown that our proposed method has better grasp performance than CNN-based methods.

[0051] 2. The present invention utilizes a feature fusion module to fuse the feature outputs of each sub-module of the Encoder, retains the feature information of each level and then inputs it into the Decoder module for grasp detection; then, it uses the current excellent 2D object detector to make a preliminary position prediction of the grasped target and remove the influence of other noise factors such as background, so that our network has excellent grasp detection performance.

[0052] 3. The present invention uses a target detector with relatively good detection effect to perform preliminary target position positioning on the RGB image input of the model, removes the background outside the grasping detection target area, and then combines it with the depth image into four channels as the input of the model. This operation allows the grasping detection model to pay more attention to the grasping target, allowing the model to obtain the feature information of the grasping target more quickly, reducing the impact of environmental background factors on grasping detection, and enhancing the generalization ability of the model.

[0053] 4. The present invention uses multiple convolutional neural networks to fuse the output features of each sub-module of the Encoder in the Transformer. In this way, the information input from the Encoder to the Decoder includes the useful feature information obtained by the modules at each level. The feature information is richer and more fine-grained features are retained. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 Schematic diagram of the Transformer-based region estimation and multi-level feature fusion capture detection model of the present invention;

[0056] Figure 2 Schematic diagram of the TransformerBlock structure of the present invention;

[0057] Figure 3 This is a schematic diagram showing the grabbing relationship of the present invention;

[0058] Figure 4 A schematic diagram showing the visualization of some experimental results on the Cornell dataset of the present invention;

[0059] Figure 5 This is a visualization diagram of some experimental results on the Jacquard dataset of the present invention;

[0060] Figure 6 Schematic diagram for comparing the grabbing and detection results of the Cornel dataset of the present invention;

[0061] Figure 7 Schematic diagram for comparison of the detection results of the Jacquard dataset of the present invention;

[0062] Figure 8 Schematic diagram for comparing the grabbing detection results of the multi-target grabbing dataset of the present invention;

[0063] Figure 9 This is a schematic diagram of capturing a real scene of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0065] See also Figure 1-9 As shown, the present invention is a Transformer-based region estimation and multi-level feature fusion capture detection method, the steps are as follows:

[0066] (1) Technical solution

[0067] 1. Network Architecture

[0068] like Figure 1 As shown in the figure, the RGB image is detected by the 2D target detector to select the approximate area of ​​the target that needs to be grasped and detected, and then combined with the depth image of the same object into a four-channel image as the input of the following module.

[0069] The four-channel input is partitioned into multiple non-overlapping patches by the Patch Partition module. Each patch is considered a "token," representing a concatenation of the original input pixels. Specifically, a 4-channel input (4*224*224) of size 224*224 is partitioned into (224 / 4)*(224 / 4) smaller 4*4 patches with 3 channels. Each token is then embedded with a fixed-position encoding and passed to the Encoder. After entering the Encoder module, the feature maps processed by each Encoder sub-block are collected and fused through a series of downsampling and convolution operations to meet the input specifications of the Decoder module. After further processing in the Decoder module, the grasp prediction module obtains the width of the grasping bounding box, the sine and cosine components of the grasping bounding box's rotation angle, and the grasp confidence score.

[0070] The final output expression is:

[0071]

[0072]

[0073]

[0074]

[0075] Among them, g represents the grasping position predicted by the grasping detection model, g Score 、g Sin 、g Cos 、g width Represents the confidence score of the predicted grasp g, the sin and cos components of the rotation angle, and the width of the grasp prediction box; x n Represents the data input from the grasp detection network to the grasp prediction module; These are the four convolution operations that the crawl detection network performs on the input data.

[0076] 2. Target Position Estimation

[0077] Object detection has developed a mature detection process and is successfully applied in various real-world fields, such as autonomous driving, facial recognition, and industrial inspection. While grasp detection and object detection differ in many ways, their technical approaches generally share many similarities. For example, both require the location of the object to be detected to be determined in advance, while grasp detection not only locates the object to be grasped but also predicts the corresponding grasping information.

[0078] Therefore, the present invention uses a relatively mature 2D target detector - Faster-RCNN (given that Yolo-Fastest has a very fast detection speed, only 5.37ms per frame, and has a high accuracy) to detect the RGB image. The position of the object to be captured is marked with a rectangular frame in advance, and then the other background within the rectangular area is removed and filled with zeros. In this way, the background area has no effect on the network weight. Considering that background removal may reduce the generalization ability of the model, the present invention fuses the processed RGB image with the unprocessed Depth image into an RGB-D four-channel image as the input of the network. Of course, no target detector can locate the target to be detected 100%. The processing method of the present invention is that in the training stage, when there is no target to be detected in the area selected by the detector, the original RGB image is not processed, and the original image information is retained; in the prediction stage, because the position label of the detected target is unknown, the measures exactly the same as those in the training stage cannot be adopted. The measures taken by the present invention are to select the area formed by the top 5 targets with the highest detection score. At the same time, the area of ​​the detected target is expanded by 30% in both training and prediction to avoid the occurrence of incomplete object detection.

[0079] 3. TransformerBlock in the network

[0080] The TransformerBlock of the present invention adopts the SwinTransformer based on the moving window. The difference from the general TransformerBlock is that the standard multi-head attention module is replaced by the multi-head attention module of the moving window. The computational complexity of the model using this moving window Transformer module is much lower than that of the general visionTransformer, and the global and local feature acquisition capabilities of the model are enhanced. Figure 2The following is a diagram of the structure of two SwinTransformerblocks. Before the data enters a single Transformerblock, it is normalized using Layernormalization (LN) and then enters the window multi-head self-attention (W-MSA) module (this is the key to the attention mechanism, which is used to establish long-distance dependencies in the image space and make up for the defect of the limited receptive field of the convolutional neural network). First, the linear mapping matrix W QKV Project the input sequence t to obtain Q, K, and V, which are defined as follows:

[0081] [Q,K,V]=tW QKV

[0082] Where t represents the input sequence, Q represents the query vector, K represents the key vector, and R represents the value vector.

[0083] Then, the similarity between Q and K is calculated by matrix dot product and then weighted with V. The calculation process of self-attention is as follows:

[0084]

[0085] Among them, Attention(Q,K,V) represents the self-attention score calculated by the three vectors Q, K, and V converted from the above t, that is, the self-attention score of the input sequence t. Q, K, and V represent the query vector, key vector, and value vector respectively; SoftMax is a general function name called the normalized exponential function, usually called the SoftMax function; d is the dimension of the vector Q; B is a learnable relative position encoding; T is a mathematical symbol, that is, the transpose symbol, used to transpose the vector K.

[0086] like Figure 2 As shown in the figure, after passing through the W-MSA module, it is fused with the initial input using a residual connection, then normalized again, and then passed through two linear connection layers (MLP) with GELU. Finally, the previous input information is fused again using a residual connection before being passed to the next Transformer block with SW-MSA, or the moving window multi-head attention mechanism. The data change process of this Transformer module group with window change mechanism and moving window change mechanism is as follows:

[0087]

[0088]

[0089]

[0090]

[0091] in, and t l They represent the outputs of the lth module (S) W-MSA module and MLP module, t l-1 Represents the output features of the previous layer, W-MSA, LN, MLP, SW-MSA are Figure 2 The module name in .

[0092] 4. Decoder and Encoder

[0093] The feature map passed into the model is encoded by the encoder so that the feature information contained in the feature map obtained by the model is more in line with the requirements of the grasp detection task. Before the image data with 4 channels and size W×H enters the encoder, it must be split into Each non-overlapping 4-channel patch of size P×P is considered a "token" representing the concatenation of the original input pixels. In the Encoder, the Linear Embedding layer converts each token into a C-dimensional embedding vector, which then passes through two consecutive TransformerBlocks without any change in the output dimension. The patch merging layer then passes through three (patch merging + Transformer Block×2) modules to return the number of tokens to the original size. The dimension becomes twice as large as before, so after three modules the number of tokens becomes The dimension becomes 8C. These stages generate features at different levels. We collect the feature maps output by these four stages and fuse them in the feature fusion module.

[0094] Patch merging layer: concatenates the features of every 2×2 adjacent patches of the input. This operation reduces the number of tokens by a factor of 2×2=4 (equivalent to 2X pooling or downsampling), because the resolution of the token is reduced by four times, and the dimension of the token is doubled through linear mapping.

[0095] The decoder of this invention is also constructed using the Swin TransformerBlock. Unlike the patch merging in the encoder, we use the Patch Expanding layer to expand the feature information transmitted from the previous module. More precisely, we perform the opposite operation of Patch Merging. Through three modules (patch expanding + Transformer Block × 2), the tokens of dimension 8C and number H / 8P × W / 8P are restored to the size before entering the encoder, that is, H / P × W / P × C.

[0096] Patch Expanding layer: First, the dimension of the token (h×w×c) is doubled by the linear layer, and then the shape of the token is reshaped, transforming the token of shape (h×w×2c) into

[0097] 5. Multi-granularity feature fusion

[0098] The residual connection within the Transformer Block enables a certain feature fusion between the networks within the Block, but there is no such connection between the individual Blocks. The Encoder module of the present invention has eight Transformer Blocks. As the network deepens, the extracted features become more abstract, but this is also accompanied by the loss of spatial details and fine-grained features. This has a great impact on the pixel-level grasp detection task, making the model's grasp detection and generalization effect in small targets worse. It is necessary to combine low-level and high-level features to describe the boundaries and details of the grasped objects. Therefore, the present invention designs a multi-level feature fusion network based on a convolutional neural network to fuse the output features of each level collected by the Encoder. Because the feature maps output by each module of the Encoder are of different scales, we use a convolutional network to transform the feature dimensions of the outputs of each level (considering that the output feature scale of the low-level modules in the Encoder is larger, the ratio of reduction to small scale is greater, so the present invention uses more convolution layers at the low level). Then, through the downsampling operation, their sizes are made consistent and fused together. Finally, a layer of 1x1 convolution is used to make their channel number consistent with the input of the Decoder. The multi-level feature fusion operation of the present invention enables the features of each level of the encoder to be retained, and the semantic information input to the decoder is richer and more feature details are retained.

[0099] 6. Grasp Position Prediction and Loss Function

[0100] This paper implements end-to-end pixel-level grasp detection, using a grasp network to generate a pixel heat map of the same size as the detection image to predict the grasp position. Each pixel in the heat map represents the grasp position of that pixel. Specific implementation: The output resolution of the decoder is converted to 224*224, which is the same size as the image, through a linear layer and a reshape operation. Then, four 1x1 convolutions are used to generate four single-channel 224*224 pixel heat maps corresponding to S, W, sin 2θ, and cos 2θ of the grasp detection. Sin 2θ and cos 2θ are used to calculate the grasp position. Determine the grasping angle θ, and the final grasp is determined by the highest score grasp in the grasp score map S. This turns the grasp position detection into an end-to-end regression problem. The model proposed in this invention finds a mapping relationship F: I~G from the input image I to the grasp position G, so that the prediction The distance to groundTruth (i.e. the real captured label G of the input image I) is minimized. The loss function of the network is defined as follows:

[0101]

[0102] In the formula, Depend on in And the i and m in the formula, It is composed of the grasping position G and i and m in the formula.

[0103] Among them, i represents the sum of the results of the subsequent smooth operations from 1 to n, m represents the sum of S, W, and θ after being substituted into the Smooth operation, and n refers to the number of pixels in the image to be detected. The definition of the Smooth(x, y) function is as follows:

[0104]

[0105] (2) Technical effects

[0106] 1. Experimental Results and Analysis

[0107] This paper validates its approach using two commonly used datasets, Cornell and Jacquard, and also tests it in a multi-target clutter scenario. Adopting the 5-fold cross-validation evaluation method employed by most studies, the experimental data samples are divided into five subsamples: four serving as training sets and one serving as validation sets, with each subsample validated once. Furthermore, to prevent overfitting of the model during training, the paper expands the dataset through random cropping, rotation, and scaling.

[0108] The evaluation criterion used in this paper is the rectangle metric on the Cornell grasping dataset. If the grasping position G predicted by the model and the ground truth G' meet both of the following conditions, the prediction is considered correct.

[0109] |G θ -G' θ |<30° (1)

[0110]

[0111] Condition (1) indicates that the angle difference between the predicted grasping box and the annotated rectangle is less than 30°, which means that the difference between the grasping angles can be in the range [-30°, 30°]. Condition (2) indicates that the intersection over union (IoU) between the predicted grasping box and the annotated rectangle is greater than 25%; in grasp position detection, this metric is also known as the Jaccard index.

[0112] For a parallel gripper, the goal of grasp detection is to detect a valid grasping rectangle in the image that satisfies the grasping task. The grasp representation method used in this invention is as follows:

[0113] G={S,W,θ}∈R 3×W×H

[0114] This representation method is for pixel-level grasp detection, S, W, θ∈R W×H Where S represents the grasp score of each pixel in the image, ranging from [0, 1]. A higher S indicates a more accurate grasp position. W represents the grasp width of each pixel, and θ represents the grasp angle of each pixel, which is determined by the Sin and Cos components. W and H are the width and height of the detection image. Figure 3 The grasping relationship is shown in the figure. For each pixel (x, y), there is a G representing the grasping. The parameters w and θ of G are shown in the figure. There is also a corresponding S to represent the grasping score of the pixel point.

[0115] The grasp detection network proposed in this paper was trained and tested on the Cornell grasp dataset and the Jacquard grasp dataset, achieving grasp accuracy rates of 98.6% and 95.9%, respectively, and a detection rate exceeding 39 frames per second. Tables 1 and 2 compare grasp detection results on the Cornell and Jacquard grasp datasets using the proposed method and several other current, well-performing grasp detection methods.

[0116] Table 1. Cornell dataset crawling detection results

[0117]

[0118]

[0119] Table 2 Jacquard dataset crawling detection results

[0120]

[0121] The present invention uses two evaluation parameters, Image-Wise split (IW) and Object-Wise split (OW), to prove the grasping prediction accuracy. Image-Wise split (IW) refers to randomly dividing all images in the data set. This division is conducive to evaluating the performance of the model in detecting the position and posture of objects. Object-Wise split (OW) refers to randomly dividing all object instances in the data set. This division method is conducive to evaluating the performance of the model when testing unknown objects. At the same time, different model inputs and model detection time efficiency are also considered for comprehensive comparison. When RGB and depth images are used as input, the model proposed in the present invention can achieve a grasping accuracy of 98.6%, which is better than other algorithms, and the detection efficiency can reach 39.2 frames per second, with excellent real-time detection effect. When only the depth image is used as input, due to the lack of the target area estimation process, the detection rate of the model is improved to more than 48 frames, but the grasping accuracy is reduced. When using only RGB images as network input, the model's generalization ability is not as good as the RGB+Depth combination because the background outside the target is removed and it is not combined with the depth image. The accuracy of OW is lower than that of IW, but the detection accuracy is still relatively high. For the Jacquard dataset, using 90% of the data as the training set and the remaining 10% as the validation set, the grasp detection network of our invention achieved an accuracy of 95.9%.

[0122] 2. Visualization effect analysis

[0123] Figure 4 and Figure 5 The visualization of some results of the present invention on the Cornell dataset and the Jacquard dataset are respectively shown. The first row of the figure shows the point with the highest grasping quality score in the Score heat map detected by the present invention on a given detection image and the grasping rectangle composed of other parameters corresponding to the point. The second, third and fourth rows respectively show the three heat maps of the grasping quality score Score, the grasping rectangle rotation angle Angel and the grasping rectangle width Width obtained by the model proposed by the present invention, where the Angle heat map is composed of the two angle components Sin and Cos through the formula We can see that the grasping detection rectangles obtained by the present invention are very suitable for grasping the objects in the detection image, and the three grasping detection heat maps obtained also accurately segment the contours of the appropriate grasping points.

[0124] To better demonstrate the detection effect of the present invention, the detection results of the present invention are compared with those of the grasping detection networks GR-CNN and GGCNN. Figure 6 、 Figure 7 、 Figure 8 The heat map comparison of some detection effects of the grasping detection network proposed in the present invention and GR-CNN and GG-CNN on the Cornell grasping dataset, Jacquard grasping dataset and multi-target detection dataset is shown. It can be seen that the grasping quality map obtained by the present invention is clearer and can better segment the grasping target and background, which makes the grasping detection of the present invention less likely to be affected by environmental background factors. At the same time, in multi-target detection, each grasped object can also be clearly distinguished, and the grasping prediction is also relatively accurate. However, GR-CNN cannot detect all grasping targets well, and the width and angle of some grasping rectangles are not suitable for grasping objects; from the detection heat map of GG-CNN, it can be found that its grasping area is very scattered and does not pay attention to the grasping position area.

[0125] The AUBO-I10 robot is used to evaluate the grasping detection method proposed in this invention. The robot is equipped with a parallel gripper with a payload of 10 kg. The zed 2i camera is placed vertically above the workbench to capture RGB images and depth images. The grasping detection method proposed in this invention uses the real-time image information collected by the camera to predict the grasping configuration in the image coordinates, which will then be transferred to the robot world coordinates, and finally the target grasping is achieved by the robot's gripper. The items in the grasping test are all items that have not appeared in the training data set, and they are randomly placed on the workbench. The test scene is as follows Figure 9 As shown in the figure, grasp prediction boxes are transmitted to the robot in descending order of confidence scores. After performing a grasp, the robot returns the grasped object and attempts the next grasp. If the object can be grasped and returned, the grasp is considered successful. Table 3 lists the results of the proposed method and some deep learning-based grasp detection methods. We conducted a total of 300 real-world robot grasp attempts, with 284 successful grasps, for a grasp success rate of 94.7%.

[0126] Table 3 Comparison results of real grasp detection

[0127]

[0128] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0129] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. The preferred embodiments do not describe all details in detail, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A Transformer-based region estimation and multi-level feature fusion grasping and detection method, characterized in that: Here are the steps: The RGB image is detected by the 2D object detector to select the target area that needs to be grasped and detected, and is combined with the depth image of the same object to form a four-channel image as input; The four-channel input is split into multiple non-overlapping patches by the Patch Partition module. Each patch is regarded as a token, representing the concatenation of the original input pixels. The fixed position code is embedded into each token and then passed to the encoder module; In the encoder, the Linear Embedding layer converts each token into a C-dimensional embedding vector, and then the number of tokens is changed through the patch merging + Transformer Block module; The decoder performs the opposite operation of patch merging and recovers the changed number of tokens through the patch expanding + transformer block module. The output features of each level collected by the Encoder are fused through a multi-level feature fusion network; Generate pixel heatmaps of the same size as the detection image to predict grasp locations.

2. The Transformer-based region estimation and multi-level feature fusion capture detection method according to claim 1, characterized in that: The Encoder module collects the feature maps processed by each Encoder sub-block, fuses the feature maps through downsampling and convolution operations, and meets the input specifications of the Decoder module.

3. The Transformer-based region estimation and multi-level feature fusion capture detection method according to claim 2, characterized in that: The fused feature map that meets the input specifications of the Decoder module is processed by the Decoder module, and the grasp prediction module obtains the width of the grasp box related to the grasp position prediction, the sine and cosine components of the grasp box's rotation angle, and the grasp confidence score.

4. The Transformer-based region estimation and multi-level feature fusion capture detection method according to claim 3, characterized in that: The grasp prediction module obtains the width of the grasp box predicted for the grasp position, the sine and cosine components of the grasp box's rotation angle, and the grasp confidence score, which is expressed as follows: Among them, g represents the grasping position predicted by the grasping detection model, g Score 、g Sin 、g Cos 、g width Represents the confidence score of the predicted grasp g, the sin and cos components of the rotation angle, and the width of the grasp prediction box; x n Represents the data input from the grasp detection network to the grasp prediction module; These are the four convolution operations that the crawl detection network performs on the input data.

5. The Transformer-based region estimation and multi-level feature fusion capture detection method according to claim 1, characterized in that: The RGB image is detected by the 2D object detector to select the target area that needs to be captured and detected. The steps are as follows: The RGB image is detected by Faster-RCNN, the position of the grasped object is marked with a rectangular box, and the other backgrounds in the rectangular area are removed and filled with 0.

6. The Transformer-based region estimation and multi-level feature fusion capture detection method according to claim 1, characterized in that: Before the data enters a single Transformerblock, it is normalized using LN and then enters the W-MSA module through the linear mapping matrix W QKV Project the input sequence t to obtain Q, K, and V, which are expressed as follows: [Q,K,V]=tW QKV Where t represents the input sequence, Q represents the query vector, K represents the key vector, and R represents the value vector; The similarity between Q and K is calculated by matrix dot product and weighted with V. The calculation expression of self-attention is as follows: Where Attention(Q,K,V) represents the self-attention score of the input sequence t, SoftMax represents the normalized exponential function, d is the dimension of the vector Q, B represents the learned relative position encoding, and T represents the transposition symbol, which is used to transpose the vector K. After passing through the W-MSA module, the residual connection is used to fuse with the initial input and then normalized. After passing through two linear connection layers (MLP) with GELU, the residual connection is used again to fuse the previous input information and then pass it to the Transformerblock with SW-MSA. The expression is as follows: in, and t l They represent the outputs of the lth module (S) W-MSA module and MLP module, t l-1 Represents the output features of the previous layer.

7. The Transformer-based region estimation and multi-level feature fusion capture detection method according to claim 1, characterized in that: Each pixel in the heat map represents the captured position of that pixel; The output resolution of the decoder is converted to the same size as the image through linear layers and shape rearrangement operations. Four 1x1 convolutions are used to generate four single-channel pixel heat maps corresponding to S, W, sin2θ, and cos2θ of the captured detection. The grasping angle θ is determined, and the final grasp is determined by the highest-scoring grasp in the grasping score graph S.

8. The Transformer-based region estimation and multi-level feature fusion capture detection method according to claim 1, characterized in that: To verify the predicted grasping position, the steps are as follows: When the grasp position G and the ground truth G' both meet the condition |G θ -G' θ |<30° and conditions Recognize that the grasping position G is correct; Conditions|G θ -G' θ |<30° means the angle difference between the predicted grab box and the annotation rectangle is less than 30°; condition Indicates that the intersection-over-union ratio of the predicted grab box and the annotated rectangle is greater than 25%.

Images