A transformer-based grasp pose detection method

Abstract

This invention discloses a Transformer-based grasping pose detection method, belonging to the field of robot recognition technology. The method includes: feeding an acquired input image into an encoder for feature extraction, outputting a feature map; the encoder containing multiple Swing transformation modules; a first decoder upsampling and feature fusion of the feature map; a second decoder decoding the feature map, outputting a grasping quality score map, a width map, and an angle map; constructing a Transformer-based grasping pose detection model; constructing a loss function, treating grasping pose prediction as a regression problem, and training and optimizing the grasping pose detection model by minimizing the distance between the predicted grasping map and the actual grasping map until the model training converges, ultimately obtaining the Transformer-based grasping pose detection model; and using the obtained Transformer-based grasping pose detection model for grasping pose detection. Addressing the problem of low grasping accuracy in existing methods, this invention improves grasping detection accuracy through a combination of Transformer and convolutional neural networks.

Description

Technical Field

[0001] This invention relates to the field of robot recognition technology, and more specifically, to a grasping pose detection method based on Transformer. Background Technology

[0002] Robotic grasping capabilities lag far behind human performance, remaining an unsolved problem in the field of robotics. When humans see new objects, they instinctively know how to grasp them. However, grasping and manipulating remains a challenge for robots.

[0003] The goal of deep learning-based grasping detection models is to identify a suitable grasping configuration from an input image. Currently, predicting the grasping configuration of objects in cluttered or stacked scenes remains a challenging task. First, grasping detection requires not only accurately predicting the object's position but also modeling its pose and contour information to predict the angle and opening distance of the end effector during robot grasping. Second, extracting and mapping the complex and ever-changing features of the robot's working environment is challenging. Since convolutional neural networks are composed of basic convolutional kernel units, which can only extract local features from the input image, the model's ability to acquire global contextual information is insufficient.

[0004] Chinese patent application CN202211126578.2, published on April 4, 2023, discloses a retinal vessel segmentation method based on the fusion of UNet and Transformer, characterized by the following steps: Step 1, preprocessing the image to be trained to obtain a preprocessed image; Step 2, inputting the preprocessed image into a retinal vessel segmentation model based on the fusion of UNet and Transformer to obtain a weight file. This model includes an encoder, a first decoder, and a fusion attention mechanism. The encoder includes multi-stream cascaded convolutional layers, multiple pooling layers, and multiple residual modules. Each convolutional layer... Using residual modules, pooling layers are placed between two convolutional units; the first decoder includes multiple cascaded improved extended convolution-based residual modules, multiple upsampling modules, and deconvolutional layers, with upsampling modules placed between two adjacent improved extended convolution-based residual units; the fusion attention mechanism uses the output of the pooling layer in the encoder and the output of the adjacent flowing pooling layer as low-level feature input and high-level feature input, respectively. The high-level feature input of the fusion attention mechanism in the third layer is the feature map formed by the corresponding encoder pooling layer through the Transformer module; step 3, load the weight file, input the tested fundus image into the model to obtain the retinal vessel segmentation result. However, this scheme has at least the following technical problems:

[0005] First, while UNet and Transformer excel in feature extraction and feature fusion respectively, their combination fails to effectively capture detailed information, resulting in low grasping detection accuracy. Second, the design of the convolutional layers, pooling layers, and residual modules used in the encoder and first decoder is ill-suited to complex structures. The encoder and first decoder structures in this scheme cannot adequately adapt to complexity, leading to low grasping detection accuracy. Furthermore, although the fusion attention mechanism introduces the outputs of the pooling layers in the encoder and the outputs of adjacent flow pooling layers as inputs to low-level and high-level features, the Transformer module is only used in the third-layer fusion attention mechanism. This may limit the model's effective fusion and utilization of features at different levels, thus affecting the accuracy of grasping detection. Summary of the Invention

[0006] 1. Technical problems to be solved

[0007] To address the problem of low grasping accuracy in existing methods, this invention proposes a grasping pose detection method based on Transformer. By combining Transformer and convolutional neural networks, the accuracy and environmental adaptability of grasping detection are improved.

[0008] 2. Technical Solution

[0009] The objective of this invention is achieved through the following technical solutions.

[0010] This specification provides a Transformer-based grasping pose detection method, comprising: acquiring an input image; feeding the acquired input image into an encoder for feature extraction, outputting a feature map, wherein the encoder includes multiple Swing transformation modules; using the feature map output by the encoder as input to a first decoder, wherein the first decoder upsamples and fuses the feature map; using the feature map output by the first decoder as input to a second decoder, wherein the second decoder decodes the feature map and outputs a grasping quality score map, a width map, and an angle map; constructing a Transformer-based grasping pose detection model, comprising an encoder, a first decoder, and a second decoder; constructing a loss function, treating grasping pose prediction as a regression problem, and training and optimizing the grasping pose detection model by minimizing the distance between the predicted grasping map and the real grasping map; repeating all steps until the grasping pose detection model training converges, ultimately obtaining the Transformer-based grasping pose detection model; and using the obtained Transformer-based grasping pose detection model for grasping pose detection.

[0011] Further, the acquired input image is fed into the encoder for feature extraction, and a feature map is output. The encoder contains multiple Swin transformation modules, including: feeding the acquired input image into the encoder for feature extraction; the encoder is composed of multiple stacked Swin transformation modules; each Swin transformation module contains a multilayer perceptron (MLP), a normalization layer (Layer Norm), a window-based multi-head self-attention mechanism, and a shift-window multi-head self-attention mechanism; the encoder maps the input features to a query Q, a key K, and a value V through linear transformation; the attention mechanism obtains attention weights by calculating the similarity between the query Q and the key K through dot product; after multiplying the attention weights with the value V, the result is normalized and processed by multiple heads to obtain the encoder's output feature map; among them, the window-based multi-head self-attention mechanism establishes global context information; the shift-window multi-head self-attention mechanism balances local and global information for environmental perception.

[0012] Furthermore, the query Q, key K, and value V are calculated using the following formula:

[0013] Q = XW Q

[0014] K = XW K

[0015] V = XW V

[0016] Among them, W Q W K W V Let Q, K, and V represent the linear projection matrices used to generate Q, K, and V, respectively, and X be the input feature map.

[0017] Furthermore, the attention mechanism obtains attention weights by calculating the similarity between query Q and key K through a dot product. The dot product is calculated using the following formula:

[0018]

[0019] Where, d K K is the channel dimension; B is the bias term, which contains context-learnable parameters; T is the transpose of the matrix.

[0020] Furthermore, the feature map output by the encoder is used as the input to the first decoder. The first decoder performs upsampling and feature fusion on the feature map, including: using the output feature map of the encoder as the input to the first decoder; the first decoder adopts a fully convolutional neural network structure to perform upsampling and feature fusion on the feature map output by the encoder; the first decoder restores the size of the feature map output by the encoder to the size of the original input image or close to the original size through convolutional layers and upsampling operations; the first decoder performs feature fusion operations, using multi-scale information to fuse feature maps of different scales; the first decoder generates a feature map with the same size as the input image, and the feature map contains the object's grasping pose information.

[0021] Furthermore, feature map fusion includes: feature fusion employs skip connections or attention mechanisms to fuse information at different scales.

[0022] Furthermore, the feature map output by the first decoder is used as the input to the second decoder. The second decoder decodes the feature map and outputs a grasping quality score map, a width map, and an angle map. This includes: inputting the feature map output by the first decoder into the second decoder for decoding to generate the grasping quality score map, the width map, and the angle map; the decoding process maps the feature map to the representation space of the grasping pose, where each pixel in the feature map corresponds to the corresponding grasping pose; obtaining the detected grasping pose information by outputting the planar grasping pose on the original image; and capturing a two-dimensional image G using the quality score map, the width map, and the angle map generated by the grasping process; wherein the grasping pose is represented by a 5-dimensional grasping g.

[0023] Furthermore, the 5D grasping g is calculated using the following formula:

[0024] g = {x, y, θ, w, h}

[0025] Where x and y are the coordinates of the center point of the grasp rectangle in the grasp pose, θ is the rotation angle of the grasp rectangle relative to the horizontal direction of the image, w is the width of the grasp rectangle, and h is the height of the grasp rectangle.

[0026] Furthermore, the two-dimensional image G is calculated using the following formula:

[0027] G={Q,W,θ}∈R3×D×H

[0028] Where Q represents the capture quality score image for each pixel, W and θ represent the capture width and angle images for each pixel; R is a matrix, D is the width of the input image, and H is the height of the input image.

[0029] Furthermore, a loss function is constructed, treating the grasping pose prediction as a regression problem. The grasping pose detection model is trained and optimized by minimizing the distance between the predicted and ground truth grasping images. This process includes: inputting an image and extracting feature maps using a Transformer-based grasping pose detection model; generating a predicted grasping image through convolution operations on the feature maps; comparing the predicted and ground truth grasping images and calculating the distance between them; using this distance as part of the loss function to transform the grasping pose prediction into a regression function; updating the weights and parameters of the grasping pose detection model using backpropagation by minimizing the loss function; and repeating all the above steps until the model converges and reaches the predetermined training objective. Here, the predicted grasping image represents the distribution of possible grasping positions and orientations on the image; the ground truth grasping image represents the labeled distribution of actual grasping positions and orientations.

[0030] 3. Beneficial effects

[0031] Compared with the prior art, the advantages of this invention are:

[0032] (1) Transformer-based encoder-first decoder architecture: The proposed model adopts a Transformer-based architecture, including an encoder, a first decoder, and a second decoder. The encoder consists of multiple Swin transformation modules used to extract features from the input image. The first decoder reconstructs, upsamples, and fuses the features to generate a feature map containing grasping pose information. This architecture can efficiently and effectively extract and decode features, achieving accurate grasping pose detection;

[0033] (2) The first decoder employs multi-scale feature fusion to capture information at different resolutions. Multi-scale feature fusion is achieved through skip connections or attention mechanisms, enabling the model to effectively detect grasping poses of objects of different sizes and postures, thereby enhancing the model's robustness and generalization ability;

[0034] (3) The model is trained using a regression loss function to predict the grasping pose. The predicted grasping image is compared with the actual grasping image, and the distance between them is minimized using techniques such as mean squared error or mean absolute error. By modeling grasping pose detection as a regression problem, the model can learn to accurately estimate the coordinates, width, and angle of the grasping center. Through iterative training until the model converges, the grasping pose detection model is finally optimized.

[0035] In summary, this invention proposes a grasping pose detection method based on Transformer. By combining Transformer and convolutional neural network, the model can not only have a strong ability to acquire global context information, but also have the inductive bias ability of convolutional neural network, thereby improving the accuracy of grasping detection and environmental adaptability, and further improving the success rate of the robot when performing grasping tasks. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the Transformer-based grasping pose detection method of the present invention.

[0037] Figure 2 This is a schematic diagram of the data flow of the Swing Transformer Block in this invention;

[0038] Figure 3 This is a diagram of the fully convolutional network structure of the present invention;

[0039] Figure 4 This is a diagram of the Swing Transformer Block network structure of the present invention;

[0040] Figure 5 The present invention provides a quality scoring heatmap, a width heatmap, and an angle heatmap for single-target grasping, as well as a grasping image.

[0041] Figure 6 The present invention provides a quality scoring heatmap, a width heatmap, and an angle heatmap for multi-target grasping, as well as a grasping image. Detailed Implementation

[0042] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0043] This embodiment presents a Transformer-based grasping pose detection method, relating to the field of robot grasping detection. The model comprises three parts: an encoder, a first decoder, and a second decoder. The encoder, constructed using a Transformer attention mechanism, extracts features from the input multi-channel image. The first decoder, composed of a fully convolutional neural network, upsamples and concatenates the multi-scale feature maps input from the encoder to fully fuse multi-scale feature information. The second decoder transforms the feature maps obtained from the first decoder into the width, angle, and grasping quality score maps required for pixel-level grasping position. This method combines the Transformer attention mechanism with convolutional neural networks, enabling the model to possess both a powerful ability to acquire global contextual information and the inductive bias capability of convolutional neural networks. Experimental verification demonstrates that the proposed grasping detection model can be widely applied in unstructured environments (excluding structured environments), showcasing its stability and generalization ability.

[0044] A Transformer-based grasping pose detection method includes the following technical steps:

[0045] Preprocessing of the S100 crawl detection dataset: In this step, the crawl detection dataset (e.g., the Cornell crawl detection dataset and the Jacquard dataset) is first preprocessed. This may include data cleaning, transformation, and annotation to ensure the suitability and consistency of the dataset.

[0046] The S200 grabbing pose detection model based on Transformer is established in this step. This model uses Transformer and other technologies, through an encoder and first decoder architecture, attention mechanism and other methods, to extract key features from the input image and predict the grabbing pose.

[0047] The S300 uses an RGB-D camera to photograph household items in Test Set 2. By using the RGB-D camera, household items such as screwdrivers, scissors, toothpaste, double-sided tape, and paper cups are photographed in different scenes and from different perspectives, serving as Test Set 2. This simulates grasping situations in real-world applications.

[0048] The S400 uses the Cornell and Jacquard datasets for model training and testing. These datasets contain samples of different objects, poses, and grasping positions. By training the model and evaluating it on the test set, the model's grasping accuracy and precision can be assessed.

[0049] Through the above technical steps, the Transformer-based grasping pose detection method can preprocess the grasping detection dataset and train and test the model. Using a real-world test set (2), the model's performance can be better validated. This method comprehensively considers multiple key steps, from data preprocessing to model building and evaluation, to improve grasping accuracy and precision.

[0050] S100 includes the following specific steps to process the grasping datasets (Cornell grasping detection dataset and Jacquard grasping detection dataset) to improve grasping accuracy: S110 uses publicly available grasping pose detection datasets, such as the Cornell grasping detection dataset and the Jacquard grasping detection dataset. These datasets contain common objects in the living environment and provide annotations for grasping bounding boxes. S120 involves preprocessing the dataset to increase data diversity and robustness, thereby improving the accuracy and generalization ability of grasping pose detection. Specific preprocessing operations include: random cropping and stitching, randomly cropping and stitching images to increase the number of grasping object samples of different sizes and positions; image scaling, arbitrarily scaling images to simulate grasping scenarios at different distances; random rotation, randomly rotating images to introduce grasping object samples at different angles; and horizontal or vertical flipping, horizontally or vertically flipping images to increase mirrored samples. Image blurring is applied to the image to simulate blurring conditions in real-world scenes. Random color and brightness adjustments are made to the image to increase the number of object samples captured under different brightness conditions. Finally, the pre-processed image is resized to a uniform 224×224 pixels to meet the model's input requirements.

[0051] Through the specific operations in step S100 above, the crawling dataset has been diversified, resulting in a richer and more diverse collection of crawlable object samples. This improves the model's training performance, enabling it to have better generalization ability and crawling detection accuracy under different scenarios and conditions.

[0052] Figure 1 This is a flowchart illustrating the Transformer-based grasping pose detection method of the present invention, as shown below. Figure 1 As shown Figure 1 As shown, the Transformer-based grasping pose detection method uses an encoder, a first decoder, and a second decoder structure.

[0053] The grasping pose detection method based on Transformer in this embodiment includes the following steps in step S200:

[0054] The S210 encoder is the core component of the entire method, consisting of a Transformer attention mechanism, with the smallest unit being the Swing Transformer Block. The encoder is responsible for feature extraction and representation learning from the input image.

[0055] Swin refers to the "Swin Transformer," a convolutional neural network architecture based on the Transformer structure. Proposed by Microsoft Research, the Swin Transformer is a novel Transformer architecture designed to achieve better performance in computer vision tasks. The core idea of ​​the Swin Transformer is to segment an image into small, non-overlapping patches and apply a Transformer structure to each patch. Compared to traditional global self-attention mechanisms, this local attention mechanism significantly reduces computational complexity and provides better scalability. The Swin Transformer utilizes a window-level cross-window attention mechanism to establish connections between windows, enabling the network to better capture global information. A Swin Transformer Block refers to the basic unit that constitutes the Swin Transformer network.

[0056] Swin consists of the following key components: Input Embedding: The Swin Transformer first performs linear projection and reshaping operations on the input image to generate an input embedding matrix. Local Attention Block: The Swin Transformer uses a local attention mechanism to process local features of the image. The local attention block consists of multiple Swin Transformer layers, each with a multi-head self-attention mechanism and a feedforward neural network. Windowed Cross Attention: The Swin Transformer introduces a window-level cross-window attention mechanism to establish associations and interactions between different windows. Output Embedding: Finally, the Swin Transformer performs an output embedding operation on the features processed by multiple local attention blocks to obtain the final feature representation.

[0057] By using the Swin Transformer Block, the Swin Transformer can effectively process images in computer vision tasks, achieving excellent performance in tasks such as image classification, object detection, and semantic segmentation. Its innovation lies in combining the Transformer's self-attention and local attention mechanisms, as well as window-level cross-window association, thereby significantly reducing computational complexity and improving computational efficiency while maintaining model performance.

[0058] The Transformer attention mechanism, which uses the Swing Transformer Block as its smallest unit, helps capture fine-grained features and local contextual information in images. This property is crucial for grasping pose detection, as the accuracy of grasping pose detection depends on sensitivity to details such as object edges and textures.

[0059] The S220's first decoder consists of a fully convolutional neural network (WCNN). Its main task is to upsample and fuse the multi-scale feature maps output by the encoder. This allows for the full integration of information at different scales, enabling the model to simultaneously perceive both the global context and local details of an object. Feature fusion aims to synthesize information from various scales and reduce the resolution loss of the feature maps. Through the processing of the WCNN, the first decoder generates feature maps with rich semantic information, providing a more accurate feature representation for further decoding and capturing poses.

[0060] The S230 second decoder decodes the fused feature map into a grasping quality score map, a width map, and an angle map, and outputs a planar grasping pose on the original image. The decoder's role is to map high-dimensional features into grasping-related prediction results. In this embodiment, the second decoder is a grasping decoder. By using a Transformer encoder and a multi-layer decoder structure, the method of this invention can fully utilize the multi-scale information and contextual relationships of image features to improve the accuracy of grasping pose detection. Simultaneously, the Swin Transformer Block, as the smallest unit of the encoder, can capture important information about image details and local context, further enhancing the model's grasping pose detection capability.

[0061] In summary, Figure 1 The Transformer-based grasping pose detection method demonstrated in the paper, through the structure of an encoder, a first decoder, and a second decoder, and by using key components such as the Swing Transformer Block, helps to improve the detection accuracy of grasping pose from the perspectives of multi-scale feature fusion and local contextual feature extraction.

[0062] Figure 2This is a schematic diagram of the data flow of the Swing Transformer Block of the present invention, as shown below. Figure 2 As shown, the S210 encoder consists of a Transformer attention mechanism, with the smallest unit being the Swing Transformer Block. This encoder processes the input multi-channel image and segments it into non-overlapping patches. Each patch is encoded by the Swing Transformer Block, a specific Transformer attention mechanism that captures local features when processing patches. Through the encoder, the Swing Transformer Block extracts local feature information from each patch. It then fuses this local feature information with the global context to obtain a more comprehensive feature representation. The fused feature information is fed into the decoder, which is responsible for reconstructing the grasping pose information from the feature representation. Throughout this process, the Swing Transformer Block plays a crucial role, effectively encoding local features while simultaneously utilizing the global context for context awareness. This comprehensive feature representation helps improve grasping accuracy, enabling the model to predict grasping poses more accurately.

[0063] In summary, Figure 2 The diagram showing the data flow of the Swing Transformer Block illustrates how the encoder uses the Transformer attention mechanism to improve capture accuracy by cutting the image into patches and fusing local features with global context.

[0064] Figure 3 This is a diagram of the fully convolutional network structure of the present invention, as shown below. Figure 3 As shown, the first decoder, S220, consists of a fully convolutional neural network used to upsample and fuse the multi-scale feature maps input from the encoder, thereby fully integrating multi-scale information. Finally, the fused feature map is passed to the second decoder, S230. The task of the second decoder, S230, is to decode the fused feature map into a grasping quality score map, a width map, and an angle map, and output a planar grasping pose on the original image. This step is the final output of the grasping pose detection and provides information about grasping quality, grasping width, and grasping angle.

[0065] In step S210, the Transformer attention mechanism is composed of the smallest unit, the Swing Transformer Block. This Swing Transformer Block includes a Multilayer Perceptron (MLP), a Layer Norm, a window-based multi-head self-attention mechanism, and a shift-window multi-head self-attention mechanism. Through these attention mechanisms, the Transformer can maintain a balance between local and global information, thus improving its environmental perception.

[0066] The S230 second decoder decodes the fused feature map into a grasping quality score map, a width map, and an angle map, and outputs the planar grasping pose on the original image. For the case where the end effector is a parallel plate gripper, a 5-dimensional grasping representation g is used:

[0067] g = {x, y, θ, w, h}

[0068] In the formula, x and y are the coordinates of the center point of the grasping rectangle in the grasping pose, θ is the rotation angle of the grasping rectangle relative to the horizontal direction of the image, w is the width of the grasping rectangle, and h is the height of the grasping rectangle. The center coordinates (x, y) represent the position of the grasping rectangle in the image. The advantage of choosing the center coordinates as a parameter is that it can accurately determine the grasping position and is applicable to objects of different sizes, shapes, and orientations. By accurately determining the grasping position, grasping errors can be reduced and grasping accuracy can be improved.

[0069] The angle between the grasping rectangle and the horizontal axis represents the grasping direction. The angle is chosen as a parameter because it describes the grasping posture. By controlling the angle, the gripper can maintain appropriate contact force with the object's surface, thereby increasing grasping stability. Furthermore, the angle can be used to handle objects with different postures and adjust the gripper's direction to adapt to different scenarios.

[0070] The width and height (w, h) of the grab rectangle represent the area covered by the grabber on the object. Choosing width and height as parameters allows the grabber to be adjusted to fit different object sizes. Properly adjusting the grabber size increases the success rate of the grab and reduces damage to the object.

[0071] Taking all factors into consideration, selecting the (x,y) center coordinates, the included angle, and the (w,h) width and height as the representation parameters for grasping can improve grasping accuracy. These parameters simultaneously take into account the grasping position, direction, and size, enabling the gripper to more accurately locate and grasp objects. By appropriately selecting these parameters, more stable, accurate, and reliable grasping operations can be achieved.

[0072] To achieve pixel-level capture, we represent 2D image capture as G, as follows:

[0073] G={Q,W,θ}∈R3×D×H

[0074] In the formula, Q represents the grasping quality score for each pixel, and W and θ represent the grasping width and orientation angle for each pixel. The grasping quality score (Q) is used to evaluate the suitable grasping performance for each pixel. By calculating and analyzing the Q value of each pixel in the image, we can determine which areas have higher grasping potential. This allows us to select the optimal grasping location for precise grasping operations. Selecting pixels with high grasping quality scores as grasping targets can improve the success rate and stability of grasping.

[0075] Grip width (D) refers to the width of the contact area between the gripper or tool and the target object during the gripping process. Selecting the grip width as a parameter allows adjustment of the opening and closing degree of the gripper or tool within a certain range to accommodate objects of different sizes. By adjusting the grip width according to the size and shape of the target object, a tighter gripping contact can be achieved, increasing gripping accuracy and stability.

[0076] The orientation angle map (θ) represents the gripping direction or posture of each pixel. Choosing the orientation angle as a parameter aims to ensure proper contact between the gripper and the target object's surface. By controlling the orientation angle, the gripper's orientation can be adjusted to maintain appropriate contact force with the target object. This helps reduce tilt between the gripper and the object, and improves gripping stability and accuracy.

[0077] These parameters were chosen to maximize the accuracy and success rate of the grasping system. By comprehensively considering the grasping quality score, grasping width, and orientation angle, we can select the optimal grasping position, posture, and opening degree, thereby achieving pixel-level grasping accuracy. This will enable the robot to effectively grasp objects of various shapes, sizes, and postures, improving the overall performance and efficiency of the grasping system.

[0078] We treat grasping pose estimation as a regression problem and use our grasping detection model to learn the mapping. The loss function is defined as follows: by minimizing the distance between the predicted values ​​(grasping heatmap) and the labeled ground truth values.

[0079]

[0080] In the formula, N is the number of samples; G is the predicted value; and L is the labeled ground truth. The predicted value (grabbing heatmap) G is a matrix of the same size as the input image, representing the probability of grabbing at different locations. The value of each pixel represents the grabbing probability score for that location. The purpose of choosing the predicted value (grabbing heatmap) as the predicted value is to provide confidence information about grabbing at different locations. By minimizing the distance between the predicted value (grabbing heatmap) and the labeled ground truth, we can enable the model to learn to accurately predict the grabbing probability for different locations.

[0081] Ground truth (L): Ground truth is a matrix of the same size as the input image, representing the correct grasping pose at each location. Ground truth can be obtained through manual annotation or other pose estimation methods. The purpose of choosing ground truth as the target value is to provide accurate grasping pose information for model training. By minimizing the distance between the predicted values ​​(grasping heatmap) and the ground truth, we can enable the model to learn to accurately predict the actual grasping pose.

[0082] By selecting these parameters and defining the corresponding loss function, we can train a model that accurately predicts grasping poses. By minimizing the distance between the predicted values ​​and the labeled ground truth values, we enable the model to learn the mapping relationship between the input image and the grasping pose, thereby improving grasping accuracy. This allows the robot to perform precise grasping operations on different scenes and objects, improving the accuracy and stability of the grasping system.

[0083] The prediction is considered correct if the following conditions are met:

[0084] The difference between the predicted grab angle and the actual labeled angle shall not exceed 30°; the intersection-union ratio (IOU) of the area between the predicted grab rectangle and the actual labeled rectangle shall not be less than 0.25.

[0085] The difference in grasping angle should not exceed 30°. The grasping angle refers to the direction or orientation of the grasping posture. By limiting the difference between the predicted grasping angle and the actual labeled value to no more than 30°, we can ensure that the predicted grasping direction is within a reasonable range, reducing the impact of errors. This limitation is based on considerations in actual grasping tasks; excessive angle differences may cause the robot to misalign or fail to grasp the object correctly.

[0086] The Intersection over Union (IOU) ratio must be at least 0.25. IOU is a metric used to evaluate the degree of overlap between two bounding boxes. By comparing the predicted grab bounding box with the labeled ground truth bounding box, a minimum IOU of 0.25 ensures that there is some overlap between the predicted and ground truth bounding boxes. This constraint helps reduce the error between the predicted and ground truth bounding boxes, ensuring that the predicted grab location matches the actual object location.

[0087] Error control of the grasping angle, limiting the difference between the predicted grasping angle and the actual value to within 30°, is to control errors in the grasping direction. Generally, a robot needs to accurately locate and grasp an object during grasping; however, excessive angle differences may cause the robot to deviate from the object's surface during grasping, resulting in grasping failure. By limiting the angle difference, we ensure that the robot can grasp more accurately based on the predicted grasping angle, improving the accuracy and success rate of grasping.

[0088] The area intersection-union ratio (IOU) constraint is crucial. IOU is a commonly used metric to measure the degree of overlap between two bounding boxes. Ensuring the IOU between the predicted grasping bounding box and the labeled ground truth bounding box is no less than 0.25 ensures a certain overlap between them. This reduces grasping bias, ensuring the robot can accurately grasp objects and avoids unnecessary contact or damage during the grasping process.

[0089] By selecting these parameters and their values, we considered key factors such as the grasping angle and object position when evaluating the prediction results. These constraints help us improve grasping accuracy, enabling the robot to perform grasping tasks more accurately and improving the reliability and success rate of the grasping system. Furthermore, the values ​​of these parameters can be adjusted according to specific applications and scenarios to meet the accuracy requirements of different tasks and needs.

[0090] Improving the accuracy of grasping and positioning is crucial. Precise planar coordinate information helps the robot accurately locate the target object, reducing the impact of positional deviations and enabling the robot to locate the object with virtually no error. Optimizing the selection of the grasping direction is also essential. Precise grasping angle information allows the robot to grasp according to the required orientation, reducing grasping failures caused by angle deviations and increasing the success rate. Ensuring appropriate contact force and stability is vital. Precise grasping width information allows the robot to adjust the distance of the gripper, ensuring appropriate contact force with the object, reducing grasping failures caused by insufficient or excessive gripping force, and improving grasping stability. Therefore, providing information such as the planar coordinates of the grasping pose, grasping angle, and grasping width is crucial for improving grasping accuracy. This information enables the robot to perform grasping operations more accurately, adapting to objects of different positions, postures, and sizes, thus improving the success rate and stability of grasping.

[0091] In summary, the fully convolutional network structure, through the cooperation of the S220 first decoder and the S230 second decoder, can appropriately upsample and fuse the multi-scale features extracted by the encoder, and finally output grasping pose information including grasping quality score map, width map, and angle map. The use of a Transformer attention mechanism based on the Swin Transformer Block helps maintain the balance between local and global information, further improving grasping accuracy. Finally, choosing a 5-dimensional grasping representation g is beneficial for accurately describing the grasping behavior of the parallel plate gripper.

[0092] Figure 4 This is a diagram of the Swing Transformer Block network structure of the present invention, as shown below. Figure 4 As shown. The calculation process of the SwinTransformer Block is as follows:

[0093] X2 = W - MSA(LN(X1)) + X1

[0094] X3 = MLP(LN(X2)) + X2

[0095] X4 = SW - MSA(LN(X3)) + X3

[0096] X5 = MLP(LN(X4)) + X4

[0097] In the formula, X1 is the input image of the Swing Transformer Block, X2, X3 and X4 are feature maps of the process stage, X5 is the output feature map of the Swing Transformer Block; W-MSA and SW-MSA are local window multi-head self-attention and global shift window multi-head self-attention, respectively; MLP is multilayer perceptron, and LN is layer normalization.

[0098] Specifically, W-MSA (Local Window Multi-Head Self-Attention) is a technique that introduces the concept of local windows to focus on the parts of the input sequence adjacent to the target location. Compared to traditional global self-attention mechanisms, it can learn the correlations between different locations in the sequence more finely. In the grasping task, W-MSA's technical effects are as follows: It provides more accurate local feature representations: By focusing on information within the local window, W-MSA can better capture the context and correlations around the target object, thus providing more accurate and refined feature representations. It improves the accuracy of location attention: The introduction of local windows allows the model to more precisely focus on the parts adjacent to the target location, thereby improving the model's accuracy in locating the target.

[0099] SW-MSA (Global Shifted Window Multi-Head Self-Attention) introduces the concept of shifted windows, enabling global contextual association learning within a self-attention mechanism. It provides more global association information by allowing positional interactions between different windows. In grasping tasks, SW-MSA offers the following advantages: Global contextual association learning: SW-MSA captures global dependencies between positions in the input sequence, allowing the model to improve grasping accuracy by learning global contextual information. Optimized long-range dependency modeling: For longer sequences, SW-MSA captures internal long-range dependencies, better modeling the associations between different positions in the sequence and improving grasping accuracy. The choice between W-MSA and SW-MSA depends on task requirements and data characteristics. For tasks requiring more attention to local context, such as precise localization in grasping tasks, W-MSA may be more suitable. For tasks requiring more global contextual association and long-range dependency modeling, SW-MSA may be more suitable. These choices are based on an understanding of task characteristics and an evaluation of experimental results.

[0100] In summary, W-MSA and SW-MSA, as variants of self-attention mechanisms, provide fine-grained feature representations and global contextual learning by focusing on local or global shifted windows, thereby contributing to improved crawling accuracy. Choosing appropriate parameters depends on task requirements and data characteristics to achieve optimal crawling performance.

[0101] The encoder consists of stacked identical transformer blocks. Attention within the Swin Transformer Block establishes long-range interactions between distant pixels and focuses on these locations in the embedding space. A linear transformation is performed on the input features X to obtain the query Q, key K, and value V, defined as follows:

[0102] Q = XW Q

[0103] K = XW K

[0104] V = XW V

[0105] In the formula, WQ, WK, and WV represent the linear projection matrices used to generate Q, K, and V, respectively, and X is the input feature map. Long-range interactions are achieved between pixels through an attention mechanism within the Swing Transformer block. This allows the model to capture associations between distant pixels and provides broader contextual information when modeling long-range dependencies in sequences or images. For grasping tasks, this long-range interaction helps the model understand the relationship between the target object and its surrounding environment, thereby improving grasping accuracy.

[0106] Spatial attention, in the Swin Transformer block, captures the relevance of features by paying attention to different locations in the embedding space. By calculating the similarity between the query Q and the key K, the model can determine which locations are relevant. By applying attention weights to the value V, the model can obtain a feature representation corresponding to the query. This spatial attention helps the model focus on regions relevant to the actual grasping task and provides more accurate and fine-grained feature representations.

[0107] Swin Transformer's attention mechanism and linear projection matrix provide a flexible and tunable way to model the relationships between input features and learn the feature representations required for the task. Through long-distance interactions and spatial attention, Swin Transformer can better capture the correlations between features, improving grasping accuracy.

[0108] Furthermore, the Swing Transformer has proven excellent performance in many computer vision tasks and has achieved good results in practice. Therefore, choosing to use the Swing Transformer block and related parameters can achieve good technical results in grasping accuracy. The multi-head attention mechanism obtains attention by calculating the similarity between the query Q and the key K through a dot product, the definition of which is as follows:

[0109]

[0110] In the formula, d K K represents the channel dimension; B is the bias term, containing context-learnable parameters; T is the transpose of the matrix. The Softmax function is used to convert similarity into an attention distribution. Similarity calculation: The similarity between the query and the key is calculated through a dot product. The multi-head attention mechanism can quantify the degree of association between input features. This allows the attention distribution to better reflect the correlation between important features, thus providing a more accurate feature representation.

[0111] Multi-head attention mechanisms compute different linear projections in parallel, generating multiple attention matrices, which are then merged. This allows the model to focus on different aspects within different representation spaces and extract rich feature information from multiple representation subspaces. This helps the model better understand and capture complex relationships in grasping tasks, thereby improving grasping accuracy.

[0112] The parameters for using a multi-head attention mechanism were chosen because of its advantages: Modeling capability: The multi-head mechanism allows the model to simultaneously focus on the correlations between different feature subspaces. This helps capture the diverse relationships between features, providing richer and more comprehensive feature representations, thereby improving grasping accuracy. Interpretability: Through attention weights, the multi-head attention mechanism can also provide the degree of attention given to different features. This makes the model's decision-making process more interpretable, helping to understand the model's attention focus and decision-making rationale in the grasping task. Parallel computation: The multi-head mechanism can compute multiple attention matrices in parallel, thereby reducing computational complexity. This is especially important for large models and large datasets, enabling more efficient training and inference.

[0113] The multi-head attention mechanism was chosen in the Swin Transformer encoder because it can better capture the correlation between input features and provide rich feature representations through dot product similarity calculation and multi-head parallel computation, which has a positive technical effect on improving the capture accuracy.

[0114] The Transformer-based grasping pose detection method in this embodiment includes the following steps in step S400: Step S410 involves training on the Cornell grasping detection dataset and the Jacquard grasping detection dataset. First, the datasets are randomly divided into training, validation, and test sets in an 8:1:1 ratio. Then, the training set is preprocessed to prepare the input data. This embodiment uses the Python 3.7 programming language, performs training and validation on the Windows 10 platform, and uses the PyTorch deep learning framework. During training, RGB and depth images are used as input, the batch size is set to 16, and training continues until the model converges.

[0115] First, the S420 tests the trained pixel-level grasping pose detection model using a test set. Then, it further tests the model using a second test set captured by an RGB-D camera to verify whether the model meets the requirements of the grasping detection task and to evaluate the model's robustness and environmental adaptability.

[0116] This embodiment uses Transformer as the base model. Transformer is a powerful attention mechanism model suitable for sequence data processing. Its advantage lies in its ability to capture long-range dependencies and global information, making it suitable for detecting and predicting grasping poses.

[0117] By using RGB and depth images as input, the model can simultaneously utilize color and depth information for grasping pose detection. This multimodal input provides richer feature representations, enhancing the model's understanding of object shape, texture, and spatial structure, thereby improving grasping accuracy. Employing a pixel-level grasping pose detection method enables precise localization of the object's grasping point. The Transformer model, through a self-attention mechanism, can identify interdependent key features in the input sequence, accurately predicting the object's grasping pose. Evaluation during the training and testing phases validates the effectiveness and accuracy of this technique in grasping detection tasks. Furthermore, validation using the Jacquard grasping detection dataset and other test sets helps assess the model's robustness and environmental adaptability. The model can generalize across different datasets and scenarios, adapting to various object shapes, sizes, and environmental conditions, improving its practicality and stability.

[0118] In summary, this Transformer-based grasping pose detection method improves grasping accuracy and model robustness by utilizing multimodal input and self-attention mechanisms. Furthermore, detailed training and testing procedures, along with validation using different datasets, ensure the model's reliability and adaptability in real-world applications.

[0119] Figure 5 The quality scoring heatmap, width heatmap, and angle heatmap for single-target grasping in this invention, as well as the grasping image, are as follows: Figure 5 The image shows the prediction results of the Transformer-based grasping pose detection method for a single target object. The prediction results demonstrate that the grasping quality of this invention for single target objects is highly accurate. The Transformer-based grasping pose detection method utilizes the attention mechanism and multi-layer autoencoder structure of the Transformer model to extract key features of the object and predict the grasping pose. This method has significant advantages over traditional methods based on manual feature engineering.

[0120] First, the Transformer model can capture long-range dependencies in the input data. In grasping pose detection tasks, factors such as the object's shape, pose, and surrounding environment can affect the grasping quality. By modeling global dependencies, the Transformer model can extract features closely related to the grasping pose and can adaptively weight the importance of different features through an attention mechanism.

[0121] Secondly, during the training phase, the Transformer-based model utilized a large amount of training data, including the Cornell grasp detection dataset and the Jacquard grasp detection dataset. These datasets cover a wide range of object shapes, sizes, and grasping poses, enabling the model to learn a broad range of grasping scenarios and generalize. Through training on large-scale data, the model can better understand the representation of objects and accurately predict grasping poses.

[0122] In addition, regarding input, this invention uses RGB and depth images as input to the model. RGB images provide color and texture information, while depth images provide the object's geometric structure and depth information. This multimodal input can provide richer feature representations, helping the model better understand the object's visual features and spatial layout, thereby improving grasping accuracy.

[0123] In summary, the Transformer-based grasping pose detection method excels in extracting key features, modeling long-range dependencies, and predicting grasping poses by leveraging the attention mechanism of the Transformer model, large-scale training data, and multimodal input. This technique improves the accuracy and stability of grasping, providing precise prediction results for single-object grasping tasks.

[0124] Figure 6 The quality scoring heatmap, width heatmap, and angle heatmap for multi-target grasping in this invention, as well as the grasping image, are as follows: Figure 6 As shown, Figure 6 As shown, the prediction results of the grasping pose detection method based on Transformer of the present invention for multiple target objects are as follows. It can be seen from the prediction results that the grasping quality of the present invention for multiple target objects is also very accurate.

[0125] Multi-target attention mechanism: To handle multiple target objects, Transformer-based methods can introduce a multi-target attention mechanism. This mechanism helps the model identify and distinguish different targets in the input, ensuring that each target receives appropriate attention weights. Through detailed modeling of different targets, the model can more accurately predict the grasping pose of each target.

[0126] Multi-task learning, based on Transformer methods, enables multi-task learning, simultaneously predicting the grasping poses of multiple targets. By sharing network layers and parameters, the model can learn the similarities and differences between different targets. This joint learning approach improves the model's generalization ability, allowing it to better adapt to various situations in multi-target grasping. Data augmentation and training strategies also play a crucial role in improving grasping accuracy when handling multi-target grasping. Introducing augmentation methods such as random transformations, rotations, and translations can increase the diversity of training data, helping the model better understand the poses and shapes of different targets. Furthermore, in training strategies, a stepwise training approach can be adopted, starting with simple targets and gradually introducing more complex targets to progressively improve the model's adaptability to multi-target grasping.

[0127] In summary, Transformer-based grasping pose detection methods can employ techniques such as multi-target attention mechanisms, multi-task learning, data augmentation, and training strategies to improve grasping accuracy when handling multi-target grasping. These techniques can help the model more accurately predict the grasping poses of multiple targets, thereby improving the quality and accuracy of multi-target grasping.

[0128] The invention and its embodiments have been described above illustratively. This description is not restrictive, and the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. The accompanying drawings are only one embodiment of the invention, and the actual structure is not limited thereto. No reference numerals in the claims should limit the scope of the claims. Therefore, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the invention, such design should fall within the scope of protection of this patent. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Multiple elements stated in the product claims may also be implemented by a single element through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A Transformer-based grasping pose detection method, comprising: Obtain the input image; The acquired input image is fed into the encoder for feature extraction and outputs a feature map. The encoder contains multiple Swing conversion modules. The feature map output by the encoder is used as the input to the first decoder, which then performs upsampling and feature fusion on the feature map. The feature map output by the first decoder is used as the input of the second decoder. The second decoder decodes the feature map and outputs a capture quality score map, a width map, and an angle map. Construct a Transformer-based grasping pose detection model, which includes an encoder, a first decoder, and a second decoder; A loss function is constructed, treating the grasping pose prediction as a regression problem. The grasping pose detection model is trained and optimized by minimizing the distance between the predicted grasping image and the real grasping image. Repeat all steps until the grasping pose detection model training converges, and finally obtain the grasping pose detection model based on Transformer. The obtained Transformer-based grasping pose detection model is used for grasping pose detection.

2. The grasping pose detection method according to claim 1, characterized in that, include: The acquired input image is fed into the encoder for feature extraction, and the output feature map is generated. The encoder contains multiple Swing transformation modules, including: The acquired input image is fed into the encoder for feature extraction; The encoder consists of multiple stacked Swing conversion modules; Each Swin transformation module includes a multilayer perceptron (MLP), a normalization layer (Layer Norm), a window-based multi-head self-attention mechanism, and a shifted-window multi-head self-attention mechanism; The encoder maps input features to query Q, key K, and value V through a linear transformation; The attention mechanism obtains attention weights by calculating the similarity between query Q and key K through dot product; After the attention weights are multiplied by the value V, the output feature map of the encoder is obtained through normalization and multi-head processing. Among them, the window-based multi-head self-attention mechanism establishes global context information; the moving window-based multi-head self-attention mechanism balances local and global information to achieve environmental perception.

3. The grasping pose detection method according to claim 2, characterized in that, include: The query Q, key K, and value V are calculated using the following formula: Q=XW Q K=XW K V=XW V Among them, W Q W K W V Let Q, K, and V represent the linear projection matrices used to generate Q, K, and V, respectively, and X be the input feature map.

4. The grasping pose detection method according to claim 3, characterized in that, include: The attention mechanism obtains attention weights by calculating the similarity between query Q and key K using a dot product. The dot product is calculated using the following formula: Where, d K K is the channel dimension; B is the bias term, which contains context-learnable parameters; T is the transpose of the matrix.

5. The grasping pose detection method according to claim 1, characterized in that, include: The feature map output by the encoder is used as the input to the first decoder. The first decoder performs upsampling and feature fusion on the feature map, including: The encoder's output feature map serves as the input to the first decoder; The first decoder uses a fully convolutional neural network structure to upsample and fuse the feature maps output by the encoder. The first decoder restores the size of the feature map output by the encoder to or close to the size of the original input image through convolutional layers and upsampling operations. The first decoder performs feature fusion operations, using multi-scale information to fuse feature maps of different scales; The first decoder generates a feature map of the same size as the input image, which contains the object's grasping pose information.

6. The grasping pose detection method according to claim 5, characterized in that, include: Feature map fusion includes: Feature fusion employs skip connections or attention mechanisms to fuse information at different scales.

7. The grasping pose detection method according to claim 1, characterized in that, include: The feature map output by the first decoder serves as the input to the second decoder. The second decoder decodes the feature map and outputs a capture quality score map, a width map, and an angle map, including: The feature map output by the first decoder is input into the second decoder for decoding to generate a capture quality score map, a width map, and an angle map. The decoding process maps the feature map to the representation space of the grasping pose, where each pixel in the feature map corresponds to the corresponding grasping pose. By outputting the planar grasping pose onto the original image, the detected grasping pose information is obtained; The two-dimensional image G is captured by extracting the generated quality score image, width image, and angle image. The grasping pose is represented by a 5-dimensional grasping g.

8. The grasping pose detection method according to claim 7, characterized in that, include: The 5D grasping g is calculated using the following formula: g = {x, y, θ, w, h} Where x and y are the coordinates of the center point of the grasp rectangle in the grasp pose, θ is the rotation angle of the grasp rectangle relative to the horizontal direction of the image, w is the width of the grasp rectangle, and h is the height of the grasp rectangle.

9. The grasping pose detection method according to claim 7, characterized in that, include: The two-dimensional image G is calculated using the following formula: G={Q,W,θ}∈R3×D×H Where Q represents the capture quality score image for each pixel, W and θ represent the capture width and angle images for each pixel; R is a matrix, D is the width of the input image, and H is the height of the input image.

10. The grasping pose detection method according to claim 1, characterized in that, include: Constructing a loss function that treats grasping pose prediction as a regression problem, the grasping pose detection model is trained and optimized by minimizing the distance between the predicted and actual grasping images. The input image is processed by a Transformer-based grasping pose detection model to extract feature maps. A predicted capture map is generated by performing a convolution operation on the feature map; The predicted crawl image is compared with the actual crawl image, and the distance between the two is calculated; By incorporating distance as part of the loss function, the grasping pose prediction is transformed into a regression function; By minimizing the loss function, the weights and parameters of the grasping pose detection model are updated using the backpropagation algorithm; Repeat all the above steps until the model converges and reaches the predetermined training objective; The predicted grab map represents the distribution of possible grab positions and orientations on the image; the true grab map represents the distribution of labeled true grab positions and orientations.

Images