A mechanical arm grabbing system and method based on an end-to-end planar grabbing network

Abstract

This invention relates to the field of robotic arm visual grasping technology, and discloses a robotic arm grasping system and method based on an end-to-end planar grasping network. The system acquires RGB-D images of the scene to be grasped using a depth camera, and preprocesses the images to obtain network input tensors. The core of the grasping pose prediction is an end-to-end planar grasping network, which sequentially includes a feature extraction and suppression layer, a global relation modeling layer, a feature restoration layer, and a prediction layer. The prediction results are filtered according to a set confidence threshold, and non-maximum suppression is used to remove overlapping candidate boxes to obtain the optimal grasping model in the image space. Based on camera intrinsic parameters and the hand-eye transformation matrix, the optimal grasping model is transformed from the image space to the camera coordinate system, and then to the robotic arm coordinate system to obtain the robotic arm's execution pose. Finally, the robotic arm is controlled to complete the grasping action according to the execution pose. This invention achieves fast and accurate planar grasping pose prediction.

Description

Technical Field

[0001] This invention relates to the field of robotic arm vision grasping technology, specifically to a robotic arm grasping system and method based on an end-to-end planar grasping network. Background Technology

[0002] Robotic arm vision grasping is a technology that, given a scene containing a target object, uses sensors to perceive external information and calculate the optimal grasping pose, enabling the robotic arm to smoothly complete the grasping operation. It is widely used in industrial production, logistics sorting, and service robots. Existing robotic arm vision grasping algorithms are mainly divided into two categories: analysis-based and data-based methods.

[0003] The analytical method requires offline creation of 3D models of known objects and a grasping model database. In the online stage, the grasping pose is obtained by identifying objects and matching them with the database. However, this method has problems such as a large workload in building the grasping model library, the ability to grasp only known objects, and poor versatility.

[0004] The data-driven approach, which trains networks using datasets containing visual information and grasping labels to directly predict grasping poses, is currently the main development direction in visual grasping. Early data-driven visual grasping networks often employed the sliding window method, requiring the generation of candidate grasping poses before searching for the optimal solution, resulting in high time costs and failing to meet real-time grasping requirements. In recent years, one-stage end-to-end networks have become a research focus, directly predicting the optimal grasping pose through convolutional neural networks, improving real-time performance. However, existing end-to-end networks still have shortcomings: traditional convolutional neural networks have local perception characteristics and lack the ability to model global image information, leading to limited grasping accuracy; while pure Transformer-structured grasping networks have high computational complexity and a large number of parameters, making it difficult to balance real-time performance and lightweight requirements.

[0005] Furthermore, traditional self-attention mechanisms suffer from problems in grasping tasks, such as equal initialization of attention weights and a quadratic relationship between computational complexity and the number of pixels. This leads to slow network convergence and poor grasping performance for small objects. Simultaneously, invalid background information in the image interferes with the network's feature learning of the target region, further affecting grasping accuracy. Therefore, there is an urgent need for an end-to-end visual grasping solution that balances high accuracy, lightweight design, and real-time performance. Summary of the Invention

[0006] To address the problems of poor versatility, insufficient real-time performance, limited accuracy, and high computational complexity in existing robotic arm vision grasping networks, this invention aims to provide a robotic arm grasping system and method based on an end-to-end planar grasping network. This system combines the lightweight advantages of convolutional neural networks and the global modeling capabilities of self-attention mechanisms, while simultaneously suppressing background interference through attention mechanisms, achieving fast and accurate planar grasping pose prediction. The technical solution is as follows:

[0007] A robotic arm grasping system based on an end-to-end planar grasping network includes:

[0008] 1) Image acquisition and input module: used to acquire RGB-D images of the scene to be captured through a depth camera, and preprocess the images to obtain network input tensors;

[0009] 2) Grasping pose prediction module: Its core is an end-to-end planar grasping network, which includes a feature extraction and suppression layer, a global relationship modeling layer, a feature restoration layer and a prediction layer in sequence;

[0010] The feature extraction and suppression layer is used to extract local image features and embed a CA attention mechanism to enhance the feature weight of the target region and suppress background interference.

[0011] The global relation modeling layer is used to establish global feature relations through a deformable self-attention mechanism;

[0012] The feature restoration layer is used to restore the low-resolution features after global modeling to the input image resolution;

[0013] The prediction layer is used to output the grasp confidence, grasp angle and grasp width of each pixel;

[0014] 3) Optimal pose selection module: It is used to select the prediction results according to the set confidence threshold, and use non-maximum suppression to remove overlapping candidate boxes to obtain the optimal grasping model in the image space.

[0015] 4) Coordinate transformation module: Based on camera intrinsic parameters and hand-eye transformation matrix, the optimal grasping model is transformed from image space to camera coordinate system, and then to robot arm coordinate system to obtain the robot arm's execution pose;

[0016] 5) Grasping execution module: used to control the robotic arm to complete the grasping action according to the execution posture.

[0017] A robotic arm grasping method based on an end-to-end planar grasping network includes the following steps:

[0018] S1: Image Acquisition and Preprocessing: RGB-D images of the scene to be captured are acquired through a depth camera, and then normalized, resized, and encapsulated to form the network input tensor;

[0019] S2: Grasping Pose Prediction: The input tensor is input into a trained end-to-end planar grasping network; the end-to-end planar grasping network extracts local features and suppresses the background through feature extraction and suppression layers, establishes global feature relationships through a global relationship modeling layer, restores resolution through a feature restoration layer, and finally outputs the grasping confidence, angle, and width of each pixel by the prediction layer; wherein, the global relationship modeling is implemented through a structure containing 3 deformable self-attention layers;

[0020] S3: Optimal pose selection: Set a confidence threshold to filter low-confidence predictions and perform non-maximum suppression on the remaining candidate capture boxes to select the optimal capture model in the image space.

[0021] S4: Coordinate system transformation: Based on the camera intrinsic parameter matrix, the optimal grasping model is transformed from the image pixel coordinate system to the camera coordinate system; then, based on the transformation matrix obtained from hand-eye calibration, it is transformed from the camera coordinate system to the robot arm base coordinate system to obtain the target pose of the robot arm end effector.

[0022] S5: Trajectory planning and grasping execution: Based on the target pose, collision detection and path planning are performed. After generating a safe trajectory, the robotic arm is controlled to move to the target point and the gripper is operated to complete the grasping.

[0023] The beneficial effects of this invention are:

[0024] 1) High accuracy: This invention combines the local feature extraction capability of convolutional neural networks with the global modeling capability of self-attention mechanisms. It achieves an accuracy of 98.39% (image segmentation) and 97.19% (object segmentation) on the Cornell dataset and 96.33% on the Jacquard dataset, both of which are superior to existing mainstream models.

[0025] 2) Strong real-time performance: This invention uses a convolutional neural network as the backbone and adopts a deformable attention mechanism to reduce computational complexity. The detection time for a single image is only 21ms, which meets the real-time grasping requirements of the robotic arm.

[0026] 3) Wide applicability: The solution of this invention has a success rate of 94.44% in capturing 9 kinds of unknown objects on the network in a real environment, and has a strong ability to capture unknown objects.

[0027] 4) Strong anti-interference ability: This invention effectively suppresses invalid background information through the CA (Coordinate Attention) attention mechanism, improves the feature learning weight of the target area, and captures a confidence heatmap that closely matches the actual shape of the object.

[0028] 5) Lightweight architecture: The network parameters of this invention are only 0.29M (3 self-attention layers), which is easy to train and deploy and suitable for low-power devices. Attached Figure Description

[0029] Figure 1 This is a network structure diagram of the SA-Grasp (Self-Attention GRASP Network, a grasping pose estimation network based on a self-attention mechanism) of this invention.

[0030] Figure 2 This is a structural diagram of the CA attention mechanism.

[0031] Figure 3 This is a structural diagram of a deformable attention mechanism.

[0032] Figure 4 Comparison of confidence heatmaps for different models.

[0033] Figure 5(a) shows the grabbing prediction results of the first group of target objects on the Cornell dataset according to the present invention.

[0034] Figure 5(b) shows the grasping prediction results of the second group of target objects on the Cornell dataset according to the present invention.

[0035] Figure 5(c) shows the grabbing prediction results of the first group of target objects on the Jacquard dataset according to the present invention.

[0036] Figure 5(d) shows the grasping prediction results of the second group of target objects on the Jacquard dataset according to the present invention.

[0037] Figure 6(a) shows the results of capturing the first group of unknown objects in a real environment.

[0038] Figure 6(b) shows the results of capturing the second group of unknown objects in a real environment.

[0039] Figure 7 This is a schematic diagram of the self-attention layer structure, including channel transformation, position encoding, deformable attention, and feedforward network. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0041] I. End-to-end planar crawling network:

[0042] This invention proposes an end-to-end planar grasping network (SA-Grasp) based on self-attention and convolutional neural networks, comprising a sequentially connected image input layer, feature extraction and suppression layer, global relation modeling layer, feature reconstruction layer, prediction layer, and output layer, as shown in the network structure below. Figure 1 As shown. The detailed design of each layer is as follows:

[0043] (1) Image input layer: Supports multimodal input and can be adapted to different sensor configurations:

[0044] Single-modal input: RGB image (3 channels), suitable for scenarios with limited sensor resources.

[0045] Multimodal input: RGB-D (Red Green Blue-Depth) four-channel image, which integrates color image and depth information to enhance feature representation capabilities in complex scenes.

[0046] The input image resolution is fixed at 300×300 to balance detection accuracy and computational efficiency.

[0047] (2) Feature extraction and suppression layer: The core function of this layer is to extract local features of the image and suppress invalid background information. The structure is designed as follows:

[0048] Convolutional layer stacking: One 11×11 and three 5×5 convolutional layers are used, with the number of channels gradually increasing. Non-linearity is introduced through the ReLU activation function to enhance feature representation ability.

[0049] Introduction of dilated convolution: Two 5×5 dilated convolutional layers are used to expand the receptive field and improve the coverage of local information without increasing the number of parameters;

[0050] Downsampling strategy: Two 2×2 max pooling layers are used to achieve 2x and 4x downsampling, reducing computation while preserving key features;

[0051] CA attention mechanism embedding: CA attention modules are added after the first convolutional layer and the second dilated convolutional layer, respectively, such as... Figure 2 As shown, spatial-channel joint attention enhances the feature weights of the target region and suppresses background interference. The calculation process is shown in formula (1):

[0052] (1);

[0053] Where F is the input feature layer, For attention weights in the width direction, For height-oriented attention weights, This indicates element-wise multiplication, and the broadcast operation adapts to spatial dimension consistency. Figure 2 In the figure, C, H, W and r are the number of feature layer channels, feature layer height, feature layer width and scaling factor, respectively.

[0054] (3) Global Relationship Modeling Layer: This layer solves the problem of insufficient global information modeling in CNN, while controlling computational complexity.

[0055] Number of self-attention layers: Experiments have shown that 3 self-attention layers can achieve the best global modeling effect with only 0.29M parameters, avoiding gradient vanishing and computational surge caused by excessively deep networks.

[0056] Channel number conversion: Three-dimensional data is unfolded into one-dimensional data through a 1×1 convolutional layer to meet the computational needs of the self-attention module.

[0057] Position encoding: Sine-cosine encoding is adopted to preserve the spatial position information of pixels and avoid the position independence of the self-attention mechanism. The encoding formula is shown in (2):

[0058] (2);

[0059] Where pos is the pixel's position index in the width or height direction. Encode the dimension index for location. The total dimension for location encoding.

[0060] Deformable attention mechanisms: such as Figure 3 As shown, the deformable attention mechanism abandons the traditional fully connected attention calculation of self-attention and only calculates the attention weight for a few randomly selected sampling points near the reference point. The computational complexity is reduced from O(N²) to O(N) (N is the number of pixels), which greatly improves the inference speed. Its calculation process is shown in formula (3):

[0061] (3);

[0062] Where m is the attention head index, k is the sampling point index, M=8 is the number of attention heads, and K=4 is the number of sampling points per reference point. and The first The sampling point at the th sampling point The attention head is relative to the reference point The coordinate offset and attention weights, and These represent the weights of the two linear mappings.

[0063] (4) Feature Restoration Layer: The low-resolution feature layer after global modeling is restored to the resolution of the input image. The structure is designed as follows:

[0064] Bilinear interpolation: Two-order bilinear interpolation achieves 2x and 4x upsampling, ensuring that the feature layer size is smoothly restored to 300×300;

[0065] Convolutional layer fusion: After each upsampling, a 3×3 convolutional layer is inserted to refine the feature information and make up for the feature loss during the upsampling process.

[0066] (5) Prediction layer: A dedicated prediction branch is designed to address the angle confusion problem, taking into account the characteristics of the captured parameters.

[0067] The confidence branch for grasping: a 1×1 convolutional layer + Sigmoid activation, outputs a confidence score in the range [0,1], representing the probability of successful grasping;

[0068] Grasping width branch: 1×1 convolutional layer + Sigmoid activation, output value multiplied by maximum width 150mm to obtain actual grasping width;

[0069] Grasping angle branch: Two parallel 1×1 convolutional layers predict the angle separately. and ,pass The calculation yields the final angle, avoiding a capture angle of 0° or... This leads to confusion in the grasping angle, resulting in chaotic network training.

[0070] (6) Output layer: The optimal grasping pose is obtained through confidence filtering and non-maximum suppression.

[0071] Confidence threshold: set to 0.5 to filter out invalid candidate poses with low confidence.

[0072] Non-maximum suppression (NMS): The IOU threshold is set to 0.3 to remove overlapping candidate boxes and avoid duplicate predictions;

[0073] Optimal selection: Select the pixel with the highest confidence in the remaining candidate box, and its corresponding coordinates, angle, and width form the optimal grasping model.

[0074] II. Crawling Model and Coordinate Transformation:

[0075] (1) Crawling model definition:

[0076] Image spatial model: This includes pixel coordinates, confidence level, angle, and width;

[0077] Camera coordinate system model: Obtained through camera intrinsic parameter conversion. Provided directly from the depth map;

[0078] Robotic arm coordinate system model: The commands are obtained through hand-eye calibration and converted into instructions for the robotic arm.

[0079] in,( () represents pixel coordinates. The confidence score for capturing at pixel coordinates. The capture angle in pixel coordinates. The capture width in pixel coordinates; () represents the camera coordinates. The capture angle in the camera coordinate system. The capture width in the camera coordinate system; () represents the coordinates of the robotic arm. The gripping angle in the robotic arm coordinate system. This represents the gripping width in the robotic arm's coordinate system.

[0080] (2) Coordinate transformation process: Camera intrinsic parameter calibration: Zhang Zhengyou calibration method is used to obtain intrinsic parameters such as focal length and principal point coordinates to realize the transformation from image space to camera space. The transformation formula is shown in (4):

[0081] (4);

[0082] in, This represents the depth value of the corresponding pixel in the depth map. , For camera focal length, , Principal coordinates; Hand-eye calibration: The Tsai-Lenz algorithm is used to obtain the conversion relationship between the camera and the end effector of the robotic arm, realizing the conversion from camera space to robotic arm space. The conversion formula is shown in (5):

[0083] (5)

[0084] Where R is a 3×3 rotation matrix and t is a 3×1 translation vector.

[0085] III. The robotic arm grasping method based on the above-mentioned planar grasping network includes the following steps:

[0086] (1) Dataset construction: The Cornell and Jacquard general crawl datasets were used. The Cornell dataset was augmented (random rotation, cropping, scaling), and the Jacquard dataset was divided into training and validation sets in a 9:1 ratio.

[0087] (2) Training platform setup: The hardware uses an Intel Core i7-11800H processor, 32G memory, and an RTX3080 graphics card. The software uses the PyTorch 1.8.0 framework and Python 3.8 language.

[0088] (3) Network training: The AdamW optimizer, batch size 32, and learning rate 0.0001 were set. The training was iterated 100 times, and the network parameters were optimized using a multi-objective loss function.

[0089] (4) Image information input: Visual information of the scene to be captured is obtained through a depth camera and input into the trained network.

[0090] (5) Grasping pose prediction: After feature extraction, global modeling, feature restoration and prediction processing, the network directly outputs the grasping prediction model for each pixel.

[0091] (6) Coordinate system transformation: The optimal grasping model is transformed from the image space to the robot coordinate system through the camera coordinate system to obtain the actual grasping pose.

[0092] (7) Grasping execution: The robotic arm performs the grasping task according to the transformed pose.

[0093] IV. Case Analysis:

[0094] (a) Network training and validation:

[0095] Step 1: Dataset Construction and Preprocessing

[0096] Step 1.1: Dataset Selection and Basic Information Confirmation:

[0097] Two internationally recognized crawling datasets, Cornell and Jacquard, were selected. The Cornell dataset contains 885 RGB-D images covering 41 objects, with an original image resolution of 640×480. The Jacquard dataset contains 3648 RGB-D images covering 21 objects, with an original resolution of 512×512. Both datasets include complete crawling pose labels (coordinates, angle, width, and confidence).

[0098] Step 1.2: Data augmentation operation (parameterized execution):

[0099] Augmentation is performed only on the Cornell dataset (the Jacquard dataset is large enough), and the specific steps are as follows:

[0100] Random rotation: angle range [-45°, 45°], step size 5°, rotation center is the center point of the image, bilinear interpolation is used to fill the blank area after rotation, and the filling value is the average pixel value of the image edge to avoid feature distortion;

[0101] Random cropping: Fixed cropping size 300×300, coordinates of the top left corner of the cropping area Randomly generated, filtered by pixel value grayscale distribution to ensure that the pixel ratio of objects within the cropping area is ≥30%;

[0102] Random scaling: The scaling ratio s∈[0.8,1.2]. After scaling by the ratio s, bilinear interpolation is used to adjust back to 300×300 resolution. The aspect ratio is kept constant during the scaling process to avoid object deformation.

[0103] Step 1.3: Dataset Partitioning:

[0104] To test the network's generalization ability for trained objects appearing at different positions and angles, the Cornell dataset was divided into an image subset (IW) and an object subset (OW) to test the network's generalization ability for untrained objects. Since the Jacquard dataset is large enough for network training, no data augmentation was performed; 90% of its data was used as the training set, and the remaining 10% as the validation set.

[0105] Step Two: Training Platform Setup and Parameter Configuration Steps:

[0106] 2.1 Hardware Platform Deployment: The system uses an Intel Core i7-11800H processor, 32GB DDR4 3200MHz dual-channel memory, an NVIDIA RTX3080 8GB graphics card, and a 1TB NVMe SSD to ensure unimpeded computing and data read / write operations.

[0107] Step 2.2: Software Environment Installation and Verification:

[0108] Install Windows 10 as the operating system; create a Python 3.8.10 virtual environment and install dependent libraries such as PyTorch 1.8.0, torchvision 0.9.0, CUDA 11.6, CuDNN 8.4.0, and OpenCV 4.5.5; run the CUDA sample program deviceQuery to confirm that the GPU is recognized normally; run print(torch.cuda.is_available()), and if True is output, the environment configuration is successful.

[0109] Step 2.3: Network parameter initialization: The weights of the convolutional and fully connected layers are initialized using Xavier uniform distribution; all bias terms are initialized to 0; the attention module parameters are fixed: the positional encoding temperature coefficient is 128, the number of deformable attention heads M=8, and the number of sampling points K=4.

[0110] Step 3: Optimize the execution steps:

[0111] Step 3.1: Fine-tuning training parameters: The optimizer is AdamW, and the parameters are set as follows: batch size = 32, initial learning rate lr = 0.0001, weight decay = 0.0001. The learning rate uses a step-decay mechanism, decreasing to 0.5% every 20 rounds; the maximum number of training rounds is 100; an early stopping strategy is adopted: if the accuracy improvement on the validation set is less than 0.1% for 5 consecutive rounds, training is stopped and the optimal model is saved.

[0112] Step 3.2: Loss Function Calculation and Backpropagation:

[0113] Using a multi-objective L1 loss function, the mean absolute error between the predicted value and the true label is calculated using the following formula:

[0114] (6);

[0115] Where H=300 and W=300; after each iteration, loss.backward() is called to calculate the gradient, optimizer.step() updates the parameters, and optimizer.zero_grad() clears the gradient to avoid gradient accumulation.

[0116] Step 3.3: Training Process Monitoring and Model Saving: After each training round, calculate the training set average loss (train_loss), validation set loss (val_loss), and accuracy (val_acc). Accuracy is determined by simultaneously meeting the angle difference criterion (the difference between the predicted and actual angles < 30°) and the Jaccard criterion (Jaccard index > 0.25). The formula is as follows:

[0117] (7);

[0118] in, and These represent the network's predicted optimal crawling model and the corresponding actual crawled labels, respectively. A checkpoint file is saved every 5 rounds.

[0119] Step 3.4: Optimal Model Selection and Validation: After training, load all checkpoint files and test on the Cornell and Jacquard test sets. Select the model with the highest accuracy as the optimal model. The test results are shown in Tables 1 and 2. The model of this invention has the best accuracy on both the Cornell and Jacquard datasets, and the detection time is only 21ms, which meets the real-time requirements.

[0120] Table 1 Performance comparison of different models on the Cornell dataset

[0121]

[0122] Table 2 Performance comparison of different models on the Jacquard dataset

[0123]

[0124] Step 4: Visual Verification and Analysis

[0125] Step 4.1: Confidence heatmap comparison:

[0126] Ten typical objects from the Cornell dataset were selected, and the grasping confidence heatmaps of this invention (SA-Grasp) were compared with those of GR-ConvNet (Generative Residual Convolutional Neural Network), TF-Grasp (Transformer-Grasp), GG-CNN2, and GG-CNN (Generative Grasping Convolutional Neural Network). Figure 4 A comparison of confidence heatmaps for different models shows that the confidence level of the background region in the heatmap of this invention is close to 0, while the confidence level of the center of the target region is the highest (≥0.8), which is highly consistent with the actual shape of the object. Other models have obvious invalid confidence levels (≥0.3) in the background.

[0127] Step 4.2: Visualize the captured pose prediction:

[0128] Object predictions were performed on the Cornell and Jacquard datasets, outputting bounding boxes (blue rectangles), as shown in Figures 5(a)-5(d). From left to right, these represent: target object, confidence score, grasp angle, grasp width, and grasp representation. The predicted bounding boxes accurately cover the optimal grasping region of the object. The SA-Grap grasp confidence heatmaps for each object are generally similar to the actual shape of the object, and the grasp is performed on the central part of the object, which has a higher success rate. This indicates that SA-Grap can predict the optimal grasp pose for different objects.

[0129] (II) Construction and Calibration of Real-World Experimental Platform:

[0130] Step 1: Hardware Deployment Steps for the Experimental Platform

[0131] Step 1.1: Core Equipment Selection and Installation: The robotic arm selected is the Kinova Jaco2 6-DOF robotic arm, with a three-finger parallel gripper as the end effector; the depth camera selected is the Intel RealSense D435i, installed at the end of the robotic arm in Eye-in-Hand mode, with the camera's optical axis perpendicular to the gripper's central axis, calibrated using a level to ensure the optical axis is parallel to the work platform; the host computer is connected to the robotic arm via a USB-C data cable and to the camera via a USB 3.0 data cable.

[0132] Step 1.2: Platform debugging and stability testing: Start the robotic arm and perform joint motion testing using the Kinova SDK to ensure that there is no jamming in each joint and the repeatability error is ≤0.1mm; start the camera and acquire 100 frames of RGB-D images to check that there is no distortion in the images and no jumps in depth values; test the stability of data transmission by continuously acquiring images for 30 minutes without any packet loss.

[0133] Step 2: Camera Intrinsic Parameter Calibration

[0134] Step 2.1: Calibration plate preparation and placement: Print an 8×8 checkerboard calibration plate with inner corner points (square size 10×10mm), and attach it to a rigid acrylic plate, ensuring there is no bending or deformation; place the calibration plate in different positions (at least 5 positions: center, four corners) and at different angles (at least 3 angles: 0°, 30°, 60°) on the work platform, and take a total of 20 calibration images, in which the inner corner points are clearly visible and unobstructed in each image.

[0135] Step 2.2: Intrinsic parameter calculation and verification: The calibration image is processed using the OpenCV function cv2.calibrateCamera(). The world coordinates of the interior corner points (with the upper left interior corner point as the origin, the x-axis along the horizontal direction of the calibration board, the y-axis vertical direction, and the z-axis = 0) and the image coordinates (extracted by cv2.findChessboardCorners()) are input. Distortion correction is performed on the test image, and the average reprojection error is calculated to be 0.35 pixels (<0.5 pixels), indicating that the calibration is effective.

[0136] Step 3: Hand-eye alignment:

[0137] Step 3.1: Calibration Data Acquisition: Fix the checkerboard calibration board at the center of the work platform, ensuring it is parallel to the platform; move the robotic arm to 15 different poses (covering the work area), ensuring each pose meets the following conditions: the camera clearly captures the calibration board, the robotic arm does not self-collision, and the joint angles do not exceed limits; record the end-effector pose of the robotic arm for each pose. Coordinates of the inner corner points of the calibration board captured by the camera .

[0138] Step 3.2: Calculation and verification of the transformation matrix:

[0139] Using the `handeyecalibrate()` function in Matlab, the robot arm's end-effector pose and camera coordinates are input, and the hand-eye transformation matrix is ​​output. Verification: Control the robot arm to the new test pose, take a picture of the calibration board with the camera, and use the transformation matrix to convert the center coordinates of the calibration board in the camera coordinate system to the robot arm coordinate system. Compare with the actual measured values ​​of the robot arm. If the error is ≤0.4mm (<0.5mm), the calibration is valid.

[0140] (III) Real-world capture experiment execution:

[0141] Step 1: Preparation and placement of the object to be grasped:

[0142] Step 1.1: Object selection and parameter recording:

[0143] Nine unknown objects (not present in the training dataset) were selected, covering different shapes, sizes, and materials.

[0144] Step 1.2: Random Placement and Positioning:

[0145] Each object was randomly placed in the test area of ​​the work platform (200×200mm), and the placement position (X,Y) and angle (0°, 360°) were recorded using a laser positioning device to ensure that the placement position and angle were not repeated each time; after placement, the objects were checked to ensure that they were not tilted or obstructed, and that the camera's field of view could be completely covered.

[0146] Step 2: Image Acquisition and Preprocessing

[0147] Step 2.1: Camera parameter settings:

[0148] Set the acquisition parameters using the Intel RealSense SDK: RGB image resolution 300×300, frame rate 30fps, exposure time 100ms, ISO 100; depth map resolution 300×300, frame rate 30fps, laser power 50%; enable pixel-level alignment between RGB and depth maps, with an alignment error ≤1 pixel.

[0149] Step 2.2: Image Acquisition and Format Conversion:

[0150] The host computer synchronously acquires one frame of RGB-D image through the ROS topics / camera / color / image_raw and / camera / depth / image_rect_raw, with an acquisition timestamp error of < 1ms; the RGB image is converted from RAW format to PNG format, and the depth map is converted from 16-bit single-channel format to 32-bit floating-point format.

[0151] Step 2.3: Preprocessing operations:

[0152] Normalization: RGB channel pixel values ​​are divided by 255 and normalized to [0,1]; depth channel pixel values ​​are divided by 1000 and converted to meters, filtering out noise points with depth values ​​< 0.1m or > 0.5m and replacing them with 3×3 domain averages;

[0153] Resizing: Use bilinear interpolation to resize the image to 300×300, matching the network input size;

[0154] Data encapsulation: The preprocessed RGB-D image is encapsulated into a 4-channel tensor (dimension 1×4×300×300), and then input into the network after adding batch dimensions.

[0155] Step 3: Network Inference and Optimal Grasping Pose Calculation Steps:

[0156] Step 3.1: Model Loading and Inference Mode Settings: Load the optimal model SA-Grasp_best.pth on the host computer, call model.eval() to set it to inference mode, disable Dropout and BatchNorm training modes to ensure stable inference results; set torch.no_grad() to disable gradient calculation and improve inference speed.

[0157] Step 3.2: Network Inference and Parameter Calculation: Input the tensor into the network, and after processing by each layer, output three 300×300 feature maps: confidence map C, angle... Image, width image For each pixel Calculation: Actual width: (Unit: mm) (This refers to the maximum opening width of the grippers).

[0158] Step 3.3: Optimal pose selection (including NMS execution):

[0159] Confidence filtering: Traverse the confidence graph and filter. The pixels are used to form a candidate point list L;

[0160] NMS execution: Sort L by confidence level from highest to lowest, take the first point P0 (highest confidence level), and capture its bounding box B0; traverse the remaining points Pi in L, calculate the IOU value between Bi and B0, and remove Pi if IOU > 0.3; take the point with the highest confidence level from the remaining points as the new P0, and repeat step 2 until all points are traversed; final selection: select the point with the highest confidence level from the candidate points after NMS. This yields an image space capture model.

[0161] Step 4: Coordinate Transformation and Collision Detection Steps:

[0162] Step 4.1: Image space to camera space transformation: According to formula (4), substitute the camera intrinsic parameters and depth values ​​to calculate: , , , , Obtain the camera space model .

[0163] Step 4.2: Camera space → robotic arm space conversion: Based on formula (5) and hand-eye conversion matrix Calculate the pose of the robotic arm in the coordinate system: obtain the pose of the robotic arm. .

[0164] Step 4.3: Collision Detection: Launch ROS+MoveIt!, load the robotic arm URDF model and the work platform collision model (Collision Mesh), and input... The RRT-Connect path planning algorithm is called to generate a trajectory; collision detection is performed using moveit_core::CollisionCheck, with a collision distance threshold of 0.01m. If there is no collision on the trajectory, the process is executed; if there is a collision, the second-highest confidence candidate point is selected and the conversion and detection are repeated until a collision-free trajectory is obtained.

[0165] Step 5: Robotic arm grasping and data recording:

[0166] Step 5.1: Capture, issue, and execute commands:

[0167] The host computer issues control commands through the ROS topic / jaco2 / goal_position, setting the motion speed to a joint speed of 0.5 rad / s and an end effector linear speed of 0.1 m / s; the robotic arm execution flow is as follows:

[0168] Move to a position 10mm above the target pose and pause for 0.5 seconds;

[0169] The grippers open and move downwards 10mm to contact the object;

[0170] When the grippers close, the closing force is adjusted to 5N via feedback from a force sensor to prevent injury or insecure gripping.

[0171] Move upwards 50mm to the placement area;

[0172] The grippers open to release the object and return to the initial position.

[0173] Step 5.2: Experimental Data Recording and Statistics:

[0174] Each object was grasped 20 times, for a total of 180 times. The experimental data are shown in Table 3. The total number of successful grasps was 170, and the success rate was 170 / 180×100%=94.44%. Some grasping results are visualized in Figures 6(a)-(b).

[0175] Step Six: Failure Root Cause Analysis and Optimization Directions:

[0176] Failure statistics: Of the 10 failures, 3 were due to excessive noise in the depth map caused by reflections on the object's surface, resulting in a gripping position deviation > 2mm; 7 were due to the object's gripping position being too narrow, leading to insecure clamping.

[0177] Optimization directions: Further improvements could be made by adding reflection suppression algorithms (such as adaptive exposure adjustment) and small object grasping strategies (such as increasing gripper friction) to further improve the success rate.

[0178] Table 3 Data Recording Table for Real-World Grasping Experiment

[0179]

[0180] (iv) Optimization and verification of key parameters:

[0181] Experimental design: The number of self-attention layers N∈{1,2,3,4,5,6,7} was set, with other parameters remaining unchanged. The experiment was conducted on the Cornell dataset, and the results are shown in Table 4.

[0182] Table 4 Performance comparison of different numbers of self-attention layers on the Cornell dataset

[0183]

[0184] Results Analysis: The accuracy was optimal (98.39%) when N=3, with only 0.29M parameters and a detection time of 21ms. When N>3, the number of parameters increased dramatically, leading to slower network convergence and decreased accuracy. Therefore, the optimal number of layers was determined to be 3, corresponding to the self-attention layer structure as follows: Figure 7 As shown.

Claims

1. A robotic arm grasping system based on an end-to-end planar grasping network, characterized in that, include: 1) Image acquisition and input module: used to acquire RGB-D images of the scene to be captured through a depth camera, and preprocess the images to obtain network input tensors; 2) Grasping pose prediction module: Its core is an end-to-end planar grasping network, which includes a feature extraction and suppression layer, a global relationship modeling layer, a feature restoration layer and a prediction layer in sequence; The feature extraction and suppression layer is used to extract local image features and embed a CA attention mechanism to enhance the feature weight of the target region and suppress background interference. The global relation modeling layer is used to establish global feature relations through a deformable self-attention mechanism; The feature restoration layer is used to restore the low-resolution features after global modeling to the input image resolution; The prediction layer is used to output the grasp confidence, grasp angle and grasp width of each pixel; 3) Optimal pose selection module: It is used to select the prediction results according to the set confidence threshold, and use non-maximum suppression to remove overlapping candidate boxes to obtain the optimal grasping model in the image space. 4) Coordinate transformation module: Based on camera intrinsic parameters and hand-eye transformation matrix, the optimal grasping model is transformed from image space to camera coordinate system, and then to robot arm coordinate system to obtain the robot arm's execution pose; 5) Grasping execution module: used to control the robotic arm to complete the grasping action according to the execution posture.

2. The robotic arm grasping system based on an end-to-end planar grasping network according to claim 1, characterized in that, The feature extraction and suppression layer in the grasping pose prediction module includes: Convolutional layer stacking unit: One 11×11 and three 5×5 convolutional layers are used, and non-linearity is introduced through the ReLU activation function to enhance feature representation ability; Dilated convolutional unit: Two 5×5 dilated convolutional layers are used to expand the receptive field and improve the coverage of local information; Downsampling unit: Two 2×2 max pooling downsampling layers are used to achieve 2x and 4x downsampling to reduce computation and retain key features; CA attention mechanism embedding unit: CA attention modules are embedded after the first 11×11 convolutional layer and the second 5×5 dilated convolutional layer, respectively; through spatial-channel joint attention, the feature weights of the target region are strengthened and background interference is suppressed. The calculation process is as follows: （1）； in, For the input feature layer, For attention weights in the width direction, For height-oriented attention weights, This indicates element-wise multiplication. This is the output feature layer.

3. The robotic arm grasping system based on an end-to-end planar grasping network according to claim 1, characterized in that, The global relationship modeling layer in the pose prediction module specifically includes: Channel conversion unit: used to unfold three-dimensional feature data into one-dimensional data through a 1×1 convolutional layer; Location encoding unit: The spatial location information of the one-dimensional data is encoded using sine-cosine encoding, and the encoding formula is as follows: （2）； in, Index of the pixel's position in the width or height direction. Encode the dimension index for location. The total dimension is encoded for the location; This is a position-encoded vector; Temperature coefficient; Deformable Self-Attention Computational Unit: Contains 3 self-attention layers, each employing a deformable attention mechanism. Its computation process is as follows: （3）； Where m is the attention head index. Here, M is the sampling point index, M is the number of attention heads, and K is the number of sampling points per reference point; and The first The sampling point at the th sampling point The attention head is relative to the reference point The coordinate offset and attention weight; and These represent the weights of the two linear mappings; For feature maps; For reference point The corresponding one-dimensional feature vector; For reference point Two-dimensional coordinates.

4. The robotic arm grasping system based on an end-to-end planar grasping network according to claim 1, characterized in that, The prediction layer in the grasping pose prediction module includes: The confidence branch is composed of a 1×1 convolutional layer and a sigmoid activation function, and outputs a confidence score in the range of [0,1]. Grab width branch: Consists of a 1×1 convolutional layer and a sigmoid activation function. The output value is multiplied by the maximum width to obtain the actual grab width. Angle branch capture: The sine and cosine values ​​of the angle are predicted by two parallel 1×1 convolutional layers, and the final angle is obtained by arctangent operation.

5. A robotic arm grasping method based on an end-to-end planar grasping network, characterized in that, Includes the following steps: S1: Image Acquisition and Preprocessing: RGB-D images of the scene to be captured are acquired through a depth camera, and then normalized, resized, and encapsulated to form the network input tensor; S2: Grasping Pose Prediction: The input tensor is input into a trained end-to-end planar grasping network; the end-to-end planar grasping network extracts local features and suppresses the background through feature extraction and suppression layers, establishes global feature relationships through a global relationship modeling layer, restores resolution through a feature restoration layer, and finally outputs the grasping confidence, angle, and width of each pixel by the prediction layer; wherein, the global relationship modeling is implemented through a structure containing 3 deformable self-attention layers; S3: Optimal pose selection: Set a confidence threshold to filter low-confidence predictions and perform non-maximum suppression on the remaining candidate capture boxes to select the optimal capture model in the image space. S4: Coordinate system transformation: Based on the camera intrinsic parameter matrix, the optimal grasping model is transformed from the image pixel coordinate system to the camera coordinate system; then, based on the transformation matrix obtained from hand-eye calibration, it is transformed from the camera coordinate system to the robot arm base coordinate system to obtain the target pose of the robot arm end effector. S5: Trajectory planning and grasping execution: Based on the target pose, collision detection and path planning are performed. After generating a safe trajectory, the robotic arm is controlled to move to the target point and the gripper is operated to complete the grasping.

6. The robotic arm grasping method based on an end-to-end planar grasping network according to claim 5, characterized in that, Step S4 specifically includes: S4.1: Define the crawling model; Image spatial model: , ( () represents pixel coordinates. For the confidence score of the capture at pixel coordinates, The capture angle in pixel coordinates. The capture width in pixel coordinates; Camera coordinate system model: Obtained through camera intrinsic parameter conversion. Provided directly from the depth map; , () represents the camera coordinates. The capture angle in the camera coordinate system. The capture width in the camera coordinate system; Robotic arm coordinate system model: The commands are obtained through hand-eye calibration and converted into instructions for the robotic arm to execute. () represents the coordinates of the robotic arm. The gripping angle in the robotic arm coordinate system. The gripping width in the robotic arm coordinate system; S4.2: Coordinate transformation process and camera intrinsic parameter calibration; The Zhang Zhengyou calibration method is used to obtain intrinsic parameters such as focal length and principal point coordinates, thereby realizing the transformation from image space to camera space. The transformation formula is shown below: （4）； in, This represents the depth value of the corresponding pixel in the depth map; , The focal length of the camera; , Principal point coordinates; Hand-eye calibration: The Tsai-Lenz algorithm is used to obtain the transformation relationship between the camera and the robotic arm end effector, realizing the transformation from camera space to robotic arm space. The transformation formula is shown below: （5） Where R is a 3×3 rotation matrix and t is a 3×1 translation vector.

Images