A robot dynamic grasping method based on sliding window self-attention mechanism

Abstract

The present invention discloses a robot dynamic grasping method based on a sliding window self-attention mechanism. The method first installs a camera and calculates the transformation matrix between the camera and the manipulator coordinate system based on the hand-eye calibration technology; uses affine transformation to process the static data set to generate a grasping dynamic data set with speed information; then constructs a dynamic grasping network model based on the sliding window multi-head self-attention mechanism, and designs a loss function optimization model that simultaneously considers the object's motion speed and grasping posture; finally, deploys the trained model into a robot grasping environment, designs a position-based visual servo grasping control scheme, and realizes the follow-up movement and grasping action of the manipulator end effector to the grasped object. The robot dynamic grasping method of the present invention can quickly and accurately predict the grasping posture and motion speed of an object based on a set of continuous single-channel depth images collected by the camera, solving the grasping problem of general objects in complex dynamic environments.

Description

Technical Field

[0001] The present invention relates to the field of machine vision and robotics technology, and in particular to a robot dynamic grasping method based on a sliding window self-attention mechanism. Background Art

[0002] Current intelligent manufacturing systems are highly automated, and industrial robots have replaced many repetitive, tedious, and low-intelligence tasks. However, these robots are typically limited to controlled environments with pre-programmed movements. Grasping unfamiliar objects, or objects whose posture and position have changed, can be challenging. The emergence of machine learning and deep learning models has reduced the need for precise modeling of the environment and objects being manipulated. By leveraging large amounts of known data, such as object environments and grasping postures, these models can learn common grasping methods, enabling robots to adapt to real-world scenarios.

[0003] When grasping dynamic objects, it's crucial to know not only the object's posture but also its speed and direction. This allows the object's future coordinates to be used as a guide for grasping. Currently, the speed of objects on production lines is often calculated using encoders on conveyor belts, but accuracy is often difficult to guarantee. Traditional open-loop control solutions can easily fail if the speed changes. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention discloses a robot visual servo dynamic grasping method based on a sliding window self-attention mechanism. The specific technical solution is as follows:

[0005] The second aspect of the present invention is a robot dynamic grasping method based on a sliding window self-attention mechanism, comprising the following steps:

[0006] Step S1: Install the camera and robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology;

[0007] Step S2: pre-processing the captured static dataset based on affine transformation to generate a captured dynamic dataset with speed information;

[0008] Step S3: Build a dynamic grasping network model based on the sliding window multi-head self-attention mechanism, design a loss function that takes into account both the object's movement speed and grasping posture, and complete the network model weight training;

[0009] Step S4: Deploy the trained model into a robot grasping environment, using a set of continuously captured depth images as input and outputting the optimal grasping posture and object movement speed;

[0010] Step S5: Design a position-based visual servo grasping control scheme to achieve the follow-up movement of the robot arm end effector to the object to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

[0011] Specifically, the installation of the camera and the robotic arm in step S1, with the camera as the eye and the robotic arm as the hand, specifically includes the following two methods:

[0012] (a) Eye outside the hand: The eye outside the hand means that the camera is separated from the robotic arm and the camera is fixed relative to the base of the robotic arm. No matter how the robotic arm moves, the camera always maintains the same position and posture;

[0013] (b) Eye on hand: The eye on hand means that the camera is fixed on the end of the robotic arm. The camera is fixed relative to the end of the robotic arm and moves in the same direction as the robotic arm. In order to avoid obstruction of the camera's field of view during the movement of the robotic arm and the movement of objects outside the camera's field of view, the eye on hand installation method is preferably adopted.

[0014] Furthermore, the hand-eye calibration technology in step S1 is implemented by the following steps:

[0015] Step S11: Fix the camera on the camera bracket installed at the end of the robotic arm;

[0016] Step S12: Prepare a checkerboard calibration plate with an accuracy of ±0.01 mm, which is clear and non-reflective, and place it horizontally on a work platform within the camera's field of view;

[0017] Step S13: Keep the calibration plate and the robotic arm base stationary, and control the end of the robotic arm to move multiple times through the teach pendant, ensuring that the calibration plate appears completely within the camera's field of view during the movement;

[0018] Step S14: Every time the end of the robotic arm moves to a new position, the calibration plate image captured by the camera and the current position of the end of the robotic arm relative to the base are saved;

[0019] Step S15: Calculate the homogeneous transformation matrix from the camera coordinate system to the robot arm end coordinate system based on the recorded data and the relationship between the coordinate systems.

[0020] Furthermore, in step S2, a grasping dynamic dataset with speed information is generated, specifically, an affine transformation process including only translation is performed on the local depth image with the grasping position at the center given by the existing robot intelligent grasping dataset, and the center point of the affine transformation is the center point of the local depth image; the original local image D without any transformation is recorded as time 0, and an affine transformation is performed every Δt time on the [0, kΔt] time segment, and the formula is as follows:

[0021] dt =Affine(d0,M t )

[0022]

[0023] Affine(A,B) represents the affine transformation of image A using matrix B, and M t ∈R 3×3 is the affine transformation matrix, v∈R 2×1 To simulate the speed of motion, E∈R 2×2 represents the identity matrix, t∈[0,kΔt] is the time of affine transformation; the grasping dynamic dataset with speed information generated based on the above method includes a time series local depth image with a time length of t, the moving speed in pixels, the grasping angle, the grasping height, and the grasping quality when grasping at time 0.

[0024] Furthermore, the construction in step S3 is a dynamic crawling network model based on the sliding window multi-head self-attention mechanism, specifically: the dynamic crawling network model refers to the Swin-Transformer model and is composed of several stacked block structures; each block structure is connected to the sliding window multi-head attention mechanism layer and the convolution layer through two residual modules, where the sliding window size is fixed to 3×3, and the convolution kernel size is 1×1, which is used for the fully connected layer; a 2×2 downsampling stage is added between blocks to reduce the image width and height to half of the original, and the number of channels to twice the original.

[0025] Furthermore, the robot grasping environment in step S4 includes a six-degree-of-freedom robotic arm, a two-finger gripper, a depth camera, and an object to be grasped; the robotic arm adopts a top-down grasping method, that is, the two-finger gripper is perpendicular to the grasping plane.

[0026] Furthermore, the visual servo grasping control scheme in step S5 adopts position-based visual servoing, which uses the error between the current posture information of the robot arm and the target posture information predicted by the network as the input of the controller and outputs the movement speed required for the end of the robot arm to follow the target.

[0027] The second aspect of the present invention is a robot dynamic grasping device based on a sliding window self-attention mechanism, comprising the following modules:

[0028] Install the calculation module: Install the camera and robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology;

[0029] Preprocessing module: preprocesses the grasping static dataset based on affine transformation to generate a grasping dynamic dataset with speed information;

[0030] Model training module: Build a dynamic grasping network model based on a sliding window multi-head self-attention mechanism, design a loss function that considers both object movement speed and grasping posture, and complete network model weight training;

[0031] Input / output module: This module deploys the trained model into a robot grasping environment, takes a set of continuously captured depth images as input, and outputs the optimal grasping posture and object movement speed.

[0032] Design and implementation module: Design a position-based visual servo grasping control scheme to realize the follow-up movement of the end effector of the robotic arm to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

[0033] The advantages and beneficial effects of the present invention are:

[0034] (1) The present invention proposes a method for generating a dynamic grasping dataset based on affine transformation simulation, that is, using affine transformation to simulate the motion of grasping features to generate a dynamic grasping dataset with speed information, which solves the problem of missing training data for deep learning models in dynamic grasping tasks.

[0035] (2) The sliding window self-attention mechanism proposed in the present invention combines the local attention capability of the convolution kernel and the feature extraction capability of the attention mechanism, which improves the network computing efficiency while also improving the network prediction accuracy. By building a dynamic grasping network skeleton that replaces the convolution layer with a sliding window multi-head attention mechanism, end-to-end prediction of the object grasping posture and movement speed is achieved, avoiding the time-consuming candidate grasping sampling and evaluation process.

[0036] (3) The present invention adopts a position-based visual servo closed-loop control scheme to replace the traditional open-loop grasping scheme, thereby improving the robot arm's ability to resist changes in objects and the environment, enabling the robot arm to accomplish more flexible and complex dynamic grasping planning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the robot dynamic grasping process in an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of a dynamic capture neural network in an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of the calculation process of the sliding window self-attention mechanism in an embodiment of the present invention;

[0040] Figure 4 Schematic diagram of the visual servo closed-loop grasping solution in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0042] like Figure 1 As shown, the robot visual servo dynamic grasping method based on the sliding window self-attention mechanism of the present invention includes a physical grasping environment, a dynamic grasping neural network and a visual servo closed-loop grasping strategy.

[0043] The physical grasping environment consists of a robot, a two-finger parallel adaptive gripper, a depth camera, and a set of objects to be grasped; the robot and the two-finger parallel adaptive gripper are the main actuators for grasping; and the depth camera is responsible for collecting RGB-D image information of the grasped object. In this embodiment, the robot is a six-axis collaborative robotic arm, the depth camera is a camera that can capture color images and 2.5D depth images, and the set of objects to be grasped is an object randomly placed horizontally in the robot's workspace and has a certain movement speed. The depth camera is installed in an eye-in-hand manner, that is, the camera is fixed relative to the end of the robot. The homogeneous transformation matrix from the camera coordinate system to the coordinate system of the end of the robotic arm obtained by the hand-eye calibration technology can obtain the position and posture of the object's graspable features relative to the coordinate system of the robotic arm base, and then the end execution coordinate system of the robotic arm is coincident with this posture to perform grasping.

[0044] The dynamic crawling neural network is based on the sliding window self-attention mechanism to build the network structure, which is composed of multiple block structures, such as Figure 2 As shown in the figure, each block structure connects the sliding window multi-head attention mechanism layer and the convolution layer through two residual modules. The sliding window size is fixed to 3×3, and the convolution kernel size is 1×1, acting as a fully connected layer. A 2×2 downsampling stage is added between blocks to halve the image width and height and double the number of channels. The network takes a series of temporally uniformly sampled depth maps as input and outputs the object grasp pose at a certain intermediate moment and the object's motion velocity at the pixel level. The final grasp position of the object is obtained by adding the planar position of the intermediate grasp pose and the position offset caused by the velocity.

[0045] The theoretical calculation process of the sliding window self-attention mechanism is as follows Figure 3 As shown, for input X∈R C×H×W , we decompose it into X = X 11 ,X 12 ,…,X HW , where X ij ∈R C×1×1 For each slice X ij , we calculate its attention with nearby slices, i.e. X ij With Xpq Attention, where p∈[ik,i+k], q∈[jk,j+k], M=2k+1 is the window size. Use three different convolutional layers with a convolution kernel size of 1×1 to map the input X to three different spaces Q, K, and V. First, calculate the correlation coefficient between the vector at ij and all vectors in the window centered on ij, that is, calculate Q pq With K ij The vector inner product of , and all correlation coefficients are normalized using the Softmax function, and then all vectors within the window are weighted averaged according to the above correlation coefficients as the final output value. The formula is as follows:

[0046]

[0047] So far, the attention calculation of the feature vector at the input feature ij has been completed. Repeat the above steps at all pixels to complete the sliding window self-attention calculation operation for all feature maps. When we perform self-attention calculation inside the window, the role played by each window can be compared to a convolution calculation performed by the convolution kernel - the self-attention calculation is performed with each pixel slice as the center, which is consistent with the inner product operation between the convolution kernel and the feature map; the repeated self-attention calculation for all pixels is consistent with the sliding of the convolution kernel on the feature map. Considering that the final output of the network contains two parts, one is the object movement speed at the pixel scale, and the other is the score when grasping at k different angles. Therefore, we use k binary classification methods for training, that is, for each angle, we can get a binary classification score, that is, whether the angle can be successfully grasped. The network training process is defined as follows:

[0048]

[0049] Among them, D t ,h is the depth image sequence and grasping height input by the network, q∈{0,1} is the binary classification label of the grasping angle, is the binary classification score of the grasp at the i-th angle output by the network, V is the object movement speed predicted by the network, and L c is the cross entropy loss, L r is the square loss, and λ is the loss weight.

[0050] Visual servo closed-loop grasping strategy such as Figure 4As shown in the figure. Dynamic grasping can obtain the final grasping posture of the target object. Based on the robot controller, the posture of the current end gripper of the robot can be obtained. The error between the grasping posture of the target object and the current posture of the end gripper of the manipulator is used as the input of the visual servo controller, and the motion speed required for the end gripper to follow the target object is output. Based on the end motion speed, the Jacobian matrix of the manipulator can be used to find the inverse kinematic solution, and the motion speed of each joint of the manipulator can be obtained and handed over to the speed controller to control the manipulator, thereby continuously approaching the target object. When certain conditions are met, such as when the grasping height is less than a certain value, it can be determined that the robot has basically overlapped with the object, and the manipulator and gripper can be controlled to complete the grasp.

[0051] The present invention also discloses a robot dynamic grasping device based on a sliding window self-attention mechanism, comprising the following modules:

[0052] Install the calculation module: Install the camera and robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology;

[0053] Preprocessing module: preprocesses the grasping static dataset based on affine transformation to generate a grasping dynamic dataset with speed information;

[0054] Model training module: Build a dynamic grasping network model based on a sliding window multi-head self-attention mechanism, design a loss function that considers both object movement speed and grasping posture, and complete network model weight training;

[0055] Input / output module: This module deploys the trained model into a robot grasping environment, takes a set of continuously captured depth images as input, and outputs the optimal grasping posture and object movement speed.

[0056] Design and implementation module: Design a position-based visual servo grasping control scheme to realize the follow-up movement of the end effector of the robotic arm to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

[0057] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

[0058] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A robot dynamic grasping method based on sliding window self-attention mechanism, characterized in that: The steps include: Step S1: Install the camera and robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology; Step S2: pre-processing the captured static dataset based on affine transformation to generate a captured dynamic dataset with speed information; Step S3: Construct a dynamic grasping network model based on a sliding window multi-head self-attention mechanism, and design a loss function that simultaneously considers the object's motion speed and grasping posture to complete the network model weight training; the construction of the dynamic grasping network model based on the sliding window multi-head self-attention mechanism is as follows: the dynamic grasping network model is based on the Swin-Transformer model and is composed of several stacked block structures; each block structure is connected to the sliding window multi-head attention mechanism layer and the convolution layer through two residual modules, wherein the sliding window size is fixed to 3×3 and the convolution kernel size is 1×1 for the fully connected layer; a 2×2 downsampling stage is added between blocks to reduce the image width and height to half of the original and the number of channels to twice the original; Step S4: Deploy the trained model into a robot grasping environment, using a set of continuously captured depth images as input and outputting the optimal grasping posture and object movement speed; Step S5: Design a position-based visual servo grasping control scheme to achieve the follow-up movement of the robot arm end effector to the object to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

2. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The installation of the camera and the robotic arm in step S1, using the camera as the eyes and the robotic arm as the hands, specifically includes the following two methods: (a) Eye outside the hand: The eye outside the hand means that the camera is separated from the robotic arm and the camera is fixed relative to the base of the robotic arm. No matter how the robotic arm moves, the camera always maintains the same position and posture; (b) Eye on hand: The eye on hand means that the camera is fixed at the end of the robotic arm. The camera is fixed relative to the end of the robotic arm and moves in the same direction as the robotic arm. In order to avoid obstruction of the camera's field of view during the movement of the robotic arm and the movement of objects outside the camera's field of view, the eye on hand installation method is adopted.

3. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The hand-eye calibration technology in step S1 is implemented by the following steps: Step S11: Fix the camera on the camera bracket installed at the end of the robotic arm; Step S12: Prepare a checkerboard calibration plate with an accuracy of ±0.01 mm, which is clear and non-reflective, and place it horizontally on a work platform within the camera's field of view; Step S13: Keep the calibration plate and the robotic arm base stationary, and control the end of the robotic arm to move multiple times through the teach pendant, ensuring that the calibration plate appears completely within the camera's field of view during the movement; Step S14: Every time the end of the robotic arm moves to a new position, the calibration plate image captured by the camera and the current position of the end of the robotic arm relative to the base are saved; Step S15: Calculate the homogeneous transformation matrix from the camera coordinate system to the robot arm end coordinate system based on the recorded data and the relationship between the coordinate systems.

4. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: In step S2, a grasping dynamic dataset with speed information is generated. Specifically, an affine transformation process including only translation is performed on the local depth image at the center of the grasping position given by the existing robot intelligent grasping dataset. The center point of the affine transformation is the center point of the local depth image. The original local image D without any transformation is recorded as time 0, and an affine transformation is performed every Δt time on the [0, kΔt] time segment. The formula is as follows: d t =Affine(d0,M t ) Affine(A,B) represents the affine transformation of image A using matrix B, and M t ∈R 3×3 is the affine transformation matrix, v∈R 2×1 To simulate the speed of motion, E∈R 2×2 represents the identity matrix, t∈[0,kΔt] is the time of affine transformation; the grasping dynamic dataset with speed information generated based on the above method includes a time series local depth image with a time length of t, the moving speed in pixels, the grasping angle, the grasping height, and the grasping quality when grasping at time 0.

5. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The robot grasping environment in step S4 includes a six-degree-of-freedom robotic arm, a two-finger gripper, a depth camera, and an object to be grasped; the robotic arm adopts a top-down grasping method, that is, the two-finger gripper is perpendicular to the grasping plane.

6. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The visual servo grasping control scheme in step S5 adopts position-based visual servoing, which uses the error between the current posture information of the robot arm and the target posture information predicted by the network as the input of the controller and outputs the movement speed required for the end of the robot arm to follow the target.

7. The robot dynamic grasping device according to any one of claims 1 to 6, characterized in that: Includes the following modules: Install the calculation module: Install the camera and robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology; Preprocessing module: preprocesses the grasping static dataset based on affine transformation to generate a grasping dynamic dataset with speed information; Model training module: Construct a dynamic grasping network model based on the sliding window multi-head self-attention mechanism, and design a loss function that simultaneously considers the object's motion speed and grasping posture to complete the network model weight training; the construction of the dynamic grasping network model based on the sliding window multi-head self-attention mechanism is as follows: the dynamic grasping network model is based on the Swin-Transformer model and is composed of several stacked block structures; each block structure is connected to the sliding window multi-head attention mechanism layer and the convolution layer through two residual modules, where the sliding window size is fixed to 3×3 and the convolution kernel size is 1×1 for the fully connected layer; a 2×2 downsampling stage is added between blocks to reduce the image width and height to half of the original and the number of channels to twice the original; Input / output module: This module deploys the trained model into a robot grasping environment, takes a set of continuously captured depth images as input, and outputs the optimal grasping posture and object movement speed. Design and implementation module: Design a position-based visual servo grasping control scheme to realize the follow-up movement of the end effector of the robotic arm to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

Images