Robot dynamic grabbing method based on sliding window self-attention mechanism

Abstract

The invention discloses a robot dynamic grabbing method based on a sliding window self-attention mechanism, and the method comprises the steps: firstly installing a camera, and calculating a transformation matrix between the camera and a mechanical arm coordinate system based on a hand-eye calibration technology; processing the static data set by adopting affine transformation to generate a grabbed dynamic data set with speed information; then, a dynamic grabbing network model based on a sliding window multi-head self-attention mechanism is constructed, and a loss function optimization model considering the object movement speed and the grabbing posture at the same time is designed; and finally, the trained model is deployed in a robot grabbing environment, a visual servo grabbing control scheme based on the position is designed, and follow-up movement and grabbing action of an end effector of a mechanical arm on a to-be-grabbed object are achieved. According to the dynamic grabbing method of the robot, the grabbing posture and the movement speed of the object can be quickly and accurately predicted based on a group of continuous single-channel depth images collected by the camera, and the grabbing problem of a universal object in a complex dynamic environment is solved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision and robotics technology, and in particular to a robot dynamic grasping method based on a sliding window self-attention mechanism. Background Art

[0002] The current intelligent manufacturing system has a high degree of automation, and many repetitive, boring and low-intelligence tasks have been replaced by industrial robots, but they are usually limited to controlled environments with pre-programmed actions. If faced with unknown objects or if the posture and position of the object have changed, the grasping work will become more difficult. The emergence of machine learning models and deep learning models has reduced the need for accurate modeling of the environment and operating objects. By utilizing a large amount of known data, such as the object environment and grasping posture, the model can learn the general grasping method of objects, so that the robot can cope with real-world scenarios.

[0003] When faced with the task of grasping dynamic objects, it is necessary not only to know the grasping posture of the object, but also the moving speed and direction of the object, so as to introduce the coordinates of the object at the future moment as the advance amount of grasping flow. At present, the moving speed of objects in actual production lines is often calculated by the encoder on the conveyor belt, and the accuracy is often difficult to guarantee. Once the speed changes, the open-loop control solution in the traditional mode is likely to fail to grasp. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention discloses a robot visual servo dynamic grasping method based on a sliding window self-attention mechanism. The specific technical solution is as follows:

[0005] The second aspect of the present invention is a robot dynamic grasping method based on a sliding window self-attention mechanism, comprising the following steps:

[0006] Step S1: Install the camera and the robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology;

[0007] Step S2: preprocessing the captured static data set based on affine transformation to generate a captured dynamic data set with speed information;

[0008] Step S3: Construct a dynamic grasping network model based on the sliding window multi-head self-attention mechanism, design a loss function that takes into account both the object's movement speed and grasping posture, and complete the network model weight training;

[0009] Step S4: deploy the trained model into the robot grasping environment, take a set of depth images obtained by continuous shooting as input, and output the optimal grasping posture and object movement speed;

[0010] Step S5: Design a position-based visual servo grasping control scheme to achieve the follow-up movement of the robot end effector to the object to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

[0011] Specifically, the installation of the camera and the robotic arm in step S1, with the camera as the eye and the robotic arm as the hand, specifically includes the following two methods:

[0012] (a) Eye outside the hand: The eye outside the hand means that the camera is separated from the robotic arm, and the camera is fixed relative to the base of the robotic arm. No matter how the robotic arm moves, the camera always maintains the same position and posture;

[0013] (b) Eye on hand: The eye on hand means that the camera is fixed on the end of the robotic arm. The camera is fixed relative to the end of the robotic arm and moves with the movement of the robotic arm. In order to avoid obstruction of the camera's field of view during the movement of the robotic arm and objects moving out of the camera's field of view, the eye on hand installation method is preferably adopted.

[0014] Furthermore, the hand-eye calibration technology in step S1 is implemented by the following steps:

[0015] Step S11: Fix the camera on a camera bracket installed at the end of the robotic arm;

[0016] Step S12: prepare a checkerboard calibration plate with an accuracy of ±0.01 mm, clear and non-reflective, and place it horizontally on a work platform within the camera's field of view;

[0017] Step S13: Keep the calibration plate and the robot base stationary, and control the end of the robot to move multiple times through the teach pendant, and ensure that the calibration plate appears completely in the camera's field of view during the movement;

[0018] Step S14: Every time the end of the robotic arm moves to a new position, the calibration plate image taken by the camera and the current position of the end of the robotic arm relative to the base are saved;

[0019] Step S15: Calculate the homogeneous transformation matrix from the camera coordinate system to the robot end coordinate system based on the recorded data and the relationship between the coordinate systems.

[0020] Furthermore, in step S2, a grasping dynamic data set with speed information is generated, specifically, an affine transformation process including only translation is performed on the local depth image at the center of the grasping position given by the existing robot intelligent grasping data set, and the center point of the affine transformation is the center point of the local depth image; the original local image D without any transformation is recorded as time 0, and an affine transformation is performed every Δt time on the [0, kΔt] time segment, and the formula is as follows:

[0021] dt =Affine(d0,M t )

[0022]

[0023] Affine(A,B) represents the affine transformation of image A using matrix B. t ∈R 3×3 is the affine transformation matrix, v∈R 2×1 To simulate the speed of motion, E∈R 2×2 represents the unit matrix, t∈[0,kΔt] is the time of affine transformation; the grasping dynamic data set with speed information generated based on the above method includes a time series local depth image with a time length of t, a moving speed in pixels, a grasping angle, a grasping height, and a grasping quality at time 0.

[0024] Furthermore, the construction of the dynamic crawling network model based on the sliding window multi-head self-attention mechanism in step S3 is specifically as follows: the dynamic crawling network model refers to the Swin-Transformer model and is composed of several stacked block structures; each block structure connects the sliding window multi-head attention mechanism layer and the convolution layer through two residual modules, where the sliding window size is fixed to 3×3, and the convolution kernel size is 1×1, which is used for the fully connected layer; a 2×2 downsampling stage is added between blocks to reduce the image width and height to half of the original, and the channel to twice the original.

[0025] Furthermore, the robot grasping environment in step S4 includes a six-degree-of-freedom robotic arm, a two-finger gripper, a depth camera, and an object to be grasped; the robotic arm adopts a top-down grasping method, that is, the two-finger gripper is perpendicular to the grasping plane.

[0026] Furthermore, the visual servo grasping control scheme in step S5 adopts position-based visual servoing, which uses the error between the current posture information of the robot arm and the target posture information predicted by the network as the input of the controller, and outputs the movement speed required for the end of the robot arm to follow the target.

[0027] The second aspect of the present invention is a robot dynamic grasping device based on a sliding window self-attention mechanism, comprising the following modules:

[0028] Install the calculation module: Install the camera and the robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology;

[0029] Preprocessing module: preprocesses the grasping static data set based on affine transformation to generate the grasping dynamic data set with speed information;

[0030] Model training module: Build a dynamic grasping network model based on the sliding window multi-head self-attention mechanism, design a loss function that takes into account both the object's movement speed and grasping posture, and complete the network model weight training;

[0031] Input-output module: deploys the trained model into the robot grasping environment, takes a set of continuously captured depth images as input, and outputs the optimal grasping posture and object movement speed;

[0032] Design and implementation module: Design a position-based visual servo grasping control solution to realize the follow-up movement of the robot end effector to the object to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

[0033] The advantages and beneficial effects of the present invention are:

[0034] (1) The present invention proposes a method for generating a dynamic grasping dataset based on affine transformation simulation, that is, using affine transformation to simulate the motion of grasping features to generate a dynamic grasping dataset with speed information, thereby solving the problem of missing training data for deep learning models in dynamic grasping tasks.

[0035] (2) The sliding window self-attention mechanism proposed in the present invention combines the local attention capability of the convolution kernel and the feature extraction capability of the attention mechanism, which improves the network computing efficiency while also improving the network prediction accuracy. By building a dynamic grasping network skeleton that replaces the convolution layer with a sliding window multi-head attention mechanism, end-to-end prediction of the object grasping posture and movement speed is achieved, avoiding the time-consuming candidate grasping sampling and evaluation process.

[0036] (3) The present invention adopts a position-based visual servo closed-loop control scheme to replace the traditional open-loop grasping scheme, thereby improving the robot arm's ability to resist changes in objects and the environment, enabling the robot arm to accomplish more flexible and complex dynamic grasping planning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the robot dynamic grasping process in an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a dynamic capture neural network in an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of the calculation process of the sliding window self-attention mechanism in an embodiment of the present invention;

[0040] Figure 4 Schematic diagram of a visual servo closed-loop grasping solution in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0042] like Figure 1 As shown, the robot visual servo dynamic grasping method based on the sliding window self-attention mechanism of the present invention includes a physical grasping environment, a dynamic grasping neural network and a visual servo closed-loop grasping strategy.

[0043] The physical grasping environment is a robot, a two-finger parallel adaptive gripper, a depth camera, and a set of objects to be grasped; the robot and the two-finger parallel adaptive gripper are the main actuators for grasping; the depth camera is responsible for collecting RGB-D image information of the grasped object. In this embodiment, the robot is a six-axis collaborative robotic arm, the depth camera is a camera that can collect color pictures and 2.5D depth pictures, and the set of objects to be grasped is an object randomly placed horizontally in the robot workspace and has a certain movement speed. The depth camera adopts the installation method of eye on hand, that is, the camera is fixed relative to the end of the robot. The homogeneous transformation matrix from the camera coordinate system to the end coordinate system of the robotic arm obtained by the hand-eye calibration technology can obtain the position and posture of the graspable features of the object relative to the coordinate system of the robotic arm base, and then the end execution coordinate system of the robotic arm is overlapped with the posture to perform grasping.

[0044] The dynamic crawling neural network is based on the sliding window self-attention mechanism to build the network structure, which is composed of multiple block structures stacked, such as Figure 2 As shown in the figure. Each block structure connects the sliding window multi-head attention mechanism layer and the convolution layer through two residual modules, where the sliding window size is fixed to 3×3 and the convolution kernel size is 1×1, which plays the role of a fully connected layer. A 2×2 downsampling stage is added between blocks to reduce the image width and height to half of the original and the number of channels to twice the original. The network takes a series of depth maps sampled uniformly in time as input, and outputs the object grasping posture at a certain moment in the middle and the object movement speed at the pixel scale. The plane position of the intermediate grasping posture and the position bias brought by the speed are added as the final grasping position of the object.

[0045] The theoretical calculation process of the sliding window self-attention mechanism is as follows Figure 3 As shown, for input X∈R C×H×W , we decompose it into X = X 11 ,X 12 ,…,X HW , where X ij ∈R C×1×1 For each slice X ij , we calculate its attention with nearby slices, namely X ij With Xpq Attention, where p∈[ik,i+k], q∈[jk,j+k], M=2k+1 is the window size. Use three different convolutional layers with a convolution kernel size of 1×1 to map the input X to three different spaces Q, K, and V. First, calculate the correlation coefficient between the vector at ij and all vectors in the window centered on ij, that is, calculate Q pq With K ij The vector inner product of , and all correlation coefficients are normalized using the Softmax function, and then all vectors inside the window are weighted averaged according to the above correlation coefficients as the final output value. The formula is as follows:

[0046]

[0047] So far, the attention calculation of the feature vector at the input feature ij has been completed. Repeat the above steps at all pixels to complete the sliding window self-attention calculation operation for all feature maps. When we perform self-attention calculation inside the window, the role of each window can be compared to a convolution calculation performed by the convolution kernel - self-attention calculation is performed with each pixel slice as the center, which is consistent with the inner product operation between the convolution kernel and the feature map; repeated self-attention calculation for all pixels is consistent with the sliding of the convolution kernel on the feature map. Considering that the final output of the network contains two parts, one is the object movement speed at the pixel scale, and the other is the score when grasping at k different angles. Therefore, we use k binary classification methods for training, that is, for each angle, we can get a binary classification score, that is, whether the angle can be successfully grasped. The network training process is defined as follows:

[0048]

[0049] Among them, D t ,h is the depth image sequence and grasping height input by the network, q∈{0,1} is the binary classification label of the grasping angle, is the binary classification score of the grasp at the i-th angle output by the network, V is the object movement speed predicted by the network, and L c is the cross entropy loss, L r is the square loss, and λ is the loss weight.

[0050] Visual servo closed-loop grasping strategy Figure 4As shown. The final grasping posture of the target object can be obtained through dynamic grasping. The posture of the current end gripper of the robot can be obtained based on the robot controller. The error between the grasping posture of the target object and the current posture of the end gripper of the robot arm is used as the input of the visual servo controller, and the movement speed required for the end of the robot arm to follow the target object is output. Based on the end movement speed, the Jacobian matrix of the robot arm can be used to find the inverse kinematic solution, and the movement speed of each joint of the robot arm can be obtained and handed over to the speed controller to control the robot arm, so as to continuously approach the target object. When certain conditions are met, such as when the grasping height is less than a certain value, it can be determined that the robot has basically overlapped with the object, and the robot arm and gripper can be controlled to complete the grasping.

[0051] The present invention also discloses a robot dynamic grasping device based on a sliding window self-attention mechanism, comprising the following modules:

[0052] Install the calculation module: Install the camera and the robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology;

[0053] Preprocessing module: preprocesses the grasping static data set based on affine transformation to generate the grasping dynamic data set with speed information;

[0054] Model training module: Build a dynamic grasping network model based on the sliding window multi-head self-attention mechanism, design a loss function that takes into account both the object's movement speed and grasping posture, and complete the network model weight training;

[0055] Input-output module: deploys the trained model into the robot grasping environment, takes a set of continuously captured depth images as input, and outputs the optimal grasping posture and object movement speed;

[0056] Design and implementation module: Design a position-based visual servo grasping control solution to realize the follow-up movement of the robot end effector to the object to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

[0057] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0058] It should be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A robot dynamic grasping method based on sliding window self-attention mechanism, characterized in that: The steps include: Step S1: Install the camera and the robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology; Step S2: preprocessing the captured static data set based on affine transformation to generate a captured dynamic data set with speed information; Step S3: Construct a dynamic grasping network model based on the sliding window multi-head self-attention mechanism, design a loss function that takes into account both the object's movement speed and grasping posture, and complete the network model weight training; Step S4: deploy the trained model into the robot grasping environment, take a set of depth images obtained by continuous shooting as input, and output the optimal grasping posture and object movement speed; Step S5: Design a position-based visual servo grasping control scheme to achieve the follow-up movement of the robot end effector to the object to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

2. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The installation of the camera and the robotic arm in step S1, using the camera as the eyes and the robotic arm as the hands, specifically includes the following two methods: (a) Eye outside the hand: The eye outside the hand means that the camera is separated from the robotic arm, and the camera is fixed relative to the base of the robotic arm. No matter how the robotic arm moves, the camera always maintains the same position and posture; (b) Eye on hand: The eye on hand means that the camera is fixed on the end of the robotic arm. The camera is fixed relative to the end of the robotic arm and moves with the movement of the robotic arm. In order to avoid obstruction of the camera's field of view during the movement of the robotic arm and objects moving out of the camera's field of view, the eye on hand installation method is preferably adopted.

3. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The hand-eye calibration technology in step S1 is implemented by the following steps: Step S11: Fix the camera on a camera bracket installed at the end of the robotic arm; Step S12: prepare a checkerboard calibration plate with an accuracy of ±0.01 mm, clear and non-reflective, and place it horizontally on a work platform within the camera's field of view; Step S13: Keep the calibration plate and the robot base stationary, and control the end of the robot to move multiple times through the teach pendant, and ensure that the calibration plate appears completely in the camera's field of view during the movement; Step S14: Every time the end of the robotic arm moves to a new position, the calibration plate image taken by the camera and the current position of the end of the robotic arm relative to the base are saved; Step S15: Calculate the homogeneous transformation matrix from the camera coordinate system to the robot end coordinate system based on the recorded data and the relationship between the coordinate systems.

4. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: In step S2, a grasping dynamic data set with speed information is generated, specifically, an affine transformation process including only translation is performed on the local depth image at the center of the grasping position given by the existing robot intelligent grasping data set, and the center point of the affine transformation is the center point of the local depth image; the original local image D without any transformation is recorded as time 0, and an affine transformation is performed every Δt time on the [0, kΔt] time segment, and the formula is as follows: d t =Affine(d0,M t ) Affine(A,B) represents the affine transformation of image A using matrix B. t ∈R 3×3 is the affine transformation matrix, c∈R 2×1 To simulate the speed of motion, E∈R 2×2 represents the unit matrix, t∈[0,kΔt] is the time of affine transformation; the grasping dynamic data set with speed information generated based on the above method includes a time series local depth image with a time length of t, a moving speed in pixels, a grasping angle, a grasping height, and a grasping quality at time 0.

5. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The step S3 constructs a dynamic crawling network model based on a sliding window multi-head self-attention mechanism, specifically: the dynamic crawling network model refers to the Swin-Transformer model and is composed of a number of stacked block structures; each block structure is connected to the sliding window multi-head attention mechanism layer and the convolution layer through two residual modules, wherein the sliding window size is fixed to 3×3, and the convolution kernel size is 1×1 for the fully connected layer; a 2×2 downsampling stage is added between blocks to reduce the image width and height to half of the original, and the channel to twice the original.

6. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The robot grasping environment in step S4 includes a six-degree-of-freedom robotic arm, a two-finger gripper, a depth camera, and an object to be grasped; the robotic arm adopts a top-down grasping method, that is, the two-finger gripper is perpendicular to the grasping plane.

7. The robot visual servo dynamic grasping method based on sliding window self-attention mechanism according to claim 1 is characterized in that: The visual servo grasping control scheme in step S5 adopts position-based visual servoing, which uses the error between the current posture information of the robot arm and the target posture information predicted by the network as the input of the controller and outputs the movement speed required for the end of the robot arm to follow the target.

8. A robot dynamic grasping device based on sliding window self-attention mechanism, characterized in that: Includes the following modules: Install the calculation module: Install the camera and the robotic arm, and calculate the transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the hand-eye calibration technology; Preprocessing module: preprocesses the grasping static data set based on affine transformation to generate the grasping dynamic data set with speed information; Model training module: Build a dynamic grasping network model based on the sliding window multi-head self-attention mechanism, design a loss function that takes into account both the object's movement speed and grasping posture, and complete the network model weight training; Input-output module: deploys the trained model into the robot grasping environment, takes a set of continuously captured depth images as input, and outputs the optimal grasping posture and object movement speed; Design and implementation module: Design a position-based visual servo grasping control solution to realize the follow-up movement of the robot end effector to the object to be grasped, and execute the grasping action of the moving object when the grasping height is less than the set threshold.

Images