A robot grasping method based on deep high-resolution convolutional neural network

Abstract

This invention relates to the field of robot vision grasping technology, and discloses a robot grasping method based on a deep high-resolution convolutional neural network to solve the problem of spatial information loss when encoding input images at low resolution, thereby enhancing its ability to grasp small or stacked objects. It also reduces decoder-introduced errors when dealing with various object types or objects far from the camera. This invention effectively improves the performance of perception tasks by using parallel branches and fusing information between different resolutions, rather than stacking convolutional layers with single branches. The framework provides accurate information to the motion planning module in the initial stage, improving trajectory smoothness and avoiding singularities in the robotic arm. It provides neural network design guidelines for robot perception tasks, such as high-resolution representation and lightweight design, addressing the challenges of different operating scenarios. It also better solves the problem of long distances between the initial camera and the target.

Description

Technical Field

[0001] This invention relates to the field of robot vision grasping technology, specifically to a robot grasping method based on a deep high-resolution convolutional neural network. Background Technology

[0002] Since the world's first robot was invented in the 1960s, robots have played an increasingly important role in fields such as intelligent manufacturing and home services. Picking and placing are among the fundamental skills of robot operation. Selecting the appropriate grasping position and posture based on visual perception is crucial for such operations.

[0003] A complete robot grasping process can be divided into three steps: grasping detection, trajectory planning, and motion control. Grasping detection is the process of locating the object and generating its grasping posture, which can be broadly divided into two categories: geometric analysis-based methods and data-driven methods. Geometric analysis-based methods typically perform grasping detection by analyzing and calculating the geometry and kinematics of a physical model. Data-driven methods are mainly based on machine learning methods and usually require a large number of manually labeled samples.

[0004] Deep CNNs, as variants of Convolutional Neural Networks (CNNs), have become the engine of visual perception and recognition. Significant advancements, including LeNet, AlexNet, VGGNet, and ResNet, have improved the perceptual capabilities of these models. Although these models were initially designed for natural image classification, much of the subsequent work in the robotics community has adopted the architectures used in these models as their backbone. Unlike general visual classification, which only requires semantic representations of objects, robotic perception tasks rely more heavily on precise spatial information about objects.

[0005] High-resolution representation is crucial for vision-based robotic grasping problems. Existing methods typically encode the input image into a low-resolution representation via a sub-network and then reconstruct the high-resolution representation. This results in the loss of spatial information, and errors introduced by the decoder become more severe when considering multiple types of objects or objects far from the camera. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a robot grasping method based on a deep high-resolution convolutional neural network.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A robot grasping method based on a deep high-resolution convolutional neural network includes the following steps:

[0009] Step 1: Use an RGB-D camera to collect color and depth images of several pairs of robotic arms grasping objects to form a dataset. Then, refer to the Cornell grasping dataset to label the grasping position and posture on the collected color and depth images as labels for each sample in the dataset.

[0010] Step 2: Based on the samples and labels in the dataset, design a network skeleton DHRNet that considers the mixing of feature maps of multiple resolutions, and train the network skeleton by fitting the dataset. The input of the network skeleton DHRNet is a color image and a depth image, and the output includes a grasping quality map Q, an angle feature map Φ, and a width feature map W. The grasping quality map Q is used to reflect the probability of successful grasping when each pixel in the depth image is used as the center point p of the grasping rectangle. The angle feature map Φ is used to describe the rotation angle to be performed by the end effector at the center point p of the grasping rectangle. The width feature map W is used to describe the image of the width of the end effector at each center point p of the grasping rectangle. Based on Q, Φ, and W, the grasping rectangle g of the end effector is obtained, and the optimal grasping position and pose of the end effector are predicted.

[0011] Step 3: During the grasping process, repeatedly capture the target image and call the trained network skeleton DHRNet. Use the best grasping position predicted by the network skeleton DHRNet as the next trajectory point of the end effector until the end effector moves to a position at a set height from the target. The end effector grasps the target based on the best grasping position and pose predicted by the network skeleton DHRNet.

[0012] Furthermore, in step two, when obtaining the grasping rectangle g of the end effector based on feature maps Q, Φ, and W:

[0013] The end effector is a parallel gripper; the five-dimensional grasping representation of the parallel gripper is used to describe the position and posture of the robotic arm's grasping;

[0014] The five-dimensional grasping representation includes the center point p of the grasping rectangle (x, y), the rotation angle θ, the width ω of the parallel clamp at p, and the height difference z between the parallel clamp and the grasping target; where x and y are the abscissa and ordinate of the center point p of the grasping rectangle, respectively.

[0015] Define the grab rectangle g as:

[0016] g=(x,y,θ,ω,z),(x,y)∈W,θ∈Φ,ω∈W.

[0017] Furthermore, in step two, when predicting the optimal grasping position and orientation of the end effector, the maximum value in the grasping quality map Q is searched to obtain the coordinates (x, y) of the optimal grasping rectangle center point. * ,y * ) = argmaxQ (x,y), based on coordinates (x * ,y * The width ω and rotation angle θ are obtained from the width feature map W and the angle feature map Φ, respectively. The height difference z between the parallel gripper and the target is measured by the depth camera, and then the optimal gripping position and posture of the robotic arm are predicted.

[0018] Furthermore, in step two, when training the network backbone DHRNet:

[0019] Select N samples (m1, m2, ..., mn) from the dataset. i ,...,m N Train the network backbone DHRNet, m i ∈R H×W Let R be the i-th sample, R be the real number field, and H and W be the height and width of each sample, respectively. The loss function L of the network skeleton is:

[0020]

[0021] Where F is the network skeleton DHRNet, the parameter Θ is the parameter of the neural network, and s i For sample m i The tag.

[0022] Furthermore, in step three, during the grasping process, the target image is repeatedly captured and the trained network skeleton DHRNet is invoked. When the optimal grasping position predicted by the network skeleton DHRNet is used as the next trajectory point of the end effector:

[0023] Let z max z min These represent the initial height of the end effector and the minimum height set for the target distance, respectively.

[0024] Before successfully capturing the target, at each trajectory point p i The DHRNet network skeleton is invoked at the point to obtain the grab rectangle g. i =F(camera(p) i ),Θ), where camera(p i ) is in p i The color image and depth map taken at the location are used to determine the optimal grab position predicted by the DHRNet backbone network as the next trajectory point p. i+1 This leads to the trajectory P = {(p0, p1, ..., p...} i ,...),p i =(p i [0],p i [1],p i [2])∈R 3 ,pi [2]-p j [2])×(ij)>0},p i [0],p i [1],p i [2] are the trajectory points p i The x-axis, y-axis, and z-axis coordinates are used to guide the robotic arm to move the end effector along trajectory P, moving it at a specific speed in the height direction from z... max Descending to z min ;p j [2] represents the trajectory point p j The z-axis coordinate.

[0025] Compared with the prior art, the beneficial technical effects of the present invention are:

[0026] Unlike the encoder-decoder structures primarily used in the past, this invention, for the first time, utilizes high-resolution feature maps to improve the perceptual quality of robot visual grasping, better addressing the problem of long distances between the initial camera and the target. The designed network skeleton, DHRNet, maintains high-resolution feature representations by using parallel branches and fusing information from different resolutions, rather than scaling down the feature map as in traditional encoders. The features learned by the DHRNet skeleton are semantically and spatially strong, which is more conducive to learning accurate spatial location information. The developed online closed-loop robot grasping framework dynamically adjusts the robot's motion based on real-time visual detection. Accurate spatial information combined with motion planning plays a crucial role as rich semantic information. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the structure of the network skeleton DHRNet of the present invention;

[0028] Figure 2 This is a schematic diagram of the possible positions of the robot's end effector in the motion planning of this invention. Detailed Implementation

[0029] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] The present invention provides a robot grasping method based on a deep high-resolution convolutional neural network, which maintains high-resolution representation throughout the robot vision grasping architecture. Its preferred implementation is as follows.

[0031] The present invention discloses a robot grasping method based on a deep high-resolution convolutional neural network, the overall network structure of which is shown in the figure below. Figure 1As shown, this invention designs a novel neural network paradigm for robot visual grasping called DHRNet. By using parallel branches and fusing information between different resolutions, rather than stacking convolutional layers with single branches, it effectively maintains high-resolution representation and repeatedly exchanges information between different resolutions, thus better addressing the problem of long distances between the initial camera and the target. This invention develops an online closed-loop robot grasping framework utilizing DHRNet. This framework dynamically adjusts the robot's motion based on real-time visual detection, using camera visual information as feedback to adjust the trajectory of the manipulating end effector in real time. It generates appropriate grasping postures for scenes with small objects or stacked objects and reduces the probability of the robotic arm entering singularities.

[0032] The visual grasping method based on deep high-resolution convolutional neural networks includes the following steps:

[0033] Step 1: Use an RGB-D camera to acquire a large number of pairs of color images and depth images, and label the acquired images according to the Cornell crawl dataset, marking the crawling position and pose.

[0034] Step 2: Based on the dataset and its labels, design a network skeleton DHRNet that considers the mixing of feature maps of multiple resolutions, and make the network fit the dataset as closely as possible.

[0035] Step two employs a five-dimensional grasping representation using a parallel gripper. The corresponding grasping configuration includes the grasping center point p = (x, y), the rotation angle θ, the width ω of the parallel gripper at p, and the height difference z between the parallel gripper and the grasped target. The complete grasping rectangle g can be defined as:

[0036] g=(x,y,θ,ω,z),(x,y)∈W,θ∈Φ,ω∈W.

[0037] The output of the DHRNet backbone consists of three feature maps Q, Φ, and W, which are the same size as the original input depth map, such as... Figure 1 As shown. Q is introduced to better reflect the probability of successful capture when each pixel in the image is used as the center of the capture rectangle. The parameters in the capture quality map Q take values ​​between 0 and 1. When the value at a certain position in the capture quality map Q is close to 1, it means that the probability of successful capture at that position is higher. Subsequently, by retrieving the capture quality map Q, the optimal capture center point (x) can be obtained. * ,y * ) = argmax Q (x, y). Φ is an image describing the gripping angle to be performed at each point p. Due to the opposite gripping around... The arc is symmetrical, therefore the grasping angle is... The range is given. W is an image describing the gripper width performed at each point p. To achieve depth invariance, the variable z is in the range of [0, 150] pixels and can be converted to a physical measurement using depth camera parameters and measured depth. The optimal gripping center point (x) is obtained from the data. * ,y * The width ω and rotation angle θ of the end effector fixture can be directly obtained from W and Φ.

[0038] To calculate the predicted position and orientation, we only need to search for the maximum value in Q to obtain the coordinates (x, y) of the optimal grab point. * ,y * ) = argmax Q (x, y). Then, based on the coordinates of that point p = (x, y). * ,y * The optimal gripping position and orientation can be predicted directly from the width map W and angle map Φ. The distance z between the end effector and the target object is measured by a depth camera, which takes into account the size of the jaws and the position of the camera.

[0039] The network skeleton DHRNet in this invention, which considers the mixing of feature maps from multiple resolutions, has the following characteristics:

[0040] (1) Parallel connections from high resolution to low resolution at all stages of the network skeleton DHRNet;

[0041] (2) Information exchange across different resolutions to enrich semantic information.

[0042] The network skeleton DHRNet in this invention is obtained by improving the HRNet (High-Resolution Network) network.

[0043] like Figure 1 As shown, the difference between the network skeleton DHRNet in this invention and the traditional HRNet network is that:

[0044] (1) The last layer is replaced with a feature map of the same size as the input map, and an upsampling step is added to generate a grabbing heatmap; the grabbing heatmap includes a grabbing quality map Q, an angle feature map Φ, and a width feature map W;

[0045] (2) Replace the activation function of stage 1 from ReLU with Leaky ReLU;

[0046] (3) Replace the optimizer with the AdamW optimizer.

[0047] Specifically, such as Figure 1As shown, the DHRNet network skeleton begins with convolutional blocks, gradually stacking convolutional blocks of different resolutions and connecting them in parallel. The entire DHRNet network skeleton consists of four stages, each composed of parallel stacked blocks with different feature resolutions, each stacked block being a residual basic block. Information interaction between different residual basic blocks is achieved through a fusion layer, where features are upsampled to a high-resolution branch and vice versa, downsampled to a low-resolution branch. Specifically, the DHRNet network skeleton utilizes 3×3 convolutional kernels and convolutions with a stride of 2 to reduce the resolution of the feature maps and uses bilinear interpolation for upsampling. By mixing convolutional layers of different resolutions in parallel, a high-resolution representation fusion is obtained. Therefore, this invention can maintain high-resolution representations by connecting high-resolution and low-resolution convolutions in parallel and iteratively performing fusion operations between parallel blocks.

[0048] Generally, the features learned by the DHRNet backbone network are strong both semantically and spatially. This is because convolutional blocks of different resolutions are connected in parallel rather than sequentially, which is more conducive to learning accurate spatial location information. The DHRNet backbone network consistently maintains high-resolution feature representations, rather than scaling down feature maps like traditional encoders. Furthermore, information from different branches is continuously fused, forming rich semantic information.

[0049] In a labeled dataset, select N samples (m1, m2, ..., m...). i ,...,m N ), m i ∈R H×W H and W represent the height and width of the sample, respectively. The goal of this invention is to construct a network skeleton F with parameter Θ to ensure that the loss function L is minimized after a certain number of updates to Θ. The loss function L is:

[0050]

[0051] Where s i For sample m i The tag.

[0052] Step 3: Throughout the grasping process, the DHRNet backbone network is repeatedly called to determine the next trajectory point, i.e., the next optimal grasping point p, until the robotic arm gripper is at a set height from the target. The best grasping position and posture are determined by observing from multiple angles. The whole process is a closed loop.

[0053] Step 3 introduces a trajectory planning framework utilizing the DHRNet network skeleton. First, the entire grasping process is a closed-loop process that uses visual information from the camera as feedback to adjust the trajectory of the manipulator's end effector in real time. When the end effector approaches the object perpendicularly, it captures the input image online. Therefore, the selection of camera trajectory points plays a crucial role in the quality of visual inspection. Active perception technology is applied, using a camera fixed to the end of the robotic arm to calculate the suboptimal viewing angle in real time.

[0054] like Figure 2 As shown, initially, the height z of the robot's end effector... max It is fixed, with its horizontal position roughly confined to the vicinity of the workspace center (x, y). During the movement of the robotic arm, if the position of the end effector is higher than z... min The DHRNet will be continuously invoked. If there are no peaks in the obtained grasping quality map Q, a conservative method will be used for horizontal detection. This invention uses a 3×3 matrix as a filter, directly convolving it with the grasping quality map Q to detect the presence of peaks. If the new feature map obtained from the convolution contains points greater than a threshold, the detection result is considered reliable, and a peak exists in the grasping quality map Q. Otherwise, it will actively move horizontally along the high-entropy direction in the grasping quality map Q. The most probable grasping rectangle g is calculated based on the grasping quality map Q, and the peak value of Q is used as the grasping score of g. When the height is not less than z... min At this time, the robotic arm saves all the stored g values ​​during this process into the grasping experience pool, and finally selects the g value with the highest grasping score and clears the grasping experience pool.

[0055] This invention addresses the problem of spatial information loss when encoding input images at low resolution, enhancing the ability to grasp small or stacked objects. It also reduces decoder-introduced errors across various object types or when objects are far from the camera. The novel robotic vision grasping framework designed in this method re-examines the design paradigm of CNN-based robotic perception tasks. Instead of stacking convolutional layers with single branches, it effectively improves perception task performance by using parallel branches and fusing information from different resolutions. This framework provides accurate information to the motion planning module in the initial stage, improving trajectory smoothness and avoiding singularities in the robotic arm. It provides neural network design guidelines for robotic perception tasks, such as high-resolution representation and lightweight design, addressing challenges in various operational scenarios. It also better solves the problem of long distances between the initial camera and the target.

[0056] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0057] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A robot grasping method based on a deep high-resolution convolutional neural network, comprising the following steps: Step 1: Use an RGB-D camera to collect color and depth images of several pairs of robotic arms grasping objects to form a dataset. Then, refer to the Cornell grasping dataset to label the grasping position and posture on the collected color and depth images as labels for each sample in the dataset. Step 2: Based on the samples and labels in the dataset, design a network skeleton DHRNet that considers the mixing of feature maps at multiple resolutions, and train the network skeleton by fitting it to the dataset. The input of the DHRNet network skeleton is a color image and a depth image, and the output includes a capture quality image. Angular feature map and width feature map ; Capture quality image This is used to reflect the use of each pixel in the depth map as the center point of the grab rectangle. The probability of successful capture, angle feature map Used to describe the end effector at the center point of the grasping rectangle. The rotation angle to be performed, and the width feature map. Used to describe the center point of each grab rectangle The image of the end effector width, based on , and Obtain the grab rectangle of the end effector It predicts the optimal grasping position and orientation of the end effector; the network skeleton DHRNet is obtained by improving the HRNet network. The network skeleton DHRNet adopts parallel connections from high resolution to low resolution in all stages and exchanges information between different resolutions; the difference between the network skeleton DHRNet and the HRNet network is: (1) The last layer is replaced with a feature map of the same size as the input image, and an upsampling step is added to generate a capture heatmap, which includes a capture quality map Q, an angle feature map Φ and a width feature map W; (2) Replace the activation function of stage 1 from ReLU with Leaky ReLU; (3) Replace the optimizer with the AdamW optimizer; Step 3: During the grasping process, repeatedly capture the target image and call the trained network skeleton DHRNet. Use the best grasping position predicted by the network skeleton DHRNet as the next trajectory point of the end effector until the end effector moves to a position at a set height from the target. The end effector grasps the target based on the best grasping position and pose predicted by the network skeleton DHRNet.

2. The robot grasping method based on a deep high-resolution convolutional neural network according to claim 1, characterized in that, In step two, based on feature maps , and Obtain the grab rectangle of the end effector hour: The end effector is a parallel gripper; the five-dimensional grasping representation of the parallel gripper is used to describe the position and posture of the robotic arm's grasping; The five-dimensional grasping representation includes the center point of the grasping rectangle. Rotation angle ,exist Width of parallel clamp and the height difference between the parallel gripper and the target being grasped. ;in These are the points to capture the center of the rectangle. The x and y coordinates; Define the grab rectangle for: 。 3. The robot grasping method based on a deep high-resolution convolutional neural network according to claim 2, characterized in that, In step two, when predicting the optimal grasping position and orientation of the end effector, a grasping quality map is searched. The maximum value in the range is used to obtain the optimal coordinates of the center point of the grab rectangle. According to coordinates In the width feature map respectively and angular feature map Get the width and rotation angle The height difference between the parallel fixture and the target The optimal gripping position and posture of the robotic arm are predicted by using depth camera measurements.

4. The robot grasping method based on a deep high-resolution convolutional neural network according to claim 1, characterized in that, In step two, when training the network skeleton DHRNet: Select in dataset Sample Train the network backbone DHRNet. For the i-th sample, For the real number field, and These are the height and width of each sample, and the loss function of the network skeleton. for: ； Where F is the network skeleton DHRNet, and the parameters are... For the parameters of the neural network, For the sample The tag.

5. The robot grasping method based on a deep high-resolution convolutional neural network according to claim 4, characterized in that, In step three, during the grasping process, the target image is repeatedly captured, and the trained network skeleton DHRNet is invoked. When the optimal grasping position predicted by the network skeleton DHRNet is used as the next trajectory point of the end effector: make , These represent the initial height of the end effector and the minimum height set for the target distance, respectively. Before successfully capturing the target, at each trajectory point... The DHRNet network skeleton is invoked to obtain the capture rectangle. ,in Is The color image and depth map captured at the location are used to determine the optimal grab position predicted by the DHRNet backbone network as the next trajectory point. Thus, the trajectory is obtained. , Trajectory points The x-axis, y-axis, and z-axis coordinates are used to guide the robotic arm to move the end effector along trajectory P, at a specific speed in the height direction. Descending to ; Represents trajectory points The z-axis coordinate.

Images