A deep reinforcement learning-based robot arm pushing and grabbing cooperation method

Abstract

This invention discloses a collaborative push-grasp method for robotic arms based on deep reinforcement learning, belonging to the field of visual robot grasping. It addresses the problems of low sample utilization in the early stages of model-free training, the inability to highlight sample importance during experience pool replay, and the resulting low grasping success rate due to unclear grasping targets. This invention selects the object with the lowest grasping difficulty as the target object by calculating the target center dispersion metric and generates a corresponding mask image based on the target object. A prediction model based on an FCN network and DQN algorithm framework is established. Features are extracted from the RGB-D image, RGB-D height image, and mask image through the network backbone. G-Net and P-Net are used for feature fusion and upsampling to obtain two Q-value heatmaps. A priority experience replay method is used to sample from the experience pool and update the model parameters, ultimately obtaining the prediction model to complete the push-grasp operation in the task scenario.

Description

Technical Field

[0001] This invention belongs to the field of visual robot grasping technology, specifically, it relates to a robotic arm push-grasp collaborative method based on deep reinforcement learning. Background Technology

[0002] Traditional industrial robots are well-suited for tasks involving fixed trajectories. However, real-world environments often present unknown objects and unstructured work environments, and traditional algorithms have limitations in handling such problems or resisting interference. Therefore, planning algorithms with adaptive energy have gained attention from researchers. For example, features can be manually extracted from previously captured images, and then traditional methods can be used to learn the mapping relationship between the extracted features and the grasping pose.

[0003] Robotic grasping can be broadly categorized into two types: analytical methods and empirical methods. Analytical methods require modeling the object's 3D form to find a relatively stable grasping bounding box or grasping points. However, in practice, it is often difficult to accurately model the object, and this method has low versatility and is difficult to transfer to other objects. With the rapid advancement of deep learning technology, with convolutional neural networks as a typical example, empirical robotic grasping methods have become a research hotspot in recent years. Common deep learning-based robotic grasping control algorithms, such as the sliding window method, require data annotation of images, which consumes a significant amount of annotation time. Furthermore, deep learning-based robotic grasping control methods have a low success rate when grasping objects in densely packed scenes, resulting in poor versatility for this type of method. The robot push-grasp collaborative control method based on deep reinforcement learning adopts a model-free training approach. In the early stage of training, there are too few positive samples, resulting in low sample utilization. Furthermore, the experience pool replay process often fails to highlight the importance of samples. To achieve a high grasping success rate, a large amount of training cost is required, which cannot achieve satisfactory training results. At the same time, the unclear grasping target leads to a low grasping success rate. Summary of the Invention

[0004] The technical problem to be solved by this invention is:

[0005] Existing methods suffer from low sample utilization in the early stages of training due to insufficient active samples, and the experience pool replay process often fails to highlight the importance of samples, thus failing to achieve satisfactory training results. At the same time, the unclear capture target leads to a low capture success rate.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] This invention provides a robotic arm push-grasp collaborative method based on deep reinforcement learning, comprising the following steps:

[0008] Step 1: Acquire RGB-D images from the task scene, perform Jacobian transformation on the images, and preprocess the images to obtain RGB-D height images;

[0009] Step 2: Perform semantic segmentation on the RGB-D height image to identify the position information of each object. Calculate the target center dispersion measure based on the position information of the objects, select the object with the least difficulty in grasping as the target object, and generate the corresponding mask image based on the target object.

[0010] Step 3: Establish a prediction model based on the FCN network and the DQN framework. The FCN network includes the network backbone, G-Net and P-Net network branches.

[0011] Step 4: Extract features from the RGB-D image, RGB-D height image, and mask image using the network backbone. Use G-Net and P-Net for feature fusion and upsampling to obtain the Q-value heatmaps of the two images.

[0012] Step 5: Generate a crawling strategy using a greedy strategy;

[0013] Step 6: Execute the grabbing action based on the grabbing strategy, calculate the reward of the action according to the scene feedback, sample a sample from the experience pool using the priority experience replay method, calculate its TD-error, update the prediction model parameters, until the preset training number threshold is reached.

[0014] Step 7: Use the trained prediction model to perform push and grab operations in the task scenario.

[0015] Furthermore, the image preprocessing described in step one includes performing bilateral filtering on the image, performing erosion, dilation, opening, and closing operations on the filtered image, and finally performing binarization on the image.

[0016] Furthermore, the calculation of the target center dispersion measure based on the object's position information in step two is specifically calculated using the following formula:

[0017]

[0018] In the formula, p a and p b Let represent the coordinates of block a and block b respectively, k represent the total number of blocks in the scene, and t represent the degree of dispersion of the blocks.

[0019] Furthermore, the network backbone mentioned in step four is DenseNet-121. The network backbone is used to extract features from the RGB-D image, RGB-D height image and mask image respectively to obtain a feature map with 3072 channels.

[0020] Furthermore, both G-Net and P-Net contain two 1x1 Conv2d layers and two BatchNorm2d layers. The 3072-channel feature map is reduced to 64 channels through the first 1x1 convolutional layer, and the 64-channel feature map is reduced to 1 channel through the second 1x1 convolutional layer. Then, the image resolution is magnified to 16 times through upsampling.

[0021] Furthermore, step five specifically includes the following process:

[0022] The mask image is multiplied by the Q-value heatmap, the maximum predicted value Q in the resulting image is converted into a motion primitive, which includes the action execution location (x, y, z) and the action angle, thus forming the current state's grasping strategy.

[0023] Furthermore, in step six, the samples in the experience pool are the set (s,a,r,s′) of the scene image s before the action is performed, the scene image s′ after the action is performed, the pose information a of the action and the reward information r.

[0024] Furthermore, in step six, the prediction model parameters are updated using the priority empirical replay method, and the sampling probability of this group is calculated based on the TD-error, i.e.:

[0025] P sample ∝|δ t |+ε

[0026] In the formula, P sample |δ represents the probability of a sample being sampled. t | represents the error estimated by the neural network, and ε represents the hyperparameter.

[0027] Furthermore, in step six, the Sumtree method is used to sample and update priorities.

[0028] Furthermore, the Sumtree method is used for priority sampling and update sampling, specifically as follows: First, a random number in the interval [0, sum] is generated. Then, starting from the root node, the left or right child node is selected according to the size of the random number until the leaf node is reached. The leaf node is the sampled experience. During the update, only the affected nodes are updated, that is, all nodes on the path from the leaf node to the root node. The update amount of each node is equal to the sum of the update amounts of its left and right child nodes.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] I. This invention proposes an index for measuring target center dispersion, which can determine the dispersion of objects. During training, it can be used as both a decision priority standard for the GNET module and a reward soft function for the PNET. This index encourages the decision module to select target objects with lower difficulty, increasing the task completion rate in the early stages of neural training, enriching the concentration of positive samples, and accelerating the training speed.

[0031] Second, this invention improves the backbone structure of the fully convolutional network by adding a backbone network for feature extraction of the mask image and achieving multi-feature fusion by designing a multi-input network structure, thereby increasing the processing power of the neural network.

[0032] Third, this invention improves the sampling strategy of experience pool replay by replacing the uniform sampling method used in existing reinforcement learning algorithms with an importance sampling method, thereby improving the effect of parameter update. Attached Figure Description

[0033] Figure 1 This is a flowchart of the robotic arm push-grab collaborative algorithm framework based on deep reinforcement learning in an embodiment of the present invention;

[0034] Figure 2 This is a diagram of the FCN network model in an embodiment of the present invention;

[0035] Figure 3 This is a flowchart of the experience playback process in an embodiment of the present invention;

[0036] Figure 4 This is a flowchart of the training iteration process in an embodiment of the present invention;

[0037] Figure 5 This is a decision flowchart of the G-NET and P-NET modules in this embodiment of the invention;

[0038] Figure 6 This is a schematic diagram of the Sumtree structure in an embodiment of the present invention;

[0039] Figure 7 These are schematic diagrams of multiple complex scenario test cases in embodiments of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are merely some, not all, of the embodiments or examples of the present invention. All other embodiments or examples obtained by those skilled in the art based on the embodiments or examples of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0041] Specific Implementation Plan 1: (e.g.) Figure 1 and Figure 6 As shown, this invention provides a robotic arm push-grasp collaborative method based on deep reinforcement learning, comprising the following steps:

[0042] Step 1: Acquire RGB-D images in the task scene, extract 3D point cloud information from the images, perform Jacobian transformation on the images, and preprocess the images except for pixels that exceed the boundary to obtain a vertical RGB-D height image from top to bottom.

[0043] Step 2: Perform semantic segmentation on the RGB-D height image to identify the position information of each object, calculate the target center dispersion measure based on the position information of the objects, determine the discreteness of each object, select the object with the least difficulty in grasping as the target grasping object, and generate the corresponding mask image based on the target object.

[0044] Step 3: Establish a prediction model based on the FCN network and the DQN framework. The FCN network includes the network backbone, G-Net and P-Net network branches.

[0045] Step 4, as follows Figure 4 As shown, features are extracted from the RGB-D image, RGB-D height image and mask image through the network backbone. G-Net and P-Net are used for feature fusion and upsampling to obtain two Q-value heatmaps. The target mask is used to limit the invalid areas in the Q-value heatmaps.

[0046] Step 5, as follows Figure 5 As shown, a greedy strategy is used to generate the crawling strategy;

[0047] Step Six, as Figure 3 As shown, the grasping action is executed based on the grasping strategy, and the reward for the action is calculated based on the scene feedback. When the data in the experience pool reaches a threshold, new data is used to replace the old data. Whenever stochastic gradient descent (SGD) is performed to minimize the loss function, a sample is sampled from the experience pool using the priority experience replay method, its TD-error is calculated, the stochastic gradient is calculated, and the prediction model parameters are updated using the stochastic gradient descent method until the preset training number threshold is reached.

[0048] Step 7: Use the trained prediction model to perform push and grab operations in the task scenario.

[0049] To enhance the success rate of object capture and accelerate training efficiency, this invention employs an image mask of the target object to provide prior knowledge to the neural network. During the research process, capturing objects one by one from sparse to dense mapping effectively improves the success rate; therefore, this invention proposes a decision-oriented prior mask design.

[0050] Specific Implementation Scheme Two: The image preprocessing described in Step One includes bilateral filtering, followed by erosion, dilation, opening, and closing operations on the filtered image, further image enhancement, and finally binarization. All other aspects of this scheme are the same as in Specific Implementation Scheme One.

[0051] Since bilateral filtering removes noise while clearly preserving image edge information and resulting in good overall image clarity, this implementation scheme adopts bilateral filtering. After image preprocessing, an enhanced task scene image from the vertical viewpoint of the workspace is obtained.

[0052] Specific Implementation Plan Three: The target center dispersion measure is calculated based on the object's position information as described in Step Two. The specific calculation formula is as follows:

[0053]

[0054] In the formula, p a and p b Let represent the coordinates of block a and block b, respectively; k represents the total number of blocks in the scene; and t represents the dispersion of the blocks. This implementation scheme is otherwise the same as specific implementation scheme one.

[0055] An object with a larger target center dispersion metric indicates a larger grasping space and thus a lower grasping difficulty. This invention selects the object with the largest target center dispersion metric as the grasping target to design the mask. The mask image can improve the data processing capability of the subsequent neural network; on the other hand, it can limit invalid action areas by performing a dot product with the output image of the neural network.

[0056] Specific Implementation Plan Four: (e.g.) Figure 2 As shown, the DenseNet-121 network backbone described in step four extracts features from the RGB-D image, RGB-D height image, and mask image, respectively, to obtain a 3072-channel feature map. In this implementation, the image undergoes an affine transformation, rotating it at the same angle to create 16 images with different viewpoints, which are then input into the backbone network, ultimately resulting in 16 different Q-value heatmaps. Other aspects of this implementation are the same as in specific implementation scheme three.

[0057] Specific implementation plan five: Since the grasping and pushing actions have different effects and goals, they are divided into two network branches to evaluate and provide feedback on them respectively.

[0058] like Figure 2 As shown, both G-Net and P-Net contain two batch normalization (BN) units, a rectified linear unit (ReLU), and a 1x1 convolutional block. Each branch first performs batch normalization to stabilize the data. The first 1x1 Cov2d reduces the dimensionality of the 3072-channel feature map to 64 channels. The second 1x1 Cov2d in each branch reduces the dimensionality of the 64-channel feature map to 1 channel. Then, an upsampling bilinear interpolation layer is used to enlarge the resolution of the resulting image, increasing the image resolution by 16 times. Other aspects of this implementation are the same as in specific implementation scheme four.

[0059] Specific Implementation Plan Six: Step Five specifically includes the following processes:

[0060] The mask image is multiplied by the Q-value heatmap to limit invalid regions in the Q-value prediction map and improve sampling efficiency in the limited action space. The maximum predicted value Q in the resulting image is converted into a motion primitive containing the action execution position (x, y, z) and the action angle, which is the current state's grasping strategy. This implementation scheme is otherwise the same as specific implementation scheme five.

[0061] Specific Implementation Scheme Seven: In Step Six, the samples in the experience pool consist of the scene image s before the action is executed, the scene image s′ after the action is executed, and the set (s,a,r,s′) of the pose information a and reward information r of the action. The scene image s before the action is executed includes a color height map, a depth height map, and a target mask map. This implementation scheme is otherwise the same as Specific Implementation Scheme Six.

[0062] This invention designs independent network parameters and reward functions for grasping and pushing, enabling the network to learn different evaluation metrics based on different actions. The reward function for the grasping module is shown in the following formula:

[0063]

[0064] The reward function for the push module is shown below:

[0065]

[0066] Specific implementation plan eight: In step six, the priority experience replay method is used to update the prediction model parameters, and the sampling probability of this group is calculated based on the TD-error, that is:

[0067] P sampl e∝|δ t |+ε

[0068] In the formula, P sample |δ represents the probability of a sample being sampled. t | represents the error estimated by the neural network, and ε represents the hyperparameter.

[0069] Specific Implementation Plan Nine: (e.g.) Figure 6 As shown, step six uses the Sumtree method for priority sampling and updating. A Sumtree is a data structure used to store and update priorities in priority experience replay. It is a complete binary tree where the value of each node is equal to the sum of the values ​​of its left and right child nodes, and the root node's value is equal to the sum of the values ​​of all nodes, which is the sum of the priorities of all experiences. This implementation scheme is otherwise the same as specific implementation scheme eight.

[0070] This invention improves upon common priority experience playback methods, thereby replacing uniform sampling methods with non-uniform priority sampling.

[0071] Specific Implementation Scheme Ten: The Sumtree method is used for priority sampling and update sampling. Specifically, a random number within the interval [0, sum] is generated first. Then, starting from the root node, the left or right child node is selected based on the magnitude of the random number, continuing until a leaf node is reached. The leaf node represents the sampled experience. This sampling process ensures that the sampling probability of each experience is proportional to its priority while maintaining a certain degree of randomness. During updates, only the affected nodes are updated, i.e., all nodes on the path from the leaf node to the root node. The update amount of each node is equal to the sum of the update amounts of its left and right child nodes. This update process avoids traversing the entire tree, improving computational efficiency. The rest of this implementation scheme is the same as Specific Implementation Scheme Nine.

[0072] Example 1

[0073] like Figure 7 As shown, this embodiment sets up test cases for various complex scenarios in the simulation scene, and uses representative baseline algorithms: the visual grasping algorithm VG and the goal-oriented push-grab collaborative algorithm T-VPG as control algorithms. In order to quantitatively evaluate the planning effect of the method of the present invention and each control algorithm, the grasping success rate (GSC), task achievement (TA), and average action degree (AAD) are used as evaluation indicators. The experimental results of the algorithm of the present invention in various test cases are shown in Table 1, the experimental results of the visual grasping algorithm VG in various test cases are shown in Table 2, and the experimental results of the goal-oriented push-grab collaborative baseline algorithm T-VPG in various test cases are shown in Table 3.

[0074] Table 1

[0075]

[0076]

[0077] Table 2

[0078]

[0079] Table 3

[0080]

[0081] As shown in Tables 1, 2, and 3, the algorithm of this invention achieves an average task completion rate of approximately 91%, far exceeding that of other algorithms. Furthermore, the grasping success rate of this invention is higher than the two control algorithms, and the average number of actions is less, demonstrating superior performance. In structured, dense scenes, objects often lack grasping space, limiting the effectiveness of the visual grasping algorithm (VG). However, it can be seen that the T-VPG algorithm outperforms the visual grasping algorithm (VG) in the six case studies, achieving an average completion rate of over 62%, highlighting the importance of the push action.

[0082] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A robotic arm push-grasp collaborative method based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Acquire RGB-D images from the task scene, perform Jacobian transformation on the images, and preprocess the images to obtain RGB-D height images; Step 2: Perform semantic segmentation on the RGB-D height image to identify the position information of each object. Calculate the target center dispersion measure based on the position information of the objects, select the object with the least difficulty in grasping as the target object, and generate the corresponding mask image based on the target object. Step 3: Establish a prediction model based on the FCN network and the DQN framework. The FCN network includes the network backbone, G-Net and P-Net network branches. Step 4: Extract features from the RGB-D image, RGB-D height image, and mask image using the network backbone. Use G-Net and P-Net for feature fusion and upsampling to obtain the Q-value heatmaps of the two images. Step 5: Generate a crawling strategy using a greedy strategy; Step 6: Execute the grabbing action based on the grabbing strategy, calculate the reward of the action according to the scene feedback, sample a sample from the experience pool using the priority experience replay method, calculate its TD-error, update the prediction model parameters, until the preset training number threshold is reached. Step 7: Use the trained prediction model to perform push and grab operations in the task scenario; The calculation of the target center dispersion measure based on the object's position information in step two is specifically calculated using the following formula: In the formula, and Let represent the coordinates of block a and block b respectively, k represent the total number of blocks in the scene, and t represent the degree of dispersion of the blocks; In step six, the priority empirical replay method is used to update the prediction model parameters, and the sampling probability of the sample is calculated based on the TD-error, i.e.: In the formula, This represents the probability that a sample will be sampled. This represents the error in the neural network estimation. This represents hyperparameters.

2. The robotic arm push-grasp collaborative method based on deep reinforcement learning according to claim 1, characterized in that, The image preprocessing described in step one includes performing bilateral filtering on the image, performing erosion, dilation, opening, and closing operations on the filtered image, and finally binarizing the image.

3. The robotic arm push-grasp collaborative method based on deep reinforcement learning according to claim 2, characterized in that, The network backbone mentioned in step four is DenseNet-121. The network backbone is used to extract features from the RGB-D image, RGB-D height image and mask image respectively to obtain a feature map with 3072 channels.

4. The robotic arm push-grasp collaborative method based on deep reinforcement learning according to claim 3, characterized in that, Both G-Net and P-Net contain two 1x1 Conv2d layers and two BatchNorm2d layers. The first 1x1 convolutional layer reduces the dimensionality of the 3072-channel feature map to 64 channels, and the second 1x1 convolutional layer reduces the dimensionality of the 64-channel feature map to 1 channel. Then, the image resolution is magnified by 16 times through upsampling.

5. The robotic arm push-grasp collaborative method based on deep reinforcement learning according to claim 4, characterized in that, Step five specifically includes the following processes: The mask image is multiplied by the Q-value heatmap, the maximum predicted value Q in the resulting image is converted into a motion primitive, which includes the action execution location (x, y, z) and the action execution angle, thus forming the current state's grasping strategy.

6. The robotic arm push-grasp collaborative method based on deep reinforcement learning according to claim 5, characterized in that, In step six, the samples in the experience pool are the scene image s before the action is performed and the scene image after the action is performed. The set of pose information 'a' and reward information 'r' for the action performed .

7. The robotic arm push-grasp collaborative method based on deep reinforcement learning according to claim 6, characterized in that, In step six, the Sumtree method is used to sample and update priorities.

8. The robotic arm push-grasp collaborative method based on deep reinforcement learning according to claim 7, characterized in that, The Sumtree method is used for priority sampling and update sampling, specifically as follows: First, a random number in the interval [0, sum] is generated. Then, starting from the root node, the left or right child node is selected according to the size of the random number until the leaf node is reached. The leaf node is the sampled experience. During the update, only the affected nodes are updated, that is, all nodes on the path from the leaf node to the root node. The update amount of each node is equal to the sum of the update amounts of its left and right child nodes.

Images