A push-grab reward design method for a grasp affordance improvement Q network
By introducing a grasping availability network into a deep reinforcement learning algorithm, a novel push reward mechanism is designed to address the problem of insufficient evaluation of the effectiveness of push actions, thereby improving the grasping success rate and efficiency, especially in grasping tasks involving tightly stacked objects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QINGDAO UNIV OF TECH
- Filing Date
- 2024-06-17
- Publication Date
- 2026-05-29
Smart Images
Figure CN118544354B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robotic grasping technology, and more specifically to an improvement in grasping availability. Q Network-driven - crawl reward design method. Background Technology
[0002] More and more researchers are applying deep reinforcement learning algorithms to unstructured robot grasping tasks. Through self-supervision, robots interact with the working environment and gradually learn grasping skills from repeated interactions. Grasping methods based on deep reinforcement learning can improve the robot's adaptive grasping ability to a certain extent. Adopting a strategy that combines pushing and grasping provides an important idea for solving the problem of grasping objects in complex environments.
[0003] In practice, it is often possible to push tightly stacked objects to a state of separation before proceeding with the subsequent grasping work, which can significantly improve the grasping success rate. Therefore, modeling these two tasks into a unified multi-task framework to achieve collaborative pushing and grasping is a promising direction for solving the problem.
[0004] To address the aforementioned technical challenges, some researchers have proposed cooperative push-grasping strategies based on deep reinforcement learning, such as the Visual Pushing for Grasping (VPG) model based on deep Q-networks. The VPG model accomplishes the task of grasping tightly stacked objects through the coordinated operation of grasping and pushing actions. It predicts the Q-values of all pixels by building a fully convolutional neural network, compares the Q-values of the grasping and pushing actions, and selects the action with the highest Q-value using a greedy strategy. This method improves grasping efficiency by changing the positional distribution of objects through pushing actions; however, the sparsity of its reward function leads to slow convergence in the early stages. Current reinforcement learning-based push-grasping algorithms still suffer from problems such as insufficient evaluation of the effectiveness of pushing actions, low efficiency of push-grasping coordination, and low grasping success rate. Summary of the Invention
[0005] The purpose of this invention is to provide an improvement in grasp availability. Q The network-driven crawling reward design method addresses the following technical issues:
[0006] How to reduce the problem of low success rate caused by overlapping objects within the crawling area, and improve the rationality of crawling and driving behavior evaluation.
[0007] The objective of this invention can be achieved through the following technical solutions:
[0008] An improvement to crawl availability QA network push-fetch reward design method, the method comprising:
[0009] S1. Acquire RGB-D image information of the working scene and obtain color height image and depth height image through projection transformation;
[0010] S2. Input the color height image and depth height image into the push network FCN respectively. push and crawling network FCN grasp And within the pre-trained availability network, pixel-level cumulative reward expectations are generated sequentially. , and availability value ;
[0011] S3. Determine whether the change in availability exceeds a set threshold after the push action is executed:
[0012] If so, then give a positive reward for the action.
[0013] If not, no action reward will be given;
[0014] The grab availability value, calculated based on the percentage of the smallest grabizable object pixel value in the scene, is used as a set threshold. Further, it is determined whether there are pixels with an availability value of 1 or higher exceeding the set threshold.
[0015] If available, prioritize grabbing objects in the scene; otherwise, perform a pushing action.
[0016] Preferably, the crawling availability network is constructed as follows:
[0017] Pre-train a parallel 121-layer DenseNet network on ImageNet;
[0018] Channel concatenation is performed, followed by two additional 1×1 convolutional layers with non-linear activation functions (ReLU) and spatial batch normalization before upsampling;
[0019] One DenseNet network takes the color channel of the heightmap as input, and another DenseNet network takes the depth channel of the heightmap as input.
[0020] Preferably, the pre-training method of the availability network is as follows:
[0021] Data is collected from no less than 3000 operations during the VPG training process, including color height images, depth height images, and angle and pixel position information of the actions performed. A zero matrix of the same size as the input image is created to establish labels for the grabbing action availability.
[0022] For grasping actions: If the distance between the opposite ends of the robot gripper fingers exceeds a set threshold, then the pixel position label for that grasping action is set to 1;
[0023] Input the crawling action availability label, the color height image and depth height image of the crawling action, and the pixel position information into the built network model respectively;
[0024] After extracting relevant features from the image and performing feature concatenation, the data is normalized and convolutional. Then, upsampling is used to obtain the availability prediction values, and the Huber loss function is used for iterative training.
[0025] The formula for calculating the loss function is as follows:
[0026] ;
[0027] in, These are the parameters of the neural network at iteration i. The target network parameters are fixed and kept constant with each update; gradients are propagated backward through a single pixel p in the network, and actions are calculated and executed through the network's forward propagation. The availability is numerically predicted; the backpropagation loss for all other pixels is 0 at iteration i.
[0028] Preferably, the method for obtaining the color height image and depth height image in step S1 is as follows:
[0029] At the current time t, the depth camera acquires RGB-D images of the working scene. Each pixel of the color image and the depth image is transformed from the camera coordinate system to the robot coordinate system through the intrinsic and extrinsic parameter matrices to obtain a series of point clouds. Then, the point clouds belonging to the working environment are mapped onto a two-dimensional plane along the direction of gravity to obtain the color image height map and the depth image height map.
[0030] Preferably, in step S2:
[0031] The reward function for the pushing action is constructed by summing the availability images of the scene in 16 directions before and after the pushing action, calculating the difference, and comparing it with a set threshold. For pushing actions where the change in the availability value before and after the pushing action is greater than the set availability threshold, the reward is set to 0.5; otherwise, it is 0. The formula for calculating the reward function for the pushing action is as follows:
[0032] ;
[0033] Where i is the direction index corresponding to the availability image. A color height map before the action is initiated. To advance the depth and height map before the action, To produce a color height map after the action, To advance the depth-height map after the action, For the availability image of the i-th direction after the push action is performed, For the availability image of the i-th direction before performing the pushing action, The availability threshold set for this embodiment.
[0034] Preferably, it also includes: designing a reward function for the grasping action:
[0035] For the grasping action, a sparse reward function is used, and the calculation formula is as follows:
[0036] .
[0037] Preferably, the formula for implementing the push and grasping collaborative strategy in step S3 is:
[0038] ;
[0039] in, The model is an RGB-D heightmap image of the scene at the current time t. These are the parameters of the Q network. To drive the cumulative reward expectation for actions, To accumulate the expected reward for the grabbing action, The number of pixels in the availability image with a value not less than 1. To capture the availability threshold.
[0040] Preferably, a reward function for the pushing action and a collaborative mechanism model for the pushing and grasping strategies are set based on the method. The model includes a visual pushing and grasping module based on a deep Q-network and a pre-trained grasping availability network module.
[0041] The beneficial effects of this invention are:
[0042] Based on VPG, this invention redesigns the push reward by introducing a grasp availability network and adopts a better decision-making method to select push and grasp actions, proposing a push reward based on improved grasp availability; it can better evaluate the effectiveness of push behavior and generate more availability areas around the grasp object.
[0043] This invention trains a grasping availability network and designs a novel push reward mechanism from the perspective of grasping availability to evaluate the rationality of the push behavior in a push operation task. The effectiveness of the push behavior is judged by the increase or decrease in the availability region around the grasping object before and after the push. A novel collaborative mechanism model of push and grasping strategies is proposed. This model includes a visual push-grasping module based on a deep Q-network and a pre-trained grasping availability network module. Depth-height images and color-height images generated by projection transformation from an RGB-D camera are input into the push network, grasping network, and pre-trained grasping availability network, respectively, to generate pixel-level cumulative reward expectation values. , and availability value By setting an availability threshold, it is possible to accurately determine whether the current environmental state meets the conditions for grasping, thus providing a basis for action decisions.
[0044] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 This is a diagram illustrating the architecture of the push-grabbing skill learning model based on grasping availability in this invention.
[0047] Figure 2 This invention captures the availability network architecture diagram;
[0048] Figure 3 This is a schematic diagram illustrating the reward design for this invention;
[0049] Figure 4 This is a schematic diagram of the action decision-making mechanism based on crawl availability of the present invention;
[0050] Figure 5 This is a schematic diagram of the simulation environment for this invention;
[0051] Figure 6 This is a graph showing the success rate of data capture in this invention.
[0052] Figure 7 This is a heatmap comparison of the action of VPG+affordance and VPG in this invention;
[0053] Figure 8This is a flowchart of the VPG+affordance capture process of the present invention;
[0054] Figure 9 The ablation experiment comparison curves of the present invention are shown in the following models: Model 1: VPG + affordance, Model 2: VPG, Model 3: VPG + affordance_without_AD, Model 4: VPG + affordance_without_AR.
[0055] Figure 10 This is a test scenario diagram for the present invention. Detailed Implementation
[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0057] In complex scenarios with tightly stacked objects, direct object grasping can face challenges such as difficulty finding suitable grasping poses, potential mechanical collisions between the object and the gripper, and low grasping success rates. Combining pushing and grasping strategies offers an important approach to solving object grasping in complex environments; however, current reinforcement learning-based push-grasp algorithms still suffer from insufficient evaluation of the effectiveness of pushing actions and low grasping success rates. Furthermore, in practice, it is often advisable to first push tightly stacked objects to a separated state before proceeding with the grasping operation, which significantly improves the success rate. Therefore, modeling these two tasks into a unified multi-task framework to achieve collaborative pushing and grasping is a promising direction for addressing this problem.
[0058] Please see Figure 1-2 As shown, the present invention is an improvement to grasp availability. Q This paper proposes a push-grab reward design method for networks. Based on VPG, it redesigns the push reward by introducing a crawl availability network and adopts a better decision-making method to select push and crawl actions, thus proposing a push reward based on improved crawl availability.
[0059] To generate more availability regions around the grasping object and better evaluate the effectiveness of pushing behavior, this embodiment trains a grasping availability network. From the perspective of grasping availability, a new pushing reward mechanism is designed to evaluate the rationality of the pushing behavior. The effectiveness of the pushing behavior is judged by the increase or decrease in availability regions around the grasping object before and after pushing. A novel collaborative mechanism between pushing and grasping strategies is proposed. This model includes a visual pushing and grasping module based on a deep Q-network and a pre-trained grasping availability network module. Depth-height images and color-height images generated by projection transformation from an RGB-D camera are input into the pushing network, grasping network, and pre-trained grasping availability network, respectively, to generate pixel-level cumulative reward expectations. , and availability value By setting an availability threshold, it is possible to accurately determine whether the current environmental state meets the conditions for grasping, thus providing a basis for action decisions.
[0060] Specific methods include:
[0061] Step S1: Acquire RGB-D image information of the working scene and obtain color height image and depth height image through projection transformation; The specific method of acquiring color height image and depth height image is as follows: At the current time t, the depth camera acquires RGB-D image of the working scene, and transforms each pixel of the color image and depth image from the camera coordinate system to the robot arm coordinate system through the intrinsic and extrinsic parameter matrix to obtain a series of point clouds. Then, the point clouds belonging to the working environment are mapped to the two-dimensional plane along the gravity direction to obtain the color image height map and the depth image height map.
[0062] Step S2: Input the color height image and depth height image of the workspace into the push network FCN. push and crawling network FCN grasp And within the pre-trained availability network, pixel-level cumulative reward expectations are generated sequentially. , and availability value The push network and the grasping network obtain state-action prediction values through a deep Q-network composed of two feedforward fully convolutional neural networks (FCNpush and FCNgrasp). The grasping availability network extracts features from the color height map and the depth height map through two DenseNet-121 pre-trained on ImageNet. Then, the color features and depth features are concatenated to achieve multimodal feature fusion. The feature fusion result is input into a fully convolutional neural network, which contains two 1×1 convolutional layers, a ReLU activation function and a batch normalization layer. Finally, it undergoes bilinear upsampling to obtain a pixel-level Agrasp value prediction heatmap.
[0063] Step S3: Determine whether the change in availability after the push action exceeds a set threshold: if yes, give a positive reward for the push action; if no, do not give an action reward; select the grab availability value calculated based on the percentage of the smallest grabizable object pixel value in the scene as the set threshold, and further determine whether there are any cases where the number of pixels with availability values not less than 1 exceeds the set threshold: if so, prioritize grabbing objects in the scene that are available for grabbing; otherwise, execute the push action; the pre-trained grab availability network generates a height map. A dense pixel map of grasping action availability values with the same image size and resolution is used. If the availability change after the push action exceeds a set threshold, a positive reward is given for the push action. If the availability change after the push action does not exceed the set threshold, no reward is given. In addition, by traversing the 16 grasping availability value images, it is determined whether there are pixels with availability values of not less than 1 whose number exceeds a set threshold. If so, the grasping operation is prioritized for graspable objects in the scene; otherwise, the push action is performed.
[0064] The background of the above technical solution is that the process of a robot pushing and grasping objects can be described as a Markov Decision Process (MDP); MDPs are typically used... These four elements represent the state space, action space, transition probability, and reward function. The goal of robot reinforcement learning is to find an optimal policy π that maximizes the expected sum of future rewards: that is, at any given time t, the distribution of graspable objects within the workspace is obtained through a camera. The agent selects and executes an action based on policy π. Then transition to a new state. And receive the corresponding reward immediately. .
[0065] As one embodiment of the present invention, the crawling availability network is constructed as follows:
[0066] Pre-train a parallel 121-layer DenseNet network on ImageNet;
[0067] Channel concatenation is performed, followed by two additional 1×1 convolutional layers with non-linear activation functions (ReLU) and spatial batch normalization before upsampling;
[0068] One DenseNet network takes the color channel of the heightmap as input, and another DenseNet network takes the depth channel of the heightmap as input.
[0069] Using the above technical solution, this embodiment employs a feedforward fully convolutional network to construct the Agrasp crawling action availability network; and assigns each state to... The model is created as an RGB-D heightmap of the scene at time t. To compute this heightmap, RGB-D images are captured from a fixed-mount camera, the data is projected onto a 3D point cloud, and then vertically back-projected in the direction of gravity to construct a heightmap with color (RGB) and height (D) channels. The edges of the heightmap are predefined based on the boundaries of the agent workspace. The network will then consider the current state. The RGB heightmap and depth heightmap below are used as input, and the output is the same as... Dense pixel mappings of availability values for grabbing actions with the same image size and resolution.
[0070] Figure 2 To capture the availability network architecture graph, this embodiment pre-trains a parallel 121-layer DenseNet on ImageNet, then performs channel concatenation, adds two additional 1×1 convolutional layers with a non-linear activation function (ReLU) and spatial batch normalization, and then performs upsampling. One DenseNet network takes the color channels of the heightmap as input, while the other takes the depth channels of the heightmap (normalized by subtracting the mean and dividing by the standard deviation) as input.
[0071] Please refer to Figure 2 As shown, in one embodiment of the present invention, the pre-training method for capturing the availability network is as follows:
[0072] Data is collected from no less than 3000 operations during the VPG training process, including color height images, depth height images, and angle and pixel position information of the actions performed. A zero matrix of the same size as the input image is created to establish labels for the grabbing action availability.
[0073] For grasping actions: If the distance between the opposite ends of the robot gripper fingers exceeds a set threshold, then the pixel position label for that grasping action is set to 1;
[0074] Input the crawling action availability label, the color height image and depth height image of the crawling action, and the pixel position information into the built network model respectively;
[0075] After extracting relevant features from the image and performing feature concatenation, the data is normalized and convolutional. Then, upsampling is used to obtain the availability prediction values, and the Huber loss function is used for iterative training.
[0076] The formula for calculating the loss function is:
[0077] ;
[0078] in, These are the parameters of the neural network at iteration i. The target network parameters are fixed and kept constant with each update; gradients are propagated backward through a single pixel p in the network, and actions are calculated and executed through the network's forward propagation. The availability is numerically predicted; the backpropagation loss for all other pixels is 0 at iteration i.
[0079] In the above technical solution, this embodiment collects data from no less than 3000 operations during the VPG training process, collecting RGB height maps, depth height maps, and pixel position information, including the pixel coordinates and angle information of the grasping point. To establish labels for grasping action availability, this embodiment creates a zero matrix of the same size as the input image. For grasping actions, if the distance between the opposite ends of the robot gripper fingers exceeds a set threshold, the pixel position label of the grasping action is set to 1. The grasping action availability labels, depth and color information of the grasping action are input into the constructed network. Relevant features are extracted from the depth and color height maps, and after feature concatenation, normalization and convolution processing are performed. Then, the grasping availability prediction value is upsampled, and iterative training is performed using the Huber loss function. After training the grasping availability network for 500 rounds, the generated weight file is evaluated, and the best training weights are selected.
[0080] In one embodiment of the present invention, in step S2:
[0081] The reward function for the pushing action is constructed by summing the availability images of the scene in 16 directions before and after the pushing action, calculating the difference, and comparing it with a set threshold. For pushing actions where the change in the availability value before and after the pushing action is greater than the set availability threshold, the reward is set to 0.5; otherwise, it is 0. The formula for calculating the reward function for the pushing action is as follows:
[0082] ;
[0083] Where i is the direction index corresponding to the availability image. A color height map before the action is initiated. To advance the depth and height map before the action, To produce a color height map after the action, To advance the depth-height map after the action, For the availability image of the i-th direction after the push action is performed, For the availability image of the i-th direction before performing the pushing action, The availability threshold set for this embodiment.
[0084] The existing standard VPG uses two deep Q-networks to learn pushing and grasping actions respectively and output their expected Q-values. It constructs a reward function for the pushing action using the difference between depth images before and after the pushing action captured by an RGB-D camera, which can increase the success rate of grasping to some extent. However, this reward setting is relatively sparse, only providing a reward based on whether the pushing action changes the environment, and using a formula... ,in To obtain the scene depth and height image after the pushing action is performed. To obtain the scene depth and height image before performing the pushing action, The depth threshold is designed; this push reward only evaluates whether the object is pushed, but does not evaluate the effectiveness of the push action; therefore, this method may cause the entire object to be pushed in a certain direction.
[0085] To address the aforementioned problems, this embodiment, based on VPG, redesigns the push reward by introducing a crawling availability network. Please refer to [link / reference needed]. Figure 3 As shown, the reward function for the pushing action is constructed by summing the availability images of the scene in 16 directions before and after the pushing action and comparing the difference with a set threshold. For pushing actions where the change in the availability value before and after the pushing action is greater than the set availability threshold, the reward is set to 0.5, and 0 for others. The calculation formula for the reward function of the pushing action is as follows:
[0086] ;
[0087] Where i is the direction index corresponding to the availability image. A color height map before the action is initiated. To advance the depth and height map before the action, To produce a color height map after the action, To advance the depth-height map after the action, For the availability image of the i-th direction after the push action is performed, For the availability image of the i-th direction before performing the pushing action, The availability threshold set for this embodiment.
[0088] As one embodiment of the present invention, it further includes: designing a reward function for the grasping action:
[0089] For the grasping action, a sparse reward function is used, and the calculation formula is as follows:
[0090] .
[0091] In the above technical solution, since the availability value is related to the direction of action execution and the availability image information is affected by the rotation direction, when calculating the push reward, it is necessary to first accumulate the 16 grasping availability value images before and after the push action, and then calculate their difference. The rationality of the push action is judged by the difference in the grasping availability image values before and after the push action. If the change in availability after the push action exceeds a set threshold, a positive reward is given for the push action; if the change in availability after the push action does not exceed the set threshold, no action reward is given. In grasping tasks in complex environments, robots are prone to failure due to insufficient grasping space. Therefore, the design of the grasping reward function is particularly important. For grasping actions, a sparse reward function is adopted:
[0092] .
[0093] As one embodiment of the present invention, the formula for implementing the push and grasping cooperative strategy in step S3 is as follows:
[0094] ;
[0095] in, The model is an RGB-D heightmap image of the scene at the current time t. These are the parameters of the Q network. To drive the cumulative reward expectation for actions, To accumulate the expected reward for the grabbing action, The number of pixels in the availability image with a value not less than 1. To capture the availability threshold.
[0096] Existing standard VPG predicts the maximum Q-value by comparing grabbing actions. And the maximum Q value of the propulsion action prediction Choose action strategy based on size .if Select the grab action and find the pixel position with the largest Q value. Then, calculate the three-dimensional spatial coordinates (x, y, z) corresponding to the pixel position through coordinate transformation. Otherwise, execute the push action.
[0097] ;
[0098] Such action primitive selection is simple and direct; however, in some test scenarios, it was found that although the predicted values... However, the block's position at this point does not meet the optimal grasping action, yet the robot still chooses the position at its maximum. Performing a grabbing action at the designated location may result in a physical collision between the gripper and the object being grabbed, leading to a failure to grab; similarly, during testing, there may also be situations where... Even if objects that can be grabbed are present in the test scenario, the system still chooses to grab them at the maximum time. The push action performed at the location resulted in unnecessary pushing motions, leading to a decrease in execution efficiency.
[0099] Therefore, a better decision-making method is needed to coordinate the push and grasp actions in a coordinated strategy. Regarding the question of how to select the push and grasp actions, this embodiment proposes a push-grasp action coordination mechanism based on improved grasp availability. Please refer to [link / reference]. Figure 4 As shown;
[0100] This embodiment generates 16 grasping availability images in 16 directions using an introduced grasping availability network. It iterates through these 16 images, calculating the grasping availability value of an object based on the percentage of pixels with the smallest graspable value in the scene, and sets this value as the availability threshold. By counting the number of pixels with an availability value of 1 or higher in each availability image, if the number of pixels with an availability value of 1 or higher exceeds the set threshold, a grasping operation is prioritized for the graspable object in the scene. Otherwise, a pushing action is selected. This reduces the number of training iterations and improves the overall grasping efficiency of the system, as shown by the calculation formula:
[0101] ;
[0102] Here, st is modeled as the RGB-D heightmap image of the scene at the current time t, and θ is the parameter of the Q network. To drive the cumulative reward expectation for actions, To accumulate the expected reward for the grabbing action, The number of pixels in the availability image with a value not less than 1. To capture the availability threshold.
[0103] Based on the above method, a collaborative mechanism model for push and grasp strategies was established. The model includes a visual push and grasp module based on a deep Q-network and a pre-trained grasping availability network module. This invention was also validated through experiments, including comparative experiments, ablation experiments, and model testing to demonstrate its effectiveness, as detailed below:
[0104] Comparative Experiment: To evaluate the effectiveness of the push-grasp skill learning (VPG+affordance) method proposed in this embodiment, the above scheme was compared with the standard VPG strategy. Each strategy was trained for 2500 steps. The grasp success rate curves of the two methods during the training process are shown in the figure. Figure 6 As shown, the VPG grasping method employs a push-grab combined strategy, actively pushing objects around the target to effectively increase the grasping space and thus significantly improve the grasping success rate. However, this method sets the push reward based on whether the environmental change exceeds a threshold, which may prevent the push model from quickly learning better push strategies, resulting in low overall training efficiency. The push-grab model (VPG+affordance) proposed in this embodiment outperforms the VPG scheme. After approximately 1000 push-grab training steps, the model converges to a grasping success rate of over 80%. The push reward function based on grasping affordance designed in this embodiment can guide the robot to learn more accurate pushing and adopt more reasonable grasping methods.
[0105] For a comparison of the visualization of the expected Q-value of cumulative rewards, please refer to [link / reference]. Figure 7 As shown, the motion visualization heatmap obtained by the algorithm in this embodiment can better locate the target block. In contrast, the motion heatmap generated by the VPG algorithm has a high heat map in a certain area, failing to achieve accurate target localization. This further verifies the important role of the scheme in the robot's final grasping effect. This embodiment uses a push-grasp model scheme based on grasping availability reward for grasping. Please refer to [link / reference]. Figure 8 As shown, the complete grasping process during training is illustrated.
[0106] VPG learns a synergistic strategy of pushing and grasping actions, but sometimes it executes ineffective pushes and sometimes pushes the grasp block outside the grasping space to complete the task. Furthermore, it doesn't fully consider the synergistic effect of pushing and grasping actions, resulting in low learning efficiency. This embodiment proposes a new strategy selection scheme. Instead of directly comparing the maximum Q-value of the action in the grasping prediction heatmap and the maximum Q-value of the action in the pushing prediction heatmap, it selects the action strategy by iterating through 16 generated grasping availability images to see if there are images with more than a set threshold of pixels with an availability of 1 or more. If so, it prioritizes grasping objects in the scene that are available for grasping, thus improving grasping efficiency. The grasping success rate curve shows that the scheme proposed in this embodiment outperforms the standard VPG strategy. Figure 5 The capture success rate curve shown suggests that the capture success rate after stable training is approximately 90% or higher, which is an overall improvement of more than 10% compared to the standard VPG strategy.
[0107] Ablation experiments: Two sets of comparative experiments were conducted to evaluate the performance of the proposed push-grab strategy based on grasp availability rewards. The purpose of the comparative experiments is as follows: to demonstrate that the push-reward strategy based on grasp availability proposed in this embodiment is effective in improving the grasp success rate in unstructured grasping environments; and to demonstrate that the push and grasp decision mechanism based on grasp availability proposed in this embodiment improves the training efficiency of the network.
[0108] The four models used for comparison are as follows:
[0109] (1) Model 1 (VPG+affordance) is a scheme that uses both push rewards based on grasp availability and push and grasp decision-making mechanisms based on grasp availability.
[0110] (2) Model 2 is trained using standard VPG and relies on the difference between depth and height images before and after a single push action as the criterion for push reward evaluation. The network output is directly compared. Value and drive network output The value determines the action strategy. ;
[0111] (3) Model 3 (VPG+affordance_without_AD) does not adopt a push and grab decision mechanism based on grab availability, but only adopts a push reward based on grab availability.
[0112] (4) Model 4 (VPG+affordance_without_AR) does not use a push reward based on grasp availability, but only a push and grasp decision mechanism based on grasp availability.
[0113] To ensure the four models share a similar experimental environment, a grasping scenario with 10 random objects was generated in the simulation environment used for training each model, and a grasping training run of 2500 steps was set. Please refer to the grasping success rate curves for the four models. Figure 9 As shown; and from Figure 8 As can be seen, adopting a push reward function based on crawl availability and a push and crawl decision mechanism based on crawl availability can lead to a higher crawl success rate.
[0114] Model Testing: In this embodiment, a robot UR5 equipped with an RG2 gripper is set up on the V-rep simulation platform. The trained model is used to test eight grasping scenarios in the simulation environment. Each test scenario consists of 4-6 grasping objects. Please refer to [the documentation / reference]. Figure 10 As shown.
[0115] These scenarios included both tightly stacked grasp blocks and single grasp blocks, facilitating observation of the push operation's performance across different test scenarios. These scenarios were challenging picking scenarios, with each grasping scenario executing 30 rounds of grasping tasks, and three metrics were used to evaluate model performance.
[0116] (1) Grasp Completion Rate (GCR): Defined as the percentage of successful grabs of the target block within a maximum of 10 actions during the entire grabbing process. A higher GCR indicates a stronger ability of the grabbing model to autonomously complete the target grabbing. GCR is defined as follows: ;in This indicates the number of rounds in which the target block was successfully captured. This represents the total number of rounds of grabbing.
[0117] (2) Grasp Success Rate (GSR) is defined as the proportion of successful grasps out of the total number of actions. A higher grasp success rate indicates that the current grasping model is trained better. GSR is defined as follows: Where NGS represents the number of successful crawls, and TNG represents the total number of crawls.
[0118] (3) Action Efficiency (AE), which describes the simplicity with which a strategy can accomplish a task. AE is defined as follows: Where NO represents the number of objects to be crawled, and NA represents the number of times the operation was performed.
[0119] The test results are shown in Table 1.
[0120] Table 1. Experimental Results of Test Scenarios
[0121]
[0122] Through experiments on the above eight test scenarios, it was observed that when tightly stacked objects and isolated objects appear simultaneously, the robot prioritizes grasping isolated objects in the grasping scenario. If there are no isolated objects available for grasping in the scenario and there are tightly stacked objects, the robot will push the stacked objects apart and then grasp the objects that become graspable after the pushing action. This further verifies that when facing complex environments, the cooperative strategy can better complete the grasping task through the cooperation of grasping and pushing.
[0123] Based on the test experiments, when there are objects available for grasping in the scene, the solution in this embodiment prioritizes grasping operation primitives to grasp isolated objects in the scene. For tightly stacked objects, the solution tries to disperse the objects before grasping. As shown in Table 1, after evaluation through the above test scenarios, the model proposed in this embodiment outperforms the standard VPG strategy in terms of grasping success rate, average completion rate, and action execution efficiency. For some complex scenarios, such as scenarios 5, 6, and 7, the solution in this embodiment performs even better.
[0124] The above description is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined in the claims, they should all fall within the protection scope of the present invention.
Claims
1. An improvement to crawl availability Q The network push-fetch reward design method is characterized by, The method includes: S1. Acquire RGB-D image information of the working scene and obtain color height image and depth height image through projection transformation; S2. Input the color height image and depth height image into the push network FCN respectively. push and crawling network FCN grasp And within the pre-trained availability network, pixel-level cumulative reward expectations are generated sequentially. , and availability value ; S3. Determine whether the change in availability exceeds a set threshold after the push action is executed: If so, then give a positive reward for the action. If not, no action reward will be given; The grab availability value, calculated based on the percentage of the smallest grabizable object pixel value in the scene, is used as a set threshold. Further, it is determined whether there are pixels with an availability value of 1 or higher exceeding the set threshold. If available, then prioritize grabbing objects in the scene; otherwise, perform a pushing action. In step S2: The reward function for the pushing action is constructed by summing the availability images of the scene in 16 directions before and after the pushing action, calculating the difference, and comparing it with a set threshold. For pushing actions where the change in the availability value before and after the pushing action is greater than the set availability threshold, the reward is set to 0.5; otherwise, it is 0. The formula for calculating the reward function for the pushing action is as follows: ； Where i is the direction index corresponding to the availability image. A color height map before the action is initiated. To advance the depth and height map before the action, To produce a color height map after the action, To advance the depth-height map after the action, For the availability image of the i-th direction after the push action is performed, For the availability image of the i-th direction before performing the pushing action, This is the availability threshold; The formula for implementing the push and grab coordination strategy in step S3 is as follows: ； in The model is an RGB-D heightmap image of the scene at the current time t. These are the parameters of the Q network. To drive the cumulative reward expectation for actions, To accumulate the expected reward for the grabbing action, The number of pixels in the availability image with a value not less than 1. To capture the availability threshold.
2. An improvement to grasp availability according to claim 1 Q The network-driven crawling reward design method is characterized by: The crawling availability network is constructed as follows: Pre-train a parallel 121-layer DenseNet network on ImageNet; Channel concatenation is performed, followed by two additional 1×1 convolutional layers with non-linear activation functions (ReLU) and spatial batch normalization before upsampling; One DenseNet network takes the color channel of the heightmap as input, and another DenseNet network takes the depth channel of the heightmap as input.
3. An improvement to grasp availability according to claim 1 Q The network-driven crawling reward design method is characterized by: The pre-training method for the availability-grabbing network is as follows: Data is collected from no less than 3000 operations during the VPG training process, including color height images, depth height images, and angle and pixel position information of the actions performed. A zero matrix of the same size as the input image is created to establish labels for the grabbing action availability. For grasping actions: If the distance between the opposite ends of the robot gripper fingers exceeds a set threshold, then the pixel position label for that grasping action is set to 1; Input the crawling action availability label, the color height image and depth height image of the crawling action, and the pixel position information into the built network model respectively; After extracting relevant features from the image and performing feature concatenation, the data is normalized and convolutional. Then, upsampling is used to obtain the availability prediction values, and the Huber loss function is used for iterative training. The formula for calculating the loss function is: ； in, These are the parameters of the neural network at iteration i. The target network parameters are fixed and kept constant with each update; gradients are propagated backward through a single pixel p in the network, and actions are calculated and executed through the network's forward propagation. The availability is numerically predicted; the backpropagation loss for all other pixels is 0 at iteration i.
4. An improvement to grasp availability according to claim 1 Q The network-driven crawling reward design method is characterized by: The method for obtaining the color height image and depth height image in step S1 is as follows: At the current time t, the depth camera acquires RGB-D images of the working scene. Each pixel of the color image and the depth image is transformed from the camera coordinate system to the robot coordinate system through the intrinsic and extrinsic parameter matrices to obtain a series of point clouds. Then, the point clouds belonging to the working environment are mapped onto a two-dimensional plane along the direction of gravity to obtain the color image height map and the depth image height map.
5. An improvement to grasp availability according to claim 1 Q The network push-fetch reward design method is characterized by, This also includes: the design of the reward function for the grasping action: For the grasping action, a sparse reward function is used, and the calculation formula is as follows: 。 6. An improvement to grasp availability according to any one of claims 1-5 Q The network push-fetch reward design method is characterized by, Based on the method, a reward function for the pushing action and a collaborative mechanism model for the pushing and grasping strategies are set. The model includes a visual pushing and grasping module based on a deep Q-network and a pre-trained grasping availability network module.