A method for learning grasping operation skills

Abstract

The present invention discloses a grasping operation skill learning method, which belongs to the field of robot grasping technology. The present invention achieves faster planning by reducing the number of actions available in any given situation, which helps to learn models from data more efficiently and accurately. Moreover, based on the visual push grasping (VPG) model based on the deep Q network, by training a feasible network model for the pushing action network and the grasping action network respectively, it can limit unreasonable grasping postures and invalid pushing operation behaviors, effectively improve the learning efficiency, and better improve the grasping efficiency and grasping success rate.

Description

Technical Field

[0001] The present invention relates to the field of robot grasping technology, and in particular to a grasping operation skill learning method. Background Art

[0002] Grasping is one of the most fundamental problems in robotics, with important applications in many scenarios, including sorting robots, industrial robots, and human-computer interaction. This problem has attracted increasing attention in recent years.

[0003] Traditional grasping methods are usually applied in environments with known object models, which limits their adaptability to different objects and scenarios. Recently, researchers have applied deep learning and reinforcement learning to robotic grasping tasks. Reinforcement learning is a machine learning method that learns the best action strategy through the interaction between the intelligent agent and the environment. It allows the robot to interact with the environment in a self-supervised manner to complete the set tasks. Some existing grasping models can achieve grasping tasks in certain scenarios, but their grasping success rate is not high when dealing with tightly stacked objects because it is difficult to find a suitable grasping point on the object and the gripper can easily collide with the grasped object. Therefore, how to design an effective strategy to grasp tightly stacked objects remains a challenge.

[0004] In real life, when grasping stacked objects, it's common to push them apart before proceeding to grasping. This not only improves efficiency but also increases the success rate of grasping. Therefore, modeling these two tasks into a unified multi-task framework to achieve synergistic effects between pushing and grasping has been a hot topic in the field of robotic grasping. In recent years, several researchers have proposed collaborative pushing grasping strategies based on deep reinforcement learning, such as the Visual Pushing Grasping (VPG) model. VPG, a deep Q-network-based model, achieves grasping of tightly stacked objects by combining grasping and pushing actions. It uses a fully convolutional neural network to predict the Q-values ​​of all pixels, compares the Q-values ​​of the grasping and pushing actions, and uses a greedy strategy to select and execute the action with the maximum Q-value. This method uses pushing actions to change the positional distribution of objects, facilitating subsequent grasping. However, the sparsity of its reward function results in slow early convergence.

[0005] Humans and animals possess a unique ability to perceive their surroundings, enabling them to discern which actions are successful in specific situations. For example, on hard surfaces, people can choose to walk or run, while on smooth surfaces, they can skate or slide. This perception, without labels or indicators, helps people understand possible action options and, by limiting the range of actions considered, enables them to respond quickly to environmental changes and make effective decisions. This involves the concept of affordance, which uses environmental characteristics to guide appropriate behavior. Affordance is a relationship between people and objects, whereby a person perceives triggers from objects, creating the possibility of performing a certain action. This is a significant assumption, as it implies that the value and meaning of objects in the environment can be directly perceived. In recent years, with the continuous expansion of affordance theory, it has also been widely applied to various fields. In robotics, with the development of deep learning, the application of affordances in robotic grasping has become a trend. However, in the robot's grasping tasks in complex scenarios, due to the randomness of the size, shape and distribution of the grasped objects, it is very difficult to complete the grasping of objects in the working area by relying on a single grasping operation. Therefore, there are many invalid pushing actions and unreasonable grasping actions, resulting in a lack of grasping efficiency and grasping success rate.

[0006] In response to the above problems, this invention document proposes a method for learning grasping operation skills. Summary of the Invention

[0007] The present invention provides a grasping operation skill learning method, which solves the above-mentioned problem that in the grasping task of the robot in complex scenes, due to the randomness of the size, shape and distribution state of the grasped objects, there are many invalid pushing actions and unreasonable grasping actions, resulting in a lack of grasping efficiency and grasping success rate.

[0008] The present invention provides the following technical solutions:

[0009] A method for learning grasping operation skills, comprising the following steps:

[0010] S1. Determine the action network based on the VPG model;

[0011] S2. Based on the action network, a feedforward fully convolutional network is used to construct an affordance network model. The affordance network model specifically includes a pushing action affordance network model and a grasping action affordance network model.

[0012] S2.1. Pretrain two parallel 121-layer feed-forward fully convolutional networks on the ImageNet database, then perform channel concatenation, add two additional 1x1 convolutional layers with nonlinear activation functions and spatial batch normalization, and then upsampling.

[0013] S2.2. One feed-forward fully convolutional network takes the color channel of the height map as input, while another feed-forward fully convolutional network takes the depth channel of the height map as input to construct the push action affordance network model and the grasp action affordance network model;

[0014] S3, training the affordance network model;

[0015] S3.1. Collect data for multiple VPG training operations to obtain RGB height maps, depth height maps, and pixel position information;

[0016] S3.2. Create a zero matrix of the same size as the input image. For a push action, calculate the sum of the pixel differences between the depth and height maps before and after the push operation. If the sum exceeds a set threshold τ, set the value of the pixel location where the push action was performed to 1.

[0017] For grasping actions, if the distance between the two ends of the robot gripper fingers exceeds the set threshold, the pixel position point label of the grasping action is set to 1;

[0018] S3.3. Input the action affordance labels, the depth and color information of the corresponding pushing and grasping actions into the constructed network respectively;

[0019] S3.4. Extract relevant features from the depth and color height maps, perform feature concatenation, and then perform normalization and convolution. Then, upsample the affordance prediction value and iteratively train it using the Huber loss function.

[0020] S4. Outputting the predicted affordance values ​​of the grabbing action and the pushing action through the affordance network model, including the predicted affordance value A for pushing. push And the predicted value of crawl availability A grasp If the push or grab affordance prediction value of a pixel position is negative, the corresponding affordance prediction value of the pixel position is set to 0;

[0021] S5. Multiply the expected value output by each action network by the availability value output by the corresponding availability network, and decide the final execution action based on the result.

[0022] In one possible design, in step S1, the VPG model uses two deep Q networks to learn the pushing and grasping actions respectively and output their respective expected Q values.

[0023] In a possible design, in step S3, the formula for iterative training using the Huber loss function is:

[0024] ;

[0025] Among them, s represents the state space, a represents the action space, is the parameter of the neural network at iteration i, the target network parameter Keep it fixed at each update, only pass the gradient back through the network through a single pixel p, and perform action a by forward propagation through the network. i The affordance value of , all other pixels have a back-propagation loss of 0 at iteration i.

[0026] In a possible design, in step S5, the specific steps are:

[0027] S5.1. Expected value Q of action rewards generated from the push and grab network push and Q grasp Performing matrix multiplication operations with the affordance prediction values ​​of the same action, we can obtain:

[0028] ;

[0029] S5.2. Compare the Q values ​​processed by the action affordance mask and finally use the greedy strategy to select the action with the largest Q':

[0030] ;

[0031] Then, the position point x, y, and z where the action is performed are obtained from the pixel coordinates corresponding to the maximum Q', and the angle at which the action is performed is obtained.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention.

[0033] The present invention achieves faster planning by reducing the number of actions available in any given situation, which helps to learn models from data more efficiently and accurately. Based on the visual push grasping (VPG) model based on the deep Q network, by training a feasible network model for the pushing action network and the grasping action network respectively, it can limit unreasonable grasping postures and invalid pushing operations, effectively improving the learning efficiency and better improving the grasping efficiency and grasping success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A model architecture diagram of a grasping operation skill learning method provided by an embodiment of the present invention;

[0035] Figure 2 A training scenario diagram for a grasping operation skill learning method provided by an embodiment of the present invention;

[0036] Figure 3A grasping success rate curve diagram of a grasping operation skill learning method provided by an embodiment of the present invention;

[0037] Figure 4 A test scenario diagram for a grasping operation skill learning method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described below with reference to the accompanying drawings.

[0039] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Example

[0040] Reference Figure 1 , a grasping operation skill learning method, comprising the following steps:

[0041] S1. Determine the action network based on the VPG model.

[0042] Specifically, in this embodiment, the action is parameterized as an action primitive ψ (push or grab), and the position q of each action is obtained by projecting a pixel p in the height map under the state st, expressed as a=(ψ,q)|ψ∈{push,grasp}, ; For the pushing action, q represents the starting position of the push in 16 directions. In this embodiment, the pushing distance is set to 10 cm in a straight line, and the task is completed by the closed jaws; for the grasping operation, before closing the jaws, the two fingers of the jaws try to move to 3 cm below q (in the direction of gravity).

[0043] S2. Based on the action network, a feedforward fully convolutional network is used to construct an affordance network model. The affordance network model specifically includes a push action affordance network model and a grasp action affordance network model. Specifically, the steps include:

[0044] S2.1. Pre-train two parallel 121-layer feed-forward fully convolutional networks on the ImageNet database, then perform channel concatenation, add two additional 1x1 convolutional layers with non-linear activation functions and spatial batch normalization, and then upsampling.

[0045] S2.2. One feed-forward fully convolutional network takes the color channel of the height map as input, while another feed-forward fully convolutional network takes the depth channel of the height map as input to construct a pushing action affordance network model and a grasping action affordance network model.

[0046] S3. Training the affordance network model. Specifically, the following steps are included:

[0047] S3.1. Perform data collection on multiple VPG training operations to obtain RGB height map, depth height map and pixel position information.

[0048] S3.2. Create a zero matrix of the same size as the input image. For a push action, calculate the sum of the pixel differences between the depth and height maps before and after the push operation. If the sum exceeds the set threshold τ, set the label value of the pixel position where the push action is performed to 1.

[0049] For a grasping action, if the distance between the two ends of the robot gripper fingers exceeds a set threshold, the pixel position point label of the grasping action is set to 1.

[0050] S3.3. Input the action affordance labels, the depth and color information of the corresponding pushing and grasping actions into the constructed network respectively.

[0051] S3.4. Extract relevant features from the depth and color height maps, perform feature concatenation, and then normalize and convolve them. Then upsample the available prediction values ​​and iteratively train them using the Huber loss function. The specific formula for iterative training using the Huber loss function is:

[0052] ;

[0053] Among them, s represents the state space, a represents the action space, is the parameter of the neural network at iteration i, the target network parameter Keep it fixed at each update, only pass the gradient back through the network through a single pixel p, and perform action a by forward propagation through the network. i The affordance value of , all other pixels have a back-propagation loss of 0 at iteration i.

[0054] S4. Outputting the predicted affordance values ​​of the grabbing action and the pushing action through the affordance network model, including the predicted affordance value A for pushing. push And the predicted value of crawl availability A grasp If the push or grab availability prediction value of a pixel position is negative, the corresponding availability prediction value of the pixel position is set to 0.

[0055] Specifically, this embodiment collects RGB images and depth images through the RGB-D camera of the simulation platform, projects the collected image information in the vertical direction to obtain relevant RGB height maps and depth height maps, rotates the relevant height images 16 times, and each rotation angle is 22.5. The rotated height images are respectively input into the available push network and the available crawl network, as well as the push network and the crawl network. Then, features are extracted from the RGB height map and the depth height map for feature splicing. Two normalization modules, linear layers, and convolution processing are used in the available push network and the available crawl network for feature embedding. Finally, the push availability prediction value A of all pixels is generated through upsampling. push And the predicted value of crawl availability A grasp .

[0056] S5. Multiply the expected value output by each action network by the affordance value output by the corresponding affordance network, and determine the final action to be executed based on the result. The specific steps are as follows:

[0057] S5.1. Expected value Q of action rewards generated from the push and grab network push and Q grasp Performing matrix multiplication operations with the affordance prediction values ​​of the same action, we can obtain:

[0058] .

[0059] S5.2. Compare the Q values ​​processed by the action affordance mask and finally use the greedy strategy to select the action with the largest Q':

[0060] ;

[0061] Then, the position point (x, y, z) where the action is performed is obtained from the pixel coordinates corresponding to the maximum Q', and the angle at which the action is performed is obtained. Example

[0062] Based on Example 1, this example provides an experiment, comprising the following steps:

[0063] S1. Model training:

[0064] S1.1. A simulation platform was built on Ubuntu 18.04. Training and testing were performed on a high-performance server equipped with an Intel Xeon(R) E5-2630 v4 CPU and an NVIDIA TITANX GPU. The operating environment was CUDA 10.0.130, CUDNN 7.6.5, Pytorch 1.1, and torchvision 0.4.0. The model was trained on the simulation platform V-rep. A UR5 robotic arm and an RG2 gripper were set up on V-rep, as well as a workspace area of ​​0.448m*0.448m. During the model training process, 10 grasping blocks were randomly selected. The robot automatically performed operations through trial and error learning until all objects in the grasping space were grasped. Then the grasping environment was updated for the next round of training. The training scenarios were as follows: Figure 2 shown.

[0065] S1.2. The affordance model network is built based on the Pytorch deep learning framework and trained using the Adam optimizer with a learning rate of 10. -4 , momentum is 0.9, and decay weight is set to 2 -5 .

[0066] S1.3. Evaluate the VPG model proposed in this invention using improved affordance (denoted as VPG+affordance), and conduct an experimental comparison between the above model and the standard VPG model. Figure 3 The following are the grasping success rate curves of the two models during the training process. The two models were trained for 2500 steps respectively.

[0067] S2. Verification test:

[0068] S2.1, using the model trained in step S1, test the eight grasping scenarios in the simulation environment. Specifically, Figure 4 As shown in the figure, each test scene consists of 4-6 grasping objects; these scenes are some challenging picking scenes, and each grasping scene performs 30 grasping tasks. Then, the model performance is evaluated using three indicators:

[0069] Metric 1, 30-time completion rate, which measures the strategy's ability to complete the task by picking up all objects without failing more than 10 times in a row;

[0070] Indicator 2: Crawl success rate , NGS is the number of successful crawls, and TNG represents the total number of crawls;

[0071] Indicator 3: Action efficiency , NO indicates the number of grasped objects, and NA indicates the number of operations performed, which describes the simplicity of the strategy in completing the task;

[0072] For all indicators, the higher the indicator, the better. Multiple experiments were conducted on these 8 test scenarios. The test scenario experimental results are shown in Table 1 below:

[0073] Table 1

[0074]

[0075] As can be seen from Table 1, after evaluating the above test scenarios, the VPG+affordance model proposed in this invention outperforms the standard VPG model in terms of grasping success rate, average completion rate, and action execution efficiency, especially for some complex scenarios such as scenarios 5, 6, and 7.

[0076] It can be concluded that the grasping operation skill learning method proposed in this invention enables faster planning by reducing the number of actions available in any given situation, which helps to learn models from data more efficiently and accurately.

[0077] Finally, a few points should be explained: Although the present invention has been described in detail above with general descriptions and specific embodiments, on the basis of the present invention, the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solution to deviate from the scope of the technical solution of the embodiments of the present invention.

Claims

1. A method for learning grasping operation skills, characterized in that: The following steps are involved: S1. Determine the action network based on the VPG model; S2. Based on the action network, a feedforward fully convolutional network is used to construct an affordance network model. The affordance network model specifically includes a pushing action affordance network model and a grasping action affordance network model. S2.

1. Pretrain two parallel 121-layer feed-forward fully convolutional networks on the ImageNet database, then perform channel concatenation, add two additional 1x1 convolutional layers with nonlinear activation functions and spatial batch normalization, and then upsampling. S2.

2. One feed-forward fully convolutional network takes the color channel of the height map as input, while another feed-forward fully convolutional network takes the depth channel of the height map as input to construct the push action affordance network model and the grasp action affordance network model; S3, training the affordance network model; S3.

1. Collect data for multiple VPG training operations to obtain RGB height maps, depth height maps, and pixel position information; S3.

2. Create a zero matrix of the same size as the input image. For a push action, calculate the sum of the pixel differences between the depth and height maps before and after the push operation. If the sum exceeds a set threshold τ, set the value of the pixel location where the push action was performed to 1. For grasping actions, if the distance between the two ends of the robot gripper fingers exceeds the set threshold, the pixel position point label of the grasping action is set to 1; S3.

3. Input the action affordance labels, the depth and color information of the corresponding pushing and grasping actions into the constructed network respectively; S3.

4. Extract relevant features from the depth and color height maps, perform feature concatenation, and then perform normalization and convolution. Then, upsample the affordance prediction value and iteratively train it using the Huber loss function. S4. Outputting the predicted affordance values ​​of the grabbing action and the pushing action through the affordance network model, including the predicted affordance value A for pushing. push And the predicted value of crawl availability A grasp If the push or grab affordance prediction value of a pixel position is negative, the corresponding affordance prediction value of the pixel position is set to 0; S5. Multiply the expected value output by each action network by the availability value output by the corresponding availability network, and decide the final execution action based on the result.

2. A grasping operation skill learning method according to claim 1, characterized in that: In step S1, the VPG model uses two deep Q networks to learn the pushing and grasping actions respectively and output their respective expected Q values.

3. A grasping operation skill learning method according to claim 1, characterized in that: In step S3, the formula for iterative training using the Huber loss function is: ； Among them, s represents the state space, a represents the action space, is the parameter of the neural network at iteration i, the target network parameter Keep it fixed at each update, only pass the gradient back through the network through a single pixel p, and perform action a by forward propagation through the network. i The affordance value of , all other pixels have a back-propagation loss of 0 at iteration i.

4. A grasping operation skill learning method according to claim 1, characterized in that: In step S5, the specific steps are: S5.

1. Expected value Q of action rewards generated from the push and grab network push and Q grasp Performing matrix multiplication operations with the affordance prediction values ​​of the same action, we can obtain: ； S5.

2. Compare the Q values ​​processed by the action affordance mask and finally use the greedy strategy to select the action with the largest Q': ； Then, the position point x, y, and z where the action is performed are obtained from the pixel coordinates corresponding to the maximum Q', and the angle at which the action is performed is obtained.

Images