A reward training method for efficient manipulation placement learning of robotic arms based on image domain

Abstract

The present invention discloses a learning reward training method for efficient manipulation and placement of a robotic arm based on an image domain, which realizes the task of autonomous placement of objects by a robotic arm in a cluttered scene. In this task, due to the existence of constraints such as limited space and complex collisions, the terminal state is unpredictable, which brings difficulties to the design of the reward function. The reward for reinforcement learning in this patent is a combination of "step reward" and "completion reward", which quantifies the performance of the terminal state by merging in the image domain, which will guide the terminal state to converge to a better domain rather than a specific value. This encourages the rapid arrangement of scattered objects in the box while minimizing the gaps between objects, and has better adaptability to objects of different numbers and irregular shapes, as well as discontinuities.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning technology, and more specifically, to an image-domain-based robotic arm efficient manipulation placement learning reward training method. Background Art

[0002] The application of robotic arms has greatly liberated human work, but traditional robotic arm control is to perform kinematic and dynamic modeling of the robotic arm and plan the end motion trajectory in Cartesian space, and then solve the angle values ​​of each joint through kinematic inverse solution, so as to control the motor to drive the robotic arm to complete specific tasks. However, model- and rule-based control methods are increasingly unable to adapt to increasingly complex application scenarios, such as autonomous grasping and placement tasks of robotic arms in cluttered scenes, that is, when several objects of arbitrary shapes are scattered in a container, how to arrange them together as much as possible to make room for future objects?

[0003] This task has several interesting and challenging features. First, there is no clear guide or quantitative standard to measure the goodness of the current operation in the process. When the objects have arbitrary shapes, gaps always exist. Second, due to the special structure of the objects and the complementarity of different shapes, the current action may have short-term or long-term correlation with the subsequence actions. Therefore, compared with traditional methods, reinforcement learning has great advantages in solving such problems without the need for modeling and rule design.

[0004] Rewards are sufficient to drive intelligent behaviors studied in the fields of nature and artificial intelligence. Reward maximization and trial-and-error experience are sufficient to develop behaviors with intelligent capabilities. A scientific reward is very important for the generalization performance of reinforcement learning networks. In this task, we cannot predict the exact final layout of the object group because the contact between the object and the environment and between the objects during the pushing and arranging process is unpredictable and difficult to control accurately. Compared with the robot movement task achieved by shaping, where the reward is relatively clear, this brings challenges to the reward design where the final state changes during the process. In summary, a robot arm efficient manipulation placement learning reward training method based on the image domain is needed to provide scientific guidance for the training of reinforcement learning-based agents, and at the same time consider issues such as simulation-to-physical migration, to achieve efficient robot arm manipulation, and to autonomously place objects of arbitrary shapes randomly scattered in a limited container to achieve maximum packing. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a robot arm efficient manipulation placement learning reward training method based on image domain, so as to realize efficient robot arm manipulation, autonomously place objects of arbitrary shapes randomly scattered in a limited container, and realize maximum packing.

[0006] The technical solution of the present invention is:

[0007] A reward training method for efficient manipulation placement learning of a robotic arm based on an image domain, comprising:

[0008] Step 1: Design the application scenario and use the RGBD camera to collect the 4-channel RGBD-heightmap of the workspace;

[0009] Step 2: Select a single-channel D-heightmap to perform an overall design of the reward function in the image domain;

[0010] Step 3: Design the single-step reward in step 2: the learned single-step reward r step Includes mobile bonus change , collision reward contact , Repeated reward r repeat ;

[0011] Step 4: Design the termination reward in step 2: the learned termination reward r terminal Including task completion termination reward r done and the sequence action reward r req ;

[0012] Step 5: Combine the rewards from step 4 and step 5 to obtain the reward calculation function for network learning: r t =r step +r terminal ;

[0013] Step 6: Combined with steps 1-5, build an enhanced simulation learning and training project for learning and training, and conduct physical migration tests on the trained network.

[0014] Furthermore, the application scenario is as follows: using a cuboid finite volume container as the robot arm workspace, and randomly placing any number of objects;

[0015] The RGBD camera is used to collect 4-channel RGBD images of the workspace, and the image is converted into a 4-channel RGBD-heightmap in the top-down direction according to the camera's intrinsic parameters. The 4-channel RGBD-heightmap is used as the neural network state input, and the original image is extracted into a high-dimensional feature vector through Resnet-50, and learning is performed based on the reinforcement learning algorithm.

[0016] Furthermore, the reinforcement learning algorithm is designed with a single-step reward r step and the termination reward r terminal to encourage fast placement of scattered objects in a box while minimizing the gaps between objects.

[0017] Furthermore, the step 3 designs the single-step reward, specifically: the learned single-step reward r step Includes mobile bonus change , collision reward contact , Repeated reward r repeat ;

[0018] (1) Movement reward design: By comparing the difference between the two environmental states D-heightmap, we can determine whether the object has moved. If it has moved, the movement reward r change =0.1;

[0019] (2) Collision reward design: When it is detected that the current action will cause the gripper to collide vertically with the object, a collision reward r is applied. contact = -0.5, judging by detecting whether there is an object in the D-heightmap corresponding to the action point;

[0020] (3) Repeated reward design: If the same action is repeated more than three times, a repeated reward r will be given. repeat =-1.5;

[0021] (4) When the mobile reward r change , collision reward contact , Repeated reward r repeat When all three rewards are not met, the single-step reward is 0.

[0022] Furthermore, the single-step reward r step Specifically:

[0023]

[0024] Furthermore, the step 4 terminates the reward r terminal Design, specifically:

[0025] (1) Task completion reward r done Design: Task completion judgment needs to meet two constraints: connected domain detection and abnormal situation detection. Only when both conditions are met at the same time, the task is considered completed and r is given. done =3;

[0026] (2) Sequence action reward r req Design: Design a sequence action reward function

[0027] r seq =4-(N-1)×0.25

[0028] Among them, the reward value decreases as the number of actions N∈(0,16) increases;

[0029] (3) When the task is completed, the reward r is terminateddone and the sequence action reward r req When none of them are satisfied or the maximum number of action steps is exceeded, the reward is considered to be 0.

[0030] Furthermore, the termination reward r terminal Specifically:

[0031]

[0032] Furthermore, the connected domain detection is specifically as follows:

[0033] (a) Remove the depth map outside the container according to the container workspace and size, and perform denoising on the depth value d(x, y) of each pixel. When d(x, y)>0.01m, assign the value to 1, otherwise assign it to 0, and obtain the binary image map1;

[0034] (b) With a pooling kernel of 2 and a step size of 2, the image in (a) is subjected to maximum pooling to obtain image map2, eliminating initial gaps and errors and reducing the detection traversal area;

[0035] (c) Initialize the number of connected domains num_component = 0, and the neighbor detection queue is empty neighbors = [ ];

[0036] (d) Traverse the value p(x, y) corresponding to each pixel point in map2 in (b), starting from the origin of the pixel coordinate system, and determine whether p(x, y) is 1. If it is 1, assign p(x, y) to 0, increase the number of num_component by 1, and add the point p(x, y) to the neighbor queue neighbors;

[0037] (e) Take a point from the end of the neighbor queue in (d), detect its upper, lower, left, and right neighbor points, and if the value is 1, assign the point a value of 0 and put it back into the neighbor queue, and perform neighbor detection again until the neighbors is empty;

[0038] (f) Output the number of connected components num_component. If num_component = 1, the objects are considered to be grouped together. Otherwise, the connected component detection is not satisfied.

[0039] Furthermore, the abnormal situation detection is specifically: when the placement termination state is reached, there is no abnormal height value in the depth heighmap.

[0040] Furthermore, the present invention also proposes a robotic arm efficient manipulation placement learning reward training system based on image domain, comprising:

[0041] Application scenario design module: Design application scenarios and collect 4-channel RGBD-heightmap of the workspace through an RGBD camera;

[0042] Reward function design module: select a single-channel D-heightmap to perform overall reward function design in the image domain; design single-step rewards: learn the single-step reward r step Includes mobile bonus change , collision reward contact , Repeated reward r repeat ; Design the termination reward: the learning termination reward r terminal Including task completion termination reward r done and the sequence action reward r req ;

[0043] Reward calculation function acquisition module: obtain the reward calculation function of network learning as r t =r step +r terminal ;

[0044] Training module: Build an enhanced simulation learning training project to conduct learning training, and conduct physical migration tests on the trained network.

[0045] The advantages of the present invention compared with the prior art are:

[0046] (1) The present invention quantifies the performance of the terminal state by merging in the image domain, which will guide the terminal state to converge to a better domain rather than a specific value, thereby having better adaptability to objects of different numbers and irregular shapes as well as discontinuities.

[0047] (2) The present invention performs learning and reward design based on the image domain, which has better portability from simulation to the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a design flow chart of the present invention;

[0049] Figure 2 It is a schematic diagram of connected domain analysis of the present invention;

[0050] Figure 3 It is a physical and virtual simulation diagram of the present invention;

[0051] Figure 4 It is a physical test diagram of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0053] like Figure 1 As shown, the specific implementation steps of the present invention are as follows:

[0054] Step 1: Design application scenarios, such as Figure 3 As shown, the cuboid finite volume container is the robot workspace, and any number of objects are randomly placed. The RGBD camera is used to collect 4-channel RGBD images of the workspace, and the image is converted into a heightmap in the top-down direction according to the camera internal parameters. The 4-channel RGBD-heightmap is used as the neural network state input, and the original image is extracted into a high-dimensional feature vector through Resnet-50, and learning is performed based on the reinforcement learning algorithm. This patented method is suitable for commonly used reinforcement learning algorithms such as DQN, PPO, etc.

[0055] Step 2: Design a reward function in the image domain based on a single-channel D-heightmap instead of evaluating it in Cartesian space. By merging in the image domain to quantify the performance of the terminal state, this will guide the terminal state to converge to a better domain rather than a specific value. We design a “single-step reward r step ” and “termination reward r terminal ” to encourage fast grouping of scattered objects in a box while minimizing the gaps between objects.

[0056] Step 3: Design the single-step reward in step 2. The learned single-step reward includes “movement reward r change ", "Collision Reward r contact ", "Repeat reward r repeat ". The following is a detailed description:

[0057] (1) Movement reward design: We encourage the movement of objects because it is necessary for the task, but we do not constrain the direction of movement because pushing objects back and forth in complex working conditions may be necessary to maximize the task reward. Therefore, we determine whether the object has moved by comparing the difference in the D-heightmap of the previous and next two environmental states. If it has moved, then r change =0.1;

[0058] (2) Collision reward design: We found that in the simulation environment, the agent often touches the upper edge of the object to flip the object to achieve a suitable posture change. Obviously, in real physical experiments, this operation often causes the robot arm to stop suddenly or damage the object, which is an unreasonable behavior. Therefore, when it is detected that the current action will cause the gripper to collide vertically with the object, a penalty reward r is applied. contact =-0.5, and judge by detecting whether there is an object in the D-heightmap corresponding to the action point.

[0059] (3) Repeated reward design: If the execution of an action does not cause the environment to change, the agent will often repeat the current action again. However, there are various constraints and collisions in the pushing process, which will affect the environmental changes and make the network easily fall into the local optimum. Therefore, we design a penalty reward. If the same action is repeated more than three times, r will be given. repeat =-1.5.

[0060] (4) When all three rewards (1), (2) and (3) are not satisfied, the single-step reward is 0. In summary,

[0061]

[0062] Step 4: Design the termination reward in step 2. The termination reward for learning includes the task completion termination reward r done ” and “sequence action reward r req ". The following is a detailed description:

[0063] (1) Task completion termination reward design: Task completion judgment needs to meet two constraints: connected domain detection and abnormal situation detection. Only when both conditions are met at the same time, the task is considered completed and a one-time reward of r is given. done =3, as explained below.

[0064] 1. Design of connected domain detection: By analyzing the connected domain in the pixel space, the degree of aggregation and discreteness of object placement can be effectively evaluated. This method uses a breadth-first search algorithm to search the number of connected domains in the image domain. When the number of connected domains is 1, it is considered that the object has been placed to the desired effect. However, when there are some irregular objects in the workspace, even if two objects have been squeezed together, there are still gaps between them due to irregular edges, such as Figure 2 As shown on the left, through the connected domain analysis, there are 3 connected domains. Continuing to push cannot effectively improve the overall placement effect. Therefore, when analyzing the original image domain, it is difficult to achieve strict connected domain detection, which leads to a long training time. This method first performs maximum pooling preprocessing on the original image to eliminate the effects caused by small gaps and noise, such as Figure 2As shown on the right. Through further connected domain analysis, we can see that there is only one connected domain, that is, the three irregular objects are considered to be grouped together. This design is an acceptable quantification scheme for the final placement effect and has better adaptability to irregular objects.

[0065] The specific steps are as follows:

[0066] (a) Remove the depth map outside the container according to the container workspace and size, and perform denoising on the depth value d(x, y) of each pixel. When d(x, y)>0.01m, assign the value to 1, otherwise assign it to 0, and obtain the binary image map1;

[0067] (b) With a pooling kernel of 2 and a step size of 2, the image in (a) is subjected to maximum pooling to obtain image map2, eliminating initial gaps and errors and reducing the detection traversal area;

[0068] (c) Initialize the number of connected domains num_component = 0, and the neighbor detection queue is empty neighbors = [ ];

[0069] (d) Traverse the value p(x, y) corresponding to each pixel point in map2 in (b), starting from the origin of the pixel coordinate system, and determine whether p(x, y) is 1. If it is 1, assign p(x, y) to 0, increase the number of num_component by 1, and add the point p(x, y) to the neighbor queue neighbors;

[0070] (e) Take a point from the end of the neighborhood queue in (d) and detect its upper, lower, left, and right neighboring points. If the value is 1, assign the point a value of 0 and put it back into the neighborhood queue, and perform neighborhood detection again until the neighbors are empty.

[0071] (f) Output the number of connected domains num_component. If num_component = 1, the objects are considered to be grouped together, otherwise the connected domain detection is not satisfied.

[0072] 2. Abnormal situation detection design: During the placement task, abnormal situations such as objects getting stuck or stacked often occur. We believe that when the placement termination state is reached, there should be no abnormal height values ​​in the depth heighmap, and the neural network can learn strategies to deal with abnormal situations.

[0073] (2) Sequential action reward design: The success rate of the task is greatly affected by the order of pushing. We hope to design a reward function to encourage the agent to complete the task with fewer actions. Therefore, we designed a sequential action reward function.

[0074] r seq =4-(N-1)×0.25

[0075] Among them, the reward value decreases as the number of actions N∈(0,16) increases.

[0076] (3) When (1) and (2) are not satisfied or the maximum number of action steps is exceeded, the reward is considered to be 0. In summary, the termination reward r terminal as follows:

[0077]

[0078] Step 5: Combine the rewards from step 4 and step 5 to obtain the reward calculation function for network learning: r t =r step +r terminal .

[0079] Step 6: Combine steps 1 / 2 / 3 / 4 / 5 to build an enhanced simulation learning and training project for learning and training, and conduct a physical migration test on the trained network. The test results are as follows: Figure 4 As shown, Figure 4 a is before the test, Figure 4 b is after the test. It can be seen that this method has good adaptability to irregular objects and can adapt to dynamically changing scenes.

[0080] Furthermore, the present invention also proposes a robotic arm efficient manipulation placement learning reward training system based on image domain, comprising:

[0081] Application scenario design module: Design application scenarios and collect 4-channel RGBD-heightmap of the workspace through an RGBD camera;

[0082] Reward function design module: select a single-channel D-heightmap to perform overall reward function design in the image domain; design single-step rewards: learn the single-step reward r step Includes mobile bonus change , collision reward contact , Repeated reward r repeat ; Design the termination reward: the learning termination reward r terminal Including task completion termination reward r done and the sequence action reward r req ;

[0083] Reward calculation function acquisition module: obtain the reward calculation function of network learning as r t =r step +r terminal ;

[0084] Training module: Build an enhanced simulation learning training project to conduct learning training, and conduct physical migration tests on the trained network.

[0085] Contents not described in detail in this specification belong to common knowledge of those skilled in the art.

Claims

1. A reward training method for efficient manipulation placement learning of a robotic arm based on an image domain, characterized by: include: Step 1: Design the application scenario and use the RGBD camera to collect the 4-channel RGBD-heightmap of the workspace; Step 2: Select a single-channel D-heightmap to perform overall reward function design in the image domain: Design a single-step reward r for the reinforcement learning algorithm step and the termination reward r terminal to encourage quick placement of scattered objects in a box while minimizing the gaps between objects; Step 3: Design the single-step reward in step 2: the learned single-step reward r step Includes mobile bonus change , collision reward contact , Repeated reward r repeat ; (1) Movement reward design: By comparing the difference between the two environmental states D-heightmap, we can determine whether the object has moved. If it has moved, the movement reward r change =0.1; (2) Collision reward design: When it is detected that the current action will cause the gripper to collide vertically with the object, a collision reward r is applied. contact = -0.5, judging by detecting whether there is an object in the D-heightmap corresponding to the action point; (3) Repeated reward design: If the same action is repeated more than three times, a repeated reward r will be given. repeat =-1.5; (4) When the mobile reward r change , collision reward contact , Repeated reward r repeat When all three rewards are not met, the single-step reward is 0; Step 4: Design the termination reward in step 2: the learned termination reward r terminal Including task completion termination reward r done and the sequence action reward r req ; (1) Task completion reward r done Design: Task completion judgment needs to meet two constraints: connected domain detection and abnormal situation detection. Only when both conditions are met at the same time, the task is considered completed and r is given. done =3; (2) Sequence action reward r req Design: Design a sequence action reward function r seq ＝4-(N-1)×0.25 Among them, the reward value decreases as the number of actions N∈(0,16) increases; (3) When the task is completed, the reward r is terminated done and the sequence action reward r req If all the conditions are not met or the maximum number of action steps is exceeded, the reward is considered to be 0; The connected domain detection is specifically: (a) Remove the depth map outside the container through the container workspace and size, and perform denoising on the depth value d(x, y) of each pixel. When d(x, y)>0.01m, assign the value to 1, otherwise assign it to 0, and obtain the binary image map1; (b) Perform maximum pooling on the image in (a) with a pooling kernel of 2 and a step size of 2 to obtain image map2, eliminating gaps and errors and reducing the detection traversal area; (c) Initialize the number of connected domains num_component = 0, and the neighbor detection queue is empty neighbors = []; (d) Traverse the value p(x, y) corresponding to each pixel point in map2 in (b), starting from the origin of the pixel coordinate system, and determine whether p(x, y) is 1. If it is 1, assign p(x, y) to 0, increase the number of num_component by 1, and add the point p(x, y) to the neighbor queue neighbors; (e) Take a point from the end of the neighbor queue in (d), detect its upper, lower, left, and right neighbor points, and if the value is 1, assign the point a value of 0 and put it back into the neighbor queue, and perform neighbor detection again until the neighbors is empty; (f) Output the number of connected domains num_component. If num_component = 1, the objects are considered to be grouped together. Otherwise, the connected domain detection is not satisfied. Step 5: Combine the rewards from step 4 and step 5 to obtain the reward calculation function for network learning: r t =r step +r terminal ; Step 6: Combined with steps 1-5, build an enhanced simulation learning and training project for learning and training, and conduct physical migration tests on the trained network.

2. According to claim 1, a method for rewarding training for efficient manipulation placement learning of a robotic arm based on an image domain, characterized in that: The specific application scenario is: using a rectangular finite volume container as the robot arm workspace, and randomly placing any number of objects; The RGBD camera is used to collect 4-channel RGBD images of the workspace, and the image is converted into a 4-channel RGBD-heightmap in the top-down direction according to the camera's intrinsic parameters. The 4-channel RGBD-heightmap is used as the neural network state input, and the original image is extracted into a high-dimensional feature vector through Resnet-50, and learning is performed based on the reinforcement learning algorithm.

3. The image-domain-based robotic arm efficient manipulation placement learning reward training method according to claim 1, characterized in that: Single step reward r step Specifically:

4. The method for rewarding training for efficient manipulation placement learning of a robotic arm based on image domain according to claim 1, characterized in that: Termination Reward r terminal Specifically:

5. The image-domain-based robotic arm efficient manipulation placement learning reward training method according to claim 1, characterized in that: The abnormal situation detection is specifically: when the placement termination state is reached, there is no abnormal height value in the depth heighmap.

6. A robotic arm efficient manipulation placement learning reward training system based on image domain, characterized by: include: Application scenario design module: Design application scenarios and collect 4-channel RGBD-heightmap of the workspace through an RGBD camera; Reward function design module: Select a single-channel D-heightmap to perform overall reward function design in the image domain, that is, design a single-step reward r for the reinforcement learning algorithm. step and the termination reward r terminal to encourage the quick placement of scattered objects in the box while minimizing the gaps between objects; design the single-step reward: the learned single-step reward r step Includes mobile bonus change , collision reward contact , Repeated reward r repeat ; Design the termination reward: the learning termination reward r terminal Including task completion termination reward r done and the sequence action reward r req ; The design of the single-step reward includes: the learned single-step reward r step Includes mobile bonus change , collision reward contact , Repeated reward r repeat ; (1) Movement reward design: By comparing the difference between the two environmental states D-heightmap, we can determine whether the object has moved. If it has moved, the movement reward r change =0.1; (2) Collision reward design: When it is detected that the current action will cause the gripper to collide vertically with the object, a collision reward r is applied. contact = -0.5, judging by detecting whether there is an object in the D-heightmap corresponding to the action point; (3) Repeated reward design: If the same action is repeated more than three times, a repeated reward r will be given. repeat =-1.5; (4) When the mobile reward r change , collision reward contact , Repeated reward r repeat When all three rewards are not met, the single-step reward is 0; Design the termination reward, including: the learning termination reward r terminal Including task completion termination reward r done and the sequence action reward r req ; (1) Task completion reward r done Design: Task completion judgment needs to meet two constraints: connected domain detection and abnormal situation detection. Only when both conditions are met at the same time, the task is considered completed and r is given. done =3; (2) Sequence action reward r req Design: Design a sequence action reward function r seq ＝4-(N-1)×0.25 Among them, the reward value decreases as the number of actions N∈(0,16) increases; (3) When the task is completed, the reward r is terminated done and the sequence action reward r req If all the conditions are not met or the maximum number of action steps is exceeded, the reward is considered to be 0; The connected domain detection is specifically: (a) Remove the depth map outside the container through the container workspace and size, and perform denoising on the depth value d(x, y) of each pixel. When d(x, y)>0.01m, assign the value to 1, otherwise assign it to 0, and obtain the binary image map1; (b) Perform maximum pooling on the image in (a) with a pooling kernel of 2 and a step size of 2 to obtain image map2, eliminating gaps and errors and reducing the detection traversal area; (c) Initialize the number of connected domains num_component = 0, and the neighbor detection queue is empty neighbors = []; (d) Traverse the value p(x, y) corresponding to each pixel point in map2 in (b), starting from the origin of the pixel coordinate system, and determine whether p(x, y) is 1. If it is 1, assign p(x, y) to 0, increase the number of num_component by 1, and add the point p(x, y) to the neighbor queue neighbors; (e) Take a point from the end of the neighbor queue in (d), detect its upper, lower, left, and right neighbor points, and if the value is 1, assign the point a value of 0 and put it back into the neighbor queue, and perform neighbor detection again until the neighbors is empty; (f) Output the number of connected domains num_component. If num_component = 1, the objects are considered to be grouped together. Otherwise, the connected domain detection is not satisfied. Reward calculation function acquisition module: obtain the reward calculation function of network learning as r t =r step + r terminal ； Training module: Build an enhanced simulation learning training project to conduct learning training, and conduct physical migration tests on the trained network.

Images