A target grasping method and system based on deep reinforcement learning and action coordination

Abstract

The application discloses a target grasping method and system based on deep reinforcement learning and action cooperation, and the method comprises the following steps: judging whether a target object is covered or not, if the target object is covered, inputting all input images into a moving action evaluation network to obtain a moving action value; completing the moving of the target object according to the moving action value until the target object is no longer covered; inputting all input images into a grasping action evaluation network to obtain a grasping action value; controlling a robot according to the grasping action value to complete the grasping of the target object; judging whether the grasping of the object is successful, if the grasping fails, inputting all input images into a pushing action evaluation network to obtain a pushing action value, and completing the pushing of the target object according to the pushing action value; inputting the image of the pushed target object into the grasping action evaluation network again to obtain a grasping action value, and completing the grasping of the target object according to the grasping action value.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a target grasping method and system based on deep reinforcement learning and action coordination. Background Technology

[0002] Grasping is a fundamental action in multi-functional robotic tasks. Researchers have proposed numerous robotic grasping methods, which typically utilize deep learning and / or reinforcement learning to generate grasping strategies and demonstrate good performance in unobstructed scenes. However, in real-world grasping scenarios, objects are often randomly placed within occluded environments, posing challenges to the grasping action. To address this issue, many researchers have investigated pre-grasping collaboration, with most methods extending non-understanding pushing actions into the action space. Some methods combine pushing and grasping strategies for sequential operations, as pushing helps rearrange objects, making grasping easier. These methods primarily target goal-independent grasping tasks and cannot grasp according to user requirements.

[0003] Object-oriented grasping aims to grasp user-specified objects. This is particularly challenging in occluded scenes, as accurately locating and grasping a target object with only partial observation data is difficult. Most existing object-oriented grasping methods employ a two-branch Q-network structure to learn push-grab synergy to facilitate object-oriented grasping in occluded scenes. For example, Xu et al. used RGB-D height maps and object segmentation masks as inputs and trained two independent Q-networks to learn the synergy between push and grab actions. Yu et al. did not use object segmentation masks but instead designed a target similarity network (TSN) to estimate object localization. They also fed visual features into two deep Q-networks to make push or grab decisions. Li et al. used a similar two-branch deep Q-network to estimate the Q-value distribution while decoupling position and angle predictions.

[0004] A key assumption of the above methods is that, in each training step, the robot should choose the action with the larger Q-value (push or grab). However, this assumption is poorly interpretable and not always reliable. Furthermore, most existing methods address the grasping challenge in occluded scenes by pushing stacked objects in a specific direction, but pushing actions are sometimes ineffective for target-oriented grasping tasks and may even further disrupt the grasping scene.

[0005] Existing goal-oriented crawling methods can be divided into two categories: goal-oriented crawling without action coordination and goal-oriented crawling with action coordination.

[0006] Regarding action-free, target-oriented grasping, some scholars have proposed methods to acquire target objects through a single grasping action. This invention categorizes these methods into two-stage and one-stage approaches. The two-stage approach refers to the fact that target designation and grasping detection are not integrated; instead, task-independent grasping detection is performed first, followed by target filtering using appropriate heuristics. Murali et al. used a binary mask of the target object to crop the scene point cloud, then performed 6-DOF grasping detection on the target object point cloud. Liri et al. used speech to designate the target object from the object detection results and then forwarded it to the grasping detection module. Another approach is to directly generate grasping suggestions in the form of images and verbal instructions based on the input target information; this approach is called the single-stage approach. Suggestions use sketches to represent target objects, and an end-to-end network is designed to learn and generate target-oriented grasping. Yang et al. designed a multi-target dense descriptor that learns target-oriented grasping capabilities through DRL, with targets defined by RGB images. However, due to limited action space, the above methods perform poorly in occluded scenes. In some closed and cluttered scenes, acquiring the target object through a single grasping action is almost impossible. Therefore, it is necessary to explore motion coordination algorithms to solve target-oriented grasping tasks in occluded scenes.

[0007] Regarding target-oriented grasping with action coordination, many studies have explored the synergistic effects of pre-grab operations (such as pushing and grabbing) to improve the success rate of grasping targets. Xu et al. proposed a target-conditional hierarchical reinforcement learning method, including a target relabeling strategy and alternative training, to address the problem of inefficient samples and accelerate training. Yang et al. proposed a Bayesian-based target exploration strategy and a classifier-based push-grab coordination strategy. To address the learning problem in a large state space, Li et al. decoupled the action space and learned the grasping position and angle separately, i.e., using an additional branch to predict the grasping angle. In addition, some new methods have gradually emerged. Among them, Zuo et al. used a graph-based deep reinforcement learning model instead of the classic DQN to explore hidden objects, thereby achieving better push-grab coordination performance.

[0008] Currently, existing motion coordination methods for target-oriented robot grasping are mainly based on pushing and grasping. However, through experimental testing of these methods, this invention found that push-grab coordination is not effective in handling some common occlusion scenarios. First, in stacked scenarios, the agent struggles to perform effective pushing actions, which is inherent to the definition of pushing actions. Second, for push-grab coordination, in some cases, multiple action steps are required to successfully grasp the target object, which is very time-consuming. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a target grasping method and system based on deep reinforcement learning and action collaboration. This invention expands the action space dimension by constructing a three-branch action collaboration network. Furthermore, this invention models the method as a hierarchical framework to improve the efficiency of policy learning.

[0010] On the one hand, it provides a target grasping method based on deep reinforcement learning and action coordination;

[0011] A target grasping method based on deep reinforcement learning and action coordination includes:

[0012] S101: Input the color height image, depth height image, and target object mask image as input images; input the input images into the first judgment module of the three-branch depth Q network to determine whether the target object is occluded. If it is occluded, input all the input images into the motion evaluation network to obtain the motion value; control the robot according to the motion value to complete the movement of the target object, and proceed to S102; if it is not occluded, proceed to S103.

[0013] S102: Repeat S101 until the target object is no longer obscured, then proceed to S103;

[0014] S103: Input all input images into the grasping action evaluation network to obtain the grasping action value; control the robot according to the grasping action value to complete the grasping of the target object;

[0015] S104: The second judgment module of the three-branch deep Q network determines whether the object grasping is successful. If the grasping is successful, the process ends; if the grasping fails, the input images are all input into the pushing action evaluation network to obtain the pushing action value. Based on the pushing action value, the robot is controlled to complete the pushing of the target object.

[0016] S105: Input the image of the target object after pushing it back into the grasping action evaluation network to obtain the grasping action value. Based on the grasping action value, control the robot to complete the grasping of the target object, and return to S104.

[0017] On the other hand, a target grasping system based on deep reinforcement learning and action coordination is provided;

[0018] A target grasping system based on deep reinforcement learning and action coordination includes:

[0019] The judgment module is configured to: take a color height image, a depth height image, and a target object mask image as input images; input the input images into the first judgment module of a three-branch depth Q network to determine whether the target object is occluded; if occluded, input the input images into the motion evaluation network to obtain motion values; control the robot according to the motion values ​​to complete the movement of the target object, and proceed to the repeat module; if the target object is not occluded, proceed to the grasping module.

[0020] The repeat module is configured to repeat the judgment module until the target object is no longer occluded, and then enter the grasping module;

[0021] The grasping module is configured to: input all input images into the grasping action evaluation network to obtain the grasping action value; and control the robot based on the grasping action value to complete the grasping of the target object.

[0022] The second judgment module is configured as follows: the second judgment module of the three-branch deep Q network determines whether the object grasping is successful. If the grasping is successful, the process ends; if the grasping fails, the input images are all input into the pushing action evaluation network to obtain the pushing action value. Based on the pushing action value, the robot is controlled to complete the pushing of the target object.

[0023] The re-grasping module is configured to: input the image of the pushed target object back into the grasping action evaluation network to obtain the grasping action value; control the robot based on the grasping action value to complete the grasping of the target object; and return to the second judgment module.

[0024] Furthermore, an electronic device is also provided, including:

[0025] Memory, used for non-transitory storage of computer-readable instructions; and

[0026] Processor, for executing the computer-readable instructions,

[0027] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0028] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0029] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0030] The above technical solution has the following advantages or beneficial effects:

[0031] (1) This invention proposes a novel three-branch deep Q-network, namely MPGNet, which can learn the coordination between movement, pushing, and grasping actions to achieve target-oriented grasping in fuzzy scenes. To the best of our knowledge, the proposed method is the first to address the challenging task of target-oriented grasping in fuzzy scenes by leveraging the coordination of movement, pushing, and grasping actions.

[0032] (2) The present invention designs a multi-stage training strategy for MPGNet, that is, to train each action individually first, and then to train them together, so as to effectively learn the coordination of movement-push-grab.

[0033] (3) The present invention conducted a large number of experiments in a simulation environment to evaluate MPGNet and further proved that the real-world robot system can reliably complete the target-oriented grasping task in occluded scenes. Attached Figure Description

[0034] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0035] Figure 1 This is a flowchart of the overall network architecture of Example 1;

[0036] Figure 2 This is a schematic diagram of the algorithm operation in Example 1; Figure 2 In this context, A represents the initial state. Figure 2 In this context, B indicates that the target object is moved; Figure 2 In this context, C represents pushing the target object. Figure 2 In this context, D indicates that the target object is being grasped;

[0037] Figure 3 This is a schematic diagram of the actual experimental platform in Example 1;

[0038] Figure 4 This is a schematic diagram of the training curve for Example 1;

[0039] Figure 5 This is a visual diagram of the network output of Example 1;

[0040] Figure 6 This is a schematic diagram of the ablation experiment results in Example 1;

[0041] Figure 7(a) is a schematic diagram showing the comparison results of the task success rate in Example 1;

[0042] Figure 7(b) is a schematic diagram showing the comparison results of the number of actions in Example 1;

[0043] Figure 8 This is a comparison diagram of the action sequence in Example 1; Figure 8 (a) in the diagram represents the processing results of the MPGNet network; Figure 8 (b) in the diagram represents the result of Efficient Push-Grasping.

[0044] Figure 9 This is a schematic diagram of the internal structure of the Mobilenet V3 block in Example 1;

[0045] Figure 10 This is a schematic diagram of the internal structure of the capture network, push network, or mobile network in Example 1;

[0046] Figure 11 This is a schematic diagram of the action evaluation network structure in Example 1;

[0047] Figure 12 This is a schematic diagram of the end effector structure of the robot arm in Example 1. Detailed Implementation

[0048] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] The focus of this invention is target-oriented grasping in occluded scenarios, where the target object is specified by a binary mask, with the goal of grasping the target object with as few robot operations as possible. Most existing methods rely on push-grasp cooperation to accomplish this task. To provide a more robust target-oriented grasping pipeline, this invention proposes MPGNet, a three-branch network for learning the cooperation between movement, push, and grasping actions. This invention also proposes a multi-stage training strategy to train MPGNet, which includes three policy networks corresponding to the three actions. The effectiveness of the method presented in this invention is demonstrated through simulations and real-world experiments.

[0050] Example 1

[0051] This embodiment provides a target grasping method based on deep reinforcement learning and action coordination;

[0052] A target grasping method based on deep reinforcement learning and action coordination includes:

[0053] S101: The target's color height image, depth height image, and target object mask image are used as input images; the input images are input into the first judgment module of the three-branch depth Q network to determine whether the target object is occluded. If it is occluded, the input images are all input into the motion evaluation network to obtain the motion value; based on the motion value, the robot is controlled to complete the movement of the target object, and proceeds to S102; if it is not occluded, proceeds to S103.

[0054] S102: Repeat S101 until the target object is no longer obscured, then proceed to S103;

[0055] S103: Input all input images into the grasping action evaluation network to obtain the grasping action value; control the robot according to the grasping action value to complete the grasping of the target object;

[0056] S104: The second judgment module of the three-branch deep Q network determines whether the object grasping is successful. If the grasping is successful, the process ends; if the grasping fails, the input images are all input into the pushing action evaluation network to obtain the pushing action value. Based on the pushing action value, the robot is controlled to complete the pushing of the target object.

[0057] S105: Input the image of the target object after pushing it back into the grasping action evaluation network to obtain the grasping action value. Based on the grasping action value, control the robot to complete the grasping of the target object, and return to S104.

[0058] Furthermore, the method also includes:

[0059] The target's color and depth images are acquired. These images are then projected upwards along the direction of gravity to obtain a color height image and a depth-height image. The target object mask image is then derived from the color image. In the simulation environment, the target object mask image can be obtained by calling the simulation environment's (CoppeliaSim) API. In real-world experiments, a finely tuned UOAIS model can be used to obtain the target object mask image.

[0060] Furthermore, the training process of the three-branch deep Q-network includes:

[0061] Construct separate evaluation networks for grasping actions, moving actions, and pushing actions;

[0062] The grasping action evaluation network is trained by targetless grasping single action learning to obtain a one-stage grasping action evaluation network.

[0063] The grasping action evaluation network of the first stage is trained with a specific grasping action learning method to obtain a pre-trained grasping action evaluation network.

[0064] The mobile motion evaluation network is trained with a specific goal of learning mobile motion to obtain a pre-trained mobile motion evaluation network.

[0065] The propulsive action evaluation network was trained with a specific purpose to obtain a pre-trained propulsive action evaluation network.

[0066] The three-branch deep Q-network is obtained by splicing together the initially trained grasping action evaluation network, the initially trained moving action evaluation network, the initially trained pushing action evaluation network, the first judgment module, and the second judgment module.

[0067] The three-branch deep Q-network is jointly trained to obtain the trained three-branch deep Q-network.

[0068] Furthermore, such as Figure 1 As shown, the trained three-branch deep Q-network includes:

[0069] The system comprises a motion capture evaluation network, a motion movement evaluation network, a push motion evaluation network, a first decision module, and a second decision module.

[0070] The first judgment module determines that the occluded end is connected to the input of the motion evaluation network; the first judgment module determines that the unoccluded end is connected to the input of the grasping motion evaluation network.

[0071] The output of the grasping action evaluation network is connected to the input of the second judgment module. The judgment result of the second judgment module is that the end of the object that has been successfully grasped is the end; the judgment result of the second judgment module is that the end of the object that has not been successfully grasped is connected to the input of the pushing action evaluation network.

[0072] Furthermore, such as Figure 9 and Figure 10 As shown, the internal structures of the grasping action evaluation network, the moving action evaluation network, and the pushing action evaluation network are consistent. The internal structure of the grasping action evaluation network includes: a first MobileNetV3 and a second MobileNetV3 connected in parallel; the output terminals of the first MobileNetV3 and the second MobileNetV3 are both connected to the input terminal of the grasping action evaluation sub-network, and the output terminal of the grasping action evaluation sub-network is used to output the grasping action Q-value map; the grasping action evaluation sub-network is implemented by a fully convolutional neural network (FCN).

[0073] The internal structure of the mobile motion evaluation network includes: a third MobileNetV3 and a fourth MobileNetV3 connected in parallel; the outputs of the third MobileNetV3 and the fourth MobileNetV3 are both connected to the input of the mobile motion evaluation sub-network, and the output of the mobile motion evaluation sub-network is used to output the Q-value map of the grasping action; the mobile motion evaluation sub-network is implemented by a fully convolutional neural network (FCN).

[0074] The internal structure of the push action evaluation network includes: a fifth MobileNetV3 and a sixth MobileNetV3 connected in parallel; the outputs of the fifth MobileNetV3 and the sixth MobileNetV3 are both connected to the input of the push action evaluation sub-network, and the output of the push action evaluation sub-network is used to output the Q-value map of the grasping action; the push action evaluation sub-network is implemented by a fully convolutional neural network (FCN).

[0075] Furthermore, the internal structures of the grasping action evaluation subnetwork, the moving action evaluation subnetwork, and the pushing action evaluation subnetwork are consistent, each including: a first batch of normalization layers, a first activation function layer, a first convolutional layer, a second batch of normalization layers, a second activation function layer, a second convolutional layer, and an upsampling layer connected in sequence.

[0076] Furthermore, the Q-function fitted by the fully convolutional neural network is extended into three feedforward fully convolutional networks (FCNs), which are named MPGNet, including a mobile network. and promote the network and motion capture evaluation network

[0077] MPGNet input target is g t State representation s t =(c t ,d t ), and output the high-density pixel Q-value mapping Q = (Q m Q g Q p ) in having with s T At the same resolution of 224*224, where Q M Q G Q P Let Q represent the action value of the corresponding movement, grasping, and pushing at the pixel position, respectively.

[0078] For each action network, two parallel MobileNetV3 networks are pre-trained on ImageNet to encode color height maps and depth height maps.

[0079] The common MobileNetV3 block is used for feature extraction of the target mask heightmap. Before sending all heightmaps to MobileNetV3, they are rotated by 16 angles to simplify the learning of directional motion primitives such as moving, grasping, and pushing.

[0080] like Figure 10 and Figure 11 As shown, the working processes of the grasping action evaluation network, the movement action evaluation network, and the pushing action evaluation network are consistent. The grasping action evaluation network is used for:

[0081] First, the color features extracted from the color height map and the mask features extracted from the mask image are concatenated in a channel-like manner. Then, the concatenated features are processed sequentially through the first batch of normalization layers, the first activation function layer, the first convolutional layer, the second batch of normalization layers, the second activation function layer, the second convolutional layer, and the upsampling layer to obtain the grasping action value Q of 16 pixels. The grasping action value Q of 16 pixels is used for grasping in different directions.

[0082] Similarly, the motion evaluation network obtains a 16-pixel motion value Q, which is used for movement in different directions.

[0083] Similarly, the push action evaluation network is used to obtain a 16-pixel push action value Q, which is then used for pushes in different directions.

[0084] The output of the three-branch deep Q-network is a Q-value map of 48 pixels (16 for movement in different directions, 16 for pushing in different directions, and 16 for grasping in different directions).

[0085] Furthermore, the loss function used during the training of the three-branch deep Q-network is:

[0086] Huber loss function:

[0087]

[0088] An ∈-greedy exploration strategy was adopted, initializing ∈ to 0.5 and annealing it to 0.1. This invention sets the future discount γ = 0.5 as a constant. The network was trained using the Adam optimizer with a fixed learning rate of 10. -4 The weight decays to 2. -5 betas is (0.9, 0.99).

[0089] Furthermore, the three-branch deep Q-network has a defined problem, state representation, action space, and reward function set during training.

[0090] The definition of the problem is:

[0091] The goal-oriented grasping task can be formulated as a goal-conditional Markov decision process:

[0092] In any given state s t At time t, given the target object g t The robot's robotic arm is considered as an intelligent agent, and the intelligent agent acts according to the policy π(s) t |g t Select and execute action a t Then transition to the new state s t+1 and receive an instant reward R(s) t ,a t ,s t+1 |g t The goal of deep reinforcement learning tasks is to explore the optimal policy π. * Optimal strategy π * The expression for determining action a, which maximizes the expected sum of future rewards, is as follows:

[0093]

[0094] Here, γ is the discount factor. Deep Q-learning is applied to train the neural network to map the action-value function Q. π (s t ,a t |g t It measures the state s at time t. t Take action a t The expected return at that time. Based on the latest Q-value, select actions to obtain a greedy deterministic strategy π(s). t |g t The learning objective is to minimize Q(s). t ,a t |g t ) and target value y t Time difference error δ t :

[0095] δ t =|Q(s) t ,a t |g t )-y t |;

[0096]

[0097] Where 'a' is the total set of available actions.

[0098] Here, state representation refers to: in order to better represent each state s tThe RGB and depth images captured by a fixed camera are projected upwards along the direction of gravity to generate a color height map c. t and depth-height map d t The state of the workspace at time t is represented as s. t =(c t ,d t The target object g is represented using a target segmentation mask map in the height direction. t It represents the area occupied by the target object in the workspace.

[0099] Among them, action space refers to:

[0100] The movement motion designed for moving obstacles is defined as follows:

[0101] a m =(p m ,θ m );

[0102] Where, p m =(x m ,y m ,z m θ is the center position of the two-finger robot gripper in the workspace. m Indicates the angle of movement. (x) m ,y m ) is obtained through eye-hand calibration from Q m The position is calculated from the pixels in the image.

[0103] z m =h m -4cm;

[0104] Among them, h m For point (x) m ,y m The height at point ).

[0105] During the movement, the robotic arm needs to move downwards to a depth below h. m You need to position your fingers 4 centimeters apart before you can close them. (Angle θ) m It is represented as one of 16 directions around the Z-axis from 0° to 360°, which means that the interval between each angle is 22.5°.

[0106] The pushing motion at the tip of the robotic arm is defined as:

[0107] a p =(p p ,d p ).

[0108] Each push is fixed at 10 centimeters in length, and the push trajectory is a straight line. p =(xp ,y p ,z p ) indicates the starting position of the pushing action, d p It is the direction vector of the push. (x) p ,y p ) is calibrated from Q p The position is calculated from the pixels in the image.

[0109] If h p If the value is greater than 0, set z. p =h p -1cm, allowing the robotic arm to lightly touch the object.

[0110] When h p When z is less than 0, set z p =2cm, to avoid the robotic arm touching the ground.

[0111] The driving direction is defined as one of 16 directions within a range of 0° to 360° around the z-axis.

[0112] The grabbing action is defined as:

[0113] a g =(p g ,θ g ),

[0114] Where, p g =(x g ,y g ,z g ) represents the middle position of the parallel claw grasping from top to bottom, while θ g This represents the gripping angle within the range of 0° to 360° around the z-axis, with 16 intervals. g For point (x) g ,y g The height at point z g =h g -4cm. The robotic arm needs to move downwards to below h. g The gripping action can only be performed at a distance of 4 centimeters. The structure of the robotic arm is as follows: Figure 12 As shown.

[0115] Furthermore, the grasping action evaluation network is trained using targetless grasping single-action learning to obtain a one-stage grasping action evaluation network. The reward function during the training process is defined as follows:

[0116] Define the grab reward function, which is unrelated to the target, as follows:

[0117]

[0118] It should be understood that, since this invention uses a grasping action evaluation network... The state is scored, so it is necessary to ensure that the grasping action evaluation network can accurately find a stable grasping position. In this stage, the robot performs grasping actions in a way that is independent of the target. Each round ends when all objects in the work area have been successfully grasped. To reduce the impact of occlusion, the positions of the m objects are set relatively discretely.

[0119] Furthermore, the grasping action evaluation network of the first stage is trained with a specific grasping action learning objective to obtain a pre-trained grasping action evaluation network. The reward function during the training process is defined as follows:

[0120] The target-oriented grabbing reward function is defined as:

[0121]

[0122] It should be understood that in the previous stage, this invention obtained a relatively stable grasping action evaluation network, which can be used for target recognition. In this stage, the goal of this invention is to train a target-oriented grasping network based on the previous stage. In practice, this invention randomly places m objects in the workspace and designates the nth object as the target. To improve learning efficiency, this invention isolates the target object from other objects.

[0123] Clearly, the agent receives zero reward in two situations: failed grasping and incorrect grasping. After sufficient training, the Q-value for successful grasping will roughly converge to a specific value. If the value exceeds this threshold, the state is considered suitable for grasping the target object. In the next stage, the trained grasping action evaluation network... It can be regarded as a discriminator.

[0124] Furthermore, the mobile action evaluation network is trained with a specific objective to learn mobile actions, resulting in a pre-trained mobile action evaluation network. The reward function during the training process is defined as follows:

[0125] The reward function for goal-oriented mobile single-action learning is defined as follows:

[0126]

[0127] in, in This indicates the improved grab Q-value achieved through movement actions. The grab Q-value after movement. The Q-value to be captured before moving.

[0128] It should be understood that, at this stage, occlusions on the target object are cleared by learning a movement strategy. To determine whether the target object is occluded, this invention proposes an occlusion checking mechanism in a simulated environment:

[0129] 1) Crop the depth-height map of the target object using a given target mask;

[0130] 2) Obtain the position and shape information of the target object through the interface functions provided by the simulation environment;

[0131] 3) Calculate the highest point (z-axis) on the surface of the target object, and compare the calculated highest point with the highest z-value on the clipped depth-height map to generate the occlusion state.

[0132] To train the mobile network, this invention first creates some simple occlusion scenarios: two objects are placed in sequence, and the first object is occluded by the other object by setting two adjacent positions.

[0133] This invention sets a flag for successful movement: after a movement action, if the obstruction of the target object is removed and the target object is not caught by the movement action, the flag is set to true.

[0134] Therefore, the agent should receive a positive reward. Each episode ends when the flag indicating a successful move changes from false to true.

[0135] This invention sets a maximum threshold of 5 for the number of movement operations. This means that if the agent performs 5 movement operations without clearing the occluded object, the simulated environment will be reset. After multiple training sessions, the agent can easily handle occluded scenes.

[0136] Next, this invention adds more objects to the workspace, allowing the agent to learn to handle more occluded scenes, making them more dispersed. In the early training phase of occluded scenes, the agent often struggles to meet the conditions for successful movement. Therefore, this invention introduces a comparison of Q before and after movement. g The method is to give rewards.

[0137] Furthermore, the pushing action evaluation network is trained with a specific purpose to learn pushing actions, resulting in a pre-trained pushing action evaluation network. The reward function during the training process is defined as follows:

[0138] The reward function for goal-oriented order-pushing action learning is defined as follows:

[0139]

[0140] in This indicates the improved grasping Q-value achieved through the push action. The grab Q-value after the push, The Q-value to be captured before moving.

[0141] It should be understood that, at this stage, the present invention focuses on training the network for the pushing action and freezing the weights of the pre-trained grasping action evaluation network and the movement network. The grasping action evaluation network also acts as a discriminator to help score the pushing action. If the pushing action improves the Q-value predicted by the grasping action evaluation network, the puzzle will receive a positive reward. In practice, the number of pushing actions in each round is limited to 5, with the grasping action performed at the end. The purpose of each pushing action is to continuously increase the Q-value of the target object. g until more than This invention constructs scenarios in which target objects are surrounded by other objects.

[0142] Furthermore, the three-branch deep Q-network is jointly trained to obtain the trained three-branch deep Q-network, and the reward function during the training process is defined as follows:

[0143] Training these three actions simultaneously, with the reward function for each action remaining consistent with the previous separate training.

[0144] In the earlier stages, the grasping action evaluation network was trained only in dispersed scenarios, the mobile network of this invention was trained in a specially designed occluded scenario, and the pushing network of this invention was trained in a relatively occluded scenario for push-grasp coordination. However, when they are combined, a distribution mismatch problem occurs due to the different training premises of each network. In addition, the mobile-push-grasp coordination requires further training to better adapt to real-world scenarios.

[0145] To address this problem, the present invention utilizes the previously trained network and Building upon this foundation, the move-push-grab strategy is further trained. In this phase, all three strategies are trained alternately while other network weights are frozen. Through this training phase, the three actions can be better coordinated with each other.

[0146] This invention sets up the same environmental conditions for movement, grasping, and pushing as for the previous stages. Ten objects are randomly placed in the workspace to construct a realistic occlusion scenario. In each episode, the agent first checks the occlusion state to decide whether to perform a movement operation. Then, a pushing action is performed until the maximum Q-value of the target object exceeds a threshold. The reward function for each action is the same as that for the previous few stages.

[0147] In the first two stages of training, this invention sets the number of objects in the scene to m = 5. After completing the first two stages, the Q-value stabilizes at 1.85. Therefore, this invention will... This is set as the threshold for determining whether a target object is graspable. Considering that the grasping and moving positions should be close to the target, this invention suggests generating a coarse target mask by calculating the maximum bounding rectangle of the target object, and assigning the Q value in the coarse mask to 0.

[0148] This invention describes a process for target-oriented grasping in occluded scenes. First, it designs three basic actions: movement, pushing, and grasping, which work together to complete the task. Second, instead of selecting actions solely based on Q-values, it proposes a workflow guided by human intuition to coordinate these three basic actions. Specifically, it designs a three-branch deep Q-network to learn a cooperative strategy for target-oriented grasping in occluded scenes. Compared to previous push-grab cooperative methods, using three primitive actions inevitably increases the learning difficulty; therefore, this invention designs a multi-stage training strategy to provide stable and rapid policy training. Figure 2 The core idea of ​​this work is illustrated through a practical example.

[0149] Figure 2 In this paper, A represents the initial state, B represents pushing occlusion, C represents moving occlusion, and D represents the grasping target. This invention models the target-oriented moving-pushing-grabbing collaborative process as a hierarchical reinforcement learning problem. This invention uses a grasping action evaluation network... Set as a discriminator, for each state s of the target object to be grasped. t Scoring is performed during the training of the mobile network. In its early stages, it fails to learn how to properly handle occlusion, resulting in negative rewards and inefficient learning. Therefore, when occlusion occurs, a well-trained grasping action evaluation network... It will be used to change the state s of the movement action. t Scoring is performed. If the increase in the improved grasping Q-value exceeds a threshold, the agent will receive a positive reward regardless of whether the movement eliminates the occlusion. When no occlusion is detected, a pushing action will be activated, and the robot will perform a pushing action to transition the state until the state becomes graspable by the discriminator.

[0150] like Figure 3As shown, this invention trains MPGNet in a simulated environment and then portes it to the real world. In the simulated environment (CoppeliaSim) of this invention, a UR5 robotic arm with an RG2 gripper is mounted in the work area. Two cameras are mounted in fixed positions with known extrinsic parameters to capture RGB-D images of the work area. In the real world, this invention constructs a similar scenario to the simulation: this invention uses a UR5 robotic arm equipped with a ROBOTIQ two-finger gripper as an agent to perform different actions, and a RealSense D435i RGB-D camera is horizontally fixed above the work area to capture color and depth images. Below, this invention will evaluate the proposed method through a series of experiments.

[0151] Grasping the Invisible is a target-oriented approach that uses a classifier-based strategy to coordinate push and grab actions to capture target objects in clutter. This method uses color segmentation to generate a target mask and detects whether the target is visible in order to coordinate or explore.

[0152] Efficient Push-Grasping employs several techniques to train object-oriented push-grass collaboration. This method utilizes object re-labeling to improve sampling efficiency. Furthermore, alternating training is employed during the training process, with push and grab learning occurring sequentially.

[0153] Grasping-Only uses a single FCN network to learn a greedy, deterministic grabbing strategy.

[0154] Target-Oriented VPG is a target-oriented approach that extends VPG by learning push-grab synergies through target masks as input. VPG outputs a Q-value map of push and grab, suitable for target-independent tasks. It executes the action with the highest Q-value in the target region.

[0155] The method of this invention is evaluated in occluded scenarios, where n objects are randomly placed in relatively adjacent positions within a simulated environment. This invention sets the first or second object as the target object to ensure that the target is occluded. The metrics are defined as follows:

[0156] Task success rate: In n test trials, the task success rate is defined as the average percentage of successful tests across n runs. A test is considered successful if the agent successfully grasps the target object within 10 actions.

[0157] Number of Actions: This invention defines the number of actions as the average number of actions required to successfully grasp the target, reflecting action efficiency. Furthermore, it also reflects scene complexity, as more complex scenes typically require more actions to grasp the target object.

[0158] Grabbing Success Rate: This invention records the average target grabbing success rate every k training iterations to reflect the model's grabbing ability during training. It can be defined as the ratio of the number of successful target grabs to the number of grabbing attempts.

[0159] This invention compares the performance of MPGNet with other baselines during training and testing. Specifically, Grasping the Invisible, Target-Oriented VPG, Efficient Push-Grasping, and MPGNet share the same training scenario: 10 objects are placed in the workspace, and the first or second object to be dropped is a target object in the occluded scene. For Grasping the Invisible and Target-Oriented VPG methods, the training process is recorded from scratch. For the Efficient Push-Grasping method, the training process is recorded starting from joint training.

[0160] This invention calculates the average target grasping success rate every 200 training iterations. During training, this invention limits the number of consecutive grasps for methods other than Grasping-Only to 2. If this number exceeds 2 without successfully grasping the target object, the system uses a heuristic method to generate a grasping action evaluation network instead of a grasping action evaluation network to accelerate training. Additionally, if the total number of actions exceeds 10 without reaching the target, the scene is reset. Compared to the baseline (e.g.) Figure 4 The success rate of Grasping-Only target crawling still shows no convergence trend and remains lower than other methods.

[0161] This invention also shows that, similar to Grasping the Invisible, the success rate of Target-Oriented VPG converges to approximately 0.35, while the training process of the former is not stable, indicating that the coordination strategy in the latter method is slightly more effective in terms of occlusion. The success rate curves show that Efficient Push-Grasping performs better than the other three methods, and the success rate converges to approximately 0.6, still lower than the proposed method. For MPGNet, the training process in stage V is recorded, as follows... Figure 4 As can be seen, MPGNet achieves the highest grasping success rate compared to other methods, converging at approximately 0.9. This demonstrates that MPGNet works effectively and efficiently for target-oriented grasping in occluded environments. A visualization of the MPGNet output is shown below. Figure 5 As shown.

[0162] Table 1 Comparison of Simulation Experiment Results

[0163]

[0164] This invention also conducted several simulation tests to assess whether the method could be generalized to other occlusion scenarios. Two scenarios with 15 and 30 objects were tested, respectively. The test scenarios were similar to the joint training scenarios but contained more objects. This invention randomly selected one of the first five objects as the target to ensure the target was highly occluded, aiming to demonstrate the robustness of MPGNet. This invention ran 50 tests on each scenario. The results, shown in Table 1, indicate that the system of this invention achieved optimal performance in terms of task success rate and number of motions.

[0165] This invention creates three variants of MPGNet: MPGNet w / o mt, MPGNet w / o at, and MPGNet w / o at&mt. MPGNet w / o mt indicates that MPGNet is trained using an alternative training method instead of a multi-stage strategy, signifying that this invention trains MPGNet from the joint training stage.

[0166] MPGNet w / o at indicates that MPGNet uses a multi-stage strategy for training, but does not perform alternative training, suggesting that action decisions during the joint training phase are based on the comparison of Q values ​​of the three action Q maps, rather than the proposed alternative training plan.

[0167] MPGNet w / o at&mt indicates that the three action policies are trained jointly without any pre-training, and action decisions are made by comparing Q-values.

[0168] Figure 6 This demonstrates that MPGNet outperforms the other three ablation versions in terms of capture success rate, proving the effectiveness of multi-stage and alternative training strategies.

[0169] The system of this invention was tested in real-world scenarios. Experiments were conducted in nine challenging scenarios containing both visible and invisible objects, where the target object was densely occluded. Each scenario contained objects with different textures and shapes. The method of this invention was compared with EfficientPush-Grasping and Grasping the Invisible, both of which showed relatively good performance in simulated tests. For each test scenario, the invention was run for 10 rounds. For each round, the invention ensured consistency in how the target object was occluded and appropriately adjusted the positions of the remaining objects. The model used in the real-world experiments of this invention was transferred from simulation to reality without any fine-tuning. The results for task success rate and number of actions are shown in Figures 7(a) and 7(b). It can be seen that MPGNet outperforms the baseline on all evaluation metrics. Figure 8 This demonstrates that the method of the present invention can capture the target object with minimal movement.

[0170] Regarding the generation of the target mask height map, this invention fine-tunes UOAIS to suit the scenario of this invention to generate the target mask height map.

[0171] The output of UOAIS is a set of segmented masks of objects in the scene with occlusion labels. In practice, this invention selects the occluded object with the smallest height value as the target to ensure that occlusion occurs.

[0172] Furthermore, in some additional real-world scenarios, this invention introduces ChatGPT4 as a multimodal large model and uses the proposed grasping algorithm to grasp occluded objects following language instructions, making the real-world testing of this invention more realistic and reliable. Specifically, this invention queries GPT4 to classify the detected masks and pre-provides candidate categories. Then, this invention prompts GPT4 to analyze the input language instructions and select the target mask as the input to the proposed algorithm.

[0173] This invention proposes MPGNet, a three-branch network designed to learn movement, pushing, and grasping collaborations for object-oriented grasping in occluded scenarios. In practice, this invention trains the network using a multi-stage strategy and incorporates domain knowledge with Q-values ​​to better coordinate these three actions. The method is compared with several baselines, demonstrating that MPGNet converges quickly and achieves a high grasping success rate. Simulation and real-world experiments further evaluate MPGNet, showing its ability to handle challenging scenarios well, including those that cannot be effectively addressed by common push-grab collaborations. Furthermore, this invention demonstrates that MPGNet can achieve instruction-based object grasping in occluded scenarios with the assistance of ChatGPT4.

[0174] Example 2

[0175] This embodiment provides a target grasping system based on deep reinforcement learning and action coordination;

[0176] A target grasping system based on deep reinforcement learning and action coordination includes:

[0177] The judgment module is configured to: take a color height image, a depth height image, and a target object mask image as input images; input the input images into the first judgment module of a three-branch depth Q network to determine whether the target object is occluded; if occluded, input the input images into the motion evaluation network to obtain motion values; control the robot according to the motion values ​​to complete the movement of the target object, and proceed to the repeat module; if the target object is not occluded, proceed to the grasping module.

[0178] The repeat module is configured to repeat the judgment module until the target object is no longer occluded, and then enter the grasping module;

[0179] The grasping module is configured to: input all input images into the grasping action evaluation network to obtain the grasping action value; and control the robot based on the grasping action value to complete the grasping of the target object.

[0180] The second judgment module is configured as follows: the second judgment module of the three-branch deep Q network determines whether the object grasping is successful. If the grasping is successful, the process ends; if the grasping fails, the input images are all input into the pushing action evaluation network to obtain the pushing action value. Based on the pushing action value, the robot is controlled to complete the pushing of the target object.

[0181] The re-grasping module is configured to: input the image of the pushed target object back into the grasping action evaluation network to obtain the grasping action value; control the robot based on the grasping action value to complete the grasping of the target object; and return to the second judgment module.

[0182] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should also be noted that the above modules, as part of a system, can be executed in a computer system such as a set of computer-executable instructions.

[0183] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0184] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0185] Example 3

[0186] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0187] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0188] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0189] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0190] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0191] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0192] Example 4

[0193] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0194] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A target grasping method based on deep reinforcement learning and action coordination, characterized in that, include: (1) Use the color height image, depth height image and target object mask image as input images; The input image is input into the first judgment module of the three-branch deep Q network to determine whether the target object is occluded. If it is occluded, the input image is input into the motion evaluation network to obtain the motion value. Based on the motion value, the robot is controlled to complete the movement of the target object and proceed to (2). If it is not occluded, proceed to (3). (2) Repeat (1) until the target object is no longer obscured, then proceed to (3); (3) Input all input images into the grasping action evaluation network to obtain the grasping action value; control the robot according to the grasping action value to complete the grasping of the target object; (4) The second judgment module of the three-branch deep Q network judges whether the object grasping is successful. If the grasping is successful, it ends; if the grasping fails, the input images are all input into the pushing action evaluation network to obtain the pushing action value. Based on the pushing action value, the robot is controlled to complete the pushing of the target object. (5) Input the image of the target object after pushing back into the grasping action evaluation network to obtain the grasping action value. Based on the grasping action value, control the robot to complete the grasping of the target object and return (4).

2. The target grasping method based on deep reinforcement learning and action coordination as described in claim 1, characterized in that, The method further includes: acquiring a color image and a depth image of the target; projecting the color image and the depth image upwards along the direction of gravity to obtain a color height image and a depth height image; and obtaining a mask image of the target object based on the color image.

3. The target grasping method based on deep reinforcement learning and action coordination as described in claim 1, characterized in that, The training process for the three-branch deep Q-network includes: Construct separate evaluation networks for grasping actions, moving actions, and pushing actions; The grasping action evaluation network is trained by targetless grasping single action learning to obtain a one-stage grasping action evaluation network. The grasping action evaluation network of the first stage is trained with a specific grasping action learning method to obtain a pre-trained grasping action evaluation network. The mobile motion evaluation network is trained with a specific goal of learning mobile motion to obtain a pre-trained mobile motion evaluation network. The propulsive action evaluation network was trained with a specific purpose to obtain a pre-trained propulsive action evaluation network. The three-branch deep Q-network is obtained by splicing together the initially trained grasping action evaluation network, the initially trained moving action evaluation network, the initially trained pushing action evaluation network, the first judgment module, and the second judgment module. The three-branch deep Q-network is jointly trained to obtain the trained three-branch deep Q-network.

4. The target grasping method based on deep reinforcement learning and action coordination as described in claim 3, characterized in that, The trained three-branch deep Q-network includes: The system comprises a motion capture evaluation network, a motion movement evaluation network, a push motion evaluation network, a first decision module, and a second decision module. The first judgment module determines that the occluded end is connected to the input of the motion evaluation network; the first judgment module determines that the unoccluded end is connected to the input of the grasping motion evaluation network. The output of the grasping action evaluation network is connected to the input of the second judgment module. The judgment result of the second judgment module is that the end of the object that has been successfully grasped is the end; the judgment result of the second judgment module is that the end of the object that has not been successfully grasped is connected to the input of the pushing action evaluation network.

5. The target grasping method based on deep reinforcement learning and action coordination as described in claim 3, characterized in that, The grasping action evaluation network is trained by targetless grasping single action learning to obtain a one-stage grasping action evaluation network. The reward function during the training process is defined as follows: Define the grab reward function, which is unrelated to the target, as follows: The first-stage grasping action evaluation network is trained with a specific grasping action learning objective to obtain a pre-trained grasping action evaluation network. The reward function during the training process is defined as follows: The target-oriented grabbing reward function is defined as:

6. The target grasping method based on deep reinforcement learning and action coordination as described in claim 3, characterized in that, The mobile action evaluation network is trained with a specific objective to learn mobile actions, resulting in a pre-trained mobile action evaluation network. The reward function during the training process is defined as follows: The reward function for goal-oriented mobile single-action learning is defined as follows: in, in This indicates the improved grab Q-value achieved through movement actions. The grab Q-value after the movement. The Q-value to be captured before moving.

7. The target grasping method based on deep reinforcement learning and action coordination as described in claim 3, characterized in that, The purposeful learning and training of the pushing action evaluation network yields a pre-trained pushing action evaluation network. The reward function during the training process is defined as follows: The reward function for goal-oriented order-pushing action learning is defined as follows: in This indicates the improved grasping Q-value achieved through the push action. The grab Q-value after the push, The Q-value to be captured before moving.

8. A target grasping system based on deep reinforcement learning and action coordination, characterized in that, include: The judgment module is configured to take a color height image, a depth height image, and a target object mask image as input images. The input image is fed into the first decision module of the three-branch deep Q-network to determine whether the target object is occluded. If it is occluded, the input image is fed into the motion evaluation network to obtain the motion value. Based on the motion value, the robot is controlled to move the target object and then enters the repeat module. If it is not obscured, proceed to the capture module; The repeat module is configured to repeat the judgment module until the target object is no longer occluded, and then enter the grasping module; The grasping module is configured to: input all input images into the grasping action evaluation network to obtain the grasping action value; and control the robot based on the grasping action value to complete the grasping of the target object. The second judgment module is configured as follows: the second judgment module of the three-branch deep Q network determines whether the object grasping is successful. If the grasping is successful, the process ends; if the grasping fails, the input images are all input into the pushing action evaluation network to obtain the pushing action value. Based on the pushing action value, the robot is controlled to complete the pushing of the target object. The re-grasping module is configured to: input the image of the pushed target object back into the grasping action evaluation network to obtain the grasping action value; control the robot based on the grasping action value to complete the grasping of the target object; and return to the second judgment module.

9. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-7.

10. A storage medium, characterized in that, Non-transitory storage of computer-readable instructions, wherein, when executed by a computer, the instructions of the method according to any one of claims 1-7 are executed.

Images