A method for a robot arm to grasp a target object in a dense scene

Abstract

The application discloses a method for a mechanical arm to grab a target object in a dense scene, comprising: building a scene similar to a real environment through V-rep dynamic simulation software; building a pushing and grabbing collaborative strategy network of an encoder-decoder structure in the simulation environment, the overall structure being composed of a dense convolutional neural network DenseNet121 and two parallel fully convolutional neural networks; building a real dense stacked complex scene, transplanting the pushing and grabbing collaborative network trained on the simulation end to a real platform, and making the mechanical arm make action decisions to grab the target object in the real environment. The method uses a deep reinforcement learning method to continuously try actions in the working environment to obtain reward values to train the pushing and grabbing collaborative strategy network, autonomously decides the optimal action from the strategy network according to the occlusion state of the target object, forms an effective action sequence to complete the target grabbing task, and improves the grabbing success rate of the target object in the dense scene.

Description

Technical Field

[0001] This invention relates to the field of robotics application technology, specifically a method for a robotic arm to grasp target objects in dense environments. Background Technology

[0002] With the industrialization of artificial intelligence, robotic arms are playing an increasingly important role, serving as a fundamental component for completing complex tasks. Essentially, robotic arms provide the core function of "grasping." Traditional teaching methods are simple and easy to operate, but robotic arms are limited to highly repetitive environments and lack adaptive grasping capabilities. Deep learning object detection methods detect the position of a target object in two-dimensional space, and then use point-based, template-based, and voting-based methods to predict the actual three-dimensional pose of the target object, allowing the robotic arm to grasp objects at arbitrary locations. However, these methods require prior knowledge of the object model and can only grasp objects in simple, structured environments. Deep reinforcement learning combines the perceptual capabilities of deep learning with the decision-making capabilities of reinforcement learning to generate a decision network suitable for object grasping tasks. It directly calculates actions that approximate the optimal decision based on the input image, improving the success rate of object grasping to some extent in different scenarios. However, in environments with densely packed objects, the target object is easily occluded by other objects, and the robotic arm still cannot find a suitable grasping pose, leading to final grasping failure. Deep reinforcement learning-based push-grabbing systems separate objects in dense scenes by "pushing" and then "grabbing" them, solving the object grasping problem in dense scenes through the synergistic effect between pushing and grasping. However, these methods grasp arbitrary objects in dense scenes one by one, rather than a specified target object, and the reward function fails to effectively evaluate the effectiveness of pushing and grasping actions, resulting in sometimes unreasonable "pushing" and "grabbing" actions.

[0003] Therefore, how to design a reasonable push and grasping collaborative strategy network so that the robotic arm can successfully grasp the specified target object in dense scenes is a problem worth considering. Summary of the Invention

[0004] This invention proposes a method for a robotic arm to grasp target objects in dense environments. Based on deep reinforcement learning, a collaborative strategy network for pushing and grasping the target object is constructed. Reward functions for pushing and grasping actions are designed to determine whether the pushing action effectively separates the target object from other objects and whether the grasping action successfully grasps the target object. The collaborative strategy network is trained by continuously trying actions in the working environment to obtain reward values ​​using the Q-learning algorithm. This method reasonably evaluates the effectiveness of pushing and grasping actions, and achieves accurate grasping of target objects in dense environments through the collaborative cooperation of "pushing" and "grasping" actions.

[0005] This invention proposes a method for a robotic arm to grasp target objects in dense environments, comprising the following main steps:

[0006] Step S1: Build a dense and complex environment for the robotic arm to work in the V-rep simulation environment;

[0007] Step S2: Construct an end-to-end deep reinforcement learning-based push and grasping cooperative policy network in the simulation environment, using an encoder-decoder structure;

[0008] Step S3: In the simulation environment built in step S1, the Q-learning algorithm is used to continuously try actions in the working environment to obtain reward values ​​in order to train the push and grasp cooperative strategy network. If the target object is severely occluded, the grasping network is blocked. Training only drives the network If the target object is not severely occluded, the push and grasping cooperative strategy network is trained in parallel.

[0009] Step S4: Build a realistic, densely stacked, complex scene, and transplant the push and grasping collaborative network trained on the simulation end to the physical platform. The robotic arm makes action decisions and grasps the target object in the real environment.

[0010] 2. Optionally, the encoder-decoder structure described in step S2 specifically includes the following steps:

[0011] Step S2-1: Construct an encoder using a dense convolutional neural network DenseNet121 to extract latent feature maps for calculating robotic arm movements;

[0012] Step S2-2: Construct a decoder using two parallel fully convolutional neural networks, namely the driver network and the driver network. and web scraping

[0013] 3. Optionally, the training of the push-and-grab collaborative strategy network described in step S3 specifically includes the following steps:

[0014] Step S3-1: Acquire RGB images within the workspace using a depth camera. c and depth image I d The target to be captured is segmented from the RGB image to form the target object mask image I. m The three types of image information are processed and fused to generate a feature map H. cdm ;

[0015] Step S3-2: Generate feature map H from step S3-1 above. cdm Rotate clockwise sequentially N rotated feature maps are obtained and sequentially fed into the encoder constructed in step S2-1 above to extract N latent feature maps at time t.

[0016] Step S3-3: If the target object is severely occluded, then block the crawling network. Training only drives the network The N latent feature maps extracted in step S3-2 Enter sequentially into the drive network Output the Q-value matrix of the driving network, and select the action corresponding to the largest Q-value as the optimal decision action A. t If the target object is not severely occluded, the push and grasping collaborative strategy network is trained in parallel, and the N latent feature maps extracted in step S3-2 are used to... Enter sequentially into the drive network and web scraping Output the Q-value matrices of the driving network and the grasping network, and select the action corresponding to the largest Q-value as the optimal decision action A. t ;

[0017] Step S3-4: The robotic arm executes the optimal motion A t If the work scenario changes, repeat steps S3-1 and S3-2 to obtain N latent feature maps at time t+1.

[0018] Steps S3-5: Based on the state S after the action t+1 Using the piecewise reward function R t The robotic arm performs the action A t The reward value after evaluation of action A t Affirmation or negation of what leads to the completion of the final task;

[0019] Step S3-6: Store the interactive data quadruple (S t A t ,S t+1 R t Add to the experience pool;

[0020] Step S3-7: Calculate the objective function based on the reward value obtained in step S3-5. Where γ∈[0,1] is the discount factor;

[0021] Step S3-8: Train the network by minimizing the Huber loss function:

[0022]

[0023] In the formula, The estimated value of the network output at time t. To backtrack the true value at time t based on the Q value at time t+1, the difference between the two is used as the loss function;

[0024] Step S3-9: Iteratively update the push and grab co-op strategy network with the Huber loss function until the success rate curve reaches a stable oscillation state, then terminate the training and save the latest push and grab co-op strategy network parameters.

[0025] 4. Optionally, the generation of feature map H in step S3-1 cdm Specifically, it includes the following steps:

[0026] Step S3-1-1: Capture a predefined workspace using a camera placed at a fixed point outside the robotic arm, and obtain an RGB color image I within that space. c and Depth image I d ;

[0027] Step S3-1-2: Apply the pre-trained segmentation network PoseCNN to the RGB color image I c The target to be captured is segmented, the target object pixels are set to 1, and all other pixels are set to 0, generating a binary mask image I of the target object. m ;

[0028] Step S3-1-3: Generate the binarized mask image I of the target object based on step S3-1-2. m Determine the occlusion status of the target object and count the number of pixels within the target object's region. If the count is below a threshold c, it indicates that the target object is currently severely occluded; otherwise, the target object is not occluded.

[0029] Step S3-1-4: Given the camera's external parameters, convert the RGB color image I... c Depth image I d and mask image I m The orthogonal projection along the direction of gravity is converted into a color height map H with a top-down view. c Depth and Height Map H d And target object mask height map H m ;

[0030] Step S3-1-5: Transfer the color height map H c Depth and Height Map H d And target object mask height map H m The height maps are convolved separately to extract feature maps of the same resolution, and then concatenated from top to bottom according to channel layers to form feature map H. cdm .

[0031] 5. Optionally, the optimal decision action A described in step S3-3... tSpecifically, it includes the following steps:

[0032] Step S3-3-1: Extract the N latent feature maps from step S3-2. Enter sequentially into the drive network Calculate the Q-value matrix of the N "push" actions at time t. The i-th latent feature map at time t The Q value of the "push" action at pixel coordinates (x, y) is If the target object is not severely occluded, N potential feature maps are also sequentially input into the push network. Calculate the Q-value matrix for N "grabbing" actions. The i-th latent feature map The value of the "grabbing" action at pixel coordinates (x, y) in the latent feature map is

[0033] Step S3-3-2: If the target object is severely occluded, combine the "push" Q-value matrix at time t in step S3-3-1 for all rotation angles. Find the maximum Q value at time t If the target object is not severely obscured, the Q-value matrix of the "push" and "grab" actions at all rotation angles at time t in step S3-3-1 is used. Find the maximum Q value at time t in,

[0034] Step S3-3-3: Using the maximum Q value Q max The corresponding action is the optimal action A of the robotic arm at time t. t A = (m, d, θ), where m represents the type of action of the robotic arm, m∈{p, g}, p represents the "pushing" action, g represents the "grasping" action, d = (x, y, z) represents the three-dimensional coordinates of the robotic arm's action point, and θ represents the angle of rotation of the robotic arm's end effector around the z-axis; if equal The robotic arm then performs a "pushing" motion, and the three-dimensional coordinates d of the motion point are (x... * y * , z * ), z * For pixels (x) * y * The depth value at point () and the rotation angle θ of the robotic arm's end effector around the z-axis are... Conversely, if equal The robotic arm then performs a "grabbing" action; the three-dimensional coordinates d of the action point are (x... * y * , z * ), z * For pixels (x) * y * The depth value at point () and the rotation angle θ of the robotic arm's end effector around the z-axis are...

[0035] 6. Optionally, the segmented reward function R described in steps S3-5 t The calculation of the reward value includes the following steps:

[0036] Step S3-5-1: Use interpolation to make the target object mask image I m Magnified to twice the original size m ′, image moments for finding I m with I m The centroid coordinates of the pixels in the target object region, I m 'Target object region pixels mapped to I' m Within the target object region, ensure that the centroid coordinates of the two objects coincide, and generate the target reward region R. area ;

[0037] Step S3-5-2: Calculate R area Area percentage of non-target objects R no_object For the target reward region R area The size of the region containing no objects, R target_are a represents the target reward region R. area The size of the area of ​​the target object within the region;

[0038] Step S3-5-3: Preset reward, when the action corresponding to the maximum Q value is "push", and the two-dimensional coordinates of the action point are located in the target reward area R. area If, after being pushed, the area P of the non-target object increases... occupy Before the push, it means that at time t, the "push" action effectively separated the target object from other objects, assigning a reward value R. t =0.5; conversely, if the area ratio P of the non-target object after pushing is less than 0.5, then... occupy If the value is greater than the value before the push, it means that the "push" action at time t failed to effectively separate the target object from other objects, and a reward value R is assigned. t =0.25;

[0039] Step S3-5-4: Similarly, when the action corresponding to the maximum Q value is "grab", and the two-dimensional coordinates of the action point are located in the target reward region R. areaIf the robotic arm successfully grasps the target object, it assigns a reward value R. t =1, otherwise, if the robotic arm fails to grasp the target object, a reward value R is assigned. t =0.5;

[0040] Step S3-5-5: Other situations are considered as push failures and grab failures, and the reward value is set to 0.

[0041] 7. Optionally, the densely stacked complex scene described in step S4 specifically includes the following steps:

[0042] Step S4-1: Perform hand-eye calibration using a calibration board and depth camera to obtain the camera's intrinsic and extrinsic parameters, and obtain the transformation matrix M between the two-dimensional pixel coordinates (x, y) and the three-dimensional spatial coordinates (x, y, z).

[0043] Step S4-2: Output the two-dimensional pixel coordinates (x, y, x) that maximize the Q value through the trained push and grasp cooperative strategy network. * y * The optimal action A is obtained by combining the transformation matrix M. t That is, the type of action of the robotic arm and the three-dimensional coordinates (x, y, z) of the action point in actual space. * y * , z * ), and the rotation angle θ of the robotic arm end about the z-axis;

[0044] Step S4-3: Control the end effector of the robotic arm to rotate to the angle θ of the feature map where the maximum Q value is located, and use the IK solver inside the robotic arm to calculate the motion to the three-dimensional coordinates (x, y, y). * y * , z * The required rotation magnitude for each degree of freedom drives the robotic arm's joint trajectory movement to execute the optimal action A. t ;

[0045] Step S4-4: Repeat steps S4-1 to S4-3 multiple times to achieve target object grabbing in dense scenes. Attached Figure Description

[0046] Figure 1 This is a flowchart of the method of the present invention;

[0047] Figure 2 This is a flowchart illustrating the actual capture process of the method of the present invention;

[0048] Figure 3 This is a scene diagram illustrating the method of grasping target objects in a dense environment according to the present invention.

[0049] Figure 4 This is a diagram of the collaborative network architecture for grasping the target object in the method of the present invention.

[0050] Figure 5 The images provided are color images, depth images, and target mask images in the simulation environment of the method of this invention.

[0051] Figure 6 These are color height maps, depth height maps, and target mask height maps in the simulation environment of the method of this invention.

[0052] Figure 7 This is a thermal height diagram of the pushing action of the method of the present invention;

[0053] Figure 8 This is a thermal height diagram of the grasping action in the method of the present invention;

[0054] Figure 9 This is the optimal motion graph for dense scenarios using the method of this invention;

[0055] Figure 10 This is an example diagram illustrating the reward mechanism of the present invention.

[0056] Figure 11 This is a graph showing the success rate of capturing target objects in dense scenes using the method of the present invention. Detailed Implementation

[0057] This invention proposes a method for a robotic arm to grasp target objects in dense environments. The technical methods described below, in conjunction with the accompanying drawings and embodiments, will be presented in detail and completely. Specific details are as follows:

[0058] Figure 1 This is a flowchart of the method of the present invention. Figure 2 This is a flowchart of the actual capture process of the method of the present invention. Figure 1 and Figure 2 This includes all the steps of this method, and the specific steps are as follows:

[0059] Step S1: Build a dense and complex environment for the robotic arm to work in the V-rep simulation environment.

[0060] Based on the trial-and-error principle of deep reinforcement learning, a large amount of empirical data needs to be collected through interaction with the environment to feed the policy network for learning. However, collecting data in the real environment is extremely costly. To solve this problem, the simulation software V-rep is used to build a scenario that is highly similar to reality for simulation experiments, which efficiently collects data and minimizes the noise interference caused by the migration from simulation to the real scene on the neural network output.

[0061] Import the robotic arm's .urdf joint configuration file and the simulated object's .obj instance file, along with other .obj instance files such as the D435i camera, ground, and table, into the V-rep simulation software. Define a 0.448×0.448×0.4 3D matrix position constraint space within the robotic arm's predefined workspace; randomly drop small wooden blocks within this space to simulate a dense object scene, allowing the robotic arm to collect data and train the policy network. The experimental platform environment is as follows... Figure 3 As shown.

[0062] Step S2: Construct an end-to-end deep reinforcement learning-based push and grasp cooperative policy network in the simulation environment, using an encoder-decoder structure. The architecture of the push and grasp cooperative policy network is as follows: Figure 4 As shown, the specific steps include the following:

[0063] Step S2-1: Construct an encoder using a dense convolutional neural network DenseNet121 to extract latent feature maps for calculating robotic arm movements.

[0064] Step S2-2: Construct a decoder using two parallel fully convolutional neural networks, namely the driver network and the driver network. and web scraping

[0065] Promote the network and web scraping The network has the same structure, including Conv2d convolutional layers, ReLU activation function layers, and BatchNorm2d normalization layers. The output Q-value matrix is ​​1×20×20, and bilinear interpolation is used to upsample and map it to an action Q-value matrix map with the same size as the original height map, which is 1×224×224.

[0066] Step S3: In the simulation environment built in step S1, the Q-learning algorithm is used to continuously try actions in the working environment to obtain reward values ​​in order to train the push and grasp cooperative strategy network. If the target object is severely occluded, the grasping network is blocked. Training only drives the network If the target object is not severely occluded, the push and grasping cooperative policy network is trained in parallel; specifically, the following steps are included:

[0067] Step S3-1: Acquire RGB images within the workspace using a depth camera. c and depth image I d The target to be captured is segmented from the RGB image to form the target object mask image I. m The three types of image information are processed and fused to generate a feature map H. cdm Specifically, it includes the following steps:

[0068] Step S3-1-1: Capture a predefined workspace using a camera placed at a fixed point outside the robotic arm, and obtain an RGB color image I within that space. c and Depth image I d .

[0069] Step S3-1-2: Apply the pre-trained segmentation network PoseCNN to the RGB color image I c The target to be captured is segmented, the target object pixels are set to 1, and all other pixels are set to 0, generating a binary mask image I of the target object. m Visual information such as Figure 5 As shown.

[0070] Step S3-1-3: Generate the binarized mask image I of the target object based on step S3-1-2. m Determine the occlusion status of the target object by counting the number of pixels within the target object's region. If the count is below a threshold c, it indicates that the target object is currently severely occluded; otherwise, the target object is not severely occluded.

[0071] Step S3-1-4: Given the camera's external parameters, convert the RGB color image I... c Depth image I d and mask image I m The orthogonal projection along the direction of gravity is converted into a color height map H with a top-down view. c Depth and Height Map H d And target object mask height map H m Visual information such as Figure 6 As shown.

[0072] Step S3-1-5: Transfer the color height map H c Depth and Height Map H d And target object mask height map H m The height maps are convolved separately to extract feature maps of the same resolution, and then concatenated from top to bottom according to channel layers to form feature map H. cdm .

[0073] Step S3-2: Generate feature map H from the above steps S3-1-5. cdm Rotate clockwise sequentially N rotated feature maps are obtained and sequentially fed into the encoder constructed in step S2-1 above to extract N latent feature maps at time t.

[0074] Step S3-3: If the target object is severely occluded, then block the crawling network. Training only drives the network The N latent feature maps extracted in step S3-2 Enter sequentially into the drive network Output the Q-value matrix of the driving network, and select the action corresponding to the largest Q-value as the optimal decision action A. t If the target object is not severely occluded, the push and grasping collaborative strategy network is trained in parallel, and the N latent feature maps extracted in step S3-2 are used to... Enter sequentially into the drive network and web scraping Output the Q-value matrices of the driving network and the grasping network, and select the action corresponding to the largest Q-value as the optimal decision action A. t Visual information such as Figure 7 and Figure 8 As shown, the specific steps include the following:

[0075] Step S3-3-1: Extract the N latent feature maps from step S3-2. Enter sequentially into the drive network Calculate the Q-value matrix of the N "push" actions at time t. The i-th latent feature map at time t The Q value of the "push" action at pixel coordinates (x, y) is If the target object is not severely occluded, N potential feature maps are also sequentially input into the push network. Calculate the Q-value matrix for N "grabbing" actions. The i-th latent feature map The value of the "grabbing" action at pixel coordinates (x, y) in the latent feature map is

[0076] Step S3-3-2: If the target object is severely occluded, combine the "push" Q-value matrix at time t in step S3-3-1 for all rotation angles. Find the maximum Q value at time t If the target object is not severely obscured, the Q-value matrix of the "push" and "grab" actions at all rotation angles at time t in step S3-3-1 is used. Find the maximum Q value at time t in,

[0077] Step S3-3-3: Using the maximum Q value Q max The corresponding action is the optimal action A of the robotic arm at time t. tA =<mn,d,θ> Where m represents the type of action of the robotic arm, m∈{p, g}, p represents the "pushing" action, g represents the "grasping" action, d=(x,y,z) represents the three-dimensional coordinates of the robotic arm's action point, and θ represents the angle of rotation of the robotic arm's end effector around the z-axis. equal The robotic arm then performs a "pushing" motion, and the three-dimensional coordinates d of the motion point are (x... * y * , z * ), z * For pixels (x) * y * The depth value at point () and the rotation angle θ of the robotic arm's end effector around the z-axis are... Conversely, if equal The robotic arm then performs a "grabbing" action; the three-dimensional coordinates d of the action point are (x... * y * , z * ), z * For pixels (x) * y * The depth value at point () and the rotation angle θ of the robotic arm's end effector around the z-axis are...

[0078] Optimal action A t The definitions are as follows: Push: The parallel grippers at the end of the robotic arm close together and push 5 cm from right to left along a straight line in the direction of rotation on the height map. Grip: The parallel grippers at the end of the robotic arm open and perform a gripping action 3 cm below the optimal motion point depth. Visualization information is as follows: Figure 9 As shown, the circle in the left figure represents the optimal action point for pushing, and the circle in the right figure represents the optimal action point for grabbing.

[0079] Step S3-4: The robotic arm executes the optimal motion A t If the work scenario changes, repeat steps S3-1 and S3-2 to obtain N latent feature maps at time t+1.

[0080] Steps S3-5: Based on the state S after the action t+1 Using the piecewise reward function R t The robotic arm performs the action A t The reward value after evaluation of action A t Visual information, such as affirmation or negation, that contributes to completing the final task. Figure 10 As shown, the specific steps include the following:

[0081] Step S3-5-1: Use interpolation to make the target object mask image I mMagnified to twice the original size m ′, image moments for finding I m with I m The centroid coordinates of the pixels in the target object region, I m 'Target object region pixels mapped to I' m Within the target object region, ensure that the centroid coordinates of the two objects coincide, and generate the target reward region R. area .

[0082] Step S3-5-2: Calculate R area Area percentage of non-target objects R no_object For the target reward region R area The size of the region containing no objects, R target_area For the target reward region R area The size of the area of ​​the target object within the region.

[0083] Step S3-5-3: Preset Reward. The action corresponding to the maximum Q value is "push," and the two-dimensional coordinates of the action point are located within the target reward region R. area If, after being pushed, the area P of the non-target object increases... occupy Before the push, it means that at time t, the "push" action effectively separated the target object from other objects, assigning a reward value R. t =0.5, conversely, if the area ratio P of the non-target object after pushing is less than 0.5, then... occupy If the value is greater than the value before the push, it means that the "push" action at time t failed to effectively separate the target object from other objects, and a reward value R is assigned. t =0.25.

[0084] Step S3-5-4: Similarly, when the action corresponding to the maximum Q value is "grab", and the two-dimensional coordinates of the action point are located in the target reward region R. area If the robotic arm successfully grasps the target object, it assigns a reward value R. t =1, otherwise, if the robotic arm fails to grasp the target object, a reward value R is assigned. t =0.5;

[0085] Step S3-5-5: Other situations are considered as push failures and grab failures, and the reward value is set to 0.

[0086] Step S3-6: Store the interactive data quadruple (S t A t S t+1 R t ) to the experience pool.

[0087] Step S3-7: Calculate the objective function based on the reward value obtained in step S3-5. Where γ∈[0,1] is the discount factor.

[0088] Step S3-8: Train the network by minimizing the Huber loss function:

[0089]

[0090] In the formula, The estimated value of the network output at time t. To backtrack the true value at time t based on the Q-value at time t+1, the difference between the two is used as the loss function. During the backpropagation of network gradients, the loss function value for other pixels is 0; only the loss function value of the pixel with the optimal action is calculated. When the absolute value of the loss function is less than 1, the MSE loss function is used, and the gradient gradually decreases as the loss value approaches its minimum, thus more accurately approximating the minimum value of the loss function. In other cases, the MAE loss function is used, where the gradient remains large, facilitating faster network training. The MSE and MAE are combined to form the Huber loss function. The gradient update method uses a stochastic optimization of adaptive momentum combined with a weight decay optimization method, summarizing past gradient history to accelerate convergence. The specific formula is as follows:

[0091]

[0092] In the above formula, g t For the gradient of network parameters, m t+1 This is a first-order moment estimate of the gradient, where β1 is the first-order momentum coefficient; v t+1 This is a second-order moment estimate of the gradient, where β2 is the second-order momentum coefficient. An unbiased estimator for the first-order moment estimator. λ is the unbiased estimate of the second-order moment estimate; α is the learning rate, which controls the magnitude and speed of convergence of the decision network, and the default value is a constant of 0.0001. λ is the weight decay coefficient, which is set to prevent the network from overfitting, and the default value is a constant of 0.00002.

[0093] Step S3-9: Iteratively update the push and grab cooperative policy network using the Huber loss function until convergence. Observe the smooth oscillation of the success rate curve, and terminate training after maintaining this state for a period of time. Save the latest push and grab cooperative policy network parameters and port them to a real-world environment.

[0094] After multiple tests, the trained push and grasp cooperative strategy network achieved an approximately 85% success rate in grasping target objects in dense scenes. The grasping success rate is as follows: Figure 11 As shown.

[0095] Step S4: Construct a realistic, densely stacked, complex scene. Port the push and grasping collaborative network trained on the simulation end to the physical platform. The robotic arm makes motion decisions and grasps the target object in a real environment. This includes the following steps:

[0096] Step S4-1: Use the calibration board and depth camera to perform hand-eye calibration, obtain the camera's intrinsic and extrinsic parameters, and obtain the transformation matrix M between the two-dimensional pixel coordinates (x, y) and the three-dimensional spatial coordinates (x, y, z).

[0097] Step S4-2: Output the two-dimensional pixel coordinates (x, y, x) that maximize the Q value through the trained push and grasp cooperative strategy network. * y * The optimal action A is obtained by combining the transformation matrix M. t That is, the type of action of the robotic arm and the three-dimensional coordinates (x, y, z) of the action point in actual space. * y * , z * ), and the rotation angle θ of the end of the robotic arm around the z-axis.

[0098] Step S4-3: Control the end effector of the robotic arm to rotate to the angle θ of the feature map where the maximum Q value is located, and use the IK solver inside the robotic arm to calculate the motion to the three-dimensional coordinates (x, y, y). * y * , z * The required rotation magnitude for each degree of freedom drives the robotic arm's joint trajectory movement to execute the optimal action A. t .

[0099] Step S4-4: Repeat steps S4-1 to S4-3 multiple times to achieve target object grabbing in dense scenes.

[0100] All the above steps and experimental results were performed on a simulation platform, which still differs significantly from the actual real-world environment. Therefore, to ensure the effectiveness of the method of this invention, transferring the trained push and grasp cooperative strategy network to a real-world environment is of practical significance. A real-world experimental platform was built, including a robotic arm, a depth camera, and objects to be picked up. After multiple tests, the real-world platform and the simulation platform have similar success rates in grasping target objects in dense scenes, demonstrating the feasibility of the method of this invention.

[0101] The foregoing description is a detailed explanation of the implementation method of the present invention, and it should not be construed that the embodiments of the present invention are limited to the above description. For the technical field to which this invention pertains, any changes made by those skilled in the art within the original scope of this invention, such as substitutions or deletions, are also within the protection scope of this invention.

Claims

1. A method for a robotic arm to grasp a target object in a dense environment, characterized in that, Includes the following steps: Step S1: Build a dense and complex environment for the robotic arm to work in the V-rep simulation environment; Step S2: Construct an end-to-end deep reinforcement learning-based push and grasping cooperative policy network in the simulation environment, using an encoder-decoder structure; Step S3: In the simulation environment built in step S1, the Q-learning algorithm is used to continuously try actions in the working environment to obtain reward values ​​in order to train the push and grasp cooperative strategy network. If the target object is severely occluded, the grasping network is blocked. Training only drives the network If the target object is not severely occluded, the push and grasping cooperative strategy network is trained in parallel. Step S4: Build a realistic, densely stacked, complex scene, and transplant the push and grasping collaborative network trained on the simulation end to the physical platform. The robotic arm makes action decisions and grasps the target object in the real environment.

2. The method for a robotic arm to grasp a target object in a dense scene according to claim 1, characterized in that, The encoder-decoder structure described in step S2 specifically includes the following steps: Step S2-1: Construct an encoder using a dense convolutional neural network DenseNet121 to extract latent feature maps for calculating robotic arm movements; Step S2-2: Construct a decoder using two parallel fully convolutional neural networks, namely the driver network and the driver network. and web scraping 3. The method for a robotic arm to grasp a target object in a dense scene according to claim 2, characterized in that, Step S3, which describes the training of the push-and-grab collaborative strategy network, specifically includes the following steps: Step S3-1: Acquire RGB images within the workspace using a depth camera. c and depth image I d The target to be captured is segmented from the RGB image to form the target object mask image I. m The three types of image information are processed and fused to generate a feature map H. cdm ; Step S3-2: Generate feature map II from the above step S3-1. cdm Rotate clockwise sequentially N rotated feature maps are obtained and sequentially fed into the encoder constructed in step S2-1 above to extract N latent feature maps at time t. Step S3-3: If the target object is severely occluded, then block the crawling network. Training only drives the network The N latent feature maps extracted in step S3-2 Enter sequentially into the drive network Output the Q-value matrix of the driving network, and select the action corresponding to the largest Q-value as the optimal decision action A. t If the target object is not severely occluded, the push and grasping collaborative strategy network is trained in parallel, and the N latent feature maps extracted in step S3-2 are used to... Enter sequentially into the drive network and web scraping Output the Q-value matrices of the driving network and the grasping network, and select the action corresponding to the largest Q-value as the optimal decision action Λ. t ; Step S3-4: The robotic arm executes the optimal motion A t If the work scenario changes, repeat steps S3-1 and S3-2 to obtain N latent feature maps at time t+1. Steps S3-5: Based on the state S after the action t+1 Using the piecewise reward function R t The robotic arm makes a movement Λ t The reward value after evaluation of action A t Affirmation or negation of what leads to the completion of the final task; Step S3-6: Store the interactive data quadruple (S t A t S t+1 R t Add to the experience pool; Step S3-7: Calculate the objective function based on the reward value obtained in step S3-5. Where γ∈[0,1] is the discount factor; Step S3-8: Train the network by minimizing the Huber loss function: In the formula, The estimated value of the network output at time t. To backtrack the true value at time t based on the Q value at time t+1, the difference between the two is used as the loss function; Step S3-9: Iteratively update the push and grab co-op strategy network with the Huber loss function until the success rate curve reaches a stable oscillation state, then terminate the training and save the latest push and grab co-op strategy network parameters.

4. The method for a robotic arm to grasp a target object in a dense scene according to claim 3, characterized in that, Step S3-1 involves generating the feature map H. cdm Specifically, it includes the following steps: Step S3-1-1: Capture a predefined workspace using a camera placed at a fixed point outside the robotic arm, and obtain an RGB color image I within that space. c and Depth image I d ; Step S3-1-2: Apply the pre-trained segmentation network PoseCNN to the RGB color image I c The target to be captured is segmented, the target object pixels are set to 1, and all other pixels are set to 0, generating a binary mask image I of the target object. m ; Step S3-1-3: Generate the binarized mask image I of the target object based on step S3-1-2. m Determine the occlusion status of the target object and count the number of pixels in the target object's region. If the count is below the threshold c, it means that the target object is currently severely occluded. Conversely, the target object was not obstructed; Step S3-1-4: Given the camera's external parameters, convert the RGB color image I... c Depth image I d and mask image I m The orthogonal projection along the direction of gravity is converted into a color height map H from a top-down perspective. c Depth and Height Map H d And target object mask height map H m ; Step S3-1-5: Transfer the color height map H c Depth and Height Map H d And target object mask height map H m The height maps are convolved separately to extract feature maps of the same resolution, and then concatenated from top to bottom according to channel layers to form feature map H. cdm .

5. The method for a robotic arm to grasp a target object in a dense scene according to claim 3, characterized in that, Step S3-3 describes the selection of the optimal decision action Λ t Specifically, it includes the following steps: Step S3-3-1: Extract the N latent feature maps from step S3-2. Enter sequentially into the drive network Calculate the Q-value matrix of the N "push" actions at time t. The i-th latent feature map at time t The Q value of the "push" action at pixel coordinates (x, y) is If the target object is not severely occluded, N potential feature maps are also sequentially input into the push network. Calculate the Q-value matrix for N "grabbing" actions. The i-th latent feature map The value of the "grabbing" action at pixel coordinates (x, y) in the latent feature map is Step S3-3-2: If the target object is severely occluded, combine the "push" Q-value matrix at time t in step S3-3-1 for all rotation angles. Find the maximum Q value at time t If the target object is not severely occluded, the Q-value matrix of the "push" and "grab" actions at all rotation angles at time t in step S3-3-1 is used. Find the maximum Q value at time t in, Step S3-3-3: Use the maximum Q value Q max The corresponding action is the optimal action At of the robotic arm at time t, A = <m, d, θ>, where m represents the action type of the robotic arm, m ∈ {p, g}, p represents the "push" action, g represents the "grab" action, d = (x, y, z) represents the three-dimensional coordinates of the action point of the robotic arm, and θ represents the rotation angle of the end of the robotic arm around the z-axis; if equals then the robotic arm executes the "push" action, and the three-dimensional coordinates d of the action point are (x * , y * , z * ), z * is the depth value at the pixel (x * , y * ), and the rotation angle θ of the end of the robotic arm around the z-axis is Conversely, if equals then the robotic arm executes the "grab" action; the three-dimensional coordinates d of the action point are (x * , y * , z * ), z * is the depth value at the pixel (x * , y * ), and the rotation angle θ of the end of the robotic arm around the z-axis is 6. The method for a robotic arm to grasp a target object in a dense scene according to claim 3, characterized in that, The segmented reward function R described in step S3-5 t The calculation of the reward value includes the following steps: Step S3-5-1: Use interpolation to make the target object mask image I m Magnified to twice the original size m ′, image moments for finding I m with I m The centroid coordinates of the pixels in the target object region, I m 'Target object region pixels mapped to I' m Within the target object region, ensure that the centroid coordinates of the two objects coincide, and generate the target reward region R. area ; Step S3-5-2: Calculate R area Area percentage of non-target objects R no_object For the target reward region R area The size of the region containing no objects, R target_area For the target reward region R area The size of the area of ​​the target object within the region; Step S3-5-3: Preset reward, when the action corresponding to the maximum Q value is "push", and the two-dimensional coordinates of the action point are located in the target reward area R. area If, after being pushed, the area P of the non-target object increases... occupy Before the push, it means that at time t, the "push" action effectively separated the target object from other objects, assigning a reward value R. t =0.5; conversely, if the area ratio P of the non-target object after pushing is less than 0.5, then... occupy If the value is greater than the value before the push, it means that the "push" action at time t failed to effectively separate the target object from other objects, and a reward value R is assigned. t =0.25; Step S3-5-4: Similarly, when the action corresponding to the maximum Q value is "grab", and the two-dimensional coordinates of the action point are located in the target reward region R. area If the robotic arm successfully grasps the target object, it assigns a reward value R. t =1, otherwise, if the robotic arm fails to grasp the target object, a reward value R is assigned. t =0.5; Step S3-5-5: Other situations are considered as push failures and grab failures, and the reward value is set to 0.

7. The method for a robotic arm to grasp a target object in a dense environment according to claim 1, characterized in that, The densely stacked complex scene described in step S4 specifically includes the following steps: Step S4-1: Use the calibration board and depth camera to perform hand-eye calibration, obtain the camera's intrinsic and extrinsic parameters, and obtain the transformation matrix M between the two-dimensional pixel coordinates (x, y) and the three-dimensional spatial coordinates (x, y, z). Step S4-2: Output the two-dimensional pixel coordinates (x, y, x) that maximize the Q value through the trained push and grasp cooperative strategy network. * y * The optimal action A is obtained by combining the transformation matrix M. t That is, the type of action of the robotic arm and the three-dimensional coordinates (x, y, z) of the action point in actual space. * y * , z * ), and the rotation angle θ of the robotic arm end about the z-axis; Step S4-3: Control the end effector of the robotic arm to rotate to the angle θ of the feature map where the maximum Q value is located, and use the IK solver inside the robotic arm to calculate the motion to the three-dimensional coordinates (x, y, y). * y * , z * The required rotation magnitude for each degree of freedom drives the robotic arm's joint trajectories to execute the optimal motion Λ t ; Step S4-4: Repeat steps S4-1 to S4-3 multiple times to achieve target object grabbing in dense scenes.

Images