Mechanical arm cooperative autonomous grabbing method based on deep reinforcement learning

Abstract

The invention provides a mechanical arm cooperative grabbing method based on deep reinforcement learning, relates to the technical field of robot application, and aims to solve the problems of unstable network, low sample efficiency and unreasonable behavior in a multi-object grabbing method in a complex scene. The method comprises the following steps: constructing a cooperative autonomous grabbing decision network model, and obtaining state action value distribution of pushing and grabbing actions through pixel-by-pixel prediction; an object mask function is introduced, invalid areas are filtered, and the influence of negative samples on the convergence speed of the model is reduced; different behavior constraint strategies are designed to suppress unreasonable behaviors in network prediction, and motion selection of the mechanical arm is optimized; a multivariate reward function is designed, a reward mechanism is dynamically adjusted, and model learning is guided to better conform to an action strategy of a task target. Through simulation and real experimental environment verification, the task success rate and the completion efficiency of the mechanical arm in an unstructured scene can be improved, and good robustness and applicability are achieved.

Description

Technical Field

[0001] The present invention relates to the field of robot application technology, and in particular to a robotic arm collaborative autonomous grasping method based on deep reinforcement learning. Background Art

[0002] With the rapid development of robotics and artificial intelligence, intelligent grasping tasks based on robotic arms have played an important role in industrial automation, logistics sorting, and service robotics. However, achieving efficient and stable grasping operations in complex scenes still faces many challenges, such as the close stacking of multiple objects, dynamic changes in scenes, and safety and accuracy issues during operations.

[0003] Traditional robotic grasping methods mostly rely on accurate environment modeling and predefined rules, and plan the operation path of the robotic arm through specific algorithms. However, such methods often show insufficient adaptability and high dependence on the environment when facing complex dynamic environments or unknown scenes, which greatly limits their scope of application. In recent years, deep learning and reinforcement learning technologies have made significant progress in the field of robot intelligent decision-making, providing new solutions for complex grasping tasks. In grasping tasks based on reinforcement learning, how to efficiently extract scene space features, design a reasonable action value evaluation mechanism, and optimize training efficiency are still key issues affecting the actual application effect. In addition, although reinforcement learning technology provides new ideas for autonomous decision-making of robotic arms, conventional Q learning or deep Q networks are prone to instability, slow convergence, and unreasonable action selection when dealing with continuous state space and large-scale action space. Therefore, how to design more efficient learning algorithms and reasonable exploration strategies is a key issue to improve the intelligence level of grasping tasks.

[0004] Therefore, it is necessary to invent a method with fast training speed, good stability, and the ability to quickly complete grasping tasks in various structured and unstructured scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide a robot arm collaborative autonomous grasping method based on deep reinforcement learning to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] Based on the environment state height map, a feature extraction network is built to extract representative spatial features from the input data to obtain feature vectors, and a network prediction model for pushing / grasping actions is built. The aforementioned feature vectors are used as inputs to perform strategy evaluation in combination with state features and action values ​​to predict the distribution of state-action evaluation values ​​for candidate actions.

[0008] The object mask function M is introduced for pushing and grasping actions respectively. p_obj and M g_obj , combining it with the network prediction results as prior information, constraining the exploration space during the interaction between the robot and the environment, reducing the interference of negative samples in the experience replay pool on the model convergence speed, and improving the efficiency and stability of model training;

[0009] Design different behavior constraint strategies to suppress unreasonable behaviors in network prediction;

[0010] In view of the speed of the robotic arm to complete the task and the feedback of the environmental state, a reward function is designed for different action tuples to guide the update of the policy network model, which is trained in a simulation environment and migrated to the real environment.

[0011] Furthermore, in the scheme, a feature extraction network is constructed based on the environment state height map, representative spatial features are extracted from the input data to obtain feature vectors, and a pushing / grasping action network prediction model is constructed. The aforementioned feature vector is used as input, and a strategy evaluation is performed in combination with the state features and action values ​​to predict the state-action evaluation value distribution of the candidate action, including: first, the single-channel depth height map in the environment state height map is copied to obtain three-channel data, and then the color height map and the depth height map are respectively rotated and input into the shared feature fusion module, and the output is used as the input of the high-level feature extraction network to obtain an intermediate feature vector, and the intermediate feature vector is spliced ​​with color and depth features and input into the behavior value evaluation module to obtain a state-action evaluation value distribution map of different actions in different directions, wherein the behavior value evaluation module includes a pushing network module and a grasping network module, and the pushing network and the grasping network are both the main network and the target network architecture.

[0012] Furthermore, in the scheme, after the network is constructed, the action is selected according to the strategy, and the object mask function M is introduced for the pushing and grasping actions respectively. p_obj and M g_obj As a priori information, the robot arm is guided to conduct meaningful spatial exploration, including: for the pushing action, the object index is obtained according to the initialization setting of the environment state in the simulation environment, the outline of the vertical projection of the object is obtained according to its coordinates, and the inside of the object outline is filled with 1 and the outside of the outline is filled with 0 through threshold segmentation to obtain the mask function in the simulation environment; in the real physical experimental environment, the mask function is obtained according to the depth height map of the workspace. Specifically: first, according to the formula Normalize the depth height map, where D(x,y) is the depth map pixel value, D min and D max is the minimum and maximum depth value, D′(x, y) is the normalized depth map pixel value; the depth threshold T for separating the foreground object and the background of the image is set according to the depth distributionthre , set the pixel points whose depth value is greater than the threshold to 1, otherwise set them to 0, and get the promoted binary mask M p_obj .

[0013] Furthermore, in the scheme, after the network is constructed, the action is selected according to the strategy, and the object mask function M is introduced for the pushing and grasping actions respectively. p_obj and M g_obj As a priori information, the robot arm is guided to conduct meaningful spatial exploration, including: For the grasping action, the processing method in the simulation environment is the same as that of the pushing action. In the real experimental environment, the depth height map is normalized and filtered, and then the threshold segmentation is performed to obtain the initial grasping mask M g_ori To prevent the grasping operation from failing due to the grasping position being at the edge of the object, the initial grasping mask is denoised and edge optimized to obtain the final grasping mask M. g_obj ; The pixel-by-pixel state-action evaluation value distribution map output by the push and grab network is respectively compared with the push mask M p_obj and grab mask M g_obj Combined with the above, an action value evaluation map with prior information is obtained, which encourages the robot arm to explore the effective area in the workspace and reduces the impact of negative samples during training.

[0014] Furthermore, in the scheme, a behavior constraint strategy is designed to suppress unreasonable behaviors in network prediction, including: performing a connected domain analysis on the push mask, extracting the number of pixels, bounding boxes and geometric information of the connected domain of the objects in the workspace, and designing a constraint strategy on this basis; the constraint on the push behavior is based on the density of objects in different directions, specifically: first, the posture angle is divided into 16 equal-angle directions θ in the action space of the robot arm d , each direction corresponds to a unit vector Secondly, for each candidate position pixel point (x, y) in each direction, calculate the centroid distribution direction in its adjacent area. The centroid is calculated as follows:

[0015]

[0016] Where (c x ,c y ) is the centroid coordinate in the connected domain, R is the set of pixels in the connected domain, (x, y) represents the pixels in the set R, and N represents the number of pixels in the connected domain; then the distribution of object density in different directions is calculated, and the formula is as follows:

[0017]

[0018] Where (c xi ,c yi) is the centroid coordinate of each connected region, is the relative position vector between the centroid of a region and the candidate action pixel point (x, y), for Normalized direction vector, S dp The similarity between each centroid and the current direction is expressed in θ d The density of objects in a direction, is the direction θ d Unit vector of ; based on the directional density S dp Generate push action weights for each direction Represents the priority of the current pixel (x, y) in each pushing direction, and combines it with the pushing action value evaluation map with prior information as a pushing behavior constraint strategy.

[0019] Furthermore, in the scheme, a behavior constraint strategy is designed to suppress unreasonable behaviors in network prediction, including: performing a connected domain analysis on the grasping mask map, extracting the number of pixels, bounding boxes and geometric information of the connected domain of the object in the workspace, and designing a grasping action constraint strategy on this basis; the constraint on the grasping behavior is to select the closest grasping direction based on the main axis direction of the object, specifically: constructing a corresponding unit direction vector for each candidate grasping action in each direction θ d For the equal-angle posture angle directions divided in the action space, calculate the unit direction vector of the main axis direction of the object where the grasping position candidate point is located in each direction where θ obj is the main axis direction of the connected domain where the current grasping position is located, and the similarity between the main axis direction and the grasping direction is calculated: Where S dg Used to calculate the grasping angle constraint factor, S dg The closer it is to 1, the closer the current grasping direction is to the main axis direction, and the higher the grasping success rate. It is used as a grasping weight strategy and combined with the grasping action value evaluation graph with prior information as a grasping behavior constraint strategy.

[0020] Furthermore, in the scheme, a reward function is designed for different action tuples according to the rapidity of the robot arm completing the task and the feedback of the environmental state to guide the update of the policy network model, so that the robot arm can learn reasonable grasping positions and postures in different scenarios, including: the reward function is divided into two parts: the driving network and the grasping network:

[0021] For the grasping network, when the robot arm performs grasping, there are only two results, success or failure. Therefore, the grasping reward function R is designed considering the behavior results and constraint strategies. g:

[0022]

[0023] Among them C g is the reward value constant, λ g is a hyperparameter, Θ g is the grasping angle constraint factor;

[0024] For the pushing network, it is necessary to consider factors such as the impact of the pushing action on subsequent grasping, the degree of aggregation of objects in the environment, the boundary constraints of the workspace, and the efficiency of task completion. The reward function is designed as follows:

[0025] r1 is the reward that considers the impact of pushing on the predictive reasoning of grasping. The grasping network is used as a discriminator to evaluate the grasping behavior in the current state. The pushing network is used as a generator to continuously improve the graspability of objects in the workspace. The quality of the pushing action is evaluated by comparing the grasping scores of the two states before and after the push. The reward function is designed according to the score change. That is, after the push action is executed, if S t+1 The best state action evaluation value predicted by the grasping network in the state is greater than S to a certain extent. t If the grab state action evaluation value is in the state, the current push is considered effective and a positive reward is given, otherwise the reward is 0:

[0026]

[0027] Where C1 is the reward value constant, λ p is a hyperparameter, is the change in the predicted value of the capture network, δ1 is the threshold value for the change in the state action evaluation value, and They are the best state action evaluation values ​​of the grasping network before and after pushing, respectively;

[0028] r2 considers the degree of aggregation of objects in the environment before and after the execution of the push. Because increasing the grasping space around the object can increase the probability of successful grasping, the average relative distance is introduced to measure the degree of discreteness between objects in the environment in the current state. The reward function is designed according to the change in the degree of aggregation before and after the action is executed:

[0029]

[0030] Where C2 is the reward value constant, Δ l = l t -l t-1 It represents the difference between the average relative distances of all objects in the working area at time t and time t-1, l t and l t-1represents the average relative distance between all objects in the working area at time t and time t-1, respectively, δ2 is the aggregation degree change threshold, when Δ l When it is greater than the threshold δ2, the pushing behavior at the previous moment is considered effective and a positive reward is given;

[0031] r3 takes into account the boundary constraints of the work area. When the robot arm performs the pushing action, it needs to keep the object in the work area to ensure the task completion rate, so a corresponding penalty is imposed for leaving the work area:

[0032] r3=aC3

[0033] Where C3 is the penalty value constant and a is the penalty coefficient;

[0034] R4 considers the efficiency of the robot arm in completing the grasping task. The better the separation effect of the push action, the higher the success rate of grasping. At the same time, the fewer the number of push actions, the shorter the overall action sequence will be, and the higher the task efficiency will be.

[0035] r4=bC4

[0036] Where C4 is the reward value constant, b is the reward coefficient;

[0037] Therefore, for driving the network, the reward function integrates into:

[0038] R p =r1+r2+r3+r4

[0039] Used to guide the robot arm to improve its strategy to obtain higher cumulative rewards.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] This method uses prior information that combines the object mask map with the network prediction state value distribution map, which effectively reduces the negative samples generated by the robot arm's exploration in invalid areas and improves the model convergence speed; different behavioral constraint strategies are designed to suppress unreasonable actions predicted by the network, thereby improving the success rate of action execution and the task completion rate; by introducing diversified reward functions for different action tuples, the optimization of the policy network model is more accurately guided to ensure the efficiency and accuracy of the robot arm in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a robot arm collaborative autonomous grasping method based on deep reinforcement learning according to the present invention;

[0043] Figure 2 It is a parameter diagram of the network prediction model of the pushing / grasping action described in the present invention;

[0044] Figure 3 This is a schematic diagram of the driving behavior constraint strategy of the present invention;

[0045] Figure 4 This is a schematic diagram of the grabbing behavior constraint strategy of the present invention;

[0046] Figure 5 It is a schematic diagram of the structure of the collaborative autonomous crawling network decision process described in the present invention;

[0047] Figure 6 This is a comparison test effect diagram of the model under the simulation environment and the real environment described in the present invention, where Figure 6 a is a comparison experiment effect diagram under simulation environment. Figure 6 b is a comparison experiment effect diagram under real physical environment. DETAILED DESCRIPTION

[0048] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0049] The present invention provides a robot arm collaborative autonomous grasping method based on deep reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0050] (1) A simulation environment is built according to the real scene, and the robot arm model and camera model are loaded. Thirty objects of different shapes are randomly loaded in each training, and the positions of each object are randomly distributed.

[0051] (2) A feature extraction network is constructed based on the environment state height map to extract representative spatial features from the input data to obtain feature vectors, and a network prediction model for pushing / grasping actions is constructed. The aforementioned feature vectors are used as inputs, and the state features and action values ​​are combined to perform strategy evaluation and predict the distribution of state-action evaluation values ​​of candidate actions.

[0052] Specific implementation: First, copy the single-channel depth height map to obtain 3-channel data; construct an affine matrix of the counterclockwise rotation angle, rotate the color height state map and the depth height state map 16 times at the specified angle, and input each rotation image into the shared feature fusion module for preliminary feature extraction; use the above output as the input of the feature extraction network, and obtain the color intermediate feature vector and the depth intermediate feature vector through the feature extraction network. The number of channels of a single feature vector is 1024, and the number of channels after channel splicing is 2048. The spliced ​​intermediate feature vectors are respectively input into the behavior value evaluation module, including the pushing network module and the grasping network module, to obtain 32 single-channel state action evaluation value distribution maps of pushing and grasping actions in 16 directions. Each direction represents the rotation angle of the end effector of the robot arm. The pixel position in the distribution map corresponds to the 3D coordinate position in the workspace, representing the specific position of the robot arm performing the action; the first part of the feature extraction network backbone consists of multiple dense blocks and transition layers, and the pushing network and the grasping network are independent convolution branches, such as Figure 2 As shown in the figure, it includes three batch normalization layers, RELU activation function layers, three convolutional layers with different configurations, and a bilinear sampling difference upsampling layer. Since this method uses the DDQN architecture for training, the designed target network has the same structure as the main network, and the parameters of the target network are synchronized with the main network in a soft update manner. The update formula is as follows:

[0053] θ target ←τθ online +(1-τ)θ target

[0054] where θ target is the target network parameter, θ online is the main network parameter, and τ = 0.005 is the target network parameter update rate.

[0055] (3) Introduce the object mask function M for pushing and grasping actions respectively p_obj and M g_obj As prior information, it constrains the exploration space during the interactive learning process between the robot and the environment.

[0056] Specific implementation: For the pushing action, in the simulation environment, the object index is obtained according to the initialization setting of the environment state, the coordinates of its current position are read according to the object index, and the outline of the vertical projection of the object is obtained according to the coordinates and the object size. The inside of the object outline is filled with 1 and the outside of the outline is filled with 0 through threshold segmentation to obtain the pushing mask function in the simulation environment; in the real physical experimental environment, the mask function is obtained according to the depth height map of the workspace. Specifically: First, according to the formula Normalize the depth height map, where D(x,y) is the depth map pixel value, D minand D max is the minimum and maximum depth value, which needs to be read in real time according to the depth map. D′(x, y) is the normalized depth image. The binary depth threshold T for separating the foreground object and the background of the image is manually set according to the depth distribution. thre , set the parameter T thre = 0.4, set the pixel points whose depth value is greater than the threshold to 1, otherwise set them to 0, and get the initial push binary mask M p_ori , and then use the 5×5 structure element to perform a single dilation process to obtain the pushing action object mask image M p_obj .

[0057] For the grasping action, the processing method in the simulation environment is the same as that of the pushing action. In the real experimental environment, the depth height map is normalized and hole filtered, and the initial grasping mask map M is obtained after threshold segmentation. g_ori To prevent the grasping position from being at the edge of the object, which may lead to incomplete grasping operation, the initial grasping mask image is filtered, including using a 3×3 structure element for denoising, eliminating noise points in the mask image and retaining the object shape, and then using a 5×5 erosion kernel for edge optimization to obtain the grasping mask function M. g_obj .

[0058] The pixel-by-pixel state-action evaluation value distribution maps output by the push network and the grasp network are respectively compared with the push action mask M p_obj and grab mask M g_obj A Hadamard product operation is performed to extract the area where objects exist in the workspace, and an action value evaluation map with prior information is obtained, thereby guiding the robot arm to explore the effective area in the workspace and reducing the impact of negative samples generated by performing operations in the object-free area during training on the model convergence speed.

[0059] (4) Design behavior constraint strategies to suppress unreasonable behaviors in network prediction.

[0060] Specific implementation: Perform a connected domain analysis on the mask image obtained in (3) to extract the number of pixels, bounding box and geometric information of the connected domain of the object in the workspace. On this basis, different constraint strategies are designed for pushing and grasping respectively; specifically:

[0061] The constraints on pushing behavior are based on the density of objects in different directions, such as Figure 3 As shown: First, the posture angle is divided into 16 equal angle directions θ in the action space of the robot arm. d ∈{0°,22.5°,45°,…,337.5°}, each direction corresponds to a unit vector Secondly, for each candidate position pixel point (x, y) in each direction, calculate the centroid distribution direction in its adjacent area. The centroid is calculated as follows:

[0062]

[0063] Where (c x ,c y ) is the centroid coordinate in the connected domain, R is the set of pixels in the connected domain, (x, y) represents the pixels in the set R, and N represents the number of pixels in the connected domain; then the distribution of object density in different directions is calculated, and the formula is as follows:

[0064]

[0065] Where (c xi ,c yi ) is the centroid coordinate of each region, is the relative position vector between the centroid of a region and the candidate action pixel point (x, y), for Normalized direction vector, S dp The similarity between each centroid and the current direction is expressed in the direction θ d The density of objects on is the direction θ d Unit vector of ; based on the direction density S dp Generate push action weights for each direction It reflects the priority of the current pixel (x, y) in each pushing direction. A higher weight means that the pushing action in this direction is more likely to successfully complete the task. The pushing action value evaluation graph with prior information obtained in (3) is combined with the direction weight:

[0066]

[0067] in Represents the driving action value evaluation graph with prior information in the d-th direction. The above operation is completed by element-by-element multiplication through the broadcast mechanism in pytorch.

[0068] The constraint design of the grasping behavior is based on the main axis direction of the object, and the grasping direction closest to it is selected, such as Figure 4 As shown: Construct the corresponding unit direction vector for each candidate grasping action in each direction θ d For the equal-angle posture angle directions divided in the action space, calculate the unit direction vector of the main axis direction of the object where the grasping position candidate point is located in each direction where θ objis the main axis direction of the connected domain where the current position is located. The similarity between the main axis direction and the 16 grasping directions is calculated by dot product: Where S dg Used to calculate the grasping angle constraint factor, S dg The closer it is to 1, the closer the current grasping direction is to the main axis direction, and the higher the grasping success rate is. It is used as a grasping weight strategy and combined with the grasping action value evaluation graph with prior information as a grasping behavior constraint strategy:

[0069]

[0070] in Represents the grasping action value evaluation graph with prior information in the d-th direction.

[0071] (5) Aiming at the speed and safety of the robotic arm in completing the grasping task, a reward function is designed for different action tuples to guide the update of the policy network model.

[0072] Specific implementation: For the grasping network, when the robot arm performs a grasping action, there are only two results: grasping the object is successful and a positive reward is given, otherwise it is a failure and the reward is 0. Therefore, the grasping reward function R is designed considering the behavior results and constraint strategies. g :

[0073]

[0074] Among them C g is the reward value constant, λ g is a hyperparameter used to measure the influence of the behavior constraint strategy of the current grasping direction on the grasping operation, Θ g =1-S dg is the grasping angle constraint factor, S dg is the similarity between the current grasping direction and the main axis direction of the object mask, and the parameter is set to C g =1.6,λ g =0.5;

[0075] For the pushing network, it is necessary to consider factors such as the impact of the pushing action on subsequent grasping, the degree of aggregation of objects in the environment, the boundary constraints of the workspace, and the efficiency of task completion. The reward function is designed as follows:

[0076] r1 is the reward that considers the impact of pushing on the predictive reasoning of grasping. The grasping network is used as a discriminator to evaluate the grasping behavior in the current state. The pushing network is used as a generator to continuously improve the graspability of objects in the workspace. The quality of the pushing action is evaluated by comparing the grasping scores of the two states before and after the push. The reward function is designed according to the score change. That is, after the push action is executed, if S t+1The best state action evaluation value predicted by the grasping network in the state is greater than S to a certain extent. t If the grab state action evaluation value is in the state, the current push is considered effective and a positive reward is given, otherwise the reward is 0:

[0077]

[0078] Where C1 is the reward value constant, λ p is a hyperparameter, Δq * is the change in the predicted value of the capture network, δ1 is the threshold value for the change in the state action evaluation value, and are the optimal state action evaluation values ​​of the network before and after pushing, respectively. The parameters are set to C1=0.5, λ p =1.5, δ1=0.1;

[0079] r2 is the degree of aggregation of objects in the environment before and after the push is executed. Because increasing the grasping space around the object can increase the probability of successful grasping, the average relative distance is introduced to measure the degree of discreteness between objects in the environment in the current state. The reward function is designed according to the change in the degree of aggregation before and after the action is executed. According to the object mask map, after image corrosion, the pixel coordinates of the center position of the circumscribed rectangle are obtained through connected domain analysis. After coordinate transformation, the three-dimensional coordinates of the object in the robot base coordinate system are obtained. Therefore, the reward function is designed according to the physical separation and aggregation degree after the push is executed as follows:

[0080]

[0081] Where C2 is the reward value constant, Δ l = l t -l t-1 It represents the difference between the average relative distances of all objects in the working area at time t and time t-1, l t and l t-1 Represent the average relative distance between all objects in the workspace at time t and time t-1 respectively, δ2 is the aggregation degree change threshold, when Δl is greater than the threshold δ2, the pushing behavior at the previous moment is considered to be effective, and a positive reward is given, and the parameters are set to c2=0.5, δ2=0.1;

[0082] r3 takes into account the boundary constraints of the work area. When the robot arm performs the pushing action, it needs to keep the object in the work area to ensure the task completion rate, so a corresponding penalty is imposed for leaving the work area:

[0083] r3=aC3

[0084] Where C3 is the penalty value constant, a is the penalty coefficient. If the object leaves the working area, a penalty is given. The parameter setting is C3 = 1.5. When the object is pushed out of the working area, a = -1, and in other cases a = 0;

[0085] R4 considers the efficiency of the robot arm in completing the grasping task. The better the separation effect of the push action, the higher the success rate of grasping. At the same time, the fewer the number of push actions, the shorter the overall action sequence will be, and the higher the task efficiency will be.

[0086] r4=bC4

[0087] Where C4 is the reward value constant, b is the reward coefficient, and the parameter setting C4=0.5. If the number of times the robot arm performs the push action is within 4 times, the grasping condition is met, and b is set to 1. In other cases, if the number of pushes exceeds the threshold, even if the grasping condition is met, b is set to 0.

[0088] Therefore, for driving the network, the reward function integrates into:

[0089] R p =r1+r2+r3+r4

[0090] Used to guide the robot arm to improve its strategy to obtain higher cumulative rewards.

[0091] (6) Conduct model training and experimental sample testing in a simulation environment and then transplant them to a real physical environment.

[0092] Specific implementation: After the state variables, action space, reward function, and network model are constructed, the DDQN architecture is used for training. The overall action decision network structure is as follows Figure 5 As shown, after the current environment state passes through the feature extraction network module, it is input into the push or grasp backbone network to predict the state action value. After the state action value is obtained, it is screened by the push mask and constraint strategy to obtain the best push and grasp prediction. The two action prediction results are selected, and the robot executes the optimal action to transfer to the next state and obtain a reward. In this process, after the main network in the behavior value evaluation module selects the action, the target network evaluates the action, and the target value is calculated for the loss function to update the main network parameters. The parameters of the target network are updated according to the network update rate. After the model training is completed in the simulation, it is migrated to the real experimental platform. The hardware platform of this method is mainly composed of a six-axis robot with a two-finger parallel gripper, a binocular camera, and a computer. The camera is calibrated with internal parameters, and the hand-eye calibration is performed on the relative position of the camera and the robot.

[0093] On this basis, a model experiment was conducted. After the test was completed in the simulation environment, a complex and closely arranged scene was set up in the real environment for testing. The grasping method alone and other collaborative methods were used to conduct comparative tests with the method of the present invention. 50 experimental tests were conducted in the scene where multiple objects were closely arranged. Figure 6 As shown in a, in the case of close stacking in the simulation environment, the grasping success rate of this method is 95%, and the task completion rate is 98%; Figure 6 As shown in b, in the real environment, the grasping success rate of this method is 90%, and the task completion rate is 92%. Compared with the grasping method only and the traditional collaborative method, this method has higher task success rate and grasping accuracy, and the model performance is better.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some or all of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope defined by the claims of the present invention.

Claims

1. A robot arm collaborative autonomous grasping method based on deep reinforcement learning, characterized in that: Contains the following: Based on the environment state height map, a feature extraction network is built to extract representative spatial features from the input data to obtain feature vectors, and a network prediction model for pushing / grasping actions is built. The aforementioned feature vectors are used as inputs to perform strategy evaluation in combination with state features and action values ​​to predict the distribution of state-action evaluation values ​​for candidate actions. The object mask function M is introduced for pushing and grasping actions respectively. p_obj and M g_obj , combining it with the network prediction results as prior information, constraining the exploration space during the interaction between the robot and the environment, reducing the interference of negative samples in the experience replay pool on the model convergence speed, and improving the efficiency and stability of model training; Design different behavior constraint strategies to suppress unreasonable behaviors in network prediction; In view of the speed of the robotic arm to complete the task and the feedback of the environmental state, a reward function is designed for different action tuples to guide the update of the policy network model, which is trained in a simulation environment and migrated to the real environment.

2. The method for autonomous robotic arm collaborative grasping based on deep reinforcement learning according to claim 1, characterized in that: A feature extraction network is constructed based on the environment state height map, representative spatial features are extracted from the input data to obtain feature vectors, and a pushing / grasping action network prediction model is constructed. The above-mentioned feature vectors are used as input, and strategy evaluation is performed in combination with state features and action values ​​to predict the state-action evaluation value distribution of candidate actions; specifically: first, the single-channel depth height map in the environment state height map is copied to obtain three-channel data, and then the color height map and the depth height map are rotated respectively and input into the shared feature fusion module, and their output is used as the input of the high-level feature extraction network to obtain an intermediate feature vector, and the intermediate feature vector is spliced ​​with color and depth features and input into the behavior value evaluation module to obtain a state-action evaluation value distribution map of different actions in different directions, wherein the behavior value evaluation module includes a pushing network module and a grasping network module.

3. The method for autonomous robotic arm collaborative grasping based on deep reinforcement learning according to claim 1, characterized in that: After the network is built, the action is selected according to the strategy, and the object mask function M is introduced for the pushing and grasping actions respectively. p_obj and M g_obj As prior information, the robot arm is guided to conduct meaningful spatial exploration. Specifically, for the pushing action, the object index is obtained according to the initialization setting of the environment state in the simulation environment, and the outline of the vertical projection of the object is obtained according to its coordinates. The inside of the object outline is filled with 1 and the outside of the outline is filled with 0 through threshold segmentation to obtain the mask function in the simulation environment. In the real physical experimental environment, the mask function is obtained according to the depth height map of the workspace. Specifically, first, according to the formula Normalize the depth height map, where D(x,y) is the depth map pixel value, D min and D max is the minimum and maximum depth value, D′(x, y) is the normalized depth map pixel value; the depth threshold T for separating the foreground object and the background of the image is set according to the depth distribution thre , set the pixel points whose depth value is greater than the threshold to 1, otherwise set them to 0, and get the promoted binary mask M p_obj .

4. The method for autonomous robotic arm collaborative grasping based on deep reinforcement learning according to claim 3 is characterized in that: After the network is built, the action is selected according to the strategy, and the object mask function M is introduced for the pushing and grasping actions respectively. p_obj and M g_obj As a priori information, it guides the robot arm to conduct meaningful spatial exploration. Specifically, for the grasping action, the processing method in the simulation environment is the same as that of the pushing action. In the real experimental environment, the depth height map is normalized and filtered, and then the threshold segmentation is performed to obtain the initial grasping mask M. g_ori To prevent the grasping operation from failing due to the grasping position being at the edge of the object, the initial grasping mask is denoised and edge optimized to obtain the final grasping mask M. g_obj ; The pixel-by-pixel state-action evaluation value distribution map output by the push and grab network is compared with the push mask M p_obj and grab mask M g_obj Combined with the above, an action value evaluation map with prior information is obtained, which encourages the robot arm to explore the effective area in the workspace and reduces the impact of negative samples during training.

5. The method for autonomous robotic arm collaborative grasping based on deep reinforcement learning according to claim 1, characterized in that: Different behavior constraint strategies are designed to suppress unreasonable behaviors in network prediction. Specifically, the connected domain analysis is performed on the push mask to extract the number of pixels, bounding boxes and geometric information of the connected domain of objects in the workspace, and the constraint strategy is designed on this basis. The constraints on the push behavior are based on the density of objects in different directions. Specifically, the posture angles are first divided into 16 equal-angle directions θ in the action space of the robot arm. d , each direction corresponds to a unit vector Secondly, for each candidate pixel point (x, y) in each direction, the centroid distribution direction in its adjacent area is calculated. The centroid is calculated as follows: Where (c x ,c y ) is the centroid coordinate in the connected domain, R is the set of pixels in the connected domain, (x, y) represents the pixels in the set R, and N represents the number of pixels in the connected domain; then the distribution of object density in different directions is calculated, and the formula is as follows: Where (c xi ,c yi ) is the centroid coordinate of each connected region, is the relative position vector between the centroid of a region and the candidate action pixel point (x, y), for Normalized direction vector, S dp The similarity between each centroid and the current direction is expressed in θ d The density of objects in a direction, is the direction θ d Unit vector of ; based on the direction density S dp Generate push action weights for each direction Represents the priority of the current pixel (x, y) in each pushing direction, and combines it with the pushing action value evaluation map with prior information as a pushing behavior constraint strategy.

6. The method for autonomous robotic arm collaborative grasping based on deep reinforcement learning according to claim 5, characterized in that: Different behavior constraint strategies are designed to suppress unreasonable behaviors in network prediction. Specifically, the connected domain analysis is performed on the grasping mask map to extract the number of pixels, bounding boxes and geometric information of the connected domain of the objects in the workspace, and on this basis, the grasping action constraint strategy is designed. The constraint on the grasping behavior is to select the closest grasping direction based on the main axis direction of the object. Specifically, the corresponding unit direction vector is constructed for each candidate grasping action in each direction. θ d For the equal-angle posture angle directions divided in the action space, calculate the unit direction vector of the main axis direction of the object where the grasping position candidate point is located in each direction where θ obj is the main axis direction of the connected domain where the current grasping position is located, and the similarity between the main axis direction and the grasping direction is calculated: Where S dg Used to calculate the grasping angle constraint factor, S dg The closer it is to 1, the closer the current grasping direction is to the main axis direction, and the higher the grasping success rate. It is used as a grasping weight strategy and combined with the grasping action value evaluation graph with prior information as a grasping behavior constraint strategy.

7. The method of autonomous robotic arm collaborative grasping based on deep reinforcement learning according to claim 1, characterized in that: According to the rapidity of the robot arm to complete the grasping task and the feedback of the environment state, a reward function is designed for different action tuples to guide the update of the strategy network model, so that the robot arm can learn reasonable grasping positions and postures in different scenarios; specifically: The reward function is divided into two parts: the push network and the grasping network. For the grasping network, there are only two results when the robot performs grasping, success or failure. Therefore, the grasping reward function R is designed considering the behavior results and constraint strategies. g : Among them C g is the reward value constant, λ g is a hyperparameter, Θ g is the grasping angle constraint factor; For the pushing network, it is necessary to consider factors such as the impact of the pushing action on subsequent grasping, the degree of aggregation of objects in the environment, the boundary constraints of the workspace, and the efficiency of task completion. The reward function is designed as follows: r1 is the reward that considers the impact of pushing on the predictive reasoning of grasping. The grasping network is used as a discriminator to evaluate the grasping behavior in the current state. The pushing network is used as a generator to continuously improve the graspability of objects in the workspace. The quality of the pushing action is evaluated by comparing the grasping scores of the two states before and after the push. The reward function is designed according to the score change. That is, after the push action is executed, if S t+1 The best state action evaluation value predicted by the grasping network in the state is greater than S to a certain extent. t If the grab state action evaluation value is in the state, the current push is considered effective and a positive reward is given, otherwise the reward is 0: Where C1 is the reward value constant, λ p is a hyperparameter, is the change in the predicted value of the capture network, δ1 is the threshold value for the change in the state action evaluation value, and They are the best state action evaluation values ​​of the grasping network before and after pushing, respectively; r2 considers the degree of aggregation of objects in the environment before and after the execution of the push. Because increasing the grasping space around the object can increase the probability of successful grasping, the average relative distance is introduced to measure the degree of discreteness between objects in the environment in the current state. The reward function is designed according to the change in the degree of aggregation before and after the action is executed: Where C2 is the reward value constant, Δ l = l t -l t-1 It represents the difference between the average relative distances of all objects in the working area at time t and time t-1, l t and l t-1 represents the average relative distance between all objects in the working area at time t and time t-1, respectively, δ2 is the aggregation degree change threshold, when Δ l When it is greater than the threshold δ2, the pushing behavior at the previous moment is considered effective and a positive reward is given; r3 takes into account the boundary constraints of the work area. When the robot arm performs the pushing action, it needs to keep the object in the work area to ensure the task completion rate, so a corresponding penalty is imposed for leaving the work area: r3=aC3 Where C3 is the penalty value constant and a is the penalty coefficient; R4 considers the efficiency of the robot arm in completing the grasping task. The better the separation effect of the push action, the higher the success rate of grasping. At the same time, the fewer the number of push actions, the shorter the overall action sequence will be, and the higher the task efficiency will be. r4=bC4 Where C4 is the reward value constant, b is the reward coefficient; Therefore, for driving the network, the reward function integrates into: R p ＝r1+r2+r3+r4 Used to guide the robot arm to improve its strategy to obtain higher cumulative rewards.

Images