A deep reinforcement learning-based robot arm cooperative autonomous grasping method

Abstract

The application provides a kind of mechanical arm cooperation grabbing method based on deep reinforcement learning, it is related to robot application technical field, to solve the problem of network instability in the method of multi-object grabbing in complex scene, low sample efficiency, unreasonable behavior.The method comprises: constructing a cooperative autonomous grabbing decision network model, obtaining the state-action value distribution of pushing and grabbing action by pixel-by-pixel prediction;Introduce object mask function, filter invalid area, reduce the influence of negative samples on model convergence speed;Design different behavior constraint strategy to suppress unreasonable behavior in network prediction, optimize the action selection of mechanical arm;Design a multi-element reward function, dynamically adjust the reward mechanism, guide the model to learn more action strategy that meets the task goal.The application is verified through simulation and real experiment environment, which can improve the task success rate and completion efficiency of the mechanical arm in unstructured scene, and has good robustness and applicability.

Description

Technical Field

[0001] This invention relates to the field of robotics application technology, specifically to a collaborative autonomous grasping method for robotic arms based on deep reinforcement learning. Background Technology

[0002] With the rapid development of robotics and artificial intelligence, intelligent grasping tasks based on robotic arms have played a vital role in industrial automation, logistics sorting, and service robots. However, achieving efficient and stable grasping operations in complex scenarios still faces many challenges, such as tightly stacked multiple objects, dynamically changing scenes, and safety and accuracy issues during operation.

[0003] Traditional robotic arm grasping methods largely rely on precise environmental modeling and predefined rules, using specific algorithms to plan the robotic arm's operational path. However, these methods often exhibit insufficient adaptability and high dependence on the environment when facing complex dynamic environments or unknown scenarios, significantly limiting their application scope. In recent years, deep learning and reinforcement learning technologies have made significant progress in the field of robot intelligent decision-making, providing new solutions for complex grasping tasks. In reinforcement learning-based grasping tasks, how to efficiently extract scene spatial features, design reasonable action value evaluation mechanisms, and optimize training efficiency remain key issues affecting practical application results. Furthermore, while reinforcement learning technology provides new ideas for autonomous decision-making in robotic arms, conventional Q-learning or deep Q-networks are prone to instability, slow convergence speed, and unreasonable action selection when dealing with continuous state spaces and large-scale action spaces. Therefore, designing more efficient learning algorithms and reasonable exploration strategies is a key issue in improving the intelligence level of grasping tasks.

[0004] Therefore, there is a need to invent a method that is fast in training, stable, and can quickly complete the grasping task in various structured and unstructured scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a collaborative autonomous grasping method for robotic arms based on deep reinforcement learning, so as to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A feature extraction network is constructed based on the environmental state height map. Representative spatial features are extracted from the input data to obtain feature vectors. A push / grab action network prediction model is constructed. The aforementioned feature vectors are used as input. The state features and action values ​​are combined to evaluate the strategy and predict the state action evaluation value distribution of candidate actions.

[0008] An object mask function M is introduced for both pushing and grasping actions. p_obj and M g_obj By combining this information with the network prediction results, the exploration space during the interaction between the robotic arm and the environment is constrained, reducing the interference of negative samples in the experience replay pool on the model's convergence speed and improving the efficiency and stability of model training.

[0009] Design different behavioral constraint strategies to suppress unreasonable behaviors in network prediction;

[0010] To address the speed at which the robotic arm completes tasks and the feedback from environmental states, reward functions are designed for different action tuples to guide the updating of the policy network model. The model is trained in a simulation environment and then transferred to a real environment.

[0011] Furthermore, in the proposed scheme, a feature extraction network is constructed based on the environmental state height map. Representative spatial features are extracted from the input data to obtain feature vectors. A push / grab action network prediction model is then constructed. The aforementioned feature vectors are used as input, and a strategy evaluation is performed by combining state features and action value to predict the distribution of state action evaluation values ​​for candidate actions. This includes: firstly, copying the single-channel depth height map from the environmental state height map to obtain 3-channel data; then, rotating the color height map and the depth height map respectively and inputting them into the shared feature fusion module, and using its output as the input to the high-level feature extraction network to obtain intermediate feature vectors; then, concatenating the color and depth features of the intermediate feature vectors and inputting them into the behavior value evaluation module to obtain the distribution map of state action evaluation values ​​for different actions in different directions. The behavior value evaluation module includes a push network module and a grasp network module, both of which are master network and target network architectures.

[0012] Furthermore, in the proposed scheme, after the network is constructed, the action to be executed is selected according to the strategy, and an object mask function M is introduced for the pushing and grasping actions respectively. p_obj and M g_obj As prior information, it guides the robotic arm to conduct meaningful spatial exploration, including: for pushing actions, in a simulation environment, obtaining the object index based on the initial settings of the environment state, obtaining the outline of the object's vertical projection based on its coordinates, and filling the area inside the object outline with 1s and the area outside the outline with 0s through threshold segmentation to obtain the object mask function in the simulation environment; in a real physical experimental environment, obtaining the object mask function based on the depth-height map of the workspace, specifically: first, according to the formula... The depth height map is normalized, where D(x,y) is the pixel value of the depth map. min and D max The minimum and maximum depth values ​​are represented by D′(x,y), which are the normalized pixel values ​​of the depth map. A depth threshold T is set based on the depth distribution to separate foreground objects from the background.thre Pixels with depth values ​​greater than a threshold are set to 1, and those with depth values ​​less than a threshold are set to 0, thus obtaining the pushing object mask function M. p_obj .

[0013] Furthermore, in the proposed scheme, after the network is constructed, the action to be executed is selected according to the strategy, and an object mask function M is introduced for the pushing and grasping actions respectively. p_obj and M g_obj As prior information, it guides the robotic arm to conduct meaningful spatial exploration, including: for grasping actions, the processing method in the simulation environment is the same as that for pushing actions; in the real experimental environment, the depth-height map is normalized and filtered, and then threshold segmentation is performed to obtain the initial object mask function M for the grasping action. g_ori To prevent the grabbing operation from failing due to the grabbing position being at the edge of the object, noise reduction and edge optimization are performed to obtain the final object mask function M. g_obj The pixel-by-pixel state-action evaluation value distribution maps output by the pushing and grasping networks are compared with the object mask function M of the pushing action, respectively. p_obj and the object mask function M for the grabbing action g_obj By combining these elements, a motion value assessment map with prior information is obtained, which encourages the robotic arm to explore effective areas in the workspace and reduces the impact of negative samples during training.

[0014] Furthermore, in the proposed scheme, a behavioral constraint strategy is designed to suppress unreasonable behaviors in network prediction. This includes: performing connected component analysis on the mask function of the pushing object to extract the number of pixels, bounding boxes, and geometric information of the connected components of the object in the working area, and designing a constraint strategy based on this. The constraint on the pushing behavior is based on the density of objects in different directions. Specifically, the posture angle is first divided into 16 equal-angle directions θ in the action space of the robotic arm. d Each direction corresponds to a unit vector. Secondly, for each candidate pixel (x, y) in each direction, the centroid distribution direction within its neighboring region is calculated. The centroid is calculated as follows:

[0015]

[0016] Among them (c x ,c y Let (x, y) be the centroid coordinates within the connected component, R be the set of pixels within the connected component, (x, y) represent the pixel in set R, and N represent the number of pixels within the connected component; then, calculate the distribution of object density in different directions using the following formula:

[0017]

[0018] Among them (cxi ,c y i) The centroid coordinates of each connected region Let be the relative position vector between the centroid of a certain region and the candidate action pixel (x,y). for The normalized direction vector, S dp For each centroid, the similarity to the current direction is represented by θ. d The density of objects in a direction, It is the direction θ d Unit vectors; based on directional density S dp Generate push action weights for each direction The priority of the current pixel (x,y) in each pushing direction is represented and combined with the pushing action value evaluation map with prior information as a pushing behavior constraint strategy.

[0019] Furthermore, in the proposed scheme, a behavioral constraint strategy is designed to suppress unreasonable behaviors in network prediction. This includes: performing connected component analysis on the object-grabbing mask function to extract the number of pixels, bounding boxes, and geometric information of the connected components of the object in the working area; and designing a grasping action constraint strategy based on this. The constraint on the grasping behavior is based on selecting the closest grasping direction according to the object's principal axis direction. Specifically, a corresponding unit direction vector is constructed for each candidate grasping action in each direction. θ d For each equiangular pose angle direction in the action space, calculate the unit direction vector of the principal axis direction of the object where the candidate grab position point is located. Where θ obj It is the main axis direction of the connected component where the current grab position is located. Calculate the similarity between the main axis direction and the grab direction: Where S dg Used to calculate the grasping angle constraint factor, S dg The closer it is to 1, the closer the current grabbing direction is to the main axis direction, and the higher the grabbing success rate. This is used as a grabbing weight strategy and combined with a grabbing action value evaluation map with prior information as a grabbing behavior constraint strategy.

[0020] Furthermore, in the proposed solution, a reward function is designed for different action tuples to guide the updating of the policy network model, taking into account the speed at which the robotic arm completes the task and the feedback from the environmental state. This enables the robotic arm to learn reasonable grasping positions and postures in different scenarios. The reward function consists of two parts: a pushing network and a grasping network.

[0021] For grasping networks, when a robotic arm performs a grasping action, there are only two outcomes: success or failure. Therefore, the grasping reward function R needs to be designed considering both the behavioral outcome and the constraint policy.g :

[0022]

[0023] Where C g λ is a constant representing the reward value. g For hyperparameters, Θ g It is the grasping angle constraint factor;

[0024] For push networks, factors such as the impact of the push action on subsequent grasping, the degree of aggregation of objects in the environment, workspace boundary constraints, and task completion efficiency need to be considered. The reward function is designed as follows:

[0025] r1 is the reward that considers the impact of pushing on the predictive inference of the grasping action. The grasping network acts as a discriminator, evaluating the grasping behavior in the current state, while the pushing network acts as a generator, continuously increasing the graspable probability of objects in the work area. The quality of the pushing action is evaluated by comparing the grasping scores before and after the pushing action. A reward function is designed based on the score change; that is, after performing the pushing action, if S... t+1 The optimal state action evaluation value predicted by the state-capturing network is greater than S to some extent. t If the grabbing action in the current state is evaluated, the current push is considered valid and a positive reward is given; otherwise, the reward is 0.

[0026]

[0027] Where C1 is the reward value constant, λ p It's a hyperparameter. To capture the change in the network's predicted values, δ1 is the threshold for the change in the state / action evaluation values. and These are the optimal state action evaluation values ​​for the grasping network before and after the push execution;

[0028] r2 considers the degree of aggregation of objects in the environment before and after the action is executed. Since increasing the grasping space around an object increases the probability of successful grasping, the average relative distance is introduced to measure the degree of dispersion between objects in the current state. The reward function is designed based on the change in the degree of aggregation before and after the action is executed.

[0029]

[0030] Where C2 is the reward value constant, Δ l =l t -l t-1 l represents the difference in average relative distance between all objects in the working area at time t and time t-1. t and l t-1δt and t-1 represent the average relative distances between all objects in the working area at time t and time t-1, respectively, and δ2 is the threshold for the change in the degree of aggregation. l If the value exceeds the threshold δ2, the previous driving action is considered effective and a positive reward is given.

[0031] r3 considers the boundary constraints of the work area. During the robotic arm's pushing motion, it needs to keep the object within the work area to ensure the task completion rate. Therefore, a penalty is imposed for objects that leave the work area.

[0032] r3=aC3

[0033] Where C3 is the penalty value constant, and a is the penalty coefficient;

[0034] r4 takes into account the efficiency of the robotic arm in completing the grasping task. The better the separation effect of the pushing action, the higher the success rate of grasping. At the same time, the fewer push actions are performed, the shorter the overall action sequence will be, and the higher the task efficiency will be.

[0035] r4=bC4

[0036] Where C4 is the reward value constant, and b is the reward coefficient;

[0037] Therefore, for the push network, the reward function is integrated as follows:

[0038] R p =r1+r2+r3+r4

[0039] This is used to guide the improvement strategy of robotic arms to obtain higher cumulative rewards.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] This method utilizes prior information combining object masking functions and network-predicted state value distribution maps to effectively reduce negative samples generated by the robotic arm exploring invalid regions, thereby improving model convergence speed. Different behavioral constraint strategies are designed to suppress unreasonable actions predicted by the network, improving the success rate of action execution and task completion rate. By introducing diversified reward functions to different action tuples, the optimization of the policy network model is more accurately guided, ensuring the efficiency and accuracy of the robotic arm in complex scenarios. Attached Figure Description

[0042] Figure 1 This is a flowchart of a collaborative autonomous grasping method for robotic arms based on deep reinforcement learning, as described in this invention.

[0043] Figure 2 This is a parameter diagram of the push / grab action network prediction model described in this invention;

[0044] Figure 3 This is a schematic diagram of the driving behavior constraint strategy described in this invention;

[0045] Figure 4 This is a schematic diagram of the crawling behavior constraint strategy described in this invention;

[0046] Figure 5 This is a schematic diagram of the collaborative autonomous crawling network decision-making process structure described in this invention;

[0047] Figure 6 The figures show the comparative experimental results of the model under the simulation environment and the real environment described in this invention. Figure 6 Figure 'a' shows the results of the comparative experiment under simulation conditions. Figure 6 b is a comparison diagram of the experimental results under real physical conditions. Detailed Implementation

[0048] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0049] The present invention describes a collaborative autonomous grasping method for robotic arms based on deep reinforcement learning, such as... Figure 1 As shown, it includes the following steps:

[0050] (1) Build a simulation environment according to the real scene, load the robotic arm model and camera model, and randomly load 30 objects of different shapes in each training session. The positions of each object are randomly distributed.

[0051] (2) Based on the environmental state height map, a feature extraction network is constructed to extract representative spatial features from the input data to obtain feature vectors. A push / grab action network prediction model is constructed, and the aforementioned feature vectors are used as input. The state features and action values ​​are combined to evaluate the strategy and predict the state action evaluation value distribution of candidate actions.

[0052] Specific implementation: First, the single-channel depth-height map is copied to obtain 3-channel data; an affine matrix with a counter-clockwise rotation angle is constructed, and the color height state map and depth height state map are rotated 16 times at a specified angle. Each rotated image is input into the shared feature fusion module for preliminary feature extraction; the above output is used as the input to the feature extraction network, which obtains color intermediate feature vectors and depth intermediate feature vectors. The number of channels for a single feature vector is 1024, and after channel concatenation, the number of channels is 2048. The concatenated intermediate feature vectors are input into the behavior value evaluation module, including the push network module and the grasping network module, to obtain a distribution map of 32 single-channel state action evaluation values ​​for push and grasping actions in 16 directions. Each direction represents the rotation angle of the end effector of the robotic arm. The pixel positions in the distribution map correspond one-to-one with the 3D coordinate positions in the workspace, representing the specific position of the robotic arm's action; the first part of the feature extraction network backbone consists of multiple dense blocks and transition layers, while the push network and grasping network are independent convolutional branches, such as... Figure 2 As shown, the network includes three batch normalization layers, a ReLU activation function layer, and three convolutional layers with different configurations, as well as a bilinear sampling interpolation upsampling layer. Since this method uses the DDQN architecture for training, the designed target network has the same structure as the main network. The parameters of the target network are synchronized with the main network through soft updates, as shown in the following formula:

[0053] θ target ←τθ online +(1-τ)θ target

[0054] Where θ target The target network parameters, θ online These are the main network parameters, and τ = 0.005 is the target network parameter update rate.

[0055] (3) Introduce object mask functions M for pushing and grasping actions respectively. p_obj and M g_obj As prior information, it constrains the exploration space during the learning process of the robotic arm interacting with the environment.

[0056] Specific implementation: For the pushing action, in the simulation environment, the object index is obtained based on the initial settings of the environment state. The coordinates of its current position are read from the object index. The outline of the object's vertical projection is obtained based on the coordinates and object size. Threshold segmentation is used to fill the area inside the outline with 1s and the area outside with 0s, resulting in the pushing object mask function in the simulation environment. In the real physical experimental environment, the object mask function is obtained based on the workspace depth-height map. Specifically: First, according to the formula... The depth height map is normalized, where D(x,y) is the pixel value of the depth map.min and D max The minimum and maximum depth values ​​need to be read in real time based on the depth map. D′(x,y) is the normalized depth image. The binarized depth threshold T for separating foreground objects and background is manually set based on the depth distribution. thre Set parameter T thre =0.4, setting pixels with depth values ​​greater than the threshold to 1, and others to 0, yields the object mask function M for the initial pushing action. p_ori Then, a single dilation process was performed on it using a 5×5 structuring element to obtain the object mask function image M of the pushing action. p_obj .

[0057] For the grasping action, the processing method in the simulation environment is the same as that for the pushing action. In the real experimental environment, the depth-height map is normalized and filtered for holes, and then thresholded to obtain the initial object mask function M for the grasping action. g_ori To prevent incomplete grasping due to the grasping position being at the object's edge, filtering is performed, including using a 3×3 structuring element to eliminate noise and preserve the object's shape, followed by edge optimization using a 5×5 erosion kernel, resulting in the object mask function M for the grasping action. g_obj .

[0058] The pixel-by-pixel state-action evaluation value distribution maps of the driving network and the grasping network outputs are compared with the object mask function M of the driving action. p_obj and the object mask function M for the grabbing action g_obj By performing Hadamard product operations, regions containing objects in the workspace are extracted, resulting in an action value assessment map with prior information. This guides the robotic arm to explore effective regions in the workspace, reducing the impact of negative samples generated during training operations in object-free regions on the model's convergence speed.

[0059] (4) Design behavioral constraint strategies to suppress unreasonable behaviors in network prediction.

[0060] Specific implementation: Perform connected component analysis on the object mask function obtained in (3), extract the number of pixels, bounding box and geometric information of the connected components of the object in the working area, and design different constraint strategies for pushing and grasping respectively; specifically:

[0061] The constraints on the pushing behavior are based on the density of objects in different directions, such as Figure 3 As shown: First, the posture angle is divided into 16 equal-angle directions θ in the motion space of the robotic arm. d ∈{0°,22.5°,45°,…,337.5°}, each direction corresponds to a unit vector. Secondly, for each candidate pixel (x, y) in each direction, the centroid distribution direction within its neighboring region is calculated. The centroid is calculated as follows:

[0062]

[0063] Among them (c x ,c y Let (x, y) be the centroid coordinates within the connected component, R be the set of pixels within the connected component, (x, y) represent the pixel in set R, and N represent the number of pixels within the connected component; then, calculate the distribution of object density in different directions using the following formula:

[0064]

[0065] Among them (c xi ,c yi The centroid coordinates of each region, Let be the relative position vector between the centroid of a certain region and the candidate action pixel (x,y). for The normalized direction vector, S dp For each centroid, the similarity to the current direction is represented by the similarity in direction θ. d The density of objects on the surface It is the direction θ d Unit vectors; based on directional density S dp Generate push action weights for each direction This reflects the priority of the current pixel (x,y) in each pushing direction. A higher weight means that the pushing action in that direction is more likely to successfully complete the task. The pushing action value evaluation map with prior information obtained in (3) is combined with the direction weight:

[0066]

[0067] in This represents the value assessment graph of the pushing action with prior information in the d-th direction. The above operation is completed by element-wise multiplication through the broadcast mechanism in PyTorch.

[0068] The constraint design for grasping behavior is based on the object's principal axis direction, selecting the grasping direction closest to it, such as... Figure 4 As shown: Construct the corresponding unit direction vector for each candidate grabbing action in each direction. θ d For each equiangular pose angle direction in the action space, calculate the unit direction vector of the principal axis direction of the object where the candidate grab position point is located. Where θ objIt is the main axis direction of the connected component where the current position is located. The similarity between the main axis direction and the 16 grab directions is calculated by dot product: Where S dg Used to calculate the grasping angle constraint factor, S dg The closer the value is to 1, the closer the current grasping direction is to the main axis direction, and the higher the grasping success rate. This value is used as a grasping weight strategy and combined with a grasping action value evaluation graph containing prior information as a grasping behavior constraint strategy.

[0069]

[0070] in This represents the value assessment diagram of the grasping action with prior information in the d-th direction.

[0071] (5) To ensure the speed and safety of the robotic arm in completing the grasping task, a reward function is designed for different action tuples to guide the updating of the policy network model.

[0072] Implementation details: For the grasping network, when the robotic arm performs a grasping action, there are only two outcomes: successfully grasping the object and receiving a positive reward, or failing and receiving a zero reward. Therefore, the grasping reward function R is designed considering the behavioral outcome and constraint strategy. g :

[0073]

[0074] Where C g λ is a constant representing the reward value. g Θ is a hyperparameter used to measure the influence of the behavior constraint policy in the current grasping direction on the grasping operation. g =1-S dg It is the grasping angle constraint factor, S dg To determine the similarity between the current grab direction and the principal axis direction of the object masking function, the parameter is set to C. g =1.6, λ g =0.5;

[0075] For push networks, factors such as the impact of the push action on subsequent grasping, the degree of aggregation of objects in the environment, workspace boundary constraints, and task completion efficiency need to be considered. The reward function is designed as follows:

[0076] r1 is the reward that considers the impact of pushing on the predictive inference of the grasping action. The grasping network acts as a discriminator, evaluating the grasping behavior in the current state, while the pushing network acts as a generator, continuously increasing the graspable probability of objects in the work area. The quality of the pushing action is evaluated by comparing the grasping scores before and after the pushing action. A reward function is designed based on the score change; that is, after performing the pushing action, if S... t+1If the evaluation value of the best state action predicted by the grasping network in the current state is greater than the evaluation value of the grasping state action in the St state to a certain extent, then the current push is considered effective and a positive reward is given; otherwise, the reward is 0.

[0077]

[0078] Where C1 is the reward value constant, λ p It's a hyperparameter, Δq * To capture the change in the network's predicted values, δ1 is the threshold for the change in the state / action evaluation values. and These are the optimal state action evaluation values ​​for the grasping network before and after the push, respectively, with parameters set to C1 = 0.5 and λ. p =1.5, δ1=0.1;

[0079] r2 considers the degree of aggregation of objects in the environment before and after the push execution. Since increasing the grasping space around the object can improve the probability of successful grasping, the average relative distance is introduced to measure the degree of dispersion between objects in the current state. The reward function is designed based on the change in the degree of aggregation before and after the action execution. Based on the object masking function, after image erosion, the pixel coordinates of the center position of the bounding rectangle are obtained through connected component analysis. After coordinate transformation, the three-dimensional coordinates of the object in the robot arm's base coordinate system are obtained. Therefore, the reward function is designed based on the degree of physical separation and aggregation after the push execution as follows:

[0080]

[0081] Where C2 is the reward value constant, Δ l =l t -l t-1 l represents the difference in average relative distance between all objects in the working area at time t and time t-1. t and l t-1 Δl represents the average relative distance between all objects in the working area at time t and time t-1, respectively. δ2 is the threshold for the change in the degree of aggregation. When Δl is greater than the threshold δ2, the pushing behavior at the previous time is considered to be effective and a positive reward is given. The parameters are set to C2 = 0.5 and δ2 = 0.1.

[0082] r3 considers the boundary constraints of the work area. During the robotic arm's pushing motion, it needs to keep the object within the work area to ensure the task completion rate. Therefore, a penalty is imposed for objects that leave the work area.

[0083] r3=aC3

[0084] Where C3 is the penalty value constant and a is the penalty coefficient. If the object leaves the working area, a penalty is imposed. The parameter is set to C3 = 1.5. When the object is pushed out of the working area, a = -1. Otherwise, a = 0.

[0085] r4 takes into account the efficiency of the robotic arm in completing the grasping task. The better the separation effect of the pushing action, the higher the success rate of grasping. At the same time, the fewer push actions are performed, the shorter the overall action sequence will be, and the higher the task efficiency will be.

[0086] r4=bC4

[0087] Where C4 is the reward value constant and b is the reward coefficient. The parameter is set to C4 = 0.5. If the gripping condition is met if the robotic arm performs the pushing action within 4 times, b is set to 1. Otherwise, if the pushing number threshold is exceeded, even if the gripping condition is met, b is set to 0.

[0088] Therefore, for the push network, the reward function is integrated as follows:

[0089] R p =r1+r2+r3+r4

[0090] This is used to guide the improvement strategy of robotic arms to obtain higher cumulative rewards.

[0091] (6) Conduct model training and experimental sample testing in the simulation environment, and then port it to the real physical environment.

[0092] Specific implementation: After the state variables, action space, reward function, and network model are constructed, the DDQN architecture is used for training. The overall action decision network structure is as follows: Figure 5 As shown, the current environmental state is processed by a feature extraction network module and then input into the pushing or grasping backbone network for state-action value prediction. After obtaining the state-action value, it is filtered by an object mask function and constraint strategy to obtain the optimal pushing or grasping prediction. The best of the two predictions is selected, and the robotic arm executes the optimal action to transition to the next state and receive a reward. During this process, after the main network in the behavior value evaluation module selects the action, the target network evaluates the action and calculates the target value, which is used to update the parameters of the main network using the loss function. The parameters of the target network are updated according to the network update rate. After model training is completed in the simulation, it is transferred to a real experimental platform. The hardware platform of this method mainly consists of a six-axis robotic arm with a two-finger parallel gripper, a binocular camera, and a computer. The camera's intrinsic parameters are calibrated, and the relative position between the camera and the robotic arm is calibrated using hand-eye calibration.

[0093] Based on this, model experiments were conducted. After testing in the simulation environment, tests were performed in a complex, tightly packed scenario in a real environment. Comparative experiments were conducted using a grasping-only method and other collaborative methods with the method of this invention. Fifty experiments were performed in each scenario with multiple objects tightly packed together. Figure 6 As shown in Figure a, in a simulated environment with tightly stacked elements, this method achieves a 95% success rate in grasping and a 98% task completion rate; Figure 6 As shown in b, in a real-world environment, this method achieves a 90% success rate in data capture and a 92% task completion rate. Compared to methods that only capture and traditional collaborative methods, this method has a higher task success rate and capture accuracy, and its model performance is superior.

[0094] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A collaborative autonomous grasping method for robotic arms based on deep reinforcement learning, characterized in that, Includes the following: A feature extraction network is constructed based on the environmental state height map. Representative spatial features are extracted from the input data to obtain feature vectors. A push / grab action network prediction model is constructed. The aforementioned feature vectors are used as input. The state features and action values ​​are combined to evaluate the strategy and predict the state action evaluation value distribution of candidate actions. An object mask function M is introduced for both pushing and grasping actions. p_obj and M g_obj By combining this information with the network prediction results, the exploration space during the interaction between the robotic arm and the environment is constrained, reducing the interference of negative samples in the experience replay pool on the model's convergence speed and improving the efficiency and stability of model training. Different behavioral constraint strategies are designed to suppress unreasonable behaviors in network prediction; connected component analysis is performed on the mask function of the pushed object to extract the number of pixels, bounding boxes, and geometric information of the connected components of the object in the working area, and constraint strategies are designed based on this; the constraints on the pushing behavior are based on the density of objects in different directions. Specifically: firstly, the pose angle is divided into 16 equal-angle directions θ in the action space of the robotic arm. d Each direction corresponds to a unit vector. Secondly, for each candidate pixel (x, y) in each direction, the centroid distribution direction within its neighboring region is calculated. The centroid is calculated as follows: Among them (c x ,c y Let (x, y) be the centroid coordinates within the connected component, R be the set of pixels within the connected component, (x, y) represent the pixel in set R, and N represent the number of pixels within the connected component; then, calculate the distribution of object density in different directions using the following formula: Among them (c xi ,c yi ) are the centroid coordinates of each connected region. Let be the relative position vector between the centroid of a certain region and the candidate action pixel (x,y). for The normalized direction vector, S dp For each centroid, the similarity to the current direction is represented by θ. d The density of objects in a direction, It is the direction θ d Unit vectors; based on directional density S dp Generate push action weights for each direction The priority of the current pixel (x,y) in each pushing direction is represented and combined with the pushing action value evaluation map with prior information as a pushing behavior constraint strategy; Connectivity analysis is performed on the mask function of the captured object to extract the number of pixels, bounding boxes, and geometric information of the connected components of the object in the working area. Based on this, a constraint strategy for the grasping action is designed. The constraint on the grasping behavior is based on selecting the closest grasping direction according to the principal axis direction of the object. Specifically, a corresponding unit direction vector is constructed for each candidate grasping action in each direction. θ d For each equiangular pose angle direction in the action space, calculate the unit direction vector of the principal axis direction of the object where the candidate grab position point is located. Where θ obj It is the main axis direction of the connected component where the current grab position is located. Calculate the similarity between the main axis direction and the grab direction: Where S dg Used to calculate the grasping angle constraint factor, S dg The closer it is to 1, the closer the current grabbing direction is to the main axis direction, and the higher the grabbing success rate. It is used as a grabbing weight strategy and combined with the grabbing action value evaluation map with prior information as a grabbing behavior constraint strategy. To address the speed at which the robotic arm completes tasks and the feedback from environmental states, reward functions are designed for different action tuples to guide the updating of the policy network model. The model is trained in a simulation environment and then transferred to a real environment.

2. The method for collaborative autonomous grasping of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: A feature extraction network is constructed based on the environmental state height map. Representative spatial features are extracted from the input data to obtain feature vectors. A push / grasp action prediction model is then built. The aforementioned feature vectors are used as input, and policy evaluation is performed by combining state features and action value to predict the state-action evaluation value distribution of candidate actions. Specifically, the single-channel depth height map in the environmental state height map is first copied to obtain 3-channel data. Then, the color height map and depth height map are rotated and input into the shared feature fusion module. The output of the fusion module is used as the input of the high-level feature extraction network to obtain intermediate feature vectors. The intermediate feature vectors are then concatenated with color and depth features and input into the behavior value evaluation module to obtain the state-action evaluation value distribution map of different actions in different directions. The behavior value evaluation module includes a push network module and a grasp network module.

3. The method for collaborative autonomous grasping of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: After the network is constructed, actions are selected according to the strategy, and object mask functions M are introduced for the pushing and grasping actions respectively. p_obj and M g_obj As prior information, it guides the robotic arm to conduct meaningful spatial exploration; specifically: for pushing actions, in the simulation environment, the object index is obtained based on the initial settings of the environment state, and the outline of the object's vertical projection is obtained based on its coordinates. Threshold segmentation is used to fill the area inside the object outline with 1s and the area outside the outline with 0s, resulting in the object mask function in the simulation environment; in the real physical experimental environment, the object mask function is obtained based on the depth-height map of the workspace. Specifically: first, according to the formula... The depth height map is normalized, where D(x,y) is the pixel value of the depth map. min and D max The minimum and maximum depth values ​​are represented by D′(x,y), which are the normalized pixel values ​​of the depth map. A depth threshold T is set based on the depth distribution to separate foreground objects from the background. thre Pixels with depth values ​​greater than a threshold are set to 1, and those with depth values ​​less than a threshold are set to 0, thus obtaining the object mask function M for the pushing action. p_obj .

4. The method for collaborative autonomous grasping of a robotic arm based on deep reinforcement learning according to claim 3, characterized in that: After the network is constructed, actions are selected according to the strategy, and object mask functions M are introduced for the pushing and grasping actions respectively. p_obj and M g_obj As prior information, it guides the robotic arm to conduct meaningful spatial exploration. Specifically, for the grasping action, the processing method in the simulation environment is the same as that for the pushing action. In the real experimental environment, the depth-height map is normalized and filtered, and then threshold segmentation is performed to obtain the initial object mask function M for the grasping action. g_ori To prevent the grabbing operation from failing due to the grabbing position being at the edge of the object, noise reduction and edge optimization are performed to obtain the final object mask function M for the grabbing action. g_obj The pixel-by-pixel distribution of state-action evaluation values ​​from the push and grasp network outputs is compared with the object mask function M of the push action. p_obj and the object mask function M for the grabbing action g_obj By combining these elements, a motion value assessment map with prior information is obtained, which encourages the robotic arm to explore effective areas in the workspace and reduces the impact of negative samples during training.

5. The method for collaborative autonomous grasping of a robotic arm based on deep reinforcement learning according to claim 1, characterized in that: To address the issues of speed in completing grasping tasks and environmental feedback, a reward function is designed for different action tuples to guide the updating of the policy network model, enabling the robotic arm to learn appropriate grasping positions and postures in different scenarios; specifically: The reward function consists of two parts: a push network and a grasping network. For the grasping network, when the robotic arm performs a grasping action, there are only two outcomes: success or failure. Therefore, the grasping reward function R is designed considering the behavioral outcome and constraint policies. g : Where C g λ is a constant representing the reward value. g For hyperparameters, Θ g It is the grasping angle constraint factor; For push networks, it is necessary to consider the impact of the push action on subsequent grasping, the degree of aggregation of objects in the environment, work area boundary constraints, and task completion efficiency factors. The reward function is designed as follows: r1 is the reward that considers the impact of pushing on the predictive inference of the grasping action. The grasping network acts as a discriminator, evaluating the grasping behavior in the current state, while the pushing network acts as a generator, continuously increasing the graspable probability of objects in the work area. The quality of the pushing action is evaluated by comparing the grasping scores before and after the pushing action. A reward function is designed based on the score change; that is, after performing the pushing action, if S... t+1 The optimal state action evaluation value predicted by the state-capturing network is greater than S to some extent. t If the grabbing action in the current state is evaluated, the current push is considered valid and a positive reward is given; otherwise, the reward is 0. Where C1 is the reward value constant, λ p It's a hyperparameter. To capture the change in the network's predicted values, δ1 is the threshold for the change in the state / action evaluation values. and These are the optimal state action evaluation values ​​for the grasping network before and after the push execution; r2 considers the degree of aggregation of objects in the environment before and after the action is executed. Since increasing the grasping space around an object increases the probability of successful grasping, the average relative distance is introduced to measure the degree of dispersion between objects in the current state. The reward function is designed based on the change in the degree of aggregation before and after the action is executed. Where C2 is the reward value constant, Δ l =l t -l t-1 l represents the difference in average relative distance between all objects in the working area at time t and time t-1. t and l t-1 δt and t-1 represent the average relative distances between all objects in the working area at time t and time t-1, respectively, and δ2 is the threshold for the change in the degree of aggregation. l If the value exceeds the threshold δ2, the previous driving action is considered effective and a positive reward is given. r3 considers the boundary constraints of the work area. During the robotic arm's pushing motion, it needs to keep the object within the work area to ensure the task completion rate. Therefore, a penalty is imposed for objects that leave the work area. r3=aC3 Where C3 is the penalty value constant, and a is the penalty coefficient; r4 takes into account the efficiency of the robotic arm in completing the grasping task. The better the separation effect of the pushing action, the higher the success rate of grasping. At the same time, the fewer push actions are performed, the shorter the overall action sequence will be, and the higher the task efficiency will be. r4=bC4 Where C4 is the reward value constant, and b is the reward coefficient; Therefore, for the push network, the reward function is integrated as follows: R p ＝r1+r2+r3+r4 This is used to guide the improvement strategy of robotic arms to obtain higher cumulative rewards.

Images