Vision-based deep reinforcement learning method for robot grasping and assembly skills

Abstract

This invention relates to a vision-based deep reinforcement learning method for robot grasping and assembly skills, comprising: acquiring at least one image containing an object to be grasped; processing the image using a grasping action generation model to obtain several Q-value maps; determining a grasping action based on the Q-value maps; the grasping action including the action execution position and the action execution angle; preset a grasping threshold; if the Q-value of the grasping action is greater than the grasping threshold, the robot performs the grasping action; calculating an assembly threshold before and after the robot performs the grasping action; if the assembly threshold remains unchanged, the robot repeatedly performs the grasping action; if the assembly threshold decreases, the robot is considered to have completed the grasping operation.

Description

Technical Field

[0001] This invention relates to a vision-based deep reinforcement learning method for robot grasping and assembly skills, belonging to the fields of intelligent manufacturing and robotics. Background Technology

[0002] Shaft and hole assembly is a classic task for industrial robots. However, traditional industrial robot programming cycles are lengthy. In the manufacturing of diverse, small-batch, and customized products, companies need to frequently perform extensive code compilation work for different products. Robots cannot learn autonomously, thus impacting programming efficiency and failing to meet the current development requirements of enterprises. To address these issues, deep reinforcement learning-based robot programming technology, through research on the interaction between intelligent robots and their environment and autonomous learning training, enables them to learn relevant skills, such as performing shaft and hole assembly. This significantly enhances the intelligence of robots, achieving a smart upgrade for them.

[0003] There are generally two methods for industrial robot grasping tasks: analysis and data-driven methods. These methods require the establishment of mathematical and physical models, which have significant limitations in practical applications.

[0004] For industrial robot shaft and hole assembly tasks, based on whether there is contact between the shaft and hole during the assembly process, it can be divided into non-contact and contact methods. Contact methods often use tactile sensing control, which has poor safety performance, complex assembly process, and low work efficiency. Although this method can obtain an optimal control strategy, the complexity of teaching sample acquisition and system design leads to poor versatility, posing difficulties for the application of robot shaft and hole assembly tasks. Non-contact methods generally use vision alignment to complete shaft and hole assembly. This method has better application safety performance, but assembly efficiency and accuracy need further improvement. In summary, a shaft and hole assembly method with higher assembly efficiency is needed.

[0005] The paper "Research on Shaft and Hole Assembly of Simulation Robot Based on Deep Reinforcement Learning" by Liu Nailong, Liu Zhaoming, and Cui Long discloses a self-learning method based on deep reinforcement learning to train a robot to learn shaft and hole assembly tasks in a 3D graphics simulation environment. It utilizes a deep neural network to encode the robot's control strategy, and then trains this deep neural network using reinforcement learning to generalize the robot's operation strategy, enabling the robot to handle shaft and hole assembly problems under more environmental uncertainties. However, this method uses a tactile perception control approach, which suffers from poor safety performance in contact assembly, a complex assembly process, and requires further improvement in work efficiency.

[0006] The paper "Research on Deep Q-Neural Network Algorithm Combining Prior Knowledge" by Chu Wei and Ru Qi discloses that prior knowledge is defined as a mapping between feature states and optimal actions. Based on prior knowledge, the action selection rules are optimized to reduce the randomness of action selection in the exploration process of the DQN algorithm. Summary of the Invention

[0007] To overcome the problems existing in the prior art, this invention designs a vision-based deep reinforcement learning method for robot grasping and assembly skills, sets grasping thresholds and assembly thresholds, and determines whether the robot should perform grasping or assembly based on the grasping thresholds and assembly thresholds, thereby improving the success rate and accuracy of grasping and assembly operations.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A vision-based deep reinforcement learning method for robot grasping and assembly skills includes the following steps:

[0010] A vision-based deep reinforcement learning method for robot grasping and assembly skills includes the following steps:

[0011] Obtain at least one image containing the item to be grabbed;

[0012] The image to be processed is processed using a grasping action generation model to obtain multiple Q-value maps; based on the multiple Q-value maps, a grasping action is determined; the grasping action includes the action execution position and the action execution angle;

[0013] A preset grasping threshold is set; if the Q value of the grasping action is greater than the grasping threshold, the robot will perform the grasping action.

[0014] The assembly threshold is calculated before and after the robot performs the grasping action. If the assembly threshold remains unchanged, the robot repeats the grasping action multiple times. If the assembly threshold decreases, the robot is considered to have completed the grasping action.

[0015] Furthermore, it also includes: training the grasping action generation model to generate multiple grasping actions; and determining a grasping threshold based on the Q-values ​​of the multiple grasping actions.

[0016] Furthermore, it also includes: obtaining a depth-height map containing the item to be grabbed; and calculating the number of binary mask pixels of the item to be grabbed in the depth-height map as the assembly threshold.

[0017] Furthermore, the process of using the grasping action to generate a model to process the image to be processed and obtain multiple Q-value maps is as follows:

[0018] The at least one image to be processed includes an RGB height map and a depth height map;

[0019] Features are extracted from the RGB height map and the depth height map respectively to obtain a first feature map and a second feature map; the first feature map and the second feature map are fused to obtain a fusion result; the fusion result is processed using an analysis network, which includes a normalization layer, an activation layer, and a convolutional layer; the analysis result is output as multiple Q-value maps using an upsampling layer.

[0020] Furthermore, the step of determining the grasping action based on multiple Q-value maps specifically involves:

[0021] Obtain prior knowledge, which is the center point of the outline of the object to be grasped; based on the prior knowledge and the Q-value map, determine the grasping action, expressed by the formula:

[0022] a t ~π(a|s t ) = argmax(C(s) t )×O(s t )×Q grasp )

[0023] In the formula, C(s) t ) represents prior knowledge; O(s t Q represents the probability of the item to be grabbed; grasp Represents the maximum Q-value of different Q-value plots; π(a|s t ) indicates the operational strategy.

[0024] Furthermore, the acquisition of prior knowledge specifically includes:

[0025] The depth-height map containing the object to be grabbed is smoothed; the smoothed depth-height map is converted to grayscale; the outline of the object to be grabbed is detected; the outline area of ​​the object to be grabbed is calculated; and the center point of the outline of the object to be grabbed is calculated.

[0026] Furthermore, it also includes:

[0027] Construct a grasping reward function r for training the grasping action generation model. g Expressed as a formula:

[0028] r g =GD-λΔθ

[0029] In the formula, G represents the grasping result; D represents the distance between the object to be grasped and the robot's end effector; Δθ is the absolute error between the current rotation angle of the object to be grasped and the rotation angle of the robot's end effector; and λ is the heuristic coefficient.

[0030] Furthermore, it also includes:

[0031] The image to be processed includes items to be captured and items to be assembled.

[0032] The assembly motion generation model processes the image to be processed, and obtains the center point and rotation angle of the item to be assembled as the environmental state; the assembly motion generation model generates assembly motions based on the environmental state; the robot executes the assembly motions.

[0033] Furthermore, the process of processing the image to obtain the center point and rotation angle of the item to be assembled includes the following steps:

[0034] The image to be processed is a depth-height map; image preprocessing is performed on the depth-height map; a binarization threshold is set according to the pixel value difference between the object to be grabbed, the object to be assembled, and the background, and the depth-height map is binarized; the outline of the object to be assembled is detected; a binary mask is generated according to the outline of the object to be assembled; the center point and rotation angle of the object to be assembled are calculated according to the binary mask using the principal component analysis method.

[0035] Furthermore, it also includes:

[0036] Construct an assembly reward function r for training the assembly action generation model. g Expressed as a formula:

[0037]

[0038] Where, d t t represents the distance between the center point of the item to be assembled and the robot's gripping position at time t; a represents the robot's maximum workspace.

[0039] Compared with the prior art, the present invention has the following features and beneficial effects:

[0040] (1) The present invention designs a collaborative grasping action generation model and an assembly action generation model, and uses visual alignment to complete different assembly tasks, such as the assembly of round shafts and round holes. Among them, the assembly action generation model extracts the center point and rotation angle of the parts to be assembled as environmental variables and inputs them into the near-end policy optimization (PPO) network, which reduces the training time and interaction data required by the policy learning algorithm and improves the assembly success rate and generalization performance.

[0041] (2) The present invention extracts prior knowledge of the target object (i.e., the center point of the object's outline) and combines the prior knowledge to generate a grasping action, thereby improving the effectiveness of the grasping action.

[0042] (3) The present invention sets a grasping threshold and an assembly threshold; determines whether the robot performs grasping or assembly based on the grasping threshold and the assembly threshold; improves the success rate and accuracy of grasping and assembly operations.

[0043] (4) The present invention sets a heuristic grab reward function to train the grab action generation model, which accelerates the convergence speed of the grab action generation model and the accuracy of the grab action.

[0044] (5) The present invention sets up a segmented assembly reward function to train the assembly action generation model, which improves the setting of the time reward function in the close-range stage and helps to improve the success rate of assembly operation tasks. Attached Figure Description

[0045] Figure 1 This is the hardware system of the present invention;

[0046] Figure 2 This is a schematic diagram of the robot grasping-assembly action generation model structure of the present invention;

[0047] Figure 3 This is a flowchart illustrating the deep reinforcement learning process for robot grasping and assembly skills in this invention. Detailed Implementation

[0048] The present invention will now be described in more detail with reference to the embodiments.

[0049] like Figure 1 As shown, the vision-based robot grasping and assembly skills deep reinforcement learning system includes the following hardware: RGB-D camera 10, camera bracket 20, computer 30, robot controller 40, industrial robot 50, hole-shaped parts to be assembled 60, shaft-shaped parts to be grasped 70, and operator 80.

[0050] The camera bracket 20 is used to fix the RGB-D camera 10. The RGB-D camera 10 captures RGB images and depth images containing the hole-shaped part 60 and the shaft-shaped part 70, and transmits them to the computer 30 for image analysis. The computer 30 issues a robot camera extrinsic parameter calibration command to the robot controller 40 through a calibration algorithm. The industrial robot 50 executes the calibration command and obtains the attitude transformation matrix of the camera coordinate system relative to the robot coordinate system. The computer 30 converts the RGB image and depth image into an RGB height map and a depth height map according to the attitude transformation matrix and the camera intrinsic parameter coefficients, and inputs the RGB height map and depth height map together into the grasping action generation model, and inputs the depth height map into the assembly action generation model. The grasping action generation model outputs grasping program instructions, and the assembly action generation model outputs assembly program instructions. The robot controller 40 receives the grasping program instructions and assembly program instructions issued by the computer, and the robot controller 40 communicates bidirectionally with the industrial robot 50, converting the grasping program instructions and assembly program instructions into corresponding control instructions and sending them to the industrial robot 50. The industrial robot 50 performs grasping and assembly actions.

[0051] like Figure 2-3 As shown, a vision-based deep reinforcement learning system for robot grasping and assembly skills is used to complete different assembly tasks, such as assembling round shafts and round holes. It includes the following functional modules: image acquisition and processing module, prior knowledge extraction module, grasping action generation model, and assembly action generation model.

[0052] The image acquisition and processing module includes an image acquisition module and a height map conversion module connected by communication. The image acquisition module includes an RGB-D camera for acquiring RGB images and depth images of the hole-shaped part 60 and the shaft-shaped part 70; the height map conversion module converts the RGB images and depth images into RGB height maps and depth height maps according to the conversion matrix and camera intrinsic parameters.

[0053] The image acquisition and processing module's processing flow is as follows: ① Acquire RGB and depth images using an RGB-D camera; ② Obtain the transformation matrix of the camera coordinate system's pose relative to the robot coordinate system based on the camera's extrinsic calibration algorithm. The transformation matrix includes rotation component R and translation component T, which are then combined with the RGB-D intrinsic parameters M. -1 The pixel coordinates O of each image i (u i ,v i Convert to robotic arm coordinates P i (X i ,Y i Z i ), where the conversion formula (1) is:

[0054] P i =R×(M) -1 ×z×O i )+T (1) Where z represents the depth of the pixel in the camera coordinate system, and then the RGB image and depth image are converted into an RGB height map and a depth height map according to formula (2).

[0055]

[0056] Among them, X g Y g These represent the minimum values ​​of the robot's workspace in the X and Y axes, respectively, and res represents the resolution of the heightmap.

[0057] The prior knowledge extraction module comprises a smoothing module, a grayscale conversion module, a binarization module, a part contour detection module, and a center point extraction module, connected in sequence. The smoothing module reduces image noise in the depth-height map; the grayscale conversion module processes the image into black and white; the binarization module filters out parts using a threshold setting; the part contour detection module detects part contours; and the center point extraction module calculates the contour area and, based on that area, calculates the contour center point.

[0058] The prior knowledge extraction module's processing flow is as follows: ① The smoothing module uses a median filter to smooth the depth-height map; ② The smoothed depth-height map is converted to grayscale; ③ A binarization threshold is set based on the pixel value difference between the hole-shaped part 60, the shaft-shaped part 70, and the background, and the depth-height map is binarized to filter out the hole-shaped part 60; ④ The contour of the shaft-shaped part 70 is detected, and the contour area of ​​the shaft-shaped part 70 is calculated using the contourArea function in the OpenCV software library; Based on the contour area, the center point of the contour of the shaft-shaped part 70 is calculated using the minAreaRect function, and the center point of the contour of the shaft-shaped part 70 is output as prior knowledge.

[0059] The grasping action generation model consists of a ResNet-101 network, a two-layer analysis network (composed of a batch normalization layer, a ReLU activation function, and a 1x1 convolutional layer), an upsampling layer, a grasping judgment module, and an assembly judgment module, connected in sequence. The ResNet-101 network, as the backbone layer, extracts features from the RGB height map and depth height map respectively, obtaining a first feature map and a second feature map. The first and second feature maps are then fused to obtain a fusion result. The two-layer analysis network module processes and analyzes the fusion result. The upsampling layer outputs multiple visualized Q-value maps, where each Q-value predicts the expected future reward for the action performed at the corresponding pixel p. The grasping judgment module determines whether the conditions for the grasping task are met. The assembly judgment module determines whether the conditions for the next assembly step are met.

[0060] The processing flow of the crawling action generation model is as follows: Figure 2 As shown, ① Height map feature extraction: The generated RGB height map and depth height map are rotated 8 times at 22.5° each, and then input into a ResNet-101 network model pre-trained on ImageNet to complete the feature extraction of the RGB height map and depth height map, and the extracted features are fused; ② Feature analysis: The fused features are fed into two analysis network modules with the same structure to output the analysis results; ③ Q-value map calculation: The analysis results are input into a bilinear upsampling layer to obtain eight Q-value maps; a greedy strategy is adopted to select the maximum Q-value in each Q-value map and denoted as Q.grasp ④ Calculate the grasping action: Pre-determine the operation strategy π(a|s) t To improve the success rate of grasping and the accuracy of grasping the center point, grasping action a is generated by combining prior knowledge. t This includes the position and angle of the action, expressed by the formula:

[0061] a t ~π(a|s t ) = argmax(C(s) t )×O(s t )×Q grasp (3)

[0062] Among them, O(s) t C(s) represents the probability of the item to be grabbed, and is output by the model generated from the grabbing action; t ) represents prior knowledge. This formula is used to obtain C(s) t )×O(s t )×Q grasp The pixel with the largest value is taken as the optimal gripping position for the robotic arm, i.e., the action execution position. The rotation angle of the Q-value map at this pixel is the rotation angle of the gripper during the gripping operation, i.e., the action execution angle. ⑤ Gripping Task Judgment: To improve the success rate and accuracy of gripping, a gripping threshold is set; if gripping action a... t If the Q value of the corresponding pixel is less than the grasping threshold, it is considered that the grasping operation is not executed. The RGB height map and depth height map are re-acquired, and a new grasping action 'a' is calculated and executed. t If the grabbing action a t If the Q value of the corresponding pixel is greater than the grasping threshold, the grasping operation is considered to be met, and the robot performs the grasping action a. t ⑥ Assembly Task Judgment: Calculate the assembly threshold. In this embodiment, the number of binary mask pixels of shaft part 70 in the depth-height map is used as the assembly threshold. Take on-site images before and after the grasping action and convert them into depth-height maps. Calculate the assembly threshold using the depth-height maps. If the assembly threshold decreases or the decrease is greater than a preset range, the robot is considered to have successfully executed the grasping task, and the robot further executes the assembly action. If the assembly threshold remains unchanged or the change is less than a preset range, the robot is considered to have failed to grasp, and the robot needs to repeat the grasping action.

[0063] The assembly action generation model comprises a smoothing module, a grayscale conversion module, a binarization module, a part contour detection module, a mask output module, a PCA pose analysis module, and a PPO module, connected in sequence. The smoothing module reduces image noise in the depth-height map; the grayscale conversion module processes the image into black and white; the binarization module filters the parts to be grasped using a threshold setting; the part contour detection module detects the contours of the parts to be assembled; the mask output module fills in the pixels within the contours of the parts to be assembled, forming a binary mask; the PCA pose analysis module infers the center point and rotation angle of the parts to be assembled; and the PPO module outputs the assembly action.

[0064] The assembly action generation model processing flow is as follows: ① Generate binary mask: The input depth-height map is smoothed using a median filter, and then the image is converted to grayscale. Based on the pixel value difference between the shaft part 70, the hole part 60, and the background, a binarization threshold is set to filter the shaft part 70. Contour detection is performed on the hole part 60, and the binary mask of the hole part 60 is output based on the detected contour. ② Estimate the 3D pose of the part: Based on the binary mask, the center point and rotation angle of the hole part 60 are calculated using Principal Component Analysis (PCA). The binary mask M of the hole part 60 is represented as:

[0065] M = {(u0,v0),(u1,v1),…,(u n ,v n )} (7)

[0066] In the formula, n is the number of pixels in the mask; (u0,v0) represents the coordinates of the pixels inside the mask. Then, the center point o of the pixel mask is calculated according to formula (8), which is the two-dimensional center coordinate point (x,y) of the hole-shaped part 60 to be assembled.

[0067]

[0068] Calculate the covariance matrix according to formulas (9)(10)(11)(12); calculate the rotation angle θ of the hole-shaped part with the angle between the principal component direction and the horizontal direction based on the covariance matrix.

[0069] Res=Mo (9)

[0070] λ1,λ2=eigenvalues(covMat(res)) (10)

[0071] α1,α2=eigenvectors(covMat(res)) (11)

[0072]

[0073] In the formula, Res represents the covariance matrix; λ1 and λ2 represent different eigenvalues ​​of the covariance matrix, respectively; and α1 and α2 represent different eigenvectors of the covariance matrix, respectively.

[0074] ③ Output assembly action b t The obtained 2D center point coordinates (x, y) and rotation angle θ of the part 60 to be assembled are compared with the center point coordinates (x, y) of the end effector of the industrial robot 50. u ,y u and rotation angle θ u Correspondingly, construct the environment state s t ,

[0075] s t =(x,y,θ,x) u ,y u ,θ u (13)

[0076] s t The input is fed into a Proximal Policy Optimization (PPO) network, which passes through multiple fully connected layers. The PPO ultimately outputs the current state s. t The action distribution on, i.e., the current state s t Assembly action b at time t t The robot performs assembly action b. t .

[0077] b t ~π(b|s t (14)

[0078] The assembly action generation model extracts the center point and rotation angle of the pixels of the item to be assembled through relevant algorithms. The center point and rotation angle of the pixels of the item to be assembled are then introduced into the proximal policy optimization (PPO) network. This aims to reduce the training time and interaction data required by the policy learning algorithm and improve the assembly success rate and generalization performance.

[0079] The following steps are taken to train the grasping action generation model and the assembly action generation model respectively:

[0080] Design a heuristic function r for crawling rewards. g To evaluate the effectiveness of the crawling:

[0081] r g =GD-λΔθ (4)

[0082]

[0083] Δθ=|θ-θ′| (6)

[0084] Where G represents the result of grasping. If the shaft part 70 to be grasped is successfully grasped, a reward of 2 is given; if the grasping fails, a reward of 0 is given. D represents the distance (in cm) between the shaft part 70 to be grasped and the robot's grasping position. Δθ is the absolute error between the current angle of the shaft part 70 to be grasped and the rotation angle of the robot's gripper. λ is the heuristic coefficient, which represents the degree of influence of the agent in the process of learning angle constraints.

[0085] Input the RGB height map and depth height map into the grasping action generation model; the grasping action generation model outputs the grasping action 'a'. t By capturing the reward function r g Calculate and execute action a t The reward is calculated; based on the reward, the parameters of the grabbing action generation model are updated.

[0086] The grasping threshold is determined based on the Q-values ​​corresponding to the grasping actions generated during training. For example, the minimum or average value of the set of Q-values ​​corresponding to the grasping actions generated during training can be used as the grasping threshold.

[0087] The training process for the assembly action generation model is as follows:

[0088] Assembly reward function setting: For assembly operation tasks, this invention designs a segmented assembly reward function r t Used to evaluate the assembly effect:

[0089]

[0090] Where d t This represents the distance (in meters) between the center point of the hole-shaped part 60 to be assembled and the robot's gripping position at time t.

[0091]

[0092] d t-1 d represents the distance (in meters) between the center of the item to be assembled and the robot's gripping position at time t-1, and a represents the maximum workspace the robot exceeds. If d t <d t-1 This means that if the robot's gripper gets close to the item to be assembled, it receives a positive reward; if it is less than a certain distance, it receives a negative reward. If the distance is less than 5mm, a larger reward is given; if it is less than 2mm, the maximum reward is given, and the assembly action is performed. If dt > a, a negative reward of -1 is given, and the robot returns to its initial position.

[0093] Input the depth and height map into the assembly action generation model; the assembly action generation model outputs assembly action b. t The robot performs assembly actions; the environmental state s at time t+1 is calculated. t+1And according to the environmental state s t+1 Generate instant rewards r t Based on the reward, the parameters of the PPO network in the assembly action generation model are updated.

[0094] The segmented reward function is designed based on the robot's motion path and actual assembly. The success of the assembly depends on the relative position and orientation of the robot gripper and the shaft-shaped part 70 to be gripped, and the relative position and orientation of the robot gripper and the hole-shaped part 60 to be assembled. Therefore, the reward function is set to be segmented.

[0095] Please see Figure 3 The flowchart of the deep reinforcement learning process for robot grasping and assembly skills in this invention is shown below.

[0096] The process of the image acquisition and processing module is as follows:

[0097] Step S01: The RGB-D camera acquires RGB images and depth images;

[0098] Step S02: Process the original RGB image and depth image using a camera extrinsic calibration algorithm to convert the image into a 3D point cloud;

[0099] Step S03: Based on the transformation matrix obtained by the camera extrinsic calibration algorithm, the intrinsic parameters of RGB-D, and the depth values ​​corresponding to the pixels of the image in the camera coordinate system, convert the RGB image and depth image into an RGB height map and a depth height map using formulas.

[0100] The process of the prior knowledge extraction module is as follows:

[0101] Step S01: Obtain the depth-height map;

[0102] Step S02: Smooth the depth-height map using a median filter;

[0103] Step S03: Convert the depth height map to grayscale;

[0104] Step S04: Set a binarization threshold, perform binarization processing on the depth height map, and filter out hole-shaped parts 60;

[0105] Step S05: Detect the outline of the shaft part 70 in the depth height map;

[0106] Step S06: Calculate the contour area using the contourArea function in OpenCV, and calculate the center point of the contour using the minAreaRect function;

[0107] Step S07: Input the center point of the contour of the shaft part 70 to be grasped as prior knowledge into step S04 of the grasping action generation model.

[0108] The process of generating the model for the grabbing action is as follows:

[0109] Step S01: Input the RGB height map and depth height map into the ResNet-101 network model respectively, and extract and fuse the features of the two height maps;

[0110] Step S02: The fusion result is fed into the analysis network module, which consists of Batch Normalization, ReLU activation function and 1x1 convolution, to perform feature analysis. The analysis network module needs to perform the analysis twice before proceeding to the next step.

[0111] Step S03: Input the analysis results into the bilinear upsampling layer;

[0112] Step S04: Combining the prior knowledge of the center point of the contour of the shaft part 70 to be grasped, the bilinear upsampling layer outputs several visualized Q-value maps, and extracts the maximum Q-value of the grasping action from each Q-value map;

[0113] Step S05: Based on the relevant operation strategy, obtain the optimal action a in the direction to be gripped of the part. t a g It includes the center point and rotation angle of the shaft-shaped part 70 to be gripped.

[0114] Step S06: Determine whether the robot should execute the next grasping task based on the grasping threshold. If the maximum Q value is less than the grasping threshold, the conditions for the grasping operation task are not met, and the robot needs to return to step S04 and re-execute the grasping task. If the maximum Q value is greater than the grasping threshold, the conditions for the grasping operation task are met.

[0115] Step S07: The ROS platform receives information about the center point and rotation angle of the shaft-shaped part 70 to be grasped, and makes robot motion planning based on the information;

[0116] Step S08: The robot control cabinet establishes a communication connection with the ROS platform and receives relevant robot motion planning information;

[0117] Step S09: The robot control cabinet transmits the corresponding grasping operation task to the industrial robot 50;

[0118] Step S10: Determine whether the robot should proceed with the next assembly task based on the assembly threshold. If the threshold for gripping parts decreases, it proves that the robot has successfully performed the gripping task, and the robot can proceed with the assembly operation. If the threshold remains unchanged, it proves that the robot's gripping operation has failed, and the robot needs to repeat the gripping task.

[0119] The process of generating the model for the assembly action is as follows:

[0120] Step S01: Obtain the depth-height map;

[0121] Step S02: Smooth the depth-height map using a median filter;

[0122] Step S03: Convert the image to grayscale;

[0123] Step S04: Filter the parts from the background by setting a threshold to achieve binarization;

[0124] Step S05: Contour inspection of the 60mm profile of the hole-shaped part to be assembled;

[0125] Step S06: Output the binary mask of the hole-shaped part 60 to be assembled based on the detected contour;

[0126] Step S07: Calculate the pose information of the binary mask of the hole-shaped part to be assembled using principal component analysis (PCA). The pose information includes the coordinates of the two-dimensional center point (x,y) and the rotation angle θ.

[0127] Step S08: The obtained two-dimensional center point coordinates (x, y) and rotation angle θ of the part 60 to be assembled are compared with the center point coordinates (x, y) of the end effector of the industrial robot 50. u ,y u and rotation angle θ u Correspondingly, this forms the environmental state s t , will s t The input is fed into the Proximal Policy Optimization (PPO) network;

[0128] Step S09: Calculate the robot's current state s using the PPO algorithm. t The optimal assembly action b at time t t The robot executes the assembly trajectory planning actions according to the time step;

[0129] Step S10: The ROS platform receives assembly trajectory planning action information;

[0130] Step S11: The robot control cabinet establishes a communication connection with the ROS platform and receives relevant robot assembly trajectory planning action information;

[0131] Step S12: The robot control cabinet transmits the corresponding assembly operation task to the industrial robot 50.

[0132] It should be noted that the vision-based deep reinforcement learning method system for robot grasping and assembly skills proposed above can also be used to achieve the above-mentioned... Figure 1 The method steps corresponding to each embodiment of the vision-based robot grasping and assembly skills deep reinforcement learning method shown are not repeated here.

[0133] It should be noted that the functional units / modules in the various embodiments of the present invention can be integrated into one processing unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated into one unit / module. The integrated unit / module described above can be implemented in hardware or in the form of software functional units / modules.

[0134] From the above description of the embodiments, those skilled in the art will clearly understand that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein, or combinations thereof. For software implementation, some or all of the processes of the embodiments can be implemented by a computer program instructing the associated hardware. During implementation, the program can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a computer. Computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible to a computer.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should be able to analyze that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A vision-based deep reinforcement learning method for robot grasping and assembly skills, characterized in that, Includes the following steps: Obtain at least one image containing the item to be grabbed; The image to be processed is processed using a grasping action generation model to obtain multiple Q-value maps; based on the multiple Q-value maps, a grasping action is determined; the grasping action includes the action execution position and the action execution angle; A preset grasping threshold is set; if the Q value of the grasping action is greater than the grasping threshold, the robot will perform the grasping action. Obtain a depth-height map containing the item to be grabbed; calculate the number of binary mask pixels of the item to be grabbed in the depth-height map as the assembly threshold; The assembly threshold is calculated before and after the robot performs the grasping action. If the assembly threshold remains unchanged, the robot repeats the grasping action multiple times. If the assembly threshold decreases, the robot is considered to have completed the grasping action.

2. The vision-based deep reinforcement learning method for robot grasping and assembly skills according to claim 1, characterized in that, Also includes: The grasping action generation model is trained to generate multiple grasping actions; a grasping threshold is determined based on the Q-values ​​of the multiple grasping actions.

3. The vision-based deep reinforcement learning method for robot grasping and assembly skills according to claim 1, characterized in that, The process of using a grasping action to generate a model to process the image to be processed, resulting in multiple Q-value images, specifically: The at least one image to be processed includes an RGB height map and a depth height map; Features are extracted from the RGB height map and the depth height map respectively to obtain a first feature map and a second feature map; the first feature map and the second feature map are fused to obtain a fusion result; the fusion result is processed using an analysis network, which includes a normalization layer, an activation layer, and a convolutional layer; the analysis result is output as multiple Q-value maps using an upsampling layer; The process of determining the grasping action based on multiple Q-value maps is as follows: Obtain prior knowledge, which is the center point of the outline of the object to be grasped; based on the prior knowledge and the Q-value map, determine the grasping action, expressed by the formula: In the formula, Represents prior knowledge; This represents the probability of the item to be grabbed. This represents the maximum Q value of the plots for different Q values; This indicates the operational strategy.

4. The vision-based deep reinforcement learning method for robot grasping and assembly skills according to claim 3, characterized in that, The acquisition of prior knowledge specifically refers to: The depth-height map containing the object to be grabbed is smoothed; the smoothed depth-height map is converted to grayscale; the outline of the object to be grabbed is detected; the outline area of ​​the object to be grabbed is calculated; and the center point of the outline of the object to be grabbed is calculated.

5. The vision-based deep reinforcement learning method for robot grasping and assembly skills according to claim 1, characterized in that, Also includes: Construct a grasping reward function for training the grasping action generation model. Expressed as a formula: In the formula, G represents the grasping result; D represents the distance between the object to be grasped and the robot's end effector. The absolute error between the current rotation angle of the object to be grasped and the rotation angle of the robot's end effector; This is the heuristic coefficient.

6. The vision-based deep reinforcement learning method for robot grasping and assembly skills according to claim 1, characterized in that, Also includes: The image to be processed includes items to be captured and items to be assembled. The assembly motion generation model processes the image to be processed, and obtains the center point and rotation angle of the item to be assembled as the environmental state; the assembly motion generation model generates assembly motions based on the environmental state; the robot executes the assembly motions.

7. The vision-based deep reinforcement learning method for robot grasping and assembly skills according to claim 6, characterized in that, The process of processing the image to obtain the center point and rotation angle of the item to be assembled includes the following steps: The image to be processed is a depth-height map; image preprocessing is performed on the depth-height map; a binarization threshold is set according to the pixel value difference between the item to be grabbed, the item to be assembled, and the background, and the depth-height map is binarized; the outline of the item to be assembled is detected; a binary mask is generated according to the outline of the item to be assembled. Principal component analysis is used to calculate the center point and rotation angle of the items to be assembled based on binary masks.

8. The vision-based deep reinforcement learning method for robot grasping and assembly skills according to claim 6, characterized in that, Also includes: Construct an assembly reward function for training the assembly action generation model. Expressed as a formula: In the formula, t represents the distance between the center point of the item to be assembled and the robot's gripping position at time t; a represents the robot's maximum workspace.

Images