A robot grasping method and system based on deep reinforcement learning and TSK-FS fuzzy reasoning
By combining deep reinforcement learning and TSK-FS fuzzy reasoning in the robot grasping system, the problem of low grasping efficiency in the existing technology is solved, and efficient object grasping in complex environments is achieved.
Patent Information
- Application Number
- CN202311479282.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-08
- Publication Date
- 2025-03-11
- Estimated Expiration
- 2043-11-08
AI Technical Summary
Existing object grasping methods cannot obtain optimal behavioral guidance strategies and are inefficient in grasping, especially in complex environments where unstructured, tightly stacked multi-objects.
A robot grasping method based on deep reinforcement learning and TSK-FS fuzzy inference is adopted to capture work scene images through a depth camera, build a driving action evaluation network and a grasping action evaluation network, and combine the TSK-FS fuzzy inference network to make action decisions and coordinate push and capture strategies.
It improves the grabbing efficiency of the target object, and can obtain the optimal behavioral guidance strategy in complex environments to avoid unnecessary collisions and empty grabbing of jaws.
Smart Images
Figure CN117260739B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of robot visual positioning and grasping, and specifically relates to a robot grasping method and system based on deep reinforcement learning and TSK-FS (Takagi-Sugeno-Kang Fuzzy System, TSK-FS) fuzzy reasoning. Background Art
[0002] In the field of robot operation, grasping is the main means of interacting with the real world. Robot grasping is widely used, not only to assist in completing logistics warehousing and palletizing tasks and assembly and processing work, but also to assist in completing housework and human-machine collaboration. However, in actual scenarios, grasping operations often face unstructured and unknown working environments. These complex environments bring huge challenges to the robot arms that work based on computer programs.
[0003] The skilled manipulation skills of the robot arm benefit from the coordination of grasping (grasping) and non-grasping (pushing) actions. Pushing can rearrange tightly stacked objects and provide necessary space for the robot arm and end gripper to complete the grasping action. Grasping adjusts the position of the object through the placement action, making the pushing action more accurate and avoiding collision. When there is a tightly stacked phenomenon, a single grasping strategy often causes unnecessary collisions and empty grasping due to the lack of a large enough operating space. Therefore, the combined pushing and grasping strategy has become an important idea for solving the grasping of objects in complex environments. However, a large number of current studies are devoted to solving the pushing and grasping planning problems separately, which will lead to low grasping efficiency. In addition, the traditional deep reinforcement learning model is used in planning, so the optimal behavior guidance strategy cannot be obtained. Relatively speaking, combining the pushing strategy with the grasping strategy for sequential operations is a less explored area.
[0004] Therefore, it is of great practical significance to study the goal-oriented grasping strategy of the robotic arm, especially to enable the robotic arm to quickly grasp the target object in a complex environment with unstructured objects and multiple objects tightly stacked. Summary of the invention
[0005] The purpose of the present invention is to solve the problem that the existing object grasping method cannot obtain the optimal behavior guidance strategy and has low grasping efficiency, and proposes a robot grasping method and system based on deep reinforcement learning and TSK-FS fuzzy reasoning.
[0006] The technical solution adopted by the present invention to solve the above technical problems is:
[0007] A robot grasping method based on deep reinforcement learning and TSK-FS fuzzy reasoning, the method specifically comprises the following steps:
[0008] Step 1: At time t, use the depth camera installed on the robot to capture the color image and depth image of the robot's working scene, and then pre-process the captured color image and depth image to obtain the color information map c in the robot arm base coordinate system. t , Depth Information Graph d t and the target object mask m t ;
[0009] Step 2: Construct the push action evaluation network Q P and the grasp action evaluation network Q G , the color infographic c t , Depth Information Map t and the target object mask m t As a driving action evaluation network Q P The input of the promotion action evaluation network Q P Output push action value q P , the color information map c t , Depth Information Map t and the target object mask m t As a grasp action evaluation network Q G The input of the network Q is evaluated by grasping action G Output the grab action value q G ;
[0010] Step 3: Push action value q P , grab action value q G , the target object edge object occupancy rate r O And the number of consecutive crawl failures f G As the input of TSK-FS fuzzy inference network, the decision action is outputted by using TSK-FS fuzzy inference network;
[0011] The decision action is pushing or grabbing;
[0012] Step 4: Use the decision action output by the TSK-FS fuzzy inference network as the input of the deep reinforcement learning model, and select the action that maximizes the output of the action value evaluation network of the deep reinforcement learning model;
[0013] Step 5: The controller controls the robot arm and the end effector of the robot arm according to the action output in step 4;
[0014] Step 6: Determine whether the target object has been grasped;
[0015] If the target object has been grasped, the entire grasping method ends.
[0016] If the task of grasping the target object is not completed, execute step seven;
[0017] Step 7: Determine whether the maximum number of crawling failures has been reached;
[0018] If the maximum number of crawling failures is not reached, go directly to step 8;
[0019] If the maximum number of failed grabs is reached, reset the robot and then execute step eight;
[0020] Step 8: Set t=t+1 and return to execute step 1.
[0021] Furthermore, in the step 1, the captured color image and depth image are preprocessed to obtain a color information map c in the robot arm base coordinate system. t , Depth Information Map t and the target object mask m t ; The specific process is:
[0022] Step 11: Map all the color pixel information in the color image and the depth pixel information in the depth image on the left eye of the depth camera to achieve matching between the color pixel information and the depth pixel information;
[0023] Step 1 and 2: Use pixel information and the camera intrinsic parameter matrix to obtain the 3D position of each pixel in the camera coordinate system, and then use the conversion relationship between the camera coordinate system and the robot base coordinate system to convert the 3D point cloud information to the robot base coordinate system;
[0024] Step 1: According to the working range of the end of the robot arm, the 3D point cloud information outside the working range is filtered out, and then the remaining 3D point cloud information is projected along the gravity direction to generate color information maps c t and depth information graphd t ;
[0025] Step 14: According to the color information map c t And the semantic segmentation algorithm obtains the binary mask information map m of the target object to be grasped t .
[0026] Furthermore, the pushing action evaluation network Q P The working process is:
[0027] The colorful infographic t , Depth Information Map t and the target object mask m t As driving action evaluation network Q PThe output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the first feature fusion module. The output of the first feature fusion module is used to promote the action value q P ;
[0028] The three convolution channels have the same structure, and each convolution channel includes a convolution layer;
[0029] The first feature fusion module includes 1 batch normalization layer, 2 deconvolution layers, 2 convolution layers and 1 upsampling layer, and each convolution layer is connected to a ReLu activation function layer.
[0030] Furthermore, the grasping action evaluation network Q G The working process is:
[0031] The colorful infographic t , Depth Information Map t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the second feature fusion module. The output of the second feature fusion module is used to promote the action value q P ;
[0032] The three convolution channels have the same structure, and each convolution channel includes a convolution layer;
[0033] The second feature fusion module includes two feature fusion units with the same structure and one upsampling layer, and each feature fusion unit includes one batch normalization layer and one convolution layer with a size of 1×1, and a ReLu activation function layer is connected after the convolution layer.
[0034] Furthermore, the output of the TSK-FS fuzzy inference network is:
[0035]
[0036] Among them, x is given by q P ,q G ,r O ,f G The feature vector composed of , y is the output of TSK-FS fuzzy inference network, is the normalized value of the activation strength of the kth fuzzy inference rule of the input feature vector x, f k(x) is the output of the input feature vector x under the kth fuzzy inference rule, and K is the total number of fuzzy inference rules.
[0037] Furthermore, when the output y of the TSK-FS fuzzy inference network is 0, the decision action representing the output is pushing; when the output y of the TSK-FS fuzzy inference network is 1, the decision action representing the output is grabbing.
[0038] A robot grasping system based on deep reinforcement learning and TSK-FS fuzzy reasoning, the system includes a data perception module, an action decision module, a controller and a host computer, wherein:
[0039] The data perception module is used to capture the color image and depth image facing the robot working scene, and send the captured color image and depth image to the host computer, and the data perception module and the host computer communicate using the USB interface;
[0040] The host computer preprocesses the received color image and depth image, and sends the preprocessing results to the action decision module;
[0041] The action decision module is used to output the action according to the image preprocessing result and send the output action to the host computer. The host computer plans the movement of the robot arm and the end effector of the robot arm according to the action;
[0042] The controller communicates with the host computer through TCP or IP communication protocol. The controller controls the robot arm and the end effector of the robot arm according to the motion planning result, and the robot arm and the end effector of the robot arm communicate with the controller through a bus communication protocol.
[0043] Further, the data perception module is a depth camera;
[0044] The specific process of preprocessing is:
[0045] Map the color pixel information in the color image and the depth pixel information in the depth image to the left eye of the depth camera to achieve matching between the color pixel information and the depth pixel information;
[0046] Using pixel information and the camera intrinsic parameter matrix, the 3D position of each pixel in the camera coordinate system is obtained, and then the conversion relationship between the camera coordinate system and the robot base coordinate system is used to convert the 3D point cloud information to the robot base coordinate system;
[0047] According to the working range of the end of the robot arm, the 3D point cloud information outside the working range is filtered out, and then the remaining 3D point cloud information is projected along the gravity direction to generate color information maps c t and depth information graphd t ;
[0048] According to the color infographic c t And the semantic segmentation algorithm obtains the binary mask information map m of the target object to be grasped t .
[0049] Furthermore, the action decision module is used to output an action according to the image preprocessing result, and the specific process is as follows:
[0050] The colorful infographic t , Depth Information Map t and the target object mask m t As a driving action evaluation network Q P The input of the promotion action evaluation network Q P Output push action value q P ;
[0051] The push action evaluation network Q P The working process is:
[0052] The colorful infographic t , Depth Information Map t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the first feature fusion module. The output of the first feature fusion module is used to promote the action value q P ;
[0053] The colorful infographic t , Depth Information Map t and the target object mask m t As a grasp action evaluation network Q G The input of the network Q is evaluated by grasping action G Output the grab action value q G ;
[0054] The grasping action evaluation network Q G The working process is:
[0055] The colorful infographic t , Depth Information Map t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the second feature fusion module. The output of the second feature fusion module is used to promote the action value qP ;
[0056] will push the action value q P , grab action value q G , the target object edge object occupancy rate r O And the number of consecutive crawl failures f G As the input of TSK-FS fuzzy inference network, the decision action is outputted by using TSK-FS fuzzy inference network;
[0057] The decision action output by the TSK-FS fuzzy inference network is used as the input of the deep reinforcement learning model, and the action that maximizes the output of the action value evaluation network of the deep reinforcement learning model is selected.
[0058] Furthermore, the decision action outputted by the TSK-FS fuzzy inference network is specifically:
[0059]
[0060] Among them, x is given by q P ,q G ,r O ,f G The feature vector composed of , y is the output of TSK-FS fuzzy inference network, is the normalized value of the activation strength of the kth fuzzy inference rule of the input feature vector x, f k (x) is the output of the input feature vector x under the kth fuzzy inference rule, and K is the total number of fuzzy inference rules.
[0061] The beneficial effects of the present invention are:
[0062] (1) The present invention adds Gauss noise values during the behavior strategy training phase of the deep reinforcement learning model, which enables the robotic arm to explore a wider action space and thus obtain the optimal behavior guidance strategy.
[0063] (2) The present invention can make autonomous decisions on pushing and grasping actions based on the status information in the current working scene, effectively improving the grasping efficiency of the target object.
[0064] (3) The fuzzy reasoning of the present invention utilizes the optimal driving action value q which is directly related to the decision action. P , the optimal grasping action value q G , the target object edge object occupancy rate r O and the number of consecutive crawl failures f G information, so that the decision result comprehensively considers the needs of the pushing action, which can effectively avoid unnecessary collisions of the grippers and the occurrence of empty grasping; and when the number of consecutive grasping failures f GWhen it is larger, the decision made by fuzzy reasoning is to take a pushing action, which is conducive to improving the efficiency of the action. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 This is a block diagram of the self-supervised learning process of the robot autonomous grasping system based on the DQN algorithm;
[0066] Figure 2 The network structure diagram of the push-grasp linkage strategy based on independent deep Q network and TSK-FS fuzzy reasoning;
[0067] Figure 3 The framework diagram of the push-grasp fuzzy inference network decision process;
[0068] Figure 4 This is the working flow chart of the autonomous grasping system of the robot arm;
[0069] Figure 5 Schematic diagram of the implementation method of the robot's autonomous push-grasp system device. DETAILED DESCRIPTION
[0070] Specific implementation method 1: A robot grasping method based on deep reinforcement learning and TSK-FS fuzzy reasoning described in this implementation method specifically includes the following steps:
[0071] Step 1: At time t, use the depth camera installed on the robot to capture the color image and depth image of the robot's working scene, and then pre-process the captured color image and depth image to obtain the color information map c in the robot arm base coordinate system. t , Depth Information Map t and the target object mask m t ;
[0072] Step 2: Construct the push action evaluation network Q P and the grasp action evaluation network Q G , the color information map c t , Depth Information Map t and the target object mask m t As a driving action evaluation network Q P The input of the promotion action evaluation network Q P Output push action value q P , the color information map c t , Depth Information Map t and the target object mask m t As a grasp action evaluation network Q G The input of the network Q is evaluated by grasping action G Output the grab action value q G ;
[0073] Step 3: Push action value q P , grab action value q G , the target object edge object occupancy rate r O (r O reflects the degree of crowding around the target object, so that the decision result comprehensively considers the needs of the pushing action and can effectively avoid unnecessary collisions between the grippers and the occurrence of empty grasping) and the number of consecutive grasping failures f G (refers to the number of times the grabbing action has failed after the robot is reset. G Reflects the experience of action attempts, when f G When it is larger, the decision made by TSK-FS is to take a pushing action, which is conducive to improving the efficiency of the action) as the input of the TSK-FS fuzzy reasoning network, and the TSK-FS fuzzy reasoning network is used to output the decision action;
[0074] The decision action is pushing or grasping; the decision action is output by the fuzzy inference network to coordinate and decide the category of the pushing and grasping action, so that the robotic arm can make the optimal action decision according to the state information in the scene and complete the grasping of the target object.
[0075] Step 4: Use the decision action output by the TSK-FS fuzzy inference network as the input of the deep reinforcement learning model (Deep Q-Leaning Network, DQN), and select the action that maximizes the output of the action value evaluation network of the deep reinforcement learning model, that is, Q y is the output of the action value evaluation network;
[0076] When the decision action is grasping, the autonomous grasping algorithm characterizes the "target object" grasping problem as a goal-oriented Markov decision process. The grasping action decision process is described as follows: At time t, the state of the robot arm is composed of s t =(c t ,d t |m t ), where c t For color infographic, d t is the depth information map, m t is the target object mask; the action consists of a t =π(s t |m t ), where π is the action strategy of the robot; R(s t ,a t |m t ) represents when the state in the workspace is s t When taking action a t Instant rewards received;
[0077] Step 5: The controller controls the robot's mechanical arm and the end effector of the mechanical arm according to the action output in step 4;
[0078] Step 6: Determine whether the target object has been grasped;
[0079] If the target object has been grasped, the entire grasping method ends.
[0080] If the task of grasping the target object is not completed, execute step seven;
[0081] Step 7: Determine whether the maximum number of failed grabbing attempts has been reached (referring to the maximum number of failed grabbing attempts set after the robot is reset this time. If the robot fails to grab successfully within the maximum number of failed grabbing attempts, the robot needs to be reset);
[0082] If the maximum number of crawling failures is not reached, go directly to step 8;
[0083] If the maximum number of failed grabs is reached, reset the robot and then execute step eight;
[0084] Step 8: Set t=t+1 and return to execute step 1.
[0085] like Figure 1 As shown, the "goal-oriented" robot autonomous grasping strategy in the present invention is obtained by training with the improved DQN (DeepQ-Leaning Network) algorithm, so that the robot can take a reasonable grasping posture according to the specific task objectives based on the observed state information of the object distribution in the current working scene. In the simulation environment, the grasping attempt of the target object is continuously performed according to the output value of the designed action value evaluation network, and the sample data in each grasping attempt behavior process is stored in the experience replay pool for network parameter update training, wherein the sample data includes the scene state information before the grasping action is executed, the grasping posture information, the scene state information after the grasping action is executed, and the reward information of the grasping action. Gaussian noise is added in the behavior strategy training stage, so that the grasping posture finally executed by the robot is the composite Gaussian noise value of the posture output by the DQN algorithm. The action value evaluation network model parameters are updated according to the training data set, thereby forming a self-supervised autonomous grasping learning system. With the iteration of the algorithm, the model parameters gradually converge to generate the optimal grasping action strategy.
[0086] In this implementation, a push-grasp linkage strategy is designed to independently construct a network for generating push actions and grasping actions, and a joint training method is adopted to learn an algorithm that can autonomously decide whether to take a push or grasping operation according to the current scenario. The proposed push-grasp linkage strategy algorithm is suitable for scenarios where the target object is visible in the scene, that is, scenarios that can be distinguished through a semantic segmentation network. The target environment can be a scene with dense crowds or occlusions.
[0087] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that: in the step 1, the captured color image and depth image are preprocessed to obtain a color information map c in the robot arm base coordinate system t , Depth Information Map t and the target object mask m t ; The specific process is:
[0088] Step 11: Map all the color pixel information in the color image and the depth pixel information in the depth image on the left eye of the depth camera to achieve matching between the color pixel information and the depth pixel information;
[0089] Step 1 and 2: Use pixel information and the camera intrinsic parameter matrix to obtain the 3D position of each pixel in the camera coordinate system, and then use the conversion relationship between the camera coordinate system and the robot base coordinate system to convert the 3D point cloud information to the robot base coordinate system;
[0090] Step 1: According to the working range of the end of the robot arm, the 3D point cloud information outside the working range is filtered out, and then the remaining 3D point cloud information is projected along the gravity direction to generate color information maps c t and depth information graphd t ;
[0091] Step 14: According to the color information map c t The binary mask information map m of the target object to be captured is obtained by using the Light-Weight-RefineNet semantic segmentation algorithm t .
[0092] The other steps and parameters are the same as those in the first embodiment.
[0093] Specific implementation method three: Combination Figure 2 The present embodiment is different from the first or second embodiment in that: the pushing action evaluation network Q P The working process is:
[0094] The colorful infographic t , Depth Information Map t and the target object mask m t As driving action evaluation network QP The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the first feature fusion module. The output of the first feature fusion module is used to promote the action value q P ;
[0095] The three convolution channels have the same structure, and each convolution channel includes a convolution layer;
[0096] The first feature fusion module includes 1 batch normalization layer, 2 deconvolution layers, 2 convolution layers and 1 upsampling layer, and each convolution layer is connected to a ReLu activation function layer.
[0097] The other steps and parameters are the same as those in the first or second embodiment.
[0098] Specific implementation method four: Combination Figure 2 The present embodiment is different from the first to third embodiments in that the grasping action evaluation network Q G The working process is:
[0099] The colorful infographic t , Depth Information Map t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the second feature fusion module. The output of the second feature fusion module is used to promote the action value q P ;
[0100] The three convolution channels have the same structure, and each convolution channel includes a convolution layer;
[0101] The second feature fusion module includes two feature fusion units with the same structure and one upsampling layer, and each feature fusion unit includes one batch normalization layer and one convolution layer with a size of 1×1, and a ReLu activation function layer is connected after the convolution layer.
[0102] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0103] Specific implementation method five: Combination Figure 3 The present embodiment is different from the first to fourth embodiments in that the output of the TSK-FS fuzzy inference network is:
[0104]
[0105] Among them, x is given by q P ,q G ,r O ,f G The feature vector composed of , y is the output of TSK-FS fuzzy inference network, is the normalized value of the activation strength of the kth fuzzy inference rule of the input feature vector x, f k (x) is the output of the input feature vector x under the kth fuzzy inference rule, and K is the total number of fuzzy inference rules.
[0106] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0107] Specific implementation method six: This implementation method is different from any one of specific implementation methods one to five in that: when the output y=0 of the TSK-FS fuzzy inference network, the decision action representing the output is pushing; when the output y=1 of the TSK-FS fuzzy inference network, the decision action representing the output is grabbing.
[0108] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.
[0109] Specific implementation method VII. A robot grasping system based on deep reinforcement learning and TSK-FS fuzzy reasoning described in this implementation method includes a data perception module, an action decision module, a controller and a host computer, wherein:
[0110] The data perception module is used to capture the color image and depth image facing the robot working scene, and send the captured color image and depth image to the host computer, and the data perception module and the host computer communicate using the USB interface;
[0111] The host computer preprocesses the received color image and depth image, and sends the preprocessing results to the action decision module;
[0112] The action decision module is used to output the action according to the image preprocessing result and send the output action to the host computer. The host computer plans the movement of the robot arm and the end effector of the robot arm according to the action;
[0113] The controller communicates with the host computer through TCP or IP communication protocol. The controller controls the robot arm and the end effector of the robot arm according to the motion planning result, and the robot arm and the end effector of the robot arm communicate with the controller through a bus communication protocol.
[0114] The perception function is completed by the depth camera module in the system. First, all the color information and depth information captured by the ZED camera are mapped on the left eye of the camera to achieve the matching of color pixel information and depth pixel information. Then, the 3D position of each pixel in the camera coordinate system and the corresponding color rendering information are restored using the depth information and the camera intrinsic parameter matrix. Then, using the conversion relationship between the camera coordinate system and the robot base coordinate system, the group of 3D point cloud information is converted to the robot base coordinate system. According to the set working range of the end of the robot, the 3D point cloud information outside the working range is filtered out, and the color information map and depth information with a resolution of are finally generated after projection along the direction of gravity. Combined with the Light-Weight-RefineNet semantic segmentation algorithm in the upper computer, the binary mask information map of the grasped target object can be obtained. The three constitute the state information of the robot's autonomous grasping strategy based on the deep reinforcement learning algorithm.
[0115] like Figure 4 As shown in the figure, when the decision action output by the fuzzy inference network is pushing, the robot control box issues a Move instruction. Under the control of the Move instruction, the robot moves and closes the gripper. Then the robot continues to move to the specified position to push, and then the robot resets to the initial position to complete the pushing work. When the decision action output by the fuzzy inference network is grasping, the robot control box issues a Move instruction. Under the control of the Move instruction, the robot moves and opens the gripper. Then the robot continues to move to the specified position and starts to close the gripper for grasping. Then the robot resets to the initial position to determine whether the grasping is successful. If the grasping fails, the depth camera is used to recapture the image until the fuzzy inference network outputs the grasping decision next time, and then the grasping is executed; if the grasping is successful, the robot moves to the storage box position and opens the gripper to put the grasped target object into the object box. The robot resets to the initial position, and the current target object is grasped.
[0116] Specific implementation eight: This implementation differs from specific implementation seven in that: the data perception module is a depth camera;
[0117] The specific process of preprocessing is:
[0118] Map the color pixel information in the color image and the depth pixel information in the depth image to the left eye of the depth camera to achieve matching between the color pixel information and the depth pixel information;
[0119] Using pixel information and the camera intrinsic parameter matrix, the 3D position of each pixel in the camera coordinate system is obtained, and then the conversion relationship between the camera coordinate system and the robot base coordinate system is used to convert the 3D point cloud information to the robot base coordinate system;
[0120] According to the working range of the end of the robot arm, the 3D point cloud information outside the working range is filtered out, and then the remaining 3D point cloud information is projected along the gravity direction to generate color information maps c t and depth information graphd t ;
[0121] According to the color infographic c t The binary mask information map m of the target object to be captured is obtained by using the Light-Weight-RefineNet semantic segmentation algorithm t .
[0122] The other steps and parameters are the same as those in the seventh embodiment.
[0123] Specific implementation method 9: This implementation method is different from specific implementation method 7 or 8 in that: the action decision module is used to output an action according to the image preprocessing result, and the specific process is as follows:
[0124] The colorful infographic t , Depth Information Map t and the target object mask m t As a driving action evaluation network Q P The input of the promotion action evaluation network Q P Output push action value q P ;
[0125] The push action evaluation network Q P The working process is:
[0126] The colorful infographic t , Depth Information Map t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the first feature fusion module. The output of the first feature fusion module is used to promote the action value q P ;
[0127] The colorful infographic t , Depth Information Map t and the target object mask m t As a grasp action evaluation network Q G The input of the network Q is evaluated by grasping action G Output the grab action value q G ;
[0128] The grasping action evaluation network Q G The working process is:
[0129] The colorful infographic t , Depth Information Map t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the second feature fusion module. The output of the second feature fusion module is used to promote the action value q P ;
[0130] will push the action value q P , grab action value q G , the target object edge object occupancy rate r O And the number of consecutive crawl failures f G As the input of TSK-FS fuzzy inference network, the decision action is outputted by using TSK-FS fuzzy inference network;
[0131] The decision action output by the TSK-FS fuzzy inference network is used as the input of the deep reinforcement learning model, and the action that maximizes the output of the action value evaluation network of the deep reinforcement learning model is selected.
[0132] The other steps and parameters are the same as those in the seventh or eighth specific implementation mode.
[0133] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 7 to 9 in that: the decision action output using the TSK-FS fuzzy inference network is specifically:
[0134]
[0135] Among them, x is given by q P ,q G ,r O ,f G The feature vector composed of , y is the output of TSK-FS fuzzy inference network, is the normalized value of the activation strength of the kth fuzzy inference rule of the input feature vector x, f k (x) is the output of the input feature vector x under the kth fuzzy inference rule, and K is the total number of fuzzy inference rules.
[0136] The other steps and parameters are the same as those in Specific Embodiments 7 to 9.
[0137] The robot single-arm autonomous push-grasp hardware device is mainly composed of a robotic arm with parallel grippers, a depth camera, and a Jetson development board. Figure 5As shown, the functional modules of the autonomous grasping system of the robot arm of the present invention are mainly composed of four main functional modules: perception, decision-making, planning and control. In this system, the communication between the camera and the host computer is realized through a USB interface, the robot arm and the end effector communicate with the robot arm controller through a bus communication protocol, and the robot arm controller communicates with the host computer through a TCP / IP communication protocol. The autonomous grasping strategy designed by the present invention provides the robot arm with the desired grasping posture, the desired pushing starting point, and the desired pushing direction. In this process, the movement form of the robot arm is point-to-point action. Therefore, the intermediate process of the robot arm moving from the starting state to the desired state of the algorithm decision is handed over to the hardware bottom-level motion planning and control algorithm.
[0138] The robot arm calls the "MoveJ" and "MoveL" motion instructions in the robot arm control box through the TCP / IP communication protocol to complete the motion planning and control of the robot arm from the initial state to the desired posture. The "MoveL" motion instruction is used to force the robot arm to move from the initial state to the desired posture determined by the autonomous grasping strategy designed by the present invention according to the state information in the current scene. The "MoveJ" command is used to realize the function of rotating the end gripper around the z-axis by a certain angle.
[0139] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. A robot grasping method based on deep reinforcement learning and TSK-FS fuzzy reasoning, characterized in that: The method specifically comprises the following steps: Step 1: At time t, use the depth camera installed on the robot to capture the color image and depth image of the robot's working scene, and then pre-process the captured color image and depth image to obtain the color information map c in the robot arm base coordinate system. t , Depth Information Graph d t and the target object mask m t ; Step 2: Construct the push action evaluation network Q P and the grasp action evaluation network Q G , the color infographic c t , Depth Information Graph d t and the target object mask m t As a driving action evaluation network Q P The input of the promotion action evaluation network Q P Output push action value q P , the color infographic c t , Depth Information Graph d t and the target object mask m t As a grasp action evaluation network Q G The input of the network Q is evaluated by grasping action G Output the grab action value q G ; Step 3: Push action value q P , grab action value q G , the target object edge object occupancy rate r O And the number of consecutive crawl failures f G As the input of TSK-FS fuzzy inference network, the decision action is outputted by using TSK-FS fuzzy inference network; The decision action is pushing or grabbing; Step 4: Use the decision action output by the TSK-FS fuzzy inference network as the input of the deep reinforcement learning model, and select the action that maximizes the output of the action value evaluation network of the deep reinforcement learning model; Step 5: The controller controls the robot's mechanical arm and the end effector of the mechanical arm according to the action output in step 4; Step 6: Determine whether the target object has been grasped; If the target object has been grasped, the entire grasping method ends. If the task of grasping the target object is not completed, execute step seven; Step 7: Determine whether the maximum number of crawling failures has been reached; If the maximum number of crawling failures is not reached, go directly to step 8; If the maximum number of failed grabs is reached, reset the robot and then execute step eight; Step 8: Set t=t+1 and return to execute step 1.
2. According to claim 1, a robot grasping method based on deep reinforcement learning and TSK-FS fuzzy reasoning is characterized in that: In the step 1, the captured color image and depth image are preprocessed to obtain a color information map c in the robot arm base coordinate system. t , Depth Information Graph d t and the target object mask m t ; The specific process is: Step 11: Map all the color pixel information in the color image and the depth pixel information in the depth image on the left eye of the depth camera to achieve matching between the color pixel information and the depth pixel information; Step 1 and 2: Use pixel information and the camera intrinsic parameter matrix to obtain the 3D position of each pixel in the camera coordinate system, and then use the conversion relationship between the camera coordinate system and the robot base coordinate system to convert the 3D point cloud information to the robot base coordinate system; Step 1: According to the working range of the end of the robot arm, the 3D point cloud information outside the working range is filtered out, and then the remaining 3D point cloud information is projected along the gravity direction to generate color information maps c t and depth information graphd t ; Step 14: According to the color information map c t And the semantic segmentation algorithm obtains the binary mask information map m of the target object to be grasped t .
3. The robot grasping method based on deep reinforcement learning and TSK-FS fuzzy reasoning according to claim 2 is characterized in that: The push action evaluation network Q P The working process is: The colorful infographic t , Depth Information Graph d t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the first feature fusion module. The output of the first feature fusion module is used to promote the action value q P ; The three convolution channels have the same structure, and each convolution channel includes a convolution layer; The first feature fusion module includes 1 batch normalization layer, 2 deconvolution layers, 2 convolution layers and 1 upsampling layer, and each convolution layer is connected to a ReLu activation function layer.
4. The robot grasping method based on deep reinforcement learning and TSK-FS fuzzy reasoning according to claim 3 is characterized in that: The grasping action evaluation network Q G The working process is: The colorful infographic t , Depth Information Graph d t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the second feature fusion module. The output of the second feature fusion module is used to promote the action value q P ; The three convolution channels have the same structure, and each convolution channel includes a convolution layer; The second feature fusion module includes two feature fusion units with the same structure and one upsampling layer, and each feature fusion unit includes one batch normalization layer and one convolution layer with a size of 1×1, and a ReLu activation function layer is connected after the convolution layer.
5. The robot grasping method based on deep reinforcement learning and TSK-FS fuzzy reasoning according to claim 4 is characterized in that: The output of the TSK-FS fuzzy inference network is: Among them, x is given by q P ,q G ,r O ,f G The feature vector composed of , y is the output of TSK-FS fuzzy inference network, is the normalized value of the activation strength of the kth fuzzy inference rule of the input feature vector x, f k (x) is the output of the input feature vector x under the kth fuzzy inference rule, and K is the total number of fuzzy inference rules.
6. A robot grasping method based on deep reinforcement learning and TSK-FS fuzzy reasoning according to claim 5, characterized in that: When the output y=0 of the TSK-FS fuzzy inference network, the decision action of the output is push; when the output y=1 of the TSK-FS fuzzy inference network, the decision action of the output is grab.
7. A robot grasping system based on deep reinforcement learning and TSK-FS fuzzy reasoning, characterized in that: The system includes a data perception module, an action decision module, a controller and a host computer, wherein: The data perception module is used to capture the color image and depth image facing the robot working scene, and send the captured color image and depth image to the host computer, and the data perception module and the host computer communicate using the USB interface; The host computer preprocesses the received color image and depth image, and sends the preprocessing results to the action decision module; The action decision module is used to output the action according to the image preprocessing result and send the output action to the host computer. The host computer plans the movement of the robot arm and the end effector of the robot arm according to the action; The action decision module is used to output actions according to the image preprocessing results. The specific process is as follows: The colorful infographic t , Depth Information Graph d t and the target object mask m t As a driving action evaluation network Q P The input of the promotion action evaluation network Q P Output push action value q P ; The push action evaluation network Q P The working process is: The colorful infographic t , Depth Information Graph d t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the first feature fusion module. The output of the first feature fusion module is used to promote the action value q P ; The colorful infographic t , Depth Information Graph d t and the target object mask m t As a grasp action evaluation network Q G The input of the network Q is evaluated by grasping action G Output the grab action value q G ; The grasping action evaluation network Q G The working process is: The colorful infographic t , Depth Information Graph d t and the target object mask m t As driving action evaluation network Q P The output of the three convolutional channels is then fed into the DenseNet backbone network. Finally, the output of the DenseNet backbone network is used as the input of the second feature fusion module. The output of the second feature fusion module is used to promote the action value q P ; will push the action value q P , grab action value q G , the target object edge object occupancy rate r O And the number of consecutive crawl failures f G As the input of TSK-FS fuzzy inference network, the decision action is outputted by using TSK-FS fuzzy inference network; The decision action output using the TSK-FS fuzzy inference network is specifically: Among them, x is given by q P ,q G ,r O ,f G The feature vector composed of , y is the output of TSK-FS fuzzy inference network, is the normalized value of the activation strength of the kth fuzzy inference rule of the input feature vector x, f k (x) is the output of the input feature vector x under the kth fuzzy inference rule, K is the total number of fuzzy inference rules; The decision action output by the TSK-FS fuzzy inference network is used as the input of the deep reinforcement learning model, and the action that maximizes the output of the action value evaluation network of the deep reinforcement learning model is selected; The controller communicates with the host computer through TCP or IP communication protocol. The controller controls the robot arm and the end effector of the robot arm according to the motion planning result, and the robot arm and the end effector of the robot arm communicate with the controller through a bus communication protocol.
8. The robot grasping system based on deep reinforcement learning and TSK-FS fuzzy reasoning according to claim 7, characterized in that: The data perception module is a depth camera; The specific process of preprocessing is: Map the color pixel information in the color image and the depth pixel information in the depth image to the left eye of the depth camera to achieve matching between the color pixel information and the depth pixel information; Using pixel information and the camera intrinsic parameter matrix, the 3D position of each pixel in the camera coordinate system is obtained, and then the conversion relationship between the camera coordinate system and the robot base coordinate system is used to convert the 3D point cloud information to the robot base coordinate system; According to the working range of the end of the robot arm, the 3D point cloud information outside the working range is filtered out, and then the remaining 3D point cloud information is projected along the gravity direction to generate color information maps c t and depth information graphd t ; According to the color infographic c t And the semantic segmentation algorithm obtains the binary mask information map m of the target object to be grasped t .