Ros distribution-based mechanical arm intelligent grabbing system

Abstract

The invention discloses an intelligent mechanical arm grabbing system based on ros distribution. The intelligent mechanical arm grabbing system comprises a self-made grabbed object data set; a target detection algorithm YOLOv5 is improved. A backbone network is replaced by MobileNetV3, model parameters and calculation amount are reduced, and lightweight deployment is facilitated. Conv of a neck network is replaced by group shuffling convolution, so that the complexity of the model is reduced, and meanwhile, the effect of an attention module is enhanced; a C3 module in a neck network is replaced by a cross-level part network module designed by a one-time aggregation method, so that the detection precision is maintained while the network complexity is reduced. And a parallel convolution attention module is added in front of the last detection head, so that the model is more sensitive to a target object in a target detection data set of the system. According to the grabbing system, objects can be grabbed in a classified mode in complex scenes such as stacking and disorder, and the transportability and robustness are effectively verified through experiments.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision detection and robot intelligent control, and in particular to a ROS-based distributed robotic arm intelligent grasping system. Background Art

[0002] In recent years, the development of robotics and artificial intelligence technologies in various fields has been changing with each passing day. At the same time, the national development strategy has made the application research in related fields of robots have a broader development prospect. The integration of robotics technology and artificial intelligence has become the mainstream development trend of intelligent robots at present, and has begun to emerge in the fields of manufacturing, medical care, agriculture, logistics and services.

[0003] As industrial technology continues to advance and labor costs are rising, industrial robots are gradually replacing some repetitive tasks that were originally done manually. Among the various tasks performed by robots, grasping is the most basic and important task of robots. At present, the field of robot grasping mainly adopts control methods such as offline programming and teach pendant programming. This point-to-point control method can only allow the robot to grasp specific objects at specific locations along the taught path, and cannot change the characteristics of the object and the working environment of the robot, and cannot work in messy and stacked working scenes.

[0004] In summary, a robot grasping system is needed that can stably and reliably perform classified grasping operations in the presence of multiple different objects, cluttered and stacked scenes. The present invention proposes a robotic arm intelligent grasping system based on a depth camera and a ros distributed framework, which solves the problem of classified grasping of different objects in cluttered and stacked scenes. Summary of the invention

[0005] The technical problem to be solved by the present invention is: to overcome the deficiencies in the prior art and provide a highly robust robotic arm intelligent grasping system, especially to solve the problem of multiple objects of different categories in the working area and the objects being cluttered and stacked.

[0006] The present invention is specifically implemented by the following technical solution: a ros distributed robotic arm intelligent grasping system, which specifically includes the following steps:

[0007] Step 1: Create your own object detection dataset

[0008] Step 2: Improve the YOLOv5 backbone network to MobileNetV3, reduce model parameters and computational complexity, and facilitate subsequent model deployment and real-time detection; introduce GSConv group shuffle convolution to reduce model complexity and enhance attention effect. Replace the neck C3 module with the VoVGSCSP cross-level partial network to reduce model complexity while maintaining detection accuracy. Add a parallel convolution attention module PCBM before the final detection head to make the model more sensitive to the target object; train to obtain the improved weight file.

[0009] Step 3: The host computer (PC) drives the realsense camera through the pyrealsense library to obtain the RGB image and the depth information aligned with the RGB image. Use a composite filter to process the image depth information to solve the problem of missing image depth information in stacked scenes. Calculate the three-dimensional coordinates of the center point of the image, and use the ros publisher to publish the category of the detected object, the detection accuracy, and the three-dimensional coordinates based on the camera coordinate system. Finally, use opencv to visualize the detection results.

[0010] Among them, "host computer (pc)" refers to the computer used to perform advanced tasks and process complex calculations in the control system. Software and hardware configuration: Ubuntu 20.04 operating system, deep learning framework Pytorch2.0.1, programming language Python3.10, CUDA version 11.8, CPU is 13th Gen Intel(R) Core(TM) i9-13900HX, GPU is NVIDIA GeForce RTX 4060.

[0011] Step 4: Perform hand-eye calibration.

[0012] Step 5: The robotic arm receives the topic information published by the ROS publisher, classifies and locates the objects, and automatically classifies and grasps the objects.

[0013] The step 1 specifically includes the following contents:

[0014] Use the makesense.ai online annotation website to annotate images of five types of objects suitable for Mycobot's robotic arm to grasp: gel pens, toothbrushes, lighters, mobile phone chargers, and glasses, and export the annotations in YOLO format. The training set and validation set are divided into a ratio of 8:2.

[0015] The step 2 specifically includes the following contents:

[0016] YOLOv5 (You Only Look Once version 5) is a single-stage target detection algorithm based on deep learning and is a version in the YOLO series. YOLOv5 is a widely used and efficient target detection algorithm in this technical field, which can achieve real-time detection while maintaining high detection accuracy. MobileNetV3 is a lightweight convolutional neural network architecture developed by Google for efficient image classification and target detection. It is the third generation model of the MobileNet series.

[0017] Replace the YOLOv5 backbone network with MobileNetV3. Add the MobileNetV3 definition in common.py. Improve the network model. The first module is the first four layers of the MobileNetV3 backbone network, which are eight times downsampling. The fifth to ninth layers are the second module, which are sixteen times downsampling. The tenth to the last layer are the third module, which are thirty-two times downsampling. Modify YOLOv5s.yaml. In backone, according to the MobileNetV3 network structure, change the output channel of the first module to 24, the output channel of the second module to 48, and the output channel of the third module to 576. Modify the layer splicing in the head. For example, the previous layer was originally spliced ​​with the sixth layer, and this part was sixteen times downsampling. The corresponding backone is the 0th layer, which is changed to the upper layer and the 0th layer. Since the original backone has 10 layers and is changed to 3 layers, the layer splicing operations in the head must be reduced by 7 on the original basis. Then register the MobileNetV3 network in the yolo file.

[0018] Introduce the GSConv module. GSConv is a hybrid convolution that uses standard convolution, depthwise separable convolution, and shuffle. Depthwise separable convolution can reduce the number of parameters in the module, but it will reduce the classification accuracy of the model. GSConv uses a uniform mixing strategy of shuffling to uniformly infiltrate the local feature information between different channels generated by standard convolution into the output features generated by depthwise separable convolution, with low implementation complexity. It can reduce the complexity of calculation and the number of parameters while maintaining the detection accuracy of the model. The GSConv network structure is as follows: First, the input feature map is subjected to a downsampled standard convolution to obtain feature map A, and then A is subjected to a depthwise separable convolution to obtain B. A and B are concatenated together and then shuffled to obtain the output feature map by juxtaposing the corresponding channels of the feature maps obtained by standard convolution and depthwise separable convolution. Add the GSConv module in the common file, register it in the yolo file, and finally replace Conv with GSConv in the yaml file.

[0019] Introduce the VoVGSCSP module. VoVGSCSP is a cross-level partial network module designed with a one-time aggregation method. The present invention adopts the most cost-effective VoVGSCSP1 as the cross-level partial network. Its structure is as follows: the input feature map is subjected to standard convolution to obtain the output x1, x1 is subjected to a bottleneck structure composed of GSConv and standard Conv to obtain the output x2, x1 and x2 are spliced ​​and then subjected to a standard convolution to obtain the final output. VoVGSCSP realizes feature fusion by splicing x1 and x2, which helps to retain more original information and feature expression capabilities; the bottleneck structure adopts two convolution paths, which is conducive to multi-scale feature extraction of the network. Add the VoVGSCSP module in the common file, register it in the yolo file, and finally replace C3 with VoVGSCSP in the yaml file.

[0020] Introduce the second improved parallel convolutional attention module PCBAM. In CBAM, channel attention and spatial attention are connected in series, and channel attention will affect the feature extraction of spatial attention. Therefore, this improved network adopts the PCBAM attention structure, which is composed of parallel channel attention and spatial attention. Parallel channel attention and spatial attention extract features from the input feature map at the same time, which strengthens the global feature extraction ability of spatial attention. The target detection data set of this system is mostly large target objects. In order to enhance the sensitivity of the model to the target object, add the PCBAM attention module before the last detection head of the YOLOv5s network. Add the PCBAM module in the common file, register it in the yolo file, and finally add the PCBAM module before the last detection head in the yaml file.

[0021] The step 3 composite filter sequentially uses four filters, extraction, space, time, and hole filling, to fill the depth holes. Note that the application of the extraction filter here will reduce the image resolution and improve the accuracy of the depth information of the detected object.

[0022] The three-dimensional coordinates of the center point of the object in step 3, the YOLO target detection algorithm can directly identify and locate the object. The position of the detection frame can be represented by the pixel coordinates of the upper left corner of the detection frame (x1, y1) and the coordinates of the lower right corner (x2, y2). Then the coordinates of the center point of the object are ((x2-x1) / 2, (y2-y1) / 2), combined with pyrealsense to obtain the depth flow information of the point, the three-dimensional coordinates of the object in the camera coordinate system can be obtained.

[0023] The step 4 specifically includes the following contents:

[0024] The hand-eye calibration of the present invention is based on the handeye-calib package and adopts the eye-outside-hand calibration method. First, print the Aruco calibration plate from the Rruco official website. The eye-outside-hand hand-eye calibration requires the position of the calibration plate in the camera and the position of the end of the robotic arm based on the base. They are all topics that publish the geometry_msg / PointStamped data format. The position information of the calibration plate in the camera can be directly obtained from the ros topic published by the realsense camera, and the position of the end of the robotic arm based on the base needs to be obtained by the Mycobot's own api get_coords. The gripper posture obtained by get_coords is the posture information in the AngleStatues format, and its posture is expressed in Euler angles. It is changed into posture data expressed in quaternions, and the published information format is changed to geometry_msgs / PoseStamped. The position of the gripper was changed to obtain multiple sets of experimental data. The Tsai-Lenz, Park, Horaud and Danielidis algorithms were used to solve the hand-eye calibration transformation matrix. The set with the smallest standard deviation of the hand-eye calibration results was used for subsequent experiments.

[0025] The step 5 specifically includes the following contents:

[0026] The lower computer (robotic arm controller) creates a ROS subscriber and subscribes to the information published by the upper computer PC, including object category, accuracy, and three-dimensional coordinates of the object in the camera coordinate system. The transformation matrix obtained by hand-eye calibration transforms the three-dimensional coordinates of the object in the camera coordinate system into the three-dimensional coordinates based on the coordinate system of the end of the robotic arm. After obtaining the transformation matrix, it cannot be directly used to grasp the object because the end coordinate system is not equivalent to the gripper coordinate system. There is no degree of freedom between the gripper and the end of the robotic arm, and the gripper coordinate system can be determined according to the end coordinate system. Let the end coordinate system be represented by x, y, and z. Through measurement, the gripper coordinate system can be obtained as x, y+6.7cm, z+0.4cm. The three-dimensional coordinates of the object in the gripper coordinate system are obtained, which can be directly used for subsequent grasping operations. At the beginning of the grasping process, the robotic arm is at the origin and the gripper is opened. When the subscriber subscribes to the object information, the inverse kinematics solution is used, and the gripper of the robotic arm reaches the position above the object and descends to grasp the object, and grasps it to different positions according to the different category labels corresponding to the object. At this time, the gripper is in the open state. After completing the classification and grasping of an object, the robot arm gripper reaches the working area to realize the classification and grasping of the next object until the working area detects no objects to be classified, completing the entire grasping process. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of intelligent crawling according to an embodiment of the present invention.

[0028] Figure 2 This is the GSConv module image of an embodiment of the present invention.

[0029] Figure 3 This is the network module image of the VoVGSCSP cross-level part according to an embodiment of the present invention.

[0030] Figure 4 The second improved PCBAM parallel convolution attention module image of the embodiment of the present invention

[0031] Figure 5 This is an improved YOLOv5s network structure image of an embodiment of the present invention.

[0032] Figure 6 This is an object three-dimensional coordinate detection image according to an embodiment of the present invention.

[0033] Figure 7 The object information image is published by the ROS publisher of the embodiment of the present invention.

[0034] Figure 8 The hand-eye calibration is a relationship graph between each ROS node and the topic according to an embodiment of the present invention.

[0035] Fig. 9 This is an image of the calibration result of the Park algorithm with the smallest standard deviation of the hand-eye calibration according to an embodiment of the present invention.

[0036] Fig.10 These are three-view images of a two-dimensional model of a robotic arm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, the robot arm intelligent grasping system of the embodiment of the present invention includes the following steps:

[0039] Step 1: Create your own target detection dataset, which includes the following:

[0040] ; The threshold of the gripper switch of the MyCobot 280pi robot arm is 0-4.5cm, so this system uses five types of target objects: gel pen, mobile phone charger, toothbrush, glasses and lighter. 2487 dataset images were collected, about 500 images of each category. In order to improve the robustness of the training model, the dataset contains multiple dataset images of complex scenes including multiple target objects. Use the makesense.ai online annotation website to annotate the images. Divide the dataset into training set and validation set in a ratio of 8:2.

[0041] Step 2: Improve the YOLOv5 backbone network to MobileNetV3, reduce model parameters and computational complexity, and facilitate subsequent model deployment and real-time detection; introduce GSConv group shuffle convolution to reduce model complexity and enhance attention effect. Replace the neck C3 module with the VoVGSCSP cross-level partial network to reduce model complexity while maintaining detection accuracy. Add a parallel convolution attention module PCBM before the final detection head to make the model more sensitive to the target object. Train to obtain the improved weight file. Specifically include the following: Replace the YOLOv5 backbone network with MobileNetV3. Add the MobileNetV3 definition in common.py. Improve the network model. The first module is the first four layers of the backbone network of MobileNetV3, which is eight times downsampling. The fifth to ninth layers are used as the second module, which is sixteen times downsampling. The tenth to the last layer are used as the third module, which is thirty-two times downsampling. Modify YOLOv5s.yaml. In backone, according to the MobileNetV3 network structure, the output channel of the first module is changed to 24, the output channel of the second module is changed to 48, and the output channel of the third module is changed to 576. Modify the layer splicing in the head. For example, the previous layer is spliced ​​with the sixth layer. This part is downsampled 16 times. The corresponding modified backone is the 0th layer. Change it to splicing the upper layer with the 0th layer. Since the original backone has 10 layers and is modified to 3 layers, the layer splicing operations in the head must be reduced by 7 on the original basis. Then register the MobileNetV3 network in the yolo file.

[0042] Introduce the GSConv module. GSConv is a hybrid convolution that uses standard convolution, depthwise separable convolution, and shuffling. Depthwise separable convolution can reduce the number of parameters in the module, but it will reduce the classification accuracy of the model. GSConv uses a shuffling uniform mixing strategy to evenly infiltrate the local feature information between different channels generated by standard convolution into the output features generated by depthwise separable convolution. It has low implementation complexity and can reduce the complexity of calculation and the number of parameters while maintaining the detection accuracy of the model. The GSConv network structure is as follows: First, the input feature map is subjected to a downsampled standard convolution to obtain feature map A, and then A is subjected to a depthwise separable convolution to obtain B. A and B are spliced ​​together and then shuffled to align the corresponding channels of the feature maps obtained by standard convolution and depthwise separable convolution to obtain the output feature map. The network structure of the GSConv module is as follows: Figure 2 Add the GSConv module to the common file, register it in the yolo file, and finally replace Conv with GSConv in the yaml file.

[0043] Introduce the VoVGSCSP module. VoVGSCSP is a cross-level partial network module designed with a one-time aggregation method. The present invention adopts the most cost-effective VoVGSCSP1 as the cross-level partial network. Its structure is as follows: the input feature map undergoes standard convolution to obtain the output x1, x1 passes through a bottleneck structure composed of GSConv and standard Conv to obtain the output x2, x1 and x2 are spliced ​​and then pass through a standard convolution to obtain the final output. VoVGSCSP achieves feature fusion by splicing x1 and x2, which helps to retain more original information and feature expression capabilities; the bottleneck structure uses two convolution paths, which is conducive to multi-scale feature extraction of the network. The VoVGSCSP network structure is as follows Figure 3 Add the VoVGSCSP module to the common file, register it in the yolo file, and finally replace C3 with VoVGSCSP in the yaml file.

[0044] The second improved parallel convolutional attention module PCBAM is introduced. In CBAM, channel attention and spatial attention are connected in series, and channel attention will affect the feature extraction of spatial attention. Therefore, this improved network adopts the PCBAM attention structure, which is composed of parallel channel attention and spatial attention. Parallel channel attention and spatial attention extract features from the input feature map at the same time, which strengthens the global feature extraction ability of spatial attention. The target detection data set of this system is mostly large target objects. In order to enhance the sensitivity of the model to the target object, a PCBAM attention module is added before the last detection head of the YOLOv5s network. The PCBAM network structure is as follows Figure 4As shown. Add the PCBAM module in the common file, register it in the yolo file, and finally add the PCBAM module before the last detection head in the yaml file.

[0045] In summary, the improved target detection network is as follows Figure 5 As shown. In order to verify the effectiveness of each added module of the improved network, relevant ablation experiments are carried out. The results of the improved target detection algorithm after ablation experiments are shown in Table 1. Precision represents the precision, Recall represents the recall rate, mAP@0.5 represents the average detection accuracy when the IOU threshold between the predicted box and the true box is 50%, and Parameters represents the parameter amount. The amount of calculation (GFLOPs) represents giga floating point operations per second, which is used to measure the computing performance index of the computer system and represents the number of operations performed per second. It can be seen that the improved algorithm has an accuracy of 0.6%, a recall rate of 3.1%, and a mAP@50 of 0.9% higher than the original YOLOv5s. The number of parameters has decreased by 49.6%, and the amount of calculation has decreased by 65.8%, which can achieve the target recognition detection task and the lightweight deployment of the model faster and better.

[0046] Table 1 Ablation experiment of improved target detection algorithm

[0047]

[0048] To further verify the performance of the improved YOLOv5s target detection algorithm on the data set of this system, it is compared with the original YOLOv5s and other common target detection algorithms. The results of the comparative test are shown in Table 2. Since the training computation of YOLOv7 is too large, it is deployed on the server for training. According to Table 2, the improved target detection algorithm based on the YOLOv5s baseline model is better than the YOLOv3-tiny, YOLOv5s and YOLOv7 algorithms. Although the number of parameters and computation of YOLOv5n is smaller than that of other algorithms, its lower precision, recall rate and mAP@50 index make it not dominant in the model comparison. The precision of the improved algorithm is 0.1% and 0.7% lower than that of YOLOv7-tiny and YOLOv5m, respectively, but the recall rate is 2.7% and 4.1% higher than that of the two, respectively. This is reasonable from the analysis of the constraint relationship between precision and recall, and the mAP@50 of this algorithm is higher than the two, with parameters of 58.7% and 12.9% of the two, and computational complexity of 40.9% and 11.3% of the two. The improved algorithm basically maintains the detection accuracy while the model has smaller parameters and computational complexity, which is more advantageous in the lightweight deployment of the model and more conducive to real-time detection of robots and implementation of classification and grasping tasks.

[0049] Table 2 Comparative experiment of improved target detection algorithm

[0050]

[0051] Step 3: The host computer (PC) drives the realsense camera through the pyrealsense library to obtain the RGB image and the depth information aligned with the RGB image. The image depth information is processed with a composite filter to solve the problem of missing image depth information. The three-dimensional coordinates of the center point of the image are calculated, and the category of the detected object, the detection accuracy, and the three-dimensional coordinates based on the camera coordinate system are published by the ros publisher. Finally, the detection results are visualized using opencv.

[0052] Furthermore, the composite filter sequentially uses four filters, namely, extraction, space, time, and hole filling, to fill the depth holes. Note that the application of the extraction filter here will reduce the image resolution and improve the accuracy of the depth information of the detected object. The three-dimensional coordinates of the center point of the object, the YOLO target detection algorithm can directly identify and locate the object. The position of the detection frame can be represented by the pixel coordinates of the upper left corner of the detection frame (x1, y1) and the coordinates of the lower right corner (x2, y2). Then the coordinates of the center point of the object are ((x2-x1) / 2, (y2-y1) / 2). Combined with pyrealsense to obtain the depth flow information of the point, the three-dimensional coordinates of the object at the point in the camera coordinate system can be obtained.

[0053] 3D object information detection visualization Figure 6 As shown, it includes the object category, detection accuracy, and the object's 3D coordinates in the camera coordinate system. The object category, detection accuracy, and the object's 3D coordinates in the camera coordinate system form a custom message type. Create a topic for the ros publisher to publish the custom message, which is used for the subsequent robot arm to create a subscriber to subscribe to the topic for grasping operations. The content of the object message published by the ros publisher is as follows Figure 7 shown.

[0054] Step 4 performs hand-eye calibration.

[0055] Specifically include the following:

[0056] The hand-eye calibration in this embodiment is based on the handeye-calib package, which adopts the eye-outside-hand calibration method. First, print the Aruco calibration plate from the Rruco official website. The eye-outside-hand hand-eye calibration requires the position of the calibration plate in the camera and the position of the end of the robotic arm based on the base. In order to unify the topics of both, the geometry_msg / PointStamped data format is used. The pose information of the calibration plate in the camera can be directly obtained from the ros topic published by the realsense camera. The pose of the end of the robotic arm based on the base needs to be obtained by the Myc obot's own api get_coords. The gripper pose obtained by get_coords is the pose information in the AngleStatues format, and its pose is expressed in Euler angles. It is changed to pose data expressed in quaternions, and the published information format is changed to geometry_msgs / PoseStamped. The ROS node topic relationship of the calibration program is as follows: Figure 8 As shown. / camera / reaslsense2_camera_manager is the ROS driver node of the RealSense camera. The topic / camera / reaslsense2_camera_manager / bond can increase the stability of the camera's communication with other nodes. First, the / aruco_single node subscribes to the internal reference data and color image of reaslsense2_camera, and converts the pose of the Arcuo code in the image coordinate system to the pose in the camera coordinate system through TF coordinate transformation. Then the / aruco_single node publishes two topics, / aru-co_single / result is used to visualize the pose of the Arcuo code in the camera coordinate system, and / aruco_single / pose is used for hand-eye calibration experiments. The node grab_pose_publisher publishes the pose message / grab_pose of the end of the robot based on the base. The node / online_hand_on_ere_calib subscribes to / aruco_single / pose and / grab_pose for hand-eye calibration. Since the eye is outside the hand, hand-eye calibration requires the position of the calibration plate in the camera and the position of the end of the robot based on the base, so the camera position and the position of the robot base must be fixed. Use the gripper to clamp the calibration plate and run the calibration program. At this time, the subscriber in the calibration program subscribes to the position of the calibration plate in the camera and the position of the end of the robot based on the base for calibration. In order to improve the hand-eye calibration effect, the robot arm's built-in slider control program is used to control the movement of the robot arm joints so that the calibration plate moves up and down, left and right, upper left, upper right, lower left, and lower right in the camera. Three groups of data are taken for a total of 24 groups for iterative calculation to solve the hand-eye calibration matrix. The position of the camera coordinate system under the robot base obtained by the Horaud algorithm with the smallest standard deviation in the calibration result is as follows Figure 8 shown.

[0057] The gripper posture was changed to obtain multiple sets of experimental data, and the four algorithms of Tsai-Lenz, Park, Horaud and Daniilidis were used to solve the hand-eye calibration transformation matrix. The group with the smallest standard deviation of the hand-eye calibration result was used for subsequent experiments. The posture calibration result was obtained in the form of quaternion (w, x, y, z), which was converted into a hand-eye homogeneous matrix as follows:

[0058]

[0059] Step 5: The robot receives the topic information published by the ROS publisher, classifies and locates the objects, and automatically classifies and grasps the objects.

[0060] The step 5 specifically includes the following contents:

[0061] The lower computer (robotic arm controller) creates a ros subscriber and subscribes to the information published by the upper computer, including object category, accuracy, and three-dimensional coordinates of the object in the camera coordinate system. The transformation matrix obtained by hand-eye calibration transforms the three-dimensional coordinates of the object in the camera coordinate system into the three-dimensional coordinates based on the coordinate system of the end of the robotic arm. After obtaining the transformation matrix, it cannot be directly used to grasp the object because the end coordinate system is not equivalent to the gripper coordinate system. There is no degree of freedom between the gripper and the end of the robotic arm, and the gripper coordinate system can be determined according to the end coordinate system. Let the end coordinate system be represented by x, y, and z. Through measurement, the gripper coordinate system can be obtained as x, y+6.7cm, z+0.4cm. The three-dimensional coordinates of the object in the gripper coordinate system are obtained, which can be directly used for subsequent grasping operations. At the beginning of the grasping process, the robotic arm is at the origin and the gripper is opened. When the subscriber subscribes to the object information, the inverse kinematics solution is used, and the gripper of the robotic arm reaches the position above the object and descends to grasp the object, and grasps it to different positions according to the different category labels corresponding to the object. At this time, the gripper is in the open state. After completing the classification and grasping of an object, the robot arm gripper reaches the work area to realize the classification and grasping of the next object until the work area detects no more objects to be classified, completing the entire grasping process. Figure 1 shown.

[0062] Since the calibration result obtained by the hand-eye calibration part is expressed as a quaternion (w, x, y, z), it is necessary to convert the quaternion into a rotation matrix, and then use the rotation matrix and translation matrix to obtain the homogeneous matrix from the camera coordinate system to the robot end coordinate system to realize the robot's grasping operation. The formula for converting the quaternion to the rotation matrix is ​​as follows:

[0063]

[0064] The camera coordinate system is A, the robot end effector (gripper) coordinate system is B, and the hand-eye calibration homogeneous matrix is ​​X, AX = XB, A = XBX -1 Thus, the three-dimensional coordinates of the object in the camera coordinate system are converted into the three-dimensional coordinates in the robot end coordinate system.

[0065] The gripper used in the grasping experiment is a two-finger gripper installed at the end of the robot arm. The coordinate system of the end of the robot arm is not directly equivalent to the gripper coordinate system. There is no degree of freedom between the two by screw installation, so the relative distance from the end flange to the center point of the two-finger gripper can be measured to convert the three-dimensional coordinates of the object in the coordinate system of the end of the robot arm into the three-dimensional coordinates of the end effector. Fig.10 As shown, the jaw coordinate system can be measured by measuring the installation relationship between the jaw and the end as x, y+6.7cm, z+0.4cm.

[0066] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A ros-based distributed robotic arm intelligent grasping system, characterized in that: The specific steps include: Step 1: Create your own target detection dataset; Step 2: Improve the YOLOv5 backbone network to MobileNetV3, reduce model parameters and computational complexity, and facilitate subsequent model deployment and real-time detection; introduce GSConv group shuffle convolution to reduce model complexity and enhance attention effect; The neck C3 module is replaced with the VoVGSCSP cross-level partial network to reduce the model complexity while maintaining the detection accuracy; a parallel convolutional attention module PCBM is added before the final detection head to make the model more sensitive to the target object; The improved weight file is obtained after training; Step 3: The host computer (PC) drives the realsense camera through the pyrealsense library to obtain the RGB image and the depth information aligned with the RGB image; Use composite filters to process image depth information to solve the problem of missing image depth information in stacking scenes; Calculate the three-dimensional coordinates of the center point of the image, and use the ros publisher to publish the category of the detected object, the detection accuracy, and the three-dimensional coordinates based on the camera coordinate system; Step 4: Perform hand-eye calibration; Step 5: The robotic arm receives the topic information published by the ROS publisher, classifies and locates the objects, and automatically classifies and grasps the objects.

2. According to the ROS distributed robotic arm intelligent grasping system according to claim 1, it is characterized in that: The specific contents of step 1 are as follows: Use the makesense.ai online annotation website to annotate images of five types of objects suitable for the Mycobot robotic arm to grasp: gel pens, toothbrushes, lighters, mobile phone chargers, and glasses. Export the annotations in YOLO format; and divide the training set and validation set into a ratio of 8:

2.

3. According to the ROS distributed robotic arm intelligent grasping system of claim 1, it is characterized in that: The specific contents of step 2 are as follows: Replace the YOLOv5 backbone network with MobileNetV3; add the MobileNetV3 definition in common.py; improve the network model. The first module is the first four layers of the MobileNetV3 backbone network, which is eight times downsampling. The fifth to ninth layers are used as the second module, which are sixteen times downsampling. The tenth to the last layer are used as the third module, which are thirty-two times downsampling; modify YOLOv5s.yaml; in backone, according to the MobileNetV3 network structure, change the output channel of the first module to 24, the output channel of the second module to 48, and the output channel of the third module to 576; modify the layer splicing in the head, for example, the original layer was spliced ​​with the sixth layer, which was sixteen times downsampling, and the modified backone was the 0th layer, which was changed to the upper layer and the 0th layer splicing; because the original backone had 10 layers and was changed to 3 layers, the layer splicing operations in the head should be reduced by 7 on the original basis; then register the MobileNetV3 network in the yolo file; Introduce GSConv module; GSConv is a hybrid convolution that uses standard convolution, depthwise separable convolution and shuffling; depthwise separable convolution can reduce the number of parameters in the module, but it will reduce the classification accuracy of the model; GSConv uses a uniform mixing strategy of shuffling to evenly infiltrate the local feature information between different channels generated by standard convolution into the output features generated by depthwise separable convolution, with low implementation complexity. It can reduce the complexity of calculation and the number of parameters while maintaining the detection accuracy of the model; The network structure of GSConv is as follows: First, input the feature map and perform a downsampled standard convolution to obtain feature map A, then A is subjected to a depthwise separable convolution to obtain B, A and B are spliced ​​together and then shuffled to obtain the output feature map by juxtaposing the corresponding channels of the feature maps obtained by standard convolution and depthwise separable convolution; Add the GSConv module in the common file, register it in the yolo file, and finally replace Conv with GSConv in the yaml file; Introducing the VoVGSCSP module; VoVGSCSP is a cross-level partial network module designed with a one-time aggregation method. The present invention adopts the most cost-effective VoVGSCSP1 as the cross-level partial network; Its structure is as follows: the input feature map is subjected to standard convolution to obtain output x1, x1 is subjected to a bottleneck structure composed of GSConv and standard Conv to obtain output x2, x1 and x2 are concatenated and then subjected to a standard convolution to obtain the final output; VoVGSCSP realizes feature fusion by concatenating x1 and x2, which helps to retain more original information and feature expression capabilities; the bottleneck structure adopts two convolution paths, which is conducive to multi-scale feature extraction of the network; add the VoVGSCSP module in the common file, register it in the yolo file, and finally replace C3 with VoVGSCSP in the yaml file; Introducing the second improved parallel convolutional attention module PCBAM; In CBAM, channel attention and spatial attention are concatenated in series, and channel attention affects the feature extraction of spatial attention; Therefore, this improved network adopts the PCBAM attention structure, which is composed of parallel channel attention and spatial attention. The parallel channel attention and spatial attention extract features from the input feature map at the same time, which strengthens the global feature extraction ability of spatial attention. The target detection data set of this system is mostly large target objects. In order to enhance the sensitivity of the model to the target object, the PCBAM attention module is added before the last detection head of the YOLOv5s network. Add the PCBAM module in the common file, register it in the yolo file, and finally add the PCBAM module before the last detection head in the yaml file.

4. According to the ROS distributed robotic arm intelligent grasping system according to claim 1, it is characterized in that: The step 3 composite filter sequentially uses four filters, namely extraction, space, time, and hole filling, to fill the depth holes. Note that the application of the extraction filter here will reduce the image resolution and improve the accuracy of the depth information of the detected object.

5. According to the ROS distributed robotic arm intelligent grasping system according to claim 1, it is characterized in that: The three-dimensional coordinates of the center point of the object in step 3, the YOLO target detection algorithm can directly identify and locate the object; the position of the detection frame can be represented by the pixel coordinates of the upper left corner of the detection frame (x1, y1) and the lower right corner coordinates (x2, y2); then the coordinates of the center point of the object are ((x2-x1) / 2, (y2-y1) / 2), combined with pyrealsense to obtain the depth flow information of the point, the three-dimensional coordinates of the object in the camera coordinate system can be obtained.

6. According to the ROS distributed robotic arm intelligent grasping system according to claim 1, it is characterized in that: The specific contents of step 4 are as follows: The hand-eye calibration of the present invention is based on the handeye-calib package, and adopts the eye-outside-hand calibration method; first, print the Aruco calibration plate from the Rruco official website; the eye-outside-hand hand-eye calibration requires the calibration plate position in the camera and the position of the end of the robot arm based on the base; they are all topics that publish the geometry_msg / PointStamped data format; The pose information of the calibration plate in the camera can be directly obtained from the ros topic published by the realsense camera. The pose of the end of the robotic arm based on the base needs to be obtained by the Mycobot's own api get_coords; the gripper pose obtained by get_coords is pose information in the AngleStatues format, and its pose is expressed in Euler angles. It is changed into pose data expressed in quaternions, and the published information format is changed to geometry_msgs / PoseStamped; the gripper pose is changed to obtain multiple sets of experimental data, and the four algorithms of Tsai-Lenz, Park, Horaud and Danielidis are used to solve the hand-eye calibration transformation matrix, and the set with the smallest standard deviation of the hand-eye calibration results is used for subsequent experiments.

7. According to the ROS distributed robotic arm intelligent grasping system of claim 1, it is characterized in that: The specific contents of step 5 are as follows: The lower computer (robotic arm controller) creates a ros subscriber and subscribes to the data published by the upper computer, including object category, accuracy and 3D coordinates of the object in the camera coordinate system; the transformation matrix obtained by hand-eye calibration transforms the 3D coordinates of the object in the camera coordinate system into 3D coordinates based on the end coordinate system of the robotic arm; after obtaining the transformation matrix, it cannot be directly used to grasp the object because the end coordinate system is not equivalent to the gripper coordinate system; there is no degree of freedom between the gripper and the end of the robotic arm, and the gripper coordinate system can be determined based on the end coordinate system; let the end coordinate system be represented by x, y, and z, and the gripper coordinate system can be obtained by measurement as x , y+6.7cm, z+0.4cm; get the three-dimensional coordinates of the object in the gripper coordinate system, which can be directly used for subsequent grasping operations; the grasping process starts, the robot arm is at the origin, and the gripper is opened; when the subscriber subscribes to the object information, through inverse kinematics solution, the robot arm gripper reaches the position above the object and descends to grip the object, and grabs it to different positions according to the different category labels corresponding to the object; at this time, the gripper is in an open state; after completing the classified grasping of an object, the robot arm gripper reaches the working area to realize the classified grasping of the next object, until the working area detects no objects to be classified, and the entire grasping process is completed.

Images