A step-by-step grasping detection method for service-type mobile robotic arms

Abstract

With the development of deep learning, the grasping detection of mobile manipulators has been widely developed and applied in the fields of industrial automation and household life. Most of the current grasping detection methods are aimed at a single target, and it is difficult to guarantee the grasping success rate of multimodal targets in practical applications. In response to such problems, the present invention combines and improves the target recognition algorithm (Yolov5) and the full convolution grasping detection algorithm (GDFCN), and proposes a step-by-step grasping detection method applied to a service-type mobile manipulator. In order to overcome the defect that a single grasping detection cannot distinguish between inherent and operable objects, the target detection is first performed, and then the local depth image under the recognition frame mapping is input into the grasping detection network, and the feasible robot grasping detection is represented as a diamond grasping, forming a new step-by-step cascade grasping algorithm. After experiments, this invention method can effectively complete the object classification task while meeting the real-time requirements, and enhances the stability and accuracy of grasping detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and intelligent grasping of a service-type mobile robot arm, and in particular to a step-by-step grasping and detection method for a mobile robot arm in an unmanned biological detection laboratory scenario. Background Art

[0002] With the outbreak and recurrence of the global COVID-19 pandemic, the field of biological sciences has received special attention from researchers around the world. Due to the need for a large number of manual experiments and massive data analysis, the low automation penetration rate has become the main factor limiting the industry from achieving greater efficiency. As a key area of ​​automation, robotic arms are currently in a rapid development stage. In order to adapt to different work requirements in fields such as health and safety, robotic arms must become more and more intelligent. Among them, whether service-type mobile robotic arms can autonomously complete intelligent grasping tasks has been given higher expectations and has become a key issue of concern to all parties. How to promote service-type mobile robotic arms in the field of medical services and use more service-type mobile robotic arms to replace manual labor to complete various complex biological experimental tasks has become an urgent problem to be solved. The unmanned biological testing laboratory project has also come into being. Using machines instead of people can not only shorten the detection time, but also improve the safety factor and avoid contact infection to medical staff.

[0003] In order to be able to perform good mobile tasks and grasping operations in unmanned biological testing laboratories and other untrained real-time scenarios, service mobile robots should have the following functions: (1) They should be able to perceive various objects in complex scenes, and this perception should have good real-time performance and good recognition capabilities for both static and moving objects. (2) They should calculate the grasping posture for all objects in the receptive field, distinguish between inherent objects in the scene and objects to be operated, and only grasp objects of interest. (3) They should be able to work normally in unstable dynamic environments, which mainly include: changes in the intensity of external light, constantly changing workspace scenes, sensors affected by noise and errors, and the robot's own disturbances and inaccurate control. Summary of the invention

[0004] In order to overcome the defects of the above-mentioned prior art and solve the problems of object grasping and operation faced by the service-type mobile manipulator of unmanned biological detection laboratories, the purpose of the present invention is to propose a step-by-step identification and grasping detection method applied to the service-type mobile manipulator. This is achieved through the following technical solutions:

[0005] Step S1, based on common instruments in unmanned biological detection laboratories, respectively construct data sets for target detection and grasping detection, wherein the data set for target detection is mainly color images, and the data set for grasping detection is mainly depth images.

[0006] Step S2, constructing a grasp detection neural network. The designed fully convolutional grasp detection algorithm (GDFCN) is a neural network that uses a single object depth image as the neural network input and is used to generate grasp poses in real time on the depth image.

[0007] Step S3, using the optimized Yolo v5s target detection algorithm to analyze the preprocessed RGB image, identify the objects to be captured and complete the classification, while excluding the inherent objects in the scene that should not be operated, and output the bounding box information.

[0008] In step S4, the target recognition algorithm (Yolov5) and the full convolution grasping detection algorithm (GDFCN) are combined and improved by using the mapping of the bounding box, and a real-time grasping detection algorithm for a robotic arm suitable for unfamiliar objects in untrained scenarios is proposed.

[0009] Furthermore, the specific process of step S1 is as follows:

[0010] S1-1, using Python scripts to obtain target images from various search engine image libraries and the second method is based on the real scene of an unmanned biological laboratory for manual shooting, using a combination of single shooting and combined shooting.

[0011] S1-2, based on the common grasping experience of humans, the graspable area is marked on the target to be grasped in the dataset. All points in this area are graspable points, including the same grasping angle and width. Each grasping point corresponds to a diamond grasping representation. In addition, the grasping confidence representation is added to the probability of executing the grasping.

[0012] S1-3, take photos of commonly used experimental equipment in unmanned biological testing laboratories. Predefine the experimental equipment into 20 categories. Use LabelImg to complete the labeling work. Label the data set in Yolo format.

[0013] Furthermore, the specific process of step S2 is as follows:

[0014] S2-1, in order to ensure the applicability of the object's depth information, the depth image must be filled and repaired first.

[0015] S2-2, GDFCN evaluation: The repaired and cropped depth image is input into the GDFCN neural network model. At the same time, a Gaussian filter is used to filter and preprocess the capture confidence.

[0016] S2-3, through the evaluation of GDFCN on the depth image, a dynamic grasping effect heat map is generated. The best grasping point is published in the form of a Topic under the ROS system.

[0017] Furthermore, the specific process of step S3 is as follows:

[0018] S3-1, in order to meet the requirements of the present invention that the mobile robot arm grasping system has good real-time performance and can meet the characteristics of dynamic grasping, try a variety of recognition and detection algorithms, and select the optimal algorithm Yolov5s suitable for the application of the method of the present invention through comparison of accuracy and processing frame rate.

[0019] S3-2, use the depth camera to complete the real-time acquisition of object information in the scene. In order to avoid the interference of noise in the scene, use the corresponding filtering algorithm to complete the preprocessing work. Then perform scaling and cropping processing to meet the input requirements of the Yolov5s target detection algorithm.

[0020] S3-3, use the target detection algorithm to complete the recognition and classification of target objects in the scene, and remove the inherent objects in the scene that cannot be grasped.

[0021] Furthermore, the specific process of step S4 is as follows:

[0022] S4-1, use the depth camera to complete the real-time collection of visual information of objects in the scene. In order to avoid the interference of noise in the scene, pre-processing is completed. Then scaling and cropping are performed to meet the input requirements of the Yolo v5s target detection algorithm.

[0023] S4-2 uses the optimized Yolo v5s target detection algorithm to analyze the preprocessed RGB image, identifies the objects to be grasped, and excludes the inherent objects in the scene that should not be operated by the robot. It directly generates bounding box information, including the coordinate information of the two corners of the bounding box, so as to determine the height, width and center position of the recognition box.

[0024] S4-3, select the type of target to be grasped through the service module under the ROS control system. If the target to be grasped does not appear, directly complete the grasping detection of the entire depth image, select the grasping point with the highest confidence, execute the grasping task, and complete the removal of the remaining objects until the target object appears in the field of view.

[0025] S4-4, using the result of the second stage target detection, the position information of the bounding box is mapped onto the depth image. The mapped checkbox is used to re-establish constraints for the depth image. Through the target detection label, if the constraint only contains the object of interest, it is marked as the correct bounding box, and a cropping operation is performed to generate a filtered local depth image.

[0026] S4-5, fill the local depth image and unify the size of the filtered local depth image to 256×256 pixels. Then input it into the trained network to quickly generate pixel-level grasping posture prediction on the local depth image, mainly including grasping confidence, pixel coordinates of grasping points, gripper rotation angle and grasping width.

[0027] To overcome the defect that a single grasp detection cannot distinguish between intrinsic objects and operable objects in the scene, this method first uses Yolo v5 to perform target detection. Then the local depth image mapped by the target recognition box is input into a lightweight fully convolutional neural network, and the feasible robot grasp is represented as a diamond grasp, forming a new step-by-step cascade grasping algorithm.

[0028] Advantages of the present invention:

[0029] Compared with the existing technology, the method of the present invention can assist the robot arm to achieve faster and more stable recognition and grasping operations. It can effectively complete the object classification task while meeting the real-time requirements and enhance the stability and accuracy of grasping detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Network diagram for the GDFCN upsampling phase

[0031] Figure 2 Schematic diagram of the downsampling phase of the GDFCN capture detection neural network

[0032] Figure 3 Skip connection strategy adopted for neural network

[0033] Figure 4 The working principle diagram of the step-by-step visual inspection algorithm

[0034] Figure 5 Figure 1 is a flowchart of grasping in a simulation environment.

[0035] Figure 6 This is a flowchart of the crawling based on this method in a real scenario DETAILED DESCRIPTION

[0036] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings in the specification and specific implementation methods in simulation environments and real scenarios.

[0037] Step S1, based on common instruments in unmanned biological detection laboratories, use three-dimensional sensors to construct data sets for target detection and grasping detection, the former mainly based on color images and the latter mainly based on depth images.

[0038] Step S2, construct Figure 1 and Figure 2The grasp detection neural network shown in the figure uses a single object depth image as the neural network input to generate grasp poses in real time on the depth image. Figure 3 The skip connection strategy shown.

[0039] Step S3, using the optimized Yolo v5s target detection algorithm to analyze the preprocessed RGB image, identify the objects to be captured and complete the classification, while excluding the inherent objects in the scene that should not be operated, and output the bounding box information.

[0040] In step S4, the target recognition algorithm (Yolov5) and the full convolution grasping detection algorithm (GDFCN) are combined and improved by using the mapping of the bounding box, and a real-time grasping detection algorithm for a robotic arm suitable for unfamiliar objects in untrained scenarios is proposed.

[0041] Furthermore, the specific process of step S1 is as follows:

[0042] S1-1 uses Python scripts to obtain target images from various search engine image libraries and conducts manual photography based on real scenes of unmanned biological laboratories, using a combination of single and combined photography.

[0043] S1-2, based on the common grasping experience of humans, the graspable area is marked on the target to be grasped in the dataset. All points in the area are graspable points with the same grasping angle and width. Each grasping point corresponds to a diamond grasping representation. In addition, the grasping confidence representation is added to the probability of executing the grasping.

[0044] S1-3, photograph the commonly used experimental equipment in unmanned biological testing laboratories. The experimental equipment is predefined into 20 categories. Since the data set is large, LabelImg is used to complete the labeling work in order to save efficiency. The data set is labeled in Yolo format

[0045] Furthermore, the specific process of step S2 is as follows:

[0046] S2-1, in order to ensure the applicability of the object's depth information, the depth image must be filled and repaired first.

[0047] S2-2, GDFCN evaluation inputs the repaired and cropped depth image into the GDFCN neural network model. At the same time, a Gaussian filter is used to filter the capture confidence. As a linear filtering method, its core idea is to use the Gaussian two-dimensional step function as the template coefficient in the filter for preprocessing operations.

[0048] S2-3, dynamic grasping effect heat map generated by GDFCN evaluation on depth images.

[0049] Furthermore, the specific process of step S3 is as follows:

[0050] S3-1, in order to meet the requirements of the present invention that the mobile robot arm grasping system has good real-time performance and can meet the characteristics of dynamic grasping, many kinds of recognition and detection algorithms are tried, and the optimal algorithm Yolov5s suitable for the application of the present invention is selected through comparison of accuracy and processing frame rate.

[0051] S3-2, use the depth camera to complete the real-time acquisition of object information in the scene. In order to avoid the interference of noise in the scene, use the corresponding filtering algorithm to complete the preprocessing work. Then perform scaling and cropping processing to meet the input requirements of the Yolov5s target detection algorithm.

[0052] S3-3, use the target detection algorithm to complete the recognition and classification of target objects in the scene.

[0053] Furthermore, the overall workflow of the step-by-step cascade grab detection algorithm in step S4 is as follows: Figure 4 As shown, the specific process is as follows:

[0054] S4-1, use the depth camera to complete the real-time collection of visual information of objects in the scene. In order to avoid the interference of noise in the scene, pre-processing is completed. Then scaling and cropping are performed to meet the input requirements of the Yolo v5s target detection algorithm.

[0055] S4-2, using the optimized Yolo v5s target detection algorithm to analyze the preprocessed RGB image, identify the objects to be grasped and exclude the inherent objects in the scene that should not be operated by the robot. Generate bounding box information, including the coordinate information of the two corners of the bounding box, so as to determine the height, width and center position of the recognition box.

[0056] S4-3, through the service module under the ROS control system, select the type of target to be grasped, select the grasping point with the highest confidence, execute the grasping task, and complete the removal of the remaining objects until the target object appears in the field of view. The operation process in the simulation environment is as follows Figure 5 shown.

[0057] S4-4, using the result of the second stage target detection, the position information of the bounding box is mapped onto the depth image. The mapped checkbox is used to re-establish constraints for the depth image, and the target bounding box is marked through the target detection label, and a cropping operation is performed to generate a filtered local depth image.

[0058] S4-5, fill the local depth image and unify the size of the filtered local depth image to 256×256 pixels. Then input it into the trained network to quickly generate pixel-level grasping posture prediction on the local depth image, mainly including grasping confidence, pixel coordinates of grasping points, gripper rotation angle and grasping width. The red part represents the peak point with high grasping confidence, and the green dot represents the best grasping point evaluated by the GDFCN network model. The grasping confidence of this pixel point is the highest.

[0059] S4-6, in the ROS system, outputs the location information of the point, as well as the corresponding grasping angle and width in the form of a Topic, as the optimal grasping posture in the image space for subsequent robotic arm grasping tasks. The overall robotic arm grasping process is as follows: Figure 6 shown.

Claims

1. A step-by-step grasping detection method for a service-type mobile robot arm, the overall method comprises the following steps: Step S1: According to the specific instrument configuration in the unmanned biological detection laboratory, the target detection and grasping detection data sets are constructed respectively; wherein, The dataset for object detection consists of color images, while the dataset for grasp detection consists of depth images; Step S2, constructing a grasp detection neural network. The designed fully convolutional grasp detection algorithm uses a single object depth image as the neural network input to generate pixel-level grasp poses on the depth image in real time; Step S3, using the optimized Yolov5s target detection algorithm to analyze the preprocessed RGB image, identify the objects to be captured and complete the classification while excluding the inherent objects in the scene that should not be operated, and output the bounding box information; Step S4, combining and improving the target recognition algorithm and the full convolution grasping detection algorithm by mapping the bounding box, and proposing a real-time grasping detection algorithm for a robotic arm suitable for unfamiliar objects in untrained scenarios; The step S1 is specifically as follows: S1-1, collects dataset images of target objects by using Python scripts to obtain target images from various search engine image libraries and manually shooting based on real scenes in unmanned biological laboratories; S1-2, based on human grasping experience, marks the graspable area on the target to be grasped in the dataset. All points in this area have the same grasping angle and width. Each grasping point corresponds to a diamond grasping representation. In addition, the grasping confidence representation is added to the probability of executing the grasping. S1-3, photograph the experimental equipment in the unmanned biological testing laboratory, and predefine the experimental equipment into 20 categories; use LabelImg to complete the labeling work, and label the data set in Yolo format; The step S2 is specifically as follows: S2-1, collect the depth image within the field of view through the three-dimensional sensor, and perform filling and repair to ensure the applicability of the object's depth information; S2-2, input the repaired and cropped depth image into the GDFCN neural network model, and use a Gaussian filter to filter the grasping confidence. The full convolutional neural network adopts a skip-level connection mode of 3 upsampling convolution kernels and 3 downsampling convolution kernels, and the sizes of the convolution kernels correspond to each other; S2-3, through GDFCN evaluation of the depth image, a dynamic grasping effect heat map is generated, and the best grasping point is published in the form of a Topic under the ROS system; The step S3 is specifically as follows: S3-1, in order to make the mobile robot grasping system have good real-time performance and meet the characteristics of dynamic grasping, try a variety of recognition and detection algorithms, and select the optimal algorithm Yolov5s suitable for the application by comparing the accuracy and processing frame rate; S3-2, using the depth camera to complete the real-time acquisition of object information in the scene. In order to avoid the interference of noise in the scene, the corresponding filtering algorithm is used to complete the preprocessing work, and then the scaling and cropping processing is performed to meet the input requirements of the Yolov5s target detection algorithm; S3-3, using the target detection algorithm to complete the recognition and classification of target objects in the scene, and remove the inherent objects in the scene that cannot be removed; The step S4 is specifically as follows: S4-1, using the depth camera to complete the real-time collection of visual information of objects in the scene, to avoid the interference of noise in the scene, complete preprocessing, and then perform scaling and cropping processing to meet the input requirements of the Yolov5s target detection algorithm; S4-2, using the optimized Yolov5s target detection algorithm to analyze the preprocessed RGB image, identify the objects to be grasped, exclude the inherent objects in the scene that should not be operated by the robot, and generate the coordinate information of the endpoints of the bounding box; S4-3, select the type of the target to be grasped through the service module under the ROS control system. If there is no target to be grasped, directly complete the grasping detection of the entire depth image, select the grasping point with the highest confidence, execute the grasping task, and complete the removal of the remaining objects; S4-4, using the result of target detection to map the position information of the bounding box onto the depth image, using the mapped check box to re-establish constraints for the depth image, marking the target object as a correct bounding box through the target detection label, and generating a filtered local depth image; S4-5, padding the local depth image, unifying the size of the filtered local depth image to 256×256 pixels, and then inputting it into the trained network to quickly generate pixel-level grasping posture prediction on the local depth image, including grasping confidence, pixel coordinates of grasping points, gripper rotation angle and grasping width; S4-6, outputs the position information of the point, as well as the corresponding grasping angle and width in the form of a Topic in the ROS system as the optimal grasping posture in the image space for subsequent robotic arm grasping tasks.

Images