Method for using coal and gangue rapid sorting mechanical arm system based on improved mask-rcnn algorithm

Abstract

The application discloses a use method of a coal and gangue rapid sorting mechanical arm system based on an improved mask-rcnn algorithm, and comprises the following steps: an improved mask-rcnn model is used to realize the algorithm for detecting and segmenting coal and gangue; the algorithm for calculating the depth information of the coal and gangue images collected by a binocular camera; and the algorithm for calculating the optimal grabbing angle of the segmented images. The application solves the function of detecting and segmenting coal and gangue, realizes the measurement of the distance between coal and gangue, and has the function of calculating the optimal grabbing angle. The system improves the efficiency of sorting coal and gangue, and has important significance for the green development of the coal industry.

Description

Technical Field

[0001] This invention relates to the field of collaborative robotic arms, and in particular to a method for using a robotic arm system for rapid sorting of coal and gangue based on an improved mask-rcnn algorithm. Background Technology

[0002] With continuous research in the field of vision, machine vision has begun to shine in areas such as object recognition, classification, and segmentation. Vision can obtain the color, size, shape, and position of objects, and it provides 80% of the information about the outside world. Among these, obtaining depth information of a given target is one of the current research hotspots.

[0003] Traditional coal and gangue sorting relies on manual sorting by workers. However, the complex environment of underground tunnels and the large amount of dust generated during coal mining pose a great threat to workers' health. With the development of intelligent coal mining, coal and gangue sorting has entered a new stage.

[0004] Therefore, a rapid coal and gangue sorting robotic arm system based on an improved mask-rcnn algorithm is needed to achieve the above functions. It should be able to autonomously identify coal and gangue in a set environment, calculate their depth information and shape and size, and guide the robotic arm to sort coal and gangue based on the acquired information, thereby improving the efficiency of coal and gangue sorting. Summary of the Invention

[0005] The purpose of this invention is to propose a method for using a robotic arm system for rapid sorting of coal and gangue based on an improved mask-R-CNN algorithm. This method first collects a large number of coal and gangue images, and then performs operations such as cropping, rotating, noise reduction, enlarging, and shrinking to expand the dataset. The dataset is then divided into training, validation, and test sets in a 10:3:2 ratio. Next, the mask-R-CNN network model is improved. The feature extraction network architecture in the backbone layer is modified by changing the structure in ResNet layer 1 to a Conv+BN structure and introducing a residual structure. The structure in layer 2 is changed to a Conv+BN+ReLU structure. Further residual structures and attention mechanisms are added to layers 3 and 4 to improve the semantic information of high-level features and the detailed information of low-level features in the feature maps obtained from the feature extraction part. Furthermore, the number of anchors in the FPN layer is reduced from 9 to 6, resulting in a 0.67-fold reduction in the total number of anchors, significantly reducing training and inference time. Feed your dataset into the improved mask-rcnn model for training and save the trained weights. A pair of images captured by a stereo camera are fed into an improved mask-R-CNN model for inference, resulting in the corresponding target bounding boxes for coal and gangue. The center position of the target bounding box is read out, and the coordinates of the center point in the left camera image and the R, G, and B values ​​of the pixels at the center point are retained. For the coordinates of the center point of the target bounding box in the right camera, the model is extended upwards and downwards by 10 pixels, and to the left and right by 49 pixels. A 5×5 square template is used to traverse each pixel within a 21×99 rectangle. The R, G, and B values ​​of the pixels covered by the template are summed and averaged to replace the R, G, and B values ​​of the original template center point pixel. The pixel values ​​of these points are compared with the pixel values ​​at the center point of the left camera, and the matching pixel with the smallest difference is selected. The image pixel coordinates of the matching pixel are obtained, and the absolute value of the difference between the two x-coordinates is taken to obtain the disparity value. Based on the camera imaging principle and the triangle similarity theorem, the depth information can be calculated. Based on the shapes of coal and gangue segmented by the improved mask-rcnn, their edge coordinates are output. The distances between these points and surrounding points are traversed sequentially, and the two points with the largest distance are selected. Their slopes in the image are calculated so that the angle of the robotic claw at the end of the robotic arm is perpendicular to it. Combined with depth information, the robotic arm is guided to reach the designated position to complete the grasping task. Attached Figure Description

[0006] Figure 1 This is a system schematic diagram of the present invention.

[0007] Figure 2 The following is a flowchart of the algorithm of the present invention.

[0008] Figure 3 This is a schematic diagram illustrating the calculation of disparity values ​​according to the present invention. Detailed Implementation

[0009] This invention relates to a method for using a rapid coal and gangue sorting robotic arm system based on an improved mask-rcnn algorithm. The system was designed from the outset to be intelligent and replace manual labor. It can be simulated on Windows and Linux systems, and the algorithm can also be imported into Raspberry Pi and Jetson Nano development boards, making it highly portable.

[0010] The specific implementation method is as follows:

[0011] (1) First, a large amount of coal and gangue datasets were collected and cropped, rotated, denoised, enlarged and reduced to expand the dataset. The datasets were divided into training set, validation set and test set in a ratio of 10:3:2.

[0012] (2) Install the necessary environment for training the improved mask-rcnn network under the windox system, then import the dataset for training and save the trained weight file.

[0013] (3) Calibrate the binocular camera and write the calibration data into the calibration function of the algorithm.

[0014] (4) Take a pair of pictures using a binocular camera and feed them into the trained improved mask-rcnn algorithm to calculate depth information for the identified coal and gangue.

[0015] (5) Based on the shape of the coal and gangue, calculate the rotation angle required for the gripping task corresponding to the included angle of the mechanical claw at the end of the robotic arm.

[0016] (6) The remaining 5 joint angles for the robotic arm to perform the grasping task are calculated from the depth information using forward and inverse kinematics algorithms.

[0017] (7) Port the algorithm to the Jetson Nano development board and extend the robotic arm for debugging.

[0018] (8) To achieve the task of sorting coal and gangue.

Claims

1. A method for using a rapid coal and gangue sorting robotic arm system based on an improved mask-rcnn algorithm, characterized in that, The method includes the following: (1) The collected coal and gangue datasets are fed into the improved mask-rcnn model for training, and the optimal weights are saved after training; (2) The trained improved mask-rcnn algorithm is used to detect and segment the images captured by the binocular camera. The disparity of the detected coal and gangue in the left and right images is read and the depth information is calculated. (3) Calculate the optimal clamping angle between the end of the robot arm and the coal and gangue based on the shape and position of the separated coal and gangue; In the content (1), the feature extraction network architecture in the Backbone layer of mask-rcnn is improved. The structure in layer 1 of ResNet50 is changed to Conv+BN structure and a residual structure is introduced. The structure in layer 2 is changed to Conv+BN+ReLU structure. The residual structure and attention mechanism are further added to the structure in layer 3 and layer 4. After the above steps, the semantic information of the high-level features and the detailed information of the low-level features of the feature map obtained by the feature extraction part are richer, which lays the foundation for subsequent detection and segmentation. In the content (1) mentioned above, coal and gangue are detected and classified. The environment and the object being tested are relatively simple. Therefore, in the FPN layer structure of mask-rcnn, the number of target box anchors of 256×256 is reduced, and only 6 anchors with areas of 128×128, 256×256, 256×128, 512×512, 1024×1024 and 1024×512 are retained. The total number of anchors is reduced by 0.67 times, which greatly reduces the training and inference time.

2. The method of using the rapid sorting robot arm system for coal and gangue based on the improved mask-rcnn algorithm according to claim 1, wherein in the content (2), a pair of images acquired by the binocular camera are sequentially fed into the trained improved mask-rcnn network for inference. For the anchor boxes that identify coal and gangue, the center position of the target box is read out. The coordinates of the center point in the left eye image and the R, G, B values ​​of the pixel at the center point are retained. For the coordinates of the center point of the target box in the right eye camera, the coordinates are extended upward and downward by 10 pixels, and extended to the left and right by 49 pixels, thus forming a rectangular box with a height and width of 21×99. Then, a 5×5 template is used to traverse each pixel in the 21×99 box. The R, G, B values ​​of the 25 pixels covered by the template are added together and averaged to replace the R, G, B values ​​of the center point of the template. Pixel values ​​are then compared with the pixel values ​​at the center point of the left eye camera. The pixel with the smallest difference is selected, and the image pixel coordinates of that pixel are recorded. The absolute value of the difference between the two in the x-coordinate is the disparity. Then, based on camera imaging and the similar triangle theorem, the depth information can be obtained.

3. The method of using the coal and gangue rapid sorting robotic arm system based on the improved mask-rcnn algorithm according to claim 1, wherein in the content (3), according to the shape of the segmented coal and gangue, their edge coordinates are output, the distance between these points and the surrounding points is traversed in turn, the two points with the largest distance are selected, and their slope in the image is calculated so that the mechanical claw at the end of the robotic arm is perpendicular to it, thereby realizing the effective gripping of coal and gangue.

Images