A multi-boom collaborative control method, system, device and medium for a drilling and anchoring robot

Abstract

The present invention discloses a multi-boom cooperative control method, system, device and medium for a drilling and anchoring robot, which relates to the field of drilling and anchoring robot control. The method includes: acquiring an image of a drilling and anchoring hole to be recognized; inputting the image of the drilling and anchoring hole to be recognized into a drilling and anchoring hole recognition model to obtain a recognized image of the drilling and anchoring hole; the drilling and anchoring hole recognition model is constructed based on the K-Means clustering algorithm and the MaskR-CNN network; positioning the drilling and anchoring holes in the recognized image of the drilling and anchoring hole to obtain the position information of the drilling and anchoring holes; planning the motion trajectories and working spaces of the booms of the target drilling and anchoring robot according to the position information of the drilling and anchoring holes to obtain boom planning information; and cooperatively controlling the booms of the target drilling and anchoring robot according to the boom planning information. The present invention can realize the cooperative control of the booms of the drilling and anchoring robot, and improve the support efficiency and personnel safety.

Description

Technical Field

[0001] The present invention relates to the field of drill-anchor robot control, and particularly to a multi-drill-arm cooperative control method, system, device and medium for a drill-anchor robot. Background Art

[0002] The intelligentization of the fully mechanized heading face in coal mines is the foundation of coal mine intelligentization, and the cooperative control of the equipment in the fully mechanized heading face is an important prerequisite for realizing coal mine intelligentization and unmanned operation. In the actual coal mine production process, when a drill-anchor robot uses multiple drill arms for bolt-net support operations, due to the harsh underground environment and complex support operation processes, the support operations are still mainly manual, which poses a threat to the safety of the support workers. Therefore, in order to improve the support efficiency and personnel safety, the multi-drill arms of the drill-anchor robot are cooperatively controlled to reduce the underground staff, which is of great significance to the safe production of coal mines. Summary of the Invention

[0003] The purpose of the present invention is to provide a multi-drill-arm cooperative control method, system, device and medium for a drill-anchor robot, so as to realize the cooperative control of each drill arm of the drill-anchor robot and improve the support efficiency and personnel safety.

[0004] To achieve the above purpose, the present invention provides the following solutions:

[0005] A multi-drill-arm cooperative control method for a drill-anchor robot, the method includes:

[0006] Obtain an image of a drill-anchor hole to be recognized;

[0007] Input the image of the drill-anchor hole to be recognized into a drill-anchor hole recognition model to obtain a drill-anchor hole recognition image; the drill-anchor hole recognition model is constructed based on the K-Means clustering algorithm and the Mask R-CNN network;

[0008] Locate the drill-anchor holes in the drill-anchor hole recognition image to obtain drill-anchor hole position information;

[0009] According to the drill-anchor hole position information, plan the motion trajectories and working spaces of each drill arm of the target drill-anchor robot to obtain drill arm planning information;

[0010] Cooperatively control each drill arm of the target drill-anchor robot according to the drill arm planning information.

[0011] Optionally, the construction method of the drill-anchor hole recognition model includes:

[0012] Obtain a drill-anchor hole image data set and drill-anchor hole position marking data; the drill-anchor hole image data set includes multiple drill-anchor hole image samples; the drill-anchor hole position marking data includes the drill-anchor hole bounding boxes and drill-anchor hole segmentation masks corresponding to each drill-anchor hole image sample;

[0013] Use the K-Means clustering algorithm to determine the anchor box data according to the drill and anchor hole image dataset;

[0014] Use the anchor box data as the anchor point box parameters of the Mask R-CNN network, and input the drill and anchor hole image dataset and the drill and anchor hole position marking data into the Mask R-CNN network for training to obtain a drill and anchor hole recognition model.

[0015] Optionally, the Mask R-CNN network includes a feature extraction module, a candidate region generation module, a region feature aggregation module, and a prediction and recognition module connected in sequence; the prediction and recognition module includes a parallel bounding box recognition layer and a mask prediction layer;

[0016] The feature extraction module is used to extract features from the input image to obtain a first feature map;

[0017] The candidate region generation module is used to generate a candidate region set according to the first feature map; the candidate region set includes multiple regions of interest;

[0018] The region feature aggregation module is used to perform region feature aggregation processing on the candidate region set to obtain a second feature map;

[0019] The bounding box recognition layer is used to perform object detection and bounding box feature extraction on the second feature map to obtain a bounding box recognition image corresponding to the input image;

[0020] The mask prediction layer is used to perform instance segmentation and mask feature extraction on the second feature map to obtain a mask prediction image corresponding to the input image.

[0021] Optionally, the use of the K-Means clustering algorithm to determine the anchor box data of the Mask R-CNN network according to the drill and anchor hole image dataset specifically includes:

[0022] Perform defogging, noise reduction, and enhancement processing on the drill and anchor hole image dataset to obtain a preprocessed drill and anchor hole image dataset;

[0023] Use the K-Means clustering algorithm to cluster the preprocessed drill and anchor hole image dataset to obtain the anchor box data of the Mask R-CNN network.

[0024] Optionally, the performing defogging, noise reduction, and enhancement processing on the drill and anchor hole image dataset to obtain a preprocessed drill and anchor hole image dataset specifically includes:

[0025] Use the histogram equalization algorithm to perform defogging processing on the drill and anchor hole image dataset;

[0026] Adopt a three-dimensional block matching algorithm to perform noise reduction processing on the dehazed drill anchor hole image dataset;

[0027] Adopt an image enhancement algorithm based on Gaussian homomorphic filtering to perform enhancement processing on the noise-reduced drill anchor hole image dataset, and obtain a preprocessed drill anchor hole image dataset.

[0028] Optionally, the motion trajectories and working spaces of the drill arms of the target drill anchor robot are planned according to the drill anchor hole position information to obtain drill arm planning information, which specifically includes:

[0029] Adopt the D-H solution method to determine the kinematic equations of the drill arms of the target drill anchor robot according to the drill anchor hole position information;

[0030] Adopt a cubic spline interpolation algorithm to plan the motion trajectories of the drill arms of the target drill anchor robot according to the kinematic equations and the drill anchor hole position information to obtain the first planning information;

[0031] According to the first planning information and the drill anchor hole position information, plan the working spaces of the drill arms of the target drill anchor robot to obtain the second planning information;

[0032] Determine the drill arm planning information according to the first planning information and the second planning information.

[0033] Optionally, the drill anchor holes in the drill anchor hole recognition image are located to obtain drill anchor hole position information, which specifically includes:

[0034] Fit the drill anchor holes in the drill anchor hole recognition image to obtain the centers of the drill anchor holes;

[0035] Adopt monocular vision positioning technology to position the centers of the drill anchor holes to obtain drill anchor hole position information; the drill anchor hole position information includes the center coordinates of the drill anchor holes.

[0036] A multi-drill arm cooperative control system for a drill anchor robot, the system includes:

[0037] An image acquisition module for acquiring an image of a drill anchor hole to be recognized;

[0038] A drill anchor hole recognition module for inputting the image of the drill anchor hole to be recognized into a drill anchor hole recognition model to obtain a drill anchor hole recognition image; the drill anchor hole recognition model is constructed based on the K-Means clustering algorithm and the Mask R-CNN network;

[0039] A drill anchor hole positioning module for positioning the drill anchor holes in the drill anchor hole recognition image to obtain drill anchor hole position information;

[0040] The boom planning module is used to plan the motion trajectories and working spaces of the booms of the target drilling and anchoring robot according to the drilling and anchoring hole position information, so as to obtain boom planning information;

[0041] The boom control module is used to perform cooperative control on the booms of the target drilling and anchoring robot according to the boom planning information.

[0042] An electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned multi-boom cooperative control method for a drilling and anchoring robot.

[0043] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned multi-boom cooperative control method for a drilling and anchoring robot.

[0044] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0045] The multi-boom cooperative control method for a drilling and anchoring robot provided by the present invention uses a drilling and anchoring hole recognition model constructed based on the K-Means clustering algorithm and the Mask R-CNN network to recognize the real-time captured underground drilling and anchoring hole image (i.e., the drilling and anchoring hole image to be recognized). Under the working conditions of high dust, low illuminance, and complex background in the coal mine underground, the drilling and anchoring holes in the image can also be accurately recognized, so as to determine the drilling and anchoring hole position information. Using this position information to plan the motion trajectories and working spaces of the booms of the drilling and anchoring robot, and then realizing the cooperative control of the booms of the drilling and anchoring robot, improving the support efficiency. In addition, since the method provided by the present invention is automatically implemented by devices such as cameras and controllers underground in the mine and does not require manual operation, it can also reduce the underground staff and avoid the occurrence of safety accidents, and has the advantages of high real-time performance and high reliability. Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0047] Figure 1 It is a flowchart of the multi-boom cooperative control method for a drilling and anchoring robot provided by the present invention;

[0048] Figure 2 It is a module diagram of the multi-boom cooperative control system for a drilling and anchoring robot provided by the present invention;

[0049] Figure 3 It is the overall process block diagram of the multi-boom collaborative control system of the drilling and anchoring robot provided by the embodiment of the present invention;

[0050] Figure 4 It is the process block diagram of the drilling and anchoring hole position recognition and positioning module provided by the embodiment of the present invention;

[0051] Figure 5 It is the flow chart of the drilling and anchoring hole recognition based on the Mask R-CNN network provided by the embodiment of the present invention;

[0052] Figure 6 It is the flow chart of the drilling and anchoring hole positioning method based on monocular vision provided by the embodiment of the present invention. Detailed implementation manners

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0054] The purpose of the present invention is to provide a multi-boom collaborative control method, system, device and medium for a drilling and anchoring robot to achieve the collaborative control of each boom of the drilling and anchoring robot and improve the support efficiency and personnel safety.

[0055] The multi-boom collaborative control method of the drilling and anchoring robot proposed by the present invention first identifies and locates the position of the anchor drilling hole, and then plans the paths of the multi-boom of the drilling and anchoring robot so that it can move to the position of the anchor drilling hole. Finally, in order to prevent the multi-boom from colliding, the movement between the multi-boom is collaboratively controlled. Through this method, the support efficiency of the multi-boom can be improved, and then the production efficiency of the coal mine can be improved.

[0056] In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0057] Embodiment 1

[0058] The present invention provides a multi-boom collaborative control method for a drilling and anchoring robot. As Figure 1 shown, the method includes:

[0059] Step 101: Obtain the image of the drilling and anchoring hole to be recognized.

[0060] Step 102: Input the drill anchor hole image to be recognized into the drill anchor hole recognition model to obtain a drill anchor hole recognition image; the drill anchor hole recognition model is constructed based on the K-Means clustering algorithm and the Mask R-CNN network.

[0061] Step 103: Locate the drill anchor holes in the drill anchor hole recognition image to obtain drill anchor hole position information.

[0062] Step 104: According to the drill anchor hole position information, plan the motion trajectories and working spaces of the drill arms of the target drill anchor robot to obtain drill arm planning information.

[0063] Step 105: Coordinate and control the drill arms of the target drill anchor robot according to the drill arm planning information.

[0064] The construction method of the drill anchor hole recognition model includes: obtaining a drill anchor hole image dataset and drill anchor hole position marking data; the drill anchor hole image dataset includes multiple drill anchor hole image samples; the drill anchor hole position marking data includes the drill anchor hole bounding boxes and drill anchor hole segmentation masks corresponding to each of the drill anchor hole image samples; using the K-Means clustering algorithm to determine anchor box data according to the drill anchor hole image dataset; using the anchor box data as the anchor point box parameters of the Mask R-CNN network, and inputting the drill anchor hole image dataset and the drill anchor hole position marking data into the Mask R-CNN network for training to obtain a drill anchor hole recognition model. Among them, the using the K-Means clustering algorithm to determine the anchor box data of the Mask R-CNN network according to the drill anchor hole image dataset specifically includes: performing defogging, noise reduction, and enhancement processing on the drill anchor hole image dataset to obtain a preprocessed drill anchor hole image dataset; using the K-Means clustering algorithm to cluster the preprocessed drill anchor hole image dataset to obtain the anchor box data of the Mask R-CNN network.

[0065] To execute the above method to achieve the corresponding functions and technical effects, the present invention also provides a multi-drill arm collaborative control system for a drill anchor robot. As Figure 2 shown, the system includes:

[0066] An image acquisition module 201 for acquiring a drill anchor hole image to be recognized.

[0067] A drill anchor hole recognition module 202 for inputting the drill anchor hole image to be recognized into the drill anchor hole recognition model to obtain a drill anchor hole recognition image; the drill anchor hole recognition model is constructed based on the K-Means clustering algorithm and the Mask R-CNN network.

[0068] The drill anchor hole positioning module 203 is used to position the drill anchor holes in the drill anchor hole recognition image to obtain the drill anchor hole position information.

[0069] The drill arm planning module 204 is used to plan the movement trajectories and working spaces of the drill arms of the target drill anchor robot according to the drill anchor hole position information to obtain drill arm planning information.

[0070] The drill arm control module 205 is used to coordinately control the drill arms of the target drill anchor robot according to the drill arm planning information.

[0071] Specifically, the image acquisition module 201 can be a camera. The image acquisition module 201, the drill anchor hole recognition module 202, and the drill anchor hole positioning module 203 together constitute a drill anchor hole position recognition and positioning module, and the drill arm planning module 204 and the drill arm control module 205 together constitute a multi-drill arm coordinated control module for the drill anchor robot. For the overall process of the system, see Figure 3 , where the drill anchor hole position recognition and positioning module is used to: collect the image information of the drill anchor holes, make a data set; preprocess the data set; perform drill anchor hole position recognition and positioning. The multi-drill arm coordinated control module for the drill anchor robot is used to: solve the kinematic equations of the multi-drill arms; plan the movement trajectories and working spaces of the multi-drill arms; and use a PLC controller to perform real-time coordinated control of the movement of the multi-drill arms.

[0072] Next, in combination with the above methods and systems, the solution provided by the present invention will be discussed in detail.

[0073] (1) Recognition and positioning of anchor drill hole positions

[0074] As Figure 4 shown, in the coal mine underground, first, a camera is used to collect the image information of the anchor drill holes to make a data set (for training). Since the environment in the coal mine underground is harsh, the collected images are unclear, interfered by dust and water vapor. Therefore, the image data set of the anchor drill holes is preprocessed, including defogging, noise reduction, enhancement, etc.; secondly, the K-Means clustering algorithm is used to cluster the image data set of the anchor drill holes to generate anchor frames; then, the Mask R-CNN network is trained using the image data set to obtain a drill anchor hole recognition model. Finally, the drill anchor hole recognition model (i.e., the trained Mask R-CNN network) is used to detect the drill anchor hole targets in the real-time collected anchor drill hole images to be recognized, so as to recognize the anchor drill holes; finally, a monocular camera is used to position the center position of the anchor drill holes.

[0075] The dataset is crucial in the field of object detection and recognition, directly affecting the performance of object detection and recognition models. The anchor drilling image dataset used in this invention all comes from the real working environment in coal mines. Due to the harsh coal mine environment, the changing illumination intensity in the underground working environment, and the fact that the collected image information is easily blocked by steel belts, etc., it is easy to cause errors in the recognition and positioning results of anchor drill holes. Therefore, in order to enhance the generalization ability of the dataset, alleviate the overfitting phenomenon, and improve the detection ability of the model under the harsh conditions in coal mines, this invention also uses data augmentation techniques to expand the number of samples. The dataset can be expanded using the following two methods. The first method is to use flipping, translation, and scaling techniques to change the image position and increase the types of the dataset. The second method for expanding the dataset is the SMOTE method, that is, the Synthetic Minority Over-sampling Technique method. It is a method for dealing with the problem of sample imbalance by artificially synthesizing new samples, thereby improving the performance of the classifier. The SMOTE method is an interpolation-based method. It can synthesize new samples for the small sample class. The main steps are as follows:

[0076] Step 0.1: Define the feature space, map each sample to a certain point in the feature space, and determine a sampling magnification N according to the sample imbalance ratio.

[0077] Step 0.2: For each sample (x, y) in the small sample class, find K nearest neighbor samples according to the Euclidean distance, and randomly select one sample point from them. Suppose the selected neighbor point is (x n , y n ). Randomly select a point on the line segment connecting the sample point and the nearest neighbor sample point in the feature space as the new sample point (x new , y new ), satisfying the following formula:

[0078] (x new , y new ) = (x, y) + rand(0 - 1) * ((x n - x), (y n - y)).

[0079] Step 0.3: Repeat the above steps 0.1 to 0.2 until the number of large and small samples is balanced.

[0080] Since there is a lot of dust and water vapor in coal mines, and the image information collected by the camera will be interfered by noise, etc., therefore, before model training (or image recognition), it is first necessary to preprocess the anchor drilling image dataset (or the drill anchor hole image to be recognized). This preprocessing includes defogging, denoising, and enhancement of the image. Specifically:

[0081] Image dehazing: The histogram equalization algorithm is used to perform dehazing processing on the drill anchor hole image dataset.

[0082] Image denoising: The 3D block matching algorithm is used to perform denoising processing on the dehazed drill anchor hole image dataset.

[0083] Image enhancement: The image enhancement algorithm based on Gaussian homomorphic filtering is used to perform enhancement processing on the denoised drill anchor hole image dataset, and the preprocessed drill anchor hole image dataset is obtained.

[0084] Similarly, for the drill anchor hole image to be recognized, in order to make the recognition result more accurate, dehazing, denoising, and enhancement processing also need to be performed before inputting it into the drill anchor hole recognition model.

[0085] 1. Image dehazing

[0086] The histogram equalization algorithm is used to perform dehazing processing on the image. The main idea of this algorithm is: by non-linearly stretching the pixel values of the image, re-distributing the pixel values to make them more uniform, so that the contrast of the part with relatively concentrated gray-scale distribution on the original image is enhanced, while the contrast of the part with relatively sparse distribution is reduced. The histogram of the processed image will show a relatively flat state, achieving the intuitive dehazing effect.

[0087] The specific steps of using the histogram equalization algorithm for image dehazing are as follows:

[0088] Step 1.1: Count the number of pixels n of each gray level in the original image i (0 ≤ i < L), where L is the total number of gray levels in the image (usually 256).

[0089] The probability that a pixel with gray level i appears in the image is:

[0090] where n is the total number of pixel points in the image.

[0091] Step 1.3: The cumulative distribution function of p x is defined as:

[0092]

[0093] Step 1.4: The histogram equalization calculation formula is:

[0094] where M and N are the sizes of the original image respectively; v is the pixel value of the original image, and cdf min is the minimum value of the cumulative distribution function.

[0095] 2. Image denoising

[0096] The Block Matching 3D (BM3D) algorithm is a noise reduction method. Its advantage is that it can better preserve some details in the image. This algorithm searches for similar blocks and filters them in the transform domain to obtain block evaluation values, and finally weights each point in the image to get the final denoising effect. Its main idea is as follows: First, an image is segmented into small pixel patches of smaller size. After selecting a reference patch, small patches similar to the reference patch are found to form a 3D block. After this process, a 3D block will be obtained. Then, all similar blocks are subjected to a 3D transform. The transformed 3D block is subjected to threshold shrinkage, which is also the process of removing noise. Then, a 3D inverse transform is performed. Finally, all the 3D blocks are restored to the image through weighted averaging.

[0097] The specific steps of the BM3D algorithm are as follows:

[0098] Step 2.1: Initial estimation

[0099] (1) Block-by-block estimation: First, the image blocks are grouped to find their similar blocks and then they are aggregated into a three-dimensional array. Then, a three-dimensional transform is performed on the formed three-dimensional array. By performing hard threshold processing on the coefficients in the transform domain, the noise is weakened, and then an inverse transform is performed to obtain the estimated values of all the image blocks in the group. Then, these estimated values are returned to their original positions.

[0100] (2) Aggregation: For the overlapping block estimates obtained, the basic estimate of the real image is obtained by performing weighted averaging on them.

[0101] Step 2.2: Final estimation

[0102] (1) Block-by-block estimation: First, the positions of the similar blocks similar to it in the basic estimate image are found through block matching. Two three-dimensional arrays are obtained through these positions, one from the noisy image and one from the basic estimate image. Then, joint Wiener filtering is performed. A three-dimensional transform is performed on both of the formed three-dimensional arrays. Using the energy spectrum in the basic estimate image as the energy spectrum, Wiener filtering is performed on the noisy three-dimensional array, and then an inverse transform is performed to obtain the estimates of all the image blocks in the group. Then, these estimated values are returned to their original positions.

[0103] (2) Aggregation: For the overlapping local block estimates obtained, the final estimate of the real image is obtained by performing weighted averaging on them.

[0104] 3. Image enhancement

[0105] Due to the low lighting level in the coal mine, the contrast of the collected anchor borehole images will be poor and the edge information will be blurred, making image recognition difficult. Therefore, image enhancement is achieved based on Gaussian homomorphic filtering to enhance the collected images and enhance the brightness features of the steel belt and anchor boreholes in the images for subsequent anchor borehole recognition. The basic idea of the image enhancement algorithm based on Gaussian homomorphic filtering is as follows: This algorithm is a method for enhancing images in the frequency domain. By using the filtering algorithm to weaken the low-frequency part and enhance the high-frequency part, the illumination change is reduced, thereby sharpening the edges of the image and highlighting the details.

[0106] The specific steps of the image enhancement algorithm based on Gaussian homomorphic filtering are as follows:

[0107] Step 3.1: Perform a logarithmic operation on the brightness information f(x, y) of the image to separate the illumination component i(x, y) and the reflection component r(x, y) of the image:

[0108] z(x, y) = lnf(x, y) = lni(x, y) + lnr(x, y).

[0109] Step 3.2: Perform a Fourier transform on the image information that has undergone the logarithmic operation in Step 3.1:

[0110] Z(u, v) = F i (u, v) + F r (u, v);

[0111] Among them, Z(u, v) is the Fourier transform result of z(x, y), F i (u, v) is the Fourier transform result of lni(x, y), and F r (u, v) is the Fourier transform result of lnr(x, y).

[0112] Step 3.3: Perform Gaussian filtering on Z(u, v):

[0113] S(u, v) = H(u, v)Z(u, v) = H(u, v)F i (u, v) + H(u, v)F r (u, v);

[0114] Among them, H(u, v) is the Gaussian filtering function.

[0115] Step 3.4: Perform an inverse Fourier transform on S(u, v):

[0116] s(x, y) = IDFT(S(u, v)).

[0117] Step 3.5: Use exponential operation to obtain the enhanced image g(x,y), as well as the illumination component i0(x,y) and reflection component r0(x,y) after image enhancement:

[0118] g(x,y) = e s(x,y) = i0(x,y) + r0(x,y).

[0119] In the field of object detection, the anchor box mechanism is adopted to generate proposed target regions, thereby improving the performance of the model. In the object detection model using the anchor box mechanism, the anchor box parameters are all manually designed, which is in line with the general situation. However, the manually designed anchor box parameters may not be suitable for the anchor drilling image dataset. If the size of the anchor box is too different from the size of the target, it will affect the prediction results of the model. The present invention uses the K-Means clustering algorithm to redesign the position information of the anchor drilling to meet the requirements of anchor drilling recognition.

[0120] K-Means clustering is a simple and commonly used unsupervised learning method. The specific workflow is to divide it into k similar clusters according to the number of dataset categories. The process of clustering the anchor drilling dataset using the K-Means method is as follows:

[0121] Step 4.1: Randomly select cluster centers according to the number of dataset categories of the anchor drilling and mark them as cluster centers.

[0122] Step 4.2: Use similarity measurement (using the Euclidean distance calculation method) for all the anchor drilling data positions, and assign each sample to the nearest cluster center.

[0123] Step 4.3: Calculate the average value of all samples in each cluster and update the cluster center position at the same time.

[0124] Step 4.4: Repeat Step 4.2 and Step 4.3 until the average cluster center no longer changes.

[0125] Use the K-means clustering algorithm to generate anchor boxes for the anchor drilling images to adapt to the special underground environment and facilitate the next step of anchor drilling recognition.

[0126] The present invention uses the Mask R-CNN network for anchor drilling recognition. Mask R-CNN is one of the most excellent object detection technologies with object recognition and semantic segmentation capabilities in the R-CNN series of algorithms. The feature extraction network of the Mask R-CNN network model combines the feature pyramid and the residual network, uses the top-down and bottom-up two paths to combine information at multiple scales to prevent network degradation, changes the region of interest pooling layer to a region of interest matching layer, and finally adds a mask branch in the output part, which can not only classify and regress the image, but also realize the instance segmentation mask of the target.

[0127] Specifically, the Mask R-CNN network includes a feature extraction module, a candidate region generation module, a region feature aggregation module, and a prediction and recognition module connected in sequence; the prediction and recognition module includes a parallel bounding box recognition layer and a mask prediction layer. The feature extraction module is used to extract features from the input image to obtain a first feature map; the candidate region generation module is used to generate a candidate region set according to the first feature map; the candidate region set includes a plurality of regions of interest; the region feature aggregation module is used to perform region feature aggregation processing on the candidate region set to obtain a second feature map; the bounding box recognition layer is used to perform object detection and bounding box feature extraction on the second feature map to obtain a bounding box recognition image corresponding to the input image; the mask prediction layer is used to perform instance segmentation and mask feature extraction on the second feature map to obtain a mask prediction image corresponding to the input image. Among them, the feature extraction module is specifically a ResNet50-FPN feature pyramid network; the candidate region generation module is specifically an RPN region proposal generation network.

[0128] In the training stage, the input images are the drill and anchor hole image samples in the drill and anchor hole image dataset, and the output images are the bounding box recognition images and mask prediction images corresponding to the drill and anchor hole image samples. By comparing the output images with the drill and anchor hole position marking data, the parameters in the Mask R-CNN network are optimized and adjusted to complete the training. In the prediction stage, the input image is the drill and anchor hole image to be recognized, and the output images are the bounding box recognition image and mask prediction image corresponding to the drill and anchor hole image to be recognized, that is, the drill and anchor hole recognition image is the bounding box recognition image corresponding to the drill and anchor hole image to be recognized and / or the mask prediction image corresponding to the drill and anchor hole image to be recognized.

[0129] In this embodiment, the framework of the Mask R-CNN network consists of four parts: a feature extraction module (ResNet50-FPN feature pyramid network), a candidate region generation module (RPN region proposal generation network), an object detection module (bounding box recognition layer), and an instance segmentation module (mask prediction layer). The steps of using the Mask R-CNN network to recognize the anchor drill holes are as Figure 5 shown, including:

[0130] Step 5.1: Use a feature pyramid network with ResNet50 as the backbone to extract features to obtain a first feature map.

[0131] Step 5.2: Based on the first feature map, use the RPN region proposal generation network to generate many regions of interest (RoI) for the image.

[0132] Step 5.3: After being processed by the Region of Interest Align (RoIAlign) operation, each region of interest generates a feature map of a fixed size, i.e., the second feature map.

[0133] Step 5.4: Based on this feature map, bounding box recognition and mask prediction are implemented.

[0134] After identifying the anchor drilling information in the image according to the Mask R-CNN network, it is necessary to locate the hole center of the anchor drilling. In this embodiment, a monocular vision positioning technique is used to locate the hole center of the anchor drilling. The positioning of the anchor drilling in the anchor drilling recognition image to obtain the anchor drilling position information specifically includes: fitting the anchor drilling in the anchor drilling recognition image to obtain the center of the anchor drilling circle; using the monocular vision positioning technique to locate the center of the anchor drilling circle to obtain the anchor drilling position information; the anchor drilling position information includes the center coordinates of the anchor drilling circle.

[0135] Specifically, as Figure 6 shown, first, calibrate the camera to obtain the parameters of the camera; second, perform defogging, noise reduction, enhancement, etc. on the collected anchor drilling image, and use the trained Mask R-CNN network to identify the anchor drilling; finally, fit the anchor drilling in the image to obtain the center of the circle, and then obtain the center coordinates of the anchor drilling according to the coordinate conversion relationship between the images.

[0136] (2) Multi-boom collaborative control module of the anchor drilling robot

[0137] According to the above anchor drilling position recognition and positioning module, the anchor drilling position information can be solved. After obtaining the anchor drilling position information, the booms of the anchor drilling robot can be planned to move to the position of the anchor drilling, so as to carry out the support operation. To prevent interference and collision between the multi-booms of the anchor drilling robot, a reasonable motion space planning for the booms is carried out, and a PLC controller is used to perform collaborative control on the motion of the multi-booms to improve the support efficiency.

[0138] The motion trajectories and working spaces of the drill arms of the target drill-anchoring robot are planned according to the drill-anchoring hole position information to obtain drill arm planning information, which specifically includes: using the D-H solution method (i.e., a general method for robot modeling proposed by Denavit and Hartenberg in 1955), determining the kinematic equations of the drill arms of the target drill-anchoring robot according to the drill-anchoring hole position information; using the cubic spline interpolation algorithm, planning the motion trajectories of the drill arms of the target drill-anchoring robot according to the kinematic equations and the drill-anchoring hole position information to obtain the first planning information; planning the working spaces of the drill arms of the target drill-anchoring robot according to the first planning information and the drill-anchoring hole position information to obtain the second planning information; and determining the drill arm planning information according to the first planning information and the second planning information.

[0139] In this embodiment, the specific steps of the multi-drill-arm cooperative control method for the drill-anchoring robot are as follows:

[0140] Step 6.1: Calculate the kinematic equations of the drill arms of the drill-anchoring robot using the D-H solution method according to the positioning information of the anchor drilling holes.

[0141] Step 6.2: Plan the motion trajectories of each drill arm of the drill-anchoring robot according to the cubic spline interpolation algorithm.

[0142] Step 6.3: To prevent collision interference during the support operation of multiple drill arms, plan the working spaces of the multiple drill arms.

[0143] Step 6.4: Use a PLC controller to perform real-time cooperative control on the motion of the multiple drill arms.

[0144] Embodiment 2

[0145] The embodiment of the present invention also provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor is used to run the computer program so that the electronic device executes the multi-drill-arm cooperative control method for the drill-anchoring robot in Embodiment 1. The electronic device can be a server.

[0146] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the multi-drill-arm cooperative control method for the drill-anchoring robot in Embodiment 1.

[0147] The present invention provides a cooperative control method, system, device and medium for multi-boom drill-anchor robots. The method first identifies and locates the positions of anchor holes, then plans the paths of the multi-boom drill-anchor robots so that they can move to the positions of the anchor holes. Finally, to prevent collisions between the multi-boom, cooperative control is performed on the movements between the multi-boom. By using the method provided by the present invention, the support efficiency of the multi-boom can be improved, and thus the production efficiency of coal mines can be enhanced.

[0148] Under the working conditions of high dust, low illuminance and complex background in underground coal mines, when the drill-anchor robot performs bolt-net support operations, by using the method provided by the present invention, the position information of the anchor holes can be obtained in real time, so as to plan the trajectories of the drill arms of the drill-anchor robot, and the multi-boom of the drill-anchor robot can be cooperatively controlled through the PLC controller, enabling the multi-boom to work in cooperation. By cooperatively controlling the multi-boom of the drill-anchor robot, the support efficiency of the bolt-net can be significantly improved, the number of underground workers can be reduced, and the occurrence of safety accidents can be avoided, thereby enhancing the production efficiency of coal mining enterprises.

[0149] Cooperative control of underground coal mine equipment is the basis for realizing coal mine intelligentization. By using the method provided by the present invention, the positions of the anchor holes can be obtained in real time, and based on the obtained positions of the anchor holes, the movement spaces of the multi-boom of the drill-anchor robot can be planned, and then the multi-boom can be cooperatively controlled by PLC. Therefore, the method of the present invention has the advantages of high real-time performance and high reliability, and can significantly improve the production efficiency of coal mines.

[0150] The various embodiments in this specification are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method section.

[0151] Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only for helping to understand the core idea of the present invention; at the same time, for those of ordinary skill in the art, based on the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A multi-boom cooperative control method for a drilling and anchoring robot, characterized in that The method includes: Obtaining a drill anchor hole image to be recognized; Inputting the drill anchor hole image to be recognized into a drill anchor hole recognition model to obtain a drill anchor hole recognition image; the drill anchor hole recognition model is constructed based on the K-Means clustering algorithm and the MaskR-CNN network; Locating the drill anchor holes in the drill anchor hole recognition image to obtain drill anchor hole position information; Planning the motion trajectories and working spaces of the drill arms of the target drill anchor robot according to the drill anchor hole position information to obtain drill arm planning information; Coordinately controlling the drill arms of the target drill anchor robot according to the drill arm planning information; The construction method of the drill anchor hole recognition model includes: Obtaining a drill anchor hole image dataset and drill anchor hole position marking data; Using the K-Means clustering algorithm to determine anchor box data according to the drill anchor hole image dataset; Using the anchor box data as the anchor point box parameters of the MaskR-CNN network, and inputting the drill anchor hole image dataset and the drill anchor hole position marking data into the MaskR-CNN network for training to obtain a drill anchor hole recognition model; The MaskR-CNN network includes a feature extraction module, a candidate region generation module, a region feature aggregation module, and a prediction and recognition module connected in sequence; the prediction and recognition module includes a parallel bounding box recognition layer and a mask prediction layer; the feature extraction module is used to extract features from the input image to obtain a first feature map; the candidate region generation module is used to generate a candidate region set according to the first feature map; the candidate region set includes a plurality of regions of interest; the region feature aggregation module is used to perform region feature aggregation processing on the candidate region set to obtain a second feature map; the bounding box recognition layer is used to perform object detection and bounding box feature extraction on the second feature map to obtain a bounding box recognition image corresponding to the input image; the mask prediction layer is used to perform instance segmentation and mask feature extraction on the second feature map to obtain a mask prediction image corresponding to the input image.

2. The multi-boom cooperative control method of the drilling and anchoring robot according to claim 1, wherein The drill anchor hole position marking data includes the drill anchor hole bounding boxes and drill anchor hole segmentation masks corresponding to each drill anchor hole image sample.

3. The multi-boom cooperative control method of the drilling and anchoring robot according to claim 2, characterized in that The using the K-Means clustering algorithm to determine the anchor box data of the MaskR-CNN network according to the drill anchor hole image dataset specifically includes: Performing defogging, noise reduction, and enhancement processing on the drill anchor hole image dataset to obtain a preprocessed drill anchor hole image dataset; Using the K-Means clustering algorithm to cluster the preprocessed drill anchor hole image dataset to obtain the anchor box data of the MaskR-CNN network.

4. The multi-boom cooperative control method of the drilling and anchoring robot according to claim 3, characterized in that, The performing defogging, noise reduction, and enhancement processing on the drill anchor hole image dataset to obtain a preprocessed drill anchor hole image dataset specifically includes: Using the histogram equalization algorithm to perform defogging processing on the drill anchor hole image dataset; Using the three-dimensional block matching algorithm to perform noise reduction processing on the defogged drill anchor hole image dataset; An image enhancement algorithm based on Gaussian homomorphic filtering is adopted to enhance the denoised drill and anchor hole image dataset, and a preprocessed drill and anchor hole image dataset is obtained.

5. The multi-boom cooperative control method of the drilling and anchoring robot according to claim 1, characterized in that Planning the motion trajectories and working spaces of the drill arms of the target drill and anchor robot according to the drill and anchor hole position information to obtain drill arm planning information, specifically including: Using the D-H solution method to determine the kinematic equations of the drill arms of the target drill and anchor robot according to the drill and anchor hole position information; Using the cubic spline interpolation algorithm to plan the motion trajectories of the drill arms of the target drill and anchor robot according to the kinematic equations and the drill and anchor hole position information to obtain the first planning information; Planning the working spaces of the drill arms of the target drill and anchor robot according to the first planning information and the drill and anchor hole position information to obtain the second planning information; Determining the drill arm planning information according to the first planning information and the second planning information.

6. The multi-boom cooperative control method of the drilling and anchoring robot according to claim 1, characterized in that, Positioning the drill and anchor holes in the drill and anchor hole recognition image to obtain drill and anchor hole position information, specifically including: Fitting the drill and anchor holes in the drill and anchor hole recognition image to obtain the centers of the drill and anchor holes; Using the monocular vision positioning technology to position the centers of the drill and anchor holes to obtain the drill and anchor hole position information; the drill and anchor hole position information includes the center coordinates of the drill and anchor holes.

7. A multi-boom cooperative control system for a drilling and anchoring robot, characterized in that, The system includes: An image acquisition module for acquiring the drill and anchor hole image to be recognized; A drill and anchor hole recognition module for inputting the drill and anchor hole image to be recognized into the drill and anchor hole recognition model to obtain a drill and anchor hole recognition image; the drill and anchor hole recognition model is constructed based on the K-Means clustering algorithm and the MaskR-CNN network; A drill and anchor hole positioning module for positioning the drill and anchor holes in the drill and anchor hole recognition image to obtain drill and anchor hole position information; A drill arm planning module for planning the motion trajectories and working spaces of the drill arms of the target drill and anchor robot according to the drill and anchor hole position information to obtain drill arm planning information; A drill arm control module for coordinately controlling the drill arms of the target drill and anchor robot according to the drill arm planning information; The construction method of the drill and anchor hole recognition model includes: Obtaining a drill and anchor hole image dataset and drill and anchor hole position marking data; Using the K-Means clustering algorithm to determine the anchor box data according to the drill and anchor hole image dataset; Using the anchor box data as the anchor point box parameters of the MaskR-CNN network, and inputting the drill and anchor hole image dataset and the drill and anchor hole position marking data into the MaskR-CNN network for training to obtain a drill and anchor hole recognition model; The Mask R-CNN network includes a feature extraction module, a candidate region generation module, a region feature aggregation module, and a prediction and recognition module connected in sequence; the prediction and recognition module includes a parallel bounding box recognition layer and a mask prediction layer; the feature extraction module is used to extract features from the input image to obtain a first feature map; the candidate region generation module is used to generate a candidate region set according to the first feature map; the candidate region set includes a plurality of regions of interest; the region feature aggregation module is used to perform region feature aggregation processing on the candidate region set to obtain a second feature map; the bounding box recognition layer is used to perform object detection and bounding box feature extraction on the second feature map to obtain a bounding box recognition image corresponding to the input image; the mask prediction layer is used to perform instance segmentation and mask feature extraction on the second feature map to obtain a mask prediction image corresponding to the input image.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the multi-drill-arm collaborative control method of the drilling and anchoring robot as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by the processor, it implements the multi-drill-arm collaborative control method of the drilling and anchoring robot as described in any one of claims 1 to 6.

Images