Radar and Vision Fusion-Based Boring Hole Positioning Method for Roadway Roof Support Steel Strips

Abstract

The present invention relates to a method for positioning the drilling holes of a roadway roof support steel strip based on the fusion of radar and vision, belonging to the technical field of roadway roof support. It includes: collecting the visual image of the roadway roof support steel strip at the current position through an industrial camera to determine the visual detection result of the drilling holes; obtaining the three-dimensional laser point cloud data by scanning the roadway roof at the current position with a lidar, and extracting the boundary point cloud of the drilling holes; using the jointly calibrated external parameter matrix of the lidar and the industrial camera to project the boundary point cloud of the drilling holes into the pixel coordinate system of the industrial camera to obtain the two-dimensional point cloud data; correlating the two-dimensional point cloud data with the visual detection result of the drilling holes, retaining the correct drilling detection result, and fusing the drilling detection results to obtain the three-dimensional coordinate information of the drilling holes. The present invention provides a hole positioning method based on the fusion of lidar and vision. This method combines the respective advantages of visual and lidar target detection, improves the accuracy of target detection, and realizes the accurate identification and precise positioning of the drilling holes.

Description

Technical Field

[0001] The present invention relates to the technical field of roadway roof support, and particularly to a positioning method for drilling holes in a roadway roof support steel strip based on the fusion of radar and vision. Background Technique

[0002] The intelligent construction of coal mines is the only way to achieve less manpower and unmanned coal mining. At present, the intelligent tunneling is still in its infancy, and there are generally problems of imbalance between excavation and support, seriously affecting the safe and efficient production of coal. Among them, the time used for roadway roof support accounts for more than half of the tunneling operation time. The fact that tunneling is fast while support is slow has become a bottleneck for improving the roadway tunneling speed. In order to improve the tunneling speed underground in coal mines, the first problem to be solved is the roadway roof support problem. At present, the roof support of most coal mine tunneling roadways still requires manual operation by workers. Not only is the support efficiency low and the labor intensity of workers high, but also the hole-forming effect of manual drilling is poor, affecting the anchoring quality. Only by realizing the automation of the support process and using an automatic drill-anchor robot to replace manual drill-anchor operation can the tunneling efficiency underground in coal mines be improved. Hole recognition is the core technical problem for realizing automatic drill-anchoring. The underground environment is harsh and the lighting conditions are poor. Especially for coal mines with poor geological conditions, it is also necessary to use anchor beam support, that is, a steel strip + anchor net in cooperation with anchor bolts for anchoring, and it is more difficult to identify the anchor hole positions. Therefore, how to achieve accurate recognition and positioning of the drilling holes in the roadway roof support steel strip is the key technical problem that urgently needs to be solved for realizing full-automatic support of the tunneling roadway.

[0003] Patent 202011315046.4 provides a method and device for identifying drilling holes in a steel strip for anchor support and an anchor net based on monocular vision. By setting a monocular vision system at the front section of the drill rig of the anchor bolt drill, it collects the visual image of the current roadway roof, identifies the drilling hole position through an image detection algorithm, and then calculates the actual positions of the drilling hole and the anchor net to generate drilling target azimuth information for the anchor bolt drill. Patent 202210465629.8 provides a positioning method and system for drilling holes in a steel strip on the roof of a mine roadway based on a laser scanner. By sampling the steel strip on the roof of the roadway with a laser scanner to obtain sampling cloud point information, it uses a point cloud segmentation algorithm to extract the boundary point cloud of the drilling hole, and calculates the three-dimensional coordinates of the anchor hole center through elliptical clustering fitting, and transmits them to the drill arm control system for anchor bolt support operation.

[0004] In summary, when the current underground coal mine drilling and anchoring robot performs the anchoring operation, the method for identifying the hole positions of the roadway roof support steel belt is mainly realized based on monocular vision or laser scanning. The monocular vision method mainly obtains the image coordinate information of the target hole positions by collecting images through the vision system and using the image target detection algorithm. The disadvantage of this method is that it is vulnerable to environmental interference. The environment in the underground coal mine is complex, with low illuminance and high dust concentration, which easily causes inaccurate image recognition and low reliability. Moreover, monocular vision recognition cannot obtain the target depth information and cannot acquire accurate position information. The laser scanning method mainly obtains the three-dimensional laser point cloud data of the target through a laser scanner or lidar, and the three-dimensional position information of the target can be obtained by extracting the point cloud data through an algorithm. However, the disadvantage of this method is that the amount of point cloud data is single, and it has sparsity and irregularity, and cannot obtain the appearance feature information of the target, making it difficult to achieve accurate target detection. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a method for positioning the drilling of the roadway roof support steel belt based on the fusion of radar and vision. The technical solution of the present invention is as follows:

[0006] A method for positioning the drilling of the roadway roof support steel belt based on the fusion of radar and vision, in which a combined sensing system of an industrial camera and a lidar is installed at the front end of the drilling arm of the drilling and anchoring robot. The method includes the following steps:

[0007] S1, collect the visual image of the roadway roof support steel belt at the current position through the industrial camera, and use the image target detection algorithm to determine the visual detection result of the drilling;

[0008] S2, scan the roadway roof at the current position through the lidar to obtain three-dimensional laser point cloud data, and use the point cloud data processing algorithm to extract the drilling boundary point cloud;

[0009] S3, use the combined calibration of the lidar and the industrial camera to project the drilling boundary point cloud into the pixel coordinate system of the industrial camera to obtain two-dimensional point cloud data;

[0010] S4, correlate the projected two-dimensional point cloud data with the visual detection result of the drilling, retain the correct drilling detection result, and fuse the drilling detection result to obtain the three-dimensional coordinate information of the drilling.

[0011] Optionally, when using the image target detection algorithm to determine the visual detection result of the drilling in S1, it includes the following steps:

[0012] S11, obtain the drilling images of the roadway roof support steel belt collected by the industrial camera at different positions and different angles, construct a drilling image data set, perform drilling annotation on the drilling image data set, and construct a complete neural network training data set;

[0013] S12. Use the neural network training dataset to train the Faster R-CNN image object detection network to obtain a network weight file, and obtain a borehole image recognition network model;

[0014] S13. Input the visual image of the roadway roof support steel strip into the borehole image recognition network model to obtain the rectangular detection frame of the borehole on the visual image of the roadway roof support steel strip and calculate the pixel coordinates of the center point of the rectangular detection frame.

[0015] Optionally, when the S2 extracts the borehole boundary point cloud by using the point cloud data processing algorithm, it includes the following steps:

[0016] S21. Use a voxel filter to filter the three-dimensional lidar point cloud data to remove noise points and outliers, and obtain the filtered three-dimensional lidar point cloud data;

[0017] S22. Use the RANSAC algorithm to fit the plane of the filtered three-dimensional lidar point cloud data to remove the roof background point cloud and the internal point cloud of the borehole, and retain the steel strip point cloud data;

[0018] S23. Use the point cloud normal estimation method to extract the boundary of the steel strip point cloud to obtain the steel strip boundary point cloud and the borehole boundary point cloud;

[0019] S24. Use the kd-tree algorithm to cluster and segment the steel strip boundary point cloud and the borehole boundary point cloud to obtain the borehole boundary point cloud.

[0020] Optionally, when the S3 projects the borehole boundary point cloud to the pixel coordinate system of the industrial camera by using the joint calibration of the lidar and the industrial camera for the external parameter matrix, it includes the following steps:

[0021] S31. Fix the installation positions of the lidar and the industrial camera, and adjust the acquisition frequencies of the industrial camera and the lidar to be consistent to ensure the spatial and temporal consistency of the sampling data;

[0022] S32. Use the industrial camera calibration algorithm to calculate the internal parameter matrix of the industrial camera, and use the joint calibration algorithm of the lidar and the industrial camera to obtain the external parameter matrix. The external parameter matrix includes a rotation matrix and a translation matrix;

[0023] S33. Project the borehole boundary point cloud obtained by the lidar to the pixel coordinate system of the industrial camera by using the rotation matrix, the translation matrix, and the internal parameter matrix of the industrial camera.

[0024] Optionally, when the S4 correlates the projected two-dimensional point cloud data with the borehole visual detection result, retains the correct borehole detection result, and fuses the borehole detection result to obtain the three-dimensional coordinate information of the borehole, it includes the following steps:

[0025] S41. Match and associate the rectangular detection frame of the drilled hole in the visual inspection result of the drilled hole with the two-dimensional point cloud data. If no less than 80% of the points in the two-dimensional point cloud data fall within any rectangular detection frame, it is determined that the association is correct. The rectangular detection frame and the boundary point cloud are both the detection results of the drilled hole, and the next step can be executed; otherwise, it is determined that the association is incorrect, and this fusion is abandoned and waiting for the next detection result.

[0026] S42. Reverse-project all the point clouds within the rectangular detection frame into the camera coordinate system, calculate the average depth value as the depth value of the drilled hole relative to the industrial camera, and then convert the pixel coordinates of the center point of the rectangular detection frame to the camera coordinate system through coordinate transformation to obtain the three-dimensional coordinate information of the drilled hole, realizing the drilling positioning of the roof support steel strip in the mine roadway.

[0027] Optionally, when the S22 uses the RANSAC algorithm to fit the plane of the filtered three-dimensional laser point cloud data to remove the roof background point cloud and the internal point cloud of the drilled hole and retain the steel strip point cloud data, it includes the following steps:

[0028] S221. Randomly select three points in the filtered three-dimensional laser point cloud data, calculate the corresponding plane model Ax + By + Cz + D = 0, and calculate the algebraic distance di = |Axi + Byi + Czi + D| from all points in the filtered three-dimensional laser point cloud data to this plane; set a threshold t. If di ≤ t, it is determined that the point is an inlier of this plane model, otherwise it is determined that the point is an outlier of this plane model; where t is an empirical value.

[0029] S222. Repeat S221 and compare the number of inliers of each plane model, and select the plane model with the largest number of inliers as the best fitting plane of the steel strip fitting plane.

[0030] S223. Judge the distance from the filtered three-dimensional laser point cloud data to the best fitting plane. If the distance from any point to the best fitting plane is less than the threshold t, it is considered that it belongs to the steel strip point cloud data and is retained. If the distance from any point to the best fitting plane is not less than the threshold t, it is removed, and finally the steel strip point cloud data is obtained.

[0031] Optionally, when the S23 uses the point cloud normal estimation method to extract the boundary of the steel strip point cloud data to obtain the steel strip boundary point cloud and the drilled hole boundary point cloud, it includes the following steps:

[0032] S231. Randomly select a point p from the steel strip point cloud data, set k1 points adjacent to the point p to form a k1 neighborhood, the point p and its k1 adjacent points form a neighborhood set N, use the least squares method to fit a micro-plane for all points in the neighborhood set N, and calculate the normal vector of this micro-plane at the point p.

[0033] S232. Repeat S231 to traverse all the steel strip point cloud data and obtain the normal vectors of all the steel strip point cloud data.

[0034] S233. Calculate the angles between the normal vector of point p and the normal vectors of the other points in the neighborhood set N. And set an angle threshold θ, where θ is an empirical value. If or then the point corresponding to this normal is considered a boundary point and retained, otherwise it is removed as an interior point. Repeat the above steps to traverse all the steel strip point cloud data to obtain the steel strip boundary point cloud and the drilling boundary point cloud.

[0035] Optionally, when using the kd - tree algorithm to cluster and segment the steel strip boundary point cloud and the drilling boundary point cloud to obtain the drilling boundary point cloud, S24 includes the following steps:

[0036] S241. Randomly select a point q from the steel strip boundary point cloud and the drilling boundary point cloud, set a k2 - neighborhood set M consisting of k2 points adjacent to point q, calculate the Euclidean distance r from point q to each point in its k2 - neighborhood set M, and set a threshold s. If r ≤ s, then put it into the clustering set Qi; s is an empirical value.

[0037] S242. Repeat S241 to traverse the steel strip boundary point cloud and the drilling boundary point cloud until the points in the clustering set Qi no longer increase, and finally obtain the clustering sets Q1 and Q2, where the clustering set with the smaller number is the drilling boundary point cloud.

[0038] All the above - mentioned optional technical solutions can be combined arbitrarily, and the present invention does not elaborate on the structures after combination one by one.

[0039] By means of the above - mentioned solution, the beneficial effects of the present invention are as follows:

[0040] By using the image target detection algorithm to determine the drilling visual detection result of the industrial camera, and using the point cloud data processing algorithm to extract the three - dimensional laser point cloud data obtained by lidar scanning to obtain the drilling boundary point cloud. Then, by using the jointly calibrated external parameter matrix of the lidar and the industrial camera to project the drilling boundary point cloud into the pixel coordinate system of the industrial camera to obtain the two - dimensional point cloud data, and then correlating the two - dimensional point cloud data with the drilling visual detection result and fusing the drilling detection result to obtain the three - dimensional coordinate information of the drilling, a roadway roof support steel strip drilling positioning method based on lidar and vision fusion is provided. This method combines the respective advantages of visual and lidar target detection, matches and fuses the target feature point information of images and point clouds, avoids the problems of false detection and missed detection that may occur in a single sensor, improves the accuracy of target detection, and uses the accurate three - dimensional coordinate information of the lidar to endow the feature of the drilling visual detection result with depth information, realizing accurate identification and precise positioning of the drilling.

[0041] In summary, the embodiments of the present invention have the following advantages:

[0042] The present invention fuses the target detection results of image vision and lidar. Compared with using only lidar or industrial cameras for detection, it not only avoids the problems of false detection and missed detection that may occur in a single sensor, but also reduces the influence of environmental factors and can work stably in the coal mine underground with poor lighting conditions.

[0043] The image target detection algorithm based on industrial cameras in the present invention adopts deep learning methods, with high detection accuracy and good robustness in complex and harsh environments. The point cloud data processing algorithm based on lidar adopts traditional algorithms, with low complexity and high reliability.

[0044] The present invention can adjust the installation positions of the lidar and industrial cameras by adjusting the algorithms and calibration parameters, and can be migrated and applied to roadway environments with a variety of different working conditions and different support conditions, with strong adaptability.

[0045] The present invention adopts a multi-sensor fusion positioning method, with high positioning accuracy, and can further realize the automation of the bolt support operation of the drilling and anchoring robot, reduce the number of support operation personnel, and improve the coal mine tunneling efficiency.

[0046] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly and implement it in accordance with the content of the specification, the following takes the preferred embodiments of the present invention and combines with the drawings to describe in detail as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of the present invention.

[0048] Figure 2 is a schematic diagram of the implementation environment of the embodiments of the present invention.

[0049] Figure 3 is a schematic diagram of a drilling vision detection result of an embodiment of the present invention.

[0050] Figure 4 is a schematic diagram of a three-dimensional lidar point cloud data sampled by an embodiment of the present invention.

[0051] Figure 5 is a schematic diagram of the steel strip boundary point cloud and the drilling boundary point cloud extracted by an embodiment of the present invention.

[0052] Figure 6 is a schematic diagram of a coordinate transformation of an embodiment of the present invention.

[0053] Figure 7 is a schematic diagram of the joint calibration effect of the lidar and the industrial camera of an embodiment of the present invention.

[0054] Figure 8 It is a schematic diagram of the positioning result of an embodiment of the present invention. Specific embodiments

[0055] The following combines the accompanying drawings and embodiments to further describe the specific embodiments of the present invention in detail. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0056] As Figure 1 shown, the present invention provides a method for positioning the drilling of the roadway roof support steel strip based on the fusion of radar and vision. A combined sensing system of an industrial camera and a lidar is installed at the front end of the drill arm of the drill-anchoring robot. As Figure 2 shown, it is a schematic diagram of the implementation environment of the method provided by an embodiment of the present invention. A support steel strip 2 is laid on the roof 1 of the mine roadway, and circular drill holes 3 are arranged horizontally on the support steel strip 2. An industrial camera 6 and a lidar 7 are installed on the drill arm 5 of the drill-anchoring robot 4. The method for positioning the drilling of the roadway roof support steel strip based on the fusion of radar and vision provided by the present invention includes the following steps S1 to S4.

[0057] S1. Collect the visual image of the roadway roof support steel strip at the current position through the industrial camera, and use the image target detection algorithm to determine the visual detection result of the drill hole.

[0058] In specific implementation, the drill-anchoring robot 4 controls the drill arm 5 to move to directly below the position of the support steel strip 2 to be drilled in the roof 1 of the mine roadway. The industrial camera 6 works, collects the visual image of the roadway roof support steel strip at the current position, and uses the image target detection algorithm to determine the visual detection result of the drill hole for the visual image of the roadway roof support steel strip.

[0059] Optionally, when S1 uses the image target detection algorithm to determine the visual detection result of the drill hole, it includes but is not limited to being implemented through the following steps S11 to S13:

[0060] S11. Obtain the drill hole images of the roadway roof support steel strip collected by the industrial camera at different positions and different angles, construct a drill hole image data set, perform drill hole annotation on the drill hole image data set, and construct a complete neural network training data set.

[0061] Among them, when performing drill hole annotation on the drill hole image data set, it can be realized through labelImg. After the drill hole annotation is completed, the annotated drill hole image data set can be randomly divided into a training data set and a test data set according to a ratio of 8:2.

[0062] S12. Use the neural network training data set to train the FasterRCNN image target detection network, obtain the network weight file, and obtain the drill hole image recognition network model.

[0063] Specifically, the Faster R-CNN image target detection network is iteratively trained using a training dataset, and the trained borehole image recognition network model is tested and verified using a test dataset to obtain a network weight file with the highest accuracy. Of course, in addition to the Faster R-CNN image target detection network, other network models can also be used for the borehole image recognition network model, and the embodiments of the present invention do not make specific limitations in this regard.

[0064] S13. Input the visual image of the roadway roof support steel strip into the borehole image recognition network model to obtain the rectangular detection frame of the borehole on the visual image of the roadway roof support steel strip and calculate the pixel coordinates of the center point of the rectangular detection frame.

[0065] As Figure 3 shown, it is a schematic diagram of the visual detection result of a borehole.

[0066] S2. Obtain three-dimensional lidar point cloud data by scanning the roadway roof at the current position using a lidar, and extract the borehole boundary point cloud using a point cloud data processing algorithm.

[0067] In specific implementation, during specific implementation, the drilling and anchoring robot 4 controls the drill arm 5 to move to directly below the position of the steel strip 2 to be drilled in the roadway roof 1 of the mine shaft, and the lidar 7 works to fully scan and sample the roadway roof steel strip. The three-dimensional lidar point cloud data obtained by sampling is as Figure 4 shown.

[0068] Specifically, when the step S2 extracts the three-dimensional lidar point cloud data using a point cloud data processing algorithm to obtain the borehole boundary point cloud, it includes but is not limited to being implemented through the following steps S21 to S24.

[0069] S21. Use a voxel filter to filter the three-dimensional lidar point cloud data to remove noise points and outliers, and obtain the filtered three-dimensional lidar point cloud data.

[0070] Specifically, when this step uses a voxel filter to filter the three-dimensional lidar point cloud data, set the side length cell of the voxel grid, equally divide the three coordinate axes X, Y, and Z of the sampled point cloud into M, N, and L parts, then divide the three-dimensional lidar point cloud into M * N * L voxel grids, calculate the centroid of each voxel grid, that is, take the average of the sum of all data points in the voxel grid, and replace all points in the voxel grid with the centroid to achieve the filtering process of the three-dimensional lidar point cloud data.

[0071] S22. Use the RANSAC algorithm to fit a plane to the filtered three-dimensional lidar point cloud data to remove the roof background point cloud and the borehole internal point cloud, and retain the steel strip point cloud data.

[0072] In specific implementation, when S22 uses the RANSAC algorithm to fit a plane to the filtered three-dimensional laser point cloud data to remove the roof background point cloud and the internal point cloud of the drill hole and retain the steel strip point cloud data, it can be carried out in the following steps S221 to S223.

[0073] S221. Randomly select three points in the filtered three-dimensional laser point cloud data, calculate the corresponding plane model Ax + By + Cz + D = 0, and calculate the algebraic distance di = |Axi + Byi + Czi + D| from all points in the filtered three-dimensional laser point cloud data to this plane; set a threshold t. If di ≤ t, determine that this point is an inlier of this plane model, otherwise determine that this point is an outlier of this plane model; where t is an empirical value.

[0074] S222. Repeat S221, compare the number of inliers of each plane model, and select the plane model with the largest number of inliers as the best fitting plane for the steel strip fitting plane.

[0075] S223. Judge the distance from the filtered three-dimensional laser point cloud data to the best fitting plane. If the distance from any point to the best fitting plane is less than the threshold t, it is considered to belong to the steel strip point cloud data and is retained. If the distance from any point to the best fitting plane is not less than the threshold t, it is removed, and finally the steel strip point cloud data is obtained.

[0076] In addition, the RANSAC algorithm can also be replaced by other plane fitting algorithms, such as the least squares method, etc.

[0077] S23. Use the point cloud normal estimation method to extract the boundaries of the steel strip point cloud data to obtain the steel strip boundary point cloud and the drill hole boundary point cloud.

[0078] When S23 uses the point cloud normal estimation method to extract the boundaries of the steel strip point cloud data to obtain the steel strip boundary point cloud and the drill hole boundary point cloud, it can include the following steps S231 to S233.

[0079] S231. Randomly select a point p from the steel strip point cloud data, set k1 points adjacent to point p to form a k1 neighborhood, point p and its adjacent k1 points form a neighborhood set N, use the least squares method to fit a micro-plane to all points in the neighborhood set N, and calculate the normal vector of this micro-plane at point p.

[0080] S232. Repeat S231, traverse all the steel strip point cloud data, and obtain the normal vectors of all the steel strip point cloud data.

[0081] S233. Calculate the angle between the normal vector of point p and the normal vectors of the remaining points in the neighborhood set N and set an angle threshold θ, θ is an empirical value; if or Then, it is considered that the point corresponding to the normal line is a boundary point and is retained, otherwise it is excluded as an interior point; repeat the above steps to traverse all the steel strip point cloud data to obtain the steel strip boundary point cloud and the drilling boundary point cloud.

[0082] In addition, the point cloud normal estimation method can also be other boundary extraction algorithms, and the embodiments of the present invention do not make specific limitations thereto.

[0083] Such as Figure 5 shown, which is a schematic diagram of the steel strip boundary point cloud and the drilling boundary point cloud extracted in the embodiments of the present invention.

[0084] S24, use the kd-tree algorithm to perform clustering segmentation on the steel strip boundary point cloud and the drilling boundary point cloud to obtain the drilling boundary point cloud.

[0085] In specific implementation, when the S24 uses the kd-tree algorithm to perform clustering segmentation on the steel strip boundary point cloud and the drilling boundary point cloud to obtain the drilling boundary point cloud, it may include the following steps S241 and S242.

[0086] S241, randomly select a point q from the steel strip boundary point cloud and the drilling boundary point cloud, set a set M of k2 points adjacent to the point q to form a k2 neighborhood set, calculate the Euclidean distance r from the point q to each point in its k2 neighborhood set M, and set a threshold s; if r ≤ s, then put it into the clustering set Qi; s is an empirical value.

[0087] S242, repeat S241, traverse the steel strip boundary point cloud and the drilling boundary point cloud until the points in the clustering set Qi no longer increase, and finally obtain the clustering sets Q1 and Q2, where the clustering set with a smaller number is the drilling boundary point cloud.

[0088] Of course, in specific implementation, the kd-tree algorithm can also be replaced by other clustering algorithms, and the embodiments of the present invention do not make specific limitations thereto.

[0089] S3, use the joint calibration of the external parameter matrix of the lidar and the industrial camera to project the drilling boundary point cloud onto the pixel coordinate system of the industrial camera to obtain two-dimensional point cloud data.

[0090] In specific implementation, when the S3 uses the joint calibration of the external parameter matrix of the lidar and the industrial camera to project the drilling boundary point cloud onto the pixel coordinate system of the industrial camera, it may include the following steps S31 to S33.

[0091] S31, fix the installation positions of the lidar and the industrial camera, and adjust the acquisition frequencies of the industrial camera and the lidar to be consistent to ensure the spatial and temporal consistency of the sampled data.

[0092] Specifically in the adjustment, the sampling frequency of the industrial camera 6 and the lidar 7 can be adjusted to be consistent with the sampling frequency of the lidar 7 as a reference to achieve the spatial and temporal synchronization of the sampled data.

[0093] S32. Calculate the internal parameter matrix of the industrial camera using the industrial camera calibration algorithm, and obtain the external parameter matrix using the joint calibration algorithm of the lidar and the industrial camera. The external parameter matrix includes a rotation matrix and a translation matrix.

[0094] Specifically, the Zhang Zhengyou calibration method, a camera calibration algorithm, can be used to calibrate the industrial camera to obtain the internal parameter matrix of the industrial camera, that is, the projection matrix, to realize the conversion of data between the camera coordinate system and the pixel coordinate system.

[0095] S33. Project the drilled hole boundary point cloud obtained by the lidar onto the pixel coordinate system of the industrial camera using the rotation matrix, the translation matrix, and the internal parameter matrix of the industrial camera.

[0096] When specifically projecting in S33, first transform the drilled hole boundary point cloud obtained in step S2 to the camera coordinate system using the matrix and the translation matrix, and then project the points of the drilled hole boundary point cloud in the camera coordinate system onto the pixel coordinates of the industrial camera using the internal parameter matrix of the industrial camera, so as to realize the projection of the drilled hole boundary point cloud onto the pixel coordinate system of the industrial camera. As Figure 6 shown, it is a schematic diagram of a coordinate transformation according to an embodiment of the present invention. Figure 6 In it, (X L , Y L , Z L ) is the coordinate point in the lidar coordinate system; (X C , Y C , Z C ) is the coordinate point in the camera coordinate system; (u, v) is the coordinate point in the pixel coordinate system; matrix A is the internal parameter matrix of the industrial camera, where (f x , f y ) is the focal length, and (u0, v0) is the pixel coordinate of the image center; matrix B is the external parameter matrix, where R is the rotation matrix and T is the translation matrix. As Figure 7 shown, it is a schematic diagram of the joint calibration effect of the lidar and the industrial camera.

[0097] S4. Correlate the projected two-dimensional point cloud data with the drilled hole visual inspection results, retain the correct drilled hole inspection results, and fuse the drilled hole inspection results to obtain the three-dimensional coordinate information of the drilled hole.

[0098] When specifically implementing, when S4 correlates the projected two-dimensional point cloud data with the drilled hole visual inspection results, retains the correct drilled hole inspection results, and fuses the drilled hole inspection results to obtain the three-dimensional coordinate information of the drilled hole, it may include the following steps S41 and S42.

[0099] S41. Match and associate the rectangular detection frame of the drill hole in the visual inspection result of the drill hole with the two-dimensional point cloud data. If no less than 80% of the points in the two-dimensional point cloud data fall within any rectangular detection frame, it is determined that the association is correct. The rectangular detection frame and the boundary point cloud are both the detection results of the drill hole, and the next step can be executed; otherwise, it is determined that the association is incorrect, and this fusion is abandoned and waiting for the next detection result.

[0100] S42. Reverse-project all the point clouds within the rectangular detection frame into the camera coordinate system, calculate the average depth value as the depth value of the drill hole relative to the industrial camera, and then convert the pixel coordinates of the center point of the rectangular detection frame to the camera coordinate system through coordinate transformation to obtain the three-dimensional coordinate information of the drill hole, realizing the drill hole positioning of the roof support steel strip in the mine roadway.

[0101] As Figure 8 shown, it is a schematic diagram of the positioning result of an embodiment of the present invention.

[0102] The present invention uses a combined sensing system based on a camera and a lidar to realize the drill hole positioning of the roof support steel strip in the mine roadway, with high stability and higher accuracy of the detection result. The image target detection algorithm is used to identify the anchor hole attributes and obtain the position of the rectangular detection frame, and the point cloud data processing algorithm of the lidar is used to obtain the drill hole boundary point cloud. The detection results of the lidar and the camera are fused to detect, identify and position the drill hole of the roof support steel strip in the mine roadway, and accurate three-dimensional coordinate information of the drill hole can be obtained.

[0103] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. It should be pointed out that for those of ordinary skill in the art of this technology, without departing from the technical principle of the present invention, several improvements and modifications can still be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for positioning the drilling of the roadway roof support steel strip based on the fusion of radar and vision, characterized in that, An integrated sensing system of an industrial camera and a lidar is installed at the front end of the drill arm of a drilling and anchoring robot. The method includes the following steps: S1. Collect a visual image of the roadway roof support steel strip at the current position through the industrial camera, and use an image target detection algorithm to determine the visual detection result of the drill hole; S2. Scan the roadway roof at the current position through the lidar to obtain three-dimensional lidar point cloud data, and use a point cloud data processing algorithm to extract the drill hole boundary point cloud; S3. Use the lidar and the industrial camera to jointly calibrate the external parameter matrix to project the drill hole boundary point cloud into the pixel coordinate system of the industrial camera to obtain two-dimensional point cloud data; S4. Associate the projected two-dimensional point cloud data with the visual detection result of the drill hole, retain the correct drill hole detection result, and fuse the drill hole detection result to obtain the three-dimensional coordinate information of the drill hole; When S4 associates the projected two-dimensional point cloud data with the visual detection result of the drill hole, retains the correct drill hole detection result, and fuses the drill hole detection result to obtain the three-dimensional coordinate information of the drill hole, it includes the following steps: S41. Match and associate the rectangular detection frame of the drill hole in the visual detection result of the drill hole with the two-dimensional point cloud data. If no less than 80% of the points in the two-dimensional point cloud data fall within any rectangular detection frame, it is determined that the association is correct. The rectangular detection frame and the boundary point cloud are both drill hole detection results, and the next step can be executed; otherwise, it is determined that the association is incorrect, and the current fusion is abandoned and waiting for the next detection result; S42. Reverse-project all the point clouds within the rectangular detection frame into the camera coordinate system, calculate the average depth value as the depth value of the drill hole relative to the industrial camera, and then convert the pixel coordinates of the center point of the rectangular detection frame to the camera coordinate system through coordinate transformation to obtain the three-dimensional coordinate information of the drill hole, realizing the drill hole positioning of the roadway roof support steel strip in the mine.

2. The method for positioning the drilling of the roadway roof support steel strip based on the fusion of radar and vision according to claim 1, wherein When S1 uses an image target detection algorithm to determine the visual detection result of the drill hole, it includes the following steps: S11. Obtain the drill hole images of the roadway roof support steel strip collected by the industrial camera at different positions and angles, construct a drill hole image data set, perform drill hole annotation on the drill hole image data set, and construct a complete neural network training data set; S12. Use the neural network training data set to train the Faster RCNN image target detection network to obtain a network weight file and obtain a drill hole image recognition network model; S13. Input the visual image of the roadway roof support steel strip into the drill hole image recognition network model to obtain the rectangular detection frame of the drill hole on the visual image of the roadway roof support steel strip and calculate the pixel coordinates of the center point of the rectangular detection frame.

3. The roadway roof support steel strip drilling positioning method based on radar and vision fusion according to claim 1, characterized in that When S2 uses a point cloud data processing algorithm to extract the drill hole boundary point cloud, it includes the following steps: S21. Use a voxel filter to filter the three-dimensional lidar point cloud data to remove noise points and outlier points to obtain the filtered three-dimensional lidar point cloud data; S22. Use the RANSAC algorithm to fit the filtered three-dimensional lidar point cloud data to remove the roof background point cloud and the drill hole internal point cloud, and retain the steel strip point cloud data; S23. Use a point cloud normal estimation method to extract the boundary of the steel strip point cloud data to obtain the steel strip boundary point cloud and the drill hole boundary point cloud; S24. Use the kd-tree algorithm to cluster and segment the steel strip boundary point cloud and the drilling boundary point cloud to obtain the drilling boundary point cloud.

4. The method for positioning the drilling of the roadway roof support steel strip based on the fusion of radar and vision according to claim 1, characterized in that, When the step S3 projects the drilling boundary point cloud onto the pixel coordinate system of the industrial camera by using the external parameter matrix jointly calibrated by the lidar and the industrial camera, it includes the following steps: S31. Fix the installation positions of the lidar and the industrial camera, and adjust the acquisition frequencies of the industrial camera and the lidar to be consistent to ensure the spatial and temporal consistency of the sampled data. S32. Use the industrial camera calibration algorithm to calculate the internal parameter matrix of the industrial camera, and use the joint calibration algorithm of the lidar and the industrial camera to obtain the external parameter matrix, where the external parameter matrix includes a rotation matrix and a translation matrix. S33. Project the drilling boundary point cloud obtained by the lidar onto the pixel coordinate system of the industrial camera by using the rotation matrix, the translation matrix, and the internal parameter matrix of the industrial camera.

5. The method for positioning the drilling of the roadway roof support steel strip based on the fusion of radar and vision according to claim 3, characterized in that When the step S22 fits and removes the roof background point cloud and the internal point cloud of the drilling from the filtered three-dimensional lidar point cloud data by using the RANSAC algorithm and retains the steel strip point cloud data, it includes the following steps: S221. Randomly select three points in the filtered three-dimensional lidar point cloud data, calculate the corresponding plane model Ax + By + Cz + D = 0, and calculate the algebraic distance di = |Axi + Byi + Czi + D| from all points in the filtered three-dimensional lidar point cloud data to this plane; set a threshold t. If di ≤ t, then determine that this point is an inlier of this plane model, otherwise determine that this point is an outlier of this plane model; where t is an empirical value. S222. Repeat S221, and compare the number of inliers of each plane model, and select the plane model with the largest number of inliers as the best fitting plane of the steel strip fitting plane. S223. Judge the distance from the filtered three-dimensional lidar point cloud data to the best fitting plane. If the distance from any point to the best fitting plane is less than the threshold t, then consider it to belong to the steel strip point cloud data and retain it. If the distance from any point to the best fitting plane is not less than the threshold t, then remove it. Finally, obtain the steel strip point cloud data.

6. The method for positioning the drilling of the roadway roof support steel strip based on the fusion of radar and vision according to claim 3, wherein When the step S23 extracts the boundary of the steel strip point cloud data by using the point cloud normal estimation method to obtain the steel strip boundary point cloud and the drilling boundary point cloud, it includes the following steps: S231. Randomly select a point p from the steel strip point cloud data, set k1 points adjacent to the point p to form a k1 neighborhood, the point p and its k1 adjacent points form a neighborhood set N, use the least squares method to fit a micro-plane for all points in the neighborhood set N, and calculate the normal vector of this micro-plane at the point p. S232. Repeat S231, traverse all the steel strip point cloud data, and obtain the normal vectors of all the steel strip point cloud data. S233. Calculate the angle ji between the normal vector of the point p and the normal vectors of the other points in the neighborhood set N, and set an angle threshold q, where q is an empirical value; if ji > q or ji < q, then consider the point corresponding to this normal line as a boundary point and retain it, otherwise remove it as an inlier; repeat the above steps, traverse all the steel strip point cloud data, and obtain the steel strip boundary point cloud and the drilling boundary point cloud.

7. The method for positioning the drilling of the roadway roof support steel strip based on the fusion of radar and vision according to claim 3, wherein, When using the kd-tree algorithm to cluster and segment the steel strip boundary point cloud and the drilling boundary point cloud to obtain the drilling boundary point cloud, S24 includes the following steps: S241: Randomly select a point q from the steel strip boundary point cloud and the drilling boundary point cloud. Set a k2-neighborhood set M consisting of k2 points adjacent to point q. Calculate the Euclidean distance r from point q to each point in its k2-neighborhood set M. Set a threshold s. If r ≤ s, put it into the clustering set Qi. s is an empirical value. S242: Repeat S241, traverse the steel strip boundary point cloud and the drilling boundary point cloud until the points in the clustering set Qi no longer increase. Finally, obtain the clustering sets Q1 and Q2, where the clustering set with the smaller quantity is the drilling boundary point cloud.

Images