Coal mine underground drill rod counting method based on multi-source sensing data fusion

Abstract

The invention relates to a coal mine underground drill rod counting method based on multi-source sensing data fusion, and belongs to the technical field of coal mine intellectualization. Comprising the steps that an infrared image is input into a first drilling machine state recognition network, and the first drilling machine state recognition network outputs the current state of a drilling machine; framing processing is carried out on the video data to obtain a plurality of original images, each original image is input into a drilling machine target detection network, and the drilling machine target detection network outputs a target frame where a drilling machine is located; the original laser radar point cloud and the original image are subjected to time alignment and space alignment, the laser point cloud marked with the drilling machine state information is converted into a laser mapping image with the drilling machine state information, the laser mapping image is input into a second drilling machine state recognition network, and the second drilling machine state recognition network outputs the current state of the drilling machine; and according to the current drilling machine states output by the first drilling machine state recognition network and the second drilling machine state recognition network, whether one is added to the drill rod count or not is determined. The counting accuracy of the drill rod can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent coal mines, and in particular to a method for counting drill rods in coal mines based on multi-source sensor data fusion. Background Art

[0002] At present, the mining depth of coal mines is constantly increasing, and the gas content of coal seams is also significantly increasing with the increase of mining depth, which makes gas outburst disasters more likely to occur. Pre-extraction of coal seam gas is the main means to prevent gas outburst disasters. Pre-extraction of coal seam gas is mainly implemented by drilling underground. This method can effectively reduce the gas content and pressure of coal seams, but there are certain requirements for the depth of the drilling hole. Generally, the depth of the drilling hole is indirectly calculated by counting the number of drill rods driven into the coal seam.

[0003] The main methods for counting drill rods of traditional mine drilling rigs include manual counting method and sensor counting method. In the manual counting method, the staff observes and records the number of drill rods on site, but long-term work can easily make people tired, resulting in counting errors, and it is time-consuming and labor-intensive, with a low level of automation. The sensor counting method is to install sensors on the drilling rig's mechanical arm, and the electric control box collects sensor data and performs data preprocessing to obtain the number of drill rods. However, in harsh environments such as dust and humidity, the sensors are easily damaged, resulting in counting errors, making the drill rod counting accuracy low. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for counting drill rods in coal mines based on multi-source sensor data fusion. The technical solution of the present invention is as follows:

[0005] A method for counting drill rods in coal mines based on multi-source sensor data fusion, comprising:

[0006] S1, obtains infrared images, video data and original LiDAR point clouds collected during the underground drilling rig construction in the coal mine in the current sampling period;

[0007] S2, inputting the collected infrared image into a pre-trained first drilling rig state recognition network, and the first drilling rig state recognition network outputs the current state of the drilling rig;

[0008] S3, performing frame processing on the collected video data to obtain multiple original images, inputting each original image into a pre-trained drilling rig target detection network, and the pre-trained drilling rig target detection network outputs a target frame where the drilling rig is located;

[0009] S4, performing time alignment and spatial alignment on the collected original laser radar point cloud and the original image to obtain the laser point cloud in the pixel coordinate system;

[0010] S5, marking the laser point cloud in the pixel coordinate system according to the target frame where the drilling rig is located, and obtaining the laser point cloud with the drilling rig status information;

[0011] S6, converting the laser point cloud with the drilling rig status information into a laser mapping image with the drilling rig status information, and inputting the laser mapping image into a pre-trained second drilling rig status recognition network, so that the second drilling rig status recognition network outputs the current status of the drilling rig;

[0012] S7, determining whether to add one to the drill rod count according to the current state of the drill rig output by the first drill rig state identification network and the current state of the drill rig output by the second drill rig state identification network in the current sampling period.

[0013] Optionally, the first drilling rig state recognition network is an improved YOLOv5s model, and the drilling rig target detection network and the second drilling rig state recognition network are both YOLOv5s models; the improved YOLOv5s model includes a backbone network, a neck network and a detection head connected in sequence, the backbone network includes CBS, iRMB*N1, iRMB*N2, iRMB*N3, iRMB*N4, SPPF and RFEC3 connected in sequence, and the detection head adopts a dynamic detection head.

[0014] Optionally, in a specific implementation of S2, the acquired infrared image is scaled according to the size of the input image of the first drilling rig state recognition network, and the scaled infrared image is input into the pre-trained first drilling rig state recognition network.

[0015] Optionally, in the specific implementation of S3, the video data collected in 1s is divided into three frames of original images, and each frame of the original image is scaled according to the size of the input image of the drilling rig target detection network, and the scaled original image is input into a pre-trained drilling rig target detection network.

[0016] Optionally, when performing spatial alignment, S4 includes:

[0017] S41, performing dedistortion processing on the collected original laser radar point cloud;

[0018] S42, projecting the dedistorted LiDAR point cloud to a camera coordinate system using a homogeneous transformation matrix between the video data and the original LiDAR point cloud;

[0019] S43, projecting the laser radar point cloud in the camera coordinate system into the pixel coordinate system using the camera intrinsic parameters to obtain the laser point cloud in the pixel coordinate system.

[0020] Optionally, during the specific implementation of S5, the laser point cloud in the pixel coordinate system is marked according to the coordinates of the target frame where the drilling rig is located in the pixel coordinate system to obtain the laser point cloud with the drilling rig status information.

[0021] Optionally, the current state of the drilling rig includes a rod-up state and a rod-down state. When S7 is specifically implemented, it includes:

[0022] S71, determining whether the current state of the drilling rig output by the first drilling rig state recognition network and the current state of the drilling rig output by the second drilling rig state recognition network in the current sampling period are transitioning from the rod-up state to the rod-down state;

[0023] S72, if at least one of the current state of the drilling rig output by the first drilling rig state identification network and the current state of the drilling rig output by the second drilling rig state identification network in the current sampling period is transitioned from the rod-on state to the rod-off state, determine to increase the drill rod count by one.

[0024] All the above optional technical solutions can be combined arbitrarily, and the present invention does not provide detailed descriptions of the structures after the combinations.

[0025] By means of the above scheme, the beneficial effects of the present invention are as follows:

[0026] By obtaining infrared images, video data and original lidar point clouds collected during the construction of underground drilling rigs in coal mines during the current sampling period, and identifying the status of drilling rig construction based on these three types of data, the drill rod counting can be supplemented by data collected by different sensors. This can not only improve the accuracy of drill rod counting under complex lighting conditions, but also significantly improve the automation and accuracy of drill rod counting, further improving work efficiency and personnel safety.

[0027] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention provides a flowchart of a method for counting drill rods in underground coal mines based on multi-source sensor data fusion.

[0029] Figure 2 It is a schematic diagram of the composition structure of the improved YOLOv5s model in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0031] The method for counting drill rods in coal mines based on multi-source sensor data fusion provided by the embodiment of the present invention can be implemented by any electronic device with computing function, such as a PC, a mobile terminal or a server. Figure 1 As shown, the method for counting drill rods in a coal mine based on multi-source sensor data fusion provided by the embodiment of the present invention includes the following steps S1 to S7:

[0032] S1, obtains the infrared images, video data and original lidar point clouds collected during the underground drilling rig construction in the coal mine in the current sampling period.

[0033] It should be noted that before executing the method provided in the embodiment of the present invention, it is necessary to first install three types of sensors, namely infrared imaging equipment, camera and laser radar, in the working scene of the underground drilling rig in the coal mine, and the three types of sensors need to maintain the same angle to ensure that the three sensors can collect the drilling rig operation data. Specifically, the infrared imaging equipment, camera and laser radar are all electrically connected to the electronic device. After the infrared imaging equipment, camera and laser radar collect infrared images, video data and original laser radar point clouds respectively, they are sent to the electronic device. The electronic device receives the data sent by the infrared imaging equipment, camera and laser radar to obtain the infrared images, video data and original laser radar point clouds collected during the construction of the underground drilling rig in the coal mine in the current sampling period.

[0034] The specific duration of the sampling cycle depends on the time it takes for the drilling rig to complete a rod loading and unloading operation. For example, if the drilling rig takes 3 seconds to complete a rod loading and unloading operation, the sampling cycle should be 3 seconds or slightly longer than 3 seconds.

[0035] Specifically, the number of infrared images and original lidar point clouds collected during the underground drilling construction in the coal mine in the current sampling period are related to the sampling frequencies of the infrared imaging equipment and the lidar, respectively. For example, if the infrared imaging equipment and the lidar are set to collect data nine times in one sampling period, then the number of infrared images and original lidar point clouds obtained in one sampling period are both nine.

[0036] S2, inputting the collected infrared image into a pre-trained first drilling rig state recognition network, and the first drilling rig state recognition network outputs the current state of the drilling rig.

[0037] It should be noted that, since multiple infrared images will be collected within a sampling cycle, when this step S2 is implemented, each infrared image collected in the current sampling cycle is input into a pre-trained first drilling rig state recognition network, and for each infrared image, the pre-trained first drilling rig state recognition network outputs the current state of the drilling rig.

[0038] It should also be noted that before the infrared image is input into the pre-trained first drilling rig state recognition network, the first drilling rig state recognition network needs to be trained first. For the specific implementation method of training the first drilling rig state recognition network, please refer to the existing classification network training method, which will not be elaborated in detail in the embodiment of the present invention.

[0039] In a specific embodiment, since the pre-trained first drilling rig state recognition network often has requirements for the size of the input image, when S2 is implemented, it is necessary to first scale the infrared image according to the size of the input image of the first drilling rig state recognition network, and input the scaled infrared image into the pre-trained first drilling rig state recognition network.

[0040] Specifically, when scaling an infrared image, it is necessary to first determine the scaling factor r, which can be calculated using the following formula:

[0041]

[0042] Among them, w o and h o They represent the size of the first rig state recognition network input image, w i and h i Respectively represent the size of the infrared image.

[0043] The specific composition structure of the first drilling rig status recognition network is not specifically limited in the embodiment of the present invention, and a conventional classification network can be used. However, in the specific implementation, due to the low contrast, lack of texture information, and less target feature information of infrared images, it will lead to difficulties in drilling rig recognition and poor recognition effect. In order to improve the recognition effect, the embodiment of the present invention improves the YOLOv5s model based on the characteristics of infrared images, that is, the first drilling rig status recognition network described in the embodiment of the present invention is an improved YOLOv5s model.

[0044] like Figure 2 As shown, the improved YOLOv5s model in the embodiment of the present invention includes a backbone network, a neck network and a detection head connected in sequence.

[0045] The backbone network includes CBS, iRMB*N1, iRMB*N2, iRMB*N3, iRMB*N4, SPPF and RFEC3 connected in sequence. The backbone network includes four efficient iRMB (Inverted Residual Mobile Block), which can improve the recognition accuracy of the network while ensuring that the number of network parameters and the amount of calculation are not increased.

[0046] The detection head adopts a dynamic detection head. The dynamic detection head applies scale-aware, space-aware and task-aware attention mechanisms to the feature tensor of each dimension. Based on task-aware, scale-aware and space-aware, the attention mechanism between feature layers is used for scale-aware, the attention mechanism between spatial positions is used for space-aware, and the attention mechanism within the output channel is used for task-aware, which makes the network focus more on the targets associated with the detection task and can improve the feature expression ability of the model.

[0047] Through experimental tests, it is found that the improved YOLOv5s model has a more stable training curve, higher detection accuracy, faster convergence speed, and achieves better target recognition effect than the YOLOv5s model.

[0048] S3, performing frame processing on the collected video data to obtain multiple original images, inputting each original image into a pre-trained drilling rig target detection network, and the pre-trained drilling rig target detection network outputs a target frame where the drilling rig is located.

[0049] In a specific embodiment, the specific number of frames during the frame processing can be consistent with the sampling frequency of the infrared imaging device and the laser radar. For example, if the infrared imaging device and the laser radar collect data three times in 1s, then when the video data is framed, the video data collected in 1s can be divided into three frames of original images. That is to say, when S3 is implemented, the video data collected in 1s is divided into three frames of original images, and each frame of the original image is scaled according to the size of the input image of the drilling rig target detection network, and the scaled original image is input into the pre-trained drilling rig target detection network. Among them, when scaling the original image, the method of scaling the infrared image described in the above step S2 can be adopted, and the embodiment of the present invention will not be repeated here.

[0050] The embodiment of the present invention does not specifically limit the specific composition structure of the drilling rig target detection network, and a conventional classification network can be used. For example, the drilling rig target detection network uses the YOLOv5s model. It should be noted that before the scaled original image is input into the pre-trained drilling rig target detection network, the drilling rig target detection network needs to be trained first. For the specific implementation method of training the drilling rig target detection network, please refer to the existing training method of the classification network, which will not be elaborated in detail in the embodiment of the present invention.

[0051] S4, aligning the collected original laser radar point cloud with the original image in time and space to obtain the laser point cloud in the pixel coordinate system.

[0052] In a specific embodiment, when S4 is implemented, time alignment is performed first, and then space alignment is performed. When performing time alignment, the sampling frequencies of the infrared imaging device and the laser radar are set to be the same, and the number of frames for framing the video data is set to be the same as the number of sampling points of the infrared imaging device and the laser radar within 1 second, and then the infrared imaging device, the camera, and the laser radar are controlled to sample at the same time.

[0053] On the basis of the above time alignment, step S4 may include the following steps S41 to S43 when performing spatial alignment:

[0054] S41, performing dedistortion processing on the collected original lidar point cloud.

[0055] Among them, the purpose of dedistortion processing is to remove some obvious discrete points and noise points in the original lidar point cloud to reduce the amount of data processing.

[0056] S42, projecting the dedistorted LiDAR point cloud to the camera coordinate system using the homogeneous transformation matrix between the video data and the original LiDAR point cloud.

[0057] Specifically, when projecting the dedistorted LiDAR point cloud to the camera coordinate system, it can be achieved by the following formula:

[0058]

[0059] Among them, P L is the coordinate of the dedistorted LiDAR point cloud in the LiDAR coordinate system, P L =[x L ,y L ,y L ] T ;P C is the coordinate of the dedistorted LiDAR point cloud in the camera coordinate system, P C =[x C ,y C , z C ] T ; T is the homogeneous transformation matrix between the video data and the original lidar point cloud, which consists of a 3×3 rotation matrix R and a 1×3 translation vector t. R and t can be determined with the assistance of external positioning equipment.

[0060] S43, projecting the laser radar point cloud in the camera coordinate system into the pixel coordinate system using the camera intrinsic parameters to obtain the laser point cloud in the pixel coordinate system.

[0061] This step can be implemented by the following formula:

[0062]

[0063] Among them, P uv is the coordinate of the laser point cloud in the pixel coordinate system; z c 、f x 、f y and c x 、c y They are all camera internal parameters and can be determined by the camera calibration algorithm.

[0064] S5, marking the laser point cloud in the pixel coordinate system according to the target frame where the drilling rig is located, and obtaining the laser point cloud with the drilling rig status information.

[0065] Specifically, the purpose of this step is to filter out point clouds with drilling rig features from the laser point cloud in the pixel coordinate system.

[0066] In a specific embodiment, when S5 is implemented, the laser point cloud in the pixel coordinate system is marked according to the coordinates of the target frame where the drilling rig is located in the pixel coordinate system to obtain the laser point cloud with the drilling rig status information. Specifically, the marking can be implemented by annotation software.

[0067] S6, converting the laser point cloud with the drilling rig status information into a laser mapping image with the drilling rig status information, and inputting the laser mapping image into a pre-trained second drilling rig status recognition network, and the second drilling rig status recognition network outputs the current status of the drilling rig.

[0068] This step can be implemented through the LIO-SAM algorithm in the SLAM algorithm. After feature extraction, factor graph optimization, map construction and loop detection, the final laser mapping image is obtained, realizing the visualization of the drilling rig in the lidar data.

[0069] It should be noted that before the laser mapping image is input into the pre-trained second drilling rig state recognition network, the second drilling rig state recognition network also needs to be trained first, and the training process is not specifically described in this embodiment of the present invention.

[0070] The specific composition structure of the second drilling rig state recognition network is not specifically limited in the embodiment of the present invention, and a conventional classification network can be used. For example, the second drilling rig state recognition network uses a YOLOv5s model. It should be noted that when the laser mapping image is input into the pre-trained second drilling rig state recognition network, the laser mapping image also needs to be scaled according to the size of the input image of the second drilling rig state recognition network.

[0071] S7, determining whether to add one to the drill rod count according to the current state of the drill rig output by the first drill rig state identification network and the current state of the drill rig output by the second drill rig state identification network in the current sampling period.

[0072] The current state of the drilling rig includes a rod loading state and a rod unloading state. In the method provided by the embodiment of the present invention, the first drilling rig state recognition network outputs a current state of the drilling rig, and the second drilling rig state recognition network outputs a current state of the drilling rig, and the current states of the two drilling rigs complement each other to improve the accuracy of drilling rig state recognition and the accuracy of drill rod counting.

[0073] In a specific embodiment, since a complete rod-raising action of the drilling rig includes the whole process from rod-raising to rod-removing, and different current states of the drilling rig can be identified based on the data collected by the three types of sensors within a sampling period, for example, if the sampling period is 3s, and the three types of sensors 1 obtain nine data, then within one sampling period, the first drilling rig state recognition network and the second drilling rig state recognition network will output nine current states of the drilling rig respectively. The embodiment of the present invention determines whether to add one to the drill rod count based on the current states of multiple drilling rigs output by the first drilling rig state recognition network and the second drilling rig state recognition network.

[0074] When S7 is specifically implemented, it includes: S71, determining whether the current state of the drilling rig output by the first drilling rig state identification network and the current state of the drilling rig output by the second drilling rig state identification network in the current sampling period are transitioned from the rod-up state to the rod-down state. S72, if at least one of the current state of the drilling rig output by the first drilling rig state identification network and the current state of the drilling rig output by the second drilling rig state identification network in the current sampling period is transitioned from the rod-up state to the rod-down state, then determining to increase the drill rod count by one.

[0075] That is to say, there are three situations in which step S7 determines to increase the drill rod count by one:

[0076] (1) In the current sampling period, the current state of the drilling rig output by the first drilling rig state recognition network and the current state of the drilling rig output by the second drilling rig state recognition network are both transitioning from the rod-up state to the rod-down state.

[0077] (2) In the current sampling period, the current state of the drilling rig outputted by the first drilling rig state recognition network is the transition from the rod-up state to the rod-down state, while the current state of the drilling rig outputted by the second drilling rig state recognition network is not the transition from the rod-up state to the rod-down state.

[0078] (3) In the current sampling period, the current state of the drilling rig outputted by the first drilling rig state recognition network is not a transition from the rod-up state to the rod-down state, and the current state of the drilling rig outputted by the second drilling rig state recognition network is a transition from the rod-up state to the rod-down state.

[0079] By adding one to the drill rod count in the above three situations, the drill rod count can be supplemented by data collected by different sensors, thereby ensuring the accuracy of the drill rod count result.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for counting drill rods in coal mines based on multi-source sensor data fusion, characterized in that: include: S1, obtains infrared images, video data and original LiDAR point clouds collected during the underground drilling rig construction in the coal mine in the current sampling period; S2, inputting the collected infrared image into a pre-trained first drilling rig state recognition network, and the first drilling rig state recognition network outputs the current state of the drilling rig; S3, performing frame processing on the collected video data to obtain multiple original images, inputting each original image into a pre-trained drilling rig target detection network, and the pre-trained drilling rig target detection network outputs a target frame where the drilling rig is located; S4, performing time alignment and spatial alignment on the collected original laser radar point cloud and the original image to obtain the laser point cloud in the pixel coordinate system; S5, marking the laser point cloud in the pixel coordinate system according to the target frame where the drilling rig is located, and obtaining the laser point cloud with the drilling rig status information; S6, converting the laser point cloud with the drilling rig status information into a laser mapping image with the drilling rig status information, and inputting the laser mapping image into a pre-trained second drilling rig status recognition network, so that the second drilling rig status recognition network outputs the current status of the drilling rig; S7, determining whether to add one to the drill rod count according to the current state of the drill rig output by the first drill rig state identification network and the current state of the drill rig output by the second drill rig state identification network in the current sampling period.

2. The method for counting drill rods in underground coal mines based on multi-source sensor data fusion according to claim 1, characterized in that: The first drilling rig state recognition network is an improved YOLOv5s model, and the drilling rig target detection network and the second drilling rig state recognition network are both YOLOv5s models; the improved YOLOv5s model includes a backbone network, a neck network and a detection head connected in sequence, the backbone network includes CBS, iRMB*N1, iRMB*N2, iRMB*N3, iRMB*N4, SPPF and RFEC3 connected in sequence, and the detection head adopts a dynamic detection head.

3. The method for counting drill rods in underground coal mines based on multi-source sensor data fusion according to claim 1, characterized in that: In the specific implementation of S2, the collected infrared image is scaled according to the size of the input image of the first drilling rig state recognition network, and the scaled infrared image is input into the pre-trained first drilling rig state recognition network.

4. The method for counting drill rods in coal mines based on multi-source sensor data fusion according to claim 1, characterized in that: In the specific implementation of S3, the video data collected in 1s is divided into three frames of original images, and each frame of the original image is scaled according to the size of the input image of the drilling rig target detection network, and the scaled original image is input into the pre-trained drilling rig target detection network.

5. The method for counting drill rods in underground coal mines based on multi-source sensor data fusion according to claim 1 or 4, characterized in that: When performing spatial alignment, S4 includes: S41, performing dedistortion processing on the collected original laser radar point cloud; S42, projecting the dedistorted LiDAR point cloud to a camera coordinate system using a homogeneous transformation matrix between the video data and the original LiDAR point cloud; S43, projecting the laser radar point cloud in the camera coordinate system into the pixel coordinate system using the camera intrinsic parameters to obtain the laser point cloud in the pixel coordinate system.

6. The method for counting drill rods in underground coal mines based on multi-source sensor data fusion according to claim 1, characterized in that: In the specific implementation of S5, the laser point cloud in the pixel coordinate system is marked according to the coordinates of the target frame where the drilling rig is located in the pixel coordinate system, so as to obtain the laser point cloud with the drilling rig status information.

7. The method for counting drill rods in underground coal mines based on multi-source sensor data fusion according to claim 1, characterized in that: The current state of the drilling rig includes a rod-up state and a rod-down state. When S7 is specifically implemented, it includes: S71, determining whether the current state of the drilling rig output by the first drilling rig state recognition network and the current state of the drilling rig output by the second drilling rig state recognition network in the current sampling period are transitioning from the rod-up state to the rod-down state; S72, if at least one of the current state of the drilling rig output by the first drilling rig state identification network and the current state of the drilling rig output by the second drilling rig state identification network in the current sampling period is transitioned from the rod-on state to the rod-off state, determine to increase the drill rod count by one.

Images