A coal mine underground drill rod counting method based on machine vision

Abstract

The application provides a coal mine underground drill rod counting method based on machine vision and belongs to the technical field of drill rod counting. The application solves the problems of low precision and large error in the existing drill rod counting method based on machine vision. The application comprises the following steps: setting a feature target on a drilling machine mechanical arm, obtaining an underground drilling machine construction video with the feature target as a data set, preprocessing the data set, constructing and training a target detection model based on deep learning, inputting the preprocessed video into the target detection model, detecting the feature target on the drilling machine mechanical arm, obtaining the motion trajectory of the feature target of the mechanical arm by using visual SLAM technology after detecting the feature target, judging whether the drilling machine mechanical arm has rod loading and rod unloading actions by using the motion trajectory of the feature target, and determining the number of drilled rods by comparing the rod values obtained according to the rod loading and rod unloading actions. The application is applied to coal mine underground drill rod counting.

Description

TECHNICAL FIELD

[0001] The application provides a coal mine underground drill rod counting method based on machine vision, and belongs to the technical field of drill rod counting. BACKGROUND

[0002] At present, the mining depth of coal mines is increasing, and the gas content of coal seams also increases significantly with the increase of mining depth, which is prone to gas outburst disasters. Pre-draining coal seam gas is a main means to prevent gas outburst disasters. Pre-draining coal seam gas is mainly implemented by drilling holes in the underground, which can effectively reduce the gas content and pressure of the coal seam, and the depth of the drilling hole has certain requirements, which is generally indirectly calculated by counting the number of drill rods drilled into the coal seam.

[0003] The traditional mine drilling machine drill rod counting method mainly includes manual counting method and sensor counting method. The manual counting method is to observe and record the number of drill rods by the staff on site, but long-time work is easy to make people tired, leading to counting errors, and it is time-consuming and labor-consuming, and the automation level is low. The sensor counting method is to install a sensor on the drilling machine mechanical arm, collect sensor data by an electric control box and perform data preprocessing, and then obtain the number of drill rods, but the sensor is easily damaged in a dusty, humid and other harsh environment, thereby causing counting errors.

[0004] With the development of machine vision technology, it has shown advantages in many fields, and different scholars have proposed different schemes for calculating the number of drill rods in recent years. For example, the patent with the application number 202110422894.3 discloses a coal mine underground drilling number counting method based on machine vision, which mainly identifies and calculates the movement period of the drilling machine power head position through image recognition, and counts the drill rods according to the drilling machine period. The patent with the application number 202210149397.5 discloses a method for automatically counting underground drill rods based on machine vision, which mainly draws a periodic motion waveform graph of the drill rod and the drilling machine through image recognition to calculate the number of drill rod advances. The patent with the application number 202210378082.8 discloses an underground drill rod counting method based on machine vision, which mainly identifies the skeleton posture of the operator through image recognition, and judges whether to advance the rod or unload the rod according to the skeleton posture of the operator to realize drill rod counting.

[0005] The above technical solutions realize the judgment and statistics of drill rod counting through different methods, but in the above methods, the recognition of the rod advancing and inclined rod action of the drill rod machine is dependent on the current image for recognition without training. Due to the influence of the coal mine underground environment, the recognition accuracy is not high and the error is large. SUMMARY

[0006] The present application aims to solve the problem of low precision and large error in the existing method of counting drill rods through machine vision, and proposes a coal mine underground drill rod counting method based on machine vision.

[0007] To solve the above technical problems, the technical scheme adopted by the present application is as follows: a coal mine underground drill rod counting method based on machine vision, comprising the following steps:

[0008] S100: setting a feature target on the drill rig mechanical arm and acquiring the underground drill rig construction video with the feature target as a data set;

[0009] S200: pre-processing the data set to increase the brightness of the images in the video;

[0010] S300: constructing and training a target detection model based on deep learning, inputting the pre-processed video into the target detection model, and detecting the feature target on the drill rig mechanical arm;

[0011] S400: after detecting the feature target on the drill rig mechanical arm, combining the target detection result, and obtaining the motion trajectory of the mechanical arm feature target using visual SLAM technology;

[0012] S500: using the motion trajectory of the feature target on the drill rig mechanical arm to determine whether the drill rig mechanical arm has rod loading and rod unloading actions, and if there are rod loading and rod unloading actions, respectively accumulating the number of rod loading and rod unloading actions in the video stream;

[0013] S600: by comparing the drill rod values obtained according to the rod loading and rod unloading actions, determine the number of drilled rods, if the number of rod loading and rod unloading actions is consistent, it means that the number of drilled rods is correct; if the number of rod loading and rod unloading actions is inconsistent, manual intervention is needed to obtain the accurate number of drill rods.

[0014] The pre-processing in step S200 is to introduce a limited contrast self-adaptive histogram equalization processing step to each frame of image collected in step S100, to increase the brightness of the video.

[0015] The target detection model based on deep learning in step S300 specifically uses an improved YOLOv5s model, which inserts a coordinate attention mechanism after the CSP module of the traditional YOLOv5s model to improve the depth of the feature extraction network.

[0016] The step of obtaining the motion trajectory of the mechanical arm feature target using visual SLAM technology in step S400 is as follows:

[0017] In the process of acquiring the motion trajectory of the feature target, a module for eliminating static feature points is added behind the visual SLAM tracking algorithm based on ORB-SLAM3, and only the dynamic mechanical arm feature target is used to estimate the pose change between adjacent frames, so as to eliminate the interference of static targets in the environment in the process of the motion trajectory of the feature target on the mechanical arm.

[0018] The convolutional neural network model is used in the step S500, and the working state of the drilling machine is judged according to the different motion trajectories of the feature target in the process of the lifting of the rod and the unloading of the rod;

[0019] The rod counting method in the step S500 needs to perform frame reduction processing on the working video of the drilling machine, so that the original video becomes 3 frames / s, and when the lifting action or the unloading action is detected and the interval between the actions is 3 s or more, the number of rods is accumulated by one.

[0020] The step of judging the working state of the drilling machine in the step S500 is:

[0021] S501: The feature target is set on the gripper of the mechanical arm of the drilling machine, and the motion trajectory of the feature target in the process of the lifting of the rod and the unloading of the rod is judged by a target detection model. In the process of the lifting of the rod, the gripper needs to move from the initial position to the rod storage to hold one rod, then the gripper moves to the fixed position to place the rod on the travel path of the propulsion motor, and finally the gripper returns to the initial position. In the process of the unloading of the rod, the gripper needs to directly move to the fixed position to hold the rod on the propulsion motor, then the gripper moves to the rod storage while holding the rod, and finally the gripper returns to the initial position;

[0022] S502: The motion trajectory data set of the feature target in the process of the lifting of the rod and the unloading of the rod is respectively made, the data set is divided into a training set and a verification set, then the convolutional neural network model is trained using the training set, and after the training is completed, the drilling motion state detection capability of the trained convolutional neural network model is tested using the verification set;

[0023] S503: The motion trajectory of the feature target obtained in the step S501 is input into the trained convolutional neural network model, and the working state of the drilling machine at this time is obtained.

[0024] The method for accumulating the number of times of the lifting of the rod and the unloading of the rod in the step S500 is:

[0025] After the lifting action of the drilling machine is recognized in the step S501, the drilling action is tracked, the lifting action ends after all the rods are driven into the coal seam, at this time the lifting action cannot be detected, and if the interval between the lifting actions is 3 s or more, it is considered that one lifting operation is completed, and the number of rods is accumulated by one;

[0026] Like the same rod number counting method, after identifying the rod unloading action of the drilling machine, the drilling machine action is tracked, the rod is screwed out in the coal seam, and the rod unloading action is ended, at this time, the rod unloading action cannot be detected, if the interval between the rod unloading actions is 3s or more, it is considered that one rod unloading operation is completed, and the number of rods is accumulated by one.

[0027] The coal mine underground drill rod counting method based on machine vision provided by the present application can automatically and accurately count the drill rods, improve the work efficiency, reduce the labor cost, and improve the safety factor of the staff. BRIEF DESCRIPTION OF DRAWINGS

[0028] The present application will be further described below in conjunction with the drawings:

[0029] Fig. 1 The flowchart of the drill rod counting method of the present application;

[0030] Fig. 2 The improved YOLOv5s network structure diagram of the present application;

[0031] Fig. 3 The flowchart of the present application for counting the number of drill rods. DETAILED DESCRIPTION

[0032] As shown in Figs. 1 to 3 The present application provides a coal mine underground drill rod counting method based on machine vision, mainly including the following steps:

[0033] S100: Obtain the underground drilling machine construction video;

[0034] S200: Due to the poor underground lighting conditions, introduce the limited contrast adaptive histogram equalization processing to each frame image in the drilling machine construction video collected in S100, increase the brightness of the video;

[0035] S300: Input the video into the target detection model based on deep learning, detect the feature target on the drilling machine mechanical arm;

[0036] S400: After detecting the feature target on the drilling machine mechanical arm, combine the target detection result, and obtain the motion trajectory of the mechanical arm feature target by using the visual SLAM technology;

[0037] S500: Use the motion trajectory of the feature target on the mechanical arm to judge whether the mechanical arm has rod feeding and rod unloading actions, if the rod feeding and rod unloading actions exist, respectively accumulate the number of rod feeding and rod unloading actions in the video stream;

[0038] S600: Determine the number of drill rods by comparing the drill rod values obtained according to the rod-up and rod-off actions. If the number of rod-up and rod-off actions is consistent, it indicates that the number of drill rods is correct. If the number of rod-up and rod-off actions is inconsistent, manual intervention is required to obtain the accurate number of drill rods.

[0039] The application utilizes a target detection model based on deep learning to detect the feature targets on the mechanical arm in the drill rod construction video. Due to the dust and water mist interference in the coal mine tunnel and the complex lighting conditions, the traditional target detection model is prone to missing detection problems, so the limited contrast adaptive histogram equalization image processing algorithm and the coordinate attention mechanism are integrated into the target detection model to improve the drill rod counting accuracy. The improved target detection network is as shown in Fig. 2 .

[0040] The application selects the YOLOv5s model as the target detection model. Due to the complex underground environment, the camera image is easily affected by water mist and dust, causing image blur and missing detection. Therefore, the limited contrast adaptive histogram equalization image processing algorithm is added to the YOLOv5s model. The limited contrast adaptive histogram equalization image processing algorithm can perform defogging processing on the image, making the picture clear, so that the drill action features are more obvious, and the feature extraction network can extract more features.

[0041] Since the drill working state feature information in the image only exists in a small area, the feature information is easily lost in the convolution process, resulting in low detection accuracy. Therefore, the coordinate attention mechanism (Coordinate Attention, CA) is introduced, which is inserted after the CSP module of the target detection network to improve the depth of the feature extraction network and increase the amount of drill working state feature information extraction, thereby improving the model detection accuracy.

[0042] The coordinate attention module adds position information to the channel attention, so that the feature extraction network can pay attention to the target features in a larger area. One direction gets long-range dependency, and the other direction gets accurate position information. The generated feature map is encoded to form a pair of direction perception and position sensitive features.

[0043] The coordinate attention module includes coordinate information embedding and coordinate attention generation. A coordinate attention module can take any intermediate tensor X=[x1, x2,..., x c ]∈R C×H×W as input and output an output Y=[y1, y2,..., y c ] of the same size with enhanced representation ability, where C is the number of channels, H and W are the height and width of the input picture, respectively.

[0044] Coordinate information embedding. In order to enable the attention module to capture remote spatial interaction with precise position information, the coordinate attention module decomposes the global pooling according to the following formula, which is converted into a one-to-one feature encoding operation for the feature target image features, where H is the height of the input feature target picture, W is the width of the input feature target picture, c represents the number of channels, x c (i, j) is the input X, which is a fixed kernel size convolutional layer, z c is the output of the cth channel.

[0045]

[0046] Specifically, using a pooling kernel of size (H, 1) and (1, W) to encode each channel along the horizontal and vertical coordinates respectively, X c (h, i) is given input X, using a pooling convolution kernel of size (H, 1) to encode each channel along the horizontal coordinate, X c (j, w) is given input X, using a pooling convolution kernel of size (1, W) to encode each channel along the vertical coordinate, is the output of the cth channel of the feature target image with height h, is the output of the cth channel with width w.

[0047]

[0048]

[0049] The above two formulas respectively aggregate the features along two spatial directions to obtain a pair of direction-aware feature maps, allowing the attention module to capture long-term dependencies along one spatial direction and preserve the precise position information of the feature target along the other spatial direction, which helps the network to more accurately locate the target to be detected.

[0050] Coordinate attention generation. After concatenating the generated z h and z w , input a 1x1 convolutional transformation function F1, which performs a transformation operation, and δ is a nonlinear activation function, f h is the encoding of spatial information in the horizontal direction, f w is the encoding of spatial information in the vertical direction, F h and F w are two 1x1 convolutional transformations that transform f h and f w into tensors with the same number of channels as the input X, g h and g w are attention weights, and σ is a sigmoid function.

[0051] f = delta(F1([z n , z w ]));

[0052] g h = sigma(F h (f h ));

[0053] g w = sigma(F w (f w ));

[0054] Finally, the output of the coordinate attention module can be written as:

[0055]

[0056] y c (i, j) is the output of the coordinate attention block, x c (i, j) is the input, and is the attention weight.

[0057] Due to the complex downhole working environment and insufficient illumination, the feature target image captured by the camera is relatively blurred, and the target feature information is easily lost in the deep learning convolution process, so the coordinate attention mechanism is embedded into the Backbone region of the original YOLOv5s network, the network layer number of the feature extraction network in the original network is increased, so that the feature extraction network can accurately locate the object of interest, thereby increasing the attention of the model to the target feature, extracting more target features, and improving the model training effect to improve the network recognition accuracy.

[0058] The feature target data set on the drilling rig mechanical arm constructed is subjected to limited contrast adaptive histogram equalization processing, and after the processed target data set is labeled, it is divided into a training set and a verification set according to a ratio of 8:2.

[0059] The training set is inserted into the target detection network of the coordinate attention mechanism, and after training, an improved target detection model is obtained; the improved target detection model is tested by the verification set to obtain the success rate of drilling rig working state classification detection.

[0060] When collecting the drilling rig working video stream, the video stream is subjected to frame reduction processing, the original video frame rate is changed to 3 frames / s, each frame image in the frame-reduced video stream is subjected to limited contrast adaptive histogram equalization processing and then input into the improved YOLOv5s model, so that the feature target on the drilling rig mechanical arm is detected.

[0061] Combining the target detection results, visual SLAM technology is used to obtain the motion trajectory of the feature target. When obtaining the motion trajectory of the feature target, a module for removing static feature points is added after the visual SLAM tracking algorithm based on ORB-SLAM3. Only the dynamic robotic arm feature target is used to estimate the pose change between adjacent frames, eliminating the interference of static targets in the environment during the motion trajectory of the feature target on the robotic arm, and ensuring that the motion trajectory of the feature target is obtained accurately.

[0062] After obtaining the motion trajectory of the characteristic target, the drilling rig is determined to be in the state of raising or lowering the rod based on the different motion trajectories of the characteristic target when raising and lowering the rod.

[0063] In this invention, the feature target is set on the gripper of the robotic arm. When loading the rod, the gripper needs to move from the initial position to the drill rod magazine to clamp a drill rod. Then, the gripper moves to a fixed position to place the drill rod on the path of the propulsion motor. Finally, the gripper returns to the initial position. When unloading the rod, the gripper needs to move directly to the fixed position to clamp the drill rod off the propulsion motor. Then, it moves towards the drill rod magazine while holding the drill rod, puts the drill rod back into the drill rod magazine, and finally returns to the initial position. Therefore, the movement trajectory of the feature target is different when the drilling rig is in the loading and unloading actions.

[0064] Data sets of feature target motion trajectories were created for the pole raising and lowering processes, and the datasets were divided into training and validation sets. The training set was then used to train a convolutional neural network, and the validation set was used to test the drilling rig motion state detection capability of the trained convolutional neural network model.

[0065] The obtained feature target motion trajectory is input into the trained convolutional neural network model to obtain the drilling rig's working status at this time.

[0066] After determining the working status of the drilling rig, the number of times the rod is raised and lowered is calculated respectively;

[0067] like Fig. 3 As shown, the method for accumulating the number of backswings and backswings is as follows:

[0068] After the drilling rig's rod-up action is detected, the drilling rig's actions are tracked. The rod-up action ends when all the drill rods are driven into the coal seam. At this time, the rod-up action will not be detected. If the interval between rod-up actions is 3 seconds or more, it is considered that one rod-up operation has been completed, and the number of drill rods is increased by one.

[0069] Similar to the method for counting the number of drill rods, after the drill rod unloading action is detected, the drill rod action is tracked. After the drill rod is unscrewed from the coal seam, the unloading action ends. At this time, the unloading action will not be detected. If the interval between unloading actions is 3 seconds or more, it is considered that one unloading operation has been completed, and the number of drill rods is counted by one.

[0070] The accuracy of the drill rod counting is determined by comparing the number of drill rods obtained by the up rod and the unloading rod actions, if the number is consistent, the drill rod counting is accurate, if the number is not consistent, manual intervention is needed to calculate the accurate number of drill rods.

[0071] It should be noted that the connection relationship between the components and modules of the present application is determined and can be realized. Except for the special description in the embodiments, the specific connection relationship can bring about the corresponding technical effects, and based on the premise of not relying on the corresponding software program execution, the technical problems proposed by the present application are solved. The model of the components, modules, specific elements, the connection mode between them, and the conventional use method and the expected technical effects brought by the above technical features, except for the specific description, all belong to the public content disclosed in the patents, journal papers, technical manuals, technical dictionaries, textbooks, etc. which can be obtained by the skilled in the art before the filing date, or belong to the existing technology such as the conventional technology and common knowledge, and do not need to be described in detail. Therefore, the technical solution provided by the present application is clear, complete and realizable, and the corresponding entity product can be reproduced or obtained according to the technical means.

[0072] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A machine vision based method for counting drill pipes in a coal mine underground, characterized by: The method comprises the following steps: S100: setting a feature target on a drilling rig mechanical arm and acquiring a downhole drilling rig construction video with the feature target as a data set; S200: preprocessing the data set to increase the brightness of the images in the video; S300: constructing and training a target detection model based on deep learning, inputting the preprocessed video into the target detection model, and detecting the feature target on the drilling rig mechanical arm; The target detection model based on deep learning in the step S300 specifically adopts an improved YOLOv5s model, and the improved YOLOv5s model inserts a coordinate attention mechanism behind a CSP module of a traditional YOLOv5s model to improve the depth of a feature extraction network; S400: after detecting the feature target on the drilling rig mechanical arm, obtaining the motion trajectory of the mechanical arm feature target by using a visual SLAM technology in combination with the target detection result; The step of obtaining the motion trajectory of the mechanical arm feature target by using the visual SLAM technology in the step S400 is as follows: When the motion trajectory of the feature target is acquired, a module for removing static feature points is added behind an ORB-SLAM3-based visual SLAM tracking algorithm, only the dynamic mechanical arm feature target is used to estimate the pose change between adjacent frames, and the interference of static targets in the environment during the motion trajectory process of the feature target on the mechanical arm is eliminated; S500: judging whether the drilling rig mechanical arm has a rod feeding and rod unloading action by using the motion trajectory of the feature target on the drilling rig mechanical arm, and if the rod feeding and rod unloading action exists, the number of times of the rod feeding and rod unloading action in the video stream is accumulated, respectively; S600: determining the number of drilled rods by comparing whether the number of drilled rods obtained according to the rod feeding and rod unloading action is the same, and if the number of drilled rods is consistent, it is indicated that the number of drilled rods is correct; if the number of drilled rods is inconsistent, manual intervention is needed to obtain the accurate number of drilled rods.

2. The machine vision based method of counting drill pipes in a coal mine shaft as claimed in claim 1, wherein: The preprocessing in the step S200 is to introduce a limited contrast self-adaptive histogram equalization processing step S100 to each frame of image collected in the drilling rig construction video, so as to increase the brightness of the video.

3. The machine vision based method of counting drill pipes in a coal mine shaft as claimed in claim 1, wherein: In the step S500, a convolutional neural network model is used to judge the drilling rig working state according to the different motion trajectories of the feature target during the rod feeding and rod unloading. The rod counting method in the step S500 needs to perform frame reduction processing on the drilling rig working video, so that the original video becomes 3 frames / s, when the rod feeding or rod unloading action is detected, and the interval between the actions is 3 s or more, the number of rods is accumulated by one.

4. The machine vision based method of counting drill pipes in a coal mine shaft as claimed in claim 1, wherein: The step of judging the drilling rig working state in the step S500 is as follows: S501: setting the feature target on the gripper of the drilling rig mechanical arm, judging the motion trajectory of the feature target during the rod feeding and rod unloading by using the target detection model, moving the gripper from the initial position to the drill rod storage to clamp a drill rod during the rod feeding, moving the gripper to the fixed position to place the drill rod on the advancing motor running path, and finally returning the gripper to the initial position; during the rod unloading, the gripper directly moves to the fixed position to clamp the drill rod on the advancing motor, then moves to the drill rod storage with the drill rod, puts the drill rod back to the drill rod storage, and finally returns the gripper to the initial position; S502: respectively make the feature target motion trajectory data set when making the rod and unloading rod, divide the data set into training set and validation set, then use the training set to train the convolution neural network model, after the training, use the validation set to test the drilling rig motion state detection ability of the trained convolution neural network model; S503: input the feature target motion trajectory obtained in step S501 into the trained convolution neural network model to obtain the working state of the drilling rig at this time.

5. The machine vision based method of counting drill pipes in a coal mine shaft as claimed in claim 4, wherein: The method for accumulating the number of rod making and rod unloading in step S500 is: After recognizing the rod making action of the drilling rig in step S501, track the drilling rig action, and the rod making action ends after the drilling rod is completely punched into the coal seam, at this time, the rod making action cannot be detected, if the interval between the rod making actions is 3s or more, it is considered that one rod making operation is completed, and the number of drilling rods is added by one; Similarly, after recognizing the rod unloading action of the drilling rig, track the drilling rig action, and the rod unloading action ends after the drilling rod in the coal seam is screwed out, at this time, the rod unloading action cannot be detected, if the interval between the rod unloading actions is 3s or more, it is considered that one rod unloading operation is completed, and the number of drilling rods is added by one.

Images