Coal mine underground drill rod counting method based on visual detection sequence smoothing algorithm

Abstract

The application discloses a coal mine underground drill pipe counting method based on a visual detection sequence smoothing algorithm, and is specifically implemented according to the following steps: step 1, drill pipe target frame-by-frame detection and detection data recording; step 2, drill pipe detection sequence timing smoothing; step 3, drill pipe continuous existence interval extraction and dynamic threshold adaptive screening; and step 4, drill pipe counting and drilling statistics output. The application can realize automatic counting of drill pipes under complex conditions in the underground, and has a counting scheme with functions of detection sequence repair, dynamic threshold self-adaption, intelligent extraction of drilling time periods and video offline analysis, so that the accuracy and reliability of drill pipe counting are improved.

Description

Technical Field

[0001] This invention belongs to the field of machine vision and video analysis technology, specifically relating to a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm. Background Technology

[0002] Drilling in coal mines is a crucial process in mining production and disaster management, involving multiple safety aspects such as water hazard prevention, ground pressure control, and gas extraction. During drilling, drill rods are connected to the drilling rig and fed into the borehole. The number of drill rods directly reflects the drilling progress and is an important statistical statistic for the aforementioned safety aspects. Currently, the counting of drill rods in coal mines mainly relies on the following methods: One method is manual counting, which involves operators recording the number of drill rods during drilling or retraction. This method relies on the worker's sense of responsibility and attention, increases labor intensity, and may have significant errors.

[0003] Secondly, measurement while drilling (MWD) systems are used for counting. This method can monitor the drill bit position, inclination angle, azimuth angle, and drilling depth in real time, and generate borehole trajectory diagrams and footage statistics. However, MWD systems are expensive and cannot be equipped on all drilling rigs, thus limiting their application.

[0004] Thirdly, video-based drill pipe counting has become a research hotspot due to the development of machine vision technology and the widespread adoption of downhole monitoring. However, the harsh downhole environment, with its dust, water mist, and drilling rig vibrations causing image jitter, can lead to temporary missed detections and fragmented results. Therefore, there is an urgent need to improve the accuracy of drill pipe counting by refining relevant algorithms. Summary of the Invention

[0005] The purpose of this invention is to provide a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm. This method can automatically count drill pipes under complex underground conditions and includes a counting scheme with functions such as detection sequence repair, dynamic threshold adaptation, intelligent extraction of drilling time periods, and offline video analysis, thereby improving the accuracy and reliability of drill pipe counting.

[0006] The technical solution adopted in this invention is a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm, which is implemented according to the following steps: Step 1: Frame-by-frame detection and data recording of drill pipe targets; Step 2: Smooth the timing of drill pipe inspection sequences; Step 3: Extraction of continuous drill pipe intervals and adaptive selection using dynamic thresholds; Step 4: Drill pipe count and drilling statistics output.

[0007] The invention is further characterized in that, Step 1 is implemented in the following steps: From historical videos of underground drilling operations in coal mines, frames are randomly extracted at intervals. The drill rod targets in the images are labeled using a labeling tool, with the category ID set to 1. The labeled data is divided into three parts: training set, validation set, and test set. The YOLO series target detection model is trained according to the proportions to obtain the weight file. Initialize the runtime environment, detect the GPU environment, and prioritize using the GPU for inference to improve inference speed; if the GPU is unavailable, fall back to the CPU. Load the drill pipe detection model, load the trained weight file, and perform the CUDA environment warm-up operation before the first inference, that is, perform one inference with an empty tensor, triggering environment initialization, model loading, CUDA kernel compilation and cache initialization. Open the drilling operation monitoring video source, open the video file through the interface, and obtain video metadata, including video frame rate and total number of frames. If the video source cannot be opened, throw an exception and terminate. Load the pre-trained YOLO object detection model mentioned earlier, perform frame-by-frame inference on the video, and for the inference results of each frame, use a dual-path detection strategy of the original detection path and the confidence filtering path to extract the detection information of the drill rod. Enter the frame-by-frame detection main loop and execute the following sub-steps in a loop: Read the current frame image from the video source; exit the loop if reading fails or the video ends; increment the frame counter current_frame by 1. The current frame image is scaled to 640×640 pixels and input into the drill pipe detection model for inference to obtain the detection results; Analyze the inference results, traverse all detection boxes, and extract the category ID, bounding box coordinates, and confidence score for each detection box; Original detection path: Filter drill rod targets, that is, all detection results of type drill rod, regardless of confidence level, as long as drill rod appears in the picture, the frame is recorded for subsequent smoothing and counting processing; Confidence filtering path: Only retain detection results with confidence exceeding a preset threshold. Set the threshold to 0.25, calculate the coordinates of the drill pipe target center point, and use them for result reporting and callback push. center_x = (x1 + x2) / 2 center_y = (y1 + y2) / 2 Where x1 and y1 represent the pixel coordinates of the top left corner of the detection box; x2 and y2 represent the pixel coordinates of the lower right corner of the detection box; center_x represents the x-coordinate of the center point of the detection box, i.e. (x1 + x2) / 2, which represents the horizontal position of the drill rod target in the image; center_y represents the ordinate of the center point of the detection box, i.e. (y1 + y2) / 2, which represents the vertical position of the drill rod target in the image; Set the drill pipe detection presence flag for the current frame: based on the original detection path, if at least one drill pipe target is detected, mark the "whether it includes drill pipe classification target" of the frame as 1, otherwise mark it as 0; Record the structured data of the current frame, including the current frame number, whether it contains drill pipe, the number of detections, and the confidence level; Release the video capture resources after the above loop ends; The entire set of frame-by-frame structured data accumulated in the main loop is directly constructed into a data structure in memory, including the current frame number, whether it contains drill rods, the number of detections, and the confidence level.

[0008] The training dataset used by the pre-trained model in step 1 includes drill pipe images under different dust concentrations, different lighting conditions, and different drilling angles captured in video, in order to improve the model's generalization ability in complex downhole environments.

[0009] Step 2 is implemented in the following steps: The memory data structure constructed in step 1 is used as the input data source for smoothing. If the frame data set is empty, an empty result is returned directly. Create a copy of the data to avoid modifying the original data, and extract the "whether it includes drill pipe classification target" column, i.e., the has_class_1 column, which is a numerical array containing drill pipe frames; The drill pipe is traversed to detect the existence marker sequence. A state machine method is used to identify all continuous undetected intervals, i.e., continuous frames where the drill pipe has not been detected, i.e., continuous intervals with a has_class_1 value of 0. The start and end indices of each interval are recorded. The state machine defines two states: Status A: The current value is not in the 0 range, meaning the drill pipe is either being monitored or has not yet started. Status B: Currently in the 0 value range, meaning the drill pipe is in an undetected state; The state transition rules are as follows: If the current frame has_class_1 = 0 and is in state A, then record the starting position of the interval start = i, and transition to state B; If the current frame has_class_1 = 1 and is in state B, then record the end position of the interval end = i-1, output the interval record (start, end), reset start to None, and transition to state A; If the system is still in state B after the traversal is complete, then record the end position of the interval as the index of the end of the array. For each identified undetected region, perform a triple condition check: Condition 1: Gap length constraint: The gap length is greater than or equal to 1 frame and less than or equal to the preset maximum allowable gap length parameter max_gap; this parameter is set according to the drilling speed and video frame rate and is used to distinguish between brief missed detections and actual departure of the drill pipe; Specifically, the gap length, gap_length = end start + 1, must satisfy the following: 1 ≤ gap_length ≤ max_gap Condition 2: Forward Existence Constraint: The drill rod detection existence flag of the previous frame in this interval is "detected", that is, the value is set to 1, that is, the drill rod was in the detected state before the gap; Specifically, check the drill rod detection marker of the previous frame in the interval. If start > 0, take has_class_1[start-1], otherwise consider it as 0. This value must be equal to 1, that is, the drill rod was being detected before the gap. Condition 3: Backward Existence Constraint: The drill rod detection existence mark of the next frame in this interval is "detected", and the value is set to 1 in the sequence, that is, the drill rod is detected again after the gap; Specifically, check the drill rod detection marker in the next frame after the interval. If end < array length - 1, then take has_class_1[end+1], otherwise consider it as 0; this value must be equal to 1, that is, the drill rod is detected again after the gap. For intervals that simultaneously meet the above three conditions, modify the drill pipe detection presence flag of all its frames to "detected", that is, set the value to 1, to fix the temporary missed detection caused by downhole factors; Output the smoothed drill pipe detection sequence data df_smoothed.

[0010] Step 3 is implemented in the following steps: Step 3.1: Use a state machine-based interval scanning algorithm to extract continuous intervals from the smoothed drill rod detection sequence, and use a dynamic threshold adaptive algorithm to calculate the optimal minimum number of consecutive frames threshold, so as to convert the frame-by-frame detection results into intervals of the actual drill rod drilling process. Step 3.2: Traverse the smoothed drill pipe detection existence marker sequence, use the state machine method to identify all continuous detected intervals, that is, the continuous frame segments in which the drill pipe is continuously detected, and record the start frame number, end frame number and interval length of each interval; Step 3.3: Use the preset minimum number of consecutive frames for preliminary screening, traverse all intervals with a value of 1, calculate the interval length, and select intervals with an interval length greater than the preset minimum number of consecutive frames as candidate drill rod intervals. Record the interval information into the candidate list, including four types of information: start frame number, end frame number, and number of consecutive frames. Discard intervals with a length less than the preset threshold, as these intervals are considered to be noise false detections. Step 3.4: Perform dynamic threshold adaptive calculation on the candidate interval list to determine the optimal minimum drill pipe duration frame threshold; Step 3.5: Use the dynamically calculated threshold to perform a secondary screening of the candidate intervals, retain only the continuous detection intervals with an interval length greater than or equal to the dynamic threshold as the final continuous drill pipe intervals, and filter out the scattered detection segments formed by noise or brief false detections. Step 3.6: Output a list of consecutive drill pipe intervals that meet the conditions. The number of intervals in the list is the drill pipe count result. Each interval includes the start frame position, end frame position, number of consecutive frames, and average confidence information within the interval, which is used for subsequent drill pipe counting and drilling time period analysis.

[0011] The minimum consecutive frame threshold in step 3.1 is calculated using a dynamic adaptive algorithm, which can be automatically calculated based on the following factors: Jump points in the candidate interval length sequence can be used to distinguish noise segments from real drill pipe segments. The minimum number of visible frames for a single drill rod within the detection area is calculated from the drill rod length, drilling speed, and frame rate. Business requirements necessitate filtering the minimum detection duration.

[0012] Step 3.4 determines the optimal minimum drill pipe duration frame threshold, and the specific algorithm is as follows: (1) Extract the length sequence of all candidate intervals and sort them in ascending order; (2) Traverse the sorted length sequence, calculate the ratio of each two adjacent intervals using the interval length, that is, the ratio of the number of frames contained in the adjacent intervals, and find the point with the maximum ratio, that is, the jump point. This jump point is the boundary between the noise interference segment and the real drill pipe segment. (3) If the maximum ratio exceeds the preset significance threshold, in this embodiment the threshold is set to 2, and the jump point index is within the effective range, then the length of the first segment after the jump point is used as the dynamic minimum drill string duration frame threshold. (4) If all segments are of similar length, i.e. there is no obvious jump, and the maximum ratio does not exceed the significance threshold, then the median of the length sequence multiplied by the preset ratio coefficient is taken as the basic dynamic threshold, and it is not less than the default minimum number of consecutive frames of 15 frames. Step 4 is implemented in the following steps: Based on the continuous intervals extracted in step 3, drill pipe counting results and drilling statistics reports are generated, specifically including: Step 4.1: Each consecutive interval that meets the threshold condition corresponds to one drill rod existence event, that is, the complete process of a drill rod from entering the screen to leaving the screen, representing a complete drilling process or drill retraction process. The number of intervals is the drill rod count result. Step 4.2: Convert the number of frames in each interval into actual time based on the video frame rate, and calculate the duration of each drill rod in the frame and the start and end times of drilling; Step 4.3: Calculate drilling statistics, including four categories: cumulative number of drill pipes, cumulative drilling footage, total drilling time, and average loading and unloading time per drill pipe. The drill pipe count results are assembled with drilling statistics into a structured dictionary output, containing the following fields: The drill pipe count result is the number of consecutive intervals that satisfy the dynamic minimum drill pipe duration frame threshold condition. Basic video information, including total number of video frames, video frame rate, total video duration, and algorithm execution time; The drill pipe details list is a detailed list of the intervals in which each drill pipe exists continuously. Each interval includes the start frame number, end frame number, duration of frames, and average confidence level.

[0013] The beneficial effects of this invention are: (1) It replaces manual counting and improves the accuracy and reliability of drill rod counting. This invention realizes fully automatic visual counting of drill rods in coal mines, avoiding the problems of missed counting and incorrect counting caused by fatigue and distraction during the counting process, reducing the labor intensity of personnel and improving the counting accuracy. (2) Triple condition constraints ensure the accuracy of filling and can adapt to complex underground environments. The gap filling algorithm of this invention ensures that the gap length is within the preset range and that the drill rod is detected in both the frame before and after the gap. It distinguishes between the temporary missed detection caused by external environmental influence and the real rod-free state, and improves the accuracy of drill rod counting in the harsh environment of high dust and low illumination in coal mines. (3) It converts frame-by-frame detection into drill rod counting. This invention provides a complete processing method for converting the frame-by-frame output of the target detection model into drill rod counting data, covering steps such as detection sequence smoothing, continuous interval extraction, threshold filtering, and counting conversion, realizing the function of counting continuous video services from single frame detection. (4) It has a configurable and adaptive parameter system to adapt to various drilling conditions. The maximum clearance parameter in this invention is a configurable parameter that can be flexibly adjusted according to different drill pipe lengths, drilling speeds, video frame rates, and downhole environmental conditions; the minimum drill pipe continuous frame threshold is automatically calculated using a dynamic adaptive algorithm, which has adaptive capabilities under different working conditions. Attached Figure Description

[0014] Figure 1This is a schematic diagram of the overall process of the coal mine underground drill pipe counting method of the present invention; Figure 2 This is a flowchart illustrating the drill pipe detection sequence gap smoothing algorithm in this invention; Figure 3 This is a flowchart illustrating the algorithm for extracting continuous drill pipe intervals in this invention. Figure 4(a) shows the original detection sequence of drill pipe gap detection in this invention; Figure 4(b) is an example schematic diagram of the drill pipe detection gap smoothing process in this invention. Detailed Implementation

[0015] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0016] Example 1 This invention relates to a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm. The flowchart is as follows: Figure 1 As shown, please follow these steps: Step 1: Frame-by-frame detection and data recording of drill pipe targets; Step 2: Smooth the timing of drill pipe inspection sequences; Step 3: Extraction of continuous drill pipe intervals and adaptive selection using dynamic thresholds; Step 4: Drill pipe count and drilling statistics output.

[0017] Example 2 This invention relates to a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm. The flowchart is as follows: Figure 1 As shown, please follow these steps: Step 1: Frame-by-frame detection and data recording of drill pipe targets; Step 1 is implemented in the following steps: like Figure 1 As shown, this invention is applied to underground drilling operations in coal mines. A high-definition explosion-proof camera is placed on the side and rear of the drilling rig, with the camera's field of view covering the drill pipe loading and unloading area, recording the entire drilling process.

[0018] From historical videos of underground drilling operations in coal mines, frames are randomly extracted at intervals. The drill rod targets in the images are labeled using a labeling tool, with the category ID set to 1. The labeled data is divided into three parts: training set, validation set, and test set. The YOLO series target detection model is trained according to the proportions to obtain the weight file. Initialize the runtime environment, detect the GPU environment, and prioritize using the GPU for inference to improve inference speed; if the GPU is unavailable, fall back to the CPU. Load the drill pipe detection model, load the trained weight file, and perform the CUDA environment warm-up operation before the first inference, that is, perform one inference with an empty tensor, triggering environment initialization, model loading, CUDA kernel compilation and cache initialization. Open the drilling operation monitoring video source, open the video file through the interface, and obtain video metadata, including video frame rate and total number of frames. If the video source cannot be opened, throw an exception and terminate. Load the pre-trained YOLO object detection model mentioned earlier, perform frame-by-frame inference on the video, and for the inference results of each frame, use a dual-path detection strategy of the original detection path and the confidence filtering path to extract the detection information of the drill rod. Enter the frame-by-frame detection main loop and execute the following sub-steps in a loop: Read the current frame image from the video source; exit the loop if reading fails or the video ends; increment the frame counter current_frame by 1. The current frame image is scaled to 640×640 pixels and input into the drill pipe detection model for inference to obtain the detection results; Analyze the inference results, traverse all detection boxes, and extract the category ID, bounding box coordinates, and confidence score for each detection box; Original detection path: Filter drill rod targets, that is, all detection results of type drill rod (the detection results of personnel and safety helmets are not processed in this invention). Regardless of the confidence level, as long as the drill rod appears in the picture, the frame is recorded for subsequent smoothing and counting processing. Confidence filtering path: Only retain detection results with confidence exceeding a preset threshold. Set the threshold to 0.25, calculate the coordinates of the drill pipe target center point, and use them for result reporting and callback push. center_x = (x1 + x2) / 2 center_y = (y1 + y2) / 2 Where x1 and y1 represent the pixel coordinates of the top left corner of the detection box; x2 and y2 represent the pixel coordinates of the lower right corner of the detection box; center_x represents the x-coordinate of the center point of the detection box, i.e. (x1 + x2) / 2, which represents the horizontal position of the drill rod target in the image; center_y represents the ordinate of the center point of the detection box, i.e. (y1 + y2) / 2, which represents the vertical position of the drill rod target in the image; Set the drill pipe detection presence flag for the current frame: Based on the original detection path, if at least one drill pipe target is detected, mark the "whether it includes drill pipe classification target (has_class_1)" of the frame as 1, otherwise mark it as 0; Record the structured data of the current frame, including the current frame number, whether it contains drill pipe, the number of detections, and the confidence level; Release the video capture resources after the above loop ends; The entire set of frame-by-frame structured data accumulated in the main loop is directly constructed into a data structure in memory, including the current frame number, whether it contains drill rods, the number of detections, and the confidence level.

[0019] The training dataset used by the pre-trained model in step 1 includes drill pipe images under different dust concentrations, different lighting conditions, and different drilling angles captured in video, in order to improve the model's generalization ability in complex downhole environments.

[0020] In step 1, the input video frame image is scaled to a preset size, such as 640×640 pixels, to balance detection accuracy and inference speed.

[0021] Step 2: Smooth the timing of drill pipe inspection sequences; Step 3: Extraction of continuous drill pipe intervals and adaptive selection using dynamic thresholds; Step 4: Drill pipe count and drilling statistics output.

[0022] Example 3 This invention relates to a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm. The flowchart is as follows: Figure 1 As shown, please follow these steps: Step 1: Frame-by-frame detection and data recording of drill pipe targets; Step 1 is implemented in the following steps: like Figure 1 As shown, this invention is applied to underground drilling operations in coal mines. A high-definition explosion-proof camera is placed on the side and rear of the drilling rig, with the camera's field of view covering the drill pipe loading and unloading area, recording the entire drilling process.

[0023] From historical videos of underground drilling operations in coal mines, frames are randomly extracted at intervals. The drill rod targets in the images are labeled using a labeling tool, with the category ID set to 1. The labeled data is divided into three parts: training set, validation set, and test set. The YOLO series target detection model is trained according to the proportions to obtain the weight file. Initialize the runtime environment, detect the GPU environment, and prioritize using the GPU for inference to improve inference speed; if the GPU is unavailable, fall back to the CPU. Load the drill pipe detection model, load the trained weight file, and perform the CUDA environment warm-up operation before the first inference, that is, perform one inference with an empty tensor, triggering environment initialization, model loading, CUDA kernel compilation and cache initialization. Open the drilling operation monitoring video source, open the video file through the interface, and obtain video metadata, including video frame rate and total number of frames. If the video source cannot be opened, throw an exception and terminate. Load the pre-trained YOLO object detection model mentioned earlier, perform frame-by-frame inference on the video, and for the inference results of each frame, use a dual-path detection strategy of the original detection path and the confidence filtering path to extract the detection information of the drill rod. Enter the frame-by-frame detection main loop and execute the following sub-steps in a loop: Read the current frame image from the video source; exit the loop if reading fails or the video ends; increment the frame counter current_frame by 1. The current frame image is scaled to 640×640 pixels and input into the drill pipe detection model for inference to obtain the detection results; Analyze the inference results, traverse all detection boxes, and extract the category ID, bounding box coordinates, and confidence score for each detection box; Original detection path: Filter drill rod targets, that is, all detection results of type drill rod (the detection results of personnel and safety helmets are not processed in this invention). Regardless of the confidence level, as long as the drill rod appears in the picture, the frame is recorded for subsequent smoothing and counting processing. Confidence filtering path: Only retain detection results with confidence exceeding a preset threshold. Set the threshold to 0.25, calculate the coordinates of the drill pipe target center point, and use them for result reporting and callback push. center_x = (x1 + x2) / 2 center_y = (y1 + y2) / 2 Where x1 and y1 represent the pixel coordinates of the top left corner of the detection box; x2 and y2 represent the pixel coordinates of the lower right corner of the detection box; center_x represents the x-coordinate of the center point of the detection box, i.e. (x1 + x2) / 2, which represents the horizontal position of the drill rod target in the image; center_y represents the ordinate of the center point of the detection box, i.e. (y1 + y2) / 2, which represents the vertical position of the drill rod target in the image; Set the drill pipe detection presence flag for the current frame: Based on the original detection path, if at least one drill pipe target is detected, mark the "whether it includes drill pipe classification target (has_class_1)" of the frame as 1, otherwise mark it as 0; Record the structured data of the current frame, including the current frame number, whether it contains drill pipe, the number of detections, and the confidence level; Release the video capture resources after the above loop ends; The entire set of frame-by-frame structured data accumulated in the main loop is directly constructed into a data structure in memory, including the current frame number, whether it contains drill rods, the number of detections, and the confidence level.

[0024] The training dataset used by the pre-trained model in step 1 includes drill pipe images under different dust concentrations, different lighting conditions, and different drilling angles captured in video, in order to improve the model's generalization ability in complex downhole environments.

[0025] In step 1, the input video frame image is scaled to a preset size, such as 640×640 pixels, to balance detection accuracy and inference speed.

[0026] Step 2: Smooth the timing of drill pipe inspection sequence; Combination Figure 2 Step 2 is implemented in the following steps: In real-world testing, fragmentation issues arise due to factors such as occlusion, lighting variations, and dust. A state machine-based gap scanning and triple-constraint filling algorithm is employed to smooth the original drill pipe testing sequence, correcting gaps caused by temporary missed detections. (Reference) Figure 3 And Figures 4(a) and 4(b), specifically including: The memory data structure constructed in step 1 is used as the input data source for smoothing. If the frame data set is empty, an empty result is returned directly. Create a copy of the data to avoid modifying the original data, and extract the "whether it includes drill pipe classification target" column, i.e., the has_class_1 column, which is a numerical array containing drill pipe frames; The drill pipe is traversed to detect the existence marker sequence. A state machine method is used to identify all continuous undetected intervals, i.e., continuous frames where the drill pipe has not been detected, i.e., continuous intervals with a has_class_1 value of 0. The start and end indices of each interval are recorded. The state machine defines two states: State A (start = None): The current value is not in the 0 range, meaning the drill pipe is in the detection state or has not yet started; State B (start = interval start index i): Currently in the 0 value interval, that is, the drill pipe is in an undetected state; The state transition rules are as follows: If the current frame has_class_1 = 0 and is in state A, then record the starting position of the interval start = i, and transition to state B; If the current frame has_class_1 = 1 and is in state B, then record the end position of the interval end = i-1, output the interval record (start, end), reset start to None, and transition to state A; If the system is still in state B after the traversal is complete, then record the end position of the interval as the index of the end of the array. For each identified undetected interval (i.e., the interval with a value of 0), perform a triple conditional judgment: Condition 1: Gap length constraint: The gap length is greater than or equal to 1 frame and less than or equal to the preset maximum allowable gap length parameter max_gap; this parameter is set according to the drilling speed and video frame rate and is used to distinguish between brief missed detections and actual departure of the drill pipe; Specifically, the gap length, gap_length = end start + 1, must satisfy the following: 1 ≤ gap_length ≤ max_gap In this embodiment, max_gap is set to 15 frames. This parameter can be set according to the typical missed detection duration in the downhole environment. It needs to be greater than the missed detection time caused by external conditions such as dust obscuring or blurry images, and less than the normal operation interval between two drill pipes, so as to distinguish between short-term missed detection and normal operation interval. Condition 2: Forward Existence Constraint: The drill rod detection existence mark of the previous frame (if it exists) in this interval is "detected", that is, the value is set to 1, that is, the drill rod was in the detected state before the gap; Specifically, check the drill rod detection marker of the previous frame in the interval. If start > 0, take has_class_1[start-1], otherwise consider it as 0. This value must be equal to 1, that is, the drill rod was being detected before the gap. Condition 3: Backward Existence Constraint: The drill rod detection existence mark of the next frame (if it exists) in this interval is "detected", and the value is set to 1 in the sequence, that is, the drill rod is detected again after the gap; Specifically, check the drill rod detection marker in the next frame after the interval. If end < array length - 1, then take has_class_1[end+1], otherwise consider it as 0; this value must be equal to 1, that is, the drill rod is detected again after the gap. For intervals that simultaneously meet the above three conditions, modify the drill pipe detection presence flag of all its frames to "detected", that is, set the value to 1, to fix the temporary missed detection caused by downhole factors such as dust obstruction, image blurring, and drilling rig vibration; Output the smoothed drill pipe detection sequence data df_smoothed.

[0027] In step 2, the maximum allowable gap length parameter `max_gap` is a configurable parameter that is dynamically adjusted based on the following factors: Video frame rate: The higher the frame rate, the more frames are in the same time interval, and max_gap should be increased accordingly; Drilling speed: The faster the drilling speed, the shorter the time the drill rod takes to pass through the detection area, and max_gap should be reduced accordingly; Environmental factors: In scenes with high dust concentration and significant image blur, the probability of brief missed detections is higher, so max_gap can be appropriately increased.

[0028] Step 3: Extraction of continuous drill pipe intervals and adaptive selection using dynamic thresholds; Step 4: Drill pipe count and drilling statistics output.

[0029] Example 4 This invention relates to a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm. The flowchart is as follows: Figure 1 As shown, please follow these steps: Step 1: Frame-by-frame detection and data recording of drill pipe targets; Step 1 is implemented in the following steps: like Figure 1 As shown, this invention is applied to underground drilling operations in coal mines. A high-definition explosion-proof camera is placed on the side and rear of the drilling rig, with the camera's field of view covering the drill pipe loading and unloading area, recording the entire drilling process.

[0030] From historical videos of underground drilling operations in coal mines, frames are randomly extracted at intervals. The drill rod targets in the images are labeled using a labeling tool, with the category ID set to 1. The labeled data is divided into three parts: training set, validation set, and test set. The YOLO series target detection model is trained according to the proportions to obtain the weight file. Initialize the runtime environment, detect the GPU environment, and prioritize using the GPU for inference to improve inference speed; if the GPU is unavailable, fall back to the CPU. Load the drill pipe detection model, load the trained weight file, and perform the CUDA environment warm-up operation before the first inference, that is, perform one inference with an empty tensor, triggering environment initialization, model loading, CUDA kernel compilation and cache initialization. Open the drilling operation monitoring video source, open the video file through the interface, and obtain video metadata, including video frame rate and total number of frames. If the video source cannot be opened, throw an exception and terminate. Load the pre-trained YOLO object detection model mentioned earlier, perform frame-by-frame inference on the video, and for the inference results of each frame, use a dual-path detection strategy of the original detection path and the confidence filtering path to extract the detection information of the drill rod. Enter the frame-by-frame detection main loop and execute the following sub-steps in a loop: Read the current frame image from the video source; exit the loop if reading fails or the video ends; increment the frame counter current_frame by 1. The current frame image is scaled to 640×640 pixels and input into the drill pipe detection model for inference to obtain the detection results; Analyze the inference results, traverse all detection boxes, and extract the category ID, bounding box coordinates, and confidence score for each detection box; Original detection path: Filter drill rod targets, that is, all detection results of type drill rod (the detection results of personnel and safety helmets are not processed in this invention). Regardless of the confidence level, as long as the drill rod appears in the picture, the frame is recorded for subsequent smoothing and counting processing. Confidence filtering path: Only retain detection results with confidence exceeding a preset threshold. Set the threshold to 0.25, calculate the coordinates of the drill pipe target center point, and use them for result reporting and callback push. center_x = (x1 + x2) / 2 center_y = (y1 + y2) / 2 Where x1 and y1 represent the pixel coordinates of the top left corner of the detection box; x2 and y2 represent the pixel coordinates of the lower right corner of the detection box; center_x represents the x-coordinate of the center point of the detection box, i.e. (x1 + x2) / 2, which represents the horizontal position of the drill rod target in the image; center_y represents the ordinate of the center point of the detection box, i.e. (y1 + y2) / 2, which represents the vertical position of the drill rod target in the image; Set the drill pipe detection presence flag for the current frame: Based on the original detection path, if at least one drill pipe target is detected, mark the "whether it includes drill pipe classification target (has_class_1)" of the frame as 1, otherwise mark it as 0; Record the structured data of the current frame, including the current frame number, whether it contains drill pipe, the number of detections, and the confidence level; Release the video capture resources after the above loop ends; The entire set of frame-by-frame structured data accumulated in the main loop is directly constructed into a data structure in memory, including the current frame number, whether it contains drill rods, the number of detections, and the confidence level.

[0031] The training dataset used by the pre-trained model in step 1 includes drill pipe images under different dust concentrations, different lighting conditions, and different drilling angles captured in video, in order to improve the model's generalization ability in complex downhole environments.

[0032] In step 1, the input video frame image is scaled to a preset size, such as 640×640 pixels, to balance detection accuracy and inference speed.

[0033] Step 2: Smooth the timing of drill pipe inspection sequence; Step 2 is implemented in the following steps: In real-world testing, fragmentation issues arise due to factors such as occlusion, lighting variations, and dust. A state machine-based gap scanning and triple-constraint filling algorithm is employed to smooth the original drill pipe testing sequence, correcting gaps caused by temporary missed detections. (Reference) Figure 3 And Figures 4(a) and 4(b), specifically including: The memory data structure constructed in step 1 is used as the input data source for smoothing. If the frame data set is empty, an empty result is returned directly. Create a copy of the data to avoid modifying the original data, and extract the "whether it includes drill pipe classification target" column, i.e., the has_class_1 column, which is a numerical array containing drill pipe frames; The drill pipe is traversed to detect the existence marker sequence. A state machine method is used to identify all continuous undetected intervals, i.e., continuous frames where the drill pipe has not been detected, i.e., continuous intervals with a has_class_1 value of 0. The start and end indices of each interval are recorded. The state machine defines two states: State A (start = None): The current value is not in the 0 range, meaning the drill pipe is in the detection state or has not yet started; State B (start = interval start index i): Currently in the 0 value interval, that is, the drill pipe is in an undetected state; The state transition rules are as follows: If the current frame has_class_1 = 0 and is in state A, then record the starting position of the interval start = i, and transition to state B; If the current frame has_class_1 = 1 and is in state B, then record the end position of the interval end = i-1, output the interval record (start, end), reset start to None, and transition to state A; If the system is still in state B after the traversal is complete, then record the end position of the interval as the index of the end of the array. For each identified undetected interval (i.e., the interval with a value of 0), perform a triple conditional judgment: Condition 1: Gap length constraint: The gap length is greater than or equal to 1 frame and less than or equal to the preset maximum allowable gap length parameter max_gap; this parameter is set according to the drilling speed and video frame rate and is used to distinguish between brief missed detections and actual departure of the drill pipe; Specifically, the gap length, gap_length = end start + 1, must satisfy the following: 1 ≤ gap_length ≤ max_gap In this embodiment, max_gap is set to 15 frames. This parameter can be set according to the typical missed detection duration in the downhole environment. It needs to be greater than the missed detection time caused by external conditions such as dust obscuring or blurry images, and less than the normal operation interval between two drill pipes, so as to distinguish between short-term missed detection and normal operation interval. Condition 2: Forward Existence Constraint: The drill rod detection existence mark of the previous frame (if it exists) in this interval is "detected", that is, the value is set to 1, that is, the drill rod was in the detected state before the gap; Specifically, check the drill rod detection marker of the previous frame in the interval. If start > 0, take has_class_1[start-1], otherwise consider it as 0. This value must be equal to 1, that is, the drill rod was being detected before the gap. Condition 3: Backward Existence Constraint: The drill rod detection existence mark of the next frame (if it exists) in this interval is "detected", and the value is set to 1 in the sequence, that is, the drill rod is detected again after the gap; Specifically, check the drill rod detection marker in the next frame after the interval. If end < array length - 1, then take has_class_1[end+1], otherwise consider it as 0; this value must be equal to 1, that is, the drill rod is detected again after the gap. For intervals that simultaneously meet the above three conditions, modify the drill pipe detection presence flag of all its frames to "detected", that is, set the value to 1, to fix the temporary missed detection caused by downhole factors such as dust obstruction, image blurring, and drilling rig vibration; Output the smoothed drill pipe detection sequence data df_smoothed.

[0034] In step 2, the maximum allowable gap length parameter `max_gap` is a configurable parameter that is dynamically adjusted based on the following factors: Video frame rate: The higher the frame rate, the more frames are in the same time interval, and max_gap should be increased accordingly; Drilling speed: The faster the drilling speed, the shorter the time the drill rod takes to pass through the detection area, and max_gap should be reduced accordingly; Environmental factors: In scenes with high dust concentration and significant image blur, the probability of brief missed detections is higher, so max_gap can be appropriately increased.

[0035] Step 3: Extraction of continuous drill pipe intervals and adaptive selection using dynamic thresholds; Step 3 is implemented in the following steps: Step 3.1: Use a state machine-based interval scanning algorithm to extract continuous intervals from the smoothed drill rod detection sequence, and use a dynamic threshold adaptive algorithm to calculate the optimal minimum number of consecutive frames threshold, so as to convert the frame-by-frame detection results into intervals of the actual drill rod drilling process. Step 3.2: Traverse the smoothed drill pipe detection existence marker sequence, use the state machine method to identify all continuous detected intervals, that is, the continuous frame segments in which the drill pipe is continuously detected, and record the start frame number, end frame number and interval length of each interval; Step 3.3: Use the preset minimum number of consecutive frames (default value 15 frames) for preliminary screening. Traverse all intervals with a value of 1, calculate the interval length, and select intervals with an interval length greater than the preset minimum number of consecutive frames as candidate drill rod intervals. Record the interval information to the candidate list, including four types of information: start frame number, end frame number, and number of consecutive frames. Discard intervals with a length less than the preset threshold, as these intervals are considered to be noise false detections. Step 3.4: Perform dynamic threshold adaptive calculation on the candidate interval list to determine the optimal minimum drill pipe duration frame threshold; Step 3.5: Use the dynamically calculated threshold to perform a secondary screening of the candidate intervals, retain only the continuous detection intervals with an interval length greater than or equal to the dynamic threshold as the final continuous drill pipe intervals, and filter out the scattered detection segments formed by noise or brief false detections. Step 3.6: Output a list of consecutive drill pipe intervals that meet the conditions. The number of intervals in the list is the drill pipe count result (the number of drill pipes that meet the dynamic minimum duration condition). Each interval includes the start frame position, end frame position, number of consecutive frames, and average confidence information within the interval, which are used for subsequent drill pipe counting and drilling time period analysis.

[0036] Step 4: Drill pipe count and drilling statistics output.

[0037] Example 5 This invention relates to a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm. The flowchart is as follows: Figure 1 As shown, please follow these steps: Step 1: Frame-by-frame detection and data recording of drill pipe targets; Step 1 is implemented in the following steps: like Figure 1 As shown, this invention is applied to underground drilling operations in coal mines. A high-definition explosion-proof camera is placed on the side and rear of the drilling rig, with the camera's field of view covering the drill pipe loading and unloading area, recording the entire drilling process.

[0038] From historical videos of underground drilling operations in coal mines, frames are randomly extracted at intervals. The drill rod targets in the images are labeled using a labeling tool, with the category ID set to 1. The labeled data is divided into three parts: training set, validation set, and test set. The YOLO series target detection model is trained according to the proportions to obtain the weight file. Initialize the runtime environment, detect the GPU environment, and prioritize using the GPU for inference to improve inference speed; if the GPU is unavailable, fall back to the CPU. Load the drill pipe detection model, load the trained weight file, and perform the CUDA environment warm-up operation before the first inference, that is, perform one inference with an empty tensor, triggering environment initialization, model loading, CUDA kernel compilation and cache initialization. Open the drilling operation monitoring video source, open the video file through the interface, and obtain video metadata, including video frame rate and total number of frames. If the video source cannot be opened, throw an exception and terminate. Load the pre-trained YOLO object detection model mentioned earlier, perform frame-by-frame inference on the video, and for the inference results of each frame, use a dual-path detection strategy of the original detection path and the confidence filtering path to extract the detection information of the drill rod. Enter the frame-by-frame detection main loop and execute the following sub-steps in a loop: Read the current frame image from the video source; exit the loop if reading fails or the video ends; increment the frame counter current_frame by 1. The current frame image is scaled to 640×640 pixels and input into the drill pipe detection model for inference to obtain the detection results; Analyze the inference results, traverse all detection boxes, and extract the category ID, bounding box coordinates, and confidence score for each detection box; Original detection path: Filter drill rod targets, that is, all detection results of type drill rod (the detection results of personnel and safety helmets are not processed in this invention). Regardless of the confidence level, as long as the drill rod appears in the picture, the frame is recorded for subsequent smoothing and counting processing. Confidence filtering path: Only retain detection results with confidence exceeding a preset threshold. Set the threshold to 0.25, calculate the coordinates of the drill pipe target center point, and use them for result reporting and callback push. center_x = (x1 + x2) / 2 center_y = (y1 + y2) / 2 Where x1 and y1 represent the pixel coordinates of the top left corner of the detection box; x2 and y2 represent the pixel coordinates of the lower right corner of the detection box; center_x represents the x-coordinate of the center point of the detection box, i.e. (x1 + x2) / 2, which represents the horizontal position of the drill rod target in the image; center_y represents the ordinate of the center point of the detection box, i.e. (y1 + y2) / 2, which represents the vertical position of the drill rod target in the image; Set the drill pipe detection presence flag for the current frame: Based on the original detection path, if at least one drill pipe target is detected, mark the "whether it includes drill pipe classification target (has_class_1)" of the frame as 1, otherwise mark it as 0; Record the structured data of the current frame, including the current frame number, whether it contains drill pipe, the number of detections, and the confidence level; Release the video capture resources after the above loop ends; The entire set of frame-by-frame structured data accumulated in the main loop is directly constructed into a data structure in memory, including the current frame number, whether it contains drill rods, the number of detections, and the confidence level.

[0039] The training dataset used by the pre-trained model in step 1 includes drill pipe images under different dust concentrations, different lighting conditions, and different drilling angles captured in video, in order to improve the model's generalization ability in complex downhole environments.

[0040] In step 1, the input video frame image is scaled to a preset size, such as 640×640 pixels, to balance detection accuracy and inference speed.

[0041] Step 2: Smooth the timing of drill pipe inspection sequence; Step 2 is implemented in the following steps: In real-world testing, fragmentation issues arise due to factors such as occlusion, lighting variations, and dust. A state machine-based gap scanning and triple-constraint filling algorithm is employed to smooth the original drill pipe testing sequence, correcting gaps caused by temporary missed detections. (Reference) Figure 3 And Figures 4(a) and 4(b), specifically including: The memory data structure constructed in step 1 is used as the input data source for smoothing. If the frame data set is empty, an empty result is returned directly. Create a copy of the data to avoid modifying the original data, and extract the "whether it includes drill pipe classification target" column, i.e., the has_class_1 column, which is a numerical array containing drill pipe frames; The drill pipe is traversed to detect the existence marker sequence. A state machine method is used to identify all continuous undetected intervals, i.e., continuous frames where the drill pipe has not been detected, i.e., continuous intervals with a has_class_1 value of 0. The start and end indices of each interval are recorded. The state machine defines two states: State A (start = None): The current value is not in the 0 range, meaning the drill pipe is in the detection state or has not yet started; State B (start = interval start index i): Currently in the 0 value interval, that is, the drill pipe is in an undetected state; The state transition rules are as follows: If the current frame has_class_1 = 0 and is in state A, then record the starting position of the interval start = i, and transition to state B; If the current frame has_class_1 = 1 and is in state B, then record the end position of the interval end = i-1, output the interval record (start, end), reset start to None, and transition to state A; If the system is still in state B after the traversal is complete, then record the end position of the interval as the index of the end of the array. For each identified undetected interval (i.e., the interval with a value of 0), perform a triple conditional judgment: Condition 1: Gap length constraint: The gap length is greater than or equal to 1 frame and less than or equal to the preset maximum allowable gap length parameter max_gap; this parameter is set according to the drilling speed and video frame rate and is used to distinguish between brief missed detections and actual departure of the drill pipe; Specifically, the gap length, gap_length = end start + 1, must satisfy the following: 1 ≤ gap_length ≤ max_gap In this embodiment, max_gap is set to 15 frames. This parameter can be set according to the typical missed detection duration in the downhole environment. It needs to be greater than the missed detection time caused by external conditions such as dust obscuring or blurry images, and less than the normal operation interval between two drill pipes, so as to distinguish between short-term missed detection and normal operation interval. Condition 2: Forward Existence Constraint: The drill rod detection existence mark of the previous frame (if it exists) in this interval is "detected", that is, the value is set to 1, that is, the drill rod was in the detected state before the gap; Specifically, check the drill rod detection marker of the previous frame in the interval. If start > 0, take has_class_1[start-1], otherwise consider it as 0. This value must be equal to 1, that is, the drill rod was being detected before the gap. Condition 3: Backward Existence Constraint: The drill rod detection existence mark of the next frame (if it exists) in this interval is "detected", and the value is set to 1 in the sequence, that is, the drill rod is detected again after the gap; Specifically, check the drill rod detection marker in the next frame after the interval. If end < array length - 1, then take has_class_1[end+1], otherwise consider it as 0; this value must be equal to 1, that is, the drill rod is detected again after the gap. For intervals that simultaneously meet the above three conditions, modify the drill pipe detection presence flag of all its frames to "detected", that is, set the value to 1, to fix the temporary missed detection caused by downhole factors such as dust obstruction, image blurring, and drilling rig vibration; Output the smoothed drill pipe detection sequence data df_smoothed.

[0042] In step 2, the maximum allowable gap length parameter `max_gap` is a configurable parameter that is dynamically adjusted based on the following factors: Video frame rate: The higher the frame rate, the more frames are in the same time interval, and max_gap should be increased accordingly; Drilling speed: The faster the drilling speed, the shorter the time the drill rod takes to pass through the detection area, and max_gap should be reduced accordingly; Environmental factors: In scenes with high dust concentration and significant image blur, the probability of brief missed detections is higher, so max_gap can be appropriately increased.

[0043] Step 3: Extraction of continuous drill pipe intervals and adaptive selection using dynamic thresholds; Step 3 is implemented in the following steps: Step 3.1: Use a state machine-based interval scanning algorithm to extract continuous intervals from the smoothed drill rod detection sequence, and use a dynamic threshold adaptive algorithm to calculate the optimal minimum number of consecutive frames threshold, so as to convert the frame-by-frame detection results into intervals of the actual drill rod drilling process. The minimum consecutive frame threshold in step 3.1 is calculated using a dynamic adaptive algorithm, which can be automatically calculated based on the following factors: Jump points in the candidate interval length sequence can be used to distinguish noise segments from real drill pipe segments. The minimum number of visible frames for a single drill rod within the detection area is calculated from the drill rod length, drilling speed, and frame rate. Business requirements necessitate filtering the minimum detection duration.

[0044] Step 3.2: Traverse the smoothed drill pipe detection existence marker sequence, use the state machine method to identify all continuous detected intervals, that is, the continuous frame segments in which the drill pipe is continuously detected, and record the start frame number, end frame number and interval length of each interval; Step 3.3: Use the preset minimum number of consecutive frames (default value 15 frames) for preliminary screening. Traverse all intervals with a value of 1, calculate the interval length, and select intervals with an interval length greater than the preset minimum number of consecutive frames as candidate drill rod intervals. Record the interval information to the candidate list, including four types of information: start frame number, end frame number, and number of consecutive frames. Discard intervals with a length less than the preset threshold, as these intervals are considered to be noise false detections. Step 3.4: Perform dynamic threshold adaptive calculation on the candidate interval list to determine the optimal minimum drill pipe duration frame threshold; Step 3.4 determines the optimal minimum drill pipe duration frame threshold, and the specific algorithm is as follows: (1) Extract the length sequence of all candidate intervals and sort them in ascending order; (2) Traverse the sorted length sequence, and calculate the ratio of each two adjacent intervals using the interval length (the length of the previous interval divided by the length of the next interval), that is, the ratio of the number of frames contained in the adjacent intervals. Find the point with the maximum ratio, that is, the jump point. This jump point is the boundary between the noise interference segment and the real drill pipe segment. (3) If the maximum ratio exceeds the preset significance threshold, in this embodiment the threshold is set to 2, and the jump point index is within the effective range, then the length of the first segment after the jump point is used as the dynamic minimum drill string duration frame threshold. (4) If all segments are of similar length, i.e. there is no obvious jump, and the maximum ratio does not exceed the significance threshold, then the median of the length sequence is multiplied by the preset ratio coefficient (0.6 in this embodiment) as the basic dynamic threshold, and it is not less than the default minimum number of consecutive frames of 15 frames. The dynamic threshold algorithm design in this embodiment has the following technical significance: (1) Jump point detection utilizes the difference in duration between noise segments and real drill pipe segments. Usually, noise segments last only a few dozen frames, while real drill pipes usually exist in the picture for hundreds of frames or more, and there is a significant difference in duration between the two. (2) The median strategy ensures that even in scenarios without obvious jump points, such as when all drill pipes have similar durations, a reasonable threshold can be obtained, thus avoiding misclassifying all segments as noise.

[0045] Step 3.5: Use the dynamically calculated threshold to perform a secondary screening of the candidate intervals, retain only the continuous detection intervals with an interval length greater than or equal to the dynamic threshold as the final continuous drill pipe intervals, and filter out the scattered detection segments formed by noise or brief false detections. Step 3.6: Output a list of consecutive drill pipe intervals that meet the conditions. The number of intervals in the list is the drill pipe count result (the number of drill pipes that meet the dynamic minimum duration condition). Each interval includes the start frame position, end frame position, number of consecutive frames, and average confidence information within the interval, which are used for subsequent drill pipe counting and drilling time period analysis.

[0046] Step 4: Drill pipe count and drilling statistics output.

[0047] Example 6 This invention relates to a method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm. The flowchart is as follows: Figure 1 As shown, please follow these steps: Step 1: Frame-by-frame detection and data recording of drill pipe targets; Step 1 is implemented in the following steps: like Figure 1As shown, this invention is applied to underground drilling operations in coal mines. A high-definition explosion-proof camera is placed on the side and rear of the drilling rig, with the camera's field of view covering the drill pipe loading and unloading area, recording the entire drilling process.

[0048] From historical videos of underground drilling operations in coal mines, frames are randomly extracted at intervals. The drill rod targets in the images are labeled using a labeling tool, with the category ID set to 1. The labeled data is divided into three parts: training set, validation set, and test set. The YOLO series target detection model is trained according to the proportions to obtain the weight file. Initialize the runtime environment, detect the GPU environment, and prioritize using the GPU for inference to improve inference speed; if the GPU is unavailable, fall back to the CPU. Load the drill pipe detection model, load the trained weight file, and perform the CUDA environment warm-up operation before the first inference, that is, perform one inference with an empty tensor, triggering environment initialization, model loading, CUDA kernel compilation and cache initialization. Open the drilling operation monitoring video source, open the video file through the interface, and obtain video metadata, including video frame rate and total number of frames. If the video source cannot be opened, throw an exception and terminate. Load the pre-trained YOLO object detection model mentioned earlier, perform frame-by-frame inference on the video, and for the inference results of each frame, use a dual-path detection strategy of the original detection path and the confidence filtering path to extract the detection information of the drill rod. Enter the frame-by-frame detection main loop and execute the following sub-steps in a loop: Read the current frame image from the video source; exit the loop if reading fails or the video ends; increment the frame counter current_frame by 1. The current frame image is scaled to 640×640 pixels and input into the drill pipe detection model for inference to obtain the detection results; Analyze the inference results, traverse all detection boxes, and extract the category ID, bounding box coordinates, and confidence score for each detection box; Original detection path: Filter drill rod targets, that is, all detection results of type drill rod (the detection results of personnel and safety helmets are not processed in this invention). Regardless of the confidence level, as long as the drill rod appears in the picture, the frame is recorded for subsequent smoothing and counting processing. Confidence filtering path: Only retain detection results with confidence exceeding a preset threshold. Set the threshold to 0.25, calculate the coordinates of the drill pipe target center point, and use them for result reporting and callback push. center_x = (x1 + x2) / 2 center_y = (y1 + y2) / 2 Where x1 and y1 represent the pixel coordinates of the top left corner of the detection box; x2 and y2 represent the pixel coordinates of the lower right corner of the detection box; center_x represents the x-coordinate of the center point of the detection box, i.e. (x1 + x2) / 2, which represents the horizontal position of the drill rod target in the image; center_y represents the ordinate of the center point of the detection box, i.e. (y1 + y2) / 2, which represents the vertical position of the drill rod target in the image; Set the drill pipe detection presence flag for the current frame: Based on the original detection path, if at least one drill pipe target is detected, mark the "whether it includes drill pipe classification target (has_class_1)" of the frame as 1, otherwise mark it as 0; Record the structured data of the current frame, including the current frame number, whether it contains drill pipe, the number of detections, and the confidence level; Release the video capture resources after the above loop ends; The entire set of frame-by-frame structured data accumulated in the main loop is directly constructed into a data structure in memory, including the current frame number, whether it contains drill rods, the number of detections, and the confidence level.

[0049] The training dataset used by the pre-trained model in step 1 includes drill pipe images under different dust concentrations, different lighting conditions, and different drilling angles captured in video, in order to improve the model's generalization ability in complex downhole environments.

[0050] In step 1, the input video frame image is scaled to a preset size, such as 640×640 pixels, to balance detection accuracy and inference speed.

[0051] Step 2: Smooth the timing of drill pipe inspection sequence; Step 2 is implemented in the following steps: In real-world testing, fragmentation issues arise due to factors such as occlusion, lighting variations, and dust. A state machine-based gap scanning and triple-constraint filling algorithm is employed to smooth the original drill pipe testing sequence, correcting gaps caused by temporary missed detections. (Reference) Figure 3 And Figures 4(a) and 4(b), specifically including: The memory data structure constructed in step 1 is used as the input data source for smoothing. If the frame data set is empty, an empty result is returned directly. Create a copy of the data to avoid modifying the original data, and extract the "whether it includes drill pipe classification target" column, i.e., the has_class_1 column, which is a numerical array containing drill pipe frames; The drill pipe is traversed to detect the existence marker sequence. A state machine method is used to identify all continuous undetected intervals, i.e., continuous frames where the drill pipe has not been detected, i.e., continuous intervals with a has_class_1 value of 0. The start and end indices of each interval are recorded. The state machine defines two states: State A (start = None): The current value is not in the 0 range, meaning the drill pipe is in the detection state or has not yet started; State B (start = interval start index i): Currently in the 0 value interval, that is, the drill pipe is in an undetected state; The state transition rules are as follows: If the current frame has_class_1 = 0 and is in state A, then record the starting position of the interval start = i, and transition to state B; If the current frame has_class_1 = 1 and is in state B, then record the end position of the interval end = i-1, output the interval record (start, end), reset start to None, and transition to state A; If the system is still in state B after the traversal is complete, then record the end position of the interval as the index of the end of the array. For each identified undetected interval (i.e., the interval with a value of 0), perform a triple conditional judgment: Condition 1: Gap length constraint: The gap length is greater than or equal to 1 frame and less than or equal to the preset maximum allowable gap length parameter max_gap; this parameter is set according to the drilling speed and video frame rate and is used to distinguish between brief missed detections and actual departure of the drill pipe; Specifically, the gap length, gap_length = end start + 1, must satisfy the following: 1 ≤ gap_length ≤ max_gap In this embodiment, max_gap is set to 15 frames. This parameter can be set according to the typical missed detection duration in the downhole environment. It needs to be greater than the missed detection time caused by external conditions such as dust obscuring or blurry images, and less than the normal operation interval between two drill pipes, so as to distinguish between short-term missed detection and normal operation interval. Condition 2: Forward Existence Constraint: The drill rod detection existence mark of the previous frame (if it exists) in this interval is "detected", that is, the value is set to 1, that is, the drill rod was in the detected state before the gap; Specifically, check the drill rod detection marker of the previous frame in the interval. If start > 0, take has_class_1[start-1], otherwise consider it as 0. This value must be equal to 1, that is, the drill rod was being detected before the gap. Condition 3: Backward Existence Constraint: The drill rod detection existence mark of the next frame (if it exists) in this interval is "detected", and the value is set to 1 in the sequence, that is, the drill rod is detected again after the gap; Specifically, check the drill rod detection marker in the next frame after the interval. If end < array length - 1, then take has_class_1[end+1], otherwise consider it as 0; this value must be equal to 1, that is, the drill rod is detected again after the gap. For intervals that simultaneously meet the above three conditions, modify the drill pipe detection presence flag of all its frames to "detected", that is, set the value to 1, to fix the temporary missed detection caused by downhole factors such as dust obstruction, image blurring, and drilling rig vibration; Output the smoothed drill pipe detection sequence data df_smoothed.

[0052] In step 2, the maximum allowable gap length parameter `max_gap` is a configurable parameter that is dynamically adjusted based on the following factors: Video frame rate: The higher the frame rate, the more frames are in the same time interval, and max_gap should be increased accordingly; Drilling speed: The faster the drilling speed, the shorter the time the drill rod takes to pass through the detection area, and max_gap should be reduced accordingly; Environmental factors: In scenes with high dust concentration and significant image blur, the probability of brief missed detections is higher, so max_gap can be appropriately increased.

[0053] Step 3: Extraction of continuous drill pipe intervals and adaptive selection using dynamic thresholds; Step 3 is implemented in the following steps: Step 3.1: Use a state machine-based interval scanning algorithm to extract continuous intervals from the smoothed drill rod detection sequence, and use a dynamic threshold adaptive algorithm to calculate the optimal minimum number of consecutive frames threshold, so as to convert the frame-by-frame detection results into intervals of the actual drill rod drilling process. The minimum consecutive frame threshold in step 3.1 is calculated using a dynamic adaptive algorithm, which can be automatically calculated based on the following factors: Jump points in the candidate interval length sequence can be used to distinguish noise segments from real drill pipe segments. The minimum number of visible frames for a single drill rod within the detection area is calculated from the drill rod length, drilling speed, and frame rate. Business requirements necessitate filtering the minimum detection duration.

[0054] Step 3.2: Traverse the smoothed drill pipe detection existence marker sequence, use the state machine method to identify all continuous detected intervals, that is, the continuous frame segments in which the drill pipe is continuously detected, and record the start frame number, end frame number and interval length of each interval; Step 3.3: Use the preset minimum number of consecutive frames (default value 15 frames) for preliminary screening. Traverse all intervals with a value of 1, calculate the interval length, and select intervals with an interval length greater than the preset minimum number of consecutive frames as candidate drill rod intervals. Record the interval information to the candidate list, including four types of information: start frame number, end frame number, and number of consecutive frames. Discard intervals with a length less than the preset threshold, as these intervals are considered to be noise false detections. Step 3.4: Perform dynamic threshold adaptive calculation on the candidate interval list to determine the optimal minimum drill pipe duration frame threshold; Step 3.4 determines the optimal minimum drill pipe duration frame threshold, and the specific algorithm is as follows: (1) Extract the length sequence of all candidate intervals and sort them in ascending order; (2) Traverse the sorted length sequence, and calculate the ratio of each two adjacent intervals using the interval length (the length of the previous interval divided by the length of the next interval), that is, the ratio of the number of frames contained in the adjacent intervals. Find the point with the maximum ratio, that is, the jump point. This jump point is the boundary between the noise interference segment and the real drill pipe segment. (3) If the maximum ratio exceeds the preset significance threshold, in this embodiment the threshold is set to 2, and the jump point index is within the effective range, then the length of the first segment after the jump point is used as the dynamic minimum drill string duration frame threshold. (4) If all segments are of similar length, i.e. there is no obvious jump, and the maximum ratio does not exceed the significance threshold, then the median of the length sequence is multiplied by the preset ratio coefficient (0.6 in this embodiment) as the basic dynamic threshold, and it is not less than the default minimum number of consecutive frames of 15 frames. The dynamic threshold algorithm design in this embodiment has the following technical significance: (1) Jump point detection utilizes the difference in duration between noise segments and real drill pipe segments. Usually, noise segments last only a few dozen frames, while real drill pipes usually exist in the picture for hundreds of frames or more, and there is a significant difference in duration between the two. (2) The median strategy ensures that even in scenarios without obvious jump points, such as when all drill pipes have similar durations, a reasonable threshold can be obtained, thus avoiding misclassifying all segments as noise.

[0055] Step 3.5: Use the dynamically calculated threshold to perform a secondary screening of the candidate intervals, retain only the continuous detection intervals with an interval length greater than or equal to the dynamic threshold as the final continuous drill pipe intervals, and filter out the scattered detection segments formed by noise or brief false detections. Step 3.6: Output a list of consecutive drill pipe intervals that meet the conditions. The number of intervals in the list is the drill pipe count result (the number of drill pipes that meet the dynamic minimum duration condition). Each interval includes the start frame position, end frame position, number of consecutive frames, and average confidence information within the interval, which are used for subsequent drill pipe counting and drilling time period analysis.

[0056] Step 4: Drill pipe count and drilling statistics output.

[0057] Step 4 is implemented in the following steps: Based on the continuous intervals extracted in step 3, drill pipe counting results and drilling statistics reports are generated, specifically including: Step 4.1: Each consecutive interval that meets the threshold condition corresponds to one drill rod existence event, that is, the complete process of a drill rod from entering the screen to leaving the screen, representing a complete drilling process or drill retraction process. The number of intervals is the drill rod count result. Step 4.2: Convert the number of frames in each interval into actual time based on the video frame rate, and calculate the duration of each drill rod in the frame and the start and end times of drilling; Step 4.3: Calculate drilling statistics, including four categories: cumulative number of drill pipes, cumulative drilling footage, total drilling time, and average loading and unloading time per drill pipe. The drill pipe count results are assembled with drilling statistics into a structured dictionary output, containing the following fields: The drill pipe count result is the number of consecutive intervals that satisfy the dynamic minimum drill pipe duration frame threshold condition. Basic video information, including total number of video frames, video frame rate, total video duration, and algorithm execution time; The drill pipe details list is a detailed list of the intervals in which each drill pipe exists continuously. Each interval includes the start frame number, end frame number, duration of frames, and average confidence level.

Claims

1. A method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm, characterized in that, The specific steps are as follows: Step 1: Frame-by-frame detection and data recording of drill pipe targets; Step 2: Smooth the timing of drill pipe inspection sequence; Step 3: Extraction of continuous drill pipe intervals and adaptive selection using dynamic thresholds; Step 4: Drill pipe count and drilling statistics output.

2. The method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm according to claim 1, characterized in that, Step 1 is implemented in the following steps: From historical videos of underground drilling operations in coal mines, frames are randomly extracted at intervals. The drill rod targets in the images are labeled using a labeling tool, with the category ID set to 1. The labeled data is divided into three parts: training set, validation set, and test set. The YOLO series target detection model is trained according to the proportions to obtain the weight file. Initialize the runtime environment, detect the GPU environment, prioritize using the GPU for inference to improve inference speed, and fall back to the CPU if the GPU is unavailable. Load the drill pipe detection model, load the trained weight file, and perform the CUDA environment warm-up operation before the first inference, that is, perform one inference with an empty tensor, triggering environment initialization, model loading, CUDA kernel compilation and cache initialization. Open the drilling operation monitoring video source, open the video file through the interface, and obtain video metadata, including video frame rate and total number of frames. If the video source cannot be opened, throw an exception and terminate. Load the pre-trained YOLO object detection model mentioned earlier, perform frame-by-frame inference on the video, and for the inference results of each frame, use a dual-path detection strategy of the original detection path and the confidence filtering path to extract the detection information of the drill rod. Enter the frame-by-frame detection main loop and execute the following sub-steps in a loop: Read the current frame image from the video source; exit the loop if reading fails or the video ends; increment the frame counter current_frame by 1. The current frame image is scaled to 640×640 pixels and input into the drill pipe detection model for inference to obtain the detection results; Analyze the inference results, traverse all detection boxes, and extract the category ID, bounding box coordinates, and confidence score for each detection box; Original detection path: Filter drill rod targets, that is, all detection results of type drill rod, regardless of confidence level, as long as drill rod appears in the picture, the frame is recorded for subsequent smoothing and counting processing; Confidence filtering path: Only retain detection results with confidence exceeding a preset threshold. Set the threshold to 0.25, calculate the coordinates of the drill pipe target center point, and use them for result reporting and callback push. center_x = (x1 + x2) / 2 center_y = (y1 + y2) / 2 Where x1 and y1 represent the pixel coordinates of the top left corner of the detection box; x2 and y2 represent the pixel coordinates of the lower right corner of the detection box; center_x represents the x-coordinate of the center point of the detection box, i.e. (x1 + x2) / 2, which represents the horizontal position of the drill rod target in the image; center_y represents the ordinate of the center point of the detection box, i.e. (y1 + y2) / 2, which represents the vertical position of the drill rod target in the image; Set the drill pipe detection presence flag for the current frame: based on the original detection path, if at least one drill pipe target is detected, mark the "whether it includes drill pipe classification target" of the frame as 1, otherwise mark it as 0; Record the structured data of the current frame, including the current frame number, whether it contains drill pipe, the number of detections, and the confidence level; Release the video capture resources after the above loop ends; The entire set of frame-by-frame structured data accumulated in the main loop is directly constructed into a data structure in memory, including the current frame number, whether it contains drill rods, the number of detections, and the confidence level.

3. The method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm according to claim 2, characterized in that, The training dataset used in step 1 for the pre-trained model includes drill pipe images under different dust concentrations, different lighting conditions, and different drilling angles captured in video, in order to improve the model's generalization ability in complex downhole environments.

4. The method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm according to claim 2, characterized in that, Step 2 is implemented in the following steps: The memory data structure constructed in step 1 is used as the input data source for smoothing. If the frame data set is empty, an empty result is returned directly. Create a copy of the data to avoid modifying the original data, and extract the "whether it includes drill pipe classification target" column, i.e., the has_class_1 column, which is a numerical array containing drill pipe frames; The drill pipe is traversed to detect the existence marker sequence. A state machine method is used to identify all continuous undetected intervals, i.e., continuous frames where the drill pipe has not been detected, i.e., continuous intervals with a has_class_1 value of 0. The start and end indices of each interval are recorded. The state machine defines two states: Status A: The current value is not in the 0 range, meaning the drill pipe is either being monitored or has not yet started. Status B: Currently in the 0 value range, meaning the drill pipe is in an undetected state; The state transition rules are as follows: If the current frame has_class_1 = 0 and is in state A, then record the starting position of the interval start = i, and transition to state B; If the current frame has_class_1 = 1 and is in state B, then record the end position of the interval end = i-1, output the interval record (start, end), reset start to None, and transition to state A; If the system is still in state B after the traversal is complete, then record the end position of the interval as the index of the end of the array. For each identified undetected region, perform a triple condition check: Condition 1: Gap length constraint: The gap length is greater than or equal to 1 frame and less than or equal to the preset maximum allowable gap length parameter max_gap; this parameter is set according to the drilling speed and video frame rate and is used to distinguish between brief missed detections and actual departure of the drill pipe; Specifically, the gap length, gap_length = end start + 1, must satisfy the following: 1 ≤ gap_length ≤ max_gap Condition 2: Forward Existence Constraint: The drill rod detection existence flag of the previous frame in this interval is "detected", that is, the value is set to 1, that is, the drill rod was in the detected state before the gap; Specifically, check the drill rod detection marker of the previous frame in the interval. If start > 0, take has_class_1[start-1], otherwise consider it as 0. This value must be equal to 1, that is, the drill rod was being detected before the gap. Condition 3: Backward Existence Constraint: The drill rod detection existence mark of the next frame in this interval is "detected", and the value is set to 1 in the sequence, that is, the drill rod is detected again after the gap; Specifically, check the drill rod detection marker in the next frame after the interval. If end < array length - 1, then take has_class_1[end+1], otherwise consider it as 0; this value must be equal to 1, that is, the drill rod is detected again after the gap. For intervals that simultaneously meet the above three conditions, modify the drill pipe detection presence flag of all its frames to "detected", that is, set the value to 1, to fix the temporary missed detection caused by downhole factors; Output the smoothed drill pipe detection sequence data df_smoothed.

5. The method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm according to claim 4, characterized in that, Step 3 is implemented in the following steps: Step 3.1: Use a state machine-based interval scanning algorithm to extract continuous intervals from the smoothed drill rod detection sequence, and use a dynamic threshold adaptive algorithm to calculate the optimal minimum number of consecutive frames threshold, so as to convert the frame-by-frame detection results into intervals of the actual drill rod drilling process. Step 3.2: Traverse the smoothed drill pipe detection existence marker sequence, use the state machine method to identify all continuous detected intervals, that is, the continuous frame segments in which the drill pipe is continuously detected, and record the start frame number, end frame number and interval length of each interval; Step 3.3: Use the preset minimum number of consecutive frames for preliminary screening, traverse all intervals with a value of 1, calculate the interval length, and select intervals with an interval length greater than the preset minimum number of consecutive frames as candidate drill rod intervals. Record the interval information into the candidate list, including four types of information: start frame number, end frame number, and number of consecutive frames. Discard intervals with a length less than the preset threshold, as these intervals are considered to be noise false detections. Step 3.4: Perform dynamic threshold adaptive calculation on the candidate interval list to determine the optimal minimum drill pipe duration frame threshold; Step 3.5: Use the dynamically calculated threshold to perform a secondary screening of the candidate intervals, retain only the continuous detection intervals with an interval length greater than or equal to the dynamic threshold as the final continuous drill pipe intervals, and filter out the scattered detection segments formed by noise or brief false detections. Step 3.6: Output a list of consecutive drill pipe intervals that meet the conditions. The number of intervals in the list is the drill pipe count result. Each interval includes the start frame position, end frame position, number of consecutive frames, and average confidence information within the interval, which is used for subsequent drill pipe counting and drilling time period analysis.

6. The method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm according to claim 5, characterized in that, The minimum consecutive frame threshold in step 3.1 is calculated using a dynamic adaptive algorithm, which can be automatically calculated based on the following factors: Jump points in the candidate interval length sequence can be used to distinguish noise segments from real drill pipe segments. The minimum number of visible frames for a single drill rod within the detection area is calculated from the drill rod length, drilling speed, and frame rate. Business requirements necessitate filtering the minimum detection duration.

7. The method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm according to claim 6, characterized in that, In step 3.4, the optimal minimum drill pipe duration frame threshold is determined using the following algorithm: (1) Extract the length sequence of all candidate intervals and sort them in ascending order; (2) Traverse the sorted length sequence, calculate the ratio of each two adjacent intervals using the interval length, that is, the ratio of the number of frames contained in the adjacent intervals, and find the point with the maximum ratio, that is, the jump point. This jump point is the boundary between the noise interference segment and the real drill pipe segment. (3) If the maximum ratio exceeds the preset significance threshold, in this embodiment the threshold is set to 2, and the jump point index is within the effective range, then the length of the first segment after the jump point is used as the dynamic minimum drill rod continuous frame number threshold. (4) If all segments are of similar length, i.e. there is no obvious jump, and the maximum ratio does not exceed the significance threshold, then the median of the length sequence multiplied by the preset ratio coefficient is taken as the basic dynamic threshold, and it is not less than the default minimum number of consecutive frames of 15 frames.

8. The method for counting drill pipes in coal mines based on a visual detection sequence smoothing algorithm according to claim 7, characterized in that, Step 4 is implemented in the following steps: Based on the continuous intervals extracted in step 3, drill pipe counting results and drilling statistics reports are generated, specifically including: Step 4.1: Each consecutive interval that meets the threshold condition corresponds to one drill rod existence event, that is, the complete process of a drill rod from entering the screen to leaving the screen, representing a complete drilling process or drill retraction process. The number of intervals is the drill rod count result. Step 4.2: Convert the number of frames in each interval into actual time based on the video frame rate, and calculate the duration of each drill rod in the frame and the start and end times of drilling; Step 4.3: Calculate drilling statistics, including four categories: cumulative number of drill pipes, cumulative drilling footage, total drilling time, and average loading and unloading time per drill pipe. The drill pipe count results are assembled with drilling statistics into a structured dictionary output, containing the following fields: The drill pipe count result is the number of consecutive intervals that satisfy the dynamic minimum drill pipe duration frame threshold condition. Basic video information, including total number of video frames, video frame rate, total video duration, and algorithm execution time; The drill pipe details list is a detailed list of the intervals in which each drill pipe exists continuously. Each interval includes the start frame number, end frame number, duration of frames, and average confidence level.

Images