A coal mine underground drill rod counting method, device and equipment based on instance segmentation

Abstract

This application discloses a method, apparatus, and equipment for counting drill rods in coal mines based on instance segmentation, relating to the field of artificial intelligence technology. The method includes: inputting a drilling rig operation video sequence into a trained instance segmentation model to segment the drilling rig body and drill rods within the current input frame image; obtaining the pixel areas of the segmented drill rod mask contour and the drilling rig body mask contour, and calculating the ratio of the drilling rig body mask area to the drill rod mask area; analyzing the changing trends of multiple mask area ratios before the current time; if the ratio trend is continuously increasing, determining whether the mask area ratio is greater than a first area ratio threshold, and if so, performing a drilling entry counting operation; if the ratio trend is continuously decreasing, determining whether the mask area ratio is less than a second area ratio threshold, and if so, performing a drilling exit counting operation. This application can improve the accuracy of drill rod counting and avoid misidentification and missed identification of drilling entry and exit.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and equipment for counting drill pipes in coal mines based on instance segmentation. Background Technology

[0002] Currently, in order to ensure coal mine safety and improve production efficiency, it is usually necessary to preserve drilling videos in accordance with regulations and have them inspected regularly by supervisory agencies. The number of drill rods driven in or out of the video is used to determine whether the drilling depth has reached the planned advance. In other words, the number of drill rods is counted to ensure that the gas content in the coal seam is reduced to a safe range before mining.

[0003] However, the current mainstream drill rod counting methods have the following shortcomings in the video analysis scenario of mine drilling: (1) Insufficient perception ability, making it difficult to adapt to the complex environment in the mine: Due to the interference factors such as uneven lighting, redundant background, and severe target occlusion in the underground images, the current models (such as YOLOv5, YOLOv7, etc.) have weak response to the edge of slender targets such as drill rods and ambiguous semantic information in the feature extraction stage, resulting in frequent misidentification and omission; (2) Lack of normalization in the recognition logic for different working states of the drilling rig, and the drill rod counting results are easily affected by external conditions: The current mainstream methods mostly rely on the action classification of video frames, trajectory peaks or worker behavior recognition as the basis for state judgment, and lack robust features that are independent of image scale and video compression. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, apparatus, and equipment for counting drill pipes in coal mines based on instance segmentation, which can improve the accuracy of drill pipe counting and avoid misidentification and omission during drill advance and retreat. The specific solution is as follows:

[0005] In a first aspect, this application discloses a method for counting drill pipes in coal mines based on instance segmentation, including:

[0006] Obtain the operation video of the drilling rig in the target coal mine to obtain the drilling rig operation video sequence;

[0007] Each frame of the drilling rig operation video sequence is sequentially input into the trained instance segmentation model to perform instance segmentation of the drilling rig body and drill rod within the current input frame image, obtaining the drill rod mask contour and the drilling rig body mask contour. The trained instance segmentation model is a model obtained by training an initial instance segmentation model based on an attention mechanism using a training set. The training set includes real drilling video data from multiple coal mine underground scenarios.

[0008] Obtain the pixel area of ​​the drill rod mask contour and the drill rig body mask contour corresponding to the current input frame image, and obtain the current drill rod mask area and the current drill rig body mask area;

[0009] Calculate the ratio of the current drilling rig body mask area to the current drill rod mask area to obtain the mask area ratio, and analyze the changing trend of multiple mask area ratios corresponding to multiple frames of images within a preset time period before the current moment to obtain the ratio changing trend;

[0010] If the ratio changes in a continuously increasing trend, the current drill pipe state is determined to be the drilling state, and it is determined whether the current mask area ratio is greater than the first area ratio threshold. If so, the drilling count is incremented by 1.

[0011] If the ratio changes in a continuously decreasing trend, the current drill pipe state is determined to be a drill retraction state, and it is determined whether the current mask area ratio is less than the second area ratio threshold. If so, the drill retraction count is incremented by 1. The second area ratio threshold is less than the first area ratio threshold.

[0012] Optionally, determining whether the current mask area ratio is greater than a first area ratio threshold, and if so, performing a count operation of incrementing the drill number by 1, includes:

[0013] Determine whether the current mask area ratio is greater than the first area ratio threshold. If the current mask area ratio is greater than the first area ratio threshold, determine whether the current system's counting state is in a countable state.

[0014] If the current system's counting state is in a countable state, then perform a count operation to increment the drilling count by 1 to obtain the drilling count statistics;

[0015] Accordingly, after the counting operation of incrementing the drilling count by 1, the method further includes:

[0016] Set the current system's counting state to uncountable.

[0017] Optionally, determining whether the current mask area ratio is less than the second area ratio threshold, and if so, performing a count operation of incrementing the number of un-drilled areas by 1, includes:

[0018] Determine whether the current mask area ratio is less than the second area ratio threshold. If the current mask area ratio is less than the second area ratio threshold, determine whether the current system's counting state is in a countable state.

[0019] If the current system's counting state is in a countable state, then perform a counting operation to increment the number of drill withdrawals by 1 to obtain the statistical result of the number of drill withdrawals;

[0020] Accordingly, after the counting operation of incrementing the number of drill retractions by 1, the process also includes:

[0021] Set the current system's counting state to uncountable.

[0022] Optionally, after setting the current system's counting state to an uncountable state, the method further includes:

[0023] Real-time monitoring of the new drill rod mask and new drill rig body mask obtained after the trained instance segmentation model performs instance segmentation on the drill rig body and drill rod in the next frame image;

[0024] Calculate the pixel areas of the new drilling rig body mask and the new drill rod mask respectively to obtain the area of ​​the new drill rod mask and the area of ​​the new drilling rig body mask;

[0025] Calculate the ratio of the mask area of ​​the new drilling rig body to the mask area of ​​the new drill pipe to obtain the new mask area ratio;

[0026] Determine whether the area ratio of the new mask is less than the third area ratio threshold or greater than the fourth area ratio threshold;

[0027] If the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, then the current system's counting state is changed from uncountable to countable.

[0028] Wherein, the fourth area ratio threshold is greater than the third area ratio threshold, the third area ratio threshold is less than the second area ratio threshold, and the fourth area ratio threshold is greater than the first area ratio threshold.

[0029] Optionally, if the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, then setting the current system's counting state from an uncountable state to a countable state includes:

[0030] If the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, then determine whether the target mask area ratios corresponding to all other frames of images preceding the next frame are less than the third area ratio threshold or greater than the fourth area ratio threshold.

[0031] If the target mask area ratio of all other frames preceding the next frame is less than the third area ratio threshold or greater than the fourth area ratio threshold, then the current system's counting state is changed from uncountable to countable.

[0032] Optionally, the instance segmentation-based method for counting drill pipes in coal mines further includes:

[0033] Real drilling video data from multiple coal mines in different scenarios were collected, and the drilling rig and drill rod in each frame of the real drilling video data were labeled to construct a dataset; the different scenarios include different angles and different lighting conditions;

[0034] The dataset is input into the YOLOv11n-seg model based on the self-attention mechanism for model training, resulting in the trained instance segmentation model.

[0035] Optionally, the model structure of the YOLOv11n-seg model based on the self-attention mechanism consists of a backbone network, a neck network, and a segmentation detection head;

[0036] The backbone network includes residual blocks based on partial dynamic convolution and attention residual modules based on dynamic statistics, which are used to extract multi-scale features from each frame of the input image; the attention residual modules based on dynamic statistics include self-attention mechanism components based on distribution statistics and learnable nonlinear mapping functions.

[0037] The neck network includes the residual block based on partial dynamic convolution and the adaptive multi-scale feature aggregation mechanism module, which is used to perform multi-level fusion of the multi-scale features output by the backbone network.

[0038] The segmentation detection head is used to perform category prediction and bounding box regression on the feature fusion results output by the neck network at multiple different scales.

[0039] Secondly, this application discloses a coal mine underground drill pipe counting device based on instance segmentation, comprising:

[0040] The video acquisition module is used to acquire the operation video of the target coal mine underground drilling rig and obtain the drilling rig operation video sequence;

[0041] The instance segmentation module is used to sequentially input each frame of the drilling rig operation video sequence into the trained instance segmentation model to segment the drilling rig body and drill rod within the current input frame image, obtaining the drill rod mask contour and the drilling rig body mask contour. The trained instance segmentation model is a model obtained by training an initial instance segmentation model based on an attention mechanism using a training set. The training set includes real drilling video data from multiple coal mine underground scenes.

[0042] The pixel area acquisition module is used to acquire the pixel area of ​​the drill rod mask contour and the drill rig body mask contour corresponding to the current input frame image, so as to obtain the current drill rod mask area and the current drill rig body mask area.

[0043] The calculation module is used to calculate the ratio of the current drilling rig body mask area to the current drill rod mask area, and obtain the mask area ratio.

[0044] The analysis module is used to analyze the changing trends of multiple mask area ratios corresponding to multiple frames of images within a preset time period before the current moment, and to obtain the ratio changing trends.

[0045] The first counting module is used to determine that the current drill pipe state is the drilling state if the ratio changes in a continuously increasing trend, and to determine whether the current mask area ratio is greater than the first area ratio threshold. If so, the counting operation of incrementing the drilling count by 1 is performed.

[0046] The second counting module is used to determine that the current drill pipe state is a drill retraction state if the ratio changes in a continuously decreasing trend, and to determine whether the current mask area ratio is less than the second area ratio threshold. If so, the counting operation of incrementing the drill retraction count by 1 is performed; the second area ratio threshold is less than the first area ratio threshold.

[0047] Thirdly, this application discloses an electronic device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, it implements the aforementioned method for counting drill pipes in coal mines based on instance segmentation.

[0048] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for counting drill pipes in coal mines based on instance segmentation.

[0049] As can be seen, this application first acquires the operation video of the target coal mine underground drilling rig to obtain a drilling rig operation video sequence. Then, each frame image in the drilling rig operation video sequence is sequentially input into an instance segmentation model based on an attention mechanism to perform instance segmentation on the drilling rig body and drill rod in the current input frame image, obtaining the drill rod mask contour and the drilling rig body mask contour. The pixel areas of the drill rod mask contour and the drilling rig body mask contour corresponding to the current input frame image are obtained to obtain the current drill rod mask area and the current drilling rig body mask area. Then, the ratio of the current drilling rig body mask area to the current drill rod mask area is calculated to obtain the mask area ratio. The changing trends of multiple mask area ratios corresponding to multiple frames within a preset time period before the current moment are analyzed to obtain the ratio changing trend. If the ratio changing trend is a continuously increasing trend, the current drill rod state is determined to be the drilling state, and it is determined whether the current mask area ratio is greater than a first area ratio threshold. If so, the drilling count is incremented by 1. If the ratio changing trend is a continuously decreasing trend, the current drill rod state is determined to be the retraction state, and it is determined whether the current mask area ratio is less than a second area ratio threshold. If so, the retraction count is incremented by 1. The second area ratio threshold is less than the first area ratio threshold.

[0050] This application utilizes an attention-based instance segmentation model to perform pixel-level precision segmentation of the drilling rig body and drill rods in drilling operation videos, significantly improving the ability to identify and segment drill rod targets in complex mine images. Furthermore, this application uses the changing trend of the ratio of the drilling rig body mask area to the drill rod mask area, combined with an area ratio threshold, to identify the drill rod's entry and exit status and count its entry and exit. This demonstrates that this application considers not only the drill rod's proportion in the image but also its spatial relationship with the drilling rig body. As a result, even if the drilling rig's angle changes, the ratio between the masks remains stable, thus adapting to different angle changes and ensuring that the relationship between the drill rod and the drilling rig body is unaffected by changes in viewing angle. Therefore, it can still provide accurate drill rod counting results even when facing changes in drilling rig angle, thereby improving the accuracy of drill rod counting and avoiding misidentification and omission of entry and exit. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1This application discloses a flowchart of a method for counting drill pipes in coal mines based on instance segmentation.

[0053] Figure 2 This is a schematic diagram of a specific instance segmentation model structure disclosed in this application;

[0054] Figure 3 This is a schematic diagram of a specific C3k2-PDCB module structure disclosed in this application;

[0055] Figure 4 This is a schematic diagram of a specific PDCB module structure disclosed in this application;

[0056] Figure 5 This is a schematic diagram of a specific Fourier weight separation process disclosed in this application;

[0057] Figure 6 This is a flowchart of a specific convolution operation disclosed in this application;

[0058] Figure 7 This is a schematic diagram of a specific C2PSA-SDAB module structure disclosed in this application;

[0059] Figure 8 This is a schematic diagram of a specific SDAB module structure disclosed in this application;

[0060] Figure 9 This is a schematic diagram of a specific TSSA component structure disclosed in this application;

[0061] Figure 10 This is a schematic diagram of a specific AMFAM module structure disclosed in this application;

[0062] Figure 11 This is a schematic diagram of a specific instance segmentation result disclosed in this application;

[0063] Figure 12 This application discloses a specific method for counting drill pipes in coal mines based on instance segmentation.

[0064] Figure 13 This is a schematic diagram of a specific coal mine underground drill pipe counting result disclosed in this application;

[0065] Figure 14 This is a schematic diagram of a coal mine underground drill pipe counting device based on instance segmentation disclosed in this application;

[0066] Figure 15 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0068] This application discloses a method for counting drill pipes in coal mines based on instance segmentation. See [link to relevant documentation]. Figure 1 As shown, the method includes:

[0069] Step S11: Obtain the operation video of the drilling rig in the target coal mine to obtain the drilling rig operation video sequence.

[0070] In this embodiment, the video of the target coal mine drilling rig to be counted is first acquired by the camera and other equipment, and then the acquired video is extracted frame by frame to obtain the drilling rig operation video sequence.

[0071] Step S12: Input each frame of the drilling rig operation video sequence into the trained instance segmentation model in sequence to perform instance segmentation of the drilling rig body and drill rod in the current input frame image, and obtain the drill rod mask contour and the drilling rig body mask contour; the trained instance segmentation model is the model obtained by training the initial instance segmentation model based on the attention mechanism using the training set; the training set includes real drilling video data of multiple coal mine underground different scenarios.

[0072] In this embodiment, after obtaining the operation video of the target coal mine underground drilling rig, each frame of the above-mentioned drilling rig operation video sequence is then sequentially input into the trained instance segmentation model obtained by training the initial instance segmentation model based on the attention mechanism using the training set, so as to sequentially perform mask instance segmentation on the drilling rig body and drill rod in each of the currently input frame images, and obtain the drill rod mask contour and drilling rig body mask contour corresponding to each frame image.

[0073] The initial instance segmentation model includes, but is not limited to, models such as YOLOv11n-seg, Mask R-CNN, YOLOv8-seg, SegFormer, and FastSAM, used to extract the mask of the drill pipe and the drill rig body; the attention mechanism includes, but is not limited to, attention mechanisms based on GhostConv structure, ECA (Efficient Channel Attention), CBAM (Convolutional Block Attention Module), SPP (Spatial Pyramid Pooling) / SPPF (Spatial Pyramid Pooling – Fast) fusion module, etc.

[0074] Specifically, the process of acquiring the trained instance segmentation model for real-time instance segmentation includes: collecting real drilling video data from multiple underground coal mines in different scenarios, and labeling the drilling rig body and drill rod in each frame of the real drilling video data to construct a dataset; the different scenarios include different angles and different lighting conditions; the dataset is input into a YOLOv11n-seg model based on a self-attention mechanism for model training to obtain the trained instance segmentation model. In this embodiment, real drilling video data from multiple underground coal mines in different scenarios (covering different angles and different lighting conditions) can be collected first, then the collected real drilling video data can be cropped frame by frame to generate image samples, and then the cropped image samples can be finely labeled to construct a dataset for model training; then, the labeled dataset can be input into a YOLOv11n-seg (You Only LookOncev11n-seg, an instance segmentation model) model based on a self-attention mechanism for model training to obtain an instance segmentation model capable of real-time instance segmentation in various scenarios.

[0075] Furthermore, to make the model more robust and flexible, new training data can be collected at preset intervals to continuously optimize and update the model, such as fine-tuning and optimizing the model parameters.

[0076] It should be noted that this application proposes three structural improvements to the traditional YOLOv11n-seg network architecture, resulting in an optimized YOLOv11n-seg model, which significantly improves the model's ability to identify and segment drill pipe targets in complex mine images. Specifically, the optimized YOLOv11n-seg model adopts a self-attention mechanism, and its structure consists of a backbone network, a neck network, and a segmentation detection head (Seg). The backbone network includes a Partial Dynamic Convolutional Residual Block (PDCB) and a Tatistically-Aware Dynamic Attention Residual Block (SDAB). The SDAB includes a Token Statistics Self-Attention (TSSA) mechanism and a learnable nonlinear mapping function (DynamicTanh, DyT). The neck network specifically includes the PDCB and an Adaptive Multi-scale Feature Aggregation Module (AMFAM) for multi-level fusion of the multi-scale features output from the backbone network. The segmentation detection head performs category prediction and bounding box regression on the feature fusion results output from the neck network at multiple different scales.

[0077] It should be noted that PDCB is used to improve the expressive diversity of feature extraction, replacing the standard Bottleneck structure in the traditional YOLOv11n-seg model in the backbone network; SDAB replaces the PSA (Pyramid Squeeze Attention) module in the traditional YOLOv11n-seg model, enabling lightweight global modeling; AMFAM can adaptively fuse multi-scale features, replacing the Concat module in the traditional YOLOv11n-seg model to enhance the flexibility of feature aggregation.

[0078] In addition, to adapt to instance segmentation tasks, a mask prediction branch can be added to the YOLOv11n-seg model. By fusing with multi-scale features shared by the backbone and neck networks, high-resolution instance masks can be generated.

[0079] In one specific implementation, see Figure 2 As shown, the optimized YOLOv11n-seg model consists of three parts: a backbone network, a neck network, and a segmentation detection head (Seg). It also includes three improved modules: the C3k2-PDCB module, the C2PSA-SDAB module, and the AMFAM module. The backbone network extracts features from the input image at multiple scales, and the extracted features are then fed into the neck network for multi-level fusion. Finally, the fused features are input into the segmentation detection head to perform class prediction and bounding box regression at multiple scales (e.g., P3 / 8, P4 / 16, P5 / 32).

[0080] Specifically, Figure 2 The structure of the C3k2 (an improved convolutional neural network architecture) - PDCB module used for feature extraction is shown in [link to documentation]. Figure 3 As shown, whether the parameter C3k in the C3k2-PDCB module is True or false will affect the internal structure of the module. Figure 3 Images (a) and (b) show the internal structure of the C3k2-PDCB module with two different C3k parameters, respectively. Figure 3 (c) illustrates the specific internal structure of the C3k module in (a). The input to the C3k2-PDCB module is the intermediate feature representation obtained after processing a single frame image from the drilling rig video sequence through the first few layers of the convolutional network. When the input single frame image enters the C3k2-PDCB module, it first passes through a combination layer of convolution + normalization + SiLU activation function and is equally split into main path and bypass features, respectively. Then, the main path features are processed non-linearly through two PDCB modules or the C3k module (depending on whether the parameter of C3k in the C3k2-PDCB module is True or false). The output of each level is collected for feature fusion. Finally, all intermediate results and bypass information are concatenated along the channel dimension and integrated into the final output features through another combination layer of convolution + normalization + SiLU activation function.

[0081] in, Figure 3 See the PDCB module in the C3k2-PDCB module. Figure 4As shown, each part of the dynamic convolutional (PDConv) operation requires batch normalization (BN) and SiLU (Sigmoid-Weighted Linear Unit) activation function after the initial operation. Residual connections further enhance the information flow of the model in deep networks. It should be noted that the original Bottleneck structure of the traditional YOLOv11n-seg model uses standard convolution operations with fixed parameters. While this provides stable feature representations, the static nature of the convolution method limits its adaptability to changes in input content. This makes it difficult to effectively extract targeted feature information in the complex and ever-changing environment of underground mines, especially when target objects such as drilling rigs and drill pipes exhibit significant differences under different lighting and background conditions; fixed convolution weights often fail to capture these changes adequately. This application replaces static convolution with partial dynamic convolution (PDConv), enhancing the model's feature extraction capability through adaptive weight adjustment.

[0082] Furthermore, while traditional dynamic convolution methods can dynamically adjust weights, they often suffer from functional redundancy among multiple parallel convolution weights. The generated n weights exhibit high functional similarity, preventing them from fully utilizing their individual strengths and thus limiting the model's effective extraction of input features. Moreover, traditional methods typically require introducing n times the number of parameters to increase the number of parallel convolution weights, which not only increases model complexity but also significantly increases computational overhead. This efficiency issue is particularly prominent for the deployment of edge computing equipment in underground mines.

[0083] To address the aforementioned issues, this application proposes an improved dynamic convolution method. The core idea of ​​this method is to perform non-overlapping frequency grouping of a fixed parameter budget in the Fourier domain. This allows for the construction of multiple convolution kernels with mutually exclusive frequency characteristics at the frequency domain level, thereby diversifying the frequency responses of the weights. Specifically, the fixed parameter budget refers to dividing the parameters originally used for a single convolution kernel into multiple non-overlapping subsets (different frequency bands) in the frequency domain without increasing the total number of model parameters. Each subset generates a convolution kernel with different frequency characteristics. This approach achieves diversity among convolution kernels without increasing the total parameter cost, thus improving feature extraction capabilities. For details, see [link to details]. Figure 5 As shown, Figure 5The process of separating the Fourier transform weights is illustrated. First, the parameters (i.e., the weight parameters of the convolution kernel) are divided into non-overlapping groups, then sorted from low frequency (center) to high frequency (edge). This example shows the case of n=2 groups, where the four corner parts in the left image are group 1, and the central part (diamond-shaped region) is group 0. The right image shows how to obtain the convolution weights from the learnable parameter group 0. First, the learnable parameters are transformed with specific Fourier indices using the inverse discrete Fourier transform (iDFT) (setting all other Fourier indices to zero). Then, the convolution weights are obtained by pruning the k×k result of the iDFT into small blocks and rearranging them into a weight tensor of size k×k×Cin×Cout. The specific implementation process is as follows:

[0084] Let a standard two-dimensional convolution kernel be... Where Cin and Cout are the number of input and output channels, respectively, and k is the kernel size. First, the convolution kernel parameters are treated as Fourier coefficients and subjected to a two-dimensional discrete Fourier transform (DFT) to convert them from the spatial domain to the frequency domain. The specific calculation formula is as follows:

[0085] ;

[0086] In the formula, k is the kernel size, and i is the input channel index, with a value range of... j is the output channel index, with a value range of... ; The values ​​are the pixel coordinates in the spatial domain (i.e., the pixel positions in the two-dimensional convolution kernel), and their range is... ; The connection weight between input channel i and output channel j is represented by a k×k two-dimensional discrete function. These are the Fourier coefficients at the frequency index (u,v); These are frequency domain coordinates, representing the frequency index.

[0087] Using the above formula, the convolution kernel can be... Convert to frequency domain representation ,in The index represents the spatial location within the convolution kernel, ranging from 0 to k-1 (i.e., the kernel size is k×k). These are the coordinates in the corresponding frequency domain. Each frequency component... Both are achieved by adjusting spatial weights. With complex exponential basis functions The weighted summation is essentially a process of projecting the spatial features of the convolution kernel onto different frequency components, and the final result is a complex matrix.

[0088] It is understandable that the Fourier coefficients This describes the intensity of different frequency components in the convolution kernel. If all convolution kernels sample from similar frequency regions, their responses will tend to overlap. To avoid overlap, the frequency domain can be optimized. These coefficients (i.e., Fourier coefficients in the frequency domain) are sorted from low to high frequency according to the Euclidean norm of the frequency index, and then divided into n sub-band regions (i.e., frequency bands). Each sub-band region contains only Fourier components within a specific frequency range. For each sub-band region, the Fourier components (i.e., frequency components) within that sub-band region (i.e., frequency band) are retained, and other frequency components are set to zero. Then, the parameters of each frequency band are processed using the inverse discrete Fourier transform (iDFT). The frequency domain representation is transformed back to the spatial domain to obtain a spatial convolution kernel with a unique frequency response. The specific calculation formula is as follows:

[0089] ;

[0090] In the formula, k is the kernel size, and i is the input channel index, with a value range of... j is the output channel index, with a value range of... ; The values ​​are the pixel coordinates in the spatial domain (i.e., the pixel positions in the two-dimensional convolution kernel), and their range is... ; These are frequency domain coordinates, representing frequency indices; These are the frequency components in the frequency domain (i.e., the Fourier parameters that are assigned to a certain frequency band).

[0091] The frequency domain representation of the convolution kernel can be obtained using the above formula. Perform a two-dimensional discrete Fourier inverse transform to recover the convolution kernel weights in the spatial domain. .in, Indicates the position index in the frequency domain. This represents the pixel location in the spatial domain, where k is the size of the convolution kernel. The weights for each spatial location are... It is a linear combination of all frequency components, where each frequency component... Multiply by the corresponding complex exponential basis function Then sum over all frequencies and divide by . This is done to normalize the data. The above process essentially restores the frequency domain features one by one back to the original spatial weight distribution, allowing the convolution kernel to be reused for spatial feature extraction from the image.

[0092] Finally, by cropping and recombining the results of the inverse Fourier transform, they are converted into standard convolution kernel shapes, thereby constructing a set of convolution kernels that are distinct in the frequency domain and have different response characteristics in the spatial domain. This ensures that the convolutional layers can extract information at different frequencies. Through the above steps, each weight can focus on extracting information within a specific frequency band, achieving "responsibility division" style feature modeling. For example, low-frequency kernels are better at extracting structural and contour information, helping to suppress background noise; while high-frequency kernels are more sensitive to details such as edges and textures, which is beneficial for accurate modeling of segmentation boundaries.

[0093] By constructing the convolution kernel as described above, not only is the diversity of feature extraction improved, but it can also effectively adapt to practical application scenarios in underground mine environments such as uneven lighting, complex backgrounds, and low contrast between the target and the background.

[0094] In addition, to improve computational efficiency and meet the actual deployment requirements of the model, a partial channel convolution method can be used. For details, see [link to documentation]. Figure 6 As shown, convolution operations can be applied to a subset of channels in the input feature map, while the remaining channels are directly passed. This reduces computational cost and memory access overhead, and significantly lowers the overall complexity of convolution computation while maintaining the integrity of feature information. For example, when the original input channel count is... The selection ratio is The former If multiple channels participate in the computation, the computational complexity is reduced from that of traditional convolution:

[0095] ;

[0096] Downgraded to:

[0097] .

[0098] The aforementioned lightweight strategy is particularly suitable for resource-constrained edge devices in mines, and can improve operational efficiency while ensuring mask quality.

[0099] For the structure of the C2PSA (Cross-Stage Progressive Spatial Attention) - SDAB module in the YOLOv11n-seg model, see [link to relevant documentation]. Figure 7As shown, this module includes a distribution statistics-driven self-attention mechanism component (TSSA) and a learnable nonlinear mapping function (DyT). Specifically, after a single frame image from a drilling rig video sequence is processed through the first few layers of the model, the resulting intermediate feature representation is used as input to the C2PSA-SDAB module. Upon receiving this intermediate feature representation, the module first splits the channel count into two branches using a convolution. One branch retains the original features, while the other undergoes deep processing through multiple SDAB sub-modules. In each sub-module, the input features first pass through a dynamically adjustable nonlinear mapping function DyT; subsequently, the features are input into the distribution statistics-driven self-attention mechanism component (TSSA); then, a second DyT and a feed-forward network (FFN) extract higher-order semantic information. Figure 7 The structure of the SDAB module can be found in [reference]. Figure 8 As shown.

[0100] It should be noted that the C2PSA-SDAB module in this application, compared to the traditional C2PSA module, adds a self-attention mechanism component based on distribution statistics (i.e., TSSA) and a learnable nonlinear mapping function (i.e., DyT). It is understandable that traditional multi-head self-attention obtains attention weights by calculating the dot product similarity between input tokens, with the specific calculation formula as follows:

[0101] ;

[0102] In the formula, Q, K, and V are the query matrix, key matrix, and value matrix, respectively. The feature dimension is [value]. However, this attention weight calculation method requires calculating the pairwise dot product of all tokens, resulting in a time and space complexity of O(n). Under high-resolution image input (such as in a mine scene), it can easily become a computational bottleneck. To reduce computational overhead, this application adds a self-attention mechanism component based on distribution statistics (TSSA) to replace the explicit similarity calculation process.

[0103] For details, see Figure 9 As shown, Figure 9 This illustrates the specific processing flow of the self-attention mechanism component based on distribution statistics (TSSA): for the input token representation... First, the token is obtained by linearly projecting it into a query Q vector, a key K vector, and a value V vector. Then, the energy of each key vector is calculated. That is, the squared L2 norm, and the global weight distribution is obtained through Softmax (normalized exponential function). The weight distribution This is used to perform a weighted summation of all value vectors V to obtain the global context vector. Finally, the output of each token (i.e., the updated token) is obtained and integrated into a new output token representation. .in Figure 9 In Indicates the first The token input from the layer, Let Q / K / V be the transpose projection matrix of the k-th head, Diagonal Attention (head k) denotes the diagonal form of the attention mechanism in the k-th attention head, and projected tokens represent the tokens generated by linear transformation. The mapping is Q, K, V required for the attention mechanism. Token statistics Diagonal Attention means that the attention weights are determined using the token itself. Updated token means that the final output representation of each token is represented.

[0104] By analyzing the energy distribution of input features across different dimensions using a distribution statistics-driven self-attention mechanism (TSSA), interference information in low-contribution directions can be automatically suppressed, thus achieving efficient feature compression. The core idea is to avoid explicitly calculating the inter-token similarity matrix, but instead... As a set of column vectors, it is assumed that it can be divided into k potential subspaces, each subspace k consisting of a set of orthogonal basis matrices. The description states that, at this point, the probability of each token belonging to each subspace can be calculated. Based on its probability distribution, its second-order statistic (i.e., the characteristic covariance matrix) in each subspace is estimated, and the specific calculation formula is as follows:

[0105] ;

[0106] To estimate the feature covariance matrix on each subspace (or attention head) We can first view the input token feature representation Z as a matrix composed of column vectors, where each column represents a feature of a token. Then, we can determine the probability distribution of each token belonging to the k-th subspace. Construct a diagonal matrix This is used to weight the features of each token; then, matrix multiplication is performed. This involves aggregating the second-order feature information of all tokens; finally, to normalize the sample size, the result can be divided by the total weight of all tokens in that subspace. This results in a weighted, non-centralized covariance matrix. .

[0107] Furthermore, using these covariance matrices Estimate the principal directions of each subspace and construct a residual compression operation:

[0108] ;

[0109] In the formula, This represents the input feature vector of the j-th token. This represents the updated characteristics (compressed output) of the j-th token. This represents the probability (membership weight) that the j-th token belongs to the k-th subspace. Let the eigenorthogonal basis matrix of the k-th subspace be denoted as . This represents the main diagonal weight matrix of the k-th subspace. Indicates features Mapped to the k-th subspace coordinate system, This indicates that the compressed low-rank component is projected back into the original space. This represents the contribution of merging all K subspaces to the token.

[0110] The above formula represents a residual feature compression mechanism used to suppress components in the input features that are correlated with redundant directions. First, for each token feature... Estimate its probability distribution in multiple subspaces. Then, in each subspace, through orthogonal basis matrices... Project it and use the main diagonal matrix Attenuation is applied to the low-variance direction to form a compression component; finally, after the compression results of all subspaces are weighted and fused, they are subtracted from the original features to obtain the updated token representation. The above operation can be viewed as a data-driven low-rank compression process performed within the original feature space, preserving directions with high information density and suppressing redundant and noisy components, thereby completing global dependency modeling in linear time complexity. This approach avoids the high computational complexity of calculating the similarity between tokens in traditional self-attention mechanisms and significantly improves inference efficiency for large-resolution images, making it particularly suitable for large-size inputs in mining scenarios.

[0111] For the learnable nonlinear mapping function (DyT) in the C2PSA-SDAB module, the input of this function is the output of the previous module, and its specific expression is as follows:

[0112] ;

[0113] In the formula, where, This is a learnable scaling factor used to control the input amplitude; and All parameters are channel-level learnable linear transformation parameters. This function (i.e., DyT) uses a parameterized hyperbolic tangent form to dynamically compress the activation range of different channels, while adding learnable scaling and offset parameters. This allows the model to maintain training stability and activation distribution control without relying on normalization layers. This approach not only replaces the redundant normalization operations in traditional networks, but also simplifies the training process and improves inference efficiency.

[0114] Compared to the C2PSA module in the traditional YOLOv11n-seg model, this application combines the C2PSA module with the SDAB module. Since the complexity of the attention operation based on statistical properties is reduced from a quadratic relationship to linear, the inference speed for large-scale images can be significantly accelerated. Furthermore, the learnable nonlinear mapping function (DyT) can suppress outliers in features through adaptive scaling, thus improving the model's adaptability to different input variations. This significantly improves the robustness and accuracy of object detection in practical applications in complex mining environments, especially demonstrating superior feature separation and localization capabilities in typical challenging scenarios such as uneven lighting and background interference with similar structures.

[0115] Furthermore, considering that the neck network of the traditional YOLOv11n-seg model often uses the Conca module to simply concatenate feature maps from different layers along the channel dimension, this static fusion strategy has limitations in multi-scale feature representation. It assigns equal weights to all input features without considering the differences in importance of features at different scales to the current detection task. To address these issues, this application replaces the Conca module with an adaptive multi-scale feature aggregation mechanism module (AMFAM). This module aims to dynamically adjust the contribution of features at different scales through intelligent feature weighting and fusion strategies, thereby improving the model's detection accuracy for multi-scale targets in complex mining environments.

[0116] For details, see Figure 10 As shown, Figure 10 The structure of the Adaptive Multi-Scale Feature Aggregation Mechanism (AMFAM) module is shown. The processing flow is from bottom to top, with the bottom two inputs being two spatially aligned feature maps of different scales. Specifically, this module uses dynamically learned weight coefficients. This is used to adjust the contribution of features at different scales. Specifically, global context analysis can be performed on the input multi-scale features, and corresponding weight distributions can be generated based on the characteristics of the current detection task. The input to this module is specifically the intermediate feature representation maps obtained after processing single-frame images from different semantic levels in the model's backbone or neck network through the first few layers of the model, denoted as:

[0117] ;

[0118] To ensure the effectiveness of feature fusion, a 1×1 convolution can be used to map both features to the same number of channels D:

[0119] ;

[0120] Aligned features First, global information in the spatial dimension is extracted. Then, global average pooling (GAP) is used to combine each feature map. Compressed into a global descriptor :

[0121] ;

[0122] Next, the two global descriptors are summed to obtain the overall semantic representation G:

[0123] ;

[0124] Furthermore, a lightweight multilayer perceptron (MLP) is used to calculate the weight coefficients of features at each scale. The calculation formula for MLP networks is:

[0125] ;

[0126] It should be noted that a Multilayer Perceptron (MLP) contains two fully connected layers. The first layer is used for dimensionality reduction to improve computational efficiency, and the second layer is used to restore the feature dimension to its original value. Finally, normalization is performed using the Softmax function. The weights... These correspond to the importance of each channel dimension of the two input feature maps. After obtaining the weight coefficients... Then, the AMFAM module performs element-wise multiplication on feature maps of different scales. Weighted fusion is performed to obtain the final fused feature map. :

[0127] ;

[0128] In the formula, This represents element-wise multiplication. Through this weighted fusion operation, the AMFAM module can effectively highlight key features and suppress redundant or irrelevant information, thereby achieving effective fusion of multi-scale features.

[0129] This application utilizes the AMFAM module in the YOLOv11n-seg model, enabling the model to adaptively adjust the importance of features at different scales. Particularly in multi-scale target detection, it flexibly handles the contribution of features at each scale and effectively improves feature fusion between small and large targets. This avoids the limitations of traditional models that apply equal weighting to features, enhances the complementarity of cross-scale information, and makes it suitable for complex mining environments. Furthermore, the AMFAM module effectively reduces redundant information, improves computational efficiency, and enhances the ability to identify key targets such as drill pipes and the drill rig body, especially in environments with insufficient lighting, occlusion, or complex backgrounds, thereby improving the accuracy and robustness of target detection.

[0130] Step S13: Obtain the pixel area of ​​the drill rod mask contour and the drill rig body mask contour corresponding to the current input frame image, and obtain the current drill rod mask area and the current drill rig body mask area.

[0131] In this embodiment, after segmenting the drill rig body and drill rod within the current input frame image to obtain the drill rod mask contour and the drill rig body mask contour, the pixel areas of the aforementioned drill rod mask contour and the aforementioned drill rig body mask contour corresponding to the current input frame image are calculated respectively to obtain the corresponding current drill rod mask area (denoted as ). ) and the current drilling rig main body mask area (denoted as That is, for each frame of input image, the mask contours of the "drill rig body" and "drill rod" are obtained through the segmentation model, and then their pixel areas are calculated respectively.

[0132] See Figure 11 As shown in the figure, there are two categories, dark and light, that play a crucial role in the counting. The "Drill_body" outline in the light-colored area is the "drill rig body" identified by the optimized YOLO11n-seg model in this application. The number 72 refers to the "drill rig body mask area," which is the area of ​​the mask at this point. =72. The dark area, the "Drill_pipe" outline, represents the "drill pipe" identified by the model. The number 35 refers to the "drill pipe mask area," which is the value at this point. =35.

[0133] Step S14: Calculate the ratio of the current drilling rig body mask area to the current drill rod mask area to obtain the mask area ratio, and analyze the changing trend of multiple mask area ratios corresponding to multiple frames of images within a preset time period before the current moment to obtain the ratio changing trend.

[0134] In this embodiment, the current drill pipe mask area is obtained ( ) and the current drilling rig body mask area ( After that, the current main body mask area of ​​the drilling rig can be further calculated. ) and the aforementioned current drill pipe mask area ( The ratio of the mask area to the area of ​​the mask is used to obtain the mask area ratio R. The specific calculation formula is as follows:

[0135] ;

[0136] The calculated mask area ratio R can help determine whether the drill pipe is currently in the drilling or retraction phase. Specifically, the changing trends of multiple mask area ratios R corresponding to multiple frames within a preset time period before the current moment can be analyzed to obtain the ratio change trend.

[0137] Step S15: If the ratio changes in a continuously increasing trend, the current drill pipe state is determined to be the drilling state, and it is determined whether the current mask area ratio is greater than the first area ratio threshold. If so, the drilling count is incremented by 1.

[0138] In this embodiment, the mask area ratio R over a period of time can be recorded in a continuous drilling operation video sequence, and then the trend of this ratio can be analyzed through linear regression. If the area ratio R is detected to be continuously increasing, it indicates that the area of ​​the drill pipe is decreasing, and at this time it can be determined that the current drill pipe state is in the drilling state; then, it is determined whether the current mask area ratio R is greater than a first area ratio threshold (e.g., 7.5). If the mask area ratio R is greater than 7.5, the corresponding drilling count operation is performed.

[0139] Specifically, determining whether the current mask area ratio is greater than a first area ratio threshold, and then incrementing the drill count by 1 if so, can include: determining whether the current mask area ratio is greater than the first area ratio threshold; if the current mask area ratio is greater than the first area ratio threshold, determining whether the current system's counting state is in a countable state; if the current system's counting state is in a countable state, then incrementing the drill count by 1 to obtain the drill count statistics; correspondingly, after incrementing the drill count by 1, it can also include: setting the current system's counting state to an uncountable state. In this embodiment, it can first determine whether the current mask area ratio R is greater than a first area ratio threshold (e.g., 7.5); if it is greater, then determine whether the current system's counting state is in a countable state; if it is not in a countable state, then incrementing the drill count by 1, and after the counting operation is completed, setting the current system's counting state to an uncountable state.

[0140] Step S16: If the ratio changes in a continuously decreasing trend, the current drill pipe state is determined to be a drill retraction state, and it is determined whether the current mask area ratio is less than the second area ratio threshold. If so, the drill retraction count is incremented by 1. The second area ratio threshold is less than the first area ratio threshold.

[0141] In this embodiment, if the area ratio R is detected to be continuously decreasing, it indicates that the area of ​​the drill pipe is increasing, and the current state of the drill pipe can be determined to be in the drill retraction state. Then, it is determined whether the current mask area ratio R is less than the second area ratio threshold (e.g., 1.8). If the mask area ratio R is less than 1.8, the corresponding drill retraction counting operation is performed. If there is no obvious trend change, the previous state can be maintained.

[0142] It should be noted that the second area ratio threshold is less than the first area ratio threshold. These two thresholds can be obtained based on experience or experiments, and can be continuously optimized and adjusted according to actual application needs.

[0143] Specifically, determining whether the current mask area ratio is less than the second area ratio threshold, and then performing a count operation of incrementing the drill-out count by 1 if so, may include: determining whether the current mask area ratio is less than the second area ratio threshold; if the current mask area ratio is less than the second area ratio threshold, determining whether the current system's counting state is in a countable state; if the current system's counting state is in a countable state, performing a count operation of incrementing the drill-out count by 1 to obtain the drill-out count statistics; correspondingly, after performing the count operation of incrementing the drill-out count by 1, it may also include: setting the current system's counting state to a non-countable state. In this embodiment, it may first determine whether the current mask area ratio R is less than the second area ratio threshold (e.g., 1.8); if it is less, it may determine whether the current system's counting state is in a countable state; if it is not in a countable state, it may perform a count operation of incrementing the drill-out count by 1, and after the counting operation is completed, set the current system's counting state to a non-countable state.

[0144] Furthermore, after setting the current system's counting state to an uncountable state, the process may further include: real-time monitoring of the new drill rod mask and new drill rig body mask obtained after the trained instance segmentation model performs instance segmentation on the drill rig body and drill rod in the next frame image; calculating the pixel areas of the new drill rig body mask and the new drill rod mask respectively to obtain the new drill rod mask area and the new drill rig body mask area; calculating the ratio of the new drill rig body mask area to the new drill rod mask area to obtain the new mask area ratio; determining whether the new mask area ratio is less than a third area ratio threshold or greater than a fourth area ratio threshold; if the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, then setting the current system's counting state from an uncountable state to a countable state; wherein, the fourth area ratio threshold is greater than the third area ratio threshold, and the third area ratio threshold is less than the second area ratio threshold, and the fourth area ratio threshold is greater than the first area ratio threshold. It should be noted that, in order to avoid duplicate counting or miscounting caused by short-term fluctuations such as jitter or partial occlusion, this application adds a state locking and counting mechanism, see [link to relevant documentation]. Figure 12As shown, the mechanism includes a "counting section" and a "counting state recovery section." Specifically, when counting begins, the "counting section" determines whether the current drill pipe movement trend is "drilling forward" or "drilling backward" based on the aforementioned trend judgment and state recognition strategy. If the current state is "drilling backward," and the mask area ratio R < 1.8 and is in a "countable" state, then a drilling backward counting operation is performed, i.e., the drilling backward count is incremented by 1. If the current state is "drilling forward," and the mask area ratio R > 7.5 and is in a "countable" state, then a "drilling forward counting" operation is performed, i.e., the drilling forward count is incremented by 1. After performing the count increment operation, the system state is immediately changed from the "countable" state to the "uncountable" state, and the historical data of the pipe change sequence used to determine whether this count was for drilling forward or drilling backward is cleared. Next, the second part, the "counting state recovery section," can determine whether the current drill pipe movement trend is "drilling forward" or "drilling backward" after completing one count, and then restore the system from the "uncountable" state to the "countable" state based on the different movement trends of the drill pipe. It should be noted that the recovery conditions differ in the two scenarios. For example, when the drill pipe movement trend is "retracting," the recovery condition is a mask area ratio R > 8.5; when the drill pipe movement trend is "advancing," the recovery condition is a mask area ratio R < 1.5. Therefore, only when the area ratio falls within the preset range is re-entry into the "countable" state allowed. This ensures that each state transition is recorded only once. Afterward, the system can re-enter the "counting section" and complete the cycle from "counting section" to "counting state recovery section" and back to "counting section."

[0145] In one specific implementation, if the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, then the current system's counting state is changed from an uncountable state to a countable state. Specifically, this may include: if the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, then determining whether the target mask area ratios corresponding to all other frames within a preset number of frames before the next frame are all less than the third area ratio threshold or greater than the fourth area ratio threshold; if the target mask area ratios corresponding to all other frames within a preset number of frames before the next frame are all less than the third area ratio threshold or greater than the fourth area ratio threshold, then the current system's counting state is changed from an uncountable state to a countable state. That is, when switching counting states, the decision to switch states is based on the mask area ratio R of all other frames within a preset number of frames before the next frame. Through the above-described drill-back counting statistics method, including the state locking and releasing mechanism with dual threshold control, misjudgment and duplicate counting during state switching can be effectively prevented. See [link to relevant documentation]. Figure 13 As shown, using the above-mentioned drilling count method, there were a total of 2 drilling advances and 3 drilling retreats (i.e., retreat-advance-retreat-push).

[0146] As can be seen, the embodiments of this application, through an instance segmentation model based on an attention mechanism, can perform pixel-level accurate instance segmentation of the drilling rig body and drill rod in drilling operation videos, thereby significantly improving the ability to identify and segment drill rod targets in complex mine images. In addition, the embodiments of this application, based on the changing trend of the ratio of the drilling rig body mask area to the drill rod mask area, and combined with the area ratio threshold, perform drill rod entry and exit status identification and drill rod entry and exit count. It can be seen that the embodiments of this application not only consider the proportion of the drill rod in the image, but also the spatial relationship between the drill rod and the drilling rig body, without relying on the target angle and tracking trajectory. In this way, even if the angle of the drilling rig changes, the ratio between the masks can remain stable, thus adapting to different angle changes and ensuring that the relationship between the drill rod and the drilling rig body is not affected by the change of viewing angle. Therefore, when facing changes in the drilling rig angle, it can still provide accurate drill rod counting results, avoiding the judgment error that may be caused by the "frame with the largest effective area" in traditional methods, thereby improving the accuracy of drill rod counting and avoiding misidentification and omission of entry and exit. Furthermore, the instance segmentation model based on the attention mechanism proposed in this application has frequency domain perception capability, which can improve edge perception and spatial semantic consistency of long targets such as drill pipes in mine images. In addition, by using the area ratio of the mask region as the basis for state switching, the influence of external variables such as image resolution, frame rate, and viewpoint on the judgment logic can be effectively suppressed.

[0147] Accordingly, this application also discloses a coal mine underground drill pipe counting device based on instance segmentation, see [link to relevant documentation]. Figure 14 As shown, the device includes:

[0148] The video acquisition module 11 is used to acquire the operation video of the target coal mine underground drilling rig and obtain the drilling rig operation video sequence;

[0149] The instance segmentation module 12 is used to sequentially input each frame image from the drilling rig operation video sequence into the trained instance segmentation model to perform instance segmentation on the drilling rig body and drill rod within the current input frame image, obtaining the drill rod mask contour and the drilling rig body mask contour; the trained instance segmentation model is a model obtained by training an initial instance segmentation model based on an attention mechanism using a training set; the training set includes real drilling video data from multiple coal mine underground scenes;

[0150] The pixel area acquisition module 13 is used to acquire the pixel area of ​​the drill rod mask contour and the drill rig body mask contour corresponding to the current input frame image, and obtain the current drill rod mask area and the current drill rig body mask area.

[0151] The calculation module 14 is used to calculate the ratio of the current drilling rig body mask area to the current drill rod mask area to obtain the mask area ratio.

[0152] Analysis module 15 is used to analyze the changing trend of multiple mask area ratios corresponding to multiple frames of images within a preset time before the current moment, and obtain the ratio changing trend.

[0153] The first counting module 16 is used to determine that the current drill pipe state is the drilling state if the ratio change trend is a continuous increasing trend, and to determine whether the current mask area ratio is greater than the first area ratio threshold. If so, the counting operation of incrementing the drilling count by 1 is performed.

[0154] The second counting module 17 is used to determine that the current drill pipe state is a drill retraction state if the ratio changes in a continuously decreasing trend, and to determine whether the current mask area ratio is less than the second area ratio threshold. If so, the counting operation of incrementing the drill retraction count by 1 is performed; the second area ratio threshold is less than the first area ratio threshold.

[0155] The specific workflow of each of the above modules can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.

[0156] As can be seen, the embodiments of this application, through an instance segmentation model based on an attention mechanism, can perform pixel-level accurate instance segmentation of the drilling rig body and drill rod in drilling operation videos, thereby significantly improving the ability to identify and segment drill rod targets in complex mine images. In addition, this application, based on the changing trend of the ratio of the drilling rig body mask area to the drill rod mask area, and combined with the area ratio threshold, performs drill rod entry and exit status identification and drill rod entry and exit count. It can be seen that this application not only considers the proportion of the drill rod in the image, but also the spatial relationship between the drill rod and the drilling rig body. In this way, even if the angle of the drilling rig changes, the ratio between the masks can remain stable, thus adapting to different angle changes and ensuring that the relationship between the drill rod and the drilling rig body is not affected by the change of viewing angle. Therefore, when facing changes in the drilling rig angle, it can still provide accurate drill rod counting results, thereby improving the accuracy of drill rod counting and avoiding misidentification and omission of entry and exit.

[0157] In some specific embodiments, the first counting module 16 may specifically include:

[0158] The first judgment unit is used to determine whether the current mask area ratio is greater than the first area ratio threshold.

[0159] The second judgment unit is used to determine whether the current system's counting state is in a countable state if the current mask area ratio is greater than the first area ratio threshold.

[0160] The first counting unit is used to increment the drilling count by 1 if the current system is in a countable state, and to obtain the drilling count statistics.

[0161] Accordingly, after the first counting module 16, the system may further include:

[0162] The first setting unit is used to set the current system's counting state to an uncountable state.

[0163] In some specific embodiments, the second counting module 17 may specifically include:

[0164] The third judgment unit is used to determine whether the current mask area ratio is less than the second area ratio threshold.

[0165] The fourth judgment unit is used to determine whether the current system's counting state is in a countable state if the current mask area ratio is less than the second area ratio threshold.

[0166] The second counting unit is used to increment the number of drill retractions by 1 if the current system counting state is in a countable state, and to obtain the statistical result of the number of drill retractions.

[0167] Accordingly, after the second counting module 17, it may also include:

[0168] The second setting unit is used to set the current system's counting state to an uncountable state.

[0169] In some specific embodiments, after the first setting unit or the second setting unit, the system may further include:

[0170] The monitoring unit is used to monitor in real time the new drill rod mask and the new drill rig body mask obtained by the trained instance segmentation model after performing instance segmentation on the drill rig body and drill rod in the next frame image;

[0171] The first calculation unit is used to calculate the pixel area of ​​the new drilling rig body mask and the new drill rod mask respectively, so as to obtain the area of ​​the new drill rod mask and the area of ​​the new drilling rig body mask.

[0172] The second calculation unit is used to calculate the ratio of the mask area of ​​the new drilling rig body to the mask area of ​​the new drill rod, and obtain the new mask area ratio;

[0173] The fifth judgment unit is used to determine whether the area ratio of the new mask is less than the third area ratio threshold or greater than the fourth area ratio threshold.

[0174] The third setting unit is used to set the counting state of the current system from uncountable state to countable state if the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold.

[0175] Wherein, the fourth area ratio threshold is greater than the third area ratio threshold, the third area ratio threshold is less than the second area ratio threshold, and the fourth area ratio threshold is greater than the first area ratio threshold.

[0176] In some specific embodiments, the third setting unit may specifically include:

[0177] The sixth judgment unit is used to determine whether the target mask area ratios corresponding to all other frames of the next frame image that are a preset number of frames before the next frame image are all less than the third area ratio threshold or greater than the fourth area ratio threshold if the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold.

[0178] The fourth setting unit is used to set the counting state of the current system from uncountable to countable if the target mask area ratio corresponding to all other frames of a preset number of frames before the next frame image is less than the third area ratio threshold or greater than the fourth area ratio threshold.

[0179] In some specific embodiments, the instance segmentation-based underground drill pipe counting device for coal mines may further include:

[0180] The dataset annotation unit is used to collect real drilling video data from multiple underground coal mines in different scenarios, and to annotate the drilling rig body and drill rod in each frame of the real drilling video data to construct a dataset; the different scenarios include different angles and different lighting conditions;

[0181] The model training unit is used to input the dataset into the YOLOv11n-seg model based on the self-attention mechanism for model training, so as to obtain the trained instance segmentation model.

[0182] In some specific embodiments, the model structure of the YOLOv11n-seg model based on the self-attention mechanism consists of a backbone network, a neck network, and a segmentation detection head;

[0183] The backbone network includes residual blocks based on partial dynamic convolution and attention residual modules based on dynamic statistics, which are used to extract multi-scale features from each frame of the input image; the attention residual modules based on dynamic statistics include self-attention mechanism components based on distribution statistics and learnable nonlinear mapping functions.

[0184] The neck network includes the residual block based on partial dynamic convolution and the adaptive multi-scale feature aggregation mechanism module, which is used to perform multi-level fusion of the multi-scale features output by the backbone network.

[0185] The segmentation detection head is used to perform category prediction and bounding box regression on the feature fusion results output by the neck network at multiple different scales.

[0186] Furthermore, embodiments of this application also disclose an electronic device, Figure 15 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0187] Figure 15 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the instance segmentation-based coal mine drill pipe counting method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0188] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0189] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0190] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the instance-segmentation-based coal mine drill pipe counting method disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0191] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned instance-segmentation-based method for counting drill pipes in coal mines. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0192] Furthermore, embodiments of this application also disclose a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the instance segmentation-based coal mine drill pipe counting method disclosed above.

[0193] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0194] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0195] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0196] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0197] The above provides a detailed description of the method, apparatus, and equipment for counting drill pipes in coal mines based on instance segmentation. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A coal mine underground drill pipe counting method based on instance segmentation, characterized in that, The method comprises the following steps: acquiring a running video of a drilling machine in a target coal mine to obtain a drilling machine operation video sequence; inputting each frame image in the drilling machine operation video sequence into a trained instance segmentation model in sequence to perform instance segmentation on the drilling machine main body and the drill rod in the current input frame image, and obtain a drill rod mask contour and a drilling machine main body mask contour; the trained instance segmentation model is a model obtained by training an initial instance segmentation model based on an attention mechanism using a training set; the training set includes real drilling video data of multiple different scenes in a coal mine underground; acquiring pixel areas of the drill rod mask contour and the drilling machine main body mask contour corresponding to the current input frame image to obtain a current drill rod mask area and a current drilling machine main body mask area; calculating a ratio of the current drilling machine main body mask area to the current drill rod mask area to obtain a mask area ratio, and analyzing a change trend of multiple mask area ratios corresponding to multiple frame images within a preset time before the current time to obtain a ratio change trend; if the ratio change trend is a continuous increasing trend, it is determined that the current drill rod state is a drilling-in state, and it is determined whether the current mask area ratio is greater than a first area ratio threshold; if yes, a drilling-in number count operation is performed; if the ratio change trend is a continuous decreasing trend, it is determined that the current drill rod state is a drilling-out state, and it is determined whether the current mask area ratio is less than a second area ratio threshold; if yes, a drilling-out number count operation is performed; the second area ratio threshold is less than the first area ratio threshold.

2. The instance segmentation based coal mine downhole drill pipe counting method according to claim 1, characterized in that, The method for determining whether the current mask area ratio is greater than the first area ratio threshold and performing the drilling-in number count operation if yes comprises the following steps: determining whether the current mask area ratio is greater than the first area ratio threshold; if the current mask area ratio is greater than the first area ratio threshold, it is determined whether a counting state of the current system is in a countable state; if the counting state of the current system is in the countable state, a drilling-in number count operation is performed to obtain a drilling-in number statistical result; correspondingly, after the drilling-in number count operation, the method further comprises the following step: setting the counting state of the current system to an uncountable state.

3. The method for counting the coal mine underground drill pipe based on instance segmentation according to claim 2, characterized in that, The method for determining whether the current mask area ratio is less than the second area ratio threshold and performing the drilling-out number count operation if yes comprises the following steps: determining whether the current mask area ratio is less than the second area ratio threshold; if the current mask area ratio is less than the second area ratio threshold, it is determined whether the counting state of the current system is in the countable state; if the counting state of the current system is in the countable state, a drilling-out number count operation is performed to obtain a drilling-out number statistical result; correspondingly, after the drilling-out number count operation, the method further comprises the following step: setting the counting state of the current system to the uncountable state.

4. The method for counting the coal mine underground drill pipe based on instance segmentation according to claim 3, characterized in that, After the counting state of the current system is set to the uncountable state, the method further comprises the following step: real-time monitoring a new drill rod mask and a new drilling machine main body mask obtained by performing instance segmentation on the drilling machine main body and the drill rod in a next frame image by the trained instance segmentation model; Calculate pixel areas of the new drill rig body mask and the new drill rod mask respectively to obtain a new drill rod mask area and a new drill rig body mask area; Calculate a ratio of the new drill rig body mask area and the new drill rod mask area to obtain a new mask area ratio; Determine whether the new mask area ratio is less than a third area ratio threshold or greater than a fourth area ratio threshold; If the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, set a counting state of a current system from an uncountable state to a countable state. The fourth area ratio threshold is greater than the third area ratio threshold, the third area ratio threshold is less than the second area ratio threshold, and the fourth area ratio threshold is greater than the first area ratio threshold.

5. The instance segmentation based coal mine downhole drill pipe counting method according to claim 4, characterized in that, If the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, set a counting state of a current system from an uncountable state to a countable state, including: If the new mask area ratio is less than the third area ratio threshold or greater than the fourth area ratio threshold, determine whether target mask area ratios corresponding to all other frame images of a preset number of frames before the next frame image are less than the third area ratio threshold or greater than the fourth area ratio threshold; If the target mask area ratios corresponding to all other frame images of the preset number of frames before the next frame image are less than the third area ratio threshold or greater than the fourth area ratio threshold, set the counting state of the current system from the uncountable state to the countable state.

6. The instance segmentation based coal mine downhole drill pipe counting method according to any one of claims 1 to 5, characterized in that, Further comprising: Collect real drilling video data of multiple different scenes in a coal mine underground, and label a drilling rig body and a drill rod in each frame image in the real drilling video data to construct a data set; the different scenes include different angles and different lighting conditions; Input the data set into a YOLOv11n-seg model based on a self-attention mechanism for model training to obtain the trained instance segmentation model.

7. The instance segmentation based coal mine downhole drill pipe counting method according to claim 6, characterized in that, The model structure of the YOLOv11n-seg model based on the self-attention mechanism is composed of a backbone network, a neck network, and a segmentation detection head; The backbone network includes a residual block based on partial dynamic convolution and an attention residual module based on dynamic statistical driving, which are used for multi-scale feature extraction of input frame images; the attention residual module based on dynamic statistical driving includes a self-attention mechanism component based on distribution statistical driving and a learnable nonlinear mapping function; The neck network includes the residual block based on partial dynamic convolution and a self-adaptive multi-scale feature aggregation mechanism module, which are used for multi-level fusion of multi-scale features output by the backbone network; The segmentation detection head is used for class prediction and bounding box regression of feature fusion results output by the neck network at multiple different scales.

8. A coal mine underground drill rod counting device based on instance segmentation, characterized in that, Comprising: A video acquisition module is configured to acquire a running video of a target coal mine drilling rig to obtain a drilling rig operation video sequence; An instance segmentation module is configured to sequentially input each frame image in the drilling rig operation video sequence into a trained instance segmentation model to perform instance segmentation on a drilling rig main body and a drill rod in a current input frame image, and obtain a drill rod mask contour and a drilling rig main body mask contour; the trained instance segmentation model is a model obtained by training an initial instance segmentation model based on an attention mechanism using a training set; the training set includes real drilling video data of multiple different scenes in a coal mine underground; A pixel area acquisition module is configured to acquire pixel areas of the drill rod mask contour and the drilling rig main body mask contour corresponding to a current input frame image, and obtain a current drill rod mask area and a current drilling rig main body mask area; A calculation module is configured to calculate a ratio of the current drilling rig main body mask area to the current drill rod mask area, and obtain a mask area ratio; An analysis module is configured to analyze a change trend of multiple mask area ratios corresponding to multiple frame images within a preset time before a current time, and obtain a ratio change trend; A first counting module is configured to, if the ratio change trend is a continuously increasing trend, determine that a current drill rod state is a drilling-in state, and determine whether the current mask area ratio is greater than a first area ratio threshold; if yes, perform a drilling-in number incrementing counting operation; A second counting module is configured to, if the ratio change trend is a continuously decreasing trend, determine that a current drill rod state is a drilling-out state, and determine whether the current mask area ratio is less than a second area ratio threshold; if yes, perform a drilling-out number incrementing counting operation; the second area ratio threshold is less than the first area ratio threshold.

9. An electronic device, comprising: A processor and a memory are included; when the processor executes a computer program saved in the memory, the instance segmentation based coal mine underground drill rod counting method of any one of claims 1 to 7 is realized.

10. A computer-readable storage medium, characterized in that, A computer program is stored; when the computer program is executed by a processor, the instance segmentation based coal mine underground drill rod counting method of any one of claims 1 to 7 is realized.

Images