A Method and System for Detecting Coal and Gangue Targets in Underground Mines Based on Multi-Source Image Fusion

Abstract

This invention belongs to the field of underground coal and gangue target detection technology, specifically disclosing an underground coal and gangue target detection method and system based on multi-source image fusion. Addressing the problem of the diversity and similarity of coal and gangue in complex underground environments, and the ineffectiveness of traditional single sensors in identification, this invention constructs an underground coal and gangue target detection model based on an improved RT-DETR model architecture. Regarding model input, this invention simultaneously acquires visible light images and infrared thermal images of mineral particles during underground coal transportation. In terms of model architecture, based on the original RT-DETR model, this invention introduces a dual-modal branch and designs a sparse-scale feature interaction module, a cross-scale feature aggregation module, and a feature fusion module suitable for cross-modal information interaction, resulting in an improved RT-DETR model that enhances the accuracy of underground coal and gangue target identification.

Description

Technical Field

[0001] This invention belongs to the field of underground coal and gangue target detection technology, specifically relating to an underground coal and gangue target detection method and system based on multi-source image fusion. Background Technology

[0002] Coal preparation technology is a key link in achieving clean coal processing and utilization. By implementing underground coal and gangue separation, not only can the quality of raw coal be improved and transportation costs reduced, but the gangue can also be used as filling material in underground goaf areas, promoting the comprehensive implementation of green mining. During the mining, transportation, and processing of coal, it typically appears in three surface states: clean, dust-adhered, and wet. Dust adhesion can obscure the texture and color differences between coal and gangue, while a wet surface makes them appear similar under visible light due to light reflection and refraction. This leads to misjudgments in existing coal and gangue target detection / identification models, reducing the accuracy of underground coal and gangue target identification. Summary of the Invention

[0003] To address the problem that traditional single sensors cannot effectively identify coal and gangue targets in complex underground environments due to their diversity and similarity, this invention proposes an underground coal and gangue target detection method based on multi-source image fusion to improve the accuracy of underground coal and gangue target identification.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A method for detecting coal and gangue targets in underground mines based on multi-source image fusion includes the following steps:

[0006] Step 1. First, acquire visible light images and infrared thermal images of mineral particles in the underground coal transportation process using a visible light camera and an infrared thermal imager; the visible light images and infrared thermal images are matched one-to-one in time and space during acquisition and form corresponding image pairs.

[0007] After preprocessing and labeling the images, a training dataset is obtained;

[0008] Step 2. Construct an underground coal and gangue target detection model based on the improved RT-DETR model architecture. The underground coal and gangue target detection model includes two backbone networks, an encoder, a query selection module, and a decoder.

[0009] Two backbone networks are defined as backbone network one and backbone network two. Backbone network one and backbone network two adopt a parallel structure and both use ResNet50 networks pre-trained on ImageNet.

[0010] Backbone network 1 is used to receive the input visible light image and extract multi-scale features from it; backbone network 2 is used to receive the input infrared thermal imaging image and extract multi-scale features from it.

[0011] The encoder is used to accept multi-scale output features from backbone network 1 and backbone network 2, and to perform cross-scale and cross-modal fusion of the input features of the two backbone networks, effectively encoding the correlation between visible light and infrared thermal imaging modalities.

[0012] The encoder outputs a sequence of converted image features. The query selection module selects a fixed number of image features from the sequence of image features output by the encoder and uses them as the initial object query for the decoder.

[0013] The decoder optimizes object queries and generates predicted bounding boxes and confidence scores for targets in underground coal and gangue images;

[0014] Step 3. Based on the training dataset constructed in Step 1, train the underground coal and gangue target detection model, and use the trained underground coal and gangue target detection model to detect and identify underground coal and gangue targets.

[0015] Furthermore, based on the aforementioned method for detecting underground coal and gangue targets using multi-source image fusion, this invention also proposes a corresponding system for detecting underground coal and gangue targets using multi-source image fusion, which adopts the following technical solution:

[0016] A coal and gangue target detection system based on multi-source image fusion includes a visible light camera, an infrared thermal imager, and a computer device, wherein the computer device includes a memory and one or more processors;

[0017] Visible light cameras and infrared thermal imagers are used to acquire images of mineral particles during underground coal transportation, respectively, to obtain visible light images and infrared thermal images, which are then sent to computer equipment.

[0018] The memory stores executable code; when the processor executes the executable code, it implements the steps of the underground coal and gangue target detection method based on multi-source image fusion as described above.

[0019] The present invention has the following advantages:

[0020] As described above, this invention addresses the problem of the diversity and similarity of coal and gangue in complex underground environments, which traditional single sensors cannot effectively identify. It proposes a multi-source image fusion-based method for underground coal and gangue target detection, which constructs an underground coal and gangue target detection model based on an improved RT-DETR model architecture. Specifically, regarding model input, this invention simultaneously acquires visible light and infrared thermal images of mineral particles during underground coal transportation. Compared to traditional single-sensor image acquisition methods, this approach not only more effectively separates coal and gangue in complex backgrounds but also utilizes the complementary visual information of visible light and infrared images to improve the overall performance of the model. In terms of model architecture, this invention makes the following improvements to the original RT-DETR model, resulting in an improved RT-DETR model. Specifically, addressing the problem that direct superposition of visible light and infrared image features makes it difficult to effectively utilize their complementary information when inputting them into the same backbone network, this invention introduces a dual-modal branch (visible light and infrared thermal imaging modal branches) to enhance the model's ability to express multimodal features and improve feature extraction. To address the issue that single-scale feature maps are insufficient to effectively capture targets at different scales due to the significant differences in the mineral scales of coal gangue, a self-attention module suitable for high-level semantic features, namely the Sparse Intra-scale Feature Interaction (SIFI) module, is designed in the encoder to enhance the model's detection capability. This module is used to capture the relationships between conceptual entities in the image. A Cross-scale Feature Aggregation Module (CFAM) is designed to combine feature maps of different scales, enhancing the model's adaptability to large and small targets and improving detection accuracy. Furthermore, considering the differences in feature distribution between visible light and infrared images, and the potential for information redundancy or insufficiency in certain regions across different modalities, this invention also designs a Spatial-Channel Aggregation Module (SCAM) suitable for cross-modal information interaction, enabling efficient complementarity and integration of features between different modalities. Furthermore, to address the issues of redundant attention heads and computational global attention in multi-head self-attention systems, which hinder the effective capture of feature differences and interactive, irrelevant feature noise between different heads, this invention designs a novel Cascaded Sparse Attention (CSA) mechanism for long-distance dependency modeling. In the encoder, a Multi-level Feature Map Aggregation (MFA) operation is used to integrate multi-level spatial location features from the decoder to form stronger encoded features, better capturing the diverse features of the target. Through these improvements in model input and structure, the method of this invention can significantly improve the accuracy of identifying underground coal and gangue targets. Attached Figure Description

[0021] Figure 1 This is a flowchart of the underground coal and gangue target detection method based on multi-source image fusion in an embodiment of the present invention;

[0022] Figure 2 These are schematic diagrams of some data examples, data acquisition platforms, and infrared imagers in embodiments of the present invention; Figure 2 (a) is a visible light coal and gangue sample, (b) is an infrared coal and gangue sample, (c) is a data acquisition platform, and (d) is an infrared imager;

[0023] Figure 3 This is a network structure diagram of the underground coal and gangue target detection model built in an embodiment of the present invention;

[0024] Figure 4 This is a network structure diagram of the SIFI module in an embodiment of the present invention;

[0025] Figure 5 This is a network structure diagram of the cascaded sparse attention module (CSA) in an embodiment of the present invention;

[0026] Figure 6 This is a network structure diagram of the hybrid expert model module in an embodiment of the present invention;

[0027] Figure 7 This is a network structure diagram of the SCAM module in an embodiment of the present invention;

[0028] Figure 8 This is a network structure diagram of the decoder in an embodiment of the present invention;

[0029] Figure 9 This is a schematic diagram illustrating a portion of the detected ground truth values ​​in the dataset used in the experiment of this invention.

[0030] Figure 10 This is a schematic diagram illustrating a partial detection example of the GM-DETR model in the dataset used in the experiments of this invention;

[0031] Figure 11 This is a schematic diagram illustrating a partial detection example of the dataset from the model presented in this invention during the experiment.

[0032] Figure 9 , Figure 10 and Figure 11 (a), (b), (c), and (d) in the text represent detection examples under the following scenarios: clean surface condition, mixed clean surface condition and dust-adhering surface condition, dust-adhering surface condition, and wet surface condition, respectively. Detailed Implementation

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] Example 1

[0035] This embodiment 1 describes a method for detecting coal and gangue targets in underground mines based on multi-source image fusion, in order to solve the problem that traditional single sensors cannot effectively identify the diversity and similarity of coal and gangue in complex underground environments.

[0036] like Figure 1 As shown, the method for detecting coal and gangue targets underground based on multi-source image fusion includes the following steps:

[0037] Step 1. First, acquire visible light images and infrared thermal images of mineral particles in the underground coal transportation process using a visible light camera and an infrared thermal imager.

[0038] The visible light images and infrared thermal images are acquired in a one-to-one correspondence in time and space, forming corresponding image pairs. After preprocessing and labeling the images, the training dataset is obtained.

[0039] During the mining, transportation, and processing of coal, coal typically exists in three surface states: clean, dust-laden, and wet. Dust adherence can obscure the texture and color differences between coal and gangue, while a wet surface makes them appear similar under visible light due to light reflection and refraction. To address this, this invention acquires visible light and infrared images of coal and gangue under multiple surface states. Some data examples, the data acquisition platform, and the infrared imager are shown below. Figure 2 As shown.

[0040] Use the LabelImg tool to annotate the images and uniformly adjust the size of all images to 640×640 pixels.

[0041] Step 2. Construct an underground coal and gangue target detection model based on the improved RT-DETR model architecture. RT-DETR (Real-Time Detection, Embedding, and Tracking) is a real-time target detection model based on Transformer.

[0042] like Figure 3 The network structure diagram of the underground coal and gangue target detection model is shown, consisting of... Figure 3 It can be seen that the underground coal and gangue target detection model includes two backbone networks, an encoder, a query selection module, and a decoder.

[0043] Two backbone networks are defined as backbone network one and backbone network two. Backbone network one and backbone network two adopt a parallel structure and both use ResNet50 networks pre-trained on ImageNet.

[0044] Two backbone networks are used to construct independent multi-level features for each modality.

[0045] Backbone network 1 is used to receive the input visible light image and extract multi-scale features from it; backbone network 2 is used to receive the input infrared thermal image and extract multi-scale features from it.

[0046] The encoder is used to accept multi-scale output features from backbone network 1 and backbone network 2, and to perform cross-scale and cross-modal fusion of the input features of the two backbone networks, effectively encoding the correlation between visible light and infrared thermal imaging modalities.

[0047] The encoder outputs a sequence of converted image features. The query selection module selects a fixed number of image features from the sequence of image features output by the encoder and uses them as the initial object query for the decoder.

[0048] The decoder optimizes object queries and generates predicted bounding boxes and confidence scores for targets in underground coal and gangue images.

[0049] The improved RT-DETR model architecture in this embodiment is obtained by making the following improvements to the original RT-DETR model:

[0050] By introducing visible light and infrared thermal imaging modal branches, the network's ability to extract multimodal information is structurally enhanced.

[0051] In the encoder, a self-attention module suitable for high-level semantic features, namely the Sparse Intra-scale Feature Interaction (SIFI) module, is designed to capture the associations between conceptual entities in the image.

[0052] We designed a feature fusion module suitable for cross-modal information interaction, namely the Cross-scale Feature Aggregation Module (CFAM), to achieve efficient complementarity and integration of features between different modalities.

[0053] Design a feature aggregation module that focuses on salient target regions and texture details for multimodal fusion features, namely the Spatial-Channel Aggregation Module (SCAM).

[0054] To reduce the computational complexity and resource consumption of the model, a more efficient attention mechanism, namely Cascaded Sparse Attention (CSA), is used in network design.

[0055] In the encoder, the multi-level feature map aggregation (MFA) operation is used to gather global attention features to obtain stronger encoded features and better capture the diverse features of the target.

[0056] The following section provides a more detailed explanation of each module in the improved RT-DETR model architecture.

[0057] The backbone network receives the input visible light image, extracts its multi-scale features, and outputs the multi-scale features of the last three stages {S3}. rgb S4 rgb S5 rgb}, which serves as the input to the encoder.

[0058] Simultaneously, backbone network 2 receives the input infrared thermal imaging image, extracts its multi-scale features, and outputs the multi-scale features of the last three stages {S3}. ir S4 ir S5 ir}, which serves as the input to the encoder.

[0059] like Figure 3 As shown, the encoder includes two SIFI modules and one CFAM module. The two SIFI modules are for high-level features S5 in visible light and infrared, respectively. rgb and S5 ir Performing sparse self-attention operations yields F5. rgb and F5 ir By using a sparse attention mechanism, the correlation between different targets within the same modality is enhanced, thereby improving the detector's understanding of coal gangue characteristics.

[0060] The CFAM module performs cross-modal and cross-scale complementary learning of visible light and infrared features to compensate for feature loss or bias in single modes and enhance the ability to distinguish coal and gangue. The SIFI and CFAM modules work together, with SIFI first enhancing the understanding of single-modal features and CFAM then performing cross-modal fusion to make the target feature representation more complete and accurate.

[0061] The input features of the CFAM module are {F5} rgb S4 rgb S3 rgb} and {F5 ir S4 ir S3 ir Finally, the encoder output X is obtained. E .

[0062] like Figure 4As shown, the SIFI module, or Sparse Intra-scale Feature Interaction (SIFI) module, includes a Cascaded Sparse Attention (CSA) module, an RMS normalization module, a hybrid expert model module, and residual connections. The processing flow of the SIFI module is as follows:

[0063] a1. The visible light high-level features S5 output from the backbone network. rgb Or the high-level infrared signature S5 output from the backbone network ir The input is fed into the SIFI module, projected onto a fixed dimension through a linear transformation, and then a positional encoding is added.

[0064] Because applying self-attention operations to high-level features with richer semantic information can capture the relationships between conceptual entities in an image, it helps subsequent modules detect and recognize image targets. Therefore, SIFI only applies to {S5}. ir S5 rgb Perform the calculation.

[0065] a2. Cascaded sparse attention using the Cascaded Sparse Attention Module (CSA) to compute features.

[0066] The computation of attention enables the model to highlight important infrared / visible light features through weighted summation, while suppressing irrelevant features. The multi-head mechanism allows the model to focus on the relationships between different feature spaces and enhances the model's ability to capture complex relationships.

[0067] In real-world environments, different coal gangue surface conditions have a significant impact on the target's performance. This embodiment, through the proposed CSA module, can help the model focus on the most important coal gangue features under various backgrounds, ignoring irrelevant interference.

[0068] The features output by the a3.CSA module are processed by the RMS normalization module. RMS normalization can stabilize the training process and improve the generalization ability of the model.

[0069] The output features of the a4.RMS normalization module and the input features of the SIFI module are added together through a residual connection to alleviate the gradient vanishing problem and accelerate convergence.

[0070] a5. The result of the residual connection summation is input into the hybrid expert model module, which further extracts deeper features and performs nonlinear transformations on the feature vectors at each position independently to enhance expressive power.

[0071] Since the characteristics of coal gangue may change under different backgrounds and conditions, using a hybrid expert model can better extract and express these changes, thereby improving the model's recognition accuracy and adaptability.

[0072] a6. The features output by the hybrid expert model module are processed by the RMS normalization module. Through RMS normalization, the training process can be stabilized and the generalization ability of the model can be improved.

[0073] a7. Add the output features from the RMS normalization module in step a6 to the input features from the hybrid expert model module in step a5 via residual connection to obtain the final output feature F5. ir Or output feature F5 rgb .

[0074] exist Figure 4 Feature S5 shown in the text represents feature S5. rgb or S5 ir Feature F5 represents F5 ir Or F5 rgb .

[0075] These features (referring to S5) rgb or S5 ir Through the meticulous processing of the above steps, the model can better identify coal gangue targets, and the final output provides accurate and rich feature representations for subsequent target identification and classification.

[0076] In step a2, the processing flow of the cascaded sparse attention module (CSA) is as follows:

[0077] b1. Provide different segmented blocks of complete infrared / visible light features for each head, and explicitly decompose the attention similarity between heads.

[0078] By assigning these features to different heads, the model can effectively capture the unique information of coal gangue under different surface conditions, thereby enhancing its adaptability to complex backgrounds and diverse surface conditions.

[0079] b2. Sparse attention is calculated for each head separately to reduce computational complexity. The calculation process is given below.

[0080] b3. To enhance information flow, the output of each head is passed layer by layer to the next head.

[0081] In coal gangue detection, this mechanism ensures that features under different surface conditions complement and cooperate with each other. The output of each head provides richer contextual information for subsequent heads, enabling the model to capture more comprehensive target features.

[0082] b4. Concatenate and map the attention maps of all heads to obtain the final result. By concatenating the attention maps of all heads, the model obtains feature fusion results from different heads, which helps to generate more accurate detection results.

[0083] In step b2, the sparse attention process for the j-th head is described as follows:

[0084] b21. After calculating the similarity between Q and K, unnecessary elements with low attention weights are masked in the attention matrix M.

[0085] b22. Adaptively select the top k scores with the highest contribution on M, retaining only the most important components.

[0086] b23. For other elements in M ​​that are less than k, use the Scatter function to replace their probabilities with 0 at a specified index, thus changing the attention computation from dense to sparse.

[0087] b24. Use the Where function to replace the position with index 0 with -∞.

[0088] b25. The first k values ​​within the interval [Δ1, Δ2] are standardized in each row of M, and the Softmax is calculated. [Δ1, Δ2] is the threshold range for standardizing the attention matrix in sparse attention calculation, where Δ1 is the minimum threshold and Δ2 is the maximum threshold.

[0089] b26. Multiply the Softmax result by V using matrix multiplication to obtain the attention map of the head.

[0090] like Figure 6 As shown, the hybrid expert model module includes n expert networks, a routing network, and an expert fusion module; the expert networks adopt a feedforward neural network (FFN); the computation flow of the hybrid expert model module is as follows:

[0091] c1. Input feature vector.

[0092] c2. The routing network dynamically selects the most relevant experts based on the input feature vector and assigns a weight p to each expert.

[0093] Since coal gangue exhibits different characteristics under different surface conditions, the routing network in this embodiment can intelligently select the expert most relevant to the current task for processing based on the input characteristics.

[0094] c3. By setting a sparsity strategy, select the k experts with the highest weights from n experts and hide the weights of the remaining experts.

[0095] c4. Each expert processes the input feature vector independently; since each expert can learn a specific task independently, they can focus on extracting specific features from different surface states.

[0096] c5. The outputs of all activation experts are fused according to the weight p to generate a unified representation. Combining the results of multiple experts provides a more comprehensive feature representation for object detection tasks.

[0097] like Figure 7 The network structure diagram of the Cross-scale Feature Aggregation Module (CFAM) is shown. The CFAM module adopts two branches: a top-down FPN and a bottom-up PAN.

[0098] Specifically, the CFAM module includes four SCAM modules, two upsampling modules (U), and two downsampling modules (D), such as... Figure 7 As shown.

[0099] The upsampling module consists of a 1×1 convolution with a stride of 1, a batch normalization (BN), and a SiLu; the downsampling module consists of a 3×3 convolution with a stride of 2, a BN, and a SiLu.

[0100] For ease of representation, four SCAM modules are defined as SCAM1, SCAM2, SCAM3 and SCAM4.

[0101] like Figure 3 As shown, the calculation process of the CFAM module is as follows:

[0102] d1. FPN branch processing flow:

[0103] The upsampled features F5 ir and feature S4 rgb and feature S4 ir The input is fed into SCAM1 to obtain the output feature F4. The upsampled feature F4 and feature S3 are then combined... rgb and feature S3 ir The input is fed into SCAM2 to obtain the output feature F3.

[0104] d2.PAN branch processing flow:

[0105] The downsampled features F3, F4, and S4 are used. rgb The input is fed into SCAM3 to obtain output feature P4. The downsampled P4 and feature F5 are then combined... rgb and F5 irThe input is fed into SCAM4 to obtain the output feature P5.

[0106] d3. Flatten features F3, P4, and P5 separately, then concatenate them together to obtain the encoder output X. E This unified feature representation integrates features from different levels, which helps improve the model's detection accuracy for minerals at different scales.

[0107] The SCAM module introduces multiple attention mechanisms, enabling the network to focus more on salient target areas, while enhancing its ability to resolve coal gangue textures and edge details, thereby improving detection robustness.

[0108] Specifically, such as Figure 7 As shown, each SCAM module consists of three 1×1 convolutional modules, one channel feature aggregation module, and one spatial feature aggregation module. The SCAM module performs multi-level feature fusion, and its calculation process is as follows:

[0109] e1. The three input features of each SCAM module are stacked in the channel dimension. By stacking these features, the model can simultaneously consider features from different modalities and different levels in a unified space.

[0110] e2. Three branches are generated through three 1×1 convolutional layers;

[0111] e3. Perform channel feature aggregation, spatial feature aggregation, and residual connection respectively. Channel feature aggregation focuses on feature combination in the channel dimension, spatial feature aggregation strengthens the feature expression at specific locations, and the output features are added and fused.

[0112] This embodiment enhances the focus on coal gangue targets through an attention mechanism. Both the heat distribution in infrared images and the texture features in visible light images help improve the model's sensitivity to coal gangue targets.

[0113] The role of the query selection module is to dynamically filter high-quality queries that are relevant to the target, while introducing an IoU (Intersection over Union) prediction network to reduce query redundancy, improve computational efficiency, and increase target detection accuracy.

[0114] The processing flow for the query selection module is as follows:

[0115] f1. The encoder generates a fixed number of embedding vectors as the initial query;

[0116] f2. An IoU score is generated for each query using an IoU prediction network (using an MLP network as the IoU prediction network) to measure the degree of matching between the query and the target. This can help evaluate whether each query accurately matches the target coal and gangue. Highly matched queries help improve the detection accuracy of coal and gangue targets.

[0117] f3. Sort all queries by IoU score.

[0118] f4. Select the n queries with the highest scores and discard the rest to save computing resources.

[0119] Because coal gangue targets are often located in complex contexts, a large number of irrelevant queries may be generated. By filtering high-quality queries, the model can focus on areas that are truly relevant to the coal gangue target, reducing computational overhead and improving processing speed and efficiency.

[0120] f5. The n queries selected are used as input to the decoder.

[0121] like Figure 8 As shown, the core components of the decoder include a multi-head attention mechanism, a global attention mechanism, a feedforward neural network, residual connections, and layer normalization. Other modules are omitted for a clearer view of the decoder.

[0122] The decoder's calculation process is as follows:

[0123] g1. Input a set of learnable features to capture the feature information of coal and gangue in multiple surface states underground. These features represent the preliminary information of the target to be detected. In this embodiment, they specifically refer to underground coal and gangue targets.

[0124] g2. After learning the feature representation through n decoder layers, each layer performs the following steps:

[0125] g21. Self-attention is computed using an attention module to model the relationships between object queries, enabling them to share information and ensuring that different queries capture different targets. Through this self-attention mechanism, the decoder can accurately identify different parts of the coal gangue and ensure that queries are coordinated, thereby enhancing the perception of the coal gangue target.

[0126] g22. Calculate global attention to learn the matching relationship between the object query and the encoder output. Through global attention, the decoder can better understand the spatial relationship between coal gangue and the background in complex environments by utilizing global features extracted by the encoder.

[0127] g23. Through a feedforward neural network, the embedding of each query is nonlinearly transformed to extract high-dimensional features and enhance expressive power. The model can extract deeper feature representations and enhance the detection capability of diverse coal gangue targets.

[0128] g24. After each sub-module, namely the attention module and the feedforward neural network, add residual connections and normalization. Residual connections alleviate the gradient vanishing problem and speed up convergence; normalization stabilizes training and improves the generalization performance of the model.

[0129] g3. Concatenate the global attention features from each decoder layer, then project them back to the original dimensions via a 1×1 convolution to construct stronger encoded features, thereby better capturing the diverse features X of the target. D .

[0130] g4. Decoder feature X D The data is input into the detection head, ultimately generating a predicted coal and gangue object category and bounding box. The model can accurately predict the category and precise location of the coal and gangue, thus providing effective information for subsequent sorting and processing.

[0131] Step 3. Based on the training dataset constructed in Step 1, train the underground coal and gangue target detection model, and use the trained underground coal and gangue target detection model to detect and identify underground coal and gangue targets.

[0132] The model training process is fairly standard and will not be detailed here. After the model training is complete, actual deployment will proceed.

[0133] After acquiring visible light images and infrared thermal images of mineral particles in the underground coal transportation process from visible light cameras and infrared thermal imagers, the images are first preprocessed according to the preprocessing method in step 1 and then input into the trained model. The model processes the input images according to the processing flow in step 2, and finally obtains the coal gangue identification results.

[0134] Furthermore, to verify the effectiveness of the method proposed in this invention, the following experimental procedure is also provided:

[0135] The experiment was conducted on a Windows 10 operating system, with hardware including an NVIDIA GeForce RTX 3090 GPU (24GB VRAM) and an Intel Xeon E5-2630 v3 @ 2.40GHz CPU.

[0136] The experiment used PyTorch 1.12.1 and CUDA 11.3 for development and training. The training parameters were set as follows: epochs of 60, batch size of 8, and the AdamW optimization algorithm. The performance of the proposed method was compared with 11 state-of-the-art methods.

[0137] The 11 comparison methods used in this experiment are YOLO-Fusion, CAFF, ProbEn, CFT, CSAA, ICAFSion, MS-DETR, LRAF-Net, RSDet, CrossFormer, and GM-DETR.

[0138] The comparative experimental results are shown in Table 1. In this experiment, the index mAP@50 was used to measure whether the model can basically identify the target location, and the index mAP@[0.5:0.95] was used to comprehensively evaluate the model performance, which measures both the target recognition ability and the accuracy and robustness of the localization.

[0139] Table 1 Comparison of experimental results

[0140]

[0141] As can be seen from Table 1 above, the model proposed in this invention achieves leading performance of 95.1% and 66.1% in both mAP@50 and mAP@[0.5:0.95] metrics. MS-DETR achieves 94.9% mAP@50, demonstrating strong competitiveness, while models such as GM-DETR and CrossFormer maintain high mAP even at high IoU thresholds. Using deep backbone networks (such as GM-DETR and LRAF-Net) to extract features results in better performance in the mAP metric, indicating the key role of deep networks in extracting visible and infrared features, which helps improve detection performance. In contrast, ProbEn's mAP@[0.5:0.95] is only 54.6%, significantly lower than other models, reflecting the insufficient adaptability of its multimodal fusion strategy in complex scenarios. MS-DETR, GM-DETR, and the proposed model fully utilize multimodal information through more advanced feature fusion modules, thus achieving significant performance advantages. The method of this invention achieves the highest mAP@50, indicating its strong overall detection capability and greater stability at low IoU settings, enabling more consistent target identification. It ranks second only to GM-DETR at mAP@[0.5:0.95], demonstrating excellent detection accuracy across different IoU thresholds and strong generalization ability. Furthermore, in some mature industrial inspection tasks (such as steel defect detection), mAP@50 is considered a strong model in over 90% of cases.

[0142] Figure 9 , Figure 10 and Figure 11 The paper presents a comparison of the object detection results of GM-DETR and the model proposed in this invention on multiple datasets, providing a more intuitive way to reflect model performance. Figure 10 and Figure 11 In the diagram, the green inverted triangle indicates that the corresponding model did not detect coal at that location. The red diagonal box indicates the portion of the predicted bounding box that is missing compared to the true bounding box. Figure 10 In (a) and (d), GM-DETR exhibits bounding box offset; Figure 10In examples (b) and (c), GM-DETR suffers from some target omissions. In contrast, the method proposed in this invention is more robust in handling complex backgrounds, lighting variations, and multi-target interference, with its bounding boxes fitting the ground truth bounding boxes better and significantly reducing missed targets. Particularly in the multimodal fusion scenario (d), the identification of coal and gangue is more accurate. Overall, the method proposed in this invention significantly improves detection accuracy and bounding box regression capability through an improved multimodal feature interaction mechanism and cross-scale fusion strategy.

[0143] Example 2

[0144] This embodiment 2 describes an underground coal and gangue target detection system based on multi-source image fusion, including a visible light camera, an infrared thermal imager, and a computer device, the computer device including a memory and one or more processors.

[0145] The visible light camera and the infrared thermal imager are connected to the computer equipment and are used to acquire images of mineral particles in the underground coal transportation process, obtain visible light images and infrared thermal images, and send them to the computer equipment.

[0146] The memory stores executable code; when the processor executes the executable code, it is used to implement the steps of the underground coal and gangue target detection method based on multi-source image fusion as described in Embodiment 1 above.

[0147] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A method for detecting coal and gangue targets in underground mines based on multi-source image fusion, characterized in that, Includes the following steps: Step 1. First, acquire visible light images and infrared thermal images of mineral particles in the underground coal transportation process using a visible light camera and an infrared thermal imager. Among them, the visible light images and infrared thermal imaging images are matched one-to-one in time and space during acquisition and form corresponding image pairs; after preprocessing and labeling the acquired images, the training dataset is obtained. Step 2. Construct an underground coal and gangue target detection model based on the improved RT-DETR model architecture. The underground coal and gangue target detection model includes two backbone networks, an encoder, a query selection module, and a decoder. Two backbone networks are defined as backbone network one and backbone network two. Backbone network one and backbone network two adopt a parallel structure and both use ResNet50 networks pre-trained on ImageNet. Backbone network 1 is used to receive the input visible light image and extract multi-scale features from it; backbone network 2 is used to receive the input infrared thermal imaging image and extract multi-scale features from it. The encoder is used to accept multi-scale output features from backbone network 1 and backbone network 2, and to perform cross-scale and cross-modal fusion of the input features of the two backbone networks, effectively encoding the correlation between visible light and infrared thermal imaging modalities. The encoder outputs a sequence of converted image features. The query selection module selects a fixed number of image features from the sequence of image features output by the encoder and uses them as the initial object query for the decoder. The decoder optimizes object queries and generates predicted bounding boxes and confidence scores for targets in underground coal and gangue images; Step 3. Based on the training dataset constructed in Step 1, train the underground coal and gangue target detection model, and use the trained underground coal and gangue target detection model to detect and identify underground coal and gangue targets; In step 2, the backbone network receives the input visible light image, extracts its multi-scale features, and outputs the multi-scale features for the last three stages. , as the input to the encoder; Meanwhile, backbone network 2 receives the input infrared thermal imaging image, extracts its multi-scale features, and outputs the multi-scale features of the last three stages. , as the input to the encoder; The encoder includes two SIFI modules and one CFAM module; A SIFI module for advanced features of visible light Perform sparse self-attention operations to obtain features. A SIFI module provides advanced infrared characterization. Perform sparse self-attention operations to obtain features. ; The CFAM module performs cross-modal and cross-scale complementary learning on visible light and infrared features. The input features of the CFAM module are respectively... and Finally, the encoder output is obtained. .

2. The method for detecting underground coal and gangue targets based on multi-source image fusion according to claim 1, characterized in that, The SIFI module includes a cascaded sparse attention module, an RMS normalization module, a hybrid expert model module, and residual connections; among them, the SIFI module provides high-level features for visible light and infrared. , The processing flow is as follows: a1. High-level visible light features output from the backbone network. Or the high-level infrared features output by the backbone network The input is fed into the SIFI module, projected onto a fixed dimension through a linear transformation, and then encoded with position. a2. Calculate cascaded sparse attention for features using a cascaded sparse attention module; a3. The features output by the cascaded sparse attention module are subjected to RMS normalization by the RMS normalization module; a4. The output characteristics of the RMS normalization module are added to the input characteristics of the SIFI module through a residual connection; a5. The result of summing the residual connections is input into the hybrid expert model module, which further extracts deeper features and performs nonlinear transformations on the feature vectors at each position independently to enhance expressive power; a6. The features output by the hybrid expert model module are subjected to RMS normalization by the RMS normalization module; a7. Add the output features from the RMS normalization module in step a6 to the input features from the hybrid expert model module in step a5 via residual connection to obtain the final output features. or output features .

3. The method for detecting underground coal and gangue targets based on multi-source image fusion according to claim 2, characterized in that, The processing flow of the cascaded sparse attention module is as follows: b1. Provide different segmented blocks of complete infrared / visible light features for each head, and explicitly decompose the attention similarity between heads; b2. Calculate sparse attention for each head separately to reduce computational complexity; b3. To enhance information flow, the output of each head is passed layer by layer to the next head; b4. Concatenate and map the attention maps of all heads to obtain the final result; In step b2, the first The process of sparse attention for calculating individual values ​​can be described as follows: b21. Calculate and After similarity, in the attention matrix The middle layer masks unnecessary elements with low attention weights; b22. In the attention matrix Before adaptive selection The score with the highest contribution is used to retain only the most important components. b23. Regarding the attention matrix medium to small Other elements, using The function replaces their probabilities with 0 at a specified index, making the attention computation change from dense to sparse. b24. Use the Where function to replace the position at index 0 with -∞; b25. Located in the interval The first k values ​​will be Each row is standardized, and the Softmax is calculated. The threshold range for standardizing the attention matrix in sparse attention computation. The minimum threshold value, The maximum value of the threshold; b26. Multiply the Softmax result by matrix multiplication. Multiply them to obtain the attention map of the head.

4. The method for detecting underground coal and gangue targets based on multi-source image fusion according to claim 2, characterized in that, The hybrid expert model module includes The system comprises an expert network, a routing network, and an expert fusion module; the expert network employs a feedforward neural network (FFN); the computational flow of the hybrid expert model module is as follows: c1. Input feature vector; c2. The routing network dynamically selects the most relevant expert based on the input feature vector and assigns weights to each expert. ; c3. By setting a sparsity strategy, from Select the expert with the highest weight. One expert is selected, and the weight of other experts is disabled. c4. Each expert processes the input feature vector independently; c5. All activation experts' outputs are based on weights. The two methods are combined to generate a unified representation.

5. The method for detecting underground coal and gangue targets based on multi-source image fusion according to claim 1, characterized in that, The CFAM module adopts two branches: a top-down FPN and a bottom-up PAN. Specifically, the CFAM module includes four SCAM modules, two upsampling modules, and two downsampling modules; the four SCAM modules are defined as SCAM1, SCAM2, SCAM3, and SCAM4; the CFAM calculation process is as follows: d1. FPN branch processing flow: Upsampled features and characteristics and characteristics The input is fed into SCAM1 to obtain the output feature F4; the upsampled feature F4, along with the feature... and characteristics The input is fed into SCAM2 to obtain the output feature F3; d2. PAN branch processing flow: The downsampled features F3, and features F4 and The input is fed into SCAM3 to obtain the output feature P4; the downsampled P4 and features are then processed together. and The input is fed into SCAM4 to obtain the output feature P5; d3. Flatten features F3, P4, and P5 separately, then concatenate them together as the encoder output. ; Each SCAM module consists of three 1×1 convolutional modules, one channel feature aggregation module, and one spatial feature aggregation module. The SCAM module fuses multi-level features, and its calculation process is as follows: e1. Each SCAM module has three input features stacked in the channel dimension. By stacking these features, the model can simultaneously consider features from different modalities and different levels in a unified space. e2. Three branches are generated through three 1×1 convolutional layers; e3. Perform channel feature aggregation, spatial feature aggregation, and residual connection respectively, and then add and fuse the output features.

6. The method for detecting underground coal and gangue targets based on multi-source image fusion according to claim 1, characterized in that, The processing flow of the query selection module is as follows: f1. The encoder generates a fixed number of embedding vectors as the initial query; f2. Use an IoU prediction network to generate an IoU score for each query to measure how well the query matches the target; f3. Sort all queries by IoU score; f4. Select the n queries with the highest scores and discard the rest to save computing resources; f5. The n selected queries are used as input to the decoder.

7. The method for detecting underground coal and gangue targets based on multi-source image fusion according to claim 1, characterized in that, The decoder includes a self-attention mechanism, a global attention mechanism, a feedforward neural network, residual connections, and layer normalization. The computation process of the decoder is as follows: g1. Input a set of learnable features to capture the feature information of coal gangue in multiple surface states underground; g2. The feature representation is learned through n decoder layers. Each layer performs the following steps: g21. Self-attention mechanism calculates self-attention and models the relationships between object queries; g22. The global attention mechanism calculates global attention and learns the matching relationship between the object query and the encoder output; g23. The embedding of each query is transformed nonlinearly through a feedforward neural network; g24. After each submodule, namely the attention module and the feedforward neural network, add residual connections and layer normalization; g3. Concatenate the global attention features from each decoder layer, then project them back to the original dimensions via a 1×1 convolution to construct stronger encoded features. This helps to better capture the diverse features of the target, i.e., decoder feature X. D ; g4. Decoder feature X D The data is input into the detection head, which ultimately generates predictions for coal gangue object categories and bounding boxes.

8. A coal and gangue target detection system based on multi-source image fusion, comprising a visible light camera, an infrared thermal imager, and a computer device, wherein the computer device includes a memory and one or more processors; Visible light cameras and infrared thermal imagers are used to acquire images of mineral particles during underground coal transportation, respectively, to obtain visible light images and infrared thermal images, which are then sent to computer equipment. The memory stores executable code; characterized in that, When the processor executes the executable code, it implements the steps of the method for detecting underground coal and gangue targets based on multi-source image fusion as described in any one of claims 1 to 7.

Images