Underground coal gangue target detection method and system based on multi-source image fusion

Abstract

The invention belongs to the technical field of underground coal gangue target detection, and particularly discloses an underground coal gangue target detection method and system based on multi-source image fusion. Aiming at the diversity and similarity of coal and gangue in an underground complex environment and the problem that a traditional single sensor cannot effectively identify coal and gangue, the invention establishes an underground coal and gangue target detection model based on an improved RT-DETR model architecture. In the aspect of model input, visible light images and infrared thermal imaging images of mineral particles in a coal mine transportation link are collected at the same time. In the aspect of model architecture, on the basis of an original RT-DETR model, the improved RT-DETR model is obtained by introducing a bimodal branch and designing structures such as a sparse scale internal feature interaction module, a cross-scale feature aggregation module and a feature fusion module suitable for cross-modal information interaction, and the recognition accuracy of an underground coal gangue target is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of underground coal gangue target detection, and specifically relates to a method and system for underground coal gangue target detection based on multi-source image fusion. Background Art

[0002] Coal preparation technology is a key link in realizing the clean processing and utilization of coal. By realizing underground coal gangue separation, not only can the quality of raw coal be improved and transportation costs be reduced, but coal gangue can also be used as a filling material for underground goafs, promoting the full implementation of green mining. During the mining, transportation, and processing of coal, coal usually appears in three surface states, namely clean surface, dust-adhered surface, and wet surface. Among these three surface states, dust adhesion can mask the texture and color differences between coal and gangue, and the wet surface makes them appear similar in visible light due to light reflection and refraction, resulting in misjudgments by existing coal gangue target detection / recognition models and reducing the recognition accuracy of underground coal gangue targets. Summary of the Invention

[0003] Aiming at the problems of the diversity and similarity of coal gangue in the complex underground environment and the inability of traditional single sensors to effectively identify them, the present invention proposes a method for underground coal gangue target detection based on multi-source image fusion to improve the recognition accuracy of underground coal gangue targets.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for underground coal gangue target detection based on multi-source image fusion, comprising the following steps:

[0006] Step 1. First, obtain visible light images and infrared thermal imaging images collected by a visible light camera and an infrared thermal imager for mineral particles in the underground coal transportation link; wherein, the visible light images and the infrared thermal imaging images are in one-to-one correspondence in terms of time and space during collection and form corresponding picture pairs;

[0007] After preprocessing and annotating the images, a training data set is obtained;

[0008] Step 2. Construct an underground coal gangue target detection model based on an improved RT-DETR model architecture, and the underground coal gangue target detection model includes two backbone networks, an encoder, a query selection module, and a decoder;

[0009] Define the two backbone networks as backbone network one and backbone network two respectively. Backbone network one and backbone network two adopt a parallel structure and both use the ResNet50 network pre-trained on ImageNet;

[0010] The backbone network one is used to receive the input visible light image and perform multi-scale feature extraction on it; the backbone network two is used to receive the input infrared thermal imaging image and perform multi-scale feature extraction on it;

[0011] The encoder is used to receive the multi-scale output features of the backbone network one and the backbone network two, and perform cross-scale and cross-modal fusion on the input features of the two backbone networks, effectively encoding the correlation between the two modalities of visible light and infrared thermal imaging;

[0012] The output features of the encoder are the converted image feature sequence, and the query selection module selects a fixed number of image features from the image feature sequence output by the encoder as the initial object query for the decoder;

[0013] The decoder optimizes the object query and generates the prediction box and confidence score of the underground coal gangue image target;

[0014] Step 3. Based on the training dataset constructed in Step 1, train the underground coal gangue target detection model, and use the trained underground coal gangue target detection model to detect and identify the underground coal gangue target.

[0015] In addition, based on the above method for detecting underground coal gangue targets based on multi-source image fusion, the present invention also proposes a corresponding system for detecting underground coal gangue targets based on multi-source image fusion, which adopts the following technical solutions:

[0016] A system for detecting underground coal gangue targets based on multi-source image fusion includes a visible light camera, an infrared thermal imager, and a computer device. The computer device includes a memory and one or more processors;

[0017] The visible light camera and the infrared thermal imager are respectively used to collect images of mineral particles in the underground coal transportation link, obtain visible light images and infrared thermal imaging images, and send them to the computer device;

[0018] The memory stores executable code; when the processor executes the executable code, it is used to implement the steps of the method for detecting underground coal gangue targets based on multi-source image fusion as described above.

[0019] The present invention has the following advantages:

[0020] As described above, in view of the diversity and similarity of coal gangue in the traditional complex underground environment and the problem that traditional single sensors cannot effectively identify them, the present invention proposes a method for detecting underground coal gangue targets based on multi-source image fusion, and constructs an underground coal gangue target detection model based on an improved RT-DETR model architecture. Among them, in terms of model input, the present invention simultaneously collects visible light images and infrared thermal imaging images of mineral particles in the underground coal transportation link. Compared with the traditional single-sensor image acquisition method, this method can not only more effectively separate coal gangue in complex backgrounds, but also improve the comprehensive performance of the model by using the complementary visual information of visible light images and infrared images. In terms of the model architecture, the present invention makes the following improvements on the basis of the original RT-DETR model to obtain an improved RT-DETR model. Specifically, in view of the significant differences in the features of visible light and infrared images, and the problem that it is difficult to effectively utilize their complementary information by directly superimposing and inputting them into the same backbone network, in order to enhance the model's ability to express multi-modal features, a dual-modal branch (visible light and infrared thermal imaging modal branches) is introduced to improve the feature extraction effect. In view of the large difference in the scales of coal gangue minerals, it is difficult for a single-scale feature map to effectively capture targets of different scales. To enhance the model's detection ability, in the encoder, a self-attention module suitable for high-level semantic features, namely the Sparse Intra-scale Feature Interaction (SIFI) module, is designed to capture the associations between conceptual entities in the image. By designing a Cross-scale Feature Aggregation Module (CFAM), combining feature maps of different scales, the model's adaptability to large and small targets is enhanced, and the detection accuracy is improved. In view of the differences in the feature distributions of visible light and infrared images, and the problem that there may be information redundancy or deficiency in some regions for different modalities, the present invention also designs a feature fusion module (Spatial-Channel Aggregation Module, SCAM) suitable for cross-modal information interaction to achieve efficient complementarity and integration of features between different modalities. In addition, in view of the problems of redundant attention heads and global attention calculation in multi-head self-attention, which make it difficult to effectively capture the feature differences between different heads and interact with irrelevant feature noises, the present invention designs a novel cascaded sparse attention mechanism CSA for long-range dependence modeling. In the encoder, a Multi-level Feature map Aggregation (MFA) operation is used to integrate the multi-level spatial position features in the decoder to form stronger encoded features and better capture the diverse features of the target. Through the above improvements in model input and structure, the method of the present invention can effectively improve the recognition accuracy of underground coal gangue targets. Description of the Drawings

[0021] Figure 1 It is a flow chart of an underground coal gangue target detection method based on multi-source image fusion in an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of some data examples, data acquisition platform and infrared imager in the embodiments of the present invention; Figure 2 (a) is a visible light coal gangue sample, (b) is an infrared coal gangue sample, (c) is a data acquisition platform, and (d) is an infrared imager;

[0023] Figure 3 A network structure diagram of an underground coal gangue target detection model constructed in an embodiment of the present invention;

[0024] Figure 4 This is a network structure diagram of the SIFI module in an embodiment of the present invention;

[0025] Figure 5 A network structure diagram of a cascaded sparse attention module CSA in an embodiment of the present invention;

[0026] Figure 6 A network structure diagram of a hybrid expert model module in an embodiment of the present invention;

[0027] Figure 7 A network structure diagram of a SCAM module in an embodiment of the present invention;

[0028] Figure 8 A network structure diagram of a decoder in an embodiment of the present invention;

[0029] Figure 9 This is a schematic diagram of an example of detecting the true value of part of the data set in the experiment of the present invention;

[0030] Figure 10 This is a schematic diagram of a partial detection example of a data set of the GM-DETR model in the experiment of the present invention;

[0031] Figure 11 This is a schematic diagram of a partial detection example of a data set of the model in the experiment of the present invention;

[0032] Figure 9 , Figure 10 and Figure 11 (a), (b), (c), and (d) represent detection examples under clean surface conditions, a mixture of clean surface conditions and dust-adherent surface conditions, and dust-adherent surface conditions and wet surface conditions, respectively. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] Example 1

[0035] This Example 1 describes a method for detecting underground coal gangue targets based on multi-source image fusion to solve the problem that in the traditional complex underground environment, due to the diversity and similarity of coal and gangue, traditional single sensors cannot effectively identify them.

[0036] As Figure 1 shown, the method for detecting underground coal gangue targets based on multi-source image fusion includes the following steps:

[0037] Step 1. First, obtain visible light images and infrared thermal imaging images obtained by collecting images of mineral particles in the underground coal transportation link by a visible light camera and an infrared thermal imager.

[0038] Among them, the visible light image and the infrared thermal imaging image correspond one by one in time and space during collection and form corresponding picture pairs. After preprocessing and annotating the images, a training dataset is obtained.

[0039] During the mining, transportation, and processing of coal, coal usually appears in three surface states, namely clean surface, dust-adhered surface, and wet surface. For these three surface states, dust adhesion will cover up the texture and color differences between coal and gangue, and the wet surface makes them look similar under visible light due to light reflection and refraction. In response to this situation, the present invention collects visible light and infrared coal gangue images in multiple surface states. Some data instances, data collection platforms, and infrared imagers are as Figure 2 shown.

[0040] Annotate the images through the LabelImg tool and uniformly adjust the size of all images to 640×640 pixels.

[0041] Step 2. Build an underground coal gangue target detection model based on the improved RT-DETR model architecture. RT-DETR (Real-Time Detection, Embedding, and Tracking) is a real-time object detection model based on Transformer.

[0042] As Figure 3 shows the network structure diagram of the underground coal gangue target detection model. It can be seen from Figure 3 that the underground coal gangue target detection model includes two backbone networks, an encoder, a query selection module, and a decoder.

[0043] Define the two backbone networks as backbone network one and backbone network two respectively. Backbone network one and backbone network two adopt a parallel structure and both use the ResNet50 network pre-trained on ImageNet.

[0044] Two backbone networks are respectively used to construct independent multi-level features for each modality.

[0045] The first backbone network is used to receive the input visible light image and perform multi-scale feature extraction on it; the second backbone network is used to receive the input infrared thermal imaging image and perform multi-scale feature extraction on it.

[0046] The encoder is used to receive the multi-scale output features of the first and second backbone networks, and perform cross-scale and cross-modal fusion on the input features of the two backbone networks to effectively encode the correlation between the visible light and infrared thermal imaging modalities.

[0047] The output features of the encoder are the converted image feature sequence, and the query selection module selects a fixed number of image features from the image feature sequence output by the encoder as the initial object query for the decoder.

[0048] The decoder optimizes the object query and generates the prediction box and confidence score of the underground coal gangue image target.

[0049] In this embodiment, the improved RT-DETR model architecture is obtained by making the following improvements on the basis of the original RT-DETR model:

[0050] Introduce visible light and infrared thermal imaging modality branches to enhance the network's ability to extract multi-modal information in terms of structure.

[0051] In the encoder, design a self-attention module suitable for high-level semantic features, namely the Sparse Intra-scale Feature Interaction (SIFI) module, to capture the correlation between concept entities in the image.

[0052] Design a feature fusion module suitable for cross-modal information interaction, namely the Cross-scale Feature Aggregation Module (CFAM), to achieve efficient complementarity and integration of features between different modalities.

[0053] Design a feature aggregation module suitable for multi-modal fusion features to focus on significant target regions and texture details, namely the Spatial-Channel Aggregation Module (SCAM).

[0054] In order to reduce the model's computational complexity and resource consumption, use a more efficient attention mechanism in the network design, namely the Cascaded Sparse Attention (CSA).

[0055] In the encoder, a multi-level feature map aggregation (MFA) operation is used to aggregate global attention features to obtain stronger encoded features and better capture the diverse features of the target.

[0056] The following further details each module of the improved RT-DETR model architecture.

[0057] Backbone network one is used to receive the input visible light image, perform multi-scale feature extraction on it, and output the multi-scale features {S3 rgb , S4 rgb , S5 rgb} of the last three stages as the input to the encoder.

[0058] Meanwhile, backbone network two is used to receive the input infrared thermal imaging image, perform multi-scale feature extraction on it, and output the multi-scale features {S3 ir , S4 ir , S5 ir} of the last three stages as the input to the encoder.

[0059] As Figure 3 shown, the encoder includes two SIFI modules and one CFAM module. The two SIFI modules respectively perform sparse self-attention operations on the high-level features S5 rgb and S5 ir of visible light and infrared to obtain F5 rgb and F5 ir , enhancing the correlation between different targets within the same modality through the sparse attention mechanism and improving the detector's understanding of coal gangue features.

[0060] The CFAM module performs cross-modal and cross-scale complementary learning on visible light and infrared features to make up for feature missing or deviation in a single modality and enhance the discrimination ability of coal gangue. The SIFI module and the CFAM module work together. That is, SIFI first enhances the understanding of single-modal features, and then CFAM performs cross-modal fusion, making the target feature representation more complete and accurate.

[0061] The input features of the CFAM module are {F5 rgb , S4 rgb , S3 rgb} and {F5 ir , S4 ir , S3 ir} respectively, and finally obtain the encoder output X E .

[0062] As Figure 4As shown, the SIFI module, i.e., the Sparse Intra-scale Feature Interaction (SIFI) module, includes a Cascaded Sparse Attention (CSA) module, an RMS normalization module, a Mixture of Experts model module, and a residual connection; the processing flow of the SIFI module is as follows:

[0063] a1. Input the visible light high-level feature S5 output by the first backbone network rgb or the infrared high-level feature S5 output by the second backbone network ir into the SIFI module, project it to a fixed dimension through a linear transformation, and add a positional encoding.

[0064] Since applying self-attention operations to high-level features with richer semantic information can capture the associations between conceptual entities in the image, which helps subsequent modules detect and recognize image targets, SIFI only calculates on {S5 ir ,S5 rgb}.

[0065] a2. Use the Cascaded Sparse Attention (CSA) module to calculate the cascaded sparse attention of the features.

[0066] The calculation of attention prompts the model to highlight important infrared / visible light feature information through weighted summation, while suppressing irrelevant features. The multi-head mechanism allows the model to focus on the relationships in different feature spaces and enhances the model's ability to capture complex relationships.

[0067] In the actual environment, different surface states of coal gangue have a great impact on the performance of the target. Through the proposed CSA module in this embodiment, the model can be helped to focus on the most important coal gangue features in various backgrounds and ignore irrelevant interferences.

[0068] a3. The features output by the CSA module are subjected to RMS normalization processing through the RMS normalization module. Through RMS normalization processing, the training process can be stabilized and the generalization ability of the model can be improved.

[0069] a4. The output features of the RMS normalization module and the input features of the SIFI module are added through a residual connection to alleviate the gradient vanishing problem and accelerate convergence.

[0070] a5. The result after adding through the residual connection is input into the Mixture of Experts model module. The Mixture of Experts model module further extracts deeper features and independently performs a non-linear transformation on the feature vectors at each position to enhance the expression ability.

[0071] Since the characteristics of coal gangue may change under different backgrounds and states, a mixture of experts model can better extract and represent these changes, thereby improving the recognition accuracy and adaptability of the model.

[0072] a6. The features output by the mixture of experts model module are RMS-normalized by the RMS normalization module. Through RMS normalization, the training process can be stabilized and the generalization ability of the model can be improved.

[0073] a7. Add the output features passing through the RMS normalization module in step a6 and the input features of the mixture of experts model module in step a5 by residual connection to obtain the final output feature F5 ir or the output feature F5 rgb 。

[0074] In Figure 4 the feature S5 shown represents the feature S5 rgb or S5 ir ,the feature F5 represents F5 ir or F5 rgb 。

[0075] These features (referring to S5 rgb or S5 ir ) can help the model better identify coal gangue targets through the refined processing of the above multiple steps. The final output provides an accurate and rich feature representation for subsequent target recognition and classification.

[0076] In step a2, the processing flow of the cascaded sparse attention module CSA is as follows:

[0077] b1. Provide different segmentation blocks of the complete infrared / visible light features for each head, and explicitly decompose the attention similarity between the heads.

[0078] By assigning these features to different heads, the model can effectively capture the unique information of coal gangue under different surface states, thereby enhancing the adaptability to complex backgrounds and diverse surface states.

[0079] b2. Each head calculates sparse attention separately to reduce the computational complexity, and its calculation process is given below.

[0080] b3. To strengthen the information flow, the output of each head is passed layer by layer to the next head.

[0081] In coal gangue detection, this mechanism can ensure the mutual complementation and cooperation of features under different surface states. The output of each head provides richer context information for the subsequent heads, enabling the model to capture more comprehensive target features.

[0082] b4. Concatenate and map the attention maps of all heads to obtain the final result. By concatenating the attention maps of all heads, the model obtains the feature fusion result from different heads, which helps to generate more accurate detection results.

[0083] In step b2, the process of calculating the sparse attention by the j-th head is expressed as:

[0084] b21. After calculating the similarity between Q and K, unnecessary elements assigned with low attention weights are masked in the attention matrix M.

[0085] b22. Adaptively select the top k contributions with the highest scores on M and only keep the most important components.

[0086] b23. For the other elements in M that are less than k, use the Scatter function to replace their probabilities with 0 at the specified indices, making the calculation of attention change from dense to sparse.

[0087] b24. Use the Where function to replace the position with index 0 with -∞.

[0088] b25. The top k values within the interval [Δ1, Δ2] are normalized for each row in M, and Softmax is calculated. [Δ1, Δ2] is the threshold range for normalizing the attention matrix in the sparse attention calculation, where Δ1 is the minimum threshold and Δ2 is the maximum threshold.

[0089] b26. Multiply the Softmax result by V through matrix multiplication to obtain the attention map of this head.

[0090] As Figure 6 shown, the mixture-of-experts model module includes n expert networks, a routing network, and an expert fusion module; among them, the expert network adopts a feed-forward neural network FFN; the calculation process of the mixture-of-experts model module is as follows:

[0091] c1. Input the feature vector.

[0092] c2. The routing network dynamically selects the most relevant experts according to the input feature vector and assigns weights p to each expert.

[0093] Since coal gangue shows different characteristics under different surface states, and in this embodiment, the routing network can intelligently select the most relevant experts for the current task according to the input features.

[0094] c3. By setting a sparse policy, select the top k experts with the highest weights from n experts and mask the weights of the remaining experts.

[0095] c4. Each expert processes the input feature vectors independently; since each expert can learn specific tasks independently, they can focus on extracting specific features from different surface states.

[0096] c5. The outputs of all activated experts are fused according to the weight p to generate a unified representation. Combining the results of multiple experts provides a more comprehensive feature representation for the object detection task.

[0097] As Figure 7 shown in the network structure diagram of the Cross-scale Feature Aggregation Module (CFAM), this CFAM module adopts two branches: the top-down FPN and the bottom-up PAN.

[0098] Specifically, the CFAM module includes four SCAM modules, two upsampling modules (Upsample, U), and two downsampling (Downsample, D) modules, as Figure 7 shown.

[0099] The upsampling module consists of a 1×1 convolution with a stride of 1, a BN (batch normalization), and a SiLu; the downsampling module consists of a 3×3 convolution with a stride of 2, a BN, and a SiLu.

[0100] For convenience of representation, four SCAM modules are defined, namely SCAM1, SCAM2, SCAM3, and SCAM4.

[0101] As Figure 3 shown, the calculation process of the CFAM module is as follows:

[0102] d1. Processing flow of the FPN branch:

[0103] Input the upsampled feature F5 ir , as well as the features S4 rgb and S4 ir into SCAM1 to obtain the output feature F4. Input the upsampled feature F4, as well as the features S3 rgb and S3 ir into SCAM2 to obtain the output feature F3.

[0104] d2. Processing flow of the PAN branch:

[0105] Input the downsampled feature F3, as well as the features F4 and S4 rgb into SCAM3 to obtain the output feature P4. Input the downsampled P4, the feature F5 rgb and F5 irInput into SCAM4 to obtain the output feature P5.

[0106] d3. After flattening the features F3, P4, and P5 respectively, concatenate them together as the output X of the encoder. E . This unified feature representation integrates features from different levels, which helps to improve the detection accuracy of the model for minerals of different scales.

[0107] Multiple attention mechanisms are introduced in the SCAM module, enabling the network to focus more on significant target regions, while enhancing the parsing ability for coal gangue texture and edge details, and improving detection robustness.

[0108] Specifically, as Figure 7 shown, each SCAM module consists of three 1×1 convolutions, a channel feature aggregation module, and a spatial feature aggregation module. The SCAM module performs multi-level feature fusion, and its calculation process is as follows:

[0109] e1. The three input features of each SCAM module are stacked in the channel dimension. By stacking these features, the model can consider features from different modalities and different levels in a unified space simultaneously.

[0110] e2. Generate three branches through three 1×1 convolutional layers.

[0111] e3. Perform channel feature aggregation, spatial feature aggregation, and residual connection respectively. Channel feature aggregation focuses on feature combination in the channel dimension, spatial feature aggregation strengthens the feature expression at specific positions, and the output features are added and fused.

[0112] In this embodiment, the attention mechanism is used to strengthen the focus on coal gangue targets. Whether it is the heat distribution in the infrared image or the texture features in the visible light image, it helps to improve the sensitivity of the model to coal gangue targets.

[0113] The role of the query selection module is to dynamically filter high-quality queries related to the target, and at the same time introduce an IoU (Intersection over Union) prediction network to reduce query redundancy, improve computational efficiency, and target detection accuracy.

[0114] The processing flow of the query selection module is as follows:

[0115] f1. The encoder generates a fixed number of embedding vectors as initial queries.

[0116] f2. Use the IoU prediction network (using an mlp network as the IoU prediction network) to generate an IoU score for each query to measure the matching degree of the query with the target, which can help evaluate whether each query accurately matches the target coal gangue. High-matching queries help to improve the detection accuracy of coal gangue targets.

[0117] f3. Sort all the queries according to the IoU score.

[0118] f4. Select the top n queries with the highest scores and discard the rest to save computing resources.

[0119] Since coal gangue targets are usually located in complex backgrounds, a large number of irrelevant queries may be generated. By screening high-quality queries, the model can focus on the areas truly related to coal gangue targets, reduce the computational overhead, and improve the processing speed and efficiency.

[0120] f5. The selected n queries are used as the input to the decoder.

[0121] As Figure 8 shown, the core components of the decoder include the multi-head attention mechanism, the global attention mechanism, the feed-forward neural network, the residual connection, and layer normalization. To show the decoder more clearly, other modules are ignored.

[0122] The computational process of the decoder is as follows:

[0123] g1. Input a set of learnable features to capture the feature information of coal gangue on multiple underground surfaces. These features represent the preliminary information of the target to be detected, which specifically refers to the underground coal gangue target in this embodiment.

[0124] g2. Learn the feature representation through n decoder layers, and each layer performs the following steps:

[0125] g21. Calculate the self-attention using the attention module to model the relationships between object queries, enabling them to share information with each other and ensuring that different queries capture different targets. Through the self-attention mechanism, the decoder can accurately identify different parts of the coal gangue and ensure the coordination of each query, thereby enhancing the perception ability of the coal gangue target.

[0126] g22. Calculate the global attention to learn the matching relationship between object queries and the encoder output. Through the global attention, the decoder can utilize the global features extracted by the encoder in a complex environment to better understand the spatial relationship between the coal gangue and the background.

[0127] g23. Pass through the feed-forward neural network to perform a non-linear transformation on the embedding of each query, extract high-dimensional features, enhance the expression ability, and enable the model to extract deeper feature representations, enhancing the detection ability for diverse coal gangue targets.

[0128] g24. After each sub-module, namely the attention module and the feed-forward neural network, add the residual connection and normalization. The residual connection alleviates the problem of gradient disappearance and accelerates convergence; normalization stabilizes the training and improves the generalization performance of the model.

[0129] g3. After concatenating the global attention features of each decoder layer together, project them back to the original dimension through a 1×1 convolution to construct stronger encoded features, which helps to better capture the diverse features X of the target. D 。

[0130] g4. Input the decoder feature X D into the detection head, and finally form the prediction of the coal gangue object category and bounding box. The model can accurately predict the category of coal gangue and its precise location, thus providing effective information for subsequent sorting and processing.

[0131] Step 3. Based on the training dataset constructed in Step 1, train the underground coal gangue target detection model, and use the trained underground coal gangue target detection model to detect and identify the underground coal gangue targets.

[0132] The model training process is relatively conventional and will not be elaborated here. After the model training is completed, actual deployment is carried out.

[0133] After obtaining the visible light images and infrared thermal imaging images of the mineral particles in the underground coal transportation link collected by the visible light camera and the infrared thermal imager, first preprocess the images according to the preprocessing method in Step 1, and input them into the trained model. The model processes the input images according to the processing flow in Step 2, and finally obtains the coal gangue recognition result.

[0134] In addition, to verify the effectiveness of the method proposed in the present invention, the following experimental process is also given:

[0135] The experiment was carried out on the Windows 10 operating system, and the hardware environment included an NVIDIA GeForce RTX 3090 GPU (24GB video memory) and an Intel Xeon E5-2630 v3 @ 2.40GHz CPU.

[0136] The experiment was developed and trained using PyTorch 1.12.1 and CUDA11.3. The training parameters were set as follows: the number of epochs was 60, the batch size was 8, and the optimization algorithm used was AdamW. The method proposed in the present invention was compared with 11 state-of-the-art methods in terms of performance.

[0137] The 11 comparison methods used in this experiment were YOLO-Fusion, CAFF, ProbEn, CFT, CSAA, ICAFusion, MS-DETR, LRAF-Net, RSDet, CrossFormer, and GM-DETR respectively.

[0138] The results of the comparative experiments are shown in Table 1. Among them, in this experiment, the selected metric mAP@50 is used to measure whether the model can basically identify the target location, and the selected metric mAP@[0.5:0.95] is used to comprehensively evaluate the model performance, which not only measures the target recognition ability but also reflects the accuracy and robustness of the positioning.

[0139] Table 1 Results of Comparative Experiments

[0140]

[0141] It is not difficult to see from Table 1 above that the model proposed in the present invention has achieved leading performances of 95.1% and 66.1% in the two metrics of mAP@50 and mAP@[0.5:0.95]. The mAP@50 of MS-DETR reaches 94.9%, showing strong competitiveness, while models such as GM-DETR and CrossFormer still maintain a high mAP under high IoU thresholds. Using deep backbone networks (such as GM-DETR and LRAF-Net) to extract features performs better in terms of the mAP metric, which indicates the key role of deep networks in extracting visible light and infrared features and helps to improve the detection performance. In contrast, the mAP@[0.5:0.95] of ProbEn is only 54.6%, significantly lower than other models, reflecting the insufficient adaptability of its multimodal fusion strategy in complex scenarios. While MS-DETR, GM-DETR, and the proposed model make full use of multimodal information through more advanced feature fusion modules, thus achieving significant advantages in performance. The method of the present invention has the highest mAP@50, indicating its strong overall detection ability, being more stable under low IoU settings, and being able to identify targets more stably; it is second only to GM-DETR in mAP@[0.5:0.95], indicating that its detection accuracy at different IoU thresholds is also excellent and its generalization ability is strong. In addition, in some mature industrial inspection tasks (such as steel defect detection), an mAP@50 above 90% already belongs to a strong model.

[0142] Figure 9 、 Figure 10 and Figure 11 show the comparison of the object detection results of GM-DETR and the model proposed in the present invention on multiple dataset examples, which is a way to more intuitively reflect the model performance. In Figure 10 and Figure 11 , the green inverted triangles indicate that the corresponding model did not detect coal at that location. The red slashed boxes indicate the missing parts of the predicted bounding boxes compared to the true bounding boxes. In Figure 10 (a) and (d), GM-DETR shows the phenomenon of bounding box offset; in Figure 10In (b) and (c) among them, GM-DETR has the problem of partial target missed detection. In contrast, the method proposed in the present invention is more robust in dealing with complex backgrounds, illumination changes, and multi-target interference. Its bounding boxes are more closely fitted to the true bounding boxes, and the missed detected targets are significantly reduced. Especially in the multi-modal fusion scenario of (d), the recognition of coal and gangue is more accurate. Generally speaking, the method proposed in the present invention significantly improves the detection accuracy and the bounding box regression ability through the improved multi-modal feature interaction mechanism and cross-scale fusion strategy.

[0143] Embodiment 2

[0144] This Embodiment 2 describes an underground coal and gangue target detection system based on multi-source image fusion, including a visible light camera, an infrared thermal imager, and a computer device. The computer device includes a memory and one or more processors.

[0145] The visible light camera and the infrared thermal imager are respectively connected to the computer device, and are used to collect images of mineral particles in the underground coal transportation link to obtain visible light images and infrared thermal images, and send them to the computer device.

[0146] The memory stores executable code; when the processor executes the executable code, it is used to implement the steps of the underground coal and gangue target detection method based on multi-source image fusion as described in Embodiment 1 above.

[0147] Of course, the above description is only the preferred embodiment of the present invention. The present invention is not limited to listing the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any person skilled in the art under the teaching of this specification fall within the substantial scope of this specification and should be protected by the present invention.

Claims

1. A method for detecting underground coal and gangue targets based on multi-source image fusion, characterized in that, It includes the following steps: Step 1. First, obtain the visible light image and the infrared thermal imaging image obtained by image acquisition of mineral particles in the underground coal transportation link by a visible light camera and an infrared thermal imager; Among them, the visible light image and the infrared thermal imaging image are in one-to-one correspondence in time and space during acquisition and form corresponding picture pairs; after preprocessing and annotation of the acquired images, a training dataset is obtained; Step 2. Construct an underground coal gangue target detection model based on the improved RT-DETR model architecture. This underground coal gangue target detection model includes two backbone networks, an encoder, a query selection module, and a decoder; Define the two backbone networks as backbone network one and backbone network two respectively. Backbone network one and backbone network two adopt a parallel structure and both use the ResNet50 network pre-trained on ImageNet; Backbone network one is used to receive the input visible light image and perform multi-scale feature extraction on it; backbone network two is used to receive the input infrared thermal imaging image and perform multi-scale feature extraction on it; The encoder is used to receive the multi-scale output features of backbone network one and backbone network two, and perform cross-scale and cross-modal fusion on the input features of the two backbone networks to effectively encode the correlation between the two modalities of visible light and infrared thermal imaging; The output feature of the encoder is a converted image feature sequence. The query selection module selects a fixed number of image features from the image feature sequence output by the encoder as the initial object query for the decoder; The decoder optimizes the object query and generates the prediction box and confidence score of the underground coal gangue image target; Step 3. Based on the training dataset constructed in Step 1, train the underground coal gangue target detection model, and use the trained underground coal gangue target detection model to detect and identify the underground coal gangue target.

2. The method for detecting underground coal gangue targets based on multi-source image fusion according to claim 1, wherein, In step 2, backbone network 1 is used to receive the input visible light image, perform multi-scale feature extraction on it, and output the multi-scale features of the last three stages as the input of the encoder; Meanwhile, the second backbone network is used to receive the input infrared thermal imaging image, perform multi-scale feature extraction on it, and output the multi-scale features of the last three stages {S3 ir , S4 ir , S5 ir}, which are used as the input of the encoder.

3. The method for detecting underground coal gangue targets based on multi-source image fusion according to claim 2, wherein, The encoder includes two SIFI modules and one CFAM module; An SIFI module for the advanced feature S5 of visible light rgb Performs sparse self-attention operation to obtain feature F5 rgb ; An SIFI module for the advanced feature S5 of infrared ir Performs sparse self-attention operation to obtain feature F5 ir ; The CFAM module performs cross-modal and cross-scale complementary learning on visible light and infrared features. Among them, the input features of the CFAM module are {F5 rgb , S4 rgb , S3 rgb} and {F5 ir , S4 ir , S3 ir} respectively, and finally the encoder output X E is obtained.

4. The method for detecting underground coal gangue targets based on multi-source image fusion according to claim 3, wherein, The SIFI module includes a cascaded sparse attention module, an RMS normalization module, a mixture of experts model module, and a residual connection; among them, the SIFI module processes the high-level features S5 rgb , S5 ir as follows: a1. Input the visible light high-level feature S5 output by the first backbone network rgb or the infrared high-level feature S5 output by the second backbone network ir into the SIFI module, project it to a fixed dimension through linear transformation, and add positional encoding; a2. Calculate the cascaded sparse attention of the features using the cascaded sparse attention module; a3. The features output by the cascaded sparse attention module are subjected to RMS normalization processing by the RMS normalization module; a4. The output features of the RMS normalization module and the input features of the SIFI module are added together through a residual connection; a5. The result after adding through the residual connection is input into the mixture of experts model module, and the mixture of experts model module further extracts deeper features and independently performs non-linear transformation on the feature vectors at each position to enhance the expression ability; a6. The features output by the mixture of experts model module are subjected to RMS normalization processing by the RMS normalization module; a7. Add the output features passing through the RMS normalization module in step a6 and the input features of the mixture of experts model module in step a5 through residual connection to obtain the final output feature F5 ir or the output feature F5 rgb .

5. The method for detecting underground coal gangue targets based on multi-source image fusion according to claim 4, wherein, The processing flow of the cascaded sparse attention module is as follows: b1. Provide different segmented blocks with complete infrared / visible light features for each head, explicitly decomposing the attention similarity between heads; b2. Each head calculates sparse attention separately to reduce the computational complexity; b3. To strengthen information flow, the output of each head is passed layer by layer to the next head; b4. Concatenate and map the attention maps of all heads to obtain the final result; Among them, in step b2, the process of calculating sparse attention for the j-th head is expressed as: b21. After calculating the similarity between Q and K, mask the unnecessary elements with low attention weights in the attention matrix M; b22. Adaptively select the top k scores with the highest contributions on the attention matrix M, and only retain the most important components; b23. For other elements in the attention matrix M that are less than k, use the Scatter function to replace their probabilities with 0 at the specified indices, changing the calculation of attention from dense to sparse; b24. Use the Where function to replace the position with index 0 with -∞; b25. The top k values within the interval [Δ1, Δ2] are normalized for each row in M, and Softmax is calculated; [Δ1, Δ2] is the threshold range for normalizing the attention matrix in the calculation of sparse attention, Δ1 is the minimum threshold, and Δ2 is the maximum threshold; b26. Multiply the Softmax result by V through matrix multiplication to obtain the attention map of this head.

6. The underground coal gangue target detection method based on multi-source image fusion according to claim 4, characterized in that The hybrid expert model module includes n expert networks, a routing network, and an expert fusion module; among them, the expert network adopts a feed-forward neural network FFN; the calculation process of the hybrid expert model module is as follows: c1. Input feature vector; c2. The routing network dynamically selects the most relevant experts based on the input feature vector and assigns weights p to each expert; c3. By setting a sparse strategy, select the k experts with the highest weights from n experts and mask the weights of the remaining experts; c4. Each expert independently processes the input feature vector; c5. The outputs of all activated experts are fused according to the weight p to generate a unified representation.

7. The underground coal gangue target detection method based on multi-source image fusion according to claim 3, characterized in that The CFAM module adopts two branches: a top-down FPN and a bottom-up PAN; Specifically, the CFAM module includes four SCAM modules, two upsampling modules, and two downsampling modules; define four SCAM modules, namely SCAM1, SCAM2, SCAM3, and SCAM4; the calculation process of CFAM is as follows: d1. Processing flow of the FPN branch: The upsampled feature F5 ir , and the feature S4 rgb and the feature S4 ir are input into SCAM1 to obtain the output feature F4; the upsampled feature F4, and the feature S3 rgb and the feature S3 ir are input into SCAM2 to obtain the output feature F3; d2. Processing flow of the PAN branch: The downsampled feature F3, as well as features F4 and S4 rgb are input into SCAM3 to obtain the output feature P4; the downsampled P4, feature F5 rgb and F5 ir are input into SCAM4 to obtain the output feature P5; d3. Flatten features F3, P4, and P5 respectively and then Concat them together as the output X of the encoder E ; Each SCAM module consists of three 1×1 convolutions, a channel feature aggregation module, and a spatial feature aggregation module. The SCAM module fuses multi-level features, and its calculation process is: e1. Stack the three input features of each SCAM module in the channel dimension. By stacking these features, the model can simultaneously consider features from different modalities and different levels in a unified space; e2. Generate three branches through three 1×1 convolutional layers; e3. Respectively perform channel feature aggregation, spatial feature aggregation and residual connection, and add and fuse the output features.

8. The method for detecting underground coal gangue targets based on multi-source image fusion according to claim 1, wherein The processing flow of the query selection module is as follows: f1. The encoder generates a fixed number of embedding vectors as the initial query; f2. Use the IoU prediction network to generate an IoU score for each query to measure the matching degree between the query and the target; f3. Score and sort all queries according to the IoU score; f4. Select the n queries with the highest scores and discard the remaining queries to save computing resources; f5. The selected n queries serve as the decoder input.

9. The method for detecting underground coal gangue targets based on multi-source image fusion according to claim 1, wherein The decoder includes a self-attention mechanism, a global attention mechanism, a feed-forward neural network, a residual connection and layer normalization. The calculation process of the decoder is as follows: g1. Input a set of learnable features to capture the feature information of underground multi-surface state coal gangue; g2. Learn the feature representation through n decoder layers, and each layer performs the following steps: g21. The self-attention mechanism calculates self-attention to model the relationship between object queries; g22. The global attention mechanism calculates global attention to learn the matching relationship between object queries and the encoder output; g23. Through the feed-forward neural network, perform a non-linear transformation on the embedding of each query; g24. After each sub-module, namely the attention module and the feed-forward neural network, add a residual connection and layer normalization; g3. After concatenating the global attention features of each decoder layer, project them back to the original dimension through a 1×1 convolution to construct stronger encoded features, so as to better capture the diverse features of the target, that is, the decoder feature X D ; g4. Input the decoder feature X D into the detection head, and finally form the prediction of coal gangue object category and bounding box.

10. An underground coal gangue target detection system based on multi-source image fusion, including a visible light camera, an infrared thermal imager and a computer device. The computer device includes a memory and one or more processors; The visible light camera and the infrared thermal imager are respectively used to collect images of mineral particles in the underground coal transportation link to obtain visible light images and infrared thermal imaging images, and send them to the computer device; The memory stores executable code; characterized in that, When the processor executes the executable code, it is used to implement the steps of the method for detecting underground coal gangue targets based on multi-source image fusion according to any one of claims 1 to 9.

Images