Underground abnormal event detection method based on multispectral image fusion

Abstract

The invention discloses an underground abnormal event detection method based on multispectral image fusion, which comprises the following steps of: firstly, acquiring a plurality of groups of multispectral underground images by adopting multispectral imaging equipment, and preprocessing the images, each group of multispectral underground images comprising an infrared image and a visible light image; then, a multispectral image detection model is constructed, the multispectral image detection model comprises a feature extraction module and a feature fusion module, finally, the multispectral image detection model is trained, and a loss function is set as a weighted MSE loss function. And performing underground abnormal event detection on each group of multispectral underground images by adopting the trained multispectral image detection model to obtain an underground abnormal event detection result. According to the method, the underground image data of different spectrum sections are fused, the accuracy and reliability of underground abnormal event detection are improved, and the method can be focused on an area containing key abnormal information, so that the feature capturing capability of the area is enhanced, and the accuracy of abnormal event detection is remarkably improved.

Description

Technical Field

[0001] The invention relates to the technical field of image data processing, and in particular to a method for detecting underground abnormal events based on multi-spectral image fusion. Background Art

[0002] In mining production, underground safety is the primary consideration. However, due to the complex underground environment and poor lighting conditions, there are many potential safety hazards, such as fire and gas leakage. Once these abnormal events occur, they may have a serious impact on mining production and personnel safety.

[0003] Traditional video surveillance systems mainly rely on image data from a single spectral band, which makes it difficult to adapt to the complex and changing environmental conditions underground. In the case of insufficient light or occlusion, traditional video surveillance methods often have difficulty effectively identifying abnormal events. Traditional image processing methods are easily affected by factors such as noise and light changes when processing underground images, resulting in reduced detection accuracy. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for detecting underground abnormal events based on multispectral image fusion. By fusing underground image data of different spectral segments, the accuracy and reliability of underground abnormal event detection are improved, and it is possible to focus on areas containing key abnormal information, thereby enhancing its feature capture capability and significantly improving the accuracy of abnormal event detection.

[0005] The technical solution of the present invention is: A method for detecting underground abnormal events based on multispectral image fusion specifically includes the following steps: (1) Using a multispectral imaging device to collect multiple groups of multispectral downhole images and perform image preprocessing to construct a downhole image data set, where each group of multispectral downhole images includes an infrared image and a visible light image; (2) Construct a multispectral image detection model. The multispectral image detection model includes a feature extraction module and a feature fusion module. The feature extraction module extracts features from infrared images and visible light images respectively, and the feature fusion module fuses the extracted infrared image features and visible light image features. (3) The multispectral image detection model is trained using the downhole image dataset. The training loss function is the weighted MSE loss function. , weighted MSE loss function The calculation formula is shown in the following formula (1): (1); In formula (1), , , They represent the height, width and number of channels of multispectral downhole images respectively; , , Respectively represent the index of each pixel in the multispectral downhole image in terms of height, width and number of channels, and are used to traverse the position and channel of each pixel in the spectral downhole image; Indicates the true value of the multispectral downhole image The pixel value of the position is the target of model learning; Indicates the predicted value of the multispectral image detection model in The pixel value of the position; is the weight decay regularization parameter; is the weight of each pixel position; (4) Use the trained multispectral image detection model to detect underground abnormal events in each group of multispectral underground images to obtain the underground abnormal event detection results.

[0006] The image preprocessing includes image denoising and image enhancement. For the smooth area in the multispectral downhole image, a denoising algorithm based on Gaussian filtering is used for image denoising; for the area containing edge and detail information, an edge-preserving filtering algorithm is used for image denoising; the image enhancement adopts a contrast-limited adaptive histogram equalization method to enhance the local contrast of the image.

[0007] The feature extraction module is a twin network, which includes a visible light image feature extraction subnetwork and an infrared image feature extraction subnetwork with the same network architecture. The input of the visible light image feature extraction subnetwork is an image obtained by splicing the denoised visible light image and its enhanced image in the channel dimension, and the input of the infrared image feature extraction subnetwork is an image obtained by splicing the denoised infrared image and its enhanced image in the channel dimension. The visible light image feature extraction subnetwork and the infrared image feature extraction subnetwork both include 3×3 convolution, depth convolution, two ReLU functions, attention mechanism, residual module and fully connected layer. The processing process is specifically shown in the following formula (2): (2); In formula (2), represents 3×3 convolution, represents depthwise convolution, represents the ReLU function, represents the attention mechanism, represents the residual module, represents the fully connected layer, is the input of the visible light image feature extraction subnetwork or the infrared image feature extraction subnetwork, is the feature map output after deep convolution, It is the output of the visible light image feature extraction subnetwork or the infrared image feature extraction subnetwork.

[0008] The attention mechanism includes a channel attention mechanism and a spatial attention mechanism. The channel attention mechanism first performs global average pooling and global maximum pooling operations on the input feature map, and then fuses and splices the vectors after global average pooling and global maximum pooling together and compresses the dimension through the fully connected layer, and then uses the Sigmoid function to normalize. The output of the channel attention mechanism is used as the input of the spatial attention mechanism. The spatial attention mechanism first performs global average pooling and global maximum pooling operations on the input, and then fuses and splices the two spatial maps output after global average pooling and global maximum pooling in the channel dimension, and then uses 1×1 convolution and Sigmoid function to compress and normalize the feature map, and outputs the attention weight map.

[0009] The feature fusion module first performs feature splicing on the feature maps output by the visible light image feature extraction subnetwork and the infrared image feature extraction subnetwork with the same network architecture, and then unifies the spliced ​​features into the same feature space through the first 3×3 convolution, and then respectively uses convolutions with different receptive field sizes to extract features from the output of the first 3×3 convolution, and finally uses the second 3×3 convolution to perform feature fusion on the features extracted by the convolutions with different receptive field sizes.

[0010] The weight of each pixel position is It is calculated by the following formula (3): (3); In formula (3), Represents the pixel value within the window; Represents the mean value of pixels in the window; Represents the local variance calculated within the window; Represents the number of pixels in the window; Represents the coordinate point within the window; represents the gradient magnitude calculated using the Sobel operator; and They are the gradients in the x and y directions obtained by convolution of the Sobel operator; is a hyperparameter that balances the contribution of local variance and gradient amplitude; It is the index of the multispectral downhole image in height, width and number of channels, which is used to traverse the whole image of the multispectral downhole image to determine the global maximum of the local variance and gradient amplitude, and provide a global reference for the current pixel weight calculation; Indicated in Take the maximum value within the value range; Indicates that The local variance calculated within a small window centered at the position; Represents the use of the Sobel operator to The gradient magnitude calculated for the image region centered at position.

[0011] During the training process of the multispectral image detection model, the multispectral image detection model is first trained once, and then the twin network is randomly pruned multiple times, and the multispectral image detection model composed of the twin network with the best pruning effect is selected for re-training.

[0012] Advantages of the present invention: (1) By fusing image data from two different spectral bands, visible light and infrared, the present invention can obtain more scene information and improve the accuracy and reliability of abnormal event detection. Multispectral image fusion technology can overcome the limitations of single spectral image data and improve the signal-to-noise ratio and contrast of the image. In the case of insufficient lighting or occlusion underground, the fused image can provide more feature information for the model.

[0013] (2) The present invention uses an improved weighted MSE loss function to improve detection performance. The improved weighted MSE loss function consists of two parts: adaptive weighting based on data features and weight decay regularization. In the detection of abnormal events in multispectral downhole images, the adaptive weighting part determines the weight by calculating the local variance and gradient amplitude, and assigns higher weights to areas with rich textures or significant gradient changes, which can guide the model to focus on areas containing key abnormal information, thereby enhancing its feature capture ability and significantly improving the accuracy of abnormal event detection; while the weight decay regularization aims to prevent overfitting. It penalizes the sum of squares of weights, constrains weight growth, and ensures the stability and generalization performance of the model; the two are organically combined to achieve the strengthening of local key information and the effective control of global weights; at the same time, its parameters have high flexibility and can be adjusted according to the downhole scene and the type of abnormality, so that the loss function has good versatility and extensibility, providing a strong guarantee for the effective detection of downhole abnormal events.

[0014] (3) The image enhancement of the present invention adopts the contrast-limited adaptive histogram equalization method CLAHE to enhance the local contrast of the image. Compared with the traditional global histogram equalization, CLAHE can adaptively enhance the local contrast according to the brightness information of the local area. While enhancing the details, it effectively avoids the noise amplification problem caused by excessive enhancement. It is particularly suitable for detail enhancement of low-contrast images. Finally, the image is enhanced by the local stitching method.

[0015] (4) During the training process of the multispectral image detection model, the present invention randomly prunes the model, so that the lightweight multispectral image detection model has a faster reasoning and detection speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention.

[0017] Figure 2 It is an architecture diagram of the multispectral image detection model of the present invention.

[0018] Figure 3 It is an architecture diagram of the twin network of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] See Figure 1-Figure 3 A method for detecting underground abnormal events based on multispectral image fusion specifically includes the following steps: (1) Using a multispectral imaging device to collect multiple groups of multispectral downhole images and perform image preprocessing to construct a downhole image data set, where each group of multispectral downhole images includes an infrared image and a visible light image; Image preprocessing includes image denoising and image enhancement. For the smooth area in the multispectral downhole image, a denoising algorithm based on Gaussian filtering is used for image denoising, which can effectively remove noise and maintain the uniformity of the smooth area; for the area containing edge and detail information, an edge-preserving filtering algorithm is used for image denoising, so as to maximize the retention of edge and texture features while removing noise, and avoid the loss of important details; image enhancement is to use the contrast-limited adaptive histogram equalization method CLAHE to enhance the local contrast of the image. CLAHE can adaptively enhance according to the brightness information of the local area, and effectively avoid the noise amplification problem caused by over-enhancement while enhancing the details. It is particularly suitable for detail enhancement of low-contrast images. Finally, the image is enhanced by a local splicing method; the image enhancement of the present invention is specifically to evenly crop the infrared image and the matching visible light image into four parts, and cross-splice them with each other; (2) Construct a multispectral image detection model. The multispectral image detection model includes a feature extraction module and a feature fusion module. The feature extraction module extracts features from infrared images and visible light images respectively, and the feature fusion module fuses the extracted infrared image features and visible light image features. S21, the feature extraction module is a twin network, which includes a visible light image feature extraction subnetwork and an infrared image feature extraction subnetwork with the same network architecture, the input of the visible light image feature extraction subnetwork is an image obtained by splicing the denoised visible light image and its enhanced image in the channel dimension, and the input of the infrared image feature extraction subnetwork is an image obtained by splicing the denoised infrared image and its enhanced image in the channel dimension; Both the visible light image feature extraction subnetwork and the infrared image feature extraction subnetwork include 3×3 convolution, depthwise convolution, two ReLU functions, attention mechanism, residual module and fully connected layer. The specific processing process is shown in the following formula (2): (2); In formula (2), represents 3×3 convolution, represents depthwise convolution, represents the ReLU function, represents the attention mechanism, represents the residual module, represents the fully connected layer, is the input of the visible light image feature extraction subnetwork or the infrared image feature extraction subnetwork, is the feature map output after deep convolution, is the output of the visible light image feature extraction subnetwork or the infrared image feature extraction subnetwork; That is, the input feature map first uses 3×3 convolution to initially extract local features of the image, such as edge, texture and other information; then deep convolution is used to further extract channel-specific feature information, which can enhance the network's expression ability without increasing the computational complexity while reducing the amount of calculation. Then, the ReLU function is applied to the output feature map of the deep convolution to enhance the network's nonlinear expression ability, and then the attention mechanism is entered; The attention mechanism includes a channel attention mechanism and a spatial attention mechanism. The channel attention mechanism first performs global average pooling AvgPool and global maximum pooling MaxPool operations on the input feature map, then fuses and splices the vectors after global average pooling and global maximum pooling together and compresses the dimension through the fully connected layer, and then uses the Sigmoid function for normalization to highlight the key channel information and improve the effectiveness and pertinence of feature representation; the output of the channel attention mechanism is used as the input of the spatial attention mechanism. The spatial attention mechanism first performs global average pooling and global maximum pooling operations on the input, then fuses and splices the two spatial maps output after global average pooling and global maximum pooling in the channel dimension, and then uses 1×1 convolution and Sigmoid function to compress and normalize the feature map, and outputs the attention weight map; Then the attention weight map is residually connected with the feature map processed by deep convolution (DConv) to obtain a new feature map, which ensures that information can be directly transferred from the shallower layer of the network to the deeper layer, maintains the effective flow of information, and avoids the performance degradation caused by the increase in depth. Then the new feature map is passed to the fully connected layer to fuse the extracted features and fuse different feature information. Finally, the ReLU function is applied to the output of the fully connected layer again for nonlinear activation to obtain the final output; S22, the feature fusion module first performs feature splicing on the feature maps output by the visible light image feature extraction subnetwork and the infrared image feature extraction subnetwork with the same network architecture, and then unifies the spliced ​​features into the same feature space through the first 3×3 convolution, and then uses convolutions with different receptive field sizes to extract features from the output of the first 3×3 convolution respectively, and finally uses the second 3×3 convolution to perform feature fusion on the features extracted by the convolutions with receptive field sizes of 3×3, 5×5, and 7×7 respectively; (3) Use the downhole image dataset to train and test the multispectral image detection model. During the training process, the multispectral image detection model is first trained once, and then the twin network is randomly pruned ten times. The multispectral image detection model composed of the twin network with the best pruning effect is selected for retraining; The loss function of training is the weighted MSE loss function , weighted MSE loss function The calculation formula is shown in the following formula (1): (1); In formula (1), , , They represent the height, width and number of channels of multispectral downhole images respectively; , , Respectively represent the index of each pixel in the multispectral downhole image in terms of height, width and number of channels, and are used to traverse the position and channel of each pixel in the spectral downhole image; Indicates the true value of the multispectral downhole image The pixel value of the position is the target of model learning; Indicates the predicted value of the multispectral image detection model in The pixel value of the position; is the weight decay regularization parameter; is the weight of each pixel position, which is calculated by the following formula (3): (3); In formula (3), Represents the pixel value within the window; Represents the mean value of pixels in the window; Represents the local variance calculated within the window; Represents the number of pixels in the window; Represents the coordinate point within the window; represents the gradient magnitude calculated using the Sobel operator; and They are the gradients in the x and y directions obtained by convolution of the Sobel operator; is a hyperparameter that balances the contribution of local variance and gradient amplitude; It is the index of the multispectral downhole image in height, width and number of channels, which is used to traverse the whole image of the multispectral downhole image to determine the global maximum of the local variance and gradient amplitude, provide a global reference for the current pixel weight calculation, and realize the comparison and normalization of local features and global features; Indicated in Take the maximum value within the value range; Indicates that The local variance calculated within a small window centered at the position; Represents the use of the Sobel operator to The gradient magnitude calculated from the image region centered at position; (4) Use the trained multispectral image detection model to detect underground abnormal events in each group of multispectral underground images to obtain the underground abnormal event detection results.

[0021] Performance Analysis: Table 1 below shows the infrared and visible light datasets for pedestrian detection collected underground. The dataset contains 1,000 infrared images and 1,000 visible light images. The experiment uses 80% of the data as a training set and 20% of the data as a test set. The experimental results in Table 1 show that the model (Ours) proposed in this embodiment leads other existing models (YOLOv10n and RT-DETR) with an accuracy of 0.954, indicating that it performs best in correctly identifying positive samples.

[0022] Table 1

[0023] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting underground abnormal events based on multispectral image fusion, characterized in that: The specific steps include: (1) Using a multispectral imaging device to collect multiple groups of multispectral downhole images and perform image preprocessing to construct a downhole image data set, where each group of multispectral downhole images includes an infrared image and a visible light image; (2) Construct a multispectral image detection model. The multispectral image detection model includes a feature extraction module and a feature fusion module. The feature extraction module extracts features from infrared images and visible light images respectively, and the feature fusion module fuses the extracted infrared image features and visible light image features. (3) The multispectral image detection model is trained using the downhole image dataset, and the training loss function is the weighted MSE loss function; (4) Use the trained multispectral image detection model to detect underground abnormal events in each group of multispectral underground images to obtain the underground abnormal event detection results.

2. The method for detecting underground abnormal events based on multispectral image fusion according to claim 1, characterized in that: The weighted MSE loss function The calculation formula is shown in the following formula (1): （1）； In formula (1), , , They represent the height, width and number of channels of multispectral downhole images respectively; , , Respectively represent the index of each pixel in the multispectral downhole image in terms of height, width and number of channels, and are used to traverse the position and channel of each pixel in the spectral downhole image; Indicates the true value of the multispectral downhole image The pixel value of the position is the target of model learning; Indicates the predicted value of the multispectral image detection model in The pixel value of the position; is the weight decay regularization parameter; is the weight of each pixel position.

3. The method for detecting underground abnormal events based on multispectral image fusion according to claim 2, characterized in that: The image preprocessing includes image denoising and image enhancement. For the smooth area in the multispectral downhole image, a denoising algorithm based on Gaussian filtering is used for image denoising; for the area containing edge and detail information, an edge-preserving filtering algorithm is used for image denoising; the image enhancement adopts a contrast-limited adaptive histogram equalization method to enhance the local contrast of the image.

4. The method for detecting underground abnormal events based on multispectral image fusion according to claim 3, characterized in that: The feature extraction module is a twin network, which includes a visible light image feature extraction subnetwork and an infrared image feature extraction subnetwork with the same network architecture. The input of the visible light image feature extraction subnetwork is an image obtained by splicing the denoised visible light image and its enhanced image in the channel dimension, and the input of the infrared image feature extraction subnetwork is an image obtained by splicing the denoised infrared image and its enhanced image in the channel dimension. The visible light image feature extraction subnetwork and the infrared image feature extraction subnetwork both include 3×3 convolution, depth convolution, two ReLU functions, attention mechanism, residual module and fully connected layer. The processing process is specifically shown in the following formula (2): （2）； In formula (2), represents 3×3 convolution, represents the depthwise convolution, represents the ReLU function, represents the attention mechanism, represents the residual module, represents the fully connected layer, is the input of the visible light image feature extraction subnetwork or the infrared image feature extraction subnetwork, is the feature map output after deep convolution, It is the output of the visible light image feature extraction subnetwork or the infrared image feature extraction subnetwork.

5. The method for detecting underground abnormal events based on multispectral image fusion according to claim 4, characterized in that: The attention mechanism includes a channel attention mechanism and a spatial attention mechanism. The channel attention mechanism first performs global average pooling and global maximum pooling operations on the input feature map, and then fuses and splices the vectors after global average pooling and global maximum pooling together and compresses the dimension through the fully connected layer, and then uses the Sigmoid function to normalize. The output of the channel attention mechanism is used as the input of the spatial attention mechanism. The spatial attention mechanism first performs global average pooling and global maximum pooling operations on the input, and then fuses and splices the two spatial maps output after global average pooling and global maximum pooling in the channel dimension, and then uses 1×1 convolution and Sigmoid function to compress and normalize the feature map, and outputs the attention weight map.

6. The method for detecting underground abnormal events based on multispectral image fusion according to claim 4, characterized in that: The feature fusion module first performs feature splicing on the feature maps output by the visible light image feature extraction subnetwork and the infrared image feature extraction subnetwork with the same network architecture, and then unifies the spliced ​​features into the same feature space through the first 3×3 convolution, and then respectively uses convolutions with different receptive field sizes to extract features from the output of the first 3×3 convolution, and finally uses the second 3×3 convolution to perform feature fusion on the features extracted by the convolutions with different receptive field sizes.

7. The method for detecting underground abnormal events based on multispectral image fusion according to claim 2, characterized in that: The weight of each pixel position is It is calculated by the following formula (3): （3）； In formula (3), Represents the pixel value within the window; Represents the mean value of pixels in the window; Represents the local variance calculated within the window; Represents the number of pixels in the window; Represents the coordinate point within the window; represents the gradient magnitude calculated using the Sobel operator; and They are the gradients in the x and y directions obtained by convolution of the Sobel operator; is a hyperparameter that balances the contribution of local variance and gradient amplitude; It is the index of the multispectral downhole image in height, width and number of channels, which is used to traverse the whole image of the multispectral downhole image to determine the global maximum of the local variance and gradient amplitude, and provide a global reference for the current pixel weight calculation; Indicated in Take the maximum value within the value range; Indicates that The local variance calculated within a small window centered at the position; Represents the use of the Sobel operator to The gradient magnitude calculated for the image region centered at position.

8. The method for detecting underground abnormal events based on multispectral image fusion according to claim 4, characterized in that: During the training process of the multispectral image detection model, the multispectral image detection model is first trained once, and then the twin network is randomly pruned multiple times, and the multispectral image detection model composed of the twin network with the best pruning effect is selected for re-training.

Images