Multi-attention mechanism UNet + + network remote sensing image strip mine area change detection method
By using the improved multi-attention mechanism UNet++ network, the problems of multi-scale and background change interference in the detection of changes in open-pit mines by traditional methods are solved, and more efficient and accurate detection of land use and illegal mining activities is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 甘肃省矿业权管理服务中心
- Filing Date
- 2023-12-06
- Publication Date
- 2026-04-24
AI Technical Summary
Traditional methods are difficult to effectively handle the diverse shapes of land features in open-pit mines and the seasonal changes and shadow interference in multi-time and multi-source remote sensing images, resulting in inaccurate change detection results.
We employ the UNet++ network with a multi-attention mechanism. By introducing temporal, spatial, and channel attention mechanisms into UNet++ and combining them with a focus loss function, we improve the feature extraction and fusion process, thereby enhancing the adaptability to multi-scale ground object shapes and the detection accuracy.
It improves the accuracy and efficiency of change detection in open-pit mines, reduces manpower and material resources, adapts to the multi-scale and diverse terrain shapes in complex open-pit mining environments, and reduces interference from background changes.
Smart Images
Figure CN121921635A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of change detection based on remote sensing images. It relates to a method for accurately detecting land use changes and illegal mining activities by making full use of the multi-resolution information of images and expanding the receptive field during feature extraction through an improved deep learning network structure. Specifically, it is a method for detecting changes in open-pit mines in remote sensing images using a multi-attention mechanism UNet++ network. Background Technology
[0002] Mineral resources have always played an indispensable role in my country's national economic development. However, traditional regulatory methods face numerous challenges in open-pit mines. The main difficulty lies in achieving comprehensive supervision and the ease with which illegal activities such as mining beyond permitted boundaries, illegal land occupation, and damage to the ecological environment can occur. Illegal mining activities not only pose a significant threat to the ecological environment but also severely disrupt normal mineral resource development and land use. Therefore, seeking a more advanced, efficient, and comprehensive regulatory method is crucial to ensure the sustainable development and resource management of mining areas.
[0003] In recent years, with the rapid development of remote sensing technology, using remote sensing to detect changes in land use and land cover in mining areas, as well as illegal mining activities, has become a research hotspot. Remote sensing change detection methods aim to obtain information on land use and land cover changes by processing remote sensing image data of the same area acquired at different times and performing comparative image analysis, while simultaneously monitoring illegal mining activities to support land planning management and resource supervision. However, traditional methods mainly rely on manually designed image features for analysis, requiring in-depth feature mining and multi-feature fusion, which limits their ability to handle complex situations such as illegal mining.
[0004] Traditional methods involve analyzing artificially designed image features such as spectral density, texture, and shape extracted from remote sensing images to obtain information on their changes. However, these features are insufficient to represent and distinguish features in complex open-pit mining environments, such as highly heterogeneous environments like mining areas, landfills, reclaimed areas, and mine buildings.
[0005] Furthermore, the UNet++ semantic segmentation network from deep learning methods also faces some challenges, limiting its application in open-pit mine change detection tasks. First, UNet++ struggles to combat scale heterogeneity and handle the diverse shapes of features in open-pit mine scenes. Second, seasonal variations and background changes such as shadows in multi-temporal, multi-source remote sensing imagery can interfere with change detection results. Therefore, it is necessary to improve and optimize the UNet++ network to enable its accurate application in open-pit mine change detection.
[0006] To address the limitations of traditional methods, this invention proposes a remote sensing image-based open-pit mining area change detection method based on the multi-attention mechanism UNet++ network. The aim is to achieve accurate detection of land use change and illegal mining activities by fully utilizing the multi-resolution information of the imagery and expanding the receptive field during feature extraction through an improved deep learning network structure. This innovative method aims to improve the accuracy of remote sensing image-based mining area change detection while simplifying the feature extraction process, providing strong technical support for land resource management and supervision. Summary of the Invention
[0007] Based on the shortcomings of the existing technology, the technical problem to be solved by the present invention is to provide a method for detecting changes in open-pit mines in remote sensing images using the UNet++ network with a multi-attention mechanism. This solves the problem that UNet++ is difficult to combat scale heterogeneity and is also difficult to handle the diverse shapes of ground features in open-pit mine scenes, as well as the interference of seasonal changes and shadows and other background changes in multi-time and multi-source remote sensing images on the change detection results.
[0008] To achieve the above objectives, the present invention adopts the following technical solution:
[0009] A method for detecting changes in open-pit mines in remote sensing images using the UNet++ network with a multi-attention mechanism is proposed. The improved algorithm integrates temporal, spatial, and channel attention (STC-Attention) mechanisms into UNet++ to enhance the semantic representation of features through multiple dimensions. Its main steps include:
[0010] A method for detecting changes in open-pit mines using remote sensing images with a multi-attention mechanism UNet++ network mainly includes the following steps:
[0011] S1. Dataset creation: Prepare the original images, then manually annotate them using the VIA data labeling tool, and finally perform data augmentation.
[0012] S2. Multi-attention mechanism: UNet++ model improvement, using temporal, spatial, and channel attention modules in the feature fusion stage at different scales; subsequently, a focus loss function is used to address the problem of target class imbalance among multiple classes.
[0013] S3. Model training and testing: The prepared training samples are input into the encoder part of UNet++ for feature extraction. The decoder part gradually restores the spatial resolution of the feature maps through upsampling operations and connects and fuses them with the corresponding feature maps of the encoder part to generate a change detection segmentation map.
[0014] S4. Post-processing of detection results: Reduce noise in the segmentation map through operations such as closing operations in image morphology to make the detection results more accurate and form the final detection result.
[0015] In summary, due to the adoption of the above technical solution, the beneficial technical effects of the invention are as follows:
[0016] It eliminates the need for manually designed features to identify and extract land use and land cover in mining areas, saving manpower and resources and improving the efficiency of mining area change detection.
[0017] It can adapt to the characteristics of multi-scale and diverse terrain shapes in complex open-pit mining environments, thus improving the accuracy of mining area change detection. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the Unet++ network structure.
[0019] Figure 2 For technology flowchart, Figure 2 Chinese operators This represents element-wise multiplication of a matrix (using a broadcast mechanism to adjust the weight dimension based on the feature map dimension), and the operator is... This indicates that element-wise addition yields a result matrix of the same size. The attention weights obtained from the temporal, spatial, and channel attention mechanisms are multiplied by the input feature map F, and then the three are summed to calculate the output feature map.
[0020] Figure 3 This is a schematic diagram of the temporal-spatial channel attention mechanism. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0022] Theoretical basis
[0023] The goal of change detection is to identify significant differences between ground targets or phenomena in multi-temporal remote sensing images. Deep learning methods, such as UNet and its various modifications, have been widely used to address the remote sensing change detection problem. UNet is a fully convolutional neural network originally used for semantic segmentation, consisting of encoder and decoder parts. The encoder and decoder parts of the network are symmetrical; the encoder is used to extract features, and the decoder is used to progressively improve the resolution of the output. Its variant, UNet++, adds skip connections between the encoder and decoder to utilize feature information from the encoder during the decoding stage, such as... Figure 1As shown, UNet++'s excellent feature extraction capabilities are better suited to the complexity of remote sensing images, thus becoming a powerful tool for remote sensing change detection. The multi-attention mechanism UNet++ method proposed in this invention is an improvement on the basic UNet++ network, specifically tailored to the characteristics of diverse land features, scale variations, and background changes in open-pit mines.
[0024] Example
[0025] Implementation process
[0026] The implementation flow of the remote sensing image open-pit mine change detection method D using the multi-attention mechanism UNet++ network is as follows: Figure 2 As shown, the process is divided into dataset creation, model improvement, model parameter fine-tuning, model training and testing, and post-processing of change detection results.
[0027] Creating a dataset
[0028] A dataset was created for training the open-pit mine change detection system. Since remote sensing images are generally large and cannot be directly input into the network for training, they were first cropped. Then, the cropped images were labeled, and data augmentation was used to enhance the diversity of the training dataset. This mainly involved random 90° rotations, horizontal flipping, scaling, and adjustments to brightness and contrast. These augmentation operations simulated image changes under different angles, weather conditions, and lighting conditions, helping the trained model to better adapt to various situations.
[0029] UNet++ model improvements
[0030] To address the characteristics of diverse land features, scales, and multi-temporal background variations in remote sensing images of open-pit mines, the UNet++ model is improved. Specifically, temporal, spatial, and channel attention modules are used in the feature fusion stage at different scales to better integrate feature information of targets at different scales; and a focus loss function is used to solve the problem of target class imbalance among multiple categories.
[0031] Temporal, spatial, and channel attention modules
[0032] UNet++ assumes that corresponding pixels at different times, as well as pixels in different channels and spatial locations at the same time, have equal importance when calculating the output feature map of an image. This makes it difficult to suppress changes in the image background and is not suitable for detecting changes in multi-scale land cover types. Therefore, this patent adds an STC-Attention module to the end of each convolutional unit of the original network to enhance the model's ability to learn more effective feature changes. STC-Attention considers the correlation of pixel features at different times, in different spaces, and in different channels.
[0033] The channel attention mechanism focuses on the feature weights of each channel, selectively emphasizing relevant channels and suppressing irrelevant ones. First, the input feature map is processed through parallel global max-pooling and global average-pooling layers to obtain background description. Then, two convolutional layers are used for dimensionality transformation and feature fusion, and finally, sigmoid activation is applied to obtain the channel attention weights.
[0034] The spatial attention mechanism primarily focuses on the spatial feature weights of pixels. This part takes the original feature map as input, passes it through a convolutional layer to perform a dimensionality transformation, reduces the dimension through convolution, and then obtains the spatial attention weights through sigmoid activation.
[0035] The temporal attention mechanism primarily focuses on the feature weights of pixels in images from different time phases. This part still takes the original feature map as input, passing it sequentially through a global max-pooling layer and a global average-pooling layer to obtain feature descriptions. After merging the two feature maps, feature fusion is performed through convolution, and then temporal attention weights are obtained through sigmoid activation.
[0036] Focus loss function
[0037] In remote sensing image change detection datasets, the number of changed samples is generally less than the number of unchanged samples. This leads to excessive attention to unchanged features during training, affecting the accuracy of remote sensing image change detection. The common loss function in this type of task is cross-entropy loss, as shown in formula (1).
[0038]
[0039] Where y is the label and y′ is the predicted value (the output of the activation function, so the value is between 0 and 1).
[0040] To address the imbalanced sample problem, the loss function is optimized by reducing the weight of positive or negative samples to make the model pay more attention to changing samples, while also making the model pay more attention to difficult or misclassified samples. The improved focus loss function is shown in Equation (2).
[0041]
[0042] α is set based on the proportion of each category in the sample, allowing the model to pay more attention to changing samples. When a changing pixel is misclassified, y′ is very small, (1-y′). γ The value is large, and the loss will also be relatively large; when the changed pixels are correctly classified, y′ is large, (1-y′) γ As the size decreases, the loss value also decreases significantly. Similarly, when unchanged pixels are misclassified, y′ is large, and y′... γThe value is large, but the loss will also be relatively large; when unchanged pixels are correctly classified, y′ is small, y′ γ The weights decrease, but the loss value also decreases significantly. This reduces the impact on simple samples, because the effect of accumulating a large number of samples with extremely low prediction probabilities may be even more limited. Therefore, γ is used to adjust the rate at which the weights of individual sample data decrease; as γ increases, the influence of the adjustment factor increases. We set γ to 2 and α to 0.25.
[0043] Model training and testing methods
[0044] The model uses an end-to-end network structure. Because the brightness and pixel value distributions of remote sensing images vary significantly, the model may get trapped in local optima during gradient descent. Therefore, normalization was applied to the experimental data, converting pixels to a range of 0-1 to avoid non-convex optimization problems and improve the network's convergence rate. Remote sensing image change detection requires processing two images; therefore, the two images are merged (channels merged) before being input into the network.
[0045] During training, it is necessary to fine-tune some hyperparameters in the model, such as the number of training iterations and the learning rate. During training, the changes in the loss function can be observed. Generally, the training is complete when the loss function converges or approximately converges.
[0046] During testing, the model can first be analyzed for overfitting and underfitting by observing the accuracy curves on the training and validation sets. Then, the model is tested, the results are analyzed, and the detection accuracy is statistically analyzed. Alternatively, the model can be first cut into smaller pieces for testing and then reassembled.
[0047] Post-processing of test results
[0048] Due to the influence of image quality and resolution, the boundaries of segmented image objects may appear irregular or jagged. Therefore, post-processing of the output remote sensing image change detection results, using image morphology operations such as closing operations to refine the detection results, including methods such as dilation, erosion, and hole filling, can effectively solve the noise interference such as isolated points and holes that still exist in the network's prediction results.
[0049] The above description is a preferred embodiment of the invention and is not intended to limit the scope of the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.
Claims
1. A method for detecting changes in open-pit mine areas using remote sensing images with a multi-attention mechanism UNet++ network, characterized by: The main steps include: S1. Dataset creation: Prepare the original images, then manually annotate them using the VIA data labeling tool, and finally perform data augmentation. S2. Improved UNet++ model with multi-attention mechanism: using temporal, spatial, and channel attention modules in the feature fusion stage at different scales; Then, the focus loss function is used to address the problem of target class imbalance among multiple categories; S3. Model training and testing: The prepared training samples are input into the encoder part of UNet++ for feature extraction. The decoder part gradually restores the spatial resolution of the feature maps through upsampling operations and connects and fuses them with the corresponding feature maps of the encoder part to generate a change detection segmentation map. S4. Post-processing of detection results: Reduce noise in the segmentation map through operations such as closing operations in image morphology to make the detection results more accurate and form the final detection result.
2. The method for detecting changes in open-pit mine areas using remote sensing images with a multi-attention mechanism UNet++ network according to claim 1, characterized in that: In step S1, since remote sensing images are generally large in size and cannot be directly input into the network for training, they are first cropped, then the cropped images are labeled with data, and finally data augmentation is used to enhance the diversity of the training dataset.
3. The method for detecting changes in open-pit mines using remote sensing images with a multi-attention mechanism UNet++ network according to claim 2, characterized in that: The data enhancements include random 90° rotations, horizontal flipping, scaling, and brightness and contrast adjustments.
4. The method for detecting changes in open-pit mine areas using remote sensing images with a multi-attention mechanism UNet++ network according to claim 4, characterized in that: The temporal, spatial, and channel attention module adds an STC-Attention module to the end of each convolutional unit of the original network to enhance the model's ability to learn more effective feature changes. STC-Attention considers the correlation of pixel features at different times, spaces, and channels.
5. The method for detecting changes in open-pit mine areas using remote sensing images with a multi-attention mechanism UNet++ network according to claim 4, characterized in that: The channel attention module primarily focuses on the feature weights of each channel, selectively emphasizing relevant channels and suppressing irrelevant channels; firstly, the input feature map F∈R... C×W×H Background description is obtained through parallel global max pooling and global average pooling layers, then dimensionality transformation and feature fusion are performed through two convolutional layers, and finally channel attention weights are obtained through Sigmoid activation.
6. The method for detecting changes in open-pit mine areas using remote sensing images with a multi-attention mechanism UNet++ network according to claim 4, characterized in that: The spatial attention mechanism focuses on the spatial feature weights of pixels. This part takes the original feature map as input, passes it through a convolutional layer to transform its dimensions, reduces the dimension to 1×H×W through convolution, and then obtains the spatial attention weights through Sigmoid activation.
7. The method for detecting changes in open-pit mines using remote sensing images with a multi-attention mechanism UNet++ network according to claim 4, characterized in that: The temporal attention mechanism focuses on the feature weights of pixels in images from different time phases. This part still takes the original feature map as input and passes it through a global max pooling layer and a global average pooling layer in sequence to obtain feature descriptions. After merging the two feature maps, feature fusion is performed through convolution, and then temporal attention weights are obtained through Sigmoid activation.
8. The method for detecting changes in open-pit mine areas using remote sensing images with a multi-attention mechanism UNet++ network according to claim 1, characterized in that: In step S2, the focus loss function is as shown in formula (1). Where y is the label, y′ is the predicted value, y′ takes a value between 0 and 1, α is determined according to the proportion of each category in the sample, so that the model pays more attention to the changing samples; γ is 2, and α is 0.
25.
9. The method for detecting changes in open-pit mine areas using remote sensing images with a multi-attention mechanism UNet++ network according to claim 1, characterized in that: In step S3, the model is an end-to-end network structure, and the testing method is as follows: before testing, the test data needs to be normalized to change the pixels to the range of 0 to 1. Then, the two images are merged by channel merging and then input into the network. At the same time, the change of the loss function is observed during the training process. The training is completed when the loss function converges or approximately converges.
10. The method for detecting changes in open-pit mine areas in remote sensing images using a multi-attention mechanism UNet++ network according to claim 1, characterized in that: In step S3, during the model testing process, the model is first analyzed for overfitting and underfitting by observing the accuracy curves on the training and validation sets. Then, the model is called to perform the test, the test results are analyzed, and the detection accuracy is statistically analyzed.