Remote sensing image small-scale fuzzy region classification method based on multiple modes and attention

Abstract

The invention discloses a remote sensing image small-scale fuzzy region classification method based on multiple modes and attention, and the method comprises the following steps: S1), obtaining a PAN image and an MSS image of the same region as original remote sensing images, and cutting image blocks according to a certain rule to obtain a data set; s2) constructing a feature extractor based on ResNet50, multi-head self-attention and cross attention and a classifier network model based on maximum global pooling for training; s3) AMP is used for training in the training process, and the gradient of the final feature output by the feature extractor is solved according to the output result of the classifier full-connection layer to obtain a CAM thermodynamic diagram; s4) testing the network model by using the test set every time the network model is trained for one round, and repeating the steps S1) to S4) if the classification accuracy of the network model in the test stage is still reduced in a specific training round; otherwise, stopping training in advance, and outputting a final overall network model. According to the method, the small-scale fuzzy features can be effectively extracted from the large-size high-resolution remote sensing image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, in particular to a remote sensing image small-scale fuzzy region classification method based on multi-modal and attention. BACKGROUND

[0002] Remote sensing images are affected by various factors in the process of acquisition, transmission and processing, which can cause image blur and affect subsequent interpretation, classification and target detection applications. The blur of remote sensing images may come from sensor hardware limitations, such as aperture and wavelength of optical systems, detector noise, may come from imaging environment interference, such as atmospheric scattering and absorption, camera system jitter, target object dynamic change, and may come from compression in the transmission process and dataset offset in the super-resolution reconstruction algorithm in the processing process. Being able to effectively extract small fuzzy regions from large remote sensing images can facilitate object analysis, high-quality data construction, and subsequent deployment of deep learning algorithms.

[0003] The existing deep learning algorithm has three shortcomings in classifying the blur problem of remote sensing images: first, most methods are trained on low-resolution images, which cannot effectively distinguish between smooth texture objects and fuzzy regions; second, the training set of remote sensing images is small, and the fuzzy region occupies a large proportion in the image, and the feature is obvious, but the generalization ability is insufficient for small-scale fuzzy regions in large remote sensing images; third, the interpretability is insufficient, most deep learning algorithms will only output "yes" or "no", and will not tell the user which region is used to make the judgment that the remote sensing image is blurred. SUMMARY

[0004] In view of this, the present application provides a remote sensing image small-scale fuzzy region classification method based on multi-modal and attention, which can effectively extract small-scale fuzzy features from large high-resolution remote sensing images.

[0005] The remote sensing image small-scale fuzzy region classification method based on multi-modal and attention includes the following steps:

[0006] S1) Obtain a panchromatic (PAN) image and a multispectral (MSS) image of the same area, adjust the size of the MSS image to the same size as the PAN image to obtain an original large-size remote sensing image, and then crop the image block according to certain rules and check the label to obtain a training set and a test set;

[0007] S2) Construct a feature extractor based on ResNet50, multi-head self-attention and cross-attention, and a classifier network model based on maximum global pooling for training;

[0008] S3) The training process uses Auto Mixed Precision (AMP) for training, and the gradient of the output of the fully connected layer of the classifier with the final feature output of the feature extractor is used to obtain the CAM heatmap, which visualizes the regions in the remote sensing image that the network model has learned to be blurry.

[0009] S4) After each training cycle, the network model is tested using a test set to calculate the classification accuracy after that training cycle. If the classification accuracy of the network model still decreases in a specific training cycle during the testing phase, then repeat steps S1) to S4). Otherwise, the training is terminated early, and the final overall network model is output and used for fuzzy classification scenarios of remote sensing images.

[0010] Furthermore, the cropping rules for step S1) are as follows: For original remote sensing images with unblurred image labels, several non-overlapping image patches are randomly cropped from them during each training session as unblurred images and input into the network model; For original remote sensing images with blurred image labels, image patches with smaller MSS image gradients and corresponding smaller PAN image gradients are cropped from them during each training session as blurred images, and input into the network model after manual inspection.

[0011] Furthermore, the feature extractor in step S2) employs a hybrid structure of ResNet50, multi-head self-attention, and cross-attention. The ResNet50 network extracts features from the input MSS and PAN images respectively, and performs multi-head self-attention and cross-attention calculations on the output features of the fourth stage of ResNet50. The query vector Q, key vector K, and value vector V of the multi-head self-attention, and the query vector Q of the cross-attention are derived from the feature map extracted from the PAN image by ResNet50, while the key vector K and value vector V of the cross-attention are derived from the feature map extracted from the MSS image by ResNet50. The final features calculated and concatenated by the multi-head self-attention module and the cross-attention module are input to the classifier. After feature compression using max global pooling, the results are output through a fully connected layer and used in subsequent loss function calculations to obtain the final classification result, determining whether there are blurred regions in the remote sensing image.

[0012] Furthermore, the loss function is calculated as follows:

[0013] The output of the fully connected layer of the classifier is compared with the ground truth to calculate the binary cross-entropy loss function. The formula for the binary cross-entropy loss function is as follows:

[0014]

[0015] in, This refers to the binary classification of true values, specifically for remote sensing images with blurred image labels. Image labels are for unblurred remote sensing images. ; These are the predicted values ​​from the network model. The number of remote sensing images in the training set.

[0016] Furthermore, in step S3), AMP training is implemented using GradScaler, where the forward and backward propagation of the ResNet50 feature extraction process is computed using FP16, and all operations of the classifier are computed using FP32.

[0017] Furthermore, in step S3), the CAM heatmap is calculated using the backpropagation gradient of the network model; let the output of the fully connected layer of the classifier be... The concatenation result of the output features of the multi-head self-attention module and the cross-attention module is as follows: ,but right The gradient map is calculated as follows:

[0018]

[0019] The above calculation can be directly performed using the `grad` method in `torch.autograd`; averaging the gradient maps across the spatial dimensions yields the weights of the feature maps for each channel in the output. ,if This indicates that the feature map of this channel has a positive effect on the output; therefore, the negative values ​​are suppressed by using the ReLU activation function. By weighted summation and normalization along the channel dimensions, the CAM heatmap can be obtained. The specific formula is as follows:

[0020] (1)

[0021] (2)

[0022] (3)

[0023] Formula (1) represents averaging the gradient map across the width and height dimensions of the remote sensing image, retaining only the feature channel dimension as the weight; Formula (2) represents weighted summation of the weights and inputs and nonlinear mapping through the ReLU activation function; Formula (3) represents normalization calculation.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] 1. In cropping the dataset, this invention does not use completely random or uniform cropping, but rather defines different cropping methods based on whether the original remote sensing image labels are blurry. Using multimodal feature input can enhance the representational power of the network model to distinguish between blurry and smooth ground features.

[0026] 2. The feature extractor of this invention adopts a hybrid structure based on ResNet50 and attention. Compared with a pure convolutional neural network structure, the network model's attention to the feature map can be changed by adjusting the computational range of attention, thus significantly affecting the classification results. Compared with a pure attention structure, it also inherently possesses the translation invariance and inductive bias of locality inherent in convolutional neural networks. Furthermore, this invention employs a hybrid mechanism of multi-head self-attention and cross-attention, which can fully utilize multimodal input information and effectively fuse multimodal features, making the fuzzy features learned by the network model more robust.

[0027] 3. The pooling layer used in the classifier of this invention is max global pooling, rather than the more common average global pooling. Max global pooling makes the network model more sensitive to small-scale local blurring, while average global pooling is more suitable for scenarios where the blurred region accounts for a large proportion of the entire remote sensing image or for multi-classification scenarios.

[0028] 4. The method of obtaining CAM heatmap in this invention is Grad-CAM based on the weight gradient of the network model. Through the CAM heatmap, users can locate whether the network model classifies remote sensing images as blurry due to blurry areas, thereby increasing the interpretability of the network model. Attached Figure Description

[0029] Figure 1 This is a flowchart of the dataset pruning process for this invention;

[0030] Figure 2 This is a schematic diagram of the overall structure of the network model of the present invention;

[0031] Figure 3 This is a schematic diagram of the multi-head self-attention module and cross-attention module of the present invention. Detailed Implementation

[0032] To better understand the technical content of the present invention, specific embodiments will be provided below in conjunction with the accompanying drawings, and the present invention will be further described in conjunction with the accompanying drawings.

[0033] Example

[0034] A method for classifying small-scale blurred regions in remote sensing images based on multimodal and attention methods includes the following steps:

[0035] S1) Obtain labeled PAN and MSS images of the same region. PAN images are typically 10000×10000 pixels, and MSS images are typically 2500×2500 pixels. Because PAN images have higher resolution and MSS images have lower resolution, their blurriness will differ slightly. Therefore, PAN and MSS images are used as input to the multimodal network model to check for blurriness issues. This invention upsamples and resizes the MSS image, using PAN and MSS images of the same size (10000×10000) as the original large-size remote sensing images.

[0036] After obtaining the original remote sensing images, they need to be cropped before they can be input into the network model for training. The cropping process is as follows: Figure 1 As shown, the cropping rules are as follows: For the original remote sensing image with an unblurred label, four non-overlapping 1024×1024 image patches are randomly cropped as unblurred images during each training session. To obtain the four non-overlapping image patches, the width of the entire remote sensing image is first... and high Calculate the maximum number of image patches that can be cropped in each dimension: Then respectively in and Random numbers are generated within a certain range, and each of these random numbers is multiplied by 1024 to serve as the starting point for cropping image patches. This starting point is recorded using a set. This process is repeated, and the set is used to determine the starting point for each cropped image patch to avoid overlapping ranges, until four non-overlapping cropped image patches are obtained. Finally, the labels of these four cropped image patches are set to 0 and input into the network model. For original remote sensing images labeled as blurred, 1024×1024 image patches are cropped during each training iteration. Image patches with smaller MSS (Maximum Saturation Level) gradients and corresponding smaller PAN (Positioning Area) gradients are selected as blurred images. The gradient range for meeting the criteria is set to 10-30. A gradient less than 10 is required because gradients less than 10 generally indicate overexposure or shadows, rather than blur. If the pixels meeting the criteria account for more than 5% of the 1024×1024 image patch, it is considered valid blur, the patch label is set to 1, and it is input into the network model. In other words, this invention can detect a blurred region with more than 52428 pixels, or a minimum blurred region area of ​​10000㎡. The final training set consists of 3580 image patches, including 2000 blurred images and 1580 unblurred images; the test set consists of 400 image patches, including 160 blurred images and 240 unblurred images.

[0037] S2) The network model structure is based on ResNet50, multi-head self-attention and cross-attention feature extractors, and a classifier based on maximum global pooling, as shown in the figure. Figure 2 As shown. Since the MSS image and the PAN image share the same ResNet50 network, this invention first copies the PAN image into three copies and stacks them along the channel dimension to align with the MSS image. The standard ResNet50 network contains four stages of output. After feature extraction by the ResNet50 network, the features output by the fourth stage of ResNet50 are denoted as follows: and Only utilize and It participates in subsequent attention calculations. The reason for selecting only the output of the fourth stage instead of the output of the first three stages is that using deep features can minimize the error caused by positional shifts, and fuzzy classification, as an image-level task, does not require the fine-grained accuracy provided by shallow features. and The attention calculation scope adopts the Shift Window method from Swin to maximize the network model's focus on small-scale blurred features without being disturbed by unblurred regions in the image. A hybrid approach of multi-head self-attention and cross-attention is used to ensure the network model can fully learn the differences between blurred and smooth features from the multimodal input and surrounding pixels. The structures of the multi-head self-attention module and the cross-attention module are as follows: Figure 3As shown. Let Q be the query vector, K be the key vector, and V be the value vector used for attention calculation. The difference between multi-head self-attention and cross-attention lies in whether Q, K, and V come from the same feature map. The query vector Q, key vector K, and value vector V for multi-head self-attention, and the query vector Q for cross-attention, come from the feature map extracted from the PAN image by ResNet50; the key vector K and value vector V for cross-attention come from the feature map extracted from the MSS image by ResNet50. Both the multi-head self-attention module and the cross-attention module consist of a multi-head attention layer and a feedforward layer. The multi-head attention layer is calculated using torch.nn.MultiheadAttention, and the position encoding uses rotational position encoding. The feedforward layer contains two linear layers: the first linear layer maps the number of feature channels from 256 to 1024, and the second linear layer maps the number of channels back from 1024 to 256. Both the multi-head attention layer and the feedforward layer use LayerNorm to normalize the output results. The output features of the multi-head self-attention module and the cross-attention module are concatenated and then input into the classifier. Max global pooling is used to compress the features to highlight local small-scale blurred features as much as possible. Then, a fully connected layer is used to convert the number of channels into the number of output classes, which is 2. The output of the fully connected layer is used in the calculation of the subsequent loss function to obtain the final classification result. At this point, the forward pass of the network model is completed.

[0038] The loss function is calculated as follows:

[0039] The output of the fully connected layer of the classifier is compared with the ground truth to calculate the binary cross-entropy loss function. The formula for the binary cross-entropy loss function is as follows:

[0040]

[0041] in, This refers to the binary classification of true values, specifically for remote sensing images with blurred image labels. Image labels are for unblurred remote sensing images. ; These are the predicted values ​​from the network model. The number of remote sensing images in the training set.

[0042] S3) AMP training achieves mixed-precision FP16 and FP32 calculations through autocast in torch.cuda.amp. Specifically, in the forward method of the network model structure definition, the ResNet50 part of the network structure is placed in the context structure of withautocast(dtype=torch.float16), and the classifier part of the network structure is placed in the context structure of withautocast(dtype=torch.float32). Then, gradient calculation, learning rate adjustment, and gradient update are performed through an instantiated object of the GradScaler class, thereby achieving mixed-precision training, reducing memory usage, accelerating training speed, and ensuring classification accuracy.

[0043] S4) The CAM heatmap is calculated using the backpropagation gradient of the network model. Let the output of the fully connected layer of the classifier be... The concatenation result of the output features of the multi-head self-attention module and the cross-attention module is as follows: ,but right The gradient map is calculated as follows:

[0044]

[0045] The above calculation can be directly performed using the `grad` method in `torch.autograd`. Averaging the gradient maps across their spatial dimensions yields the weights of each channel's feature map in relation to the output. ,if This indicates that the feature map of this channel has a positive effect on the output; therefore, the negative values ​​are suppressed by using the ReLU activation function. By weighted summation and normalization along the channel dimensions, the CAM heatmap can be obtained. The specific formula is as follows:

[0046] (1)

[0047] (2)

[0048] (3)

[0049] Formula (1) represents averaging the gradient map across the width and height dimensions of the remote sensing image, retaining only the feature channel dimension as the weight; Formula (2) represents weighted summation of the weights and inputs and nonlinear mapping through the ReLU activation function; Formula (3) represents normalization calculation.

[0050] S5) The classification accuracy of the network model when tested on the test set is calculated as follows: the result after softmax + argmax is compared with the labels set on the remote sensing images before inputting them into the network model. If they are equal, the classification accuracy is incremented by one. The final classification accuracy is then divided by the number of remote sensing images in the test set to obtain the classification accuracy. The network model employs an early termination training method; that is, if the classification accuracy during the testing phase does not decrease within 50 training epochs, the training process is terminated early, and the final overall network model is output.

[0051] The specific configuration and hyperparameters for training the network model are as follows: parallel training using 4 RTX 3090 graphics cards, NCCL communication protocol, AdamW optimizer, learning rate set to 0.001667 according to the Ultralytic framework calculation formula, betas parameter set to (0.99, 0.999), batch size set to 8, training epochs set to 100, and learning rate update strategy set to polyscheduler.

[0052] Table 1 Ablation experiments with different mechanisms for combining multimodal features

[0053]

[0054] The ablation experiment results in Table 1 demonstrate the effectiveness of the method of this invention. In Table 1, feature concatenation refers to directly concatenating the features of the PAN image and MSS image along the channel dimension using the concat method, and then mapping the channel dimension through a convolutional layer; multi-head self-attention + feature concatenation refers to first calculating multi-head self-attention based on the features of the PAN image and MSS image, then concatenating the multi-head self-attention maps, and finally mapping the channel dimension through a convolutional layer; cross-attention uses the feature map of the PAN image as the query vector Q, and the feature map of the MSS image as the key vector K and value vector V to calculate cross-attention. Experimental results show that the hybrid mechanism of multi-head self-attention and cross-attention in this invention, which combines multimodal features, can effectively improve the classification accuracy of the network model. It is worth noting that the classification accuracy using only cross-attention is almost no better than that of feature concatenation.

[0055] Table 2 Comparison of Single-Mode Input and Multi-Mode Input

[0056]

[0057] Table 2 shows the importance of using multimodal input in improving classification accuracy. Using only MSS or PAN images as input, the network model cannot accurately distinguish which areas are blurred in the remote sensing image and which are smooth terrain features.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Those skilled in the art can make various improvements and modifications to the present invention without departing from its technical principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for classifying small-scale blurred regions in remote sensing images based on multimodal and attention principles, characterized in that: Includes the following steps: S1) Obtain PAN and MSS images of the same area. By adjusting the size of the MSS image to be the same as that of the PAN image, the original large-size remote sensing image is obtained. Then, the original remote sensing image is cropped into image patches according to certain rules and the labels are checked to obtain the training set and test set. S2) Construct a feature extractor based on ResNet50, multi-head self-attention and cross-attention, and a classifier network model based on maximum global pooling for training; S3) The training process uses AMP for training, and the gradient of the output of the fully connected layer of the classifier with the final feature output of the feature extractor is used to obtain the CAM heatmap, which visualizes the regions in the remote sensing image that the network model has learned to have blurred areas. S4) After each training cycle, the network model is tested using a test set to calculate the classification accuracy after that training cycle. If the classification accuracy of the network model still decreases within a specific training round during the testing phase, repeat steps S1) to S4); otherwise, terminate the training early, output the final overall network model, and use it for fuzzy classification scenarios of remote sensing images.

2. The method for classifying small-scale blurred regions in remote sensing images based on multimodal and attention principles according to claim 1, characterized in that, The cropping rules for step S1) are as follows: For original remote sensing images with non-blurred image labels, several non-overlapping image patches are randomly cropped from them during each training session as non-blurred images and input into the network model; For original remote sensing images with blurred image labels, image patches with smaller MSS image gradients and corresponding smaller PAN image gradients are cropped from them during each training session as blurred images, and input into the network model after manual inspection.

3. The method for classifying small-scale blurred regions in remote sensing images based on multimodal and attention principles according to claim 2, characterized in that, The feature extractor in step S2) employs a hybrid structure of ResNet50, multi-head self-attention, and cross-attention. The ResNet50 network extracts features from the input MSS and PAN images respectively, and performs multi-head self-attention and cross-attention calculations on the output features of the fourth stage of ResNet50. The query vector Q, key vector K, and value vector V of the multi-head self-attention, and the query vector Q of the cross-attention are derived from the feature map extracted from the PAN image by ResNet50, while the key vector K and value vector V of the cross-attention are derived from the feature map extracted from the MSS image by ResNet50. The final features calculated and concatenated by the multi-head self-attention and cross-attention modules are input to the classifier. After feature compression using max global pooling, the results are output through a fully connected layer and used in subsequent loss function calculations to obtain the final classification result, determining whether there are blurred regions in the remote sensing image.

4. The method for classifying small-scale blurred regions in remote sensing images based on multimodal and attention principles according to claim 3, characterized in that, The loss function is calculated as follows: The output of the fully connected layer of the classifier is compared with the ground truth to calculate the binary cross-entropy loss function. The formula for the binary cross-entropy loss function is as follows: in, This refers to the binary classification of true values, specifically for remote sensing images with blurred image labels. Image labels are for unblurred remote sensing images. ; These are the predicted values ​​from the network model. The number of remote sensing images in the training set.

5. The method for classifying small-scale blurred regions in remote sensing images based on multimodal and attention principles according to claim 1, characterized in that, In step S3), AMP training is implemented using GradScaler, where the forward and backward propagation of the ResNet50 feature extraction process is computed using FP16, and all operations of the classifier are computed using FP32.

6. The method for classifying small-scale blurred regions in remote sensing images based on multimodal and attention principles according to claim 5, characterized in that, In step S3), the CAM heatmap is calculated using the backpropagation gradient of the network model; the output of the fully connected layer of the classifier is denoted as... The concatenation result of the output features of the multi-head self-attention module and the cross-attention module is as follows: ,but right The gradient map is calculated as follows: The above calculation can be directly performed using the `grad` method in `torch.autograd`; averaging the gradient maps across the spatial dimensions yields the weights of the feature maps for each channel in the output. ,if This indicates that the feature map of this channel has a positive effect on the output; therefore, the negative values ​​are suppressed by using the ReLU activation function. By weighted summation and normalization along the channel dimensions, the CAM heatmap can be obtained. The specific formula is as follows: （1） （2） （3） Formula (1) represents averaging the gradient map across the width and height dimensions of the remote sensing image, retaining only the feature channel dimension as the weight; Formula (2) represents weighted summation of the weights and inputs and nonlinear mapping through the ReLU activation function; Formula (3) represents normalization calculation.

Images