Spectrum self-attention and cloud mask guided remote sensing image cross-modal cloud detection method

Abstract

The invention provides a spectrum self-attention and cloud mask guided remote sensing image cross-modal cloud detection method. According to the method, the spectral self-attention sub-network in the remote sensing image cross-modal cloud detection network is constructed, and the dependency relationship between different spectral bands of the remote sensing image is effectively mined and utilized, so that the recognition accuracy of cloud region features is improved. According to the method, cross-modal migration from the source domain image to the target domain image is realized by using the migration sub-network of the unsupervised image domain in the remote sensing image cross-modal cloud detection network, and the multi-spectral image with the target domain features is generated, so that the defect of the source domain image in spectral information is overcome, the false detection and omission ratio is effectively reduced, and the detection efficiency is improved. And the robustness of cloud detection of the model in a complex scene is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically relates to a spectral self-attention and cloud mask-guided cross-modal cloud detection method for remote sensing images within the field of remote sensing image detection technology. This invention can be used to accurately identify and detect cloud regions in complex remote sensing scenarios. Background Technology

[0002] Cloud detection in remote sensing images is a crucial step in surface information extraction and remote sensing image preprocessing, significantly impacting applications such as meteorological monitoring and environmental assessment. In recent years, with the development of remote sensing technology, hyperspectral and multispectral images have demonstrated excellent performance in cloud detection due to their rich optical feature information. However, acquiring hyperspectral remote sensing images relies on advanced imaging sensors. Currently, most remote sensing satellites widely used in China are equipped with optical sensors that only support imaging of a limited number of spectral bands. Their spectral resolution and optical information density are lower than those of hyperspectral sensors. This makes it difficult to distinguish clouds from other interfering objects in complex surface environments, often leading to decreased detection accuracy. To address these issues, a cloud detection method is needed that can infer richer spectral information from limited spectral band images without relying on hyperspectral imaging sensors, thereby improving the feature representation capabilities in cloud detection tasks.

[0003] Wuhan University proposed a cloud detection method based on a cascaded U-shaped network with an attention mechanism in its patent application, "A Cloud Detection Method and System Based on a Cascaded U-Shaped Network with an Attention Mechanism" (Application No.: CN202311293882.0, Publication No.: CN117036984A, Publication Date: 2023-11-10). The core idea of ​​this method is to construct a deep learning model composed of two cascaded U-shaped networks, introduce an attention mechanism in the first stage to enhance feature extraction capabilities, and further refine the cloud detection results in the second stage, thereby achieving accurate detection and refined identification of cloud regions in multispectral remote sensing images. The main steps of its implementation are as follows: (1) Preprocessing remote sensing image data; (2) Constructing a cascaded U-shaped network with an attention mechanism, including two cascaded U-Net structures, each U-shaped network containing three parts: a shrinking path, a skip connection, and an expanding path. The first U-Net has five layers and adopts a residual structure and channel and spatial attention mechanism to be responsible for preliminary feature extraction and cloud region detection. The second U-Net has four layers and is used to further refine and optimize the detection results. The two U-Nets are connected by a convolutional module and a Bridge layer is introduced into the network to enhance feature flow and avoid the risk of overfitting. (3) The network is trained using training data and the model parameters are optimized using a loss function and Adam optimizer. (4) The test data is input into the trained model and the probability map of cloud detection is output. The shortcomings of this method are: the number of remote sensing image spectral bands on which this method depends is limited, and the spectral features of clouds and highly reflective ground objects are easily confused, resulting in less than ideal detection results. Although this method enhances the feature extraction capability by cascading two U-Net structures and introducing channel and spatial attention mechanisms, its attention mechanism does not explicitly model the dependency relationship between bands, so it is difficult to fully explore and utilize the intrinsic relationship between different spectral bands. Furthermore, the loss function used in this method is relatively simple and does not fully incorporate factors such as edge information and image quality. Therefore, it still needs improvement in terms of enhancing the clarity of cloud detection boundaries and overall accuracy. Summary of the Invention

[0004] The purpose of this invention is to overcome the defects of the prior art and provide a cross-modal cloud detection method for remote sensing images guided by spectral self-attention and cloud masking. This method addresses the problem in the prior art where limited spectral information in some remote sensing images and the failure to effectively model the dependencies between different spectral bands make it difficult to accurately distinguish clouds from similar interfering objects, resulting in low cloud detection accuracy and high rates of missed and false detections.

[0005] The underlying principle of this invention is as follows: Addressing the low cloud detection accuracy of existing GF-1WFV remote sensing images due to the limited number of spectral bands, this invention constructs a cross-modal cloud detection network comprising a spectral self-attention sub-network and an unsupervised image domain transfer sub-network. In the unsupervised image domain transfer sub-network, GF-1WFV images undergo cross-modal transfer to generate multi-spectral band images with a Sentinel-2A style, supplementing the original image's spectral information deficiencies. In the spectral self-attention sub-network, a U-shaped spectral self-attention module is designed and introduced to fully utilize the dependencies between different spectral bands. A dual-branch structure synchronously generates cross-modal multi-spectral band images and cloud mask results, achieving joint optimization of image enhancement and cloud detection. To address issues such as detail blurring, style confusion, and unclear cloud boundaries during cross-modal generation, this invention designs structural style separation loss and Gaussian edge loss, and combines multiple loss functions to optimize the network training process, effectively improving the quality of the generated images and the clarity of cloud detection boundaries. This invention combines cross-modal image transfer technology and image segmentation technology to address the problem of high false detection and false negative rates in cloud detection in complex scenarios, which is caused by the limited number of spectral bands and insufficient spectral information in GF-1WFV remote sensing images.

[0006] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0007] Step 1: Construct a cross-modal cloud detection network including a spectral self-attention subnetwork and an unsupervised image domain transfer subnetwork;

[0008] Step 2: Select source domain X-domain images and target domain Y-domain images from remote sensing image data collected by different sensors to form a sample set. After preprocessing each image in the sample set, a training set is generated.

[0009] Step 3: Design the structural style separation loss function and the Gaussian edge loss function;

[0010] Step 4, train the remote sensing image cross-modal cloud detection network:

[0011] Step 5: Perform cloud detection on the remote sensing image to be detected.

[0012] Using the same method as in step 2, the preprocessed remote sensing image to be detected is input into the trained cross-modal cloud detection network for cloud detection, and the cloud detection results are output.

[0013] Furthermore, the spectral self-attention subnetwork is composed of a U-shaped spectral self-attention module SAU and a bi-branched U-shaped spectral self-attention module DSAU connected in series;

[0014] The U-shaped spectral self-attention module (SAU) adopts an encoder-decoder structure. The encoder part contains a three-stage cascaded feature extraction structure: the first stage consists of an embedded convolutional layer with a kernel size of 1×1 and a stride of 1, a spectral self-attention module (SAB), and a downsampling convolutional layer with a kernel size of 3×3 and a stride of 2, cascaded together; the second stage consists of a spectral self-attention module (SAB) and a downsampling convolutional layer with a kernel size of 3×3 and a stride of 2, cascaded together; the third stage is a spectral self-attention module (SAU). The encoder consists of a self-attention module (SAB); the decoder is a two-stage cascaded decoding structure: the first stage consists of a spectral self-attention module (SAB) and an upsampling convolutional layer, with the upsampling operation using a deconvolutional layer with a kernel size of 3×3 and a stride of 2; the second stage consists of a spectral self-attention module (SAB) and an upsampling convolutional layer, with the upsampling operation using a deconvolutional layer with a kernel size of 3×3 and a stride of 2; the encoder's first-stage output is connected to the second-stage input of the decoding branch, and the encoder's second-stage output is connected to the first-stage input of the decoding branch.

[0015] The dual-branch U-shaped spectral self-attention module DSAU is an U-shaped spectral self-attention module SAU with an added decoding branch. The two decoding branches have the same structure as the decoder in the SAU module, and the two decoding branches are in parallel. The first-stage output of the encoder is simultaneously connected to the second-stage input of the two decoding branches, and the second-stage output of the encoder is simultaneously connected to the first-stage input of the two decoding branches, forming three parallel linear transformation paths; the input image X... in X is obtained after three linear transformations. Q X K X V , for X K After performing matrix transpose and X Q Perform matrix multiplication to obtain attention map A, then combine A with X. V Matrix multiplication is performed to obtain the output feature map X. A The obtained X A After performing a linear transformation, the output feature X of the network is obtained by adding it to the original input feature. out .

[0016] Furthermore, the unsupervised image domain transfer subnetwork structure includes two generators and two discriminators. The original X-domain image is converted into a Y-domain image by the first generator and then input into the first discriminator to determine whether it is a real target domain image. The second generator reconstructs the Y-domain image back into the original X-domain image, and then the second discriminator determines whether the reconstructed X-domain image is real. Each generator is composed of an initial convolutional layer cascaded with a spectral self-attention subnetwork. The kernel size of the initial convolutional layer is set to 1×1, and the kernel movement stride is... The structure of each discriminator consists of one input convolutional layer, three intermediate convolutional layers, and one output convolutional layer. The kernel size of the input convolutional layer is set to 3×3, the kernel stride is 1, and the number of kernels is set to 64. The three intermediate convolutional layers are all 5×5 convolutional layers with the following kernel sizes: 128, 256, and 512, respectively. The kernel stride is 2 for each kernel. Normalization and ReLU activation function processing are performed after each convolutional layer. The kernel size of the output convolutional layer is 5×5, and the kernel stride is 1.

[0017] Furthermore, the steps for generating the training set are as follows:

[0018] The first step is to select remote sensing image data from different sources that contain cloud areas and various ground features as source domain X-domain images and target domain Y-domain images, respectively. Among them, there are at least 108 source domain images and at least 36 target domain images; and the cloud areas in the selected images contain various cloud types such as thin clouds, fragmented clouds and thick clouds.

[0019] The second step is to crop the source domain image into a sub-pattern with a size of 512×512 pixels and the target domain image into a sub-pattern with a size of 384×384 pixels.

[0020] The third step is to treat the cropped source domain sub-plots and target domain sub-plots as independent datasets to form an unpaired training set.

[0021] Furthermore, the structural style separation loss function L s as follows:

[0022] L s = l(x,y)·c(x,y)·s(x,x')

[0023]

[0024] Where l(x,y) represents brightness loss, used to measure the consistency of the generated image and the target image in overall brightness; c(x,y) represents contrast loss, used to compare local intensity changes in the image; s(x,x') represents structure loss, used to measure the similarity of the images in structural information; x represents the generated X-domain image, y represents the Y-domain image, and x' represents the dark channel of the X-domain image; μ represents the mean, σ represents the variance; C1, C2, and C3 represent adjustment factors, where C1 = (K1L) 2 C2 = (K2L) 2 C3 = C2 / 2, K1 = 0.01, K2 = 0.03, and L represents the dynamic range of pixel values ​​in the range [0, 255].

[0025] The Gaussian edge loss function L b as follows:

[0026] L b =MSE(sg(x' m ),sg(x m ))+MSE(sg(y' m ),sg(y m ))

[0027] Where MSE represents mean squared error, and sg represents a combination of Sobel and Gaussian processing. Sobel processing calculates the gray-level gradients of the image in the horizontal and vertical directions using a 3×3 convolution kernel, while Gaussian processing convolves the image using a 3×3 Gaussian filter kernel. This involves weighting each pixel in the image and its eight neighboring pixels according to the weights in the filter kernel, and then averaging the sums. The weights in the filter kernel are determined by a two-dimensional Gaussian function G(i,j). i and j represent the coordinates of the offset position of a pixel relative to the center in the filter kernel, π represents pi, σ is the standard deviation, and exp represents an exponential function with the natural constant e as the base; x m 、x' m y m and y' m These represent the predicted cloud detection result in domain X, the actual cloud label in domain X, the predicted cloud detection result in domain Y, and the actual cloud label in domain Y, respectively.

[0028] Furthermore, the steps for training the cross-modal cloud detection network are as follows: input the training set into the cross-modal cloud detection network, train it using backpropagation and the Adam Optimizer, iteratively update the network parameters until the total loss function of the network converges, and obtain the trained cross-modal cloud detection network.

[0029] The total loss function of the network is as follows:

[0030] L=L(D,G)+L cyc +L s +L b

[0031] Where L(D,G) represents the adversarial loss, L cyc L represents the cycle-consistent loss. s L represents the structural style separation loss. b This represents Gaussian edge loss.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] First, this invention constructs a cross-modal cloud detection network comprising a spectral self-attention subnetwork and an unsupervised image domain transfer subnetwork, thereby mapping source domain remote sensing images with limited spectral bands to target domain remote sensing images with multiple spectral bands, thus expanding the spectral dimension of the input image. Compared to remote sensing image data with a limited number of spectral bands used in existing technologies, this invention effectively compensates for the deficiency in spectral information dimension of remote sensing image data with limited spectral bands, improving the ability to distinguish clouds from other complex highly reflective surface features.

[0034] Secondly, the spectral self-attention subnetwork designed in this invention introduces a U-shaped spectral self-attention module, which can deeply explore the dependencies between different spectral bands. This overcomes the shortcomings of existing technologies that cause confusion between clouds and highly reflective interference objects due to insufficient spectral information or band feature aliasing. As a result, the model of this invention enhances the expression of spectral information while strengthening the expression of spatial structure features of cloud regions, further improving the accuracy of cloud detection.

[0035] Third, this invention designs structural style separation loss and Gaussian edge loss, which, together with adversarial loss and cycle consistency loss, constitute a loss function combination to jointly optimize the network training process. This overcomes the shortcomings of existing technologies where the loss function is relatively simple, leading to structural ambiguity, style confusion, and unclear cloud boundary recognition in cloud detection tasks in cross-modal generated images. This invention, through the joint optimization of multiple loss functions, not only effectively improves the fidelity of cross-modal generated images at the structural and stylistic levels but also enhances the recognition accuracy and stability of the cloud detection model in complex scenes. Attached Figure Description

[0036] Figure 1 This is a flowchart of the present invention;

[0037] Figure 2 This is a schematic diagram of the spectral self-attention module structure constructed in this invention;

[0038] Figure 3 This is a schematic diagram of the spectral self-attention subnetwork constructed in this invention;

[0039] Figure 4 This is a schematic diagram of the structure of the unsupervised image domain transfer subnetwork constructed in this invention; Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] Reference Figure 1 The implementation steps of the embodiments of the present invention will be further described below.

[0042] Step 1: Obtain source domain data and target domain data.

[0043] In this embodiment of the invention, the source domain data and target domain data are remote sensing images from the domestic remote sensing satellite GF-1WFV, acquired through the data sharing platform of the China Resources Satellite Application Center, serving as the source domain data (X domain). A total of 108 remote sensing images were collected. The X domain data was divided into a training set of 86 images and a test set of 22 images. Preprocessing was then performed on the X domain images. In the preprocessing stage, black-bordered areas containing no valid information were first removed, and then the images were uniformly rotated to 11264×11264×4. Each image was then cropped into sub-patterns of 512×512 pixels, ultimately resulting in 41,624 training samples and 10,648 test samples.

[0044] Sentinel-2A images from the WHUS2-CD benchmark dataset released by Wuhan University were selected as the target domain data (Y domain) for the cross-modal transfer task. In the preprocessing stage, the Y domain images were unified to a 60-meter spatial resolution using bilinear interpolation across the 10-meter and 20-meter resolution spectral bands, and a small number of invalid black borders were removed, resulting in images of 10752×10752×13 pixels. Each image was then cropped into 384×384 pixel blocks to fit the network input requirements. After processing, 18816 Y domain training samples were used to form the training set, and 9408 Y domain test samples were used to form the test set.

[0045] In embodiments of the present invention, the preprocessed training samples are used to train the remote sensing image cross-modal cloud migration network, and the preprocessed test samples are used to test the remote sensing image cross-modal cloud migration network.

[0046] Step 2: Construct a cross-modal cloud detection network for remote sensing images.

[0047] A cross-modal cloud detection network is constructed, comprising a spectral self-attention subnetwork and an unsupervised image domain transfer subnetwork. First, the spectral self-attention subnetwork is used to extract spectral features from the input image. Then, the unsupervised image domain transfer subnetwork generates a cross-modal image with rich spectral style. Finally, the generated cross-modal image is input into the cloud detection network to output the cloud detection results.

[0048] Step 2.1, construct the spectral self-attention module SAB.

[0049] Reference Figure 2 The spectral self-attention module SAB constructed in this invention will be further described below.

[0050] Input image X in Through the three parallel linear transformation paths of the spectral self-attention module SAB, the linear transformation corresponds to Figure 2 Linear, X in Each is associated with three learnable weight matrices W. Q W K W V Multiply to get X Q X K X V The parameters in the weight matrix are generated randomly during network initialization and continuously optimized through backpropagation during training. For X... K After performing matrix transpose and X Q Perform matrix multiplication to obtain attention map A, then combine A with X. V Matrix multiplication is performed to obtain the output feature map X. A The obtained X A After performing a linear transformation, the output feature X of the network is obtained by adding it to the original input feature. out .

[0051] Step 2.2: Construct a spectral self-attention quantum network.

[0052] Reference Figure 3 The spectral self-attention quantum network constructed in this invention will be further described below.

[0053] In the embodiments of this invention, the spectral self-attention subnetwork is composed of a U-shaped spectral self-attention module (SAU) and a two-branched U-shaped spectral self-attention module (DSAU) connected in series. The U-shaped spectral self-attention module (SAU) and the two-branched U-shaped spectral self-attention module (DSAU) respectively correspond to Figure 3 In Figure 3 (a) Figure 3 (b) The U-shaped spectral self-attention module (SAU) adopts an encoder-decoder structure. The input image size is H×W×C, where H is the image height, W is the image width, and C is the number of channels in the input image. The encoder part consists of a three-stage cascaded feature extraction structure: the first stage consists of an embedding convolutional layer with a kernel size of 1×1 and a stride of 1, a spectral self-attention module (SAB), and a downsampling convolutional layer with a kernel size of 3×3 and a stride of 2, cascaded together. Figure 3In the encoder, `Embedding` refers to the embedded convolutional layer, and `DownSample` refers to the downsampled convolutional layer. The second stage consists of a cascaded spectral self-attention module (SAB) and a downsampled convolutional layer with a kernel size of 3×3 and a stride of 2. The third stage is the bottom-level spectral self-attention module (SAB). The image sizes output by the encoder's first and second stages are H / 2×W / 2×2C and H / 2×W / 2×4C, respectively. The decoder is a two-stage cascaded decoding structure: the first stage consists of a cascaded spectral self-attention module (SAB) and an upsampled convolutional layer. The upsampling operation uses a deconvolutional layer with a kernel size of 3×3 and a stride of 2. Figure 3 UpSample in the code is an upsampling convolutional layer. The second stage consists of a cascaded spectral self-attention module (SAB) and an upsampling convolutional layer. The upsampling operation uses a deconvolutional layer with a kernel size of 3×3 and a stride of 2. The image sizes output by the first and second stages of the decoder are H / 2×W / 2×2C and H×W×C, respectively. The input image in the encoder is multiplied element-wise by the output features of the spectral self-attention module (SAB) in the first stage and the output features of the upsampled image in the second stage, and then input into the spectral self-attention module (SAB) of the second stage of the decoder. The output features of the second-stage spectral self-attention module (SAB) of the encoder and the upsampled output features of the first stage of the decoder are multiplied element-wise and then input into the first-stage spectral self-attention module (SAB) of the decoder. The output of the second-stage spectral self-attention module (SAB) of the decoder is connected to a convolutional layer with a kernel size of 3×3.

[0054] The dual-branch U-shaped spectral self-attention module (DSAU) adds a decoding branch to the U-shaped spectral self-attention module (SAU). The two decoding branches have the same structure as the decoder in the SAU module and are connected in parallel. The output features of the input image from the encoder, after passing through the first-stage spectral self-attention module (SAB), are element-wise multiplied with the upsampled output features of the second-stage decoder branches, and then simultaneously input into the second-stage spectral self-attention modules (SAB) of both decoders. The output features of the second-stage spectral self-attention modules (SAB) of the encoder, after element-wise multiplication with the upsampled output features of the first-stage decoder branches, are input into the first-stage spectral self-attention module of the decoder. Each of the outputs from the second-stage spectral self-attention modules (SAB) of the two decoder branches is connected to a convolutional layer with a 3×3 kernel.

[0055] Step 2.3: Construct an unsupervised image domain transfer subnetwork.

[0056] Reference Figure 4 The unsupervised image domain transfer subnetwork constructed in this invention will be further described below.

[0057] The constructed unsupervised image domain transfer subnetwork structure includes two generators and two discriminators, such as... Figure 4 As shown, the Generator is the generator, and the Discriminator is the discriminator. The original X-domain image first passes through the first generator. The first branch converts the X-domain image into a Y-domain image, i.e., the synthesized Setinel-2A data. The generated Y-domain image is then input into the first discriminator to determine whether it is the true target domain image. Another branch of the first generator outputs the predicted GF-1 cloud mask. The difference between the predicted GF-1 cloud mask and the true GF-1 cloud mask is measured by calculating the loss between the two. The second generator reconstructs the original X-domain image from the generated Y-domain image. The second discriminator then determines whether the reconstructed X-domain image is true and calculates the loss between the reconstructed GF-1 image and the true GF-1 image to evaluate the image reconstruction quality. Each generator consists of an initial convolutional layer cascaded with a spectral self-attention subnetwork. The kernel size of the initial convolutional layer is set to 1×1, and the kernel stride is 1. Each discriminator consists of an input convolutional layer, three intermediate convolutional layers, and an output convolutional layer. The input convolutional layer has a 3×3 kernel size, a kernel stride of 1, and 64 kernels. The three intermediate convolutional layers are all 5×5 layers with 128, 256, and 512 kernels respectively, and a kernel stride of 2 for each. Normalization and ReLU activation are applied after each convolutional layer. The output convolutional layer has a 5×5 kernel size and a kernel stride of 1.

[0058] Step 3, Design the loss function:

[0059] Step 3.1: Design the structure-style separation loss function. Perform a structure-contrast loss on the structure of each channel of the Y-domain image and the dark channel of the X-domain image, and perform a style-contrast loss on the style of each channel of the Y-domain image and the corresponding channel of the Y-domain image. The structure-style separation loss function is as follows:

[0060] L=l(x,y)·c(x,y)·s(x,x')

[0061]

[0062] Where l(x,y) represents brightness loss, used to measure the consistency of the generated image and the target image in overall brightness; c(x,y) represents contrast loss, used to compare local intensity changes in the image; s(x,x') represents structure loss, used to measure the similarity of the images in structural information; x represents the generated X-domain image, y represents the Y-domain image, and x' represents the dark channel of the X-domain image; μ represents the mean, σ represents the variance; C1, C2, and C3 represent adjustment factors, where C1 = (K1L) 2 C2 = (K2L) 2C3 = C2 / 2, K1 = 0.01, K2 = 0.03, and L represents the dynamic range of pixel values ​​in the range [0, 255].

[0063] Step 3.2, design the Gaussian edge loss function as follows:

[0064] L b =MSE(sg(x' m ),sg(x m ))+MSE(sg(y' m ),sg(y m ))

[0065] Where MSE represents mean squared error, and sg represents a combination of Sobel and Gaussian processing. Sobel processing calculates the gray-level gradients of the image in the horizontal and vertical directions using a 3×3 convolution kernel, while Gaussian processing convolves the image using a 3×3 Gaussian filter kernel. This involves weighting each pixel in the image and its eight neighboring pixels according to the weights in the filter kernel, and then averaging the sums. The weights in the filter kernel are determined by a two-dimensional Gaussian function G(i,j). i and j represent the coordinates of the offset position of a pixel relative to the center in the filter kernel, π represents pi, σ is the standard deviation, and exp represents an exponential function with the natural constant e as the base; x m 、x' m y m and y' m These represent the predicted cloud detection result in domain X, the actual cloud label in domain X, the predicted cloud detection result in domain Y, and the actual cloud label in domain Y, respectively.

[0066] Step 3.3, the formulas for the adversarial loss function and the cycle-consistent loss function are as follows:

[0067]

[0068] in, To minimize the loss of generator G, To maximize the loss of the discriminator D, log(·) represents the base-2 logarithmic operation, x ~ P data (x) represents real data from the training set, x ~ P G(X) This refers to generated samples created by the generator. Let $\frac{ ... This is the loss of the discriminator on the generated data, representing the expected value of the discriminator correctly classifying the generated data as "forgery".

[0069] In the cycle-consistent loss function, G and F represent two generator networks, x and y come from images in two different domains, and ||·|| denotes the L1 norm. x~P data (x) represents real data from the training set X, and y ~ P. data (y) represents real data from the Y domain of the training set.

[0070] Step 3.4, the overall loss function used for cross-modal transfer is the sum of all loss functions, and the formula for the overall loss function is as follows:

[0071] L=L(D,G)+L cyc +L s +L b

[0072] Where L(D,G) is the adversarial loss, L cyc For cycle-consistent loss, L s For structural style separation loss, L b It uses Gaussian edge loss.

[0073] Step 4: Train the network using cross-modal data.

[0074] Data generated from cross-modal transfer was input into the cloud detection network RS-Net, and trained using backpropagation and the AdamOptimizer. During training, binary cross-entropy loss was used as the loss function to iteratively update the network parameters until the loss function converged. The binary cross-entropy loss function is as follows:

[0075] L' = -ylogy' - (1-y)log(1-y')

[0076] Where y represents the actual cloud label, and y' represents the cloud detection result predicted by the network.

[0077] Step 5: Test the model:

[0078] The trained network model was applied to the test set to perform cloud detection on the input remote sensing images.

[0079] The effects of this invention will be further illustrated below with simulation experiments:

[0080] 1. Simulation conditions:

[0081] The hardware testing platform used in the simulation experiment of this invention is a server equipped with an AMD Epyc 730216-Core Processor, 128GB of DDR4 memory, and two NVIDIA GeForce RTX 3090 graphics cards.

[0082] The software platform used in this invention is the Ubuntu 18.04LTS operating system, and the experimental environment is based on the PyTorch 1.8.1 deep learning framework.

[0083] The datasets used in this invention include multispectral images from the domestic remote sensing satellite GF-1WFV and Sentinel-2A images from the WHUS2-CD dataset. The GF-1 dataset contains 108 images, which, after cropping and sliding window processing, generated 41,624 training samples and 10,648 test samples. The Sentinel-2A dataset contains 36 images, which, after uniform resampling and cropping, generated 18,816 training samples and 9,408 test samples, serving as source and target domain data for training and testing the cross-modal cloud detection network.

[0084] 2. Simulation content and result analysis:

[0085] The simulation experiment of this invention uses RS_Net as the basic cloud detection network. The training set is used to train the remote sensing image cross-modal cloud detection network constructed in this invention and two existing cross-modal networks (Pix2pix network and SwitchGAN network). After obtaining the trained network, the test set is input into each trained network to perform cloud detection and obtain the cloud detection results.

[0086] The two existing technologies used in the simulation experiment are:

[0087] The existing technology method Pix2pix network refers to an image-to-image translation network based on conditional generative adversarial networks, proposed by Isola et al. in their paper "Image-to-image translation with conditional adversarial networks" (CVPR 2017), abbreviated as Pix2pix network.

[0088] The existing technical method SwitchGAN refers to a switchable generative adversarial network based on the CycleGAN framework proposed by Yang et al. in their paper "Continuous conversion of CT kernel using switchable CycleGAN with AdaIN" (IEEE Transactions on Medical Imaging, 2021, 40(11): 3015–3029), abbreviated as SwitchGAN network.

[0089] The classification results of the four methods were evaluated using three metrics: overall accuracy (OA), F1-Score, and mean intersection-over-union (mIOU). The overall accuracy (OA), F1-Score, and mIOU were calculated using the following formulas, and all calculation results are plotted in Table 1.

[0090]

[0091] Table 1. Performance indicators of cloud detection by the present invention and various existing technologies in simulation experiments.

[0092]

[0093] As can be seen from Table 1, the overall accuracy (OA) of the method of the present invention is 97.50%, the F1-Score is 92.43%, and the average crossover ratio (mIoU) is 86.41%. These three indicators are all higher than the two existing technologies, proving that the present invention can achieve better detection performance in cross-modal cloud detection tasks.

Claims

1. A method for cross-modal cloud detection in remote sensing images guided by spectral self-attention and cloud masking, characterized in that, The detection method involves constructing a spectral self-attention subnetwork and an unsupervised image domain transfer subnetwork, respectively. The steps include the following: Step 1: Construct a cross-modal cloud detection network including a spectral self-attention subnetwork and an unsupervised image domain transfer subnetwork; Step 2: Select source domain X-domain images and target domain Y-domain images from remote sensing image data collected by different sensors to form a sample set. After preprocessing each image in the sample set, a training set is generated. Step 3: Design the structural style separation loss function and the Gaussian edge loss function; Step 4, train the remote sensing image cross-modal cloud detection network: Step 5: Perform cloud detection on the remote sensing image to be detected. Using the same method as in step 2, the preprocessed remote sensing image to be detected is input into the trained cross-modal cloud detection network for cloud detection, and the cloud detection results are output.

2. The detection method according to claim 1, characterized in that, The spectral self-attention subnetwork described in step 1 is composed of a U-shaped spectral self-attention module SAU and a bi-branched U-shaped spectral self-attention module DSAU connected in series.

3. The detection method according to claim 2, characterized in that, The U-shaped spectral self-attention module (SAU) adopts an encoder-decoder structure. The encoder part contains a three-stage cascaded feature extraction structure: the first stage consists of an embedded convolutional layer with a kernel size of 1×1 and a stride of 1, a spectral self-attention module (SAB), and a downsampling convolutional layer with a kernel size of 3×3 and a stride of 2; the second stage consists of a spectral self-attention module (SAB) and a downsampling convolutional layer with a kernel size of 3×3 and a stride of 2; the third stage is a spectral self-attention module (SAB). The decoder section is a two-stage cascaded decoding structure: the first stage consists of a spectral self-attention module (SAB) and an upsampling convolutional layer cascaded together. The upsampling operation uses a deconvolutional layer with a kernel size of 3×3 and a stride of 2. The second stage consists of a cascaded spectral self-attention module (SAB) and an upsampling convolutional layer. The upsampling operation uses a deconvolutional layer with a kernel size of 3×3 and a stride of 2. The output of the first stage of the encoder is connected to the input of the second stage of the decoding branch, and the output of the second stage of the encoder is connected to the input of the first stage of the decoding branch.

4. The detection method according to claim 2, characterized in that, The dual-branch U-shaped spectral self-attention module DSAU is a U-shaped spectral self-attention module SAU with an added decoding branch. The two decoding branches have the same structure as the decoder in the SAU module, and the two decoding branches are connected in parallel. The first-stage output of the encoder is simultaneously connected to the second-stage input of both decoding branches, and the second-stage output of the encoder is simultaneously connected to the first-stage input of both decoding branches. The input image X... in X is obtained after three linear transformations. Q X K X V , for X K After performing matrix transpose and X Q Perform matrix multiplication to obtain attention map A, then combine A with X. V Matrix multiplication is performed to obtain the output feature map X. A The obtained X A After performing a linear transformation, the output feature X of the network is obtained by adding it to the original input feature. out .

5. The detection method according to claim 1, characterized in that, The unsupervised image domain transfer subnetwork structure described in step 1 includes two generators and two discriminators. The original X-domain image is converted into a Y-domain image by the first generator and then input into the first discriminator to determine whether it is a real target domain image. The second generator reconstructs the original X-domain image from the Y-domain image, and then the second discriminator determines whether the reconstructed X-domain image is real. Each generator consists of an initial convolutional layer cascaded with a spectral self-attention subnetwork. The kernel size of the initial convolutional layer is set to 1×1, and the kernel stride is [missing value].

1. Each discriminator consists of an input convolutional layer, three intermediate convolutional layers, and an output convolutional layer. The kernel size of the input convolutional layer is set to 3×3, the kernel stride is 1, and the number of kernels is set to 64. The three intermediate convolutional layers are all 5×5 convolutional layers with the following kernel sizes: 128, 256, and 512, respectively. The kernel stride is 2 for each kernel. Normalization and ReLU activation are performed after each convolutional layer. The kernel size of the output convolutional layer is 5×5, and the kernel stride is 1.

6. The detection method according to claim 1, characterized in that, The steps for generating the training set in step 2 are as follows: The first step is to select remote sensing image data from different sources that contain cloud areas and various ground features as source domain X-domain images and target domain Y-domain images, respectively. Among them, there are at least 108 source domain images and at least 36 target domain images; and the cloud areas in the selected images contain various cloud types such as thin clouds, fragmented clouds and thick clouds. The second step is to crop the source domain image into a sub-pattern with a size of 512×512 pixels and the target domain image into a sub-pattern with a size of 384×384 pixels. The third step is to treat the cropped source domain sub-plots and target domain sub-plots as independent datasets to form an unpaired training set.

7. The detection method according to claim 1, characterized in that, The structural style separation loss function L mentioned in step 3 s as follows: L s =l(x,y)·c(x,y)·s(x,x') Where l(x,y) represents brightness loss, used to measure the consistency of the generated image and the target image in overall brightness; c(x,y) represents contrast loss, used to compare local intensity changes in the image; s(x,x') represents structure loss, used to measure the similarity of the images in structural information; x represents the generated X-domain image, y represents the Y-domain image, and x' represents the dark channel of the X-domain image; μ represents the mean, σ represents the variance; C1, C2, and C3 represent adjustment factors, where C1 = (K1L) 2 C2 = (K2L) 2 C3 = C2 / 2, K1 = 0.01, K2 = 0.03, and L represents the dynamic range of pixel values ​​in the range [0, 255].

8. The detection method according to claim 7, characterized in that, The Gaussian edge loss function L mentioned in step 3 b as follows: L b =MSE(sg(x' m ),sg(x m ))+MSE(sg(y' m ),sg(y m )) Where MSE represents mean squared error, and sg represents a combination of Sobel and Gaussian processing. Sobel processing calculates the gray-level gradients of the image in the horizontal and vertical directions using a 3×3 convolution kernel, while Gaussian processing convolves the image using a 3×3 Gaussian filter kernel. This involves weighting each pixel in the image and its eight neighboring pixels according to the weights in the filter kernel, and then averaging the sums. The weights in the filter kernel are determined by a two-dimensional Gaussian function G(i,j). i and j represent the coordinates of the offset position of a pixel relative to the center in the filter kernel, π represents pi, σ is the standard deviation, and exp represents an exponential function with the natural constant e as the base; x m 、x' m y m and y' m These represent the predicted cloud detection result in domain X, the actual cloud label in domain X, the predicted cloud detection result in domain Y, and the actual cloud label in domain Y, respectively.

9. The detection method according to claim 1, characterized in that, The steps for training the cross-modal cloud detection network in step 4 are as follows: input the training set into the cross-modal cloud detection network, train it using backpropagation and the Adam Optimizer, iteratively update the network parameters until the total loss function of the network converges, and obtain the trained cross-modal cloud detection network.

10. The detection method according to claim 8, characterized in that, The total loss function of the network is as follows: L=L(D,G)+L cyc +L s +L b Where L(D,G) represents the adversarial loss, L cyc L represents the cycle-consistent loss. s L represents the structural style separation loss. b This represents Gaussian edge loss.

Images