Cloud Detection Method Based on Spatial-Spectral Joint Multispectral Remote Sensing Image

Abstract

This invention proposes a multispectral remote sensing image cloud detection method based on spatial-spectral joint processing. The steps are as follows: acquiring training and testing sample sets; constructing a multispectral remote sensing image cloud detection network based on spatial-spectral joint processing and iteratively training the network; and obtaining the cloud detection results. The spatial-spectral information extraction module in the multispectral remote sensing image cloud detection network effectively extracts and fuses the spatial and spectral information of the multispectral remote sensing image. Furthermore, the context-based dynamic convolution module fuses the shallow features of the encoding module with the deep features of the decoding module, making the network more attentive to cloud edge information and identifying more continuous cloud boundaries, thereby effectively improving the accuracy of multispectral remote sensing image cloud detection.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a cloud detection method for multispectral remote sensing images. Specifically, it relates to a cloud detection method for multispectral remote sensing images based on spatial-spectral joint method, which can be used to classify and remove clouds in multispectral remote sensing images. Background Technology

[0002] Remote sensing satellite imagery consists of image data acquired by sensors in remote sensing satellites and is currently widely used in fields such as weather forecasting, geological disaster prevention and mitigation, and agricultural production. According to cloud cover data from the International Satellite Cloud Project, more than half of the Earth's surface is obscured by clouds most of the time, significantly reducing the amount of ground information obtainable and greatly hindering the development of remote sensing. Therefore, designing an efficient cloud detection algorithm is of great significance for the utilization and processing of remote sensing images.

[0003] Traditional cloud detection algorithms mostly employ spectral thresholding, analyzing the spectral characteristics of clouds and other areas in remote sensing images and selecting appropriate thresholds for different channels to achieve cloud detection. However, it is difficult to analyze the differences between clouds and highly reflective ground features solely based on spectral characteristics, and its generalization ability across different scenarios is poor, resulting in relatively low accuracy for cloud detection using traditional thresholding methods. In recent years, with the continuous development of hardware and the rise of deep learning theory, deep neural networks, represented by convolutional neural networks, can automatically extract and process features from images and have been widely applied in the field of remote sensing image processing. Currently, deep learning methods can achieve high accuracy in cloud detection in remote sensing images and show good results in some challenging problems, such as distinguishing clouds from highly reflective ground features. However, most current deep learning cloud detection methods are designed for three-channel images, and in the processing of multispectral remote sensing images, information extraction is often lacking, leading to a decrease in cloud detection accuracy. For example, application publication number CN 114494821 A, entitled "A Remote Sensing Image Cloud Detection Method Based on Feature Multi-Scale Perception and Adaptive Aggregation," provides a remote sensing image cloud detection method, UNet-Cloud, based on feature multi-scale perception and adaptive aggregation. Since the semantic information of clouds and cloud shadows differs at different scales, the feature multi-scale perception module of this invention perceives contextual information at different scales through parallel dilated convolution between the encoder and decoder. The feature adaptive aggregation module, based on a self-attention mechanism, adaptively learns the contribution of features at different scales to cloud and cloud shadow detection, achieving weighted aggregation of multi-scale features. The vector post-processing workflow includes operations such as hole filling, buffer analysis, edge smoothing, and erasing, thereby solving problems such as fragmented and jagged areas in some regions of the detection results from deep learning networks. Ultimately, it obtains an effective image area that is closer to, or even more accurate than, manually produced effective image areas, saving significant time and manpower costs for remote sensing image production. Furthermore, the final detection effect is superior, facilitating subsequent image applications. However, due to the large amount of interference information contained in multispectral remote sensing images, this network did not focus on extracting the spectral information of multispectral remote sensing images. Therefore, when performing cloud detection on multispectral remote sensing images, there are problems such as false detection and missed detection, especially in the detection of cloud boundaries and the differentiation between clouds and highly reflective ground features. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a cloud detection method for multispectral remote sensing images based on spatial-spectral joint method, which is used to solve the technical problem of high false positives and false negatives caused by the lack of information in multispectral remote sensing images in the existing technology.

[0005] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0006] (1) Obtain the training sample set and the test sample set:

[0007] K multispectral remote sensing images with cloud region labels were obtained from the satellite database. Each multispectral remote sensing image was normalized. M normalized multispectral remote sensing images and their corresponding labels were selected as the training sample set, and the remaining N normalized multispectral remote sensing images and their corresponding labels were selected as the test sample set, where K > 10000 and M > N.

[0008] (2) Constructing a multispectral remote sensing image cloud detection network H based on spatial-spectral joint model:

[0009] A multispectral remote sensing image cloud detection network H is constructed, comprising a sequentially cascaded input layer, an encoding module, a decoding module, and an output layer, as well as multiple context-dynamic convolutional modules CD. The encoding module includes multiple sequentially cascaded convolutional layers and multiple downsampling modules D. Each downsampling module D includes a spatial-spectral information extraction module SC composed of a parallel spatial feature extraction module S and a spectral feature extraction module C, and a max-pooling layer cascaded with SC. The decoding module includes multiple sequentially cascaded upsampling modules U and multiple convolutional layers. Each upsampling module U includes a sequentially cascaded convolutional layer, a regularization layer, and an upsampling layer. The number of upsampling modules U in the decoding module is equal to the number of downsampling modules D in the encoding module. Each downsampling module D is connected to its symmetrically positioned upsampling module U through a context-dynamic convolutional module CD. The output layer includes sequentially cascaded convolutional layers and activation function layers.

[0010] (3) Iteratively train the cloud detection network H for multispectral remote sensing images:

[0011] (3a) Initialize the number of iterations to r, the maximum number of iterations to R, R≥30, and the multispectral remote sensing image cloud detection network H in the r-th iteration. r The weight parameter is α r And let r = 0;

[0012] (3b) The training sample set is used as the input of the multispectral remote sensing image cloud detection network H. Multiple convolutional layers in the encoding module extract the overall features of each training sample. After the spatial and spectral information extraction module SC in multiple downsampling modules D extracts the spatial and spectral information of the overall features, it is pooled by the max pooling layer to obtain multiple spatial and spectral information at different scales. Multiple context dynamic convolution modules CD adjust the output features of the last downsampling module D into an adaptive convolution kernel and fuse the spatial and spectral information at different scales into the decoding module through the adaptive convolution kernel. Then, the decoding module decodes the fused information, and the output layer performs binary classification on the decoding result to obtain the cloud detection prediction results of M training samples.

[0013] (3c) Using the Focalloss loss function, calculate the loss value θ between the cloud detection prediction result and the corresponding true label for each training sample. m And using gradient descent, through the loss value θ m With weight α r The partial derivative of the parameter with respect to α r The network was updated to obtain the multispectral remote sensing image cloud detection network H after this iteration. r ;

[0014] (3d) Determine whether r = R holds true. If so, obtain the trained multispectral remote sensing image cloud detection network H. R Otherwise, let r = r + 1, H r =H, and perform step (3b);

[0015] (4) Obtain the cloud detection prediction results of the test sample set:

[0016] The test sample set is used as the trained multispectral remote sensing image cloud detection network H. R The input is propagated forward, utilizing the network's weights α. R Obtain the cloud detection prediction results for the corresponding test sample images.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] First, the multispectral remote sensing image cloud detection network constructed in this invention includes a spatial-spectral information extraction module composed of a parallel spatial feature extraction module and a spectral feature extraction module. During the iterative training of this network and the acquisition of multispectral remote sensing image cloud detection results, the spatial-spectral information extraction module can extract spatial information and spectral information separately and fuse them. This effectively extracts the spatial-spectral information of multispectral remote sensing images, avoiding the shortcomings of existing multi-scale sensing methods that only focus on spatial scale information while ignoring spectral information. This reduces the probability of missed detections and false detections, and effectively improves the cloud detection accuracy of multispectral remote sensing images.

[0019] Second, the multispectral remote sensing image cloud detection network constructed in this invention contains multiple context-dynamic convolution modules, which can adjust the spatial spectrum information output by the last downsampling module D into multiple adaptive convolution kernels and fuse spatial spectrum information of different scales into the decoding module through the adaptive convolution kernels. This makes the network pay more attention to cloud edge information and long-distance information, the identified cloud boundaries are more continuous, and the identification of cloud regions is more complete, further improving the cloud detection accuracy of multispectral remote sensing images. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0021] Figure 2 This is a schematic diagram of the structure of the remote sensing image cloud detection network constructed in this invention.

[0022] Figure 3 This is a schematic diagram of the spatial spectrum information extraction module constructed according to an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the structure of the context-based dynamic convolution module constructed in this invention. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Reference Figure 1 The present invention includes the following steps:

[0026] Step 1) Obtain the training sample set and the test sample set:

[0027] Two hundred thousand multispectral remote sensing images with cloud region labels were obtained from the dataset as the overall sample set, and each image was normalized. Two hundred thousand Sentinel-2 remote sensing satellite images and their manually labeled cloud region labels were also selected as the overall sample set. The label size was the same as the original remote sensing satellite image size, and the images were binarized. If a pixel represented a cloud, the corresponding pixel value was 255; otherwise, the pixel value was 0. The Sentinel-2 remote sensing satellite images have 13 channels, including RGB, near-infrared, and far-infrared channels, making them multispectral remote sensing images with rich spectral and spatial information. One hundred and fifty thousand images and their corresponding labels from the overall sample set were selected as the training sample set, and the remaining five hundred thousand images and their corresponding labels were selected as the test sample set.

[0028] Step 2) Construct a multispectral remote sensing image cloud detection network H based on spatial-spectral joint model:

[0029] A multispectral remote sensing image cloud detection network H is constructed, comprising a sequentially cascaded input layer, an encoding module, a decoding module, and an output layer, as well as multiple context-dynamic convolutional modules CD. The encoding module includes multiple sequentially cascaded convolutional layers and multiple downsampling modules D. Each downsampling module D includes a sequentially cascaded spatial-spectral information extraction module SC and a max-pooling layer. The spatial-spectral information extraction module SC includes a parallel spatial feature extraction module S and a spectral feature extraction module C. The decoding module includes multiple sequentially cascaded upsampling modules U and multiple convolutional layers. Each upsampling module U includes a sequentially cascaded convolutional layer, a regularization layer, and an upsampling layer. The number of upsampling modules U in the decoding module is equal to the number of downsampling modules D in the encoding module. Each downsampling module D is connected to its symmetrically positioned upsampling module U through a context-dynamic convolutional module CD. The output layer includes sequentially cascaded convolutional layers and activation function layers.

[0030] The spatial and spectral information extraction module SC in the downsampling module D is used to extract spatial and spectral information. The pooling window size of the max pooling layer is 2×2, which can remove redundant information in the image, enhance the invariance of image features, and retain more texture information.

[0031] The spatial spectrum information extraction module SC, such as Figure 3 As shown, the spatial spectral feature extraction module S consists of four densely connected cascaded 3D convolutional layers, a batch regularization layer, and an activation function layer. The spectral feature extraction module C also consists of four densely connected cascaded 3D convolutional layers, a batch regularization layer, and an activation function layer. The 3D convolutional layer kernel size in the spatial spectral feature extraction module S is 3×3×1, while the 3D convolutional layer kernel size in the spectral feature extraction module C is 1×1×3. The batch regularization layer standardizes the data, ensuring the output follows a normal distribution with a mean of 0 and a variance of 1. This accelerates network training and prevents gradient vanishing and gradient exploding. The activation function layer uses the ReLU activation function, enhancing the network's nonlinear representation capability. The spatial-spectral joint module employs a densely connected approach and uses 3D convolutional layers. By setting different 3D convolutional kernel sizes for the two modules, the spatial and spectral information of the multispectral remote sensing image can be extracted more effectively and fully.

[0032] In the upsampling module U, the convolutional kernel size is 3×3, the kernel movement stride is 1, and the regularization layer adopts dropout regularization, which can reduce the complex co-adaptation relationship between neurons and prevent the network from overfitting. The overall structure of the upsampling module U is symmetrical with that of the downsampling module D, which is conducive to the fusion of features at both ends of the network.

[0033] Context-based dynamic convolution module CD Figure 4 As shown, the deepest spatial spectral information features are convolved with 3×3 convolution kernels formed by 1×1 convolution kernels to encode features. By representing deep features, the network's representation ability can be effectively improved, and the network can pay more attention to the edge information of the cloud.

[0034] The output layer has a 1×1 kernel size and a stride of 1 for the convolutional layer. The activation function is the softmax activation function, which normalizes the prediction results to obtain the binary classification result of cloud detection.

[0035] Step 3) Iteratively train the remote sensing image cloud detection network H:

[0036] (3a) Let the initial number of iterations be r, the maximum number of iterations be , and let r = 0. The multispectral remote sensing image cloud detection network H in the r-th iteration is... r The weight parameter is αr ;

[0037] (3b) The training sample set is used as the input to the multispectral remote sensing image cloud detection network H. First, it passes through two convolutional layers in the encoding module to extract the overall features of the input sample. Then, it enters the spatial-spectral information extraction module SC in the downsampling module D. The spatial information extraction module S and the spectral information extraction module C, which are in parallel, extract spatial and spectral information respectively. Since the spatial information extraction module and the spectral information extraction module are composed of densely connected three-dimensional convolutional layers, they can extract rich spatial and spectral information. After pooling by the max pooling layers in the four downsampling modules D, each input sample obtains four different scales of spatial-spectral information. Among them, the spatial-spectral information obtained after the last downsampling module D is... As deep features, they enter the context dynamic convolution module CD to obtain an adaptive convolution kernel with a kernel size of 3×3, which can adaptively adjust the network parameters. Spatial spectral information at different scales is fused into the decoding module after passing through the adaptive convolution kernel, so that the encoded features and decoded features are fused together, enabling the network to focus on long-range information and pay more attention to the edges of clouds. The fused features enter the decoding module and are decoded step by step through each upsampling module U to obtain the final decoded information. The decoded information passes through the output layer and is activated by the softmax function to obtain the predicted probability of each pixel in the sample. A threshold method is used for binary classification. If the probability value is greater than 0.5, it is predicted as a cloud region with a pixel value of 255; otherwise, it is predicted as a non-cloud region with a pixel value of 0.

[0038] (3c) Using the Focalloss loss function, the loss between the cloud detection prediction result and the corresponding ground truth label for each training sample is calculated to obtain the corresponding loss value θ. m And use gradient descent to pass through the loss value θ m With weight α r The partial derivative of the parameter with respect to α r Update, θ m Let represent the loss value between the cloud detection prediction result and the corresponding true label for the m-th training sample. The update formula is:

[0039]

[0040] Where η represents the learning step size, η = 0.01, α r+1 Represents α r The update results This represents partial derivative operations.

[0041] (3d) Determine whether r = R holds true. If so, obtain the trained remote sensing cloud detection network H. R Otherwise, let r = r + 1, H r =H, and perform step (3b);

[0042] Step 4) Obtain cloud detection results from remote sensing images:

[0043] The test sample set is used as the trained multispectral remote sensing image cloud detection network H. R The input is propagated forward, utilizing the network's weights α. R Obtain the cloud detection prediction result for the corresponding test sample image. Each pixel in the cloud detection result image represents the detection result of the corresponding pixel in the input image. If the pixel is detected as a cloud pixel, the corresponding pixel in the cloud detection result is 255; otherwise, it is 0.

Claims

1. A cloud detection method for multispectral remote sensing images based on spatial-spectral joint method, characterized in that, Includes the following steps: (1) Obtain the training sample set and the test sample set: K multispectral remote sensing images with cloud region labels were obtained from the satellite database. Each multispectral remote sensing image was normalized. M normalized multispectral remote sensing images and their corresponding labels were selected as the training sample set, and the remaining N normalized multispectral remote sensing images and their corresponding labels were selected as the test sample set, where K > 10000 and M > N. (2) Constructing a multispectral remote sensing image cloud detection network H based on spatial-spectral joint model: A multispectral remote sensing image cloud detection network H is constructed, comprising a sequentially cascaded input layer, an encoding module, a decoding module, and an output layer, as well as multiple context-dynamic convolutional modules CD. The encoding module includes multiple sequentially cascaded convolutional layers and multiple downsampling modules D. Each downsampling module D includes a spatial-spectral information extraction module SC composed of a parallel spatial feature extraction module S and a spectral feature extraction module C, and a max-pooling layer cascaded with SC. The decoding module includes multiple sequentially cascaded upsampling modules U and multiple convolutional layers. Each upsampling module U includes a sequentially cascaded convolutional layer, a regularization layer, and an upsampling layer. The number of upsampling modules U in the decoding module is equal to the number of downsampling modules D in the encoding module. Each downsampling module D is connected to its symmetrically positioned upsampling module U through a context-dynamic convolutional module CD. The output layer includes sequentially cascaded convolutional layers and activation function layers. (3) Iteratively train the cloud detection network H for multispectral remote sensing images: (3a) Initialize the number of iterations to r, the maximum number of iterations to R, R≥30, and the multispectral remote sensing image cloud detection network H in the r-th iteration. r The weight parameter is α r And let r = 0; (3b) The training sample set is used as the input of the multispectral remote sensing image cloud detection network H. Multiple convolutional layers in the encoding module extract the overall features of each training sample. After the spatial and spectral information extraction module SC in multiple downsampling modules D extracts the spatial and spectral information of the overall features, it is pooled by the max pooling layer to obtain multiple spatial and spectral information at different scales. Each context dynamic convolution module CD adjusts the spatial and spectral information output by the last downsampling module D into multiple adaptive convolutional kernels and fuses the spatial and spectral information at different scales into the decoding module through the adaptive convolutional kernels. Then, the decoding module decodes the fused information, and the output layer performs binary classification on the decoding result to obtain the cloud detection prediction results of M training samples. (3c) Using the Focalloss loss function, calculate the loss value θ between the cloud detection prediction result and the corresponding true label for each training sample. m And using gradient descent, through the loss value θ m With weight α r The partial derivative of the parameter with respect to α r The network was updated to obtain the multispectral remote sensing image cloud detection network H after this iteration. r ; (3d) Determine whether r = R holds true. If so, obtain the trained multispectral remote sensing image cloud detection network H. R Otherwise, let r = r + 1, H r =H, and perform step (3b); (4) Obtain the cloud detection prediction results of the test sample set: The test sample set is used as the trained multispectral remote sensing image cloud detection network H. R The input is propagated forward, utilizing the network's weights α. R Obtain the cloud detection prediction results for the corresponding test sample images.

2. The cloud detection method for multispectral remote sensing images based on spatial-spectral joint method as described in claim 1, characterized in that, The multispectral remote sensing image cloud detection network H mentioned in step (2) includes: The encoding module contains two convolutional layers and four downsampling modules D. The convolutional kernel size of each convolutional layer is 3×3, and the kernel stride is 1. The max-pooling window size in the downsampling module D is 2×2. The spatial feature extraction module S consists of multiple densely connected cascaded three-dimensional convolutional layers, batch regularization layers, and activation function layers. The kernel size of the three-dimensional convolutional layers is a×a×p, where a < 5 and p < 3. The spectral feature extraction module C consists of multiple densely connected cascaded three-dimensional convolutional layers, batch regularization layers, and activation function layers. The kernel size of the three-dimensional convolutional layers is 1×1×a, where a < 5. The decoding module contains four upsampling modules U and two convolutional layers, where all convolutional layers have a kernel size of 3×3 and a kernel stride of 1. The output layer contains convolutional layers with a kernel size of 1×1 and a stride of 1, and activation function layers with a softmax activation function. The context dynamic convolution module CD, wherein deep features are passed through convolutional layers to form convolutional layers with a kernel size of 3×3, wherein the kernel size of the convolutional layer is 1×1, and the stride is 1.

3. The cloud detection method for multispectral remote sensing images based on spatial-spectral joint method as described in claim 1, characterized in that, The Focalloss loss function mentioned in step (3c) is expressed as follows: Where y (m) Let y' represent the true label of the m-th training sample. (m) This represents the prediction result for the m-th training sample.

Images