Fine segmentation and detection method of clouds and cloud shadows in remote sensing images based on deep learning

Abstract

The present invention discloses a method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning, which belongs to the field of image processing. The detection method comprises: obtaining and processing a training data set, dividing the data set into a training set and a validation set; constructing a segmentation network for deep learning semantics; using the data set to train a network model in the segmentation network; using the trained network model to predict and generate mask images of clouds and cloud shadows; using the segmentation network ResNet-18 network as the backbone to extract feature information at different levels, strengthening the model's attention to channel information and spatial information through a multi-scale global attention module to improve the accuracy of segmentation, and using a strip pyramid channel attention module to learn spatial information at multi-scale to better detect small cloud clusters. Finally, through a hierarchical feature aggregation module, high-dimensional features are fused with low-dimensional features, and layer-by-layer upsampling is performed to obtain the final segmentation effect. The method can detect smaller cloud clusters and obtain finer edges.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning. Background Art

[0002] With the advancement of remote sensing technology, remote sensing images are widely used in many fields such as military, agriculture, and meteorology. In remote sensing images, all ground targets may be obscured by clouds or fall into cloud shadows, resulting in lower brightness and visibility than the actual ones. This affects our exploration of real ground targets, so the identification and segmentation of clouds and cloud shadows in remote sensing images is a very important issue.

[0003] The existing cloud detection methods mainly rely on manual identification based on the thresholds of different cloud and cloud shadow bands or by utilizing the normalized differences between different cloud bands. However, traditional algorithms generally have the problems of complex operation, requiring a lot of manpower for feature extraction, and low detection accuracy.

[0004] In recent years, with the rise of artificial intelligence, many detection methods using machine learning and deep learning have emerged. Many scholars have applied deep learning to cloud detection, such as J (2018) proposed a segmented convolutional neural network for cloud classification. Li Z et al. (2019) proposed a multi-scale convolutional feature fusion algorithm for cloud detection. Xu K (2019) used the deep convolutional neural network ResNet to study pixel-level cloud mask generation. Chai D (2019) proposed a deep neural network-based method to detect clouds and cloud shadows in Landsat images. Segal-Rozenhaimer M (2020) proposed a cloud detection algorithm based on the red, green, blue and near-infrared channels of remote sensing images, attempting to use the inherent spectral and spatial information in satellite images to learn depth-invariant features for cloud detection.

[0005] Although deep learning has shown great results in the field of image processing, the performance of many networks in specific scenarios is not so ideal. In the task of detecting clouds and cloud shadows, most networks have a series of problems such as inaccurate segmentation results, unsatisfactory edge information recovery, inaccurate detection of many thin clouds, and interference from complex backgrounds when detecting cloud shadows. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a fine segmentation and detection method for clouds and cloud shadows in remote sensing images based on deep learning.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] The deep learning-based fine segmentation and detection method for clouds and cloud shadows in remote sensing images includes the following steps:

[0009] S1, obtain and process the training data set, and divide the data set into a training set and a validation set;

[0010] S2, building a segmentation network for deep learning semantics;

[0011] S3, using the data set obtained in S1, trains the network model in the segmentation network established in S2;

[0012] S4 uses the trained network model to predict and generate mask images of clouds and cloud shadows.

[0013] Furthermore, in S1, the original images of size 4800×2742 pixels are uniformly cropped to 224×224, and finally images of size 3×224×224 are obtained. Then all images are divided into training set and validation set in a ratio of 8:2.

[0014] Furthermore, the network constructed in S2 includes: backbone network ResNet-18, multi-scale global attention module, strip pyramid channel attention module and hierarchical feature aggregation module;

[0015] Among them, the backbone network ResNet-18 extracts feature information at different levels, strengthens the model's attention to channel information and spatial information through the multi-scale global attention module, and uses the strip pyramid channel attention module to learn spatial information at multiple scales. Finally, the hierarchical feature aggregation module is used to fuse high-dimensional features with low-dimensional features, and upsamples layer by layer to obtain the final segmentation effect.

[0016] Furthermore, the multi-scale global attention module includes a hierarchical multi-scale convolution module, a global spatial attention module, and a global channel attention module; the implementation process of the multi-scale global attention module is:

[0017] Shallow feature F L ∈R C×H×W After the layered multi-scale convolution module, the features of shallow feature-rich spatial information are obtained. Then combine it with the high-dimensional feature F output by the next layer of backbone network H ∈R 2C×H / 2×W / 2 Add to get the preliminary features F∈R 2C×H / 2×W / 2 ; The preliminary feature F is passed through the parallel global spatial attention module and channel attention module to obtain the weight W S and W C ; The above process formula is expressed as:

[0018]

[0019]

[0020]

[0021] in, Represents the permutation operation, which is used to adjust the dimension of the matrix, converting the matrix dimension from H*W*C to C*W*H. MLP represents the full connection operation in the channel dimension, and σ represents the Sigmoid function.

[0022] The final weight W s and W C Multiplying with the preliminary feature F, we get the feature M that strengthens the spatial and channel attention of the target on a global scale s and M C , and combine it with the initial feature F after convolution C Add and finally output F O ∈R 2C×W / 2×H / 2 ; The above process can be expressed mathematically as:

[0023]

[0024] The formula of the hierarchical multi-scale convolution module is expressed as:

[0025]

[0026] Get the output z i After that, i Merge in the channel dimension, and finally adjust the number of channels to the desired number for output through 1×1 convolution.

[0027] Furthermore, the strip pyramid channel attention module includes strip convolution and a squeeze-excitation weight module (SEWeight); the strip pyramid channel attention module is implemented through the following four steps:

[0028] 1) Use four strip convolutions with different kernel sizes to extract features of different scales and construct a feature pyramid;

[0029] 4) Use SEWeight module to calculate the channel attention of different feature blocks;

[0030] 5) Recalibrate the channel attention vector using Softmax to obtain the recalibrated weights of the multi-scale channels;

[0031] 4) Multiply the recalibrated weights and the corresponding features by their corresponding elements, and finally obtain a refined feature map of multi-scale feature information.

[0032] Furthermore, the implementation process of the compression-excitation weight module can be mathematically expressed as:

[0033]

[0034] w c =σ(W1r(W0(g c ))

[0035] Among them, GAP represents global average pooling, r represents the ReLU function, W0 is a fully connected network with input C and output C / r, W1 is a fully connected network with input C / r and output C, and the symbol σ represents the activation function Sigmoid.

[0036] Furthermore, the hierarchical feature aggregation module aggregating shallow and deep features includes the following steps:

[0037] 1) The shallow feature F1 is improved through the self-attention module to obtain the shallow feature F11 with enhanced position information;

[0038] 2) The deep feature F2 is upsampled by 3×3 convolution and bilinear interpolation, and its size is changed to F21, the same as the shallow feature;

[0039] 3) Weight it and multiply the corresponding elements with the shallow features to obtain the shallow features U1 with enhanced classification ability;

[0040] 4) The shallow feature F1 is average pooled to obtain F12 of the same size as the deep feature;

[0041] 5) F12 is multiplied by the deep feature F22 that has undergone two layers of 3×3 convolution and Sigmoid weighting to obtain the deep feature U2 that enhances spatial information;

[0042] 6) Finally, U1 is added to U2 which has been upsampled by bilinear interpolation and sent to a 3×3 convolution to complete the fusion of deep features and shallow features.

[0043] Furthermore, S3 adopts the supervised training mode, and the specific steps are as follows:

[0044] S31, first perform data augmentation operations on the images in the dataset, and randomly rotate, horizontally flip, or vertically flip the images in the dataset with a probability of 0.2;

[0045] S32, convert the image and label into tensors. The image is used as the input sequence of the model. The output and label are used to calculate the loss function. The batch size of each input is set to 8. The "ploy" strategy is used to reduce the learning rate accordingly as the number of training times increases to achieve better training results. Iter is the current number of iterations, the iteration coefficient power is set to 0.9, and the maximum number of training times maxiter is 300.

[0046] The learning rate calculation formula is as follows:

[0047]

[0048] Select cross entropy as the maximum loss function, the formula is expressed as:

[0049]

[0050] Where x is the predicted image output by the model, class is the label, and the Adam optimizer is used.

[0051] Furthermore, in S4, when making predictions, the model parameters with the highest MioU on the validation set during the training iteration are used for prediction; a color image is input, and the features are extracted using the trained weights, and encoding and decoding operations are performed to output a mask image of the cloud and cloud shadow.

[0052] A deep learning-based fine segmentation and detection system for clouds and cloud shadows in remote sensing images, including:

[0053] Data processing module: used to obtain and process the training data set and divide the data set into training set and validation set;

[0054] Network building module: building a segmentation network for deep learning semantics;

[0055] Training module: Use the data set to train the network model in the segmentation network;

[0056] Image generation module: Use the trained network model to predict and generate mask images of clouds and cloud shadows.

[0057] Beneficial effects of the present invention: The detection method of the present invention can effectively distinguish between clouds and cloud shadows. Compared with the current method, it is more accurate in detecting clouds and cloud shadows, can effectively reduce the interference of background information, and enhance the detection ability of cloud edges and thin clouds. Similarly, the network model proposed in the present invention also has good effects in segmenting other objects and has excellent generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0059] Figure 1 It is the overall structure of the CRSNet model in the present invention;

[0060] Figure 2 It is the multi-scale global attention module MGA module in the present invention;

[0061] Figure 3 It is the hierarchical multi-scale convolution module HMC in the present invention;

[0062] Figure 4 It is the compression-excitation weight module SEWeight in the present invention;

[0063] Figure 5 It is the strip pyramid channel attention module SPCA in the present invention;

[0064] Figure 6 It is the middle-level feature aggregation module HFA of the present invention;

[0065] Figure 7 It is the remote sensing image input and the mask image generated in the present invention. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0067] like Figure 1 As shown in FIG, a method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning includes the following steps:

[0068] S1, obtain the training data set, which includes a training set and a validation set;

[0069] The cloud and cloud shadow datasets used are mainly from high-resolution remote sensing image data taken by the Landsat 8 satellite and selected from Google Earth (GE); the original image size directly obtained from GE is 4800×2742 pixels. Due to the limitation of graphics card memory, the original images are uniformly cropped to 224×224 size. The cropping method is sliding coverless cutting, and finally a 3×224×224 size image is obtained, indicating that the cropped image has 3 channels and a width and height of 224. Then all images are divided into training and validation sets in a ratio of 8:2.

[0070] The selected images include multiple angles, heights and backgrounds. The purpose is to ensure the complexity of the image background so that the data will be more authentic. The image background mainly includes woodlands, waters, deserts, buildings, farmlands and other scenes.

[0071] S2 builds a deep learning semantic segmentation network. The model inputs remote sensing images converted into tensors and outputs images that classify clouds, cloud shadows, and background in the image at the pixel level.

[0072] like Figure 1 As shown, the segmentation network includes a ResNet-18 backbone, a multi-scale global attention module (MGA), a strip-pyramid channel attention module (SPCA), and a hierarchical feature aggregation module (HFA). The fitted residual mapping introduced by ResNet effectively addresses the problems of vanishing and exploding gradients in deep networks, enabling better feature extraction at different levels. During feature extraction, each layer of features undergoes the MGA module to strengthen spatial and channel attention, improving the accuracy of cloud and cloud shadow segmentation. Simultaneously, each layer of downsampled features after MGA undergoes further independent multi-scale spatial information learning through the SPCA module, establishing cross-channel interactions in a local manner to enhance the detection of clouds and cloud shadows at different scales. Finally, the HFA module strengthens the contextual connection between each layer of features while fusing high-dimensional and low-dimensional features layer by layer, fully leveraging the spatial information preserved by shallow features and the rich semantic information of deep features.

[0073] The specific steps to build a segmentation network are:

[0074] S21, build a multi-scale global attention module;

[0075] like Figure 2 As shown, the multi-scale global attention module includes a hierarchical multi-scale convolution module (HMC), a global spatial attention module, and a global channel attention module;

[0076] High-resolution features often inevitably contain noise, so convolution is used to pre-filter the information. Multi-scale features are very important for many image tasks. Most existing methods represent multi-scale features in a hierarchical manner. By building hierarchical connections, multi-scale features can be represented in a finer-grained manner and the receptive field of each layer can be increased. In order to achieve the above functions, the following methods are designed: Figure 3 The hierarchical multi-scale convolution module (HMC) shown in the figure filters the feature map before attention enhancement; the hierarchical multi-scale convolution module performs convolution of different scales on different channels of the feature vector, and iteratively adds the results and repeats the convolution to obtain more comprehensive feature information; the filtered features are passed through the global spatial attention module and the global channel attention module; both the global spatial attention and the global channel attention parts discard the pooling operation commonly used in previous attention mechanisms (such as SENet and CBAM), reducing the loss of spatial information; the global channel attention module also avoids the traditional method of expanding the features into one dimension for full connection to obtain channel dimension weights, so that the relationship between channels can be more completely preserved.

[0077] In terms of implementation details, the relatively high-resolution shallow feature F L ∈R C×H×W After the layered multi-scale convolution module, the features of shallow feature-rich spatial information are obtained. Then combine it with the high-dimensional feature F output by the next layer of backbone network H ∈R 2C×H / 2×W / 2 Add to get the preliminary features F∈R 2C×H / 2×W / 2 ; The preliminary feature F is passed through the parallel global spatial attention module and channel attention module to obtain the weight W S and W C ; The above process formula is expressed as:

[0078]

[0079]

[0080]

[0081] in, Represents the permutation operation, which is used to adjust the dimension of the matrix, converting the matrix dimension from H*W*C to C*W*H. MLP represents the full connection operation in the channel dimension, and σ represents the Sigmoid function.

[0082] The final weight W s and W C Multiplying with the preliminary feature F, we get the feature M that strengthens the spatial and channel attention of the target on a global scale s and MC , and combine it with the initial feature F after convolution C Add and finally output F O ∈R 2C×W / 2×H / 2 ; The above process can be expressed mathematically as:

[0083]

[0084] The formula of the hierarchical multi-scale convolution module (HMC) is expressed as:

[0085]

[0086] Get the output z i After that, i Merge in the channel dimension, and finally adjust the number of channels to the desired number for output through 1×1 convolution.

[0087] S22, constructing a strip pyramid channel attention module;

[0088] In the process of downsampling features layer by layer, semantic information will be gradually lost, so it is crucial for deep neural networks to capture long-range correlations; however, the receptive field of convolution is limited, and it is difficult to capture the long-range correlations of features; in order to improve the semantic information of deep features, it is necessary to increase the receptive field. In the past, in models involving pyramid structures, pooling was often used to obtain feature information of different scales, and pooling is very serious for the loss of details, which will cause the pooling pyramid model to be only effective for the recognition of large-scale targets in the cloud and cloud shadow detection task, and it is not helpful for the detection of small cloud clusters and cloud shadows. In order to make up for this shortcoming, the present invention proposes to use strip convolution to extract features of different scales. There are three benefits of using strip convolution to extract features: 1) Using convolution instead of pooling for downsampling can reduce information loss; 2) Strip convolution deploys a long strip pooling kernel shape along one spatial dimension, so it can capture long-distance relationships in isolated areas; and, strip convolution maintains a narrow kernel shape in the spatial dimension, which makes it easier to capture local context and prevent irrelevant areas from interfering with predictions; 3) In the pyramid structure, as the size of the convolution kernel increases, the number of parameters also increases greatly, and the strip convolution module can reduce memory consumption.

[0089] The channel attention mechanism allows the network to adapt the weight of each channel to generate more representative features. Therefore, after obtaining features at different scales, each feature block is passed through a compression-excitation weight module (SEWeight) to obtain its own channel weight vector, which is then passed through Softmax and multiplied by the corresponding features, allowing each feature block to independently learn multi-scale spatial information. The compression-excitation weight module consists of two parts: compression and excitation. The compression part is used to encode global information, and the excitation part is used to adaptively generate importance weights between channels. Global average pooling is used to embed global spatial information into channel information.

[0090] The structure of the compression-excitation weight module is as follows Figure 4 As shown; the process of compression-excitation weight module can be expressed mathematically as:

[0091]

[0092] w c =σ(W1r(W0(g c ))

[0093] Among them, GAP represents global average pooling, r represents the ReLU function, W0 is a fully connected network with input C and output C / r, and W1 is a fully connected network with input C / r and output C. The two fully connected operations can effectively and adaptively learn the relationship between channels. The symbol σ represents the activation function Sigmoid.

[0094] Through the strip convolution and compression-excitation weight module (SEWeight) proposed above, as Figure 5 As shown, the present invention proposes a strip pyramid channel attention module including strip convolution and compression-excitation weight module (SEWeight); the strip pyramid channel attention module is implemented through four steps: 1) using four strip convolutions with different convolution kernel sizes to extract features of different scales to construct a feature pyramid; 2) using the SEWeight module to calculate the channel attention of different feature blocks; 3) by using Softmax to recalibrate the channel attention vector, the recalibrated weight of the multi-scale channel is obtained; 4) the recalibrated weight is multiplied by the corresponding feature element, and finally a more refined feature map with richer multi-scale feature information is obtained.

[0095] When constructing a feature pyramid using strip convolution, increasing the kernel size in parallel will lead to a significant increase in the number of parameters. To fully utilize multi-scale spatial position information without increasing the computational effort, grouped convolution is used to extract feature maps of different scales. The features of different scales are then concatenated in the channel dimension, and their dimensions are reorganized to multiply with the multi-scale channel attention obtained below. The process can be expressed as:

[0096]

[0097] F=Cat(F0,F1,F2,F3)

[0098] F∈R C×H×W →F∈R 4×C / 4×H×W

[0099] Among them, k i and G i Represents the size of the strip convolution kernel; Cat represents the splicing operation in the channel dimension; F i are features of different scales extracted by the strip convolution pyramid, and F is the feature after splicing the channel dimension.

[0100] Different scale features F i Through the compression-excitation weight (SEWeight) module, the channel attention vectors w of different scales are obtained i , and then splice them in the channel dimension; then we reorganize the dimensions of the obtained multi-scale channel attention vector to further realize the interaction of channel attention:

[0101]

[0102]

[0103] Get the multi-scale channel weight W i After that, we multiply it with the corresponding elements of the different scale features on the corresponding channels and concatenate them in the channel dimension as the output; the above process is mathematically expressed as:

[0104] Y i =F i *W i

[0105] Out=Cat([Y0,Y1,Y2,Y3]),Out∈R C×H×W

[0106] where Y i Represents the weight of each channel W i And each feature block F i The features after multiplication, * indicates the corresponding multiplication in the channel dimension; Out indicates the output features.

[0107] The SPCA module allows us to have a larger receptive field and detect clouds and cloud shadows in a more fine-grained manner. At the same time, the multi-scale channel attention obtained through the interaction between local and global factors allows the model to pay more attention to global semantic information, improving detection accuracy and noise resistance.

[0108] S23, building a hierarchical feature aggregation module;

[0109] After fully extracting the category information of clouds and cloud shadows during the encoding phase, it is necessary to restore them to the original image size layer by layer during the decoding phase to achieve pixel-by-pixel classification. When the encoder downsamples, the image size is compressed and the position information of the target is gradually lost. The prediction results obtained by simple bilinear interpolation upsampling will have low accuracy and rough edges. To solve the above problems, the present invention proposes a hierarchical feature aggregation module (HFA);

[0110] The hierarchical feature aggregation module uses the category information of deep features to guide the upsampling of shallow features, achieving enhanced classification while preserving location information. Simultaneously, the shallow features are subjected to an improved self-attention module (ISA). This module uses a 3×3 convolution on the key to learn contextual information, then concatenates the learned query and context information, and uses two consecutive 1×1 convolutions to extract local information. This operation not only pays attention to the relationship between independent pixels, but also to the semantics surrounding the pixels, thereby grasping the long-distance dependencies between pixels at a global level. This ensures that the shallow features have more accurate location information.

[0111] like Figure 6 As shown in Figure 2, the aggregation process of shallow and deep features by the hierarchical feature aggregation module includes the following steps:

[0112] 1) The shallow feature F1 is processed through an improved self-attention module (ISA) to obtain the shallow feature F11 with enhanced position information;

[0113] 2) Let the deep feature F2 be upsampled by 3×3 convolution and bilinear interpolation, and its size is changed to F21, the same as the shallow feature;

[0114] 3) Weight it and multiply the corresponding elements with the shallow features to obtain the shallow features U1 with enhanced classification ability;

[0115] 4) The shallow feature F1 is average pooled to obtain F12 of the same size as the deep feature;

[0116] 5) F12 is multiplied by the deep feature F22 that has undergone two layers of 3×3 convolution and Sigmoid weighting to obtain the deep feature U2 that enhances spatial information;

[0117] 6) Finally, U1 is added to U2 which has been upsampled by bilinear interpolation and sent to a 3×3 convolution to complete the fusion of deep features and shallow features.

[0118] The above process can be expressed mathematically as:

[0119] F 11 =attention(F1)

[0120] F 12 =Avpooling{BN[Conv 3×3 (F1)]}

[0121] F 21 =BN[Conv 3×3 (F2)]

[0122] F 22 =BN{Conv 3×3 [DWConv 3×3 (F2)]}

[0123] U1=F 11 ×Sigmoid(F 21 )

[0124] U2=F 12 ×Sigmoid(F 22 )

[0125] Among them, Conv 3×3 Represents 3×3 convolution, DWConv 3×3 Stands for depthwise separable convolution; we will DWConv 3×3 The parameter group is set to the number of channels of deep features; BN represents batch normalization operation; Avpooling represents average pooling operation; × and + represent multiplication and addition of corresponding elements respectively; the symbol σ represents the activation function ReLU.

[0126] S3, uses the data set obtained in S1 to train the above network model;

[0127] The present invention adopts a supervised training mode, and the specific steps are:

[0128] S31, first perform data augmentation operations on the images in the dataset, and randomly rotate, horizontally flip, or vertically flip the images in the dataset with a probability of 0.2;

[0129] S32 converts images and labels into tensors. Images serve as the model's input sequence, and outputs and labels are used to calculate the loss function. The batch size for each input is set to 8. The "ploy" strategy is used to reduce the learning rate as the number of training times increases to achieve better training results. Iter is the current number of iterations, and the iteration coefficient power is set to 0.9. To ensure sufficient convergence and comparison accuracy, the maximum number of training times (maxiter) is 300. The learning rate calculation formula is as follows:

[0130]

[0131] Select cross entropy as the maximum loss function, the formula is expressed as:

[0132]

[0133] Where x is the predicted image output by the model, class is the label, and the Adam optimizer is used.

[0134] S4, uses the trained network model to predict and generate mask images of clouds and cloud shadows;

[0135] When predicting, the model parameters with the highest MioU (mean intersection-over-union) of the validation set during the training iteration are used for prediction; a color image is input, and the trained weights are used to extract features and perform encoding and decoding operations to output the mask image of the cloud and cloud shadow, such as Figure 7 shown.

[0136] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0137] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention as claimed.

Claims

1. A deep learning-based method for fine segmentation and detection of clouds and cloud shadows in remote sensing images, characterized by: The following steps are involved: S1, obtain and process the training data set, and divide the data set into a training set and a validation set; S2, building a segmentation network for deep learning semantics; S3, using the data set obtained in S1, trains the network model in the segmentation network established in S2; S4, uses the trained network model to predict and generate mask images of clouds and cloud shadows; The network constructed in S2 includes: backbone network ResNet-18, multi-scale global attention module, strip pyramid channel attention module and hierarchical feature aggregation module; The backbone network ResNet-18 extracts feature information at different levels, uses a multi-scale global attention module to strengthen the model's attention to channel and spatial information, and uses a strip pyramid channel attention module to learn spatial information at multiple scales. Finally, a hierarchical feature aggregation module fuses high-dimensional features with low-dimensional features, and upsamples layer by layer to achieve the final segmentation effect. The strip pyramid channel attention module includes strip convolution and compression-excitation weight modules; the strip pyramid channel attention module is implemented through the following four steps: 1) Use four strip convolutions with different kernel sizes to extract features of different scales and construct a feature pyramid; 2) Use the SEWeight module to calculate the channel attention of different feature blocks; 3) Recalibrate the channel attention vector using Softmax to obtain the recalibrated weights of the multi-scale channels; 4) Multiply the recalibrated weights and the corresponding features by their corresponding elements, and finally obtain a refined feature map of multi-scale feature information.

2. The method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning according to claim 1, characterized in that: In S1, the original images of size 4800×2742 pixels are uniformly cropped to 224×224, and finally images of size 3×224×224 are obtained. Then all images are divided into training set and validation set in a ratio of 8:

2.

3. The method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning according to claim 1, characterized in that: The multi-scale global attention module includes a hierarchical multi-scale convolution module, a global spatial attention module, and a global channel attention module. The implementation process of the multi-scale global attention module is as follows: Shallow feature F L ∈R C×H×W After the layered multi-scale convolution module, the features of shallow feature-rich spatial information are obtained. Then combine it with the high-dimensional feature F output by the next layer of backbone network H ∈R 2C×H / 2×W / 2 Add to get the preliminary features F∈R 2C×H / 2×W / 2 ; The preliminary feature F is passed through the parallel global spatial attention module and channel attention module to obtain the weight W S and W C ; The above process formula is expressed as: in, Represents the permutation operation, which is used to adjust the dimension of the matrix, converting the matrix dimension from H*W*C to C*W*H. MLP represents the full connection operation in the channel dimension, and σ represents the Sigmoid function. The final weight W s and W C Multiplying with the preliminary feature F, we get the feature M that strengthens the spatial and channel attention of the target on a global scale s and M C , and combine it with the initial feature F after convolution C Add and finally output F O ∈R 2C×W / 2×H / 2 ; The above process can be expressed mathematically as: The formula of the hierarchical multi-scale convolution module is expressed as: Get the output z i After that, i Merge in the channel dimension, and finally adjust the number of channels to the desired number for output through 1×1 convolution.

4. The method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning according to claim 1, characterized in that: The implementation process of the compression-excitation weight module can be mathematically expressed as: w c =σ(W1r(W0(g c )) Among them, GAP represents global average pooling, r represents the ReLU function, W0 is a fully connected network with input C and output C / r, W1 is a fully connected network with input C / r and output C, and the symbol σ represents the activation function Sigmoid.

5. The method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning according to claim 1, characterized in that: The hierarchical feature aggregation module aggregates shallow and deep features in the following steps: 1) The shallow feature F1 is improved through the self-attention module to obtain the shallow feature F11 with enhanced position information; 2) Let the deep feature F2 be upsampled by 3×3 convolution and bilinear interpolation, and its size is changed to F21, the same as the shallow feature; 3) Weight it and multiply the corresponding elements with the shallow features to obtain the shallow features U1 with enhanced classification ability; 4) The shallow feature F1 is average pooled to obtain F12 of the same size as the deep feature; 5) F12 is multiplied by the deep feature F22 that has undergone two layers of 3×3 convolution and Sigmoid weighting to obtain the deep feature U2 that enhances spatial information; 6) Finally, U1 is added to U2 which has been upsampled by bilinear interpolation and sent to a 3×3 convolution to complete the fusion of deep features and shallow features.

6. The method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning according to claim 1, characterized in that: S3 adopts the supervised training mode, and the specific steps are as follows: S31, first perform data augmentation operations on the images in the dataset, and randomly rotate, horizontally flip, or vertically flip the images in the dataset with a probability of 0.2; S32, convert the image and label into tensors. The image is used as the input sequence of the model. The output and label are used to calculate the loss function. The batch size of each input is set to 8. The "ploy" strategy is used to reduce the learning rate accordingly as the number of training times increases to achieve better training results. Iter is the current number of iterations, the iteration coefficient power is set to 0.9, and the maximum number of training times maxiter is 300. The learning rate calculation formula is as follows: Select cross entropy as the maximum loss function, the formula is expressed as: Where x is the predicted image output by the model, class is the label, and the Adam optimizer is used.

7. The method for fine segmentation and detection of clouds and cloud shadows in remote sensing images based on deep learning according to claim 1, characterized in that: In S4, when making predictions, the model parameters with the highest MioU on the validation set during the training iteration are used for prediction; a color image is input, and the features are extracted using the trained weights, and encoding and decoding operations are performed to output a mask image of the cloud and cloud shadow.

8. A fine segmentation and detection system for clouds and cloud shadows in remote sensing images based on deep learning, characterized by: include: Data processing module: used to obtain and process the training data set and divide the data set into training set and validation set; Network building module: building a segmentation network for deep learning semantics; Training module: Use the data set to train the network model in the segmentation network; Image generation module: Use the trained network model to predict and generate cloud and cloud shadow mask images; The network constructed in S2 includes: backbone network ResNet-18, multi-scale global attention module, strip pyramid channel attention module and hierarchical feature aggregation module; The backbone network ResNet-18 extracts feature information at different levels, uses a multi-scale global attention module to strengthen the model's attention to channel and spatial information, and uses a strip pyramid channel attention module to learn spatial information at multiple scales. Finally, a hierarchical feature aggregation module fuses high-dimensional features with low-dimensional features, and upsamples layer by layer to achieve the final segmentation effect. The strip pyramid channel attention module includes strip convolution and compression-excitation weight modules; the strip pyramid channel attention module is implemented through the following four steps: 1) Use four strip convolutions with different kernel sizes to extract features of different scales and construct a feature pyramid; 2) Use the SEWeight module to calculate the channel attention of different feature blocks; 3) Recalibrate the channel attention vector using Softmax to obtain the recalibrated weights of the multi-scale channels; 4) Multiply the recalibrated weights and the corresponding features by their corresponding elements, and finally obtain a refined feature map of multi-scale feature information.

Images