A lightweight cloud detection method from satellite images based on dark channel features

Abstract

The present invention discloses a lightweight satellite image cloud detection method based on dark channel feature guidance, which belongs to an image data processing method. In the lightweight satellite image cloud detection method based on dark channel feature guidance proposed by the present invention, in order to utilize the dark channel prior in multispectral imagery to guide network learning features, a multi-scale dark channel extractor is first used to predict the dark channel, and then the dark channel features and image features are input into a dark channel guided context aggregation module based on an attention mechanism to enhance the image features, thereby making the cloud detection results more accurate. Then, in order to enhance the network's migration capability between different satellite sensors, a channel adaptation module is proposed to handle the situation where the number of bands of different satellite sensors is inconsistent. This method can outperform the mainstream method while keeping the number of parameters and computational complexity small, and the model also has a certain migration capability.

Description

Technical Field

[0001] The present invention relates to an image data processing method, and in particular to a lightweight satellite image cloud detection method based on dark channel feature guidance. Background Art

[0002] Optical remote sensing imagery is one of the most important data sources for Earth remote sensing observations. It has been widely used in many fields, including surface cover mapping and vegetation and water monitoring, providing irreplaceable support for global environmental monitoring. However, optical remote sensing imagery is inevitably affected by cloud obscuration, resulting in invalid pixels with low or even no value in the imagery. This causes ground object information to be obscured or distorted, affecting interpretation and analysis. Therefore, it is necessary to detect clouds in the imagery to improve the utilization rate of cloud-containing imagery. Cloud detection is often the first and most critical preprocessing step. When processing large quantities of satellite imagery, manually annotating cloud masks is time-consuming and labor-intensive. Therefore, it is crucial to design fully automatic algorithms for detecting clouds in optical remote sensing imagery.

[0003] Automated cloud detection using optical remote sensing satellites has always faced numerous challenges. First, clouds come in a wide variety of types, and they can easily be confused with bright ground objects. This is especially true when remote sensing imagery has a limited number of spectral bands, such as in common multispectral images consisting of near-infrared, red, green, and blue bands. Misdetection due to ground objects is particularly common. Second, near thin cloud boundaries, cloud information is intermingled with surface information, making it difficult to distinguish between cloud and non-cloud areas. To address these challenges, researchers have proposed a number of effective methods from various perspectives. Traditional cloud detection methods using optical remote sensing satellite imagery can be divided into two categories: single-temporal and multi-temporal. Currently, deep learning-based methods can achieve end-to-end cloud detection and outperform traditional methods. However, existing methods still have several challenges: 1) By stacking convolutional and pooling layers, existing methods only extract local spatial features within the image, while ignoring the global semantic information of the remote sensing image patches. 2) Existing methods are generally trained and tested on images from the same sensor, resulting in significant accuracy degradation when tested on a different sensor than the training image. 3) Existing methods have large parameter and computational requirements, making them difficult to deploy. Consequently, existing methods are insufficient for addressing practical problems and applications, limiting the widespread adoption of deep learning cloud detection technology in remote sensing. In this context, developing a lightweight and high-performance deep learning cloud detection model is a significant challenge. Summary of the Invention

[0004] Compared to convolutional neural networks, the Vision Transformer (ViT) divides the image into multiple small patches and uses a self-attention mechanism to model the global relationships between these patches. Therefore, ViT can capture long-range dependencies, allowing the network to extract higher-order, more abstract features to improve classification accuracy. In the proposed method, ViT is used as the network encoder. Furthermore, it is observed that the dark channel image in multispectral imagery can clearly distinguish between cloud areas and other landform types. Therefore, the dark channel prior can be used to guide the network's feature learning. When the trained network is tested on data from other sensors, the accuracy drops significantly. Therefore, it is necessary to propose a module to enhance the network's transferability and improve the model's generalization performance across different sensors.

[0005] Therefore, this paper proposes a lightweight cloud detection method using satellite imagery guided by dark channel features. First, a multi-scale dark channel extractor is used to predict dark channels. Dark channel features and image features are then fed into an attention-based dark channel-guided context aggregation module to enhance image features, thereby improving cloud detection accuracy. Furthermore, to enhance the network's transferability across different satellite sensors, a channel adaptation module is proposed to address the inconsistency of the number of bands across different satellite sensors.

[0006] The technical solution adopted by the present invention is: a lightweight satellite image cloud detection method based on dark channel feature guidance, comprising the following steps:

[0007] Step 1: First, input the training image data into the encoder network to extract multiple features at different levels;

[0008] Step 2: Cascade the multiple features obtained in step 1 to obtain the image semantic feature F sem ;

[0009] Step 3: Input the multiple features obtained in step 1 into the multi-scale dark channel extractor module to obtain new features;

[0010] Step 4: Calculate the dark channel prediction loss using the new features calculated in step 3

[0011] Step 5: Perform a cascade operation on the new features calculated in step 3 to obtain the dark channel feature F dark ;

[0012] Step 6: The image semantic feature F obtained in step 2 sem And the dark channel feature F obtained in step 5 dark Input into the dark channel guided context aggregation module to obtain the final feature map;

[0013] Step 7: Calculate the cross entropy loss between the final feature map obtained in step 6 and the actual cloud mask

[0014] Step 8: Perform backpropagation through the gradient descent algorithm and update the network parameters;

[0015] Step 9: Steps 1 to 8 are iterated repeatedly until the training is completed to obtain a model for predicting remote sensing image segmentation results;

[0016] Step 10: During the testing phase, a window is set to slide on the image, and the image blocks of each window are input into the model to obtain the prediction results of each window, and finally the segmentation results of the remote sensing image are obtained.

[0017] Furthermore, in step 1, the encoder network uses MobileViT.

[0018] Furthermore, the multi-scale dark channel extractor module in step 3 includes multiple 1×1 convolutional layers, which are used to map the multiple features in step 1 to another space to obtain new features.

[0019] Furthermore, the dark channel prediction loss in step 4 The definition is as follows:

[0020]

[0021] The first regression loss The reverse Huber loss is used:

[0022]

[0023] in, represents the predicted dark channel, y dark Indicates the real dark channel;

[0024] Second loss is the L1 loss of the gradient between the dark channel and the predicted value:

[0025]

[0026] Among them, g x and g y Represent the gradients in the horizontal and vertical directions respectively;

[0027] The third loss The definition is as follows:

[0028]

[0029] Furthermore, in step 6, the image semantic features are first convolved twice with 1×1 to obtain Fkey and F value , and also perform a 1×1 convolution on the dark channel feature to obtain F query ; Then transform these feature dimensions into Where C = 128, N is the number of pixels; for F key and F query Perform matrix multiplication and apply the SoftMax function to obtain the similarity feature map F; then, the similarity feature map F is combined with the feature F value Perform matrix multiplication and add it to the image semantic features to obtain the final feature map.

[0030] Furthermore, in step 1, in order to increase the mobility of the cloud detection network across different sensors, the original training image data is first input into the channel adaptation module to obtain the output three-channel feature map, which is then input into the encoder network. The specific implementation process is as follows:

[0031] For a C-band multispectral image I c , and transform it as follows:

[0032] f({I1,…,I c})=g(h(I1),…,h(I c ))#(5)

[0033] The function h is a simple convolutional network, and g is a symmetric function;

[0034] If the function h is a convolutional layer and g is a cascade operation followed by maximum pooling in the channel dimension, it can be expressed as:

[0035] f=max(cat(conv(I1),…,conv(I c )))#(6)

[0036] in H, W are the height and width of the feature map respectively;

[0037] Assuming there are N such functions h, we can get N feature maps {f1,…,f N}, in order to finally output the 3-channel feature map, all feature maps are cascaded and passed through a convolution layer to obtain the final feature map Right now:

[0038] f in =conv(cat(f1,…,f N ))#(7).

[0039] Furthermore, the function h also includes a combination of instance normalization and residual convolution.

[0040] Compared with the existing technology, the advantages and beneficial effects of the present invention are as follows: the lightweight satellite image cloud detection method based on dark channel feature guidance proposed in the present invention adopts a lightweight backbone network as an encoder; inspired by the dark channel prior, the present invention incorporates a dark channel-guided context aggregation module to enhance image features, thereby making the cloud detection results more accurate; in order to enhance the network's migration capability between different satellite sensors, a channel adaptation module is proposed to handle the situation where the number of bands of different satellite sensors is inconsistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 : The overall network structure diagram of the method designed by the present invention;

[0042] Figure 2 : The channel adaptive module structure diagram designed by the present invention;

[0043] Figure 3 :Visualization results of the method of the present invention; (a) false color image (b) true value (c) Fmask (d) baseline method (e) PPM (f) ASPP (g) non-local (h) dual attention (i) method of the present invention. DETAILED DESCRIPTION

[0044] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0045] Please see Figure 1 The overall network structure diagram of the method designed by the present invention, a remote sensing image deep network semi-supervised semantic segmentation method based on transformation consistency regularization provided by the present invention comprises the following steps:

[0046] Step 1: The overall network structure diagram of the method designed by the present invention is as follows Figure 1 As shown in the figure, the training image data is first fed into the encoder network to extract deep features. To make the entire network lightweight, the encoder network used here is MobileViT. MobileViT consists of four stages, and thus its features {F1, F2, F3, F4} can be obtained.

[0047] Step 2: Cascade the four features obtained in step 1 to obtain the image semantic feature F sem .

[0048] Step 3: Input the four features obtained in step 1 into the multi-scale dark channel extractor module (see Figure 1), this module uses both low-level detail features (i.e., F1 and F2) and high-level semantic features (i.e., F3 and F4) extracted by the encoder network to predict the dark channel. To reduce feature dimensionality, the features of the four stages in this module are mapped to another space using 1×1 convolution to obtain four new features.

[0049] Step 4: Calculate the dark channel prediction loss for each of the four features calculated in step 3 Dark channel prediction loss It can be defined as follows:

[0050]

[0051] The first regression loss The reverse Huber loss is used:

[0052]

[0053] in, represents the dark channel predicted by the network, y dark Indicates the real dark channel.

[0054] Second loss is the L1 loss of the gradient between the dark channel and the predicted value:

[0055]

[0056] Among them, g x and g y Represents the gradient in the horizontal and vertical directions respectively, and n represents the tensor g x The number of elements.

[0057] The third loss The most commonly used indicator in the field of image reconstruction is SSIM. Since the upper limit of SSIM is 1, the definition as follows:

[0058]

[0059] Step 5: Cascade the four features calculated in step 3 to obtain the dark channel feature F dark .

[0060] Step 6: The image semantic feature F obtained in step 2 sem And the dark channel feature F obtained in step 5 dark Input to the dark channel guided context aggregation module (see Figure 1 ) to obtain the final feature map. In detail, first perform two 1×1 convolutions on the image semantic features to obtain F key and Fvalue , and also perform a 1×1 convolution on the dark channel feature to obtain F query . These feature dimensions are then transformed into Where C = 128, N is the number of pixels. key and F query Perform matrix multiplication and apply SoftMax function to obtain similarity feature map F. Then, the similarity feature map F is combined with feature F value Perform matrix multiplication and add it to the image semantic features to obtain the final feature map.

[0061] Step 7: Calculate the cross entropy loss between the final feature map obtained in step 6 and the actual cloud mask

[0062] Step 8: In order to increase the mobility of the cloud detection network on different sensors, a channel adaptation module is proposed without reducing the performance on the current training dataset. Figure 2 As shown. The original training image data is first input into the channel adaptation module before inputting into the network to obtain the output three-channel feature map, and finally input into the pre-training model. In detail, for the multispectral image I with C bands C , and transform it as follows:

[0063] f({I1,…,I c})=g(h(I1),…,h(I c ))#(5)

[0064] The function h is a simple network such as a convolutional layer, and g is a symmetric function.

[0065] If the function h is a convolutional layer and g is a cascade operation followed by maximum pooling in the channel dimension, it can be expressed as:

[0066] f=max(cat(conv(I1),…,conv(I c )))#(6)

[0067] in

[0068] Assuming there are N such functions h, we can get N feature maps {f1,…,f N In order to finally output the 3-channel feature map, all feature maps are cascaded and passed through a convolution layer to obtain the final feature map Right now:

[0069] f in =conv(cat(f1,…,f N ))#(7)

[0070] After experiments, the function h adopts a combination of instance normalization and residual convolution.

[0071] Step 9: Perform backpropagation through the gradient descent algorithm and update the network parameters θ.

[0072] Step 10: Repeat steps 1 to 9 until the training is completed.

[0073] Step 11: During the testing phase, a window is set to slide on the image, and the image blocks of each window are input into the network to obtain the prediction results of each window, and finally the segmentation results of the remote sensing image are obtained.

[0074] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0075] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A lightweight satellite image cloud detection method based on dark channel feature guidance, characterized by: The following steps are involved: Step 1: First, input the training image data into the encoder network to extract multiple features at different levels; Step 2: Cascade the multiple features obtained in step 1 to obtain the image semantic feature F sem ; Step 3: Input the multiple features obtained in step 1 into the multi-scale dark channel extractor module to obtain new features; Step 4: Calculate the dark channel prediction loss using the new features calculated in step 3 Dark channel prediction loss in step 4 The definition is as follows: The first regression loss The reverse Huber loss is used: in, represents the predicted dark channel, y dark Indicates the real dark channel; Second loss is the L1 loss of the gradient between the dark channel and the predicted value: Among them, g x and g y Represents the gradient in the horizontal and vertical directions respectively, and n represents the tensor g x The number of elements in ; The third loss The definition is as follows: Among them, SSIM is the image reconstruction index; Step 5: Perform a cascade operation on the new features calculated in step 3 to obtain the dark channel feature F dark ; Step 6: The image semantic feature F obtained in step 2 sem And the dark channel feature F obtained in step 5 dark Input into the dark channel guided context aggregation module to obtain the final feature map; Step 7: Calculate the cross entropy loss between the final feature map obtained in step 6 and the actual cloud mask Step 8: Perform backpropagation through the gradient descent algorithm and update the network parameters; Step 9: Steps 1 to 8 are iterated repeatedly until the training is completed to obtain a model for predicting remote sensing image segmentation results; Step 10: During the testing phase, a window is set to slide on the image, and the image blocks of each window are input into the model to obtain the prediction results of each window, and finally the segmentation results of the remote sensing image are obtained.

2. The lightweight satellite image cloud detection method based on dark channel feature guidance according to claim 1 is characterized by: In step 1, the encoder network uses MobileViT.

3. The method for lightweight satellite image cloud detection based on dark channel feature guidance according to claim 1, characterized in that: The multi-scale dark channel extractor module in step 3 includes multiple 1×1 convolutional layers, which are used to map the multiple features in step 1 to another space to obtain new features.

4. The method for lightweight satellite image cloud detection based on dark channel feature guidance according to claim 1, characterized in that: In step 6, the image semantic features are first convolved twice with 1×1 to obtain F key and F value , and also perform a 1×1 convolution on the dark channel feature to obtain F query ; Then transform these feature dimensions into Where C = 128, N is the number of pixels; for F key and F query Perform matrix multiplication and apply the SoftMax function to obtain the similarity feature map F; then, the similarity feature map F is combined with the feature F value Perform matrix multiplication and add it to the image semantic features to obtain the final feature map.

5. The lightweight satellite image cloud detection method based on dark channel feature guidance according to claim 1 is characterized by: In step 1, in order to increase the mobility of the cloud detection network across different sensors, the original training image data is first input into the channel adaptation module to obtain the output three-channel feature map, which is then input into the encoder network. The specific implementation process is as follows: For a C-band multispectral image I C , and transform it as follows: f({I1,…,I c })=g(h(I1),…,h(I c ))#(5) The function h is a simple convolutional network, and g is a symmetric function; Setting the function h as a convolution layer, g as a cascade operation and then taking the maximum pooling in the channel dimension, it can be expressed as: f=max(cat(conv(I1),…,conv(I c )))#(6) in H, W are the height and width of the feature map respectively; If there are N such functions h, then we can get N feature maps {f1,…,f N }, in order to finally output the 3-channel feature map, all feature maps are cascaded and passed through a convolution layer to obtain the final feature map Right now: f in =conv(cat(f1,…,f N ))#(7)。 6. The lightweight satellite image cloud detection method based on dark channel feature guidance according to claim 5 is characterized by: Function h also includes a combination of instance normalization and residual convolution.

Images