A multi-level semantic fusion cloud and cloud shadow detection method, device and storage medium

Abstract

The present invention discloses a cloud and cloud shadow detection method, device and storage medium with multi-level semantic fusion, which relate to the field of image processing technology. A residual network is used as a backbone network to extract features from input image data. The overall structure of an encoder-decoder is adopted. In the model, three modules, namely a multi-branch residual context semantic module, a multi-scale convolution sub-channel attention module and a feature fusion upsampling module, are proposed to strengthen feature extraction, refine cloud and cloud shadow edge information, and enhance the actual segmentation ability of the model. Compared with the current method, the detection of clouds and cloud shadows is more accurate, the interference of background information is effectively reduced, the detection ability of cloud edges and thin clouds is enhanced, and the technical problem that the detection accuracy of most networks for cloud shadows is not high enough is solved, the detection accuracy of clouds and cloud shadows is improved, and the robust performance of the algorithm is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multi-level semantic fusion cloud and cloud shadow detection method, device and storage medium. Background Art

[0002] At present, due to the rapid development of aerospace technology, the acquisition of high-resolution remote sensing images has become increasingly simple. However, the acquired remote sensing images are usually affected by cloud cover due to weather reasons, which hinders the extraction of ground object information, greatly reduces the effective information extracted, and even leads to wrong judgments. At the same time, clouds also play an important role in meteorology and are an important indicator of climate change. Therefore, accurate segmentation of clouds and cloud shadows in remote sensing images plays an important role in the later use of remote sensing images.

[0003] The existing cloud detection method is mainly to manually identify clouds and cloud shadows based on the thresholds of different bands or by using the normalized differences between different cloud bands. However, traditional algorithms generally have problems such as complex operation, requiring a lot of manpower for feature extraction, and low detection accuracy.

[0004] In recent years, with the rise of artificial intelligence, many detection methods using machine learning and deep learning have emerged. Many scholars have applied deep learning to cloud detection. Although deep learning has a good effect in the field of image processing, the effect of many networks applied to specific scenarios is not so ideal. In the cloud and cloud shadow detection task, most networks have inaccurate segmentation results, unsatisfactory edge information recovery, many thin clouds cannot be accurately detected, and cloud shadow detection is disturbed by complex background. For this reason, a multi-level semantic fusion cloud and cloud shadow detection method is proposed. Summary of the invention

[0005] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a cloud and cloud shadow detection method, device and storage medium with multi-level semantic fusion, which solves the technical problem in the prior art that the detection accuracy of cloud shadows is not high enough.

[0006] The object of the present invention can be achieved by the following technical solution: a cloud and cloud shadow detection method with multi-level semantic fusion, the method comprising the following steps:

[0007] The original image was directly obtained from Google Earth, with a size of 4800×2742 pixels. The original image was then cropped uniformly and manually labeled. The clouds in the image were labeled red, the cloud shadows were labeled green, and the background was labeled black, so that the labels were consistent with the original image. Figure 1 One by one, all images are divided into training set and validation set in a ratio of 8:2;

[0008] Construct a multi-level feature context semantic fusion network and three related modules, including a multi-branch residual context semantic module, a multi-scale convolutional sub-channel attention module, and a feature fusion upsampling module;

[0009] The images in the dataset are augmented, and randomly rotated, horizontally flipped, or vertically flipped with a probability of 0.2. The images and labels are then converted into tensors as input sequences. The input batch size is set to 8, the initial learning rate is set to 0.001, the decay coefficient is 0.98, and the learning rate is updated every 3 training times. A total of 200 training times are performed. The learning rate calculation formula is as follows:

[0010] 1. N =lr0×β N / s

[0011] Among them lr N is the learning rate size of the Nth training, lr0 is the initial learning rate, β is the decay coefficient, s is the update interval, and the loss function selects the cross entropy loss function. The calculation formula is as follows:

[0012]

[0013] Where x is the output tensor of the network, class is the real label, and the Adam algorithm is used as the optimizer during the training process to obtain the trained multi-level feature context semantic fusion network;

[0014] A color image is input, and the color image is encoded and decoded by the trained multi-level feature context semantic fusion network to output the mask image of clouds and cloud shadows.

[0015] Furthermore, the multi-branch residual context semantic module adopts a residual structure to achieve identity mapping by performing a shortcut connection on the input tensor, without introducing additional parameters or computational complexity, and retaining some original parameters of the input;

[0016] The main part adopts a dual-branch connection method. The left branch is a contextual semantic branch, which is used to extract contextual semantic information and incorporates a self-attention mechanism to focus on meaningful information in the feature map. The right branch is an enhanced feature extraction branch, which introduces two strip convolutions while using a 3x3 convolution block to enhance feature extraction and improve the effect.

[0017] First, the input tensor is compressed through a 1x1 convolution to reduce the channel size to half of the original value. Then, the output tensors are fused after being output through two branches, and a residual connection is formed with the input end before passing through a dual-channel attention module to model the importance of each channel feature.

[0018] The context semantic branch is introduced in the multi-branch residual context semantic module, which combines convolution with attention and uses convolution to extract local information in the feature map. First, for the input features, a 3×3 convolution is used to perform feature mapping on the input to obtain the key with local context information, marked as K. A 1×1 convolution is used to perform feature mapping on the input to obtain the value, marked as V, while the query still uses the original value, marked as Q. Then, the key and the query are concat and two consecutive convolution operations are performed to obtain the weighted Attention map of each pixel, marked as A:

[0019] A = Conv(Conv(concat[K,Q]))

[0020] Conv(.) represents the convolution operation, and concat(.) represents the connection of multiple tensors in the channel dimension. Different from the traditional attention mechanism, A here is obtained by the interaction between Q and local context semantic information K. The interaction between local context semantic information is used to enhance the attention mechanism. Then, A is multiplied by V to obtain the dynamic context information X:

[0021] X=A*V

[0022] Finally, K with local context information is added to X with global context information to get the output result of the left branch;

[0023] The right branch introduces a set of 1×3 and 3×1 strip convolution blocks, which use 3×3, 1×3 and 3×1 convolution kernels to replace the single 3×3 convolution kernel for calculation, and then add the three outputs to get the final output;

[0024] Furthermore, the multi-scale convolution sub-channel attention module is used to extract the connection between deep channels. After extracting information of different scales through grouped convolution, the improved pyramid pooling module further extracts deep semantic information. Strip pooling is used to accurately extract the boundary information of clouds and cloud shadows, improve the problem of rough segmentation boundaries, and use multi-scale convolution to extract information of different scales to improve the expression ability of the model. In the main branch, convolution kernels of different sizes of 1×1, 3×3, 5×5, and 7×7 are first used to extract information of different scales in deep features, so as to better extract information in the feature map. Then, the improved pyramid pooling layer is used to refine the edge information of clouds and cloud shadows in the feature map, and finally spliced ​​and output. The calculation process of the main branch is as follows:

[0025] C i =Conv(k i ×k i )(x)i=0,1,2,3

[0026] A i1 =Avg(k i ×1)(C i )i=0,1,2,3

[0027] A i2 =Avg(1×k i )(C i )i=0,1,2,3

[0028] A i =A i1 +A i2

[0029] Y=DWConv(concat[A0,A1,A2,A3])

[0030] Where Conv(.) represents convolutions of different scales, k i Represents the convolution kernel size, Avg(.) represents strip pooling of different sizes, DWConv(.) represents depthwise separable convolution, and concat(.) represents concatenation operation;

[0031] The input feature map is split on the channel and divided into different sub-channels. Then, different convolution kernels are used on each sub-channel to obtain different feature maps on the channel. Then, the attention at different scales is extracted through channel attention to obtain the channel attention vector. Finally, Softmax is used to further calibrate the channel information.

[0032] The channel attention module allows the importance of each channel to be evaluated, so as to focus on important information. Like most channel attention modules, an adaptive global average pooling layer is first used to extract channel information from the feature map, and then two layers of one-dimensional convolutional layers are used to compress and recalibrate the global information. Finally, the weight is calculated by the nonlinear activation function Sigmoid. The calculation process of the entire auxiliary branch is as follows:

[0033] F i =Conv(k i ×k i )(x)i=0,1,2,3

[0034] g i =GAP(F i )

[0035] S i =Sigmoid(W1σ(W0(g i )))

[0036] Q i =Softmax(S i )

[0037] W = concat(Q1,Q2,Q3,Q4)

[0038] Where σ represents the nonlinear activation function ReLU, Represents a one-dimensional convolution operation. By compressing and restoring the channels, the relationship between channels can be more effectively combined, which is conducive to information interaction between dimensions.

[0039] The important information in the feature map is obtained by extracting the enhanced features of the main branch and calculating the channel weights of the auxiliary branch. The two are multiplied and the channel is adjusted using a depthwise separable convolution. The calculation formula is as follows:

[0040] output = DWConv(W*Y).

[0041] Furthermore, the feature fusion upsampling module fuses the semantic information of high-level features with the detail information of low-level features, uses the rich category information of high-level features to guide the classification of low-level features, and adopts a dual-branch mode to extract information from low-level feature maps. The "bottleneck structure" mode is adopted in the left branch, and a hole convolution is introduced to increase its receptive field area. The right branch also introduces a hole convolution, and then a maximum pooling layer is passed to reduce the error of increased variance of the estimated value caused by the limited neighborhood size, retaining more image background information. For high-level features, the bilinear interpolation upsampling method is first used to upsample the high-level feature map to the same size as the low-level feature map, and then the category information therein is refined and extracted through the convolution layer, the high-level features are weighted and fused with the low-level features, and finally the results on both sides are added and the channel is adjusted and output through a 1×1 convolution.

[0042] Furthermore, the multi-level feature context semantic fusion network mainly adopts a multi-level feature fusion method to fuse deep features with shallow features and add an attention mechanism. The overall structure is an encoder-decoder. First, the residual network ResNet-18 is used as the backbone network and feature extraction is performed by downsampling layer by layer. Then, the output features are respectively passed through a multi-branch residual context semantic module and a multi-scale convolutional sub-channel attention module for context semantic extraction and feature enhancement. A large number of feature transmission channels and fusion operations are added to fuse information from different feature layers. The encoder part aggregates deep information and shallow information while upsampling layer by layer, so that the context information in the network is propagated to higher resolution layers. Finally, a residual module is used at the output end for channel step refinement.

[0043] Furthermore, the cropping method is sliding coverless cutting, and the cropped image has 3 channels and a size of 224 in width and height.

[0044] Furthermore, the size of the cropped image is 224×224.

[0045] Furthermore, after the dual-branch feature extraction, it enters a dual-channel attention module. Since the common channel attention mostly directly adopts global average pooling, which will lead to the loss of edge information, two types of global pooling operations are used, namely global average pooling and global maximum pooling, to make the extracted feature information richer. Then, the 1x1 convolution operation is used to extract the mutual information between channels, focusing on the importance of different feature information between different channels. The calculation formula is as follows:

[0046] g max =Conv1d(GMP(x))

[0047] g avg=Conv1d(GAP(x))

[0048] Among them, GMP(.) and GAP(.) are global maximum pooling and global average pooling operations respectively, Conv1d(.) is a one-dimensional convolution operation, and then the results of the two branches are spliced ​​and then subjected to a convolution operation, adaptively focusing on the feature information of global average pooling and maximum pooling, strengthening the focus on useful information, such as the edge contours and pixel color values ​​of clouds and cloud shadows, and weakening the focus on useless information, such as background farmland and house buildings. Finally, the nonlinear activation function Sigmoid(.) is used to perform a product operation with the input original feature layer. The calculation process is as follows:

[0049] g = concat(g max ,g avg )

[0050] W = Sigmoid(Conv1d(g))

[0051] y out =x*W

[0052] Among them, concat(.) represents the concatenation operation, Sigmoid(.) represents the nonlinear activation function, and Conv1d(.) represents the one-dimensional convolution operation.

[0053] A device, characterized in that it comprises:

[0054] one or more processors;

[0055] A memory for storing one or more programs;

[0056] When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-level semantic fusion cloud and cloud shadow detection method as described above.

[0057] A storage medium containing computer executable instructions, characterized in that the computer executable instructions are used to execute a multi-level semantic fusion cloud and cloud shadow detection method as described above when executed by a computer processor.

[0058] Beneficial effects of the present invention:

[0059] During use, the present invention uses a residual network as the backbone network to extract features from the input image data, and adopts an encoder-decoder structure as a whole. In the model, we propose three modules: a multi-branch residual context semantic module, a multi-scale convolutional sub-channel attention module, and a feature fusion upsampling module to strengthen feature extraction, refine the edge information of clouds and cloud shadows, and enhance the actual segmentation ability of the model. Compared with the current method, it is more accurate to detect clouds and cloud shadows, can effectively reduce the interference of background information, and enhance the detection ability of cloud edges and thin clouds. Similarly, the model proposed in this article also has good results in segmenting other objects, and has excellent generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 It is a schematic diagram of the cloud detection network structure of the present invention;

[0062] Figure 2 Schematic diagram of a multi-branch residual context semantic module of the present invention;

[0063] Figure 3 Schematic diagram of the multi-scale convolutional sub-channel attention module of the present invention;

[0064] Figure 4 It is a schematic diagram of a feature fusion upsampling module of the present invention;

[0065] Figure 5 This is a training set sample diagram of an embodiment of the present invention, wherein the upper half is the original remote sensing image and the lower half is the corresponding label diagram;

[0066] Figure 6 The original remote sensing image for prediction in the embodiment of the present invention;

[0067] Figure 7 The predicted image is the original remote sensing image predicted in the embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] like Figure 1 As shown, a multi-level semantic fusion cloud and cloud shadow detection method includes:

[0070] Acquisition of training sets and data sets:

[0071] The first step is to obtain the training set. The size of the image is 4800×2742 pixels. Due to the limitation of graphics card memory, the original images are uniformly cropped to 224×224 size. The cropping method is sliding coverless cutting, and finally a 3×224×224 image is obtained, which means that the cropped image has 3 channels and a width and height of 224. Then all the images are divided into training set and validation set in a ratio of 8:2.

[0072] The pictures we selected include multiple angles, heights and backgrounds. The purpose is to ensure the complexity of the picture background so that the data will be more authentic. The picture background mainly includes forests, waters, deserts, buildings, farmlands and other scenes, such as Figure 5 As shown, some images in the training set are selected and machine labels are displayed.

[0073] 2. Constructing a multi-level feature context semantic fusion network

[0074] The structure of a remote sensing image cloud and cloud shadow detection method based on a multi-level feature context semantic fusion network is as follows: Figure 1 As shown in the figure. The MFCSF adopts an end-to-end training method. It is mainly composed of a backbone network, a multi-branch residual context semantics module (MRCS), a multi-scale convolutional sub-channel attention module (MCSA), and a feature fusion upsampling module (FFU). The network mainly adopts a multi-level feature fusion method to fuse deep features with shallow features and add an attention mechanism. The overall structure is an encoder-decoder. The network first uses ResNet-18 as the backbone network to extract features by downsampling layer by layer. Then, the output features are respectively subjected to MRCS and MCSA modules for context semantic extraction and feature enhancement. In the network, we add a large number of feature transmission channels and fusion operations to fuse information from different feature layers. The encoder part aggregates deep information and shallow information by upsampling layer by layer, which can propagate the context information in the network to higher resolution layers. Finally, a residual module is used at the output end for channel step refinement.

[0075] It should be further explained that, in the specific implementation process, the multi-branch residual context semantic module adopts the form of a residual structure, and realizes the identity mapping by performing a shortcut connection on the input tensor, which neither introduces additional parameters nor computational complexity, and retains some original parameters of the input;

[0076] The main part adopts a dual-branch connection method. The left branch is a contextual semantic branch, which is used to extract contextual semantic information and incorporates a self-attention mechanism to focus on meaningful information in the feature map. The right branch is an enhanced feature extraction branch, which introduces two strip convolutions while using a 3x3 convolution block to enhance feature extraction and improve the effect.

[0077] First, the input tensor is compressed through a 1x1 convolution to reduce the channel size to half of the original value. Then, the output tensors are fused after being output through two branches, and a residual connection is formed with the input end before passing through a dual-channel attention module to model the importance of each channel feature.

[0078] The context semantic branch is introduced in the multi-branch residual context semantic module, which combines convolution with attention and uses convolution to extract local information in the feature map. First, for the input features, a 3×3 convolution is used to perform feature mapping on the input to obtain the key with local context information, marked as K. A 1×1 convolution is used to perform feature mapping on the input to obtain the value, marked as V, while the query still uses the original value, marked as Q. Then, the key and the query are concat and two consecutive convolution operations are performed to obtain the weighted Attention map of each pixel, marked as A:

[0079] A = Conv(Conv(concat[K,Q]))

[0080] Conv(.) represents the convolution operation, and concat(.) represents the connection of multiple tensors in the channel dimension. Different from the traditional attention mechanism, A here is obtained by the interaction between Q and local context semantic information K. The interaction between local context semantic information is used to enhance the attention mechanism. Then, A is multiplied by V to obtain the dynamic context information X:

[0081] X=A*V

[0082] Finally, K with local context information is added to X with global context information to get the output result of the left branch;

[0083] The right branch introduces a set of 1×3 and 3×1 strip convolution blocks, which use 3×3, 1×3 and 3×1 convolution kernels to replace the single 3×3 convolution kernel for calculation, and then add the three outputs to get the final output.

[0084] It needs to be further explained that, in the specific implementation process, the multi-scale convolution sub-channel attention module is used to extract the connection between deep channels. After extracting information of different scales through grouped convolution, the improved pyramid pooling module further extracts deep semantic information, in which strip pooling is used to accurately extract the boundary information of clouds and cloud shadows, improve the problem of rough segmentation boundaries, and use multi-scale convolution to extract information of different scales to improve the expression ability of the model. In the main branch, convolution kernels of different sizes of 1×1, 3×3, 5×5, and 7×7 are first used to extract information of different scales in deep features, so as to better extract information in the feature map, and then the improved pyramid pooling layer is used to refine the edge information of clouds and cloud shadows in the feature map, and finally spliced ​​and output. The calculation process of the main branch is as follows:

[0085] C i =Conv(k i ×k i )(x)i=0,1,2,3

[0086] A i1 =Avg(k i ×1)(C i )i=0,1,2,3

[0087] A i2 =Avg(1×k i )(C i )i=0,1,2,3

[0088] A i =A i1 +A i2

[0089] Y=DWConv(concat[A0,A1,A2,A3])

[0090] Where Conv(.) represents convolutions of different scales, k i Represents the convolution kernel size, Avg(.) represents strip pooling of different sizes, DWConv(.) represents depthwise separable convolution, and concat(.) represents concatenation operation;

[0091] The input feature map is split on the channel and divided into different sub-channels. Then, different convolution kernels are used on each sub-channel to obtain different feature maps on the channel. Then, the attention at different scales is extracted through channel attention to obtain the channel attention vector. Finally, Softmax is used to further calibrate the channel information.

[0092] The channel attention module allows the importance of each channel to be evaluated, so as to focus on important information. Like most channel attention modules, an adaptive global average pooling layer is first used to extract channel information from the feature map, and then two layers of one-dimensional convolutional layers are used to compress and recalibrate the global information. Finally, the weight is calculated by the nonlinear activation function Sigmoid. The calculation process of the entire auxiliary branch is as follows:

[0093] F i =Conv(k i ×k i )(x)i=0,1,2,3

[0094] g i =GAP(F i )

[0095] S i =Sigmoid(W1σ(W0(g i )))

[0096] Q i =Softmax(S i )

[0097] W = concat(Q1,Q2,Q3,Q4)

[0098] Where σ represents the nonlinear activation function ReLU, Represents a one-dimensional convolution operation. By compressing and restoring the channels, the relationship between channels can be more effectively combined, which is conducive to information interaction between dimensions.

[0099] The important information in the feature map is obtained by extracting the enhanced features of the main branch and calculating the channel weights of the auxiliary branch. The two are multiplied and the channel is adjusted using a depthwise separable convolution. The calculation formula is as follows:

[0100] output = DWConv(W*Y).

[0101] It needs to be further explained that the feature fusion upsampling module fuses the semantic information of high-level features with the detailed information of low-level features, uses the rich category information of high-level features to guide the classification of low-level features, and adopts a dual-branch mode to extract information from low-level feature maps. The "bottleneck structure" mode is adopted in the left branch, and a hole convolution is introduced to increase its receptive field area. The right branch also introduces a hole convolution, and then a maximum pooling layer is passed to reduce the error of increased variance of the estimated value caused by the limited neighborhood size, retaining more image background information. For high-level features, the bilinear interpolation upsampling method is first used to upsample the high-level feature map to the same size as the low-level feature map, and then the category information is refined and extracted through the convolution layer, the high-level features are weighted and fused with the low-level features, and finally the results on both sides are added and the channel is adjusted and output through a 1×1 convolution.

[0102] Since remote sensing images contain rich category information, ordinary networks are easily disturbed by complex backgrounds when identifying clouds or cloud shadows. Therefore, in order to accurately identify the clouds and cloud shadows in the image, we added a multi-branch residual context semantic module (MRCS) to the network. The overall structure is shown in the figure below. Figure 2 As shown in the figure, adding the MRCS module to the overall network can effectively extract the feature interaction information between different spatial positions and the mutual connection between channels.

[0103] In this module, the whole adopts the form of residual structure, and realizes the identity mapping by making a shortcut connection to the input tensor. There is no introduction of additional parameters or computational complexity, and some original parameters of the input can be retained. The main part of this module adopts a dual-branch connection method. The left branch is a contextual semantic branch, which is used to extract contextual semantic information and incorporates a self-attention mechanism to focus on meaningful information in the feature map; the right branch is an enhanced feature extraction branch, which introduces two strip convolutions while using a 3×3 convolution block to enhance feature extraction and achieve improved results.

[0104] In the process of cloud and cloud shadow detection, the boundaries of clouds and cloud shadows are full of irregular shapes and are disturbed by complex backgrounds, so it is very difficult to accurately identify the boundary information of clouds and cloud shadows. Therefore, we propose a multi-scale convolutional sub-channel attention module (MCSA) here. The entire module can not only extract the connection between deep channels, but also extract information of different scales through grouped convolutions and further extract deep semantic information through the improved pyramid pooling module. Strip pooling is used to accurately extract the boundary information of clouds and cloud shadows and improve the problem of rough segmentation boundaries. The entire module structure is as follows: Figure 3 shown.

[0105] Finally, a feature fusion upsampling module (FFU) is added when the decoder upsamples and outputs the segmented images of clouds and cloud shadows, such as Figure 4 The figure shows the structure of the entire module. During the upsampling process, the semantic information of high-level features is integrated with the detailed information of low-level features, and the rich category information of high-level features is used to guide the classification of low-level features.

[0106] 3. Use the data set to train the network model:

[0107] The present invention adopts a supervised training mode, firstly performs data enhancement operation on the pictures in the data set, randomly rotates the pictures in the data set with a probability of 0.2, and performs horizontal flip or vertical flip operation, then converts the pictures and labels into tensors as input sequences and sets the batch size of each input to 8, adopts an equal interval adjustment learning rate (StepLR) strategy to reduce the learning rate accordingly as the number of training times increases to achieve better training effect, wherein the initial learning rate is set to 0.001, the attenuation coefficient is 0.98, and the learning rate is updated once every 3 training times, and a total of 200 training times are performed. The Adam algorithm is used for optimization during the training process, and the parameters of the network are updated by back propagation.

[0108] 4. Use the trained network model to predict and generate mask images of clouds and cloud shadows:

[0109] After the training is completed, the weight of the model is obtained, and then the prediction stage of the model prediction is entered. When predicting, the present invention uses the trained multi-level feature context semantic fusion network for prediction, and inputs a color picture into the model, such as Figure 6 The original color remote sensing image for the test is shown in Figure 1. After the trained weights are used to extract features and perform encoding and decoding operations, the mask image of clouds and cloud shadows is output, as shown in Figure 1. Figure 7 shown.

[0110] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0111] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A multi-level semantic fusion cloud and cloud shadow detection method, characterized in that: The method comprises the following steps: The original images directly obtained from Google Earth are 4800×2742 pixels in size. The original images are then cropped uniformly and manually labeled. The clouds in the images are labeled red, the cloud shadows are labeled green, and the background is labeled black, so that the labels correspond to the original images one by one. All images are then divided into training and validation sets in a ratio of 8:

2. Construct a multi-level feature context semantic fusion network and three related modules, including a multi-branch residual context semantic module, a multi-scale convolutional sub-channel attention module, and a feature fusion upsampling module; The multi-branch residual context semantic module adopts the form of a residual structure, and the main part adopts a dual-branch connection method. The left branch is a context semantic branch, which is used to extract the semantic information of the context, and incorporates a self-attention mechanism to focus on the meaningful information in the feature map; the right branch is an enhanced feature extraction branch, which introduces two strip convolutions while using a 3x3 convolution block to enhance feature extraction and achieve improved effects; The multi-scale convolution sub-channel attention module is used to extract the connection between deep channels, and further extract deep semantic information through the improved pyramid pooling module after extracting information of different scales through grouped convolution; The images in the dataset are augmented, and randomly rotated, horizontally flipped, or vertically flipped with a probability of 0.

2. The images and labels are then converted into tensors as input sequences. The input batch size is set to 8, the initial learning rate is set to 0.001, the decay coefficient is 0.98, and the learning rate is updated every 3 training times. A total of 200 training times are performed. The learning rate calculation formula is as follows: lr N ＝lr0×β N / s Among them lr N is the learning rate size of the Nth training, lr0 is the initial learning rate, β is the decay coefficient, s is the update interval, and the loss function selects the cross entropy loss function. The calculation formula is as follows: Where x is the output tensor of the network, class is the real label, and the Adam algorithm is used as the optimizer during the training process to obtain the trained multi-level feature context semantic fusion network; A color image is input, and the color image is encoded and decoded by the trained multi-level feature context semantic fusion network to output the mask image of clouds and cloud shadows.

2. The cloud and cloud shadow detection method of multi-level semantic fusion according to claim 1, characterized in that: The multi-branch residual context semantic module realizes identity mapping by performing a shortcut connection on the input tensor, which neither introduces additional parameters nor computational complexity, and retains some original parameters of the input; First, the input tensor is compressed through a 1x1 convolution to reduce the channel size to half of the original value. Then, the output tensors are fused after being output through two branches, and a residual connection is formed with the input end before passing through a dual-channel attention module to model the importance of each channel feature. The context semantic branch is introduced in the multi-branch residual context semantic module, which combines convolution with attention and uses convolution to extract local information in the feature map. First, for the input features, a 3×3 convolution is used to map the input features to obtain the key with local context information, marked as K. A 1×1 convolution is used to map the input features to obtain the value, marked as V, while the query still uses the original value, marked as Q. Then, the key and query are concat and two consecutive convolution operations are performed to obtain the weighted Attentionmap of each pixel, marked as A: A = Conv(Conv(concat[K,Q])) Conv(.) represents the convolution operation, and concat(.) represents the connection of multiple tensors in the channel dimension. Different from the traditional attention mechanism, A here is obtained by the interaction between Q and local context semantic information K. The interaction between local context semantic information is used to enhance the attention mechanism. Then, A is multiplied by V to obtain the dynamic context information X: X=A*V Finally, K with local context information is added to X with global context information to get the output result of the left branch; The right branch introduces a set of 1×3 and 3×1 strip convolution blocks, which use 3×3, 1×3 and 3×1 convolution kernels to replace the single 3×3 convolution kernel for calculation, and then add the three outputs to get the final output.

3. The cloud and cloud shadow detection method of multi-level semantic fusion according to claim 1, characterized in that: The multi-scale convolution sub-channel attention module uses strip pooling to accurately extract the boundary information of clouds and cloud shadows, improves the problem of rough segmentation boundaries, and uses multi-scale convolution to extract information of different scales to improve the expression ability of the model. In the main branch, convolution kernels of different sizes of 1×1, 3×3, 5×5, and 7×7 are first used to extract information of different scales in deep features, so as to better extract information in the feature map. Then, the improved pyramid pooling layer is used to refine the edge information of clouds and cloud shadows in the feature map, and finally spliced ​​and output. The calculation process of the main branch is as follows: C i =Conv(k i ×k i )(x) i=0,1,2,3 A i1 =Avg(k i ×1)(C i ) i=0,1,2,3 A i2 =Avg(1×k i )(C i ) i=0,1,2,3 A i =A i1 +A i2 Y=DWConv(concat[A0,A1,A2,A3]) Where Conv(.) represents convolutions of different scales, k i Represents the convolution kernel size, Avg(.) represents strip pooling of different sizes, DWConv(.) represents depthwise separable convolution, and concat(.) represents concatenation operation; The input feature map is split on the channel and divided into different sub-channels. Then, different convolution kernels are used on each sub-channel to obtain different feature maps on the channel. Then, the attention at different scales is extracted through channel attention to obtain the channel attention vector. Finally, Softmax is used to further calibrate the channel information. The channel attention module allows the importance of each channel to be evaluated, so as to focus on important information. Like most channel attention modules, an adaptive global average pooling layer is first used to extract channel information from the feature map, and then two layers of one-dimensional convolutional layers are used to compress and recalibrate the global information. Finally, the weight is calculated by the nonlinear activation function Sigmoid. The calculation process of the entire auxiliary branch is as follows: F i =Conv(k i ×k i )(x) i=0,1,2,3 g i =GAP(F i ) S i =Sigmoid(W1σ(W0(g i ))) Q i =Softmax(S i ) W = concat(Q1,Q2,Q3,Q4) Where σ represents the nonlinear activation function ReLU, Represents a one-dimensional convolution operation. By compressing and restoring the channels, the relationship between channels can be more effectively combined, which is conducive to information interaction between dimensions. GAP(.) represents a global average pooling operation. The important information in the feature map is obtained by extracting the enhanced features of the main branch and calculating the channel weights of the auxiliary branch. The two are multiplied and finally the channel is adjusted using the depthwise separable convolution. The calculation formula is as follows: output = DWConv(W*Y).

4. The cloud and cloud shadow detection method of multi-level semantic fusion according to claim 1, characterized in that: The feature fusion upsampling module fuses the semantic information of high-level features with the detail information of low-level features, uses the rich category information of high-level features to guide the classification of low-level features, and adopts a dual-branch mode to extract information from low-level feature maps. The "bottleneck structure" mode is adopted in the left branch, and a hole convolution is introduced to increase its receptive field area. The right branch also introduces a hole convolution, and then a maximum pooling layer is used to reduce the error of increased variance of the estimated value caused by the limited neighborhood size, retaining more image background information. For high-level features, the bilinear interpolation upsampling method is first used to upsample the high-level feature map to the same size as the low-level feature map, and then the category information therein is refined and extracted through the convolution layer, the high-level features are weighted and fused with the low-level features, and finally the results on both sides are added and the channel is adjusted and output through a 1×1 convolution.

5. The cloud and cloud shadow detection method of multi-level semantic fusion according to claim 1, characterized in that: The multi-level feature context semantic fusion network adopts a multi-level feature fusion method to fuse deep features with shallow features and add an attention mechanism. The overall structure is an encoder-decoder. First, the residual network ResNet-18 is used as the backbone network and feature extraction is performed by layer-by-layer downsampling. Then, the output features are respectively passed through a multi-branch residual context semantic module and a multi-scale convolutional sub-channel attention module for context semantic extraction and feature enhancement. A large number of feature transmission channels and fusion operations are added to fuse information from different feature layers. The encoder part aggregates deep information and shallow information while upsampling layer by layer, so that the context information in the network is propagated to higher resolution layers. Finally, a residual module is used at the output end for channel step refinement.

6. The cloud and cloud shadow detection method of multi-level semantic fusion according to claim 1, characterized in that: The cropping method is sliding coverless cutting, and the cropped image has 3 channels and a size of 224 in width and height.

7. The cloud and cloud shadow detection method of multi-level semantic fusion according to claim 6, characterized in that: The size of the cropped image is 224×224.

8. The cloud and cloud shadow detection method of multi-level semantic fusion according to claim 2, characterized in that: After the dual-branch feature extraction, it enters a dual-channel attention module. Since the common channel attention mostly directly adopts global average pooling, which will lead to the loss of edge information, two types of global pooling operations are used, namely global average pooling and global maximum pooling, to make the extracted feature information richer. Then, the 1x1 convolution operation is used to extract the mutual information between channels, focusing on the importance of different feature information between different channels. The calculation formula is as follows: g max =Conv1d(GMP(x)) g avg =Conv1d(GAP(x)) Among them, GMP(.) and GAP(.) are global maximum pooling and global average pooling operations respectively, Conv1d(.) is a one-dimensional convolution operation, and then the results of the two branches are spliced ​​and then subjected to a convolution operation, adaptively focusing on the feature information of global average pooling and maximum pooling, strengthening the focus on useful information and weakening the focus on useless information. Finally, the nonlinear activation function Sigmoid(.) is multiplied with the input original feature layer to obtain the final output result. The calculation process is as follows: g=concat(g max ,g avg ) W = Sigmoid(Conv1d(g)) y out =x*W Among them, concat(.) represents the concatenation operation, Sigmoid(.) represents the nonlinear activation function, and Conv1d(.) represents the one-dimensional convolution operation.

9. A device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a multi-level semantic fusion cloud and cloud shadow detection method as described in any one of claims 1-8.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute a multi-level semantic fusion cloud and cloud shadow detection method as described in any one of claims 1-8 when executed by a computer processor.

Images