A multi-scale aggregation cloud and cloud shadow identification method, system, device and storage medium

Abstract

The present invention discloses a multi-scale aggregation cloud and cloud shadow identification method, system, device and storage medium, which belong to the field of image processing technology. The method comprises: obtaining a test picture; inputting the test picture into a pre-trained multi-scale attention feature aggregation network to obtain a mask image of clouds and cloud shadows, and completing the identification of clouds and cloud shadows; using trained weight extraction features to perform encoding and decoding operations and then outputting the mask image of clouds and cloud shadows. The method can effectively improve the accuracy of cloud and cloud shadow identification, effectively reduce the interference of complex background and noise in the image, effectively capture scattered small-scale cloud and cloud shadow targets, enhance the detection ability of thin clouds, and refine the segmentation of irregular junctions of clouds and cloud shadows, improve the segmentation accuracy of complex edge details of clouds and cloud shadows, and also has good results in segmentation experiments of other targets, with excellent generalization ability and good robustness.

Description

Technical Field

[0001] The present invention relates to a multi-scale aggregated cloud and cloud shadow identification method, system, device and storage medium, and belongs to the field of image processing technology. Background Art

[0002] With the development of remote sensing technology, remote sensing images have been widely used in many fields such as agriculture, meteorology, and military. Since 67% of the Earth's surface is covered by clouds, many areas in remote sensing images are often covered by clouds, which causes the ground information we obtain to be attenuated or even directly lost. Therefore, the accurate identification of clouds and cloud shadows is of great significance for the application of optical remote sensing images. Due to the susceptibility to factors such as ground interference and noise interference and the lack of generalization ability, traditional deep learning networks, if directly applied to cloud detection, are prone to loss of details and spatial information, resulting in rough segmentation of cloud and cloud shadow boundaries, image omissions, and false detections. Summary of the Invention

[0003] The purpose of the present invention is to provide a multi-scale aggregation cloud and cloud shadow identification method, system, device and storage medium to solve the problems of rough segmentation of cloud and cloud shadow boundaries, image omission and false detection, and improve the accuracy of cloud and cloud shadow identification.

[0004] To achieve the above objectives, the present invention is implemented by adopting the following technical solutions:

[0005] In a first aspect, the present invention provides a multi-scale aggregation cloud and cloud shadow identification method, comprising:

[0006] Get the image to be tested;

[0007] The image to be tested is input into the pre-trained multi-scale attention feature aggregation network to obtain the mask image of clouds and cloud shadows, thus completing the recognition of clouds and cloud shadows.

[0008] In combination with the first aspect, further, the multi-scale attention feature aggregation network is trained by the following method:

[0009] Get training data;

[0010] Perform data augmentation on the images in the training data, and then convert the images and corresponding labels into tensors;

[0011] The tensor is input into a multi-scale attention feature aggregation network for training to obtain a trained multi-scale attention feature aggregation network.

[0012] In combination with the first aspect, further, the multi-scale attention feature aggregation network includes a multi-scale strip pooling attention module, which is composed of 4 parallel strip average pooling branches and an adaptive average pooling branch, two parallel strip convolution branches, a spatial attention module, and a channel attention module, for extracting multi-scale contextual information and deep spatial and channel information;

[0013] Four strip average pooling branches and one adaptive average pooling branch are used to extract and add the test image in parallel, obtain multi-scale feature maps, restore them to the size of the test image, and connect them together in the height dimension to obtain the weight vector;

[0014] The weight vectors are input into two parallel strip convolution branches respectively. The first branch consists of a convolution kernel of 1×7 and a convolution kernel of 7×1, and the second branch consists of a convolution kernel of 7×1 and a convolution kernel of 1×7. The first branch extracts the feature map and inputs it into the spatial attention module to extract the first feature map containing spatial information. The second branch extracts the feature map and inputs it into the channel attention module to extract the second feature map containing channel information. Then, the final feature map is output after the connection interaction.

[0015] The calculation process of the channel attention module is as follows:

[0016] Use global average pooling and global maximum pooling to extract features respectively:

[0017]

[0018]

[0019] Among them, x represents the input feature map, and Represents the second weight vector and the third weight vector of the global maximum pooling branch and the global average pooling branch output, Gmax and Gavg represent the global maximum pooling and the global average pooling, respectively. C2D 1×1 Indicates a two-dimensional convolution with a convolution kernel of 1×1;

[0020] Concatenate the features extracted by global average pooling and global maximum pooling:

[0021]

[0022] Among them, CAT3 represents splicing in the width dimension. is the image after splicing in the width dimension;

[0023] Size recovery, feature selection, reweighting:

[0024]

[0025] Among them, CA(x) represents the first feature map output by the channel attention module, DWC2D 1×2 Denotes a two-dimensional depth-separable convolution with a convolution kernel of 1×2, DWC2D 1×1 represents a two-dimensional depth-separable convolution with a convolution kernel of 1×1, and σ represents the nonlinear activation function Sigmoid;

[0026] The calculation process of the spatial attention module is as follows:

[0027] After extracting features using global average pooling and global maximum pooling, they are connected along the channel dimension, a convolution operation is performed, and then a nonlinear activation function is used to generate the second feature map:

[0028] SA(x)=σ(C2D 7×7 (CAT1(MP(x),AP(x))))

[0029] Among them, SA(x) represents the second feature map output by the spatial attention module, C2D 7×7 It represents a two-dimensional convolution with a convolution kernel of 7×7, CAT1 represents splicing in the channel dimension, MP and AP represent maximum pooling and average pooling respectively.

[0030] In combination with the first aspect, the multi-scale attention feature aggregation network further includes a deep multi-head feedforward transfer attention module, which is used to promote the mutual guidance of two adjacent layers of the backbone network in the multi-scale attention feature aggregation network to perform feature mining, and fuse the feature map information of the two adjacent layers extracted from the backbone network;

[0031] First, perform layer normalization on the feature maps output by two adjacent layers to generate the first layer normalized tensor and the second layer normalized tensor. The query vector is generated from the first layer normalized tensor, and the key vector and value vector are generated from the second layer normalized tensor. The calculation process is as follows:

[0032]

[0033]

[0034]

[0035] Where X is the first layer normalized tensor, Y is the second layer normalized tensor, Q is the query vector, K is the key vector, and V is the value vector. Indicates that the convolution kernel for calculating the query vector is a 1×1 two-dimensional convolution. Indicates that the convolution kernel for calculating the query vector is a 3×3 two-dimensional depth-separable convolution, Indicates that the convolution kernel for calculating the key vector is a 1×1 two-dimensional convolution, Indicates that the convolution kernel for calculating the key vector is a 3×3 two-dimensional depth-separable convolution, The convolution kernel of the calculated value vector is a 1×1 two-dimensional convolution. The convolution kernel representing the calculated value vector is a 3×3 two-dimensional depth-wise separable convolution;

[0036] Reshape the query vector and key vector so that their dot products interact to generate the transposed attention map:

[0037] Attention(Q′,K′,V′)=V′·Softmax(K′·V′ / β)

[0038] P′=C2D 1×1 Attention(Q′,K′,V′)+x+y

[0039] Where x and y represent the feature maps of the shallow and deep inputs respectively, P′ is the transposed attention feature map of the output, Q′, K′, V′ are the three matrices obtained after reshaping the tensor from the original size, β is the scaling parameter, and Attention(Q′, K′, V′) is the attention function;

[0040] Input the transposed attention feature map into the feedforward network to obtain the feedforward feature map:

[0041] Z1=DWC2D 3×3 ((C2D 1×1 (Z))

[0042] Z2=δ(Z1)⊙Z1

[0043] Z′=C2D 1×1 (Z2)+Z

[0044] Among them, Z is the third layer normalized tensor obtained by layer normalization of the transposed attention feature map, C2D 1×1 It is a two-dimensional convolution with a convolution kernel of 1×1, DWC2D 3×3 It is a two-dimensional depth-wise separable convolution with a convolution kernel of 3×3, Z1 is the first feedforward intermediate map, ⊙ represents the dot product, δ is the Gelu nonlinear activation function, Z2 is the second feedforward intermediate map, and Z′ is the output feedforward feature map.

[0045] In combination with the first aspect, further, the multi-scale attention feature aggregation network includes a bilateral feature fusion module, including a detail branch and a context branch;

[0046] The feature map output by the detail branch is input into two branches at the same time to obtain the detail output value after the two feature maps:

[0047]

[0048]

[0049] Among them, x_d is the feature map output by the detail branch, represents a two-dimensional depth-wise separable convolution with a convolution kernel of 3×3 and a dilation factor of 2. BN represents batch normalization, R represents the ReLU activation function, AP represents average pooling, and Y1 and Y2 represent the detail output values ​​after the two feature maps respectively.

[0050] The feature map output by the context branch is input into two branches at the same time, and the context output value after the two feature maps is obtained:

[0051] y1=γ1·σ(x_c up )

[0052]

[0053] Among them, x_c up is the feature map output by the context branch, σ represents the nonlinear activation function Sigmoid, Indicates a two-dimensional convolution with a convolution kernel of 3×3 and a dilation factor of 2, DWC2D 3×3 Indicates that the convolution kernel is a 3×3 two-dimensional depth-separable convolution, y1 and y2 represent the context output values ​​after the two feature maps respectively;

[0054] The context output values ​​after the two feature maps are added together and then subjected to a two-dimensional depth-wise separable convolution, followed by batch normalization and activation processing to obtain a bilateral feature fusion feature map:

[0055] y out =R(BN(DWC2D 3×3 (y1+y2)))

[0056] Among them, y out It is the bilateral feature fusion feature map.

[0057] In combination with the first aspect, the multi-scale attention feature aggregation network further includes a boundary refinement boosting module for enhancing the detection of complex edge information of clouds and cloud shadows. The calculation process of the boundary refinement boosting module is as follows:

[0058] x′=C2D 3×3 (C2D 3×3 (x))+x

[0059] y=drop(C2D 3×3 (x′))

[0060] y′=Up(C2D 1×1 (y))

[0061] Among them, x and y′ represent the input value and output value of the boundary refinement boosting module respectively, C2D 1×1 Indicates a 1×1 two-dimensional convolution kernel, Up indicates 2x upsampling, C2D 3×3 represents a two-dimensional convolution with a convolution kernel of 3×3, drop represents the dropout algorithm, x′ represents the first refined intermediate value, and y represents the second refined intermediate value.

[0062] In a second aspect, the present invention further provides a multi-scale aggregated cloud and cloud shadow identification system, comprising:

[0063] Image acquisition module: used to obtain the image to be tested;

[0064] Cloud and cloud shadow recognition module: This module inputs the image to be tested into a pre-trained multi-scale attention feature aggregation network to obtain mask images of clouds and cloud shadows, thereby completing the recognition of clouds and cloud shadows.

[0065] In a third aspect, the present invention further provides a multi-scale aggregated cloud and cloud shadow identification device, comprising a processor and a storage medium;

[0066] The storage medium is used to store instructions;

[0067] The processor is configured to operate according to the instructions to execute the steps of the method according to any one of the first aspects.

[0068] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] The present invention provides a multi-scale aggregation cloud and cloud shadow identification method, system, device, and storage medium. The method inputs a test image into a pre-trained multi-scale attention feature aggregation network, uses trained weights to extract features, performs encoding and decoding operations, and outputs a mask image of the cloud and cloud shadow. This method can effectively improve the accuracy of cloud and cloud shadow identification.

[0071] A multi-scale strip pooling attention module is used to further extract multi-scale contextual information and deep spatial and channel information. Clouds and cloud shadows are classified based on this contextual information, and the edges between clouds and cloud shadows are segmented and detailed. Strip pooling can reduce interference from other irrelevant areas in the image, effectively reducing the interference of complex background and noise in the image, and can effectively capture scattered small-scale cloud and cloud shadow targets.

[0072] A deep multi-head feedforward transfer attention module is used to enhance the communication capabilities of the two channels, promote mutual guidance between two adjacent layers of the backbone network for feature mining, and fuse the feature map information extracted from the two adjacent layers of the backbone network, thereby enhancing the detection ability of thin clouds and refining the segmentation of irregular junctions between clouds and cloud shadows.

[0073] A bilateral feature fusion module is used to fuse low-level semantic information and high-level detail information. The semantic information of the context branch is used to guide the feature response of the detail branch, achieving efficient information exchange and reducing the impact of interfering objects on recognition.

[0074] A boundary refinement boosting module is used to enhance feature representation during the training phase, which improves the segmentation accuracy of complex edge details of clouds and cloud shadows. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 Schematic diagram of the structure of the multi-scale attention feature aggregation network provided by an embodiment of the present invention;

[0076] Figure 2 is a schematic diagram of the structure of a channel attention module provided by an embodiment of the present invention;

[0077] Figure 3 is a schematic structural diagram of a spatial attention module provided by an embodiment of the present invention;

[0078] Figure 4 2 is a schematic diagram of the structure of a deep multi-head feedforward attention transfer module provided by an embodiment of the present invention;

[0079] Figure 5 is a structural diagram of a bilateral feature fusion module provided by an embodiment of the present invention;

[0080] Figure 6 2 is a schematic structural diagram of a boundary refinement and boosting module provided by an embodiment of the present invention;

[0081] Figure 7 This is a flowchart of a multi-scale aggregation cloud and cloud shadow identification method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0082] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0083] Example 1

[0084] like Figure 7 As shown, an embodiment of the present invention provides a multi-scale aggregation cloud and cloud shadow identification method, which is characterized by including:

[0085] S1. Obtain the image to be tested.

[0086] Collect an original color image as the image to be tested.

[0087] S2. Input the image to be tested into the pre-trained multi-scale attention feature aggregation network to obtain the mask image of the cloud and cloud shadow, and complete the identification of the cloud and cloud shadow.

[0088] Build as Figure 1 The multi-scale attention feature aggregation network shown in the figure has an encoder-decoder structure and adopts an end-to-end training method. It is mainly composed of a multi-scale strip pooling attention module, a deep multi-head feedforward transfer attention module, a bilateral feature fusion module and a boundary refinement boosting module.

[0089] Since the extraction of feature information in the image is very important in the process of identifying clouds and cloud shadows in remote sensing images, selecting a suitable backbone network can greatly improve the detection efficiency of the network. In this embodiment of the present invention, a residual network is selected as the backbone network to perform feature extraction operations.

[0090] The multi-scale strip pooling attention module (MSPA) is used to further extract multi-scale contextual information and deep spatial and channel information; it consists of a parallel 4-strip average pooling branch and an adaptive average pooling branch, two parallel strip convolution branches, a spatial attention module, and a channel attention module.

[0091] The pooling kernel of the 4-band average pooling branch is N×1 (N=1, 3, 5, 6), and the pooling kernel of the adaptive average pooling branch is 1×N, which is used to extract and add input features in parallel to obtain a multi-scale feature map. The size of the multi-scale feature map is restored to the size of the input image to be tested, and they are connected together in the height dimension to obtain a weight vector to complete the extraction of multi-scale features.

[0092] The weight vectors are input into two parallel strip convolution branches respectively. The first branch consists of convolution kernel 1×7 and convolution kernel 7×1, and the second branch consists of convolution kernel 7×1 and convolution kernel 1×7. After the first branch extracts the feature map, it is input into the spatial attention module to extract the first feature map containing spatial information. After the second branch extracts the feature map, it is input into the channel attention module to extract the second feature map containing channel information. Finally, the image information of the two branches is connected and interacted to output the final feature map.

[0093] Horizontal strip convolution tends to learn some horizontal details in cloud and cloud shadow images, while vertical strip convolution tends to learn some vertical details. The edges of clouds and cloud shadows are often closely connected. The above operation can well capture the edge feature information at the connection between clouds and cloud shadows, and improve the edge segmentation effect.

[0094] The structure of the channel attention module is as follows Figure 2 As shown in the figure, its core content is to use two types of global pooling to extract high-level features, namely global average pooling and global maximum pooling. Using different global pooling means that the extracted high-level features are richer. Then, two-dimensional depth-separable convolution is used as the extractor of information between channels. The end point focuses on the importance of features in different channels. The calculation process of the channel attention module is as follows:

[0095] Use global average pooling and global maximum pooling to extract features respectively:

[0096]

[0097]

[0098] Among them, x represents the input feature map, and Represents the second weight vector and the third weight vector of the global maximum pooling branch and the global average pooling branch output, Gmax and Gavg represent the global maximum pooling and the global average pooling, respectively. C2D 1×1 Indicates a two-dimensional convolution with a convolution kernel of 1×1;

[0099] Then the features extracted by global average pooling and global maximum pooling are spliced:

[0100]

[0101] Among them, CAT3 represents splicing in the width dimension. is the image after splicing in the width dimension;

[0102] Next, a two-dimensional depth-wise separable convolution with a convolution kernel of 1×2 is used to restore the size of the spliced ​​image, focusing on the detailed feature information of the image; then a two-dimensional depth-wise separable convolution with a convolution kernel of 1×1 is used as a selector to adaptively focus on the feature representation of the global average pooling branch and the global maximum pooling branch; finally, after the selector output, the nonlinear activation function Sigmoid is used to reweight the original feature map.

[0103] The calculation process of the above size recovery, feature selection, and reweighting is as follows:

[0104]

[0105] Among them, CA(x) represents the first feature map output by the channel attention module, DWC2D 1×2 Denotes a two-dimensional depth-separable convolution with a convolution kernel of 1×2, DWC2D 1×1represents a two-dimensional depth-wise separable convolution with a convolution kernel of 1×1, and σ represents the nonlinear activation function Sigmoid.

[0106] The structure of the spatial attention module is as follows Figure 3 As shown in the figure, it uses average pooling and maximum pooling to extract feature information. Unlike the channel attention module, the two aggregation methods in the spatial attention module are performed along the channel dimension. After connecting the feature map results generated by average pooling and maximum pooling in the channel dimension, a convolution operation with a convolution kernel of 7×7 is performed to reduce the number of channels from 2 to 1. Through the convolution operation with a large convolution kernel such as 7×7, a relatively large receptive field can be extracted. Finally, the nonlinear activation function Sigmoid is used to generate the final feature map.

[0107] The calculation process of the above spatial attention module is as follows:

[0108] After extracting features using global average pooling and global maximum pooling, they are connected along the channel dimension, a convolution operation is performed, and then a nonlinear activation function is used to generate the second feature map:

[0109] SA(x)=σ(C2D 7×7 (cAT1(MP(x),AP(x))))

[0110] Among them, SA(x) represents the second feature map output by the spatial attention module, C2D 7×7 It represents a two-dimensional convolution with a convolution kernel of 7×7, CAT1 represents splicing in the channel dimension, MP and AP represent maximum pooling and average pooling respectively.

[0111] The structure of the deep multi-head feedforward transfer attention module (DMFA) is as follows Figure 4 As shown, it is used to promote the mutual guidance of two adjacent layers of the backbone network in the multi-scale attention feature aggregation network for feature mining, and to fuse the feature map information of the two adjacent layers extracted from the backbone network.

[0112] The deep multi-head feedforward transfer attention module first performs layer normalization on the feature maps output by two adjacent layers to generate the first layer normalized tensor X∈R H×W×C and the second layer normalized tensor Y∈R H×W×C , the query vector Q is generated by the first layer normalized tensor, the key vector K and value vector V are generated by the second layer normalized tensor, and are enriched with local context information. This is achieved by using 1×1 convolution to aggregate pixel-level cross-channel context information, and then using 3×3 depth convolution to encode channel-level spatial context information. The calculation process is as follows:

[0113]

[0114]

[0115]

[0116] Where X is the first layer normalized tensor, Y is the second layer normalized tensor, Q is the query vector, K is the key vector, and V is the value vector. Indicates that the convolution kernel for calculating the query vector is a 1×1 two-dimensional convolution. Indicates that the convolution kernel for calculating the query vector is a 3×3 two-dimensional depth-separable convolution, Indicates that the convolution kernel for calculating the key vector is a 1×1 two-dimensional convolution, Indicates that the convolution kernel for calculating the key vector is a 3×3 two-dimensional depth-separable convolution, The convolution kernel of the calculated value vector is a 1×1 two-dimensional convolution. The convolution kernel representing the calculated value vector is a 3×3 two-dimensional depth-wise separable convolution.

[0117] Then reshape the query vector and key vector so that their dot product interaction produces a shape of R C×C The transposed attention map is calculated as follows:

[0118] Attention(Q′,K′,V′)=V′·Softmax(K′·V′ / β)

[0119] P′=C2D 1×1 Attention(Q′,K′,V′)+x+y

[0120] Among them, x and y represent the feature maps of the shallow and deep input respectively, P′ is the output transposed attention feature map, Q′, K′, V′ are three matrices obtained after reshaping the tensor from the original size, Q′∈R HW×C , K′∈ R C×HW , V′∈R HW×C , β is a scaling parameter used to control the size of the dot product of K′ and V′ before performing the Softmax function, and Attention(Q′, K′, V′) is the attention function.

[0121] After the feature map information is processed, the transposed attention feature map is input into the feedforward network, which performs the same operation on each pixel position of the input transposed attention feature map. First, the third layer normalized tensor Z is obtained through layer normalization operation, Z∈R H×W×C, then a two-dimensional convolution with a convolution kernel of 1×1 is used to expand the feature channel, and a two-dimensional depth-separable convolution with a convolution kernel of 3×3 is used to encode information from spatially adjacent pixel positions; then a gating mechanism is used to split the feature map information after the depth-separable convolution in the channel dimension and pass it through two parallel branches, one of which passes through the Gelu nonlinear activation function, multiplies the feature maps output by the two parallel branches, and then passes through a 1×1 convolution to reduce the channel back to the original input dimension to obtain the feedforward feature map. The above calculation process is as follows:

[0122] Z1=DWC2D 3×3 ((C2D 1×1 (Z))

[0123] Z2=δ(Z1)⊙Z1

[0124] Z′=C2D 1×1 (Z2)+Z

[0125] Among them, Z is the third layer normalized tensor obtained by layer normalization of the transposed attention feature map, C2D 1×1 It is a two-dimensional convolution with a convolution kernel of 1×1, DWC2D 3×3 It is a two-dimensional depth-wise separable convolution with a convolution kernel of 3×3, Z1 is the first feedforward intermediate map, ⊙ represents the dot product, δ is the Gelu nonlinear activation function, Z2 is the second feedforward intermediate map, and Z′ is the output feedforward feature map.

[0126] The structure of the bilateral feature fusion module (BFF) is as follows Figure 5 As shown in the figure, it includes detail branches and context branches; it is used to fuse low-level semantic information and high-level detail information, improve the overall anti-interference ability of the model, and reduce the impact of interference objects and noise in the image on cloud and cloud shadow prediction; this module uses the semantic information of the context branch to guide the feature response of the detail branch. Through guidance at different scales, we can extract feature representations at different scales.

[0127] The feature map output by the detail branch is input into two branches at the same time to obtain the detail output values ​​after two feature maps; one branch enters the depthwise separable dilated convolution with a convolution kernel of 3×3 and a dilation factor of 2. The use of dilated convolution can greatly improve the receptive field without adding additional parameters. By increasing the receptive field and enhancing the contextual information, the accuracy of the segmentation boundary can be better improved; the other branch enters the depthwise separable dilated convolution with a convolution kernel of 3×3 and a dilation factor of 2, as well as the average pooling layer; batch normalization and ReLU activation functions are added to both branches to make the network converge faster, more stable, and prevent overfitting; the above calculation process is as follows:

[0128]

[0129]

[0130] Among them, x_d is the feature map output by the detail branch, It represents a two-dimensional depth-wise separable convolution with a convolution kernel of 3×3 and a dilation factor of 2. BN represents batch normalization, R represents the ReLU activation function, AP represents average pooling, and Y1 and Y2 represent the detail output values ​​after the two feature maps, respectively.

[0131] Since the pixel sizes of the feature maps output by the detail branch and the context branch are different, the size of the feature map output by the detail branch is twice that of the context branch. Therefore, the feature map output by the context branch is input into both branches at the same time, and both branches are first upsampled. In one branch, the feature map is directly Sigmoid activated after upsampling. In the other branch, the feature map after upsampling is first subjected to dilated convolution and then to depthwise separable convolution. After the feature map passes through the convolution layer, Sigmoid activation is performed. After the detail branch feature mapping, the Sigmoid activation value is reweighted to the feature map to obtain the context output value after the two feature maps:

[0132] y1=γ1·σ(x_c up )

[0133]

[0134] Among them, x_c up is the feature map output by the context branch, σ represents the nonlinear activation function Sigmoid, Indicates a two-dimensional convolution with a convolution kernel of 3×3 and a dilation factor of 2, DWC2D 3×3 It indicates that the convolution kernel is a 3×3 two-dimensional depth-separable convolution, and y1 and y2 represent the context output values ​​after the two feature maps respectively.

[0135] The results at different scales are summarized and feature information is further extracted. That is, y1 and y2 are added and then passed through a depthwise separable convolution layer. That is, the context output values ​​after the two feature maps are added and then passed through a two-dimensional depthwise separable convolution. Then, batch normalization and activation processing are added to obtain the bilateral feature fusion feature map:

[0136] y out =R(BN(DWC2D 3×3 (y1+y2)))

[0137] Among them, y out It is the bilateral feature fusion feature map.

[0138] The structure of the boundary refinement boosting module (BRB) is as follows Figure 6As shown in the figure, it is used to enhance the detection of complex edge information of clouds and cloud shadows. It re-predicts the boundary details of clouds and cloud shadows through end-to-end training. In addition, to address the problem of unsatisfactory segmentation prediction images when segmentation accuracy is low, the BRB module proposes an enhanced training strategy. Feature representation is enhanced during the training phase. A dropout step is added during the intermediate training process. During the prediction phase, neurons in the network can be discarded with a probability of 0.1. This can improve segmentation accuracy to a certain extent, obtain prediction images with good segmentation effects, and prevent network overfitting. The calculation process of the boundary refinement boosting module is as follows:

[0139] x′=C2D 3×3 (C2D3×3(x))+x

[0140] y=drop(C2D 3×3 (x′))

[0141] y′=Up(C2D 1×1 (y))

[0142] Among them, x and y′ represent the input value and output value of the boundary refinement boosting module respectively, C2D 1×1 Indicates a 1×1 two-dimensional convolution kernel, Up indicates 2x upsampling, C2D 3×3 represents a two-dimensional convolution with a convolution kernel of 3×3, drop represents the dropout algorithm, x′ represents the first refined intermediate value, and y represents the second refined intermediate value.

[0143] For deep neural networks, capturing long-range correlations is crucial; however, convolution operations are used to process local areas and have a limited receptive field, making it difficult to capture the correlation of long-range feature information; pooling operations using large square kernels can increase the sharing of global information. This method works well when detecting large-scale clouds, but it is ineffective for scattered small-scale cloud clusters. Because the large square kernel extracts too much information from irrelevant areas, this will interfere with the model's final prediction and reduce the accuracy of segmentation; to address the above problems, the present invention proposes a multi-scale strip pooling attention module (MSPA) to further extract multi-scale contextual information and deep spatial and channel information.

[0144] On the one hand, clouds and cloud shadows have similar shapes, so they can be classified through contextual information, and the edges between clouds and cloud shadows are processed and segmented in detail. Strip pooling can reduce the interference of other irrelevant areas in the image and more effectively identify scattered small-sized clouds and cloud shadows, thereby reducing the probability of missed detection and false detection of detection targets and improving the segmentation effect; on the other hand, after the strip pooling operation, the attention mechanism operation is immediately entered to extract multi-scale deep spatial information and multi-scale deep channel information in parallel, and better pay attention to the category information and location information of clouds and cloud shadows. In this way, the model can focus on important information in the image and further improve the segmentation effect.

[0145] In order to meet the requirements of segmenting thin cloud layers and irregular junctions of clouds and cloud shadows in the cloud and cloud shadow segmentation task, the solution of the present invention chooses to promote the mutual guidance of two adjacent layers of the backbone network for feature mining, and fuses the feature map information of the two adjacent layers extracted from the backbone network; however, simply combining two feature maps of different scales will lead to the loss of the diversity of the two types of information. Therefore, the solution of the present invention designs a deep multi-head feedforward transfer attention module (DMEA) to enhance the communication capability of the two channels, so that the two adjacent layers of the backbone network guide each other for feature mining, promote the fusion of image feature information to provide more useful feature information for the upsampling process.

[0146] In order to minimize the impact of interfering objects in the image on cloud and cloud shadow prediction when segmenting clouds and cloud shadows, improve the overall anti-interference ability of the model, reduce the probability of false detection and missed detection, and further solve the problem that the irregular shapes of clouds and cloud shadows are difficult to accurately predict; in the decoding stage, the valve stem of the present invention proposes a bilateral feature fusion module (BFF) for fusing low-level semantic information and high-level detail information; the feature representations of the detail branch and the context branch are complementary, and one party does not know the information of the other party; there are several different methods to combine the two feature responses, namely element summation connection; however, the outputs of the two branches have different levels of feature representation, the detail branch is used for the lower level, and the semantic branch is used for the higher level; therefore, a simple combination will ignore the diversity of these two types of information, resulting in poor performance and difficulty in optimization; the bilateral feature fusion module can greatly improve the fusion of the feature maps output by the two branches. The module uses the semantic information of the context branch to guide the feature response of the detail branch; through guidance at different scales, we can extract feature representations of different scales. At the same time, compared with simple combination, this guidance method can achieve efficient information exchange between the two branches.

[0147] Because the size and shape of clouds and cloud shadows are arbitrary and irregular, it is difficult to detect boundary information. The segmentation boundaries of some existing methods are very rough and lack details. To address this problem, the present invention proposes a new module (BRB) to re-predict the boundary details of clouds and cloud shadows through end-to-end training. In addition, to address the problem that the segmentation prediction image is very unsatisfactory when the segmentation accuracy is not high, the BRB module proposes an enhanced training strategy, which enhances feature representation in the training phase and adds a dropout link in the intermediate training process. In the prediction phase, neurons in the network can be discarded with a probability of 0.1. Therefore, the segmentation accuracy can be improved to a certain extent, a prediction image with good segmentation effect can be obtained, and network overfitting can be prevented.

[0148] After building a multi-scale attention feature aggregation network, pre-train it.

[0149] Obtaining the training data set:

[0150] The cloud and cloud shadow datasets used in the embodiments of the present invention come from Google Earth, a virtual earth software developed by Google. It places satellite photos, aerial photos, and geographic information systems on a three-dimensional model of the earth. The effective resolution of global terrain images on Google Earth is at least 100 meters, usually 30 meters, and the observation altitude (Eyealt) is 15 kilometers.

[0151] This dataset consists of high-definition remote sensing images randomly collected by professional meteorological experts in Qinghai, Yunnan Plateau, Qinghai Plateau, Qinghai-Tibet Plateau and Yangtze River Delta. To better reflect the performance of the model, the embodiment of the present invention selected several sets of high-resolution cloud images with different shooting angles and altitudes. Due to the limitation of GPU video storage capacity, the high-definition cloud remote sensing images with an original resolution of 4800×2692 were cut into a size of 224×224. After screening, a total of 12,280 images were obtained, of which 9,824 were used as training sets and 2,456 were used as validation sets. The ratio of training sets to validation sets in the dataset is 8 / 2.

[0152] Deep neural networks require large amounts of training data, but these learning samples are difficult to obtain. Therefore, when training samples are scarce, data augmentation is essential to avoid overfitting. Therefore, the present invention employs data augmentation through translation, flipping, and rotation. High-resolution cloud and cloud shadow images obtained from Google Earth are categorized into five different background types: water, forest, field, town, and desert. These labels are manually labeled into three types: cloud (red), cloud shadow (green), and background (black).

[0153] The embodiment of the present invention adopts a supervised training method. First, data augmentation processing is performed on the images in the data set. Then, the original images and corresponding labels are converted into tensors and input into the model for training. The batch size of each training is set to 16. The embodiment of the present invention adopts an equal interval adjustment learning rate (StepLR) strategy to reduce the learning rate accordingly as the number of training times increases to achieve better training effect. The initial learning rate is set to 0.001, the decay coefficient is 0.98, and the learning rate is updated every 3 training times, for a total of 200 training times. The Adam algorithm is used as the optimizer during the training process.

[0154] After the training is completed, the model weights will be obtained. Next, the model enters the prediction stage, and the collected test images (original color images) are input into the pre-trained multi-scale attention feature aggregation network to obtain the mask images of clouds and cloud shadows, completing the identification of clouds and cloud shadows.

[0155] Example 2

[0156] An embodiment of the present invention provides a multi-scale aggregated cloud and cloud shadow identification system, comprising:

[0157] Image acquisition module: used to obtain the image to be tested;

[0158] Cloud and cloud shadow recognition module: This module inputs the image to be tested into a pre-trained multi-scale attention feature aggregation network to obtain mask images of clouds and cloud shadows, thereby completing the recognition of clouds and cloud shadows.

[0159] Example 3

[0160] An embodiment of the present invention provides a multi-scale aggregated cloud and cloud shadow identification device, comprising a processor and a storage medium;

[0161] The storage medium is used to store instructions;

[0162] The processor is configured to operate according to the instructions to execute the steps of the following method:

[0163] Get the image to be tested;

[0164] The image to be tested is input into the pre-trained multi-scale attention feature aggregation network to obtain the mask image of clouds and cloud shadows, thus completing the recognition of clouds and cloud shadows.

[0165] Example 4

[0166] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the following method are implemented:

[0167] Get the image to be tested;

[0168] The image to be tested is input into the pre-trained multi-scale attention feature aggregation network to obtain the mask image of clouds and cloud shadows, thus completing the recognition of clouds and cloud shadows.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0170] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0173] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-scale aggregation cloud and cloud shadow identification method, characterized by: include: Get the image to be tested; The image to be tested is input into the pre-trained multi-scale attention feature aggregation network to obtain the mask image of the cloud and cloud shadow, thus completing the recognition of the cloud and cloud shadow. The multi-scale attention feature aggregation network includes a multi-scale strip pooling attention module, which is composed of four parallel strip average pooling branches and an adaptive average pooling branch, two parallel strip convolution branches, a spatial attention module, and a channel attention module, which is used to extract multi-scale contextual information and deep spatial and channel information; Four strip average pooling branches and one adaptive average pooling branch are used to extract and add the test image in parallel, obtain multi-scale feature maps, restore them to the size of the test image, and connect them together in the height dimension to obtain the weight vector; The weight vectors are input into two parallel strip convolution branches respectively. The first branch consists of a convolution kernel of 1×7 and a convolution kernel of 7×1, and the second branch consists of a convolution kernel of 7×1 and a convolution kernel of 1×7. The first branch extracts the feature map and inputs it into the spatial attention module to extract the first feature map containing spatial information. The second branch extracts the feature map and inputs it into the channel attention module to extract the second feature map containing channel information. Then, the final feature map is output after the connection interaction. The calculation process of the channel attention module is as follows: Use global average pooling and global maximum pooling to extract features respectively: Among them, x represents the input feature map, and Represents the second weight vector and the third weight vector of the global maximum pooling branch and the global average pooling branch output, Gmax and Gavg represent the global maximum pooling and the global average pooling, respectively. C2D 1×1 Indicates a two-dimensional convolution with a convolution kernel of 1×1; Concatenate the features extracted by global average pooling and global maximum pooling: Among them, CAT3 represents splicing in the width dimension. is the image after splicing in the width dimension; Size recovery, feature selection, reweighting: Among them, CA(x) represents the first feature map output by the channel attention module, DWC2D 1×2 Denotes a two-dimensional depth-separable convolution with a convolution kernel of 1×2, DWC2D 1×1 represents a two-dimensional depth-separable convolution with a convolution kernel of 1×1, and σ represents the nonlinear activation function Sigmoid; The calculation process of the spatial attention module is as follows: After extracting features using global average pooling and global maximum pooling, they are connected along the channel dimension, a convolution operation is performed, and then a nonlinear activation function is used to generate the second feature map: SA(x)=σ(C2D 7×7 (CAT1(MP(x),AP(x)))) Among them, SA(x) represents the second feature map output by the spatial attention module, C2D 7×7 It represents a two-dimensional convolution with a convolution kernel of 7×7, CAT1 represents splicing in the channel dimension, MP and AP represent maximum pooling and average pooling respectively.

2. The multi-scale aggregation cloud and cloud shadow identification method according to claim 1, characterized in that: The multi-scale attention feature aggregation network is trained by the following method: Get training data; Perform data augmentation on the images in the training data, and then convert the images and corresponding labels into tensors; The tensor is input into a multi-scale attention feature aggregation network for training to obtain a trained multi-scale attention feature aggregation network.

3. The multi-scale aggregation cloud and cloud shadow identification method according to claim 1, characterized in that: The multi-scale attention feature aggregation network includes a deep multi-head feedforward transfer attention module, which is used to promote the mutual guidance of two adjacent layers of the backbone network in the multi-scale attention feature aggregation network to perform feature mining and fuse the feature map information of the two adjacent layers extracted from the backbone network; First, perform layer normalization on the feature maps output by two adjacent layers to generate the first layer normalized tensor and the second layer normalized tensor. The query vector is generated from the first layer normalized tensor, and the key vector and value vector are generated from the second layer normalized tensor. The calculation process is as follows: Where X is the first layer normalized tensor, Y is the second layer normalized tensor, Q is the query vector, K is the key vector, and V is the value vector. Indicates that the convolution kernel for calculating the query vector is a 1×1 two-dimensional convolution. Indicates that the convolution kernel for calculating the query vector is a 3×3 two-dimensional depth-separable convolution, Indicates that the convolution kernel for calculating the key vector is a 1×1 two-dimensional convolution, Indicates that the convolution kernel for calculating the key vector is a 3×3 two-dimensional depth-separable convolution, The convolution kernel of the calculated value vector is a 1×1 two-dimensional convolution. The convolution kernel representing the calculated value vector is a 3×3 two-dimensional depth-wise separable convolution; Reshape the query vector and key vector so that their dot products interact to generate the transposed attention map: Attention(Q ′ ,K ′ ,V ′ )=V ′ ·Softmax(K ′ ·V ′ / β) P ′ =C2D 1×1 Attention(Q ′ ,K ′ ,V ′ )+x+y Among them, x and y represent the feature maps of shallow and deep input respectively, P ′ is the output transposed attention feature map, Q ′ ,K ′ ,V ′ are the three matrices obtained after reshaping the tensor from its original size, β is the scaling parameter, Attention(Q ′ ,K ′ ,V ′ ) is the attention function; Input the transposed attention feature map into the feedforward network to obtain the feedforward feature map: Z1=DWC2D 3×3 ((C2D 1×1 (WITH)) Z2=δ(Z1)⊙Z1 WITH ′ =C2D 1×1 (Z2)+Z Among them, Z is the third layer normalized tensor obtained by transposing the attention feature map and performing layer normalization, C2D 1×1 It is a two-dimensional convolution with a convolution kernel of 1×1, DWC2D 3×3 It is a two-dimensional depth-separable convolution with a convolution kernel of 3×3, Z1 is the first feedforward intermediate graph, ⊙ represents the dot product, δ is the Gelu nonlinear activation function, Z2 is the second feedforward intermediate graph, Z ′ is the output feedforward feature map.

4. The multi-scale aggregation cloud and cloud shadow identification method according to claim 1, characterized in that: The multi-scale attention feature aggregation network includes a bilateral feature fusion module, including a detail branch and a context branch; The feature map output by the detail branch is input into two branches at the same time to obtain the detail output value after the two feature maps: Among them, x_d is the feature map output by the detail branch, represents a two-dimensional depth-wise separable convolution with a convolution kernel of 3×3 and a dilation factor of 2, BN represents batch normalization, R represents the ReLU activation function, AP represents average pooling, and γ1 and γ2 represent the detail output values ​​after the two feature maps respectively; The feature map output by the context branch is input into two branches at the same time, and the context output value after the two feature maps is obtained: y1=γ1·σ(x_c up ) Among them, x_c up is the feature map output by the context branch, σ represents the nonlinear activation function Sigmoid, Indicates a two-dimensional convolution with a convolution kernel of 3×3 and a dilation factor of 2, DWC2D 3×3 Indicates that the convolution kernel is a 3×3 two-dimensional depth-separable convolution, y1 and y2 represent the context output values ​​after the two feature maps respectively; The context output values ​​after the two feature maps are added together and then subjected to a two-dimensional depth-wise separable convolution, followed by batch normalization and activation processing to obtain a bilateral feature fusion feature map: y out =R(BN(DWC2D 3×3 (y1+y2))) Among them, y out It is the bilateral feature fusion feature map.

5. The multi-scale aggregation cloud and cloud shadow identification method according to claim 1, characterized in that: The multi-scale attention feature aggregation network includes a boundary refinement boosting module to enhance the detection of complex edge information of clouds and cloud shadows. The calculation process of the boundary refinement boosting module is as follows: x ′ =C2D 3×3 (C2D 3×3 (x))+x y=drop(C2D 3×3 (x ′ )) and ′ =Up(C2D 1×1 (and)) Among them, x and y ′ Represent the input value and output value of the boundary refinement boosting module, C2D 1×1 Indicates a 1×1 two-dimensional convolution kernel, Up indicates 2x upsampling, C2D 3×3 Indicates a two-dimensional convolution with a convolution kernel of 3×3, drop indicates the dropout algorithm, and x ′ represents the first refined intermediate quantity, and y represents the second refined intermediate quantity.

6. A multi-scale aggregation cloud and cloud shadow identification system based on the method of claim 1, characterized in that: include: Image acquisition module: used to obtain the image to be tested; Cloud and cloud shadow recognition module: This module inputs the image to be tested into a pre-trained multi-scale attention feature aggregation network to obtain mask images of clouds and cloud shadows, thereby completing the recognition of clouds and cloud shadows.

7. A multi-scale aggregation cloud and cloud shadow identification device, characterized in that: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images