Foundation cloud picture multi-class segmentation method based on multi-scale feature extraction and screening

Abstract

The invention discloses a foundation cloud picture multi-class segmentation method based on multi-scale feature extraction and screening. Comprising the following steps: 1) extracting multi-level visual features by using four stacked residual blocks as a trunk structure; 2) a multi-scale feature extraction module processes a feature map output by the backbone network through a parallel pooling layer, reduces the number of channels by using a 1 * 1 convolution kernel, and realizes global feature extraction through bilinear interpolation up-sampling; 3) further screening important features from the multi-scale features by a feature screening module, and performing feature fusion by using multi-branch depth separable convolution and a mixed attention mechanism; and 4) the decoder performs up-sampling through transposition convolution and fuses deep and shallow layer features, and finally completes pixel-level classification through a softmax function. Compared with other algorithms, when a semantic segmentation task of the foundation cloud picture is processed, the segmentation accuracy is remarkably improved, particularly, the capability of subdividing different types of cloud classes is excellent, misclassification conditions are greatly reduced, and the advantages of the method in the aspect of global feature extraction are verified.

Description

Technical Field

[0001] The present invention belongs to the field of meteorological observation and artificial intelligence technology, and specifically relates to a ground-based cloud image multi-class segmentation method based on multi-scale feature extraction and screening. Background Art

[0002] Clouds are an important part of the water cycle and energy balance in the climate system. By observing the types, distribution, and changes of clouds, meteorologists can more accurately predict the weather, understand climate change, and issue early warnings for extreme weather events. The cloud information obtained through manual observations carries subjective errors and high labor costs. Therefore, researchers use ground-based equipment such as all-sky imagers to collect a large number of ground-based cloud images, and then use computer vision technology to automatically extract cloud information from the cloud images, thereby realizing automated observation of clouds.

[0003] The automated observation of ground-based clouds mainly focuses on two tasks: cloud classification and cloud segmentation. In the cloud classification task, the researchers aim to identify the largest cloud type in the image and give the corresponding label. The goal of cloud segmentation is to treat the clouds in the image as the foreground and the sky as the background, and perform pixel-level binary segmentation. However, when multiple types of clouds appear in a cloud image at the same time, most of the classification and segmentation methods proposed in previous studies have limitations: the labels given by cloud classification do not contain the location and area information of the clouds, while cloud segmentation can only distinguish between cloud pixels and sky pixels, ignoring the type of clouds. Therefore, a method is needed to segment ground-based cloud images by cloud type, so as to obtain the cloud amount of each type of cloud, making the automated observation results of ground-based cloud images more detailed. In fact, segmentation by cloud type is a semantic segmentation problem for ground-based cloud images, which integrates cloud classification and cloud segmentation tasks. Compared with binary segmentation, semantic segmentation of ground-based cloud images is a very challenging task because different cloud types are difficult to distinguish due to their high similarity.

[0004] At present, only a few research works are related to the semantic segmentation of ground-based cloud images. Dev et al. used clustering algorithms to perform semantic segmentation for thin clouds and thick clouds; Fabel et al. used semi-supervised learning to perform semantic segmentation for high clouds, medium clouds, and low clouds. These works only classified clouds into 2 or 3 categories, while current work related to cloud image classification generally divides clouds into 6 or 9 categories or even more. Therefore, a ground-based cloud image semantic segmentation method with more detailed cloud classification is needed. Ye et al. performed semantic segmentation on 8 types of clouds through superpixel feature extraction and class-by-class measurement learning methods. Although limited by the amount of data, the performance of the fully convolutional network was not good in the comparative experiment they proposed. However, with the emergence of better performing networks such as UNet in recent years, as well as the increasingly abundant cloud image data, there is reason to believe that convolutional neural networks are more suitable for handling the task of ground-based cloud image semantic segmentation. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-class segmentation method for ground-based cloud images based on global feature extraction and screening. Through the powerful feature extraction capability of convolutional neural networks, and the use of deep separable convolution to reduce network parameters, and the attention mechanism to help the network understand the dependencies between distant pixels, this method achieves accurate segmentation of 10 types of ground-based clouds. In order to achieve the above technical objectives, the present invention constructs a semantic segmentation network for ground-based cloud images based on multi-scale feature extraction and screening. Specifically, it includes the following steps:

[0006] Step 1: Main structure;

[0007] The structure uses 4 stacked residual blocks to extract multi-level visual features from the ground cloud map. The 2nd, 3rd, and 4th residual blocks all contain a 2×2 pooling layer to expand the receptive field, and apply the ReLU activation function to help the network learn complex patterns. After the input image passes through the backbone structure, the length and width of the final output feature map are reduced to 1 / 8 of the original.

[0008] Step 2: Multi-scale feature extraction module;

[0009] In order to enhance the global feature extraction capability of the network, the feature map F output by the backbone network H×W×C are input into x parallel pooling layers, where H, W, and C represent the length, width, and number of channels of the feature map, respectively. For the i-th pooling layer, the feature map is first pooled into a feature map of a smaller size. After that, a convolution kernel of size 1×1×(C / x) is used to reduce the feature map F i The number of channels is Then, bilinear interpolation is used to upsample the pooled feature map to the size of the original feature map, and we get All restored feature maps are concatenated with the original feature maps according to the channel dimension, and the output P of the final multi-scale feature extraction module is:

[0010]

[0011] in, Represents a concatenation operation.

[0012] Step 3: Feature screening module;

[0013] The feature filtering module is inserted after the multi-scale feature extraction module declared in step 2 to help the model select more important features from the multi-scale features. H×W×C First, it is input into x parallel convolution branches, each of which uses a depth-separable convolution kernel of different sizes to extract features and generate a feature map Then, the obtained x feature maps are added together to obtain the multi-scale fusion feature map UH×W×C Through global average pooling, the feature map on each channel is compressed into a real number, namely the feature vector S 1×1×C The formula for global average pooling is as follows:

[0014]

[0015] Among them U c (i, j) represents the pixel value of the cth channel of the feature map U at position (i, j), S c is the output value of the cth channel after global average pooling. 1×1×C After being input into x parallel 1-dimensional convolutions with kernel size k, x feature vectors with the same dimension as S are obtained. Using the sigmoid function, the normalized weights Feature map after separation input Multiply each channel to generate x weighted feature maps. Finally, add these feature maps to get the output feature map K of the feature filtering module. H×W×C .

[0016] Step 4: Decoder;

[0017] The decoder uses transposed convolution to upsample the output of the feature filtering module with a step size of 2. The length and width of the upsampled feature map are twice the length and width before upsampling. After that, the obtained feature map is concatenated with the output feature map of the third residual block in step 1 to fuse the deep features with the shallow features. Finally, the fused feature map is restored to the same length and width as the input cloud map through upsampling. The softmax function is used to determine the category of each pixel as a certain cloud or background.

[0018] The present invention designs a semantic segmentation model for ground-based cloud images based on multi-scale feature extraction and screening. The model uses parallel pooling layers to extract features of different scales, and uses the attention mechanism to dynamically select the scale of the convolution kernel, forcing the network to tilt computing resources to more important feature scales to achieve the effect of feature screening. Compared with other networks with similar structures, the present invention enhances the network's ability to extract global features, reduces the probability of classifying a certain cloud as another cloud, and thus improves the accuracy of semantic segmentation of ground-based cloud images. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the semantic segmentation network of the ground-based cloud image proposed in the present invention;

[0020] Figure 2 It is a structural schematic diagram of the feature screening module of the present invention; DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0022] The architecture of the ground-based cloud image semantic segmentation model based on multi-scale feature extraction and screening is as follows: Figure 1 The model specifically includes the following steps:

[0023] S1: main structure;

[0024] The main structure is as follows Figure 1 As shown in (a), it includes 4 stacked convolution blocks with residual structures, convolutional layers 1, 2 and 3, and 3 downsamplings to extract shallow and deep cloud map features. First, an RGB ground cloud map of size 256×256 is input. After the convolution head, a feature map with a length×width×number of channels of 256×256×64 is output. Then, after an average pooling of size 2, the length and width of the feature map are halved to a size of 128×128×64. After that, the number of channels of the feature map doubles each time it passes through a convolution block. After each pooling, the length and width are halved. Finally, the backbone network obtains a feature map F of size 32×32×512 32×32×512 .

[0025] S2: multi-scale feature extraction module;

[0026] In order to enhance the global feature extraction capability of the network, the feature map F output by the backbone network 32×32×512 are input into 4 parallel pooling layers. These pooling layers have different pooling sizes, namely 1×1, 2×2, 3×3, and 6×6, and each size corresponds to a level. At each level, the feature map is first average-pooled into a feature map of a smaller size. For example, at the third layer, that is, the layer with a pooling size of 3×3, the feature map will be pooled into After that, the feature map F is reduced using a convolution kernel of size 1×1×128 i The number of channels is Then, bilinear interpolation is used to upsample the pooled feature map to the size of the original feature map, and we get All restored feature maps are concatenated with the original feature maps according to the channel dimension, and the output P of the final multi-scale feature extraction module is:

[0027]

[0028] in, Represents a concatenation operation.

[0029] S3: feature screening module;

[0030] The feature screening module is used to achieve adaptive selection of features of different scales. Its structure is as follows: Figure 2As shown in Figure 2, the module mainly consists of three parts: multi-branch deep separation convolution, hybrid attention mechanism and feature selection fusion.

[0031] Multi-branch deep separation convolution structure such as Figure 2 (a) shows the output P of the multi-scale feature extraction module. 32×32×1024 First, they are input into four parallel convolution branches. These branches use convolution kernels of different sizes to capture spatial features of different scales. Unlike the traditional convolution used in S1, this module uses depthwise separable convolution to reduce the computational complexity and number of parameters of the model. Specifically, the convolution kernel in each convolution branch is divided into two parts: depthwise convolution and pointwise convolution. In depthwise convolution, each input channel is processed by a separate convolution kernel. 32×32×1024 Each branch has 1024 deep convolution kernels, corresponding to the number of channels, with sizes of 1×1, 2×2, 3×3, and 5×5. Each convolution kernel slides and performs convolution operations only on its corresponding single input channel. After the deep convolution, the point-by-point convolution uses a 1×1 convolution kernel to process the output of the deep convolution. The point-by-point convolution not only increases nonlinearity, but also effectively extracts features from each independently processed channel. For the input P 32×32×1024 , the multi-branch deep separation convolution structure will output 4 feature maps Where i represents the i-th branch.

[0032] The structure of the hybrid attention mechanism is as follows Figure 2 (b) First, the feature maps obtained by the four branches Add together to get the fusion feature map U 32×32×1024 After global average pooling of the feature map U, the feature vector S is obtained. 1×1×1024 After S is input into four parallel one-dimensional convolutions with a kernel size of 5, four feature vectors are obtained. Finally, the obtained feature vector is normalized using the sigmoid activation function to obtain The expression of Sigmoid is defined as follows:

[0033]

[0034] where e is the base of natural logarithms.

[0035] Feature selection fusion structure such as Figure 2 (c) shows the four normalized feature vectors. Feature map after separation input Multiply each channel to generate 4 weighted feature maps. Add these feature maps to get the final output feature map K of the feature filtering module. 32×32×1024 .

[0036] S4: decoder;

[0037] The structure of the decoder is as follows Figure 1 As shown in (b), it includes three upsamplings and three convolution blocks: convolution layer 4, convolution layer 5 and convolution layer 6. The first upsampling uses transposed convolution to convert the output K of the feature filtering module 32×32×1024 The length and width of the K2 are doubled, and the number of channels is reduced to 1 / 4, resulting in K 64×64×256 After that, the obtained feature map is concatenated with the output feature map of convolutional layer 2 in step 1 to obtain feature map C 64×64×512 This step fuses the deep features with the shallow features, enhancing the robustness of the network. Feature map C 64×64×512 After three convolution blocks and two upsamplings, we finally get a feature map of size 256×256×11. The channel vector z corresponding to each pixel is normalized by the softmax function to get a probability mask to determine which category the pixel belongs to. The expression of softmax is as follows:

[0038]

[0039] Among them, z i Represents the i-th element in vector z.

[0040] In the application example of the present invention, the ground-based cloud map used was taken in China at different times from 6 am to 6 pm. The collected images were corrected to eliminate the influence of optical distortion caused by the fisheye lens. Following the World Meteorological Organization's classification of clouds, ground-based clouds are divided into 10 categories. In addition, a cloudless category is included. According to this classification standard, each ground-based cloud map is marked with a pixel-level label. Each label is a mask image of the same size as the ground-based cloud map, and its pixel value represents the category of the corresponding position. According to the test set results, the pixel accuracy of the segmentation of the ground-based cloud map is 92.6%, which reflects the effectiveness of the present invention. Table 1 shows the details of the ground-based cloud map data set used in the present invention, and Table 2 shows the comparison of the experimental results of the present invention with other algorithms.

[0041] Table 1 Ground-based cloud image dataset

[0042]

[0043] Table 2 Comparison of experimental results of the present invention and other algorithms

[0044]

Claims

1. A multi-class segmentation method for ground-based cloud images based on multi-scale feature extraction and screening, characterized in that: The following steps are involved: S1: Use a set of stacked residual blocks as the backbone structure to extract multi-level visual features of the input ground-based cloud image; S2: The feature map output after the backbone structure is extracted is input into multiple parallel pooling layers. The feature map output by each pooling layer is processed by convolution kernel size adjustment and bilinear interpolation, and then concatenated with the original feature map according to the channel dimension. S3: The feature screening module selects more important features from the multi-scale features. The feature screening module uses multiple parallel depth-wise separable convolution kernels for feature extraction, and combines global average pooling and one-dimensional convolution to generate and normalize feature vectors. S4: Use the decoder module to upsample features through transposed convolution, concatenate the upsampled feature map with the low-level feature map, and finally output the cloud category of each pixel through the softmax function.

2. The method for multi-class segmentation of ground-based cloud images based on multi-scale feature extraction and screening according to claim 1, characterized in that: A backbone structure is designed to extract visual features at multiple levels; the backbone structure includes 4 superimposed convolution blocks with residual structures, convolution layers 1, 2 and 3, and 3 downsamplings to extract shallow and deep cloud features; first, an RGB ground cloud image of size 256×256 is input; after the convolution head, a feature map of length×width×number of channels of 256×256×64 is output; then after average pooling of size 2, the length and width of the feature map are halved to a size of 128×128×64; thereafter, the number of channels of the feature map is doubled each time it passes through a convolution block; each time it passes through a pooling, the length and width are halved; finally, the backbone network obtains a feature map F of size 32×32×512 32×32×512 .

3. The method for multi-class segmentation of ground-based cloud images based on multi-scale feature extraction and screening according to claim 1, characterized in that: Use a multi-scale feature extraction module to enhance the global feature extraction capability of the network; the feature map F output by the backbone network 32×32×512 are input into 4 parallel pooling layers respectively; the pooling layers have different pooling sizes, namely 1×1, 2×2, 3×3 and 6×6, each size corresponds to a level; at each level, the feature map is first averagely pooled into a feature map of a smaller size; at the third layer, that is, the layer with a pooling size of 3×3, the feature map will be pooled into After that, the feature map F is reduced using a convolution kernel of size 1×1×128 i The number of channels is Use bilinear interpolation to upsample the pooled feature map to the size of the original feature map, and get All restored feature maps are concatenated with the original feature maps according to the channel dimension, and the output P of the final multi-scale feature extraction module is:

4. The method for multi-class segmentation of ground-based cloud images based on multi-scale feature extraction and screening according to claim 1, characterized in that: The feature screening module includes a depth-separable convolution module and an adaptive attention mechanism module; the output P of the multi-scale feature extraction module 32×32×1024 First, they are input into four parallel convolution branches respectively; the convolution branches use convolution kernels of different sizes to capture spatial features of different scales; feature The filtering module uses depthwise separable convolution to reduce the computational complexity and number of parameters of the model. Specifically, the convolution kernel in each convolution branch is divided into two parts: depthwise convolution and pointwise convolution. In depthwise convolution, each input channel is processed by a separate convolution kernel. 32×32×1024 Each branch has 1024 deep convolution kernels, corresponding to the number of channels, with sizes of 1×1, 2×2, 3×3, and 5×5 respectively; each convolution kernel slides and performs convolution operations only on its corresponding single input channel; after the deep convolution, the point-by-point convolution uses a 1×1 convolution kernel to process the output of the deep convolution; the point-by-point convolution adds nonlinearity to extract features from each independently processed channel; for the input P 32×32×1024 , the multi-branch deep separation convolution structure will output 4 feature maps Where i represents the i-th branch; the feature maps obtained by the four branches Add together to get the fusion feature map U 32×32×1024 ; After global average pooling of the feature map U, the feature vector S is obtained 1×1×1024 ; After S is input into 4 parallel 1D convolutions with a kernel size of 5, 4 feature vectors are obtained Finally, the obtained feature vector is normalized using the sigmoid activation function to obtain The four normalized feature vectors Feature map after separation input Multiply each channel to generate 4 weighted feature maps, and add the feature maps to get the final output feature map K of the feature filtering module 32×32×1024 .

5. The method for multi-class segmentation of ground-based cloud images based on multi-scale feature extraction and screening according to claim 1, characterized in that: Design a decoding method that integrates features at different levels; use transposed convolution for the first upsampling to transform the output K of the feature filtering module 32×32×1024 The length and width of the K2 are doubled, and the number of channels is reduced to 1 / 4, resulting in K 64×64×256 ; After that, the obtained feature map is concatenated with the output feature map of convolutional layer 2 to obtain feature map C 64×64×512 ; The deep features are integrated with the shallow features to enhance the robustness of the network; Feature map C 64×64×512 After three convolution blocks and two upsamplings, a feature map of size 256×256×11 is finally obtained. The channel vector z corresponding to each pixel is normalized by the softmax function to obtain a probability mask to determine which category the pixel belongs to.

Images