Ground-based cloud picture segmentation method and system based on improved U-Net model
By improving the architecture of the U-Net model, using hollow convolution and multi-head self-attention mechanisms, combined with bicubital interpolation, the problems of loss of detailed information and low segmentation accuracy in cloud image segmentation are solved, and the accuracy and clarity of cloud image segmentation are improved .
Patent Information
- Application Number
- CN202411039405.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-22
AI Technical Summary
The existing cloud map segmentation algorithm has the problems of loss of detailed information and low segmentation accuracy, especially the cloud map segmentation method based on the U-Net model is not robust enough when extracting cloud map features, making it difficult to accurately obtain cloud information.
By improving the architecture of the U-Net model, hollow convolution and hollow space convolution pooling pyramids are used to build hollow blocks, combining deep separable convolution and multi-head self-attention mechanisms, introducing bicubital interpolation method to enhance feature extraction and segmentation accuracy .
It improves the accuracy of cloud image segmentation, reduces cloud information loss, captures a larger range of context information, enhances feature extraction capabilities, and improves the clarity and resolution of segmented images.
Smart Images

Figure CN119006481A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection, and in particular to a ground-based cloud image segmentation method and system based on an improved U-Net model. Background Art
[0002] Clouds are common and important natural phenomena in daily life. They not only reflect the current atmospheric movement and water vapor changes, but also predict the trend of future weather changes. Cloud movement affects the earth's thermal balance, radiation balance and water cycle, and ultimately affects global climate change. Therefore, analyzing and observing cloud map changes and accurately obtaining cloud information have positive significance for the fields of meteorology and astronomy. Cloud map segmentation is an important step in cloud map observation. The current cloud map segmentation algorithms mainly include traditional segmentation algorithms and cloud map segmentation algorithms based on deep learning. The traditional methods mainly include: threshold segmentation through the relative difference of the red and blue channels in the RGB cloud map; and segmentation of cloud and non-cloud areas through clustering algorithms and superpixel algorithms. The above traditional algorithms are often greatly disturbed by factors such as light intensity and parameter selection, and have poor robustness. The quality of cloud map segmentation needs to be further improved. Another important method of cloud map segmentation is the algorithm based on deep learning, which mainly uses networks based on U-Net and SegNet architectures, but there are problems such as loss of detail information and low segmentation accuracy. Therefore, it is necessary to improve the deep learning model architecture, strengthen cloud map feature extraction, and improve cloud map segmentation accuracy.
[0003] The above problems need to be solved urgently. Summary of the invention
[0004] The purpose of the present invention is to provide a ground-based cloud image segmentation method and system diagram based on an improved U-Net model, so as to improve the accuracy of cloud image segmentation.
[0005] In order to overcome at least one of the above-mentioned shortcomings of the prior art, on the one hand, the present invention provides a ground-based cloud image segmentation method based on an improved U-Net model, the method comprising: building a traditional U-Net model with an encoder, a skip connection layer and a decoder as the overall architecture; replacing the traditional convolution in the traditional U-Net model with a dilated convolution, and embedding a dilated spatial convolution pooling pyramid to construct a dilated block Dilate-Block, thereby improving the encoder of the traditional U-Net model; adding a boundary refinement module DS-Path with a residual structure in the skip connection layer, and replacing the convolution layer of the traditional residual block with a depth-separable convolution ; The Im-CSAM module is constructed by combining deep separable convolution and multi-head self-attention to transform the CBAM attention mechanism architecture and connect it to the boundary refinement module DS-Path with a residual structure; the bicubic interpolation method is introduced to perform upsampling in the decoder; an improved U-Net cloud image segmentation model is constructed based on the Dilate-Block, the boundary refinement module DS-Path with a residual structure, the Im-CSAM module and the improvement of the decoder upsampling; the corresponding cloud image segmentation image is obtained by inputting the cloud image to be detected into the improved U-Net cloud image segmentation model.
[0006] Furthermore, the improved encoder contains four layers of dilated blocks, namely Dilate-Block1, Dilate-Block2, Dilate-Block3 and Dilate-Block4, and four pooling downsampling processes.
[0007] Furthermore, the atrous block includes a first layer of atrous convolution, a atrous spatial convolution pooling pyramid, and a second layer of atrous convolution, which are connected in sequence; the first layer of atrous convolution is used to perform convolution calculation on the input cloud image to be detected to extract early low-level features; the atrous spatial convolution pooling pyramid is used to enhance the low-level features extracted by the first layer of atrous convolution; the second layer of atrous convolution is used to perform convolution operation on the enhanced low-level features output by the atrous spatial convolution pooling pyramid to extract higher-level features.
[0008] Furthermore, the atrous spatial convolution pooling pyramid includes a 1x1 convolution layer, a pooling pyramid and an adaptive pooling layer; the 1x1 convolution layer is used to perform dimensionality reduction operations on early low-level features of the input; the pooling pyramid includes three multi-scale atrous convolutions with different expansion factors, and obtains receptive fields at different scales through different filling and expansion, extracts features of different scales, and adds a layer of 1x1 convolution after the three layers of atrous convolution respectively to adjust the feature channel; a layer of 1x1 convolution is added after the adaptive pooling layer to adjust the feature channel; the 1x1 convolution layer, the pooling pyramid and the adaptive pooling layer are calculated in parallel, and the outputs are fused and superimposed through the Concat layer, and the 1x1 convolution layer is used again after the Concat layer to adjust the feature channel to form a new feature.
[0009] Furthermore, the method includes: using a pooling layer to perform a downsampling operation on higher-level features extracted from the dilate-block to reduce the size of the feature map, and outputting it to the next dilate block.
[0010] Furthermore, the boundary refinement module DS-Path with a residual structure is composed of three depth-separable convolutions, and the convolution kernel size is fixed to 3x3, which is used to receive the output of the corresponding dilate block Dilate-Block.
[0011] Further, the Im-CSAM module is constructed by combining the deep separable convolution and multi-head self-attention to transform the CBAM attention mechanism architecture, so that it is connected to the boundary refinement module DS-Path with a residual structure, including: replacing the MLP in the CAM module with the deep separable convolution to obtain an optimized CAM module; replacing the original global average pooling and global maximum pooling in the SAM module by adopting multi-head self-attention to obtain an optimized SAM module; obtaining the Im-CSAM module based on the optimized CAM module and the SAM module; obtaining the feature map X based on the boundary refinement module DS-Path with a residual structure 2 The processing flow in the Im-CSAM module includes: obtaining a feature map X based on the boundary refinement module DS-Path with a residual structure 2 After downsampling through the maximum pooling layer and the average pooling layer, they enter the deep separable module with a two-layer structure. The output is summed based on the element-wise operation, and the channel attention feature is generated through the Sigmod activation function. 2 Perform element-wise multiplication operations to generate the input features required by the SAM module; the optimized channel attention calculation includes:
[0012]
[0013] Where W d1 and W d2 is the weight of the depthwise convolution, W p1 and W p2 is the weight of the point-wise convolution, and Represent the results of global average pooling and global maximum pooling respectively;
[0014] Multi-head self-attention is used to replace the original global average pooling and global maximum pooling operations, and the features from the CAM module are accepted as input X 2c ; Among them, the optimized spatial attention calculation includes:
[0015] M s (X 2c )=σ(MultiHeadAttention(X 2c ));
[0016] In the formula, MultiHeadAttention represents the multi-head self-attention mechanism;
[0017] Calculate the attention score and weighted sum value for each head independently, concatenate the outputs of N heads together, and perform linear transformation through the forward propagation network to obtain the final output feature map X 3 .
[0018] Furthermore, the improved U-Net model also includes a bottleneck layer; the bottleneck layer is located at the bottom layer of the network, and is used to receive the output from the encoder, filter and compress information through the operation of two convolutional layers, so that important features can be retained and passed to the decoder part.
[0019] Furthermore, each dilate-block of the encoder is directly connected to the corresponding up-block convolution block of the decoder in a skip connection manner.
[0020] In a second aspect, the present invention provides a ground-based cloud map segmentation system based on an improved U-Net model, the system comprising: building a U-Net traditional model unit, suitable for building a U-Net traditional model with an encoder, a jump connection layer and a decoder as an overall architecture; an improved encoder unit, suitable for replacing traditional convolutions in the U-Net traditional model with dilated convolutions, and embedding a dilated spatial convolution pooling pyramid to construct a dilated block Dilate-Block, thereby improving the encoder of the U-Net traditional model; an improved jump connection layer unit, suitable for adding a boundary refinement module DS-Path with a residual structure in the jump connection layer, replacing the convolution layer of the traditional residual block with a depth-separable convolution; by combining the depth-separable convolution The Im-CSAM module is constructed by transforming the CBAM attention mechanism architecture with multi-head self-attention, so that it is connected to the boundary refinement module DS-Path with a residual structure; the decoder unit is improved, which is suitable for introducing the bicubic interpolation method to perform upsampling in the decoder; an improved U-Net cloud image segmentation model unit is constructed, which is suitable for constructing an improved U-Net cloud image segmentation model based on the void block Dilate-Block, the boundary refinement module DS-Path with a residual structure, the Im-CSAM module and the improvement of the decoder upsampling; a cloud image segmentation image generation unit is suitable for obtaining a corresponding cloud image segmentation image by inputting the cloud image to be detected into the improved U-Net cloud image segmentation model.
[0021] On the other hand, the present invention also provides a computer-readable storage medium, wherein one or more instructions are stored in the computer-readable storage medium, and the computer instructions are used to enable the computer to execute the above-mentioned ground-based cloud image segmentation method based on the improved U-Net model.
[0022] On the other hand, the present invention provides an electronic device, comprising: a memory and a processor; the memory stores at least one program instruction; the processor implements the above-mentioned ground-based cloud image segmentation method based on the improved U-Net model by loading and executing the at least one program instruction.
[0023] The beneficial effects of the present invention are as follows: the present invention provides a ground-based cloud image segmentation method based on an improved U-Net model, which has the following beneficial effects:
[0024] 1. By constructing a dilate-block in the encoder, the fine structure of the cloud image is retained and the loss of cloud layer information is reduced, while capturing a wider range of contextual information and completing the shallow cloud image feature extraction.
[0025] 2. By introducing deep separable convolution, the input features of the encoder and the high-level features of the decoder are effectively combined, the semantic gap that may exist between the encoder features and the decoder features at the same level is improved, and the feature combination of shallow information and deep contextual information is strengthened.
[0026] 3. By improving the CBAM structure, enhancing and combining the feature extraction capabilities of CBAM in both spatial and channel dimensions, focusing on important information and key features in the cloud map, capturing more spatial relationships and dependencies in the feature map, and finding the key areas in the key features.
[0027] 4. By adopting the bicubic interpolation upsampling method in the decoder, the image resolution is restored and the clarity of the segmented image is further improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0029] Figure 1 This is a flow chart of a ground-based cloud image segmentation method based on an improved U-Net model provided in Example 1 of the present invention.
[0030] Figure 2 This is a schematic diagram of an improved U-Net model structure provided in Example 1 of the present invention.
[0031] Figure 3 This is a schematic diagram of a hollow block structure provided in Example 1 of the present invention.
[0032] Figure 4 It is a schematic diagram of the structure of a DS-Path module provided in Example 1 of the present invention and a schematic diagram of the connection relationship with the Im-CSAM module.
[0033] Figure 5 This is a schematic diagram of the structure of an Im-CSAM module provided in Example 1 of the present invention.
[0034] Figure 6a It is the input RGB cloud image provided by Example 1 of the present invention.
[0035] Figure 6b It is a schematic diagram of the segmentation results of segmentation using the SegNet model provided in Example 1 of the present invention.
[0036] Figure 6c It is a schematic diagram of the segmentation results of segmentation using the traditional U-Net model provided in Example 1 of the present invention.
[0037] Figure 6d It is a schematic diagram of the segmentation result of segmentation using the FCN model provided in Example 1 of the present invention.
[0038] Figure 6e It is a schematic diagram of the segmentation results of segmentation using the improved U-Net model provided in Example 1 of the present invention.
[0039] Figure 7 This is a schematic diagram of the structure of a ground-based cloud image segmentation system based on an improved U-Net model provided in Example 2 of the present invention.
[0040] Figure 8 This is a partial block diagram of an electronic device provided in Example 4 of the present invention. DETAILED DESCRIPTION
[0041] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.
[0042] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.
[0043] The present invention will now be described in detail with reference to the accompanying drawings. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.
[0044] For ease of understanding, the overall inventive concept of the present invention is explained here: The present invention proposes a ground-based cloud image segmentation method based on an improved U-Net model, which adopts an encoder-skip connection layer-decoder structure as a whole. First, the dilated convolution and dilated spatial convolution pooling pyramid are combined to enhance early feature extraction; second, a boundary refinement module with a residual structure is introduced in the skip connection layer to promote information exchange between high-level and low-level features; third, an improved CBAM mechanism is added to focus on important cloud features in the spatial and channel dimensions to improve segmentation accuracy; finally, bicubic interpolation is used for upsampling to restore the feature map resolution and improve the clarity of the segmented image.
[0045] The specific embodiments are as follows:
[0046] Example 1
[0047] like Figure 1-2 As shown, a flow chart of a ground-based cloud image segmentation method based on an improved U-Net model and a structural schematic diagram of the improved U-Net model provided by the present invention.
[0048] As an example, the method includes:
[0049] S110: Build a U-Net traditional model with encoder, skip connection layer and decoder as the overall architecture.
[0050] Preferably, the encoder part includes four layers, usually composed of traditional convolution and pooling layers. Each convolution layer usually includes two 3x3 convolution operations, followed by a ReLU activation function. The encoder accepts a two-dimensional image as input, learns the corresponding shallow features after convolution operation, and then performs downsampling operation. The downsampling process mainly adopts the method of convolution and maximum pooling. After each feature extraction, the model uses maximum pooling to reduce the size of the feature map, which reduces the number of parameters of the model and relieves the model operation pressure. In the design, each time the maximum pooling is used, the resolution of the feature map will be reduced by 1 / 2. After four maximum poolings, the resolution of the output feature map is 1 / 16 of the input original image. The decoder also includes four layers, usually composed of a series of upsampling layers and traditional convolution layers, which mainly restores the output compressed feature map to the original feature map size, and adopts the method of feature fusion and deconvolution. Each upsampling layer expands the spatial dimension of the feature map through deconvolution. After upsampling, it is followed by two 3x3 convolution operations, and there is a ReLU activation function after each convolution. After four deconvolution operations, the resolution and size of the feature map are restored to the original image. The last layer of U-Net is a convolution layer with a convolution kernel size of 1x1, which converts the feature map containing 64 channels into classification results. The jump connection layer establishes a direct connection between the encoder and the decoder. Specifically, each layer of feature maps in the encoder is directly spliced with the corresponding feature map after the upsampling layer in the decoder. The jump connection layer in the traditional U-Net model does not embed any modules.
[0051] S120: Use dilated convolution to replace the traditional convolution in the traditional U-Net model, and embed dilated spatial convolution pooling pyramid to construct dilate-Block to improve the encoder of the traditional U-Net model.
[0052] Preferably, combined Figure 3As shown, the improved encoder includes four layers of dilated blocks, namely Dilate-Block1, Dilate-Block2, Dilate-Block3 and Dilate-Block4, and four pooling downsampling processes. The dilated block includes the first layer of dilated convolution, dilated spatial convolution pooling pyramid and the second layer of dilated convolution connected in sequence; the first layer of dilated convolution is used to perform convolution calculation on the input cloud image to be detected to extract early low-level features; the dilated spatial convolution pooling pyramid is used to enhance the low-level features extracted by the first layer of dilated convolution; the second layer of dilated convolution is used to perform convolution operation on the enhanced low-level features output by the dilated spatial convolution pooling pyramid to extract higher-level features.
[0053] Preferably, the atrous block includes a first layer of atrous convolution, a atrous spatial convolution pooling pyramid, and a second layer of atrous convolution connected in sequence; the first layer of atrous convolution is used to perform convolution calculation on the input cloud image to be detected to extract early low-level features; the atrous spatial convolution pooling pyramid is used to enhance the low-level features extracted by the first layer of atrous convolution; the second layer of atrous convolution is used to perform convolution operation on the enhanced low-level features output by the atrous spatial convolution pooling pyramid to extract higher-level features.
[0054] Specifically, the processing flow of the input cloud image to be detected in the spatial downsampling module includes: the input feature shape is a three-dimensional tensor, which first passes through the first layer of dilated convolution Dilated Conv1 with a size of 3x3, and extracts early low-level features through convolution calculation. The second layer of the dilated downsampling block is the dilated spatial convolution pooling pyramid (ASPP), which accepts the output features from Dilated Conv1. ASPP mainly consists of the following parts: (1) 1x1 convolution layer. The input features enter the 1x1 convolution layer to perform dimensionality reduction operations; (2) pooling pyramid. The main components of the pooling pyramid are three multi-scale dilated convolutions with different expansion factors, the expansion rates are 1, 3, and 5 respectively, and the convolution kernel size is fixed to 3x3. Through different padding and expansion, the receptive fields at different scales are obtained, and the features of different scales are further extracted. After the three layers of dilated convolution, a layer of 1x1 convolution is added to adjust the feature channel; (3) adaptive pooling layer, followed by a layer of 1x1 convolution to adjust the feature channel. The 1x1 convolution layer, the pooling pyramid, and the adaptive pooling layer are calculated in parallel, and their outputs are fused and superimposed through the Concat layer. Finally, the 1x1 convolution layer is used again to adjust the feature channels to form new features, and the number of channels is 512. The last layer of the hole block is the dilated convolution Dilated Conv2, and the convolution kernel is 3x3. While enhancing the nonlinear expression ability and extracting higher-level features, it smoothes the feature map and reduces noise, so that subsequent layers can better process these features. The pooling layer is used to perform downsampling operations, reduce the size of the feature map, and output it to the next hole downsampling block. This embodiment constructs a dilate-block to capture a wider range of contextual information while retaining the fine structure of the cloud map and reducing the loss of cloud information, thereby completing the shallow cloud map feature extraction.
[0055] S130: Add a boundary refinement module DS-Path with a residual structure to the jump connection layer, and replace the convolution layer of the traditional residual block with a depth-wise separable convolution.
[0056] Preferably, the DS-Path module (Depthwise Separable Path, DS-Path) consists of three depthwise separable convolutions, and the convolution kernel size is fixed to 3x3, which is used to receive the output of the corresponding dilate-block. Specifically, the skip connection mechanism combines high-level semantic features from the decoder and low-level detailed features from the encoder, aiming to bridge the semantic gap between the encoder and the decoder. Each dilate-block of the encoder is directly connected to the corresponding up-block of the decoder in a skip connection manner. A residual structure is introduced in the skip connection layer to strengthen the feature combination of shallow information and deep contextual information, and a depthwise separable convolution module is used in the residual structure to replace the traditional standard convolution module to construct a depthwise separable residual block DS-Path for feature extraction. Reference Figure 4 , the DS-Path module consists of three depth-separable convolutions, and the convolution kernel size is fixed to 3x3, which accept the output of the corresponding dilate-block. Formally, the input feature of each layer of DS-Path module is X, which is directly transmitted to the output through shortcut. Assuming that the stacked nonlinear change layer, that is, the two layers of depth-separable residual blocks are F(X), the output can be expressed as F(X)+X, and finally the feature map X is obtained through the ReLu layer. 2 , as the input of the next module. This embodiment effectively combines the input features of the encoder with the high-level features of the decoder by introducing a deep separable residual block, improves the semantic gap that may exist between the encoder features and the decoder features at the same level, and strengthens the feature combination of shallow information and deep context information.
[0057] S140: The Im-CSAM module is constructed by combining deep separable convolution and multi-head self-attention to transform the CBAM attention mechanism architecture and connect it to the boundary refinement module DS-Path with the residual structure.
[0058] Preferably, combined Figure 4 and Figure 5The improved channel spatial attention module (Im-CSAM) is constructed by combining deep separable convolution and multi-head self-attention to transform the CBAM attention mechanism architecture, so that it is connected to the boundary refinement module DS-Path with a residual structure, including: replacing the MLP (Multilayer Perceptron: shared multi-layer perceptron) in the CAM module with deep separable convolution to obtain an optimized CAM module; replacing the original global average pooling and global maximum pooling in the SAM module with multi-head self-attention to obtain an optimized SAM module; and obtaining an Im-CSAM module based on the optimized CAM module and SAM module.
[0059] Specifically, the feature map X obtained by the DS-Path module 2 The processing flow in the Im-CSAM module includes:
[0060] The feature map X obtained by the DS-Path module 2 After downsampling through the maximum pooling layer and the average pooling layer, they enter the deep separable module with a two-layer structure. The output is summed based on the element-wise operation, and the channel attention feature is generated through the Sigmod activation function. 2 Perform element-wise multiplication operations to generate the input features required by the SAM module; the optimized channel attention calculation includes:
[0061]
[0062] Where W d1 and W d2 is the weight of the depthwise convolution, W p1 and W p2 is the weight of the point-wise convolution, and Represent the results of global average pooling and global maximum pooling respectively.
[0063] Multi-head self-attention is used to replace the original global average pooling and global maximum pooling operations, and the features from the CAM module are accepted as input X 2c ; Among them, the optimized spatial attention calculation includes:
[0064] M s (X 2c )=σ(MultiHeadAttention(X 2c ));
[0065] In the formula, MultiHeadAttention represents the multi-head self-attention mechanism;
[0066] Calculate the attention score and weighted sum value for each head independently, concatenate the outputs of N heads together, and perform linear transformation through the forward propagation network to obtain the final output feature map X 3 Among them, the multi-head self-attention mechanism consists of N single self-attention components, and the feature vector is mapped to the key K n (key vector), query Q n (query vector), value V n (value vector), where n represents the number of self-attention heads. n and Q n After the new matrix is generated by Matmul (denoted as M), the scaled dot product operation (denoted as S) is applied to V n Perform the M operation again to obtain the attention score matrix, which is calculated as follows:
[0067]
[0068] Here, d represents dimension, Q, K, and V represent query, key, and value respectively.
[0069] More specifically, the improvement process is as follows:
[0070] Suppose given a tensor As input, CBAM will sequentially derive 1D channel attention feature maps and 2D spatial attention feature map The whole attention process can be summarized as:
[0071]
[0072] Therefore, the final output feature map can be obtained
[0073] Regarding CAM (Channel Attention Module): CAM is mainly dedicated to discovering key features in images. In general, the traditional approach of CAM is: for the input feature F, firstly, the spatial information of the feature map is aggregated by using average pooling and maximum pooling operations to generate different spatial context descriptors: and They are then fed into a shared multi-layer perceptron (MLP) to generate output features, which are then added pixel by pixel, and the combined sum is finally sent to a sigmoid function to obtain the channel weighted attention feature map M. c , in short, the original channel attention is calculated as follows:
[0074]
[0075] like Figure 5 As shown in the improved CAM structure, MLP is replaced by depth-separable convolution, and the feature map X 2 The output is downsampled through the maximum pooling layer and the average pooling layer, and then enters the deep separable module with a two-layer structure. The output is element-wise summed, and the channel attention feature is generated through the Sigmod activation function. Finally, it is combined with the input feature X 2 Perform element-wise multiplication to generate the input features required by the SAM module. Therefore, the improved channel attention calculation is as follows:
[0076] M c (X 2 )=σ(DSConv(AvgPool(X 2 ))+DSConv(MaxPool(X 2 )));
[0077] Among them, DSConv stands for depth-wise separable convolution operation;
[0078] Since the depth-wise separable convolution contains depth-wise convolution and point-wise convolution, it can be specifically expanded as follows:
[0079]
[0080] Regarding SAM (Spatial Attention Module): SAM is different from CAM but complements CAM. It focuses more on finding the key areas of key features. In general, the traditional approach of SAM is: first perform channel-based global maximum pooling and global average pooling operations to obtain two H×W×1 feature maps and Then let the two be concatenated based on the channel, and then pass a 7×7 convolution to generate a 2D spatial attention map. In short, the spatial attention is calculated as:
[0081]
[0082] like Figure 5 As shown in the improved SAM structure, multi-head self-attention is used to replace the original global average pooling and global maximum pooling operations, and the features from CAM are accepted as input X 2c , the improved spatial attention is calculated as follows:
[0083] M s (X 2c)=σ(MultiHeadAttention(X 2c ));
[0084] Among them, MultiHeadAttention represents the multi-head self-attention mechanism.
[0085] This embodiment improves the CBAM structure, enhances and combines the feature extraction capabilities of CBAM in both spatial and channel dimensions, focuses on important information and key features in the cloud map, captures more spatial relationships and dependencies in the feature map, and finds out the key areas in the key features.
[0086] S150: Introduce a bicubic interpolation method to perform upsampling in the decoder.
[0087] Preferably, the decoding module consists of four Up-Blocks and a 1x1 convolution. Each Up-Block module contains two convolutional layers of size 3x3, and adds batch normalization and activation layer functions. The high-level cloud features are alternately passed through the upsampling layer and the Up-Block convolution block, and finally a cloud mask of size 128x128 is obtained, which has the same resolution as the input image.
[0088] The upsampling method is bicubic interpolation. Bicubic interpolation is an interpolation method based on cubic polynomials. Assume that the value P of the unknown function f at point (x, y) is obtained by weighted averaging of 4x4 sampling points in a rectangular grid, so the interpolation function is:
[0089]
[0090] Among them, i and j represent the horizontal and vertical positions of the image respectively, and a ij = represents the weighting coefficient; since the bicubic interpolation method is already very mature in the prior art, this application does not improve it, so it will not be described in detail here. This embodiment uses the bicubic interpolation upsampling method to restore the image resolution and further improve the clarity of the segmented image.
[0091] Preferably, in order to verify the segmentation results of the improved U-Net model on the ground-based cloud image segmentation, the proposed improved U-Net is compared with several other deep learning-based segmentation methods. The comparison networks include U-Net, SegNet and FCN, and their segmentation results are as follows: Figure 6b-6e shown.
[0092] like Figure 6a The test cloud image is shown as follows: Figure 6b-6eAs shown in the figure, SegNet has the worst segmentation effect. It can simply identify the approximate area and distribution of clouds and sky, but cannot accurately describe and segment the edge details of clouds. The segmentation effect of traditional U-Net and FCN is better, and the segmentation effect has been significantly improved compared with SegNet. It can obtain more shape contour information of clouds, and the edge contour information of clouds has been further optimized. However, for the clouds closest to the edge of the cloud map, the segmentation effect of traditional U-Net is poor, and it cannot identify the sky and clouds. FCN also finds it difficult to achieve more detailed edge detail description.
[0093] Taking recall rate, precision rate and MIoU as the evaluation indicators of segmentation effect, Table 1 shows the comparison results of the evaluation indicators of the improved U-Net and the other three comparison networks. The improved U-Net has achieved the best performance in all evaluation indicators and has the highest recall rate, precision rate and MIoU. Therefore, it can be seen that the proposed improved U-Net has the best segmentation performance in the segmentation of ground-based cloud images.
[0094] Table 1:
[0095]
[0096] From the comprehensive segmentation results, the segmentation effect of the improved U-Net is better than that of the other three segmentation models, especially the edge part of the ground-based cloud image, which can maximize the segmentation of clouds with similar features to the sky background.
[0097] Example 2
[0098] See also Figure 7 , this embodiment provides a schematic diagram of the structure of a ground-based cloud image segmentation system based on an improved U-Net model.
[0099] As an example, the system includes:
[0100] The U-Net traditional model building unit 710 is suitable for building a U-Net traditional model with an encoder, a skip connection layer and a decoder as an overall architecture.
[0101] The improved encoder unit 720 is suitable for replacing the traditional convolution in the U-Net traditional model with the dilated convolution, and embedding the dilated spatial convolution pooling pyramid to construct the dilate-Block, so as to improve the encoder of the U-Net traditional model.
[0102] The improved skip connection layer unit 730 is suitable for adding a boundary refinement module DS-Path with a residual structure in the skip connection layer, and replacing the convolution layer of the traditional residual block with a depthwise separable convolution; the Im-CSAM module is constructed by combining the depthwise separable convolution and multi-head self-attention to transform the CBAM attention mechanism architecture, so that it is connected to the boundary refinement module DS-Path with a residual structure.
[0103] The decoder unit 740 is improved and adapted to introduce bicubic interpolation for upsampling in the decoder.
[0104] The improved U-Net cloud image segmentation model construction unit 750 is suitable for constructing an improved U-Net cloud image segmentation model based on the void block Dilate-Block, the boundary refinement module DS-Path with a residual structure, the Im-CSAM module, and the improvement of the decoder upsampling.
[0105] The cloud image segmentation image generating unit 760 is adapted to obtain a corresponding cloud image segmentation image by inputting the cloud image to be detected into the improved U-Net cloud image segmentation model.
[0106] It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.
[0107] It is worth mentioning that all units involved in this embodiment are logical units. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.
[0108] Example 3
[0109] The embodiment of the present invention further proposes a storage medium, on which a ground-based cloud image segmentation method based on an improved U-Net model is stored, and when the ground-based cloud image segmentation program based on the improved U-Net model is executed by a processor, the steps of the ground-based cloud image segmentation method based on the improved U-Net model as described above are implemented. Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be repeated here one by one.
[0110] Example 4
[0111] See also Figure 8 An embodiment of the present invention further provides an electronic device, comprising: a memory and a processor; the memory stores at least one program instruction; the processor loads and executes the at least one program instruction to implement the ground-based cloud image segmentation method based on the improved U-Net model provided in Example 1.
[0112] The memory 602 and the processor 601 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 601 and the memory 602 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 601 is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor 601.
[0113] The processor 601 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 602 can be used to store data used by the processor 601 when performing operations.
[0114] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can know all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.
Claims
1. A ground-based cloud image segmentation method based on an improved U-Net model, characterized in that: The method comprises: The U-Net traditional model is built with encoder, skip connection layer and decoder as the overall architecture; The traditional convolution in the traditional U-Net model is replaced by dilated convolution, and the dilated spatial convolution pooling pyramid is embedded to construct the dilate-Block, which improves the encoder of the traditional U-Net model. A boundary refinement module DS-Path with a residual structure is added to the jump connection layer, and the convolution layer of the traditional residual block is replaced by a depth-wise separable convolution. The Im-CSAM module is constructed by combining deep separable convolution and multi-head self-attention to transform the CBAM attention mechanism architecture, and connected to the boundary refinement module with residual structure; Introducing bicubic interpolation for upsampling in the decoder; An improved U-Net cloud image segmentation model is constructed based on the dilate-Block, the boundary refinement module DS-Path with a residual structure, the Im-CSAM module, and the improvement of the decoder upsampling; The cloud image to be detected is input into the improved U-Net cloud image segmentation model to obtain the corresponding cloud image segmentation image.
2. The ground-based cloud image segmentation method based on the improved U-Net model according to claim 1, characterized in that: The improved encoder contains four layers of hole blocks, namely Dilate-Block1, Dilate-Block2, Dilate-Block3 and Dilate-Block4, and four pooling downsampling processes.
3. The ground-based cloud image segmentation method based on the improved U-Net model according to claim 1, characterized in that: The hole block includes a first layer of hole convolution, a hole spatial convolution pooling pyramid and a second layer of hole convolution connected in sequence; The first layer of hole convolution is used to perform convolution calculation on the input cloud image to be detected to extract early low-level features; The atrous spatial convolution pooling pyramid is used to enhance the low-level features extracted by the first layer of atrous convolution; The second layer of atrous convolution is used to perform a convolution operation on the enhanced low-level features output by the atrous spatial convolution pooling pyramid to extract higher-level features.
4. The ground-based cloud image segmentation method based on the improved U-Net model according to claim 3 is characterized in that: The atrous spatial convolutional pooling pyramid includes a 1x1 convolutional layer, a pooling pyramid and an adaptive pooling layer; The 1x1 convolutional layer is used to perform dimensionality reduction on the early low-level features of the input; The pooling pyramid includes three multi-scale dilated convolutions with different dilation factors. Through different padding and dilation, the receptive fields at different scales are obtained to extract features at different scales. A 1x1 convolution is added after each of the three dilated convolutions to adjust the feature channels. A 1x1 convolution layer is added after the adaptive pooling layer to adjust the feature channel; The 1x1 convolution layer, the pooling pyramid and the adaptive pooling layer are calculated in parallel, and the outputs are fused and superimposed through the Concat layer. After the Concat layer, the 1x1 convolution layer is used again to adjust the feature channel to form a new feature.
5. The ground-based cloud image segmentation method based on the improved U-Net model according to claim 3, characterized in that: The method comprises: using a pooling layer to perform a downsampling operation on higher-level features extracted from a dilate-block to reduce the size of a feature map, and outputting the feature map to a next dilate block.
6. The ground-based cloud image segmentation method based on the improved U-Net model according to claim 1, characterized in that: The boundary refinement module DS-Path with a residual structure consists of three depth-separable convolutions, and the convolution kernel size is fixed to 3x3, which is used to receive the output of the corresponding dilate block Dilate-Block.
7. The ground-based cloud image segmentation method based on the improved U-Net model according to claim 1, characterized in that: The Im-CSAM module is constructed by combining the deep separable convolution and multi-head self-attention to transform the CBAM attention mechanism architecture, so that it is connected to the boundary refinement module DS-Path with the residual structure, including: The optimized CAM module is obtained by replacing the MLP in the CAM module with depth-wise separable convolution; The optimized SAM module is obtained by replacing the original global average pooling and global maximum pooling in the SAM module with multi-head self-attention; Obtaining an Im-CSAM module based on the optimized CAM module and SAM module; The feature map X obtained based on the boundary refinement module DS-Path with residual structure 2 The processing flow in the Im-CSAM module includes: The feature map X obtained based on the boundary refinement module DS-Path with residual structure 2 After downsampling through the maximum pooling layer and the average pooling layer, they enter the deep separable module with a two-layer structure. The output is summed based on the element-wise operation, and the channel attention feature is generated through the Sigmod activation function. 2 Perform element-wise multiplication operations to generate the input features required by the SAM module; the optimized channel attention calculation includes: Where W d1 and W d2 is the weight of the depthwise convolution, W p1 and W p2 is the weight of the point-wise convolution, and Represent the results of global average pooling and global maximum pooling respectively; Multi-head self-attention is used to replace the original global average pooling and global maximum pooling operations, and the features from the CAM module are accepted as input X 2c ; Among them, the optimized spatial attention calculation includes: M s (X 2c )=σ(MultiHeadAttention(X 2c )); In the formula, MultiHeadAttention represents the multi-head self-attention mechanism; Calculate the attention score and weighted sum value for each head independently, concatenate the outputs of N heads together, and perform linear transformation through the forward propagation network to obtain the final output feature map X 3 .
8. The ground-based cloud image segmentation method based on the improved U-Net model according to claim 1, characterized in that: The improved U-Net model also includes a bottleneck layer; The bottleneck layer is located at the bottom layer of the network and is used to receive the output from the encoder, filter and compress information through the operation of two convolutional layers, so that important features can be retained and passed to the decoder part.
9. The ground-based cloud image segmentation method based on the improved U-Net model according to claim 1, characterized in that: Each dilate-block of the encoder is directly connected to the corresponding up-block convolution block of the decoder in a skip connection manner.
10. A ground-based cloud image segmentation system based on an improved U-Net model, characterized in that: The system comprises: Build U-Net traditional model unit, which is suitable for building U-Net traditional model with encoder, skip connection layer and decoder as the overall architecture; Improved encoder unit, suitable for replacing traditional convolution in the traditional U-Net model with dilated convolution, and embedding dilated spatial convolution pooling pyramid to construct dilate-Block, improving the encoder of the traditional U-Net model; Improve the skip connection layer unit, which is suitable for adding a boundary refinement module DS-Path with a residual structure in the skip connection layer, and replacing the convolution layer of the traditional residual block with a depthwise separable convolution; construct an Im-CSAM module by combining the depthwise separable convolution and multi-head self-attention to transform the CBAM attention mechanism architecture, so that it is connected to the boundary refinement module DS-Path with a residual structure; Improved decoder unit adapted to introduce bicubic interpolation for upsampling in said decoder; Constructing an improved U-Net cloud image segmentation model unit, which is suitable for constructing an improved U-Net cloud image segmentation model based on the dilate-block, the boundary refinement module DS-Path with a residual structure, the Im-CSAM module, and the improvement of the decoder upsampling; The cloud image segmentation image generating unit is adapted to obtain a corresponding cloud image segmentation image by inputting the cloud image to be detected into the improved U-Net cloud image segmentation model.
