All-sky nephogram detection and classification method based on attention pyramid network

Abstract

The invention discloses an attention pyramid network-based all-sky cloud picture detection and classification method, and aims to solve the problems of accuracy and efficiency of an existing cloud detection technology in processing multi-cloud layer coverage, cloud type diversity and dynamic cloud layer change. According to the method, a deep learning technology and an advanced attention mechanism are utilized, and a pyramid space pooling (PSP) module and a convolutional block attention module (CBAM) are combined, so that the recognition and classification capability of cloud features is effectively improved. By applying the method, various cloud types including but not limited to cirrus cloud, layer cloud, cumulus cloud and the like can be accurately identified and classified in the all-sky cloud picture, and more accurate data support is provided for meteorological monitoring and climate research. The method has the advantages of being high in precision, high in efficiency, high in adaptability and the like, cloud detection and classification performance can be remarkably improved, and the method is particularly suitable for meteorological analysis under complex weather conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of meteorological observation and image processing, and in particular relates to a full-sky cloud image detection and classification method based on an attention pyramid network. Background Art

[0002] Ground-based cloud observations play an important role in meteorological monitoring and climate research. The accuracy of all-sky cloud image detection and classification directly affects the data quality in application areas such as weather forecasting, environmental monitoring, and photovoltaic power generation prediction. However, existing cloud detection and classification methods are mostly based on traditional image processing techniques, which face significant challenges in dealing with complex all-sky cloud images, such as the diversity of cloud morphology, the complex interactions of cloud layers, and the fuzziness of the boundaries between different cloud types.

[0003] Traditional cloud detection and classification methods usually rely on simple threshold segmentation and rule-based classification. These methods may perform well in simple cloud scenes, but often fail to accurately identify and segment different types of clouds in complex cloud environments with multiple layers and types. In addition, traditional methods lack the ability to adapt to the dynamic changes of clouds, making it difficult to accurately reflect the actual changes of clouds in real time, thus limiting their effectiveness in real-time monitoring applications.

[0004] In recent years, deep learning techniques, especially convolutional neural networks (CNNs) and attention mechanisms, have made significant progress in the field of image recognition, greatly improving the accuracy of object detection and image classification. However, in terms of cloud detection and classification, existing deep learning methods still have difficulty in fully coping with the diversity and complexity of clouds. In particular, in the recognition task of multi-scale and multi-type clouds, existing methods still have deficiencies in accuracy and stability. Summary of the invention

[0005] In view of the technical problems existing in the prior art, the present invention provides a full-sky cloud image detection and classification method based on an attention pyramid network. In view of the problems that the existing methods have insufficient accuracy in dealing with complex cloud morphologies and dynamic cloud layer changes, and limited ability to distinguish different cloud types, the accuracy and efficiency of cloud detection and classification are significantly improved by introducing a pyramid spatial pooling module and a convolutional block attention mechanism.

[0006] The above method of the present invention aims to improve the accuracy and efficiency of cloud detection and classification, and is suitable for applications such as meteorological monitoring, climate research and environmental assessment.

[0007] A full-sky cloud image detection and classification method based on an attention pyramid network comprises: collecting full-sky cloud images and preprocessing the cloud images; inputting the preprocessed full-sky cloud images into a trained attention pyramid network, and finally outputting a corresponding semantic segmentation mask image to complete full-sky cloud image detection and classification; the attention pyramid network is composed of an encoder and a decoder, and the encoder is used to perform feature extraction and downsampling to extract a feature map of the full-sky cloud image; and the decoder is used to gradually restore the spatial resolution of the image on the input feature map to generate a pixel-level semantic segmentation result.

[0008] Furthermore, the preprocessing includes image enhancement, denoising and normalization. The preprocessing is used to improve the recognizability and processing efficiency of cloud features. These preprocessing steps provide higher quality data input for subsequent feature extraction and classification. The above preprocessing steps can all adopt existing methods. For example, image enhancement can use methods such as histogram equalization to improve the contrast of the image; Gaussian filtering, median filtering and other methods can be used to remove noise in the image; and normalization is used to normalize the pixel values ​​of the image to the interval [0,1] or [-1,1] to eliminate the lighting differences between different images.

[0009] In the present invention, the encoder's task is to extract high-level features of clouds from the input full-sky cloud image to provide basic information for subsequent classification and segmentation, specifically including feature extraction and downsampling:

[0010] Feature extraction: Using the pre-trained residual network (ResNet-50) as the backbone of the encoder, high-level feature extraction is performed on the input full-sky cloud image through multi-layer convolution and residual blocks. This part can capture the spatial structure, boundary information and texture features of the cloud layer, providing strong support for subsequent classification.

[0011] Downsampling: At the end of the encoder, the resolution of the input image is significantly reduced. Pooling is used to reduce the spatial dimensions of the feature map, improving the computational efficiency of the model while retaining key cloud features.

[0012] The task of the decoder is to gradually restore the spatial resolution of the image based on the high-level features extracted by the encoder and generate pixel-level semantic segmentation results. The innovation of the decoder part of the present invention is that the pyramid spatial pooling module (PSP Module) and the convolutional block attention module (CBAM) are organically combined to achieve mutual enhancement of multi-scale context information aggregation and feature selection.

[0013] Furthermore, the decoder uses a pyramid space pooling module to perform multi-scale pooling on the input feature map to obtain multi-scale context features, and then uses a multi-scale convolutional attention mechanism module to obtain an attention-enhanced feature map, and finally outputs the semantic segmentation result after upsampling.

[0014] Pyramid Spatial Pooling Module: The Pyramid Spatial Pooling Module (PSP Module) is introduced in the decoder part to capture multi-scale contextual information through pooling operations at different scales. This module enhances the model's understanding of global and local cloud features and helps to more accurately classify complex cloud structures.

[0015] Multi-scale convolutional attention mechanism module: The convolutional block attention module (CBAMBlock) is further introduced. This module uses the dual mechanisms of channel attention and spatial attention to refine the feature map. First, the channel attention mechanism screens the input feature map for importance and highlights the key feature channels; then, the spatial attention mechanism further highlights the important spatial regions in the feature map. This design enables the model to focus on important feature regions more accurately when processing complex cloud structures, improving the accuracy of cloud detection and classification.

[0016] Upsampling: Upsampling is used in each decoding stage to restore the image size, ensuring that the final output segmentation result has high resolution and fine boundaries. The upsampling is performed using bilinear interpolation.

[0017] Furthermore, the multi-scale convolutional attention mechanism module includes a channel attention module and a spatial attention module, and performs point-by-point multiplication of the feature maps output by the channel attention module and the PSP module and the result of the spatial attention module to generate an attention-enhanced feature map.

[0018] Furthermore, in the channel attention module, channel features are extracted through adaptive average pooling and maximum pooling operations for the multi-scale context features of the input, and the importance weight of each channel is calculated through a shared multi-layer perceptron.

[0019] Furthermore, in the spatial attention module, average pooling and maximum pooling operations are performed on the channel dimension for the multi-scale contextual features of the input, and then the spatial attention weights are calculated through convolution.

[0020] During the model training phase, the attention pyramid network uses a large amount of annotated all-sky cloud image data to perform end-to-end training on the model, and uses a hybrid loss function that combines cross entropy loss and Dice loss for optimization. While improving the overall classification accuracy, the hybrid loss enhances the ability to finely segment cloud boundaries, allowing the model to better learn the characteristics of different cloud types.

[0021] Specifically, the attention pyramid network uses a mixed loss function to optimize model parameters during the training phase. Furthermore, the mixed loss function is formed by multiplying the cross entropy loss function and the Dice loss function by the weight parameters respectively and then adding them together.

[0022] After the model training is completed, the validity of the model proposed in this paper is verified through verification experiments. This model can be used for new all-sky cloud maps to achieve real-time cloud detection and classification, accurately identify multiple cloud types (such as cirrus, stratus, cumulus, etc.), and provide reliable data support for meteorological analysis and environmental monitoring.

[0023] By applying this method, various cloud types can be accurately identified and classified in the full sky cloud map, including but not limited to cirrus, stratus and cumulus, etc., providing more accurate data support for meteorological monitoring and climate research. The present invention has the advantages of high precision, high efficiency and strong adaptability, and can significantly improve the performance of cloud detection and classification, and is particularly suitable for meteorological analysis under complex weather conditions.

[0024] The method of the present invention not only improves the accuracy and efficiency of cloud detection and classification through the combination of multi-scale feature extraction, attention mechanism enhancement and layer-by-layer refinement decoding, but also has strong adaptability and can achieve efficient cloud recognition under a variety of complex meteorological conditions. The method has important application value in the fields of weather forecasting, climate research and environmental monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a technical roadmap for the full-sky cloud image detection and classification method based on the attention pyramid network of the present invention. DETAILED DESCRIPTION

[0026] In order to more clearly explain the technical solution of the present invention, the full sky cloud image detection and classification method based on the attention pyramid network of the present invention is described in detail below through specific embodiments. However, the scope of protection of the present invention is not limited thereto.

[0027] (1) Data collection and preprocessing

[0028] In the present invention, in the model training stage, it is first necessary to use a panoramic cloud camera to collect a large amount of full-sky cloud image data; in the model detection stage, the full-sky cloud image collected online can also be directly used. In this embodiment, these image data are acquired in a high-resolution format of 1024×1024 pixels, and the sampling time interval is 5 minutes. In order to improve the training efficiency and classification accuracy of the model, the image data needs to go through the following preprocessing steps:

[0029] ① Image enhancement: Use methods such as histogram equalization to improve the contrast of the image and make the features of the clouds more obvious.

[0030] ② Denoising: Remove noise from the image through Gaussian filtering, median filtering and other methods to reduce interference introduced by sensors or environmental factors.

[0031] ③ Normalization: Normalize the pixel values ​​of the image to the range [0,1] or [-1,1] to eliminate the illumination differences between different images. The normalization formula is as follows:

[0032]

[0033] Among them, I is the original image after enhancement and denoising, I min and I max Respectively represent the minimum and maximum pixel values ​​of the image, I norm This is the normalized cloud image of the entire sky.

[0034] (2) Encoder part:

[0035] ① Feature extraction: Use the pre-trained ResNet-50 as the backbone network to extract high-level features from the input full sky cloud image (i.e., the full sky cloud image normalized in step (1)). ResNet-50 can effectively capture the local and global information in the image and generate the feature map X:

[0036] X = ResNet50(I norm )

[0037] ② Downsampling: In the downsampling process, the maximum pooling operation is used to reduce the spatial size of the feature map, thereby improving the computational efficiency and obtaining the output feature map F of the encoder. The formula is as follows:

[0038]

[0039] (3) Decoder part:

[0040] ① Use the PSP module to perform multi-scale pooling on the extracted feature maps. This module converts the input feature maps into feature representations of multiple scales through adaptive average pooling operations of different sizes:

[0041] P i (F)=Upsample(Conv(AdaptiveAvgPool i×i (F)))

[0042] Among them, F is the input feature map, AdaptiveAvgPool i×i represents the adaptive average pooling operation, Conv is a 1×1 convolution operation, Upsample is a bilinear interpolation upsampling operation, and Pi(F) represents the feature representation of the i-th scale. Features of all scales are concatenated to form a rich multi-scale context feature representation:

[0043] F psp =Concat(F,P1(F),P2(F),P3(F),P6(F))

[0044] This approach enables the model to use multi-scale contextual information to better understand the different forms and structures of clouds. At the same time, the PSP module is used to obtain multi-scale contextual features corresponding to the image.

[0045] ② Multi-scale convolutional attention mechanism module (CBAM): The CBAM module consists of two main parts: channel attention module and spatial attention module. This invention uses a combination of four CBAM modules of different scales. After each CBAM operation, the image size doubles. The following is the detailed operation process of a single CBAM module:

[0046] a. Channel attention module: In this module, channel features are extracted through adaptive average pooling (Global Average Pooling) and maximum pooling (Global Max Pooling) operations.

[0047] The importance weight of each channel is calculated through a shared multi-layer perceptron (MLP).

[0048] The channel attention calculation formula is as follows:

[0049] M c (F) = σ(MLP(AvgPool(F psp ))+MLP(MaxPool(F psp )))

[0050] Among them, σ represents the Sigmoid function, MLP is a multi-layer perceptron containing two fully connected layers, AvgPool represents the adaptive average pooling function, and MaxPool represents the maximum pooling function.

[0051] b. Spatial attention module: Through this module, the feature map output by the PSP module is average pooled and max pooled in the channel dimension, and then the spatial attention weight is calculated through 7×7 convolution. The spatial attention calculation formula is:

[0052] M S (F psp )=σ(Conυ([AυgPool(F psp );MaxPool(F psp )]))

[0053] Perform element-wise multiplication on the result of channel attention, the feature map output by the PSP module, and the result of the spatial attention module to generate the attention-enhanced feature map F cbam :

[0054] Fcban =M c (F psp )·F psp ·M s (F psp )

[0055] In this way, the model can more accurately focus on important features related to cloud detection and classification, improving the overall detection performance.

[0056] ③ Upsampling: In the final stage of the decoder, the present invention gradually restores the spatial resolution of the feature map through upsampling operations to make it consistent with the size of the input original image. Upsampling is also performed using bilinear interpolation to obtain the final output result Figure F output , the formula is as follows:

[0057] F output =Upsample(F cbam )

[0058] (4) Model training

[0059] In order to improve the accuracy of cloud detection and classification, the present invention adopts a hybrid loss function to optimize the model parameters. This loss function combines the cross-entropy loss and the Dice loss, which can achieve better results in processing unbalanced data and refining cloud boundaries.

[0060] ① Cross entropy loss Lce: used to measure the difference between the predicted category probability distribution and the actual category. Its formula is:

[0061]

[0062] Where N is the number of pixels, C is the number of cloud types, and y ic represents the true category label of pixel i, p ic Represents the class probability predicted by the model.

[0063] ②Dice loss L Dice : It is used to measure the overlap between the predicted results and the true results, which is particularly suitable for unbalanced data and fine-grained segmentation tasks. The calculation formula of Dice loss is:

[0064]

[0065] Among them, p i is the probability predicted by the model, t i is the true label of the target, and smooth is a smoothing factor to prevent division by zero.

[0066] ③Mixed loss function L Combined:The present invention combines the cross entropy loss and the Dice loss into a hybrid loss function, which is defined as follows:

[0067] L Combined =αL CE +βL Dice

[0068] Among them, α and β are two weight parameters used to balance the contribution of these two losses in the total loss. By adjusting α and β, the loss function can be flexibly adjusted according to the needs of specific tasks. In the present invention, α = 0.5, β = 0.5, that is, the weights of cross entropy loss and Dice loss are equal.

[0069] During the model training process, the present invention sets the following hyperparameters to achieve the best model performance:

[0070] Learning rate: The initial learning rate is set to 1*10-5. An adaptive decay strategy is used during training to dynamically adjust the learning rate based on the performance of the validation set, thereby accelerating convergence and avoiding overfitting.

[0071] Batch size: The size of each batch of data is 16, that is, 16 images are used for parameter update in each training iteration

[0072] Training method: Use dual-card distributed training and multi-GPU parallel computing to improve the efficiency of model training.

[0073] Using a large amount of annotated all-sky cloud image data, the hybrid loss function is continuously optimized through an end-to-end training method to improve the model's ability to distinguish different cloud types. In each training step, the difference between the model's prediction results and the true label is measured using the hybrid loss function, and the model parameters are updated using the back-propagation algorithm until the loss function is minimized.

[0074] Through the above steps, the attention pyramid network EAPNet of the present invention is finally obtained.

[0075] (5) Model evaluation

[0076] After the model training is completed, in order to evaluate the classification and segmentation effect of the model in practical applications, the present invention uses two indicators, mean intersection over union (mIOU) and pixel accuracy (PA), to evaluate the segmentation effect, as described below:

[0077] Mean intersection over union (mIOU): mIOU is used to measure the overlap between the predicted segmentation area and the actual segmentation area, and is an important indicator for measuring segmentation accuracy. The calculation formula for mIOU is:

[0078]

[0079] Among them, C is the number of categories, P c and T c They represent the predicted area and the true area of ​​category c, ∩ represents the intersection, and ∪ represents the union. The higher the mIOU value, the higher the segmentation accuracy of the model.

[0080] Pixel Accuracy (PA): PA measures the overall classification accuracy of the model for all pixels and is calculated as:

[0081]

[0082] Where N is the total number of pixels, y i is the true label, is the predicted label, Π is the indicator function, which is 1 when the classification is correct, otherwise it is 0. The larger the PA value, the better the effect.

[0083] The present invention compares the classification and segmentation effects of different mainstream semantic segmentation models on the full sky cloud map through experimental results, as shown in the following table:

[0084] Metrics U-Net PSPNet DeepLabV3+ EAPNet mIOU(%) 36.14 63.90 64.87 69.00 PA(%) 83.78 85.68 88.32 89.37

[0085] As can be seen from the table, the MIoU of EAPNet is 4.13% higher than the second best method DeepLabV3+; similarly, in terms of the PA indicator, EAPNet also performs well, which is 1.05% higher than DeepLabV3+, highlighting its effectiveness in pixel-level cloud classification and segmentation tasks.

[0086] The trained EAPNet model is applied to new all-sky cloud image data, which can identify and classify various cloud types in real time, such as cirrus, stratus, cumulus, etc., providing high-precision data support for meteorological analysis and environmental monitoring.

Claims

1. A method for detecting and classifying all-sky cloud images based on an attention pyramid network, characterized in that: include: Collect the whole sky cloud image and pre-process the cloud image; The preprocessed full-sky cloud map is input into the trained attention pyramid network, and finally the corresponding semantic segmentation mask map is output to complete the detection and classification of the full-sky cloud map; the attention pyramid network is composed of an encoder and a decoder, and the encoder is used to perform feature extraction and downsampling to extract the feature map of the full-sky cloud map; the decoder is used to gradually restore the spatial resolution of the image on the input feature map to generate a pixel-level semantic segmentation result.

2. The method for detecting and classifying all-sky cloud images based on an attention pyramid network according to claim 1, characterized in that: The preprocessing includes image enhancement, denoising and normalization.

3. The method for detecting and classifying all-sky cloud images based on the attention pyramid network according to claim 1, characterized in that: In the encoder, feature extraction uses the pre-trained ResNet-50 as the backbone network to extract high-level features from the input full-sky cloud image; downsampling uses the maximum pooling operation.

4. The method for detecting and classifying all-sky cloud images based on an attention pyramid network according to claim 1, characterized in that: The decoder uses a pyramid spatial pooling module to perform multi-scale pooling on the input feature map to obtain multi-scale context features, and then uses a multi-scale convolutional attention mechanism module to obtain an attention-enhanced feature map, and finally outputs a semantic segmentation result after upsampling.

5. The method for detecting and classifying all-sky cloud images based on an attention pyramid network according to claim 4, characterized in that: The multi-scale convolutional attention mechanism module includes a channel attention module and a spatial attention module. The feature map output by the channel attention module and the pyramid space pooling module and the result of the spatial attention module are multiplied point by point to generate an attention-enhanced feature map; the upsampling is performed by bilinear interpolation.

6. The method for detecting and classifying all-sky cloud images based on an attention pyramid network according to claim 5, characterized in that: In the channel attention module, channel features are extracted through adaptive average pooling and maximum pooling operations for the multi-scale context features of the input, and the importance weight of each channel is calculated through a shared multi-layer perceptron.

7. The method for detecting and classifying all-sky cloud images based on an attention pyramid network according to claim 5, characterized in that: In the spatial attention module, average pooling and maximum pooling operations are performed on the channel dimension for the multi-scale contextual features of the input, and then the spatial attention weights are calculated through convolution.

8. The method for detecting and classifying all-sky cloud images based on an attention pyramid network according to any one of claims 1 to 7, characterized in that: The attention pyramid network adopts a hybrid loss function to optimize model parameters during the training phase.

9. The method for detecting and classifying all-sky cloud images based on an attention pyramid network according to claim 8, characterized in that: The hybrid loss function is formed by multiplying the cross entropy loss function and the Dice loss function by the weight parameters respectively and then adding them together.

Images