Cloud cover identification system and method for double-camera all-sky imager

Abstract

The invention relates to the technical field of cloud cover recognition, and discloses a double-camera all-sky imager cloud cover recognition system, which comprises a visible light camera, an infrared camera, a graphics processor connected with the visible light camera and the infrared camera, and a deep learning framework connected with the graphics processor, according to the system, an SSTD-Unet cloud cover segmentation network is adopted to train independent models for visible light and infrared images, and the SSTD-Unet cloud cover segmentation network is of a symmetrical structure composed of a down-sampling coding part, a bottleneck part and an up-sampling decoding part. The cloud cover identification system can realize stable and high-precision cloud cover identification under various weather and illumination conditions, and compared with a traditional threshold segmentation method, the cloud cover identification system has the advantages that the performance is obviously improved, and a reliable solution is provided for realizing high-precision, full-automatic and all-weather cloud cover identification.

Description

Technical Field

[0001] This invention relates to the field of cloud cover recognition technology, specifically to a cloud cover recognition system and method using a dual-camera all-sky imager. Background Technology

[0002] The formation and evolution of clouds are closely related to weather changes, significantly impacting climate model assessments and weather forecasts. Furthermore, continuous observation and analysis of cloud cover data can be used for solar energy production forecasting and photovoltaic power generation performance evaluation. Currently, automated cloud cover observation equipment, such as automatic cloud cover meters, is widely used in various meteorological stations and also has applications in demanding precision meteorological observation fields such as meteorological science, telemetry, solar energy resource research, and aviation / ship meteorology.

[0003] Accurate identification of cloud cover across the entire sky is crucial for meteorological, climate research, and aerospace fields. Traditional cloud cover identification algorithms largely rely on manually designed features and threshold segmentation. These algorithms suffer from poor generalization ability and insufficient stability under complex and variable atmospheric conditions. Furthermore, visible light-based imagers cannot operate at night, leading to data interruptions. Summary of the Invention

[0004] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a cloud cover identification system and method for a dual-camera all-sky imager.

[0005] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: a cloud cover recognition system for a dual-camera all-sky imager, comprising a visible light camera and an infrared camera, a graphics processor connected to the visible light camera and the infrared camera, and a deep learning framework connected to the graphics processor. The system uses the SSTD-Unet cloud cover segmentation network to train independent models for visible light and infrared images respectively. The SSTD-Unet cloud cover segmentation network has a symmetrical structure consisting of a downsampling encoding part, a bottleneck part, and an upsampling decoding part. The downsampling encoder path consists of four downsampling... The cloud segmentation network consists of several stages. Each downsampling stage contains a dual multipath convolutional block and a POOL block. The bottleneck layer contains a dual multipath convolutional block and a channel-space fusion attention mechanism module. The upsampling decoder path consists of four upsampling stages, each containing a decoder block. The decoder block consists of a 2×2 transposed convolutional layer and two consecutive 3×3 convolutional layers. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. The input of the SSTD-Unet cloud segmentation network is a full-sky image, and the output is a pixel-level cloud segmentation map.

[0006] Preferably, the visible light camera is a fisheye visible light camera.

[0007] Furthermore, each dual multipath convolutional block consists of two consecutive multipath convolutional blocks, and each multipath convolutional block contains the following four parallel branches: a 1×1 convolutional branch, using a 1×1 convolutional kernel, responsible for extracting local detail features; a 3×3 convolutional branch, composed of 1×1 convolution and 3×3 convolution, which first performs channel dimensionality reduction and then extracts medium-scale features; a 5×5 convolutional branch, composed of 1×1 convolution and 5×5 convolution, which obtains global contextual information with a large receptive field; and a pooling branch, which uses 3×3 max pooling followed by 1×1 convolution to maintain feature invariance and enhance model robustness. The outputs of the four branches are concatenated in the channel dimension and processed by a batch normalization layer (BN) and a ReLU activation function to form the final output.

[0008] Furthermore, the channel-space fusion attention mechanism module comprises two cascaded sub-modules: channel attention and spatial attention. The channel attention sub-module first compresses the spatial dimension using global average pooling and global max pooling, generating two 1×1×C channel descriptors. These descriptors then share a two-layer MLP (Multilayer Perceptron). The first layer compresses the number of channels to C / r, where C is the original number of channels and r is the reduction ratio (default 16). The second layer restores the original number of channels to C. The outputs of the two branches are summed element-wise, and a channel attention matrix is ​​generated using the Sigmoid function. Finally, the matrix is ​​multiplied channel-wise with the original feature map to enhance key features in the channel dimension. The spatial attention sub-module performs average pooling and max pooling in the channel dimension to obtain two H×W×1 feature maps. These two feature maps are concatenated in the channel dimension to form an H×W×2 input. A 7×7 convolutional layer compresses the number of channels to 1, and a spatial attention matrix is ​​generated using the Sigmoid function. Finally, the matrix is ​​multiplied position-wise with the input feature map to focus on key regions in the spatial dimension. H is the feature map height, and W is the feature map width.

[0009] The cloud cover recognition system of the dual-camera all-sky imager uses a cloud cover recognition method in which the fisheye visible light camera and the infrared camera automatically switch and work independently during the day and night according to the sunrise and sunset times. The cloud cover segmentation network of SSTD-Unet trains two models with the same structure but independent parameters for visible light images and infrared images respectively, and processes cloud images during the day and night respectively.

[0010] The visible light cloud image processing method is as follows: the original visible light images are manually labeled, 70% of the images are used as the training set, 20% of the images are used as the validation set, and 10% of the images are used as the test set. Data augmentation is performed on the training set, including random rotation of 5-10 degrees, horizontal flipping, and random scaling.

[0011] The infrared cloud image processing method is as follows: the original infrared images are manually labeled, 70% of the images are used as the training set, 20% of the images are used as the validation set, and 10% of the images are used as the test set. Data augmentation is performed on the training set, including random rotation of 5-10 degrees, horizontal flipping, and random scaling.

[0012] (III) Beneficial Effects Compared with existing technologies, the present invention provides a cloud cover recognition system for a dual-camera all-sky imager, which has the following beneficial effects: This system employs a dual-camera configuration, including a fisheye visible light camera and an infrared camera. It can automatically switch between day and night based on sunrise and sunset times, operating independently to achieve continuous observation throughout the day. Addressing the characteristics of dual-modal data, the cloud cover segmentation network SSTD-Unet of this invention, based on the U-Net architecture, incorporates a multi-path convolutional structure to achieve multi-scale feature extraction and cross-scale information interaction, thereby effectively improving the accuracy of cloud layer recognition and cloud cover calculation. Furthermore, a channel-space fusion attention mechanism is introduced between the encoder and decoder to enhance the model's ability to focus on key cloud features. This invention trains two models (SSTD-Unet-VIS and SSTD-Unet-IR) with identical structures but independent parameters for visible light and infrared images, respectively, to process daytime and nighttime cloud images.

[0013] This method can achieve stable and high-precision cloud cover identification under various weather and lighting conditions. Compared with traditional threshold segmentation methods, it has a significant performance improvement and provides a reliable solution for achieving high-precision, fully automatic, and all-weather cloud cover identification. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the U-Net network structure.

[0015] Figure 2 This is a schematic diagram of the cloud data segmentation network of the present invention.

[0016] Figure 3 This is a schematic diagram of the structure of the dual multipath convolution module of the present invention.

[0017] Figure 4 This is a schematic diagram of the channel space fusion attention mechanism module of the present invention.

[0018] Figure 5 This is the original visible light data and its manually annotated diagram.

[0019] Figure 6 This consists of the original infrared data and its manually annotated images. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1-6 .

[0022] A cloud cover recognition system using a dual-camera all-sky imager includes a visible light camera and an infrared camera, a graphics processor connected to both cameras, and a deep learning framework connected to the graphics processor. The system uses the SSTD-Unet cloud cover segmentation network to train independent models for the visible light and infrared images respectively.

[0023] like Figure 1 As shown, U-Net is a typical encoder-decoder image segmentation network, originally proposed by Ronneberger et al. in 2015 for medical image segmentation tasks. The network structure is symmetrical, resembling the letter "U," hence the name U-Net. It can be divided into three parts: an encoder (downsampling encoding part), a decoder (upsampling decoding part), and skip connections.

[0024] In the encoder part, the network gradually extracts high-level semantic features of the image through multiple layers of convolution and max pooling operations, while reducing the spatial resolution of the feature maps. Each stage includes two 3×3 convolutions and one 2×2 max pooling operation, which increases the number of feature channels layer by layer (e.g., 64→128→256→512→1024) and halves the spatial size layer by layer.

[0025] In the decoder section, the network uses upsampling (such as deconvolution or bilinear interpolation) to gradually restore the spatial resolution of the image. After each upsampling, the feature map from the corresponding encoder stage is fused with the feature map from the decoder through feature concatenation, achieving complementarity between high- and low-level semantic information.

[0026] This skip connection mechanism preserves detailed features such as edges and textures while recovering spatial information of the image, enabling U-Net to perform excellently in pixel-level tasks. Finally, the decoder maps the number of feature map channels to the number of classes through 1×1 convolutional layers, achieving pixel-by-pixel classification prediction.

[0027] U-Net prediction principle The output of U-Net is denoted as Here, H and W represent the height and width of the image, respectively, and C represents the number of categories. The network outputs a vector of length C for each pixel (x, y) of the input image, representing the predicted probability of that pixel belonging to each category.

[0028] The Softmax function is used to calculate the probability that the (x, y)th pixel belongs to class C: ; Where Zx,y,c represents the linear output of the network to class C at that pixel location (i.e., the feature values ​​after the last convolutional layer). The final predicted class is determined by the following formula: ; That is, the category with the highest probability is selected as the prediction result for that pixel, thus obtaining a complete semantic segmentation image.

[0029] During the training phase, U-Net typically uses a pixel-wise cross-entropy loss function for optimization, which takes the following form: ; Here, yx,y,c are the true labels (one-hot encoded) of pixels (x,y). Furthermore, to address issues such as small target regions or class imbalance, improved loss functions such as Dice Loss or Focal Loss are often introduced to enhance the model's segmentation accuracy.

[0030] like Figure 2 As shown, the SSTD-Unet structure of this invention includes a symmetrical structure of Encoder + Bottleneck + Decoder. Unlike the standard U-Net, this paper uses a dual multi-path convolutional module instead of a conventional convolutional block in the encoder part to achieve multi-scale feature extraction, and introduces a channel-space fusion attention module in the bottleneck layer to enhance feature representation capabilities. The network input is a 592×592 all-sky image, and the output is a 592×592 pixel-level cloud segmentation map.

[0031] The various functional modules in the network, such as Figure 2 As shown, the downsampling encoding block (ENCODE block) consists of dual multipath convolutional blocks, which enable deeper multi-scale feature extraction and enhance the model's ability to express complex cloud features. The upsampling decoding block (DECODER block) consists of a 2×2 transposed convolutional layer (ConvTranspose) and two consecutive 3×3 convolutional layers. Each convolutional layer is followed by a batch normalization (BN) layer and a ReLU activation function. This module implements feature map upsampling and feature optimization. Figure 3As shown, the multi-path convolutional block consists of four parallel branches, each containing 1×1 convolution, 3×3 convolution, 5×5 convolution, and max pooling operations. The outputs of each branch are concatenated along the channel dimension and then processed by a batch normalization (BN) layer and a ReLU activation function. This module achieves multi-scale feature extraction, effectively capturing the morphological features of clouds at different scales; for example... Figure 4 As shown, the channel-space fusion attention module consists of a channel attention submodule and a spatial attention submodule connected in series. The channel attention submodule generates channel weights through global average pooling and global max pooling, while the spatial attention submodule generates spatial weights through channel-dimensional pooling operations and 7×7 convolutions. This module can adaptively enhance cloud-related features and suppress background interference. The POOL block uses a 2×2 max pooling operation with a stride of 2 to achieve feature map downsampling. The upsampling block uses a 2×2 transposed convolution with a stride of 2 to achieve feature map upsampling. SKIP_CONNECTION achieves feature fusion between corresponding layers of the encoder and decoder, preserving spatial detail information.

[0032] The encoder path consists of four downsampling stages, each containing a dual multipath convolutional block and a pool block. Specifically, stage one uses a dual multipath convolutional block with 3 input channels and 4 output channels, connected to a pool block to achieve 2x downsampling; stage two uses a dual multipath convolutional block with 4 input channels and 8 output channels, connected to a pool block; stage three uses a dual multipath convolutional block with 8 input channels and 16 output channels, connected to a pool block; and stage four uses a dual multipath convolutional block with 16 input channels and 32 output channels, connected to a pool block. If the input image shape is 592×592×3, the dimensions of the four feature maps output by the encoder are [296×296×4], [148×148×8], [74×74×16], and [37×37×32], respectively.

[0033] The bottleneck layer consists of a dual multipath convolutional block and a channel-space fusion attention block. The dual multipath convolutional block has 32 input channels and 64 output channels. The channel-space fusion attention block applies dual attention weights to the features, enhancing the model's ability to focus on key cloud regions. The dual multipath convolutional block outputs [37×37×64] features, while the channel-space fusion attention block does not change the feature size.

[0034] The decoder path consists of four upsampling stages, each containing a DECODER block. Stage 1 uses a DECODER block with 64 input channels and 32 output channels; Stage 2 has 32 input channels and 16 output channels; Stage 3 has 16 input channels and 8 output channels; and Stage 4 has 8 input channels and 4 output channels. Each DECODER block is fused with the corresponding layer features of the encoder via SKIP_CONNECTION to preserve spatial detail information. The feature sizes are [74×74×32], [148×148×16], [296×296×8], and [592×592×4], respectively.

[0035] The output layer consists of a 1×1 convolutional layer and a sigmoid activation function, which compresses the number of feature channels from 4 to the number of output categories to 2, generating the final segmentation probability map.

[0036] This network architecture achieves deeper multi-scale feature extraction through dual multi-path convolutional blocks, enhances the model's ability to focus on key features through a channel-space fusion attention mechanism, and effectively fuses detailed and semantic information through an encoder-decoder symmetric structure and skip connections. With a compact design of only 4 initial feature layers, the network maintains high performance while exhibiting high computational efficiency, demonstrating excellent performance in all-sky cloud cover recognition tasks.

[0037] Dual multipath convolution modules, such as Figure 3 As shown. In the encoder feature extraction part, a dual multipath convolution module is used instead of the double convolutional block in the traditional U-Net. Each dual multipath convolution module consists of two consecutive multipath convolutional blocks, and each multipath convolutional block contains four parallel branches, which extract features at different scales respectively: 1×1 Convolution Branch: Employs a 1×1 convolution kernel to extract local detail features, resulting in high computational efficiency and a small number of parameters. 3×3 convolution branch: Consists of cascaded 1×1 and 3×3 convolutions, performing channel dimensionality reduction first and then extracting medium-scale features. 5×5 convolution branch: Consists of cascaded 1×1 and 5×5 convolutions to obtain global contextual information with a large receptive field of view. Pooling branch: 3×3 max pooling followed by 1×1 convolution is used to preserve feature invariance and enhance model robustness. The outputs of the four branches are concatenated along the channel dimension, and then processed by a batch normalization (BN) layer and a ReLU activation function to form the final output. Dual multipath convolution significantly enhances the model's ability to represent complex cloud morphologies through two consecutive multi-scale feature fusions. If the number of input channels is C_in and the number of output channels is C_out, then each branch is evenly allocated C_out / 4 output channels to ensure a balanced contribution of multi-scale features.

[0038] Channel space fusion attention mechanism such as Figure 4 As shown. A channel-space fusion attention mechanism is introduced at the bottleneck layer. This module consists of two cascaded sub-modules: channel attention and spatial attention. The channel attention submodule first compresses the spatial dimension using global average pooling and global max pooling respectively, generating two 1×1×C channel descriptors. Then, it shares a two-layer MLP (Multilayer Perceptron). The first layer compresses the number of channels to C / r (r is the reduction ratio, default 16), and the second layer restores the original number of channels C. The outputs of the two branches are added element-wise and then the channel attention matrix is ​​generated by the Sigmoid function. Finally, the matrix is ​​multiplied with the original feature map channel by channel to complete the important feature enhancement of the channel dimension.

[0039] The spatial attention submodule performs average pooling and max pooling in the channel dimension to obtain two H×W×1 feature maps. The two feature maps are concatenated in the channel dimension to form an H×W×2 input. The number of channels is compressed to 1 through a 7×7 convolutional layer, and then the spatial attention matrix is ​​generated by the sigmoid function. Finally, the matrix is ​​multiplied position by position with the input feature map to achieve focusing on key regions in the spatial dimension.

[0040] The channel-space fusion attention module enables the model to adaptively focus on key features related to cloud identification through a dual attention mechanism of channels and space, effectively improving segmentation accuracy in complex sky backgrounds.

[0041] The platform configuration for deep learning training of the model of this invention is as follows:

[0042] Visible light data processing. The original visible light data consisted of 2200 images. The images were manually labeled, such as... Figure 5 The image on the left is the original image, and the image on the right is the annotation. Black represents the background and buildings, white represents clouds, and gray represents the sky.

[0043] The original visible light data images were 3504*3504 pixels in size, which were reduced to 592*592 pixels. 1540 images (70%) were used as the training set, 440 images (20%) as the validation set, and 220 images (10%) as the test set.

[0044] Data augmentation was performed on the training set, including random rotation of 5-10 degrees, horizontal flipping, and random scaling. After data augmentation, the training set contained 6160 images. The validation and test sets remained unchanged at 440 and 220 images, respectively.

[0045] Infrared data processing. A total of 1000 infrared images were processed, and the images were manually labeled, such as... Figure 6 The image on the left is the original image, and the image on the right is the annotation.

[0046] The original size of the infrared data images is 640*512, and this size is maintained. 700 images (70%) are used as the training set, 200 images (20%) as the validation set, and 100 images (10%) as the test set.

[0047] Data augmentation was performed on the training set, including horizontal flipping and random scaling. After data augmentation, the training set contained a total of 2100 images.

[0048] The proposed SSTD-UNet model architecture trains the SSTD-Unet-VIS and SSTD-Unet-IR models using visible light and infrared data, respectively. Aside from the different input data, the model training strategy and parameter settings remain consistent. Model training is implemented on the Python platform using the PyTorch deep learning framework, with Adam as the optimizer, a batch size of 16, and a total of 100 training epochs. The initial learning rate is set to 0.001, and an automatic learning rate decay strategy is used.

[0049] It should be noted that the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cloud cover recognition system using a dual-camera all-sky imager, comprising a visible light camera and an infrared camera, a graphics processor connected to the visible light camera and the infrared camera, and a deep learning framework connected to the graphics processor; the system employs a cloud cover segmentation network to train independent models for the visible light and infrared images respectively. The cloud cover segmentation network is a symmetrical structure composed of a downsampling encoding part, a bottleneck part, and an upsampling decoding part. The downsampling encoder path consists of four downsampling stages, each containing a dual multipath convolutional block and a pooling layer block. The bottleneck layer contains a dual multipath convolutional block and a channel-space fusion attention mechanism module. The upsampling decoder path consists of four upsampling stages, each containing a decoder block. The decoder block consists of a 2×2 transposed convolutional layer and two consecutive 3×3 convolutional layers. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The input of the cloud cover segmentation network is an all-sky image, and the output is a pixel-level cloud layer segmentation map.

2. The cloud cover recognition system for a dual-camera all-sky imager according to claim 1, characterized in that, The visible light camera is a fisheye visible light camera.

3. The cloud cover recognition system for a dual-camera all-sky imager according to claim 1, characterized in that, Each dual multipath convolutional block consists of two consecutive multipath convolutional blocks. Each multipath convolutional block contains the following four parallel branches: a 1×1 convolutional branch, using a 1×1 convolutional kernel, responsible for extracting local detail features; a 3×3 convolutional branch, composed of a 1×1 convolution and a 3×3 convolution, which first performs channel dimensionality reduction and then extracts medium-scale features; a 5×5 convolutional branch, composed of a 1×1 convolution and a 5×5 convolution, which obtains global contextual information with a large receptive field; and a pooling branch, which uses 3×3 max pooling followed by a 1×1 convolution to maintain feature invariance and enhance model robustness. The outputs of the four branches are concatenated along the channel dimension and processed by a batch normalization layer and a ReLU activation function to form the final output.

4. The cloud cover recognition system for a dual-camera all-sky imager according to claim 1, characterized in that, The channel-space fusion attention mechanism module comprises two cascaded sub-modules: channel attention and spatial attention. The channel attention sub-module first compresses the spatial dimension using global average pooling and global max pooling respectively, generating two 1×1×C channel descriptors. Then, they share a two-layer MLP multilayer perceptron. The first layer compresses the number of channels to C / r, where C is the original number of channels and r is the reduction ratio, which is 16 by default. The second layer restores the original number of channels to C. The outputs of the two branches are added element-wise and then the channel attention matrix is ​​generated by the Sigmoid function. Finally, the matrix is ​​multiplied with the original feature map channel by channel to complete the important feature enhancement of the channel dimension. The spatial attention submodule performs average pooling and max pooling in the channel dimension to obtain two H×W×1 feature maps. The two feature maps are concatenated in the channel dimension to form an H×W×2 input. The number of channels is compressed to 1 through a 7×7 convolutional layer, and then a spatial attention matrix is ​​generated by the Sigmoid function. Finally, the matrix is ​​multiplied positionally with the input feature map to achieve focus on key regions in the spatial dimension. H is the feature map height and W is the feature map width.

5. The cloud cover recognition method used in the cloud cover recognition system of the dual-camera all-sky imager according to claims 1-4 is characterized in that, The fisheye visible light camera and infrared camera automatically switch between day and night and work independently according to sunrise and sunset times. The cloud segmentation network of the SSTD-Unet is trained with two models with the same structure but independent parameters for visible light images and infrared images, respectively, to process cloud images during the day and at night.

6. The cloud cover identification method according to claim 5, characterized in that, The visible light cloud image processing method is as follows: the original visible light images are manually labeled, 70% of the images are used as the training set, 20% of the images are used as the validation set, and 10% of the images are used as the test set. Data augmentation is performed on the training set, including random rotation of 5-10 degrees, horizontal flipping, and random scaling.

7. The cloud cover identification method according to claim 5, characterized in that, The infrared cloud image processing method is as follows: the original infrared images are manually labeled, 70% of the images are used as the training set, 20% of the images are used as the validation set, and 10% of the images are used as the test set. Data augmentation is performed on the training set, including random rotation of 5-10 degrees, horizontal flipping, and random scaling.

Images