A pseudo-visible light cloud image generation method for night sea fog monitoring

Abstract

This invention provides a pseudo-visible light cloud image generation method for nighttime sea fog monitoring, relating to the field of artificial intelligence technology. The method includes the following steps: S1: Constructing a training dataset based on Himawari-8 satellite cloud images, selecting visible light and far-infrared band data from them; S2: Constructing a multi-scale feature fusion generative adversarial network under an attention mechanism, based on the Pix2Pix framework and a multi-scale feature fusion network; S3: Training a mapping network from far-infrared band cloud images to visible light band cloud images using image translation technology to generate nighttime pseudo-visible light cloud images; S4: Verifying the model's accuracy for nighttime sea fog monitoring by combining the pseudo-visible light generation model with a sea fog detection model. This method can solve the problem of accurately distinguishing between low clouds and sea fog and improve the accuracy of nighttime sea fog monitoring.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method for generating pseudo-visible light cloud images for nighttime sea fog monitoring. Background Technology

[0002] Sea fog is a hazardous weather phenomenon occurring in the lower atmosphere over the sea, coastlines, and islands, caused by the condensation of water vapor into water droplets or ice crystals, resulting in horizontal visibility of less than 1000 meters. As a hazardous weather event, sea fog severely impacts the safety of marine and coastal activities, making accurate monitoring of sea fog crucial for shipping, fisheries, and other production activities. Conventional sea fog monitoring involves deploying sea fog observation stations, but this method is limited by station density and observation time, making it impossible to achieve large-scale, long-term, continuous monitoring of sea fog. Satellite remote sensing data offers numerous advantages, including wide coverage and high spatiotemporal resolution, enabling continuous monitoring of the distribution range of sea fog, particularly its formation and dissipation dynamics. This capability is irreplaceable by other conventional monitoring methods and is becoming an important means of sea fog monitoring.

[0003] Since the 1970s, scholars have attempted to utilize the differences in the radiation characteristics of clouds and fog to conduct research on fog monitoring using satellite remote sensing, with the threshold method being the most commonly used approach. However, because the threshold method primarily targets the radiation characteristics of sea fog, it struggles to utilize the textural features of sea fog, thus limiting the monitoring accuracy. Due to the influence of solar radiation, different sea fog monitoring models need to be designed for daytime and nighttime, making it difficult to establish an all-weather sea fog remote sensing algorithm. Furthermore, since low clouds and sea fog are not fundamentally different in physical properties and their spectral characteristics are extremely similar, the identification of low clouds and sea fog mainly relies on the differences in their textural characteristics. Although the accuracy of distinguishing between sea fog and low clouds can be improved during daytime sea fog monitoring by mining the rich textural features contained in the visible light band, the lack of visible light band coverage at night, coupled with the fact that existing nighttime sea fog monitoring methods mostly classify clouds at the pixel level and cannot perceive the overall textural features of sea fog, makes the differentiation between low clouds and sea fog a challenging problem. Summary of the Invention

[0004] The problem addressed by this invention is how to solve the difficulty in accurately distinguishing between low clouds and sea fog, and to improve the accuracy of sea fog monitoring at night.

[0005] To address the above problems, this invention provides a method for generating pseudo-visible light cloud images for nighttime sea fog monitoring, comprising the following steps:

[0006] S1: Construct a training dataset based on Himawari-8 satellite cloud images and selecting visible light and far-infrared band data from them;

[0007] S2: Based on the Pix2Pix framework and combined with a multi-scale feature fusion network, a multi-scale feature fusion generative adversarial network under the attention mechanism was constructed.

[0008] S3: Use image translation technology to train a mapping network from far-infrared band cloud images to visible light band cloud images to generate pseudo visible light cloud images at night.

[0009] S4: The accuracy of the model for nighttime sea fog monitoring was verified by combining a pseudo-visible light generation model with a sea fog detection model.

[0010] In the aforementioned method, a study on pseudo-visible light cloud image generation for nighttime sea fog monitoring was conducted using multispectral information from Himawari-8 satellite cloud images based on a deep learning model. Firstly, a multi-scale feature fusion generative adversarial network (GAN) for nighttime pseudo-visible light generation was proposed, targeting cloud and fog characteristics under an attention mechanism. This network, based on the spectral characteristics of cloud images in different imaging channels, designed multi-level network modules to enhance its ability to mine cloud image features. The introduction of multi-scale fusion enables the network to capture structural and detailed information at different scales of cloud images, thereby improving the accuracy of nighttime sea fog monitoring.

[0011] Further, step S1 includes:

[0012] S11: Establish the mapping relationship between other bands of Himawari-8 satellite cloud images and visible light bands, select daytime far-infrared bands and visible light bands data to train the network, and use nighttime far-infrared band data to invert pseudo-visible cloud images.

[0013] Furthermore, in step S2, the multi-scale feature fusion generative adversarial network consists of a generator network and a discriminator network;

[0014] The generator network employs an encoder and decoder structure.

[0015] In the encoding stage of the encoder structure, the input cloud map is processed layer by layer through multiple first group modules to extract cloud map features;

[0016] In the decoding stage of the decoder structure, the second group of modules receives feature maps from the encoder and decoder, unifies the number of feature maps, and splices multi-scale feature maps. The spliced ​​feature maps are then fused using convolution operations to generate a pseudo visible light cloud map.

[0017] The discriminant network uses a Markov PatchGAN structure, consisting of five convolutional layers, to distinguish between pseudo visible light cloud images and real cloud images.

[0018] Furthermore, the first group of modules consists of two convolutional layers and one channel attention module;

[0019] Each convolutional layer consists of a 3×3 convolutional kernel, BN, and ReLU. After two convolutional layers, the feature map is input into the channel attention layer. The channel attention module perceives the contribution of the cloud maps of different channels to the model training process and assigns different weights to different channels.

[0020] Furthermore, the second set of modules consists of a multi-scale feature stitching module and a multi-scale feature fusion module;

[0021] The feature stitching module stitches together cloud map features from different layers of different codecs, and the feature fusion module fuses the stitched cloud map features.

[0022] Furthermore, in the multi-scale stitching module of the second group of modules, feature maps of the same level from the encoder are received and the number of feature maps is unified by convolution.

[0023] Max pooling or bilinear interpolation is used to unify the feature map resolution for encoder-decoder layers of different sizes, convolution is used to unify the number of features, and skip connections are used to obtain low-level feature information and high-level semantic information of the cloud map.

[0024] After obtaining five feature maps at different scales, the feature maps are concatenated. In the feature fusion module, convolution operations are performed on the concatenated feature maps, and features are fused to reduce redundant information. The feature map is calculated as follows:

[0025]

[0026] Where A represents the first group of encoder modules, B represents the second group of decoder modules, N represents the total number of the second group of decoder modules, C(·) represents convolution operation, H(·) represents the feature fusion mechanism implemented by convolution, batch normalization and ReLU activation function, D(·) and U(·) represent downsampling and upsampling operations respectively, and [·] represents concatenation operation.

[0027] Furthermore, the objective function in training a generative adversarial network consists of two parts: adversarial loss and L1 loss.

[0028] The formula for adversarial loss is:

[0029] L CGAN (G,D)=E y [logD(x,y)]+E x [log(1-D(x,G(x))]

[0030] Where x represents the far-infrared cloud image as input to the generator G, resulting in the generated image G(x); {x,y} and {x,G(x)} are input to the discriminator D, where y represents the visible light cloud image corresponding to x;

[0031] To obtain low-frequency information from the cloud image, an L1 loss function is introduced to constrain the difference between the generated image G(x) and the real image y on a global pixel basis, as follows:

[0032] L L1 (G)=E x,y [||yG(x)||1]

[0033] The objective function of the model is expressed as:

[0034]

[0035] Furthermore, the pseudo-visible light generation model uses daytime visible light and far-infrared bands 14 to 16 as training samples; the sea fog detection model uses daytime sea fog recognition dataset as training samples for training.

[0036] After model training, the nighttime far-infrared band cloud image to be detected is input into the pseudo-visible light generation model to generate a nighttime pseudo-visible light cloud image; the pseudo-visible light cloud image is then input into the sea fog detection model to generate a nighttime sea fog recognition result.

[0037] The present invention employing the above technical solution has the following beneficial effects:

[0038] This invention addresses the difficulties in current satellite cloud imagery for sea fog monitoring. Based on the analysis of the spectral characteristics of cloud imagery, it proposes a pseudo-visible light cloud image generation method for nighttime sea fog monitoring by constructing a multi-scale fusion generative adversarial network. This method solves the problem that traditional methods have difficulty in accurately distinguishing between low clouds and sea fog, thereby improving the accuracy of nighttime sea fog monitoring. Attached Figure Description

[0039] Figure 1 The flowchart of the pseudo-visible light cloud image generation method for nighttime sea fog monitoring provided in this embodiment of the invention. Figure 1 ;

[0040] Figure 2 The flowchart of the pseudo-visible light cloud image generation method for nighttime sea fog monitoring provided in this embodiment of the invention. Figure 2 ;

[0041] Figure 3 This is a diagram of the multi-scale feature fusion generative adversarial network structure under the attention mechanism of the pseudo-visible light cloud image generation method for nighttime sea fog monitoring provided in this embodiment of the invention.

[0042] Figure 4 This is a schematic diagram of the module structure of the pseudo visible light cloud image generation method for nighttime sea fog monitoring provided in an embodiment of the present invention.

[0043] Figure 5A schematic diagram of the channel attention module network structure of the pseudo visible light cloud image generation method for nighttime sea fog monitoring provided in an embodiment of the present invention;

[0044] Figure 6 This is a multi-scale feature stitching and fusion image in module B3 of the second group of modules in the pseudo visible light cloud image generation method for nighttime sea fog monitoring provided in an embodiment of the present invention.

[0045] Figure 7 The flowchart illustrates the nighttime sea fog detection method, which is a pseudo-visible light cloud image generation method for nighttime sea fog monitoring provided in this embodiment of the invention. Detailed Implementation

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0047] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings to further illustrate the technical solutions of the present invention. However, the present invention is not limited to these embodiments.

[0048] Example

[0049] This embodiment provides a method for generating pseudo-visible light cloud images for nighttime sea fog monitoring, such as... Figure 1 and Figure 2 As shown, this method includes the following steps:

[0050] S1: Construct a training dataset based on Himawari-8 satellite cloud images and selecting visible light and far-infrared band data from them;

[0051] S2: Based on the Pix2Pix framework and combined with a multi-scale feature fusion network, a multi-scale feature fusion generative adversarial network under the attention mechanism was constructed.

[0052] S3: Use image translation technology to train a mapping network from far-infrared band cloud images to visible light band cloud images to generate pseudo visible light cloud images at night.

[0053] S4: The accuracy of the model for nighttime sea fog monitoring was verified by combining a pseudo-visible light generation model with a sea fog detection model.

[0054] Specifically, Himawari-8 is Japan's new generation geostationary weather satellite. The satellite carries an Advanced Himawari Imager (AHI) with 16 channels, covering the visible, near-infrared, mid-infrared, and far-infrared bands.

[0055] Specifically, in order to utilize the texture features contained in visible light cloud images in nighttime sea fog monitoring, we used image translation technology to train a mapping network from far-infrared cloud images to visible light cloud images to generate pseudo-visible light cloud images at night.

[0056] See Figure 2 Step S1 includes:

[0057] S11: Establish the mapping relationship between other bands of Himawari-8 satellite cloud images and visible light bands, select daytime far-infrared bands and visible light bands data to train the network, and use nighttime far-infrared band data to invert pseudo-visible cloud images.

[0058] Specifically, in order to train a generative adversarial network to generate pseudo-visible light cloud images at night, it is necessary to establish a mapping relationship between other bands of satellite cloud images and the visible light band. Since the radiation characteristics of the far-infrared band are basically the same during the day and at night, data from various far-infrared bands and the visible light band during the day are selected to train the network, so as to achieve the goal of inverting pseudo-visible cloud images using far-infrared band data at night.

[0059] In step S2, the multi-scale feature fusion generative adversarial network consists of a generator network and a discriminator network.

[0060] The generator network employs an encoder and decoder structure.

[0061] In the encoding stage of the encoder structure, the input cloud map is processed layer by layer through multiple first group modules to extract cloud map features;

[0062] In the decoding stage of the decoder structure, the second group of modules receives feature maps from the encoder and decoder, unifies the number of feature maps, and splices multi-scale feature maps. The spliced ​​feature maps are then fused using convolution operations to generate a pseudo visible light cloud map.

[0063] The discriminant network uses a Markov PatchGAN structure, consisting of five convolutional layers, to distinguish between pseudo visible light cloud images and real cloud images.

[0064] See Figure 3The diagram shows the architecture of a multi-scale feature fusion generative adversarial network (MFC-GAN) with an attention mechanism. Based on the Pix2Pix (Image-to-Image Translation with Conditional Adversarial Networks) framework combined with a multi-scale feature fusion network, an MFC-GAN is constructed. The input cloud image is processed by the generator, which consists of multiple layers: Encoder Module Group A and Decoder Module Group B, to obtain a pseudo-visible light cloud image. The pseudo-visible light cloud image, the input cloud image, and the label cloud image serve as input to the discriminator to generate a pseudo-visible light cloud image for nighttime conditions.

[0065] The first group of modules consists of two convolutional layers and one channel attention module;

[0066] Each convolutional layer consists of a 3×3 convolutional kernel, BN, and ReLU. After two convolutional layers, the feature map is input into the channel attention layer. The channel attention module perceives the contribution of the cloud maps of different channels to the model training process and assigns different weights to different channels.

[0067] See Figure 4 The illustrated module structure consists of two convolutional layers and a channel attention module. Each convolutional layer comprises a 3×3 kernel, Batch Normalization (BN), and ReLU. After passing through these two convolutional layers, the feature maps are input into the channel attention layer. The channel attention module assigns different weights to different channels by perceiving the contribution of the feature maps from different channels to the model training process. The second module (Decoder Module Group B) consists of a multi-scale feature concatenation module and a multi-scale feature fusion module. The feature concatenation module concatenates feature maps from different layers of different encoders and decoders, while the feature fusion module fuses the concatenated feature maps, thereby reducing feature loss during network sampling and enabling deeper feature extraction.

[0068] See Figure 5The channel attention module network structure shown first performs channel-wise max pooling and average pooling on the depth feature map, resulting in two one-dimensional vectors, the dimension of which is the number of channels. Then, the two vectors are input into a shared-weight MLP (Multilayer Perceptron), the output vectors are fused, and after passing through an activation function, they are used as channel attention weights and distributed sequentially to each depth feature map.

[0069] The second group of modules consists of a multi-scale feature stitching module and a multi-scale feature fusion module;

[0070] The feature stitching module stitches together cloud map features from different layers of different codecs, and the feature fusion module fuses the stitched cloud map features.

[0071] In the multi-scale stitching module of the second group of modules, the feature maps of the same level of the encoder are received and the number of feature maps is unified by convolution.

[0072] Max pooling or bilinear interpolation is used to unify the feature map resolution for encoder-decoder layers of different sizes, convolution is used to unify the number of features, and skip connections are used to obtain low-level feature information and high-level semantic information of the cloud map.

[0073] After obtaining five feature maps at different scales, the feature maps are concatenated. In the feature fusion module, convolution operations are performed on the concatenated feature maps, and features are fused to reduce redundant information. The feature map is calculated as follows:

[0074]

[0075] Where A represents the first group of encoder modules, B represents the second group of decoder modules, N represents the total number of the second group of decoder modules, C(·) represents convolution operation, H(·) represents the feature fusion mechanism implemented by convolution, batch normalization and ReLU activation function, D(·) and U(·) represent downsampling and upsampling operations respectively, and [·] represents concatenation operation.

[0076] See Figure 6The diagram shows the multi-scale feature concatenation and fusion map of module B3 in the second group of modules. Because the second group of modules operates in different decoding layers, the source of features acquired by its multi-scale feature concatenation module and the processing operations on different features vary. To further illustrate the structure and feature concatenation and fusion strategy of the multi-scale feature concatenation and fusion module in the second group of modules, the multi-scale feature concatenation and fusion process of module B3 in the second group of modules is used as an example. In the multi-scale concatenation module of module B3 in the decoder layer group of modules, it receives feature maps from the same level of the encoder and uses convolution to unify the number of feature maps. For encoder-decoder layers of different scales, max pooling or bilinear upsampling is first used to unify the feature map resolution, convolution is used to unify the number of features, and skip connections are used to obtain low-level feature information and high-level semantic information of the cloud map. Finally, after obtaining five feature maps of different scales, the feature maps are concatenated. In the feature fusion module, convolution and other operations are performed on the concatenated feature maps to further fuse features and reduce redundant information.

[0077] The objective function in training a generative adversarial network consists of two parts: adversarial loss and L1 loss.

[0078] The formula for adversarial loss is:

[0079] L CGAN (G,D)=E y [logD(x,y)]+E x [log(1-D(x,G(x))]

[0080] Where x represents the far-infrared cloud image as input to the generator G, resulting in the generated image G(x); {x,y} and {x,G(x)} are input to the discriminator D, where y represents the visible light cloud image corresponding to x;

[0081] To obtain low-frequency information from the cloud image, an L1 loss function is introduced to constrain the difference between the generated image G(x) and the real image y on a global pixel basis, as follows:

[0082] L L1 (G)=E x,y [||yG(x)||1]

[0083] The objective function of the model is expressed as:

[0084]

[0085] Specifically, G needs to minimize the objective function through multiple iterations of training, while D aims to maximize the objective function.

[0086] Among them, the pseudo-visible light generation model uses daytime visible light and far-infrared bands 14 to 16 as training samples; the sea fog detection model uses daytime sea fog recognition dataset as training samples for training.

[0087] After model training, the nighttime far-infrared band cloud image to be detected is input into the pseudo-visible light generation model to generate a nighttime pseudo-visible light cloud image; the pseudo-visible light cloud image is then input into the sea fog detection model to generate a nighttime sea fog recognition result.

[0088] Specifically, a pseudo-visible light cloud image is generated based on a far-infrared band cloud image, and then the generated pseudo-visible light cloud image is used for sea fog monitoring. Therefore, the performance of the proposed method is evaluated from two aspects: cloud image generation and sea fog monitoring.

[0089] See Figure 7 To verify the effectiveness of the proposed model for nighttime sea fog monitoring, a nighttime sea fog identification method is proposed, consisting of a pseudo-visible light generation model and a sea fog detection model, combined with pseudo-visible light cloud imagery and model transfer. First, the pseudo-visible light generation model uses daytime visible light and far-infrared bands 14-16 as training samples, while the sea fog detection model is trained using the daytime sea fog identification dataset. After training, the nighttime far-infrared cloud imagery to be detected is input into the pseudo-visible light generation model to generate a nighttime pseudo-visible light cloud imagery. Finally, the pseudo-visible light cloud imagery is input into the sea fog detection model to generate the nighttime sea fog identification result.

[0090] This method, based on a deep learning model, utilizes multispectral information from Himawari satellite cloud images to conduct research on pseudo-visible light cloud image generation for nighttime sea fog monitoring. Firstly, considering cloud and fog characteristics under an attention mechanism, a multi-scale feature fusion generative adversarial network (GAN) for nighttime pseudo-visible light generation is proposed. This network, based on the spectral characteristics of cloud images in different imaging channels, designs multi-level network modules to enhance its ability to mine cloud image features. The introduction of multi-scale fusion enables the network to capture structural and detailed information at different scales of the cloud image. Finally, a pseudo-visible light cloud image generation method for nighttime sea fog monitoring is proposed to address the difficulty of accurately distinguishing between low clouds and sea fog using traditional methods, thereby improving the accuracy of nighttime sea fog monitoring.

[0091] While the disclosure is as stated above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure, and all such changes and modifications will fall within the protection scope of this invention.

Claims

1. A method for generating pseudo-visual cloud images for night-time sea fog monitoring, characterized in that, The method comprises the steps of: S1: constructing a training data set based on a Himawari-8 satellite cloud image and selecting visible light band and far infrared band data therefrom; S2: constructing a multi-scale feature fusion generative adversarial network under an attention mechanism based on a Pix2Pix framework combined with a multi-scale feature fusion network; the multi-scale feature fusion generative adversarial network is composed of a generative network and a discriminative network; The generative network adopts an encoder and decoder structure; In the encoding stage of the encoder structure, the input cloud image is extracted layer by layer through a plurality of first group modules; the first group modules are composed of two convolution layers and one channel attention module; each convolution layer is composed of 3 3 convolution kernels and BN and ReLU After passing through the two convolution layers, the feature map is input into the channel attention layer; the channel attention module gives different weights to different channels by perceiving the contribution of the cloud image of different channels to the model training process. In the decoding stage of the decoder structure, the second group of modules receives feature maps from the encoder and the decoder, unifies the number of feature maps for multi-scale feature map splicing, performs multi-scale feature fusion on the spliced feature maps through convolution operation to generate pseudo visible light cloud images; the second group of modules is composed of a multi-scale feature splicing module and a multi-scale feature fusion module; the feature splicing module splices cloud image features from different layers of different encoders and decoders, and the feature fusion module performs fusion processing on the spliced cloud image features; The discriminative network adopts a Markov PatchGAN structure and is composed of five convolution layers, and is used for discriminating pseudo visible light cloud images and real cloud images; S3: training a mapping network from a far infrared band cloud image to a visible light band cloud image by using image translation technology to generate a night pseudo visible light cloud image; S4: verifying the precision of the model for night sea fog monitoring by combining the pseudo visible light generation model with a sea fog detection model.

2. The method for generating pseudo-visual cloud images for night sea fog monitoring according to claim 1, characterized in that, The step S1 comprises: S11: establishing a mapping relationship between Himawari-8 satellite cloud images of other bands and a visible light band, selecting data of daytime far infrared bands and a visible light band to train a network, and using nighttime far infrared band data to retrieve a pseudo visible cloud image.

3. The method for generating pseudo-visual cloud images for night sea fog monitoring according to claim 1, characterized in that, In the multi-scale splicing module of the second group of modules, the feature maps of the same level of the encoder are received and the number of feature maps is unified by using convolution; The feature resolution of different scale encoder-decoder layers is unified by using maximum pooling or bilinear interpolation operation, the number of features is unified by using convolution, and cloud image bottom layer feature information and high-level semantic information are obtained through jump connection; After obtaining five feature maps of different scales, the feature maps are spliced, convolution operation is performed on the spliced feature maps in the feature fusion module, and the features are fused to reduce redundant information; the feature map calculation is: ； wherein denotes the first set of modules of the encoder, denotes the second set of modules of the decoder, denotes the total number of the second set of modules of the decoder, denotes a convolution operation, denotes a convolution, batch normalization and ReLU the activation function implements a feature fusion mechanism, and denote down-sampling and up-sampling operations, respectively, denotes a concatenation operation.

4. The method for generating pseudo-visual cloud images for night sea fog monitoring according to claim 1, characterized in that, The objective function in the training of the generative adversarial network is composed of the adversarial loss and the loss two parts; The adversarial loss formula is: ； wherein, represents a far-infrared cloud picture as an input of a generator , obtaining a generated image ; and as an input of a discriminator D, wherein represents a visible light cloud picture corresponding to ; The low-frequency information of the cloud image is acquired, and The loss function constrains the generated image from the overall pixels and the real image The difference between them is represented as: ； The objective function of the model is represented as: 。 5. The method for generating pseudo-visual cloud images for night sea fog monitoring according to claim 1, characterized in that, The pseudo visible light generation model adopts daytime visible light and far infrared fourteenth to sixteenth band data as training samples; the sea fog detection model adopts a daytime sea fog recognition data set as a training sample for training; After the model is trained, the nighttime far infrared band cloud image to be detected is input into the pseudo visible light generation model to generate a nighttime pseudo visible light cloud image; the pseudo visible light cloud image is input into the sea fog detection model to generate a nighttime sea fog recognition result.

Images