True color cloud image generation method, device, electronic device and storage medium

Abstract

The present invention provides a true-color cloud image generation method, device, electronic device, and storage medium, belonging to the field of meteorological technology. The method comprises: obtaining infrared cloud image data and numerical model product data of a target area at the same time; inputting the preprocessed infrared cloud image data and numerical model product data into a true-color cloud image generation model to obtain a true-color cloud image at the corresponding time, output by the true-color cloud image generation model; the true-color cloud image generation model is generated by pre-training an improved Pix2Pix network. Through reasonable model structure design, the present invention proposes a true-color cloud image generation model based on the improved Pix2Pix network. While maintaining the overall structure of the model, the model is suitable for generating true-color cloud images of any size, solving the problem of missing visible light channel data at night. Experimental verification and quantitative indicator evaluation show that the model's MAE and RMSE indicators are superior to existing simulation cloud image generation methods.

Description

Technical Field

[0001] The present invention relates to the field of meteorological technology, and in particular to a true color cloud image generation method, device, electronic equipment and storage medium. Background Art

[0002] The FY-4A satellite (Fengyun-4 satellite) is equipped with the advanced geostationary radiation imager AGRI, whose observation band covers visible light to long-wave infrared. The cloud images in the visible and near-infrared bands have high resolution and rich information, which are very important for observing weather processes such as typhoons. Their energy comes from solar energy reflected by the surface and the atmosphere, and targets can be distinguished based on the different reflectivities of different objects. However, due to the lack of solar radiation at night, AGRI cannot observe this information at night. In addition, the surface radiation characteristics and temperature at night are different from those during the day, which greatly increases the difficulty of nighttime cloud prediction, resulting in serious limitations on real-time weather monitoring and satellite remote sensing at night. Therefore, the simulation generation of nighttime visible light true color cloud images (hereinafter referred to as true color cloud images) can make up for the lack of AGRI visible light channel data at night.

[0003] Researchers have made considerable progress in the simulation generation of nighttime visible light true-color cloud images. The key approach is to utilize meteorological products such as satellite infrared cloud images from the same time and region that can be detected at night, establish a mapping relationship, and generate the corresponding true-color cloud images. This method does not require the model to extract information from the temporal dimension and can simulate true-color cloud images at any time of night. In 2021, Cheng Wencong et al. proposed a satellite cloud image simulation generation method based on generative adversarial networks (GANs) and numerical model products. This method simulates 12-channel infrared cloud images from the FY-4A satellite and fused cloud images from 1, 2, and 3 channels of visible light during both daytime and nighttime. In the same year, Cheng Wencong et al. further proposed a method for simulating true-color cloud images from nighttime meteorological satellites based on generative adversarial networks. This method combines satellite infrared channel cloud image data from the same time and region with numerical model products into a GAN model to generate simulated true-color cloud images at night.

[0004] However, the above-mentioned true-color cloud map generation method based on the GAN model cannot easily switch the size of the simulated cloud map. Once the size of the simulated cloud map changes, the discriminator network structure needs to be redesigned. The model cannot be flexibly applied to the generation of simulated cloud maps in any geographical range and does not have scalability. Summary of the Invention

[0005] The present invention provides a true-color cloud image generation method, device, electronic device and storage medium to address the defect in the prior art that the model cannot be flexibly applied to the generation of simulated cloud images in any geographical range. The method has good expansion capabilities and can perform all-weather true-color cloud image monitoring for typhoons moving over a large area or the interaction of multiple typhoon weather systems.

[0006] In a first aspect, the present invention provides a true color cloud image generation method, comprising:

[0007] Obtain infrared cloud image data and numerical model product data of the target area at the same time;

[0008] Performing geolocation, radiometric calibration, and data normalization processing on the infrared cloud image data, and performing normalization processing on the numerical model product data;

[0009] Inputting the processed infrared cloud image data and the processed numerical model product data into a true color cloud image generation model to obtain a true color cloud image of a corresponding time output by the true color cloud image generation model;

[0010] The true color cloud image generation model is generated by training the improved Pix2Pix network using a pre-built data training set.

[0011] According to a true color cloud image generation method provided by the present invention, the improved Pix2Pix network specifically includes:

[0012] An upsampling module for the numerical model product data is added to the input ends of the generator and the discriminator of the original Pix2Pix network, and the upsampling module is composed of multiple transposed convolutional layers;

[0013] The structure of the discriminator is set to a U-Net structure including a downsampling module and an upsampling module;

[0014] Adding a convolutional attention module to the generator;

[0015] All convolutional layers of the improved Pix2Pix network are set as depthwise separable convolutional layers.

[0016] According to a true color cloud image generation method provided by the present invention, a pre-built data training set is used to train an improved Pix2Pix network, including:

[0017] Acquire multiple training samples; the sample data of each training sample includes infrared cloud image data and numerical model product data of the same time and the same area, and the label of each training sample is a true color cloud image of the corresponding time and the corresponding area;

[0018] Performing geolocation, radiometric calibration, visible light channel reflectance correction, and data normalization processing on the infrared cloud image data in each sample data and the true color cloud image in the corresponding label, and performing data normalization processing on the numerical model product data in each sample data;

[0019] Performing data cleaning on all training samples, and constructing the data training set using all the cleaned training samples, so as to train the improved Pix2Pix network using the data training set;

[0020] The data cleaning of all training samples includes deleting some training samples with missing values or invalid filling values in all training samples, and then screening out all training samples that are within the preset daytime period;

[0021] The improved Pix2Pix network is trained iteratively using all training samples after data cleaning and the labels corresponding to each training sample.

[0022] According to a true color cloud image generation method provided by the present invention, the improved Pix2Pix network is trained using the data training set, comprising iteratively performing the following steps based on each training sample until the mean absolute error determined using the validation set is less than a preset threshold:

[0023] Fix the parameters of the generator, and update the parameters of the discriminator by backpropagation using the discriminator loss function according to the label input by the discriminator and the true color cloud image generated by the generator based on the sample data corresponding to the label;

[0024] The parameters of the discriminator are fixed, and the parameters of the generator are updated by back propagation using the generator loss function according to the sample data input by the generator and the labels corresponding to the sample data.

[0025] According to a true color cloud image generation method provided by the present invention, the discriminator loss function is jointly determined based on the loss function of the discriminator of the Pix2Pix network before improvement and the loss function of the U-Net structure; the generator loss function is jointly determined based on the loss function of the generator of the Pix2Pix network before improvement and the discriminator of the U-Net structure.

[0026] According to a true color cloud image generation method provided by the present invention, the infrared cloud image data includes full disk data of the FY-4A satellite AGRI; the numerical model product data includes ERA5 data.

[0027] According to a true color cloud map generation method provided by the present invention, cloud map data is geographically located, comprising: projecting the cloud map data onto a grid of equal longitude and latitude, and extracting corresponding data from the grid of equal longitude and latitude according to the longitude and latitude information of the area corresponding to the cloud map data;

[0028] Performing radiometric calibration on the cloud image data includes: using the digital quantized value of each pixel in the cloud image of each channel in the cloud image data as an index; extracting the radiometric brightness temperature or reflectivity corresponding to the index of each pixel in the calibration table, and assigning the value to each pixel;

[0029] When the cloud image data includes visible light cloud image data of channels 1 to 3 and infrared cloud image data of channels 7 to 14 of the FY-4A satellite AGRI, performing visible light channel reflectance correction on the cloud image data, including: correcting the reflectance of the visible light cloud image data of channels 1 to 3 after radiometric calibration to the apparent reflectance;

[0030] The calculation formula for data normalization is:

[0031]

[0032] Among them, x is the data value before normalization; x * is the normalized data value; for the visible light cloud image data of channels 1 to 3, the max value is 1 and the min value is 0; for the infrared cloud image data of channels 7 to 14, the max value is 350 and the min value is 0; for any level of data in the ERA5 data, the max value is the maximum value of the level data, and the min value is the minimum value of the level data.

[0033] In a second aspect, the present invention further provides a true color cloud image generating device, comprising:

[0034] Data acquisition unit, used to obtain infrared cloud image data and numerical model product data of the target area at the same time;

[0035] A data preprocessing unit, configured to perform geolocation, radiometric calibration, and data normalization on the infrared cloud image data, and to perform normalization on the numerical model product data;

[0036] a cloud image generation unit, configured to input the normalized infrared cloud image data and the normalized numerical model product data into a true color cloud image generation model, so as to obtain a true color cloud image of a corresponding time output by the true color cloud image generation model;

[0037] The true color cloud image generation model is generated by training the improved Pix2Pix network using a pre-built data training set.

[0038] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating a true color cloud image as described above is implemented.

[0039] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for generating a true color cloud image.

[0040] The true-color cloud map generation method, device, electronic device, and storage medium provided by the present invention, after reasonable model structure design, propose a true-color cloud map generation model based on an improved Pix2Pix network. While keeping the overall structure of the model unchanged, it is suitable for generating true-color cloud maps of any size, solving the problem of missing visible light channel data at night. Experimental verification and quantitative indicator evaluation show that its MAE and RMSE indicators are superior to existing simulation cloud map generation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 It is a flow chart of the true color cloud image generation method provided by the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the true color cloud image generation model of the original Pix2Pix network;

[0044] Figure 3 Schematic diagram of the structure of the true color cloud image generation model based on the improved Pix2Pix network provided by the present invention;

[0045] Figure 4 This is a schematic diagram of the process of generating an all-weather typhoon true color cloud map provided by the present invention;

[0046] Figure 5 It is a structural schematic diagram of the true color cloud image generating device provided by the present invention;

[0047] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0049] It should be noted that, in the description of the embodiments of the present invention, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device comprising the elements. The orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0050] The terms "first," "second," and the like in this application are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, the objects distinguished by "first," "second," and the like generally refer to a class of objects and do not limit the number of objects. For example, the first object may be one or more.

[0051] Researchers have made significant progress in the simulation and generation of nighttime visible-light true-color cloud images (also known as true-color cloud images). The key approach to this is to utilize meteorological products such as satellite infrared cloud images, which are available at night and cover the same region, to establish a mapping relationship and generate the corresponding true-color cloud images. This approach eliminates the need for the model to extract temporal information and can simulate true-color cloud images at any time of the night.

[0052] In 2021, Cheng Wencong and others proposed a method for simulating satellite cloud imagery based on generative adversarial networks (GANs) and numerical model products. This method simulates 12-channel infrared cloud images from the FY-4A satellite and fused cloud images from 1, 2, and 3 channels of visible light during the day and at night.

[0053] In the same year, Cheng Wencong further proposed a GAN-based method for simulating and generating true-color cloud images from nighttime meteorological satellites. The satellite infrared channel cloud image data and numerical model products (ERA5 data) of the same time and region were input into the GAN model to generate nighttime visible light simulation cloud images.

[0054] However, the discriminator in the existing GAN model is a typical convolutional classification network, which outputs a scalar representing the global true and false information of the image. The convolutional layer setting of the discriminator network is related to the overall size of the image, which makes it impossible to conveniently switch the size of the simulated cloud map. Once the size of the simulated cloud map changes, the discriminator network structure needs to be redesigned. The model cannot be flexibly applied to the generation of simulated cloud maps of any geographical range and does not have scalability.

[0055] In order to completely or partially overcome the deficiencies of the existing methods, the present invention makes improvements on the existing technology and provides a new true color cloud image generation method.

[0056] The following combination Figures 1-6 The present invention describes a method, device, electronic device, and storage medium for generating a true color cloud image provided by an embodiment of the present invention.

[0057] Figure 1 This is a flow chart of the true color cloud image generation method provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps:

[0058] Step 101: Obtain infrared cloud image data and numerical model product data of the target area at the same time.

[0059] As an optional embodiment, the infrared cloud image data mainly includes the full disk data of the FY-4A satellite AGRI, specifically the 4km resolution full disk data of the FY-4A satellite AGRI, especially the infrared cloud image data of channels 7-14. The numerical model product data mainly includes ERA5 data.

[0060] Specifically, AGRI is one of the main payloads of the FY-4A satellite, and the present invention mainly utilizes its 4km resolution full disk cloud image data.

[0061] AGRI has 14 detection bands. Channels 1, 2, and 3 are visible light channels, capable of detecting true-color cloud images only during the day. These can be used as label data for the corresponding time periods when constructing training data sets. Channels 7 through 14 are medium- and long-wave infrared channels, capable of detecting infrared cloud image data both day and night. These can be used as sample data for the corresponding time periods when constructing training data sets.

[0062] Specifically, when constructing the data training set, any 7- to 14-channel infrared cloud image data collected at a certain time and the numerical model product data at that time can be used as one of the sample data, and the 1-, 2-, and 3-channel true color cloud images collected at that time can be used as labels to generate a training sample.

[0063] By adopting the above method, by collecting the above data at different times in a certain research area, multiple training samples can be generated to form a data training set.

[0064] The detailed information of the full disk data of AGRI used is shown in Table 1:

[0065] Table 1 AGRI data list

[0066]

[0067]

[0068] Furthermore, the ERA5 data used in the present invention refers to the fifth generation global atmospheric reanalysis data (ECMWF Reanalysis V5, ERA5) of the European Center for Medium-Range Weather Forecasts (ECMWF), which is currently the most widely used numerical weather prediction (NWP) product.

[0069] As an optional embodiment, the present invention comprehensively considers the types and hierarchical characteristics of ERA5 data required for true color cloud image simulation and selects the ERA5 data shown in Table 2 together with the infrared cloud image data of the same time as the input data of the true color cloud image generation model:

[0070] Table 2 ERA5 data list

[0071]

[0072]

[0073] It should be noted that when the present invention generates true color cloud maps, it can analyze the infrared cloud map data and numerical model product data at any time of the day to obtain the true color cloud map at each time. Finally, by splicing the true color cloud maps of all times in time sequence, the true color cloud map of all weather conditions can be obtained.

[0074] Step 102: geo-locating, radiometrically calibrating, and normalizing the infrared cloud image data, and normalizing the numerical model product data.

[0075] As an optional embodiment, geo-locating the infrared cloud image data mainly includes: projecting the infrared cloud image data onto an equal longitude and latitude grid, and extracting corresponding data from the equal longitude and latitude grid according to the longitude and latitude information of the area corresponding to the infrared cloud image data.

[0076] Optionally, the infrared cloud image data is subjected to radiation calibration, including: using the digital quantization value (DN value) of each pixel in the infrared cloud image of each channel in the infrared cloud image data as an index; extracting the radiation brightness temperature corresponding to the index of each pixel in the calibration table to assign a value to each pixel.

[0077] Optionally, the calculation formula for the above data normalization is:

[0078]

[0079] Among them, x is the data value before normalization; x * is the normalized data value; for the visible light cloud image data of channels 1 to 3, the max value is 1 and the min value is 0; for the infrared cloud image data of channels 7 to 14, the max value is 350 and the min value is 0; for any level of data in the ERA5 data, the max value is the maximum value of the level data, and the min value is the minimum value of the level data.

[0080] Before using the true color cloud image generation model to identify infrared cloud image data and numerical model product data, the present invention needs to pre-process the input data according to the recognition requirements of the model, mainly including:

[0081] (1) Geographic Positioning

[0082] The full-disk data can be projected onto a grid of equal latitude and longitude using the conversion formula between row and column numbers provided on the official website of the National Satellite Meteorological Center. Then, based on the latitude and longitude range of the target area, the corresponding data on the latitude and longitude grid is extracted to achieve geolocation of the cloud image data for each AGRI channel.

[0083] (2) Radiation calibration

[0084] The digital quantization value (DN value) in each channel of the AGRI cloud map data (NOMChannel xx, xx is the channel name) is used as the index, and the radiation brightness temperature or reflectivity corresponding to this index position is extracted from the calibration table (CALChannel xx, xx is the channel name) and assigned to the corresponding position of the cloud map data.

[0085] It should be noted that the present invention calibrates the visible light channel 1, 2, and 3 data in the AGRI cloud image data as reflectivity, and calibrates the infrared channel 7 to 14 data as radiation brightness temperature.

[0086] (3) Visible light channel reflectivity correction

[0087] For a fixed geographical area on Earth, there will be certain differences in brightness and darkness at different times due to the different distances between the Sun and the Earth and the solar zenith angle. This will interfere with the training process of the true-color cloud image generation model, and will also lead to blurred concepts of time and physical meanings during normal cloud image simulations, resulting in brightness and darkness changes at night that should not exist.

[0088] In order to eliminate the brightness and darkness differences of the label data (i.e., the true color cloud image) at different times during model training, the reflectance of the AGRI cloud image data channels 1, 2, and 3 after calibration is corrected to the apparent reflectance. The calculation formula is:

[0089]

[0090] Where Ref is the reflectivity after radiometric calibration of channels 1, 2, and 3, ρ is the corresponding apparent reflectivity, and D ES is the distance between the Sun and the Earth, and u is the solar zenith angle.

[0091] After reflectivity correction, the brightness of each pixel in the true color cloud image used as a label and the true color cloud image obtained by simulation is equivalent to the brightness under direct sunlight at noon. The image is brighter, has higher contrast, and is richer in information.

[0092] It should be noted that the present invention only requires the visible light channel reflectance correction of the true color cloud image used as the training label of the improved Pix2Pix network.

[0093] (4) Data normalization

[0094] The data used in the present invention to generate true color cloud maps mainly include AGRI cloud map data and ERA5 data. The above data include various physical quantities with different value ranges. To prevent the range differences between the data from affecting the improved Pix2Pix network training process, the present invention performs normalization operations on the AGRI cloud map data and ERA5 data in the data preprocessing stage. The calculation method is as follows:

[0095]

[0096] Among them, x is the data value before normalization; x * The data values are normalized. For visible light cloud data in channels 1 to 3 of the AGRI cloud data, the max value is 1 and the min value is 0. For infrared cloud data in channels 7 to 14, the max value is 350 and the min value is 0. For any layer of data in the ERA5 data, the max value is the maximum value of the layer, and the min value is the minimum value of the layer. After normalization, all data are mapped to the range [0, 1].

[0097] As an optional embodiment, during the construction of the training data set, after normalizing the AGRI cloud data and ERA5 data related to all training samples, the normalized data can be further cleaned to filter out data suitable for model training. The specific filtering conditions are mainly two points: first, each sample data must not contain missing values or invalid fill values to avoid interference with model training due to these incomplete data; second, the sample data must be daytime data in the geographical area where it is located.

[0098] Among them, the method of filtering out daytime data is:

[0099] According to the time zone of the longitude of the sampling area where the AGRI cloud map data and ERA5 data are obtained, the hourly sample data from 10:00 to 15:00 in this time zone are screened and extracted from all sample data, and the data training set is constructed from all the screened and extracted sample data.

[0100] The true-color cloud image generation method provided by the present invention corrects the reflectance of the visible light channel used for label data (referred to as the labeled true-color cloud image) during the data preprocessing stage and shortens the daytime interval extracted from the training dataset. This eliminates the brightness differences between the labeled true-color cloud image at different times. This ensures that the brightness of each pixel in the true-color cloud image and the labeled true-color cloud image output by the model is equivalent to the brightness under direct sunlight at noon. This unifies the temporal concept of the cloud image and prevents unnecessary cloud image changes in the nighttime simulated cloud image. In particular, the output nighttime true-color cloud image is bright, high-contrast, rich in information, with complete land information and clear land edges.

[0101] Step 103: Input the normalized infrared cloud image data and the normalized numerical model product data into a true color cloud image generation model to obtain a true color cloud image of a corresponding time output by the true color cloud image generation model.

[0102] The true color cloud image generation model is generated by training the improved Pix2Pix network using a pre-built data training set.

[0103] The original Pix2Pix network, as an important variant of the GAN network, is another Image-to-Image Translation similar to the CycleGAN network, also known as an image translation (conversion) network.

[0104] Figure 2 This is a schematic diagram of the structure of the true color cloud image generation model of the original Pix2Pix network. When using the original Pix2Pix network to generate simulated cloud images, the main defects are as follows:

[0105] First, the resolution of ERA5 data is lower than that of cloud image data, so ERA5 data and infrared cloud image data cannot be directly used as the input of the original Pix2Pix network.

[0106] Secondly, the existing Pix2Pix network extracts and learns the channel features and spatial features of the input data indiscriminately. When the number of input data channels increases or the spatial size increases, it cannot ensure that the model can accurately extract important features and suppress redundant features, which is not conducive to model convergence.

[0107] In addition, although the discriminator of the existing Pix2Pix network can identify the authenticity of local blocks of the image, its accuracy is still insufficient for the details of the cloud map, and it cannot achieve pixel-by-pixel authenticity identification of the generated image.

[0108] In order to overcome the shortcomings of the existing technology, the true color cloud map generation method provided by the present invention improves the original Pix2Pix network, and obtains a true color cloud map generation model after training the improved Pix2Pix network. The true color cloud map generation model is used to extract features from the input preprocessed infrared cloud map data and numerical model product data, and output the corresponding true color cloud map.

[0109] Figure 3 : is a structural diagram of a true color cloud image generation model based on an improved Pix2Pix network provided by the present invention. As an optional embodiment, the improved Pix2Pix network provided by the present invention specifically includes:

[0110] An upsampling module for the numerical model product data is added to the input ends of the generator and the discriminator of the original Pix2Pix network;

[0111] The structure of the discriminator is set to a U-Net structure including a downsampling module and an upsampling module;

[0112] Adding a convolutional attention module to the generator;

[0113] All convolutional layers of the improved Pix2Pix network are set as depthwise separable convolutional layers.

[0114] The true-color cloud image generation method provided by this invention uses the Pix2Pix network as its foundational framework. By improving and pre-training the original Pix2Pix network, a true-color cloud image generation model is obtained. During pre-training, the sample data input to the true-color cloud image generation model can be infrared cloud images from AGRI channels 7 to 14 and ERA5 data from the same time period, with labels representing visible light cloud images from channels 1 to 3 of the corresponding AGRI time period (i.e., true-color cloud images).

[0115] Specifically, the improvements made by this invention to the original Pix2Pix network mainly include but are not limited to the following four aspects:

[0116] (1) At the input of the generator and discriminator, an upsampling module for ERA5 data is introduced;

[0117] (2) Referring to the structure of U-Net GAN, the discriminator is improved to the U-Net structure to simultaneously improve the generation effect of the global and detailed simulation cloud map;

[0118] (3) Introducing the Convolutional Block Attention Module (CBAM) into the generator to enable the model to capture important channel and spatial features;

[0119] (4) All traditional convolutional layers in the model are replaced with depthwise separable convolution (DSC) layers to reduce the number of model parameters, improve training speed, and prevent overfitting.

[0120] The original Pix2Pix network uses PatchGAN as the discriminator. One benefit is that the model can handle images of any size. This is because unlike the discriminator of traditional GAN models, which outputs a scalar representing the global true / false information of an image, PatchGAN outputs a matrix where each element represents the local true / false information of each small region in the image, independent of the overall image size. This gives the Pix2Pix network tremendous flexibility, allowing it to generate images of any size given sufficient computing resources.

[0121] This paper uses the Pix2Pix network as its foundational model, improving upon the original Pix2Pix network. The improved U-Net-based discriminator and the generator, which incorporates a convolutional attention module (CBAM), can both operate on cloud maps of any size. The overall structure of the model remains unchanged when generating true-color cloud maps of varying sizes. The only adjustment required is the upsampling module for ERA5 data (based on the relationship between the input ERA5 data size and the infrared cloud map size, and the output true-color cloud map).

[0122] The present invention provides a method for adjusting the upsampling module of ERA5 data according to the size requirement of outputting true color cloud images of corresponding size:

[0123] Since the ERA5 data upsampling module consists of several layers of transposed convolutional layers, it is necessary to reasonably design the convolution kernel and the number of layers of the transposed convolutional layer in the ERA5 upsampling module based on the relationship between the transposed convolution output image and the input image size. The specific calculation formula is:

[0124] H out =(H in -1)×stride-2×padding+dilation×(kernel_size-1)+output_padding+1;

[0125] Among them H in is the size of ERA5 data before upsampling; H out is the size of the cloud data, that is, the target size of the ERA5 data that we hope to obtain through upsampling. stride, padding, dilation, kernel_size, and output_padding are all transposed convolution parameters.

[0126] By replacing the ERA5 upsampling modules at the front end of the generator and discriminator in the model, we have completed the partial construction of the improved Pix2Pix network that can generate cloud maps of other sizes.

[0127] In summary, the improved Pix2Pix network provided by the present invention can be applied to the generation of true color cloud images of any size.

[0128] Figure 4 This is a flow chart of generating an all-weather typhoon true color cloud map provided by the present invention, such as Figure 4As shown, the true color cloud map generation method provided by the present invention can be applied to the simulation generation of all-weather satellite true color cloud maps. By integrating the ERA5 upsampling module that can participate in model training, the discriminator of the U-Net structure, the convolutional attention mechanism, the depth-separable convolution and the original Pix2Pix network, after reasonable model structure design, a network based on the improved Pix2Pix is proposed, and then the true color cloud map generation model is obtained by fully pre-training the improved Pix2Pix network. The constructed true color cloud map generation model can better realize the simulation generation of all-day true color cloud maps, and solve the problem of missing data of the AGRI night visible light channel. After experimental verification and quantitative index evaluation, the MAE and RMSE indicators of the true color cloud map generated by the simulation of the present invention are better than the current mainstream simulation methods.

[0129] like Figure 4 As shown, the method for generating all-weather typhoon true color cloud images provided by the present invention, to obtain a true color cloud image generation model, first needs to construct a data training set, and pre-train the improved Pix2Pix network on the constructed data training set, including:

[0130] First, select the target area for all-weather typhoon true color cloud image monitoring based on the scope of the geographical area of concern;

[0131] Then, sample data were collected, including 4 km resolution full disk data from the FY-4A satellite AGRI and ERA5 data, the fifth generation reanalysis field product of the European Centre for Medium-Range Weather Forecasts.

[0132] Furthermore, the method provided in the above embodiment is used to pre-process the collected data in sequence, including geographic positioning, radiometric calibration, visible light channel reflectance correction of the AGRI cloud map data, and normalization operations of the AGRI cloud map data and ERA5 data.

[0133] Because the true-color cloud image generation model is pre-trained using a supervised training method, real true-color cloud images are required as training ground truth (labels). Since true-color cloud images are only available during the day and not at night, only daytime data can be used to construct the training data set. Therefore, data cleaning is required on the normalized AGRI cloud image data and ERA5 data. This primarily involves removing data containing missing values or invalid fill values and filtering out daytime data in the target area (e.g., hourly data from 10:00 AM to 3:00 PM). This ensures that all training samples and labels corresponding to each training sample are available for model training. The improved Pix2Pix network is iteratively trained using these training samples until the model training results converge (i.e., the mean absolute error determined using the validation set is less than a preset threshold).

[0134] Since the improved Pix2Pix network includes a generator and a discriminator, during model pre-training, the following steps are iteratively performed based on each training sample until the mean absolute error determined using the validation set is less than a preset threshold:

[0135] Fix the parameters of the generator, and update the parameters of the discriminator by backpropagation using the discriminator loss function according to the labels input by the discriminator and the true color cloud image generated by the generator based on the sample data corresponding to the generated labels;

[0136] The parameters of the discriminator are fixed, and the parameters of the generator are updated by back propagation using the generator loss function according to the sample data input by the generator and the labels corresponding to the sample data.

[0137] Specifically, the Adam optimizer is used for model pre-training. The parameters of the generator and discriminator can be updated alternately, with the learning rate set to 0.0002 and the momentum parameters β1 = 0.5 and β2 = 0.999.

[0138] It should be emphasized that when updating the parameters of the discriminator, the parameters of the generator must be fixed, and the gradient is calculated using the discriminator's loss function and back propagation is performed; when updating the parameters of the generator, the parameters of the discriminator must be fixed, and the gradient is calculated using the generator's loss function and back propagation is performed.

[0139] The batch size of the input data for each parameter update is set to 16. The mean absolute error (MAE) on the validation set is used as an indicator during the training process, and the model with the smallest MAE on the validation set is saved as the final true color cloud image generation model.

[0140] Finally, by collecting infrared cloud image data and ERA5 data continuously during the typhoon period in the target area, preprocessing the data and inputting it into the trained true color cloud image generation model, all-weather true color cloud image monitoring of typhoons in any geographical area can be achieved.

[0141] The present invention combines the loss function of the original Pix2Pix network with the loss function of the U-Net GAN network for gradient calculation and parameter update.

[0142] Based on the contents of the above embodiments, as an optional embodiment, the discriminator loss function of the improved Pix2Pix network used in the true color cloud image generation method provided by the present invention is jointly determined based on the loss function of the discriminator of the Pix2Pix network before the improvement and the loss function of the U-Net structure; the generator loss function is jointly determined based on the loss function of the generator of the Pix2Pix network before the improvement and the discriminator of the U-Net structure.

[0143] Specifically, the loss function of the original Pix2Pix network is combined with the loss function of the U-Net GAN network to symbolize D U Represents the discriminator based on the U-Net structure, where the encoder part of the discriminator is represented by D enc U , the decoder part is represented by D dec U , as shown below, the discriminator loss function is set to the sum of the encoder and decoder loss functions:

[0144]

[0145] Among them, the loss functions of the encoder and decoder are as follows:

[0146]

[0147]

[0148] The loss function of the generator is:

[0149]

[0150] in, and Represents the true or false judgment of the discriminator for each pixel (i, j) of the input image, The L1 distance between the true color cloud image generated by the generator and the true value of the true color cloud image of the corresponding label.

[0151] Optionally, λ in the above formula may be set to 100.

[0152] The true-color cloud image generation method provided by this invention applies a true-color cloud image generation model to all-weather typhoon monitoring, enabling continuous, all-day observation of typhoons. This demonstrates the value of nighttime true-color cloud image simulation for real-time monitoring of weather systems, weather forecasting, and disaster prevention and mitigation. Because the true-color cloud image generation model employed is based on an improved Pix2Pix network, it is applicable to cloud image generation for any size. While maintaining the overall structure of the model, it enables the generation of simulated true-color cloud images for larger areas. This allows for all-weather true-color cloud image monitoring of typhoons moving over a large area or the interaction of multiple typhoon weather systems, providing a framework for real-time, daytime, and nighttime observation of full-disk visible light remote sensing images.

[0153] Observations show that the cloud simulation effect of the improved true-color cloud image generation model is significantly improved compared to that of the original Pix2Pix model. The distortion of cloud details in the daytime simulated cloud images has been resolved. The land area information of the nighttime simulated cloud images is complete, the contrast with the ocean area is improved, and the cloud details are richer. The quantitative indicators are shown in Table 3:

[0154] Table 3 Quantitative comparison table

[0155] index MSE RMSE MAE PSNR(dB) True color cloud image generation model 0.005 0.0683 0.0454 23.6677 Original Pix2Pix 0.0072 0.0768 0.0516 22.9864

[0156] Compared with the cloud map generation model based on the original Pix2Pix network, the various indicators of the true color cloud map generation model provided by the present invention are significantly improved, among which the MSE, RMSE, and MAE are reduced by 30.56%, 11.07%, and 12.02% respectively, and the PSNR is increased by 0.6813dB.

[0157] The above experimental phenomena fully verify that the improved true color cloud image generation model has better performance than the original Pix2Pix network in the simulation generation of true color cloud images.

[0158] Figure 5 This is a schematic diagram of the structure of the true color cloud image generating device provided by the present invention. Figure 5 As shown, it mainly includes a data acquisition unit 51, a data pre-processing unit 52 and a cloud map generation unit 53, wherein:

[0159] The data acquisition unit 51 is mainly used to obtain infrared cloud image data and numerical model product data of the target area at the same time;

[0160] The data pre-processing unit 52 is mainly used to perform geolocation, radiometric calibration and data normalization on the infrared cloud image data, and to perform normalization on the numerical model product data;

[0161] The cloud map generation unit 53 is mainly used to input the normalized infrared cloud map data and the normalized numerical model product data into the true color cloud map generation model to obtain the true color cloud map of the corresponding time output by the true color cloud map generation model.

[0162] The true color cloud image generation model is generated by training the improved Pix2Pix network using a pre-built data training set.

[0163] It should be noted that the true color cloud image generation device provided in the embodiment of the present invention can execute the true color cloud image generation method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0164] The true-color cloud map generation method provided by the present invention, through reasonable model structure design, proposes a true-color cloud map generation model based on an improved Pix2Pix network. While maintaining the overall structure of the model, it is applicable to the generation of true-color cloud maps of any size, solving the problem of missing visible light channel data at night. Experimental verification and quantitative indicator evaluation show that its MAE and RMSE indicators are superior to existing simulation cloud map generation methods.

[0165] Figure 6 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communications bus 640. The processor 610 may call logic instructions in the memory 630 to execute a true color cloud image generation method, which includes: obtaining infrared cloud image data and numerical model product data of a target area at the same time; performing geolocation, radiometric calibration, and data normalization on the infrared cloud image data, and normalizing the numerical model product data; inputting the normalized infrared cloud image data and the normalized numerical model product data into a true color cloud image generation model to obtain a true color cloud image of the corresponding time output by the true color cloud image generation model; the true color cloud image generation model is generated by training an improved Pix2Pix network using a pre-constructed data training set.

[0166] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the true color cloud map generation method provided by the above methods, which includes: obtaining infrared cloud map data and numerical model product data of the target area at the same time; performing geolocation, radiation calibration and data normalization processing on the infrared cloud map data, and normalizing the numerical model product data; inputting the normalized infrared cloud map data and the normalized numerical model product data into a true color cloud map generation model to obtain a true color cloud map of the corresponding time output by the true color cloud map generation model; the true color cloud map generation model is generated after training the improved Pix2Pix network using a pre-constructed data training set.

[0168] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the true color cloud map generation method provided in the above embodiments, the method comprising: obtaining infrared cloud map data and numerical model product data of the target area at the same time; performing geolocation, radiation calibration and data normalization processing on the infrared cloud map data, and normalizing the numerical model product data; inputting the normalized infrared cloud map data and the normalized numerical model product data into a true color cloud map generation model to obtain a true color cloud map of the corresponding time output by the true color cloud map generation model; the true color cloud map generation model is generated after training the improved Pix2Pix network using a pre-constructed data training set.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0170] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A true color cloud image generation method, characterized in that: include: Obtain infrared cloud image data and numerical model product data of the target area at the same time; Performing geolocation, radiometric calibration, and data normalization processing on the infrared cloud image data, and performing normalization processing on the numerical model product data; Inputting the processed infrared cloud image data and the processed numerical model product data into a true color cloud image generation model to obtain a true color cloud image of a corresponding time output by the true color cloud image generation model; The true color cloud image generation model is generated by training the improved Pix2Pix network using a pre-built data training set; The improved Pix2Pix network specifically includes: An upsampling module for the numerical model product data is added to the input ends of the generator and the discriminator of the original Pix2Pix network, and the upsampling module is composed of multiple transposed convolutional layers; The structure of the discriminator is set to a U-Net structure including a downsampling module and an upsampling module; Adding a convolutional attention module to the generator; All convolutional layers of the improved Pix2Pix network are set as depthwise separable convolutional layers.

2. The true color cloud image generation method according to claim 1, characterized in that: The improved Pix2Pix network is trained using a pre-built data training set, including: Acquire multiple training samples; the sample data of each training sample includes infrared cloud image data and numerical model product data of the same time and the same area, and the label of each training sample is a true color cloud image of the corresponding time and the corresponding area; Performing geolocation, radiometric calibration, visible light channel reflectance correction, and data normalization processing on the infrared cloud image data in each sample data and the true color cloud image in the corresponding label, and performing data normalization processing on the numerical model product data in each sample data; Performing data cleaning on all training samples, and constructing the data training set using all the cleaned training samples, so as to train the improved Pix2Pix network using the data training set; The data cleaning of all training samples includes deleting some training samples with missing values or invalid filling values in all training samples, and then screening out all training samples that are within the preset daytime period; The improved Pix2Pix network is trained iteratively using all training samples after data cleaning and the labels corresponding to each training sample.

3. The true color cloud image generation method according to claim 2, characterized in that: The improved Pix2Pix network is trained using the data training set, including iteratively performing the following steps based on each training sample until the mean absolute error determined using the validation set is less than a preset threshold: Fix the parameters of the generator, and update the parameters of the discriminator by backpropagation using the discriminator loss function according to the label input by the discriminator and the true color cloud image generated by the generator based on the sample data corresponding to the label; The parameters of the discriminator are fixed, and the parameters of the generator are updated by back propagation using the generator loss function according to the sample data input by the generator and the labels corresponding to the sample data.

4. The true color cloud image generation method according to claim 3, characterized in that: The discriminator loss function is jointly determined based on the loss function of the discriminator of the Pix2Pix network before improvement and the loss function of the U-Net structure discriminator; the generator loss function is jointly determined based on the loss function of the generator of the Pix2Pix network before improvement and the discriminator of the U-Net structure.

5. The true color cloud image generation method according to any one of claims 1 to 4, characterized in that: The infrared cloud image data includes the full disk data of the FY-4A satellite AGRI; the numerical model product data includes ERA5 data.

6. The true color cloud image generation method according to claim 5, characterized in that: Geolocating the cloud image data includes: projecting the cloud image data onto a grid of equal longitude and latitude, and extracting corresponding data from the grid of equal longitude and latitude according to the longitude and latitude information of the area corresponding to the cloud image data; Performing radiometric calibration on the cloud image data includes: using the digital quantized value of each pixel in the cloud image of each channel in the cloud image data as an index; extracting the radiometric brightness temperature or reflectivity corresponding to the index of each pixel in the calibration table, and assigning the value to each pixel; When the cloud image data includes visible light cloud image data of channels 1 to 3 and infrared cloud image data of channels 7 to 14 of the FY-4A satellite AGRI, performing visible light channel reflectance correction on the cloud image data, including: correcting the reflectance of the visible light cloud image data of channels 1 to 3 after radiometric calibration to the apparent reflectance; The calculation formula for data normalization is: Among them, x is the data value before normalization; x * is the normalized data value; for the visible light cloud image data of channels 1 to 3, the max value is 1 and the min value is 0; for the infrared cloud image data of channels 7 to 14, the max value is 350 and the min value is 0; for any level of data in the ERA5 data, the max value is the maximum value of the level data, and the min value is the minimum value of the level data.

7. A true color cloud image generation device, characterized in that: include: Data acquisition unit, used to obtain infrared cloud image data and numerical model product data of the target area at the same time; A data preprocessing unit, configured to perform geolocation, radiometric calibration, and data normalization on the infrared cloud image data, and to perform normalization on the numerical model product data; a cloud image generation unit, configured to input the normalized infrared cloud image data and the normalized numerical model product data into a true color cloud image generation model, so as to obtain a true color cloud image of a corresponding time output by the true color cloud image generation model; The true color cloud image generation model is generated by training an improved Pix2Pix network using a pre-built data training set; the improved Pix2Pix network specifically includes: An upsampling module for the numerical model product data is added to the input ends of the generator and the discriminator of the original Pix2Pix network, and the upsampling module is composed of multiple transposed convolutional layers; The structure of the discriminator is set to a U-Net structure including a downsampling module and an upsampling module; Adding a convolutional attention module to the generator; All convolutional layers of the improved Pix2Pix network are set as depthwise separable convolutional layers.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the true color cloud image generating method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the true color cloud image generation method according to any one of claims 1 to 6 is implemented.

Images