Surface temperature space-time fusion method and system based on conditional generative adversarial network

Abstract

The invention belongs to the field of meteorological data processing, and provides a surface temperature space-time fusion method and system based on a conditional generative adversarial network, and the method only needs to predict a coarse-resolution image of a date and another fine-resolution reference image at any time in the same region for model input, and does not limit the reference acquisition time. Therefore, the influence of time inconsistency on a prediction result can be greatly reduced, and meanwhile, a multi-scale mechanism is integrated to adapt to the input of reference image pairs with different resolutions. From the perspective of quantitative indexes and visual evaluation, the CGAN-LSTFM has robustness for reference images at different moments and resolution ratios of different scales, and meanwhile, due to less input and limitation, the model also has more flexibility.

Description

Technical Field

[0001] The present invention belongs to the field of meteorological data processing, and in particular to a method and system for spatiotemporal fusion of surface temperature based on a conditional generative adversarial network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As one of the key indicators of the earth's climate system, surface temperature is of great significance for ecological monitoring, urban heat island effect, climate change and other research. Monitoring surface temperature from high altitudes through remote sensing technology helps track and analyze climate warming, urban heat island effect and other environmental issues. This monitoring not only helps to scientifically plan climate change adaptation and mitigation measures, but also improves the management efficiency of agricultural planting patterns and urban planning. Accurate acquisition of surface temperature information has a far-reaching impact on promoting sustainable development of mankind and improving the quality of life.

[0004] In view of the problem that it is difficult to have high spatial and temporal resolution for surface temperature, spatiotemporal fusion technology provides a feasible solution. Spatiotemporal fusion of remote sensing images refers to the technology of synthesizing dense temporal images with high spatial resolution from at least two different data sources. In most cases, one is an image with high temporal resolution but low spatial resolution, while the other is an image with low temporal resolution but high spatial resolution.

[0005] With the rapid development of artificial intelligence, deep learning is being used more and more widely in various fields. It is an algorithm that uses artificial neural networks as the framework to learn hierarchical representations of input data. These hierarchical representations are generally learned based on neural networks. This learning process is a process of mapping input data to targets by learning from observed instances. For each layer of data changes in the neural network, a set of the most appropriate weight values ​​is found so that the network has the ability to establish a correct correspondence between input data and output targets. Therefore, spatiotemporal fusion of surface temperature based on deep learning methods has become a new idea. The spatiotemporal fusion methods applied to surface temperature data can be divided into two categories: weight function-based methods and learning-based methods.

[0006] The weight function-based method designs a weight function to assign weights to data points in different data sources, where the weights are calculated based on the temporal and spatial proximity, quality, and environmental factors of the data. Using the designed weight function, multi-source data can be effectively fused through weighted averaging or advanced data fusion techniques. However, the weight function-based method relies on the accurate design and parameter setting of the weight function, which requires a deep understanding and precise control of the temporal and spatial properties and quality of the data. In addition, the subjectivity of weight assignment may lead to instability of the results, especially when data is missing or of uneven quality. The weight function method may also face the problem of high computational complexity, especially when large-scale data sets are involved, which will increase processing time and resource consumption.

[0007] Learning-based methods use learning algorithms to establish the relationship between observed high- and low-resolution image pairs, and then predict unobserved fine images. These methods are based on the strong nonlinear representation ability between input and output images, which is the main advantage of this method compared to other methods. It can be roughly divided into dictionary learning-based and deep learning-based methods. Dictionary learning-based methods mainly establish the correspondence between high- and low-resolution images to predict high-resolution images under the prediction date.

[0008] Most spatiotemporal fusion models require two pairs of high- and low-resolution surface temperature images as references, and strict constraints must be applied to the reference image selection to ensure fusion accuracy. However, due to the inconsistency of the sensor time system, it is difficult to obtain a completely time-consistent reference image pair. No effective solution has been proposed for the problems in related technologies. Summary of the invention

[0009] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a method and system for spatiotemporal fusion of surface temperature based on conditional generative adversarial network, which comprehensively utilizes the spatiotemporal resolution advantages of multiple surface temperature data sources to obtain a fusion product with high spatiotemporal resolution, thereby improving the quality and applicability of the data.

[0010] In order to achieve the above object, the present invention adopts the following technical solution:

[0011] The first aspect of the present invention provides a spatiotemporal fusion method of surface temperature based on a conditional generative adversarial network, comprising the following steps:

[0012] Acquire the surface temperature image of the first spatial resolution at the target time and the second spatial resolution at the reference time;

[0013] The constructed conditional generative adversarial network is trained based on the acquired surface temperature images of the first spatial resolution at the target moment and the second spatial resolution at the reference moment to obtain a trained conditional generative adversarial network; the training process of the conditional generative adversarial network includes:

[0014] Inputting the surface temperature image of the first spatial resolution at the target time into the generator, and obtaining the prediction result of the second spatial resolution at the initial target time;

[0015] The objective function of the generator is to minimize the distribution distance between the initial second spatial resolution prediction result at the target time and the second spatial resolution surface temperature image at the reference time, and the objective function of the discriminator is to maximize the distribution distance, thereby generating the maximum and minimum objective function of the adversarial network.

[0016] The set conditional labels are input into the generator and discriminator to limit the maximum and minimum objective functions of the adversarial network. The network searches along the desired direction to obtain the optimal solution.

[0017] The second spatial resolution prediction result at the target moment is obtained based on the trained conditional generative adversarial network prediction.

[0018] Furthermore, the training process of the generator includes:

[0019] Extracting the common features of the surface temperature images of the first spatial resolution at the target time and the second spatial resolution at the reference time based on the encoder, and concatenating the common features along the channel dimension to obtain the first surface temperature features;

[0020] The first surface temperature feature is input into a decoder for decoding, and restored to the original pixel space through a deconvolution operation to obtain a second surface temperature feature;

[0021] The second surface temperature feature and the surface temperature of the second spatial resolution at the reference moment are used to calculate the content loss function, and the generator is optimized based on the content loss function.

[0022] Furthermore, the encoder includes three parts, each of which is connected by a BN layer, a ReLU layer and a Conv layer, and the output of the first part is used as the input of the next part. The difference is that the Conv layer sequentially uses 3×3Conv 32 layers, 3×3Conv64 layers and 3×3Conv 128 layers;

[0023] The decoder includes three parts, each of which is a BN layer, a ReLU layer and a Deconv layer. The output of the first part is used as the input of the next part. The difference is that the Deconv layer uses 3×3Deconv 128 layers, 3×3Deconv 64 layers, and 3×3Deconv 32 layers in sequence.

[0024] Furthermore, the training process of the discriminator includes:

[0025] The second spatial resolution prediction result at the target time and the first resolution surface temperature at the target time are used as the first set of data, and the first resolution surface temperature at the target time and the second resolution surface temperature at the target time are used as the second set of data;

[0026] The first set of data and the second set of data are respectively mapped to the probability of belonging to the real image, and the ideal output of the mapping of the first set of data is defined as 0, and the ideal output of the mapping of the second set of data is defined as 1;

[0027] The difference between the discriminator's output and the ideal output is used as the discriminator loss to update the discriminator's parameters.

[0028] Furthermore, the additional condition of the generator is the second resolution surface temperature image at the reference time, and the additional condition of the discriminator is the first resolution surface temperature image at the target time.

[0029] Furthermore, the total loss function of the conditional generative adversarial network is:

[0030] L CGAN-LSTFM =L CGAN +L Content +L Feature ,

[0031]

[0032] Among them, L CGAN is the conditional adversarial loss, L Content is the content loss, L Feature represents feature loss, y is the additional condition, D(x|y) is the discriminant result of the second-resolution surface temperature at the target moment under the additional condition, D(G(z|y)) is the discriminant result of the second-resolution prediction generated under the additional condition, D(x) is the discriminant result of the second-resolution surface temperature at the target moment, D(G(z)) is the discriminant result of the second-resolution surface temperature predicted at the target moment, x is the second-resolution surface temperature at the target moment, is the logarithmic expectation of the discriminator’s output for all x, is the expectation of the inverse logarithmic value of the discriminator's output for all generated data; δ is a hyperparameter, represents the generator prediction result, M represents the real label image, N represents the number of pixels in the feature map, and F represents the predicted image features obtained by the encoder. Represents the features extracted from the true labeled images.

[0033] Furthermore, after obtaining the surface temperature images with the first spatial resolution at the target time and the second spatial resolution at the reference time, preprocessing is performed, including radiation calibration, atmospheric correction and cropping.

[0034] The second aspect of the present invention provides a spatiotemporal fusion system of surface temperature based on a conditional generative adversarial network, comprising:

[0035] A data acquisition module, which is used to acquire a surface temperature image with a first spatial resolution at a target time and a second spatial resolution at a reference time;

[0036] The conditional generative adversarial network training module is used to train the constructed conditional generative adversarial network based on the acquired surface temperature images of the first spatial resolution at the target moment and the second spatial resolution at the reference moment to obtain the trained conditional generative adversarial network; wherein the training process of the conditional generative adversarial network includes:

[0037] Inputting the surface temperature image of the first spatial resolution at the target time into the generator, and obtaining the prediction result of the second spatial resolution at the initial target time;

[0038] The objective function of the generator is to minimize the distribution distance between the initial second spatial resolution prediction result at the target time and the second spatial resolution surface temperature image at the reference time, and the objective function of the discriminator is to maximize the distribution distance, thereby generating the maximum and minimum objective function of the adversarial network.

[0039] The set conditional labels are input into the generator and discriminator to limit the maximum and minimum objective functions of the adversarial network. The network searches along the desired direction to obtain the optimal solution.

[0040] The prediction module is used to predict the second spatial resolution prediction result at the target moment based on the trained conditional generative adversarial network.

[0041] A third aspect of the present invention provides a computer-readable storage medium.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the above-mentioned method for spatiotemporal fusion of surface temperature based on a conditional generative adversarial network.

[0043] A fourth aspect of the present invention provides a computer device.

[0044] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the spatiotemporal fusion method of surface temperature based on a conditional generative adversarial network as described above are implemented.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The present invention performs spatiotemporal fusion of surface temperature based on conditional generative adversarial networks to solve the limitation of strict time constraints on the fusion model in the spatiotemporal fusion problem. This method only requires a coarse-resolution image of the prediction date and another fine-resolution reference image at any time in the same area for model input, and there is no restriction on the time of obtaining the reference. This can greatly reduce the impact of temporal inconsistency on the prediction results, and at the same time incorporates a multi-scale mechanism to adapt to the input of reference image pairs with different resolutions. From the perspective of quantitative indicators and visual evaluation, CGAN-LSTFM is robust to reference images at different times and resolutions of different scales. At the same time, due to fewer inputs and restrictions, the model also has more flexibility.

[0047] 2. By combining remote sensing data with different temporal and spatial resolutions, the present invention can effectively improve the accuracy and reliability of surface temperature data in complex meteorological conditions and different types of surface cover. This technology particularly enhances the ability to estimate surface temperature under conditions of cloud cover, smoke interference, and surface heterogeneity, making large-scale, long-time series surface temperature monitoring more universal and practical. In addition, the present invention can also provide solid technical support for fields such as climate change research, agricultural monitoring, and urban planning, and provide a scientific basis for rationally responding to and adapting to climate change.

[0048] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 It is a flow chart of a method for spatiotemporal fusion of surface temperature based on a conditional generative adversarial network provided by an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of a conditional generative adversarial network provided by an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of a prediction result with a spatial resolution of 1 km provided by an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of comparison results of different models provided by an embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of prediction results of input data of different scales provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0056] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0057] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0058] Embodiment 1

[0059] like Figure 1 and Figure 2 As shown, this embodiment provides a spatiotemporal fusion method of surface temperature based on a conditional generative adversarial network, comprising the following steps:

[0060] Step 1: Obtain the surface temperature image and preprocess the surface temperature image;

[0061] The acquired surface temperature images include the surface temperature images of the first spatial resolution at the target time and the surface temperature images of the second spatial resolution at any time in the same area;

[0062] In this embodiment, the first spatial resolution is lower than the second spatial resolution; for example, the first spatial resolution is 4 km and the second spatial resolution is 1 km;

[0063] Among them, the preprocessing of the acquired surface temperature image includes radiometric calibration, atmospheric correction, cropping and other preprocessing of the image;

[0064] Step 2: Generate a prediction result of the second spatial resolution at the target time based on the surface temperature image of the first spatial resolution at the target time, the surface temperature image of the second spatial resolution at any time in the same area, and the generator;

[0065] Specifically include:

[0066] Step 201: Input the surface temperature image of the first spatial resolution at the target time and the surface temperature image of the second spatial resolution at any time in the same area into the encoder to extract common high-level features, and connect them in series along the channel dimension.

[0067] The first spatial resolution image at the target moment is recorded as L1, and the second spatial resolution image at any moment is recorded as H1. The encoder extracts common high-level features from the two images respectively:

[0068] L1 feature =G_Encode(L1),

[0069] H1 feature =G_Encode(H1),

[0070] Among them, G_Encode represents the encoder of the generator.

[0071] The high-level feature L1 feature and H1 feature The new features are concatenated along the channel dimension as the final output of the encoder.

[0072] In this embodiment, the encoder includes three parts, each of which is connected by a BN layer, a ReLU layer and a Conv layer. The output of the first part is used as the input of the next part. The difference is that the Conv layer uses 3×3Conv (output channels are 32), 3×3Conv (output channels are 64) and 3×3Conv (output channels are 128) in sequence;

[0073] Step 202: Input the output of the encoder into the decoder for decoding to obtain a decoding result; after L1 and H1 are extracted by the encoder, rough abstract features L1 and H1 in the high-level feature space are obtained respectively. feature and fine abstract features H1 feature The mixed features of the two are concatenated and input into the decoder, and restored to the original pixel space through deconvolution operation.

[0074] In this embodiment, the decoder includes three parts, each of which is a BN layer, a ReLU layer and a Deconv layer. The output of the first part is used as the input of the next part. The difference is that the Deconv layer uses 3×3Deconv (output channel 128), 3×3Deconv (output channel 64), and 3×3Deconv (output channel 32) in sequence, and the channel change of the input data is opposite to the encoding process;

[0075] Step 203: Input the decoding result to the fully connected layer, output the high-resolution prediction result at the target time, use the prediction result of the generator and the original high-resolution surface temperature at the target time to calculate the generator loss, and feed the loss back to the generator to optimize the performance of the generator;

[0076] Step 3: Use the discriminator to determine whether the generated high-resolution prediction result of the target moment is close to the real image, and optimize the discriminator according to the discriminant loss to make it more discriminative; then optimize the generator so that the generated image is more difficult for the discriminator to recognize. The purpose of the generator is to generate images that can "fool" the discriminator, while the purpose of the discriminator is to try its best to recognize the predicted image generated by the generator. The two are optimized alternately and trained adversarially until the generation adversarial loss is stable. At this point, the performance of the generator is sufficient to generate images that are very different from the real image, and even the already strong discriminator cannot judge it;

[0077] Specifically include:

[0078] Step 301: downsample the high-resolution prediction result at the target moment to different scales, input the downsampled data into the discriminator, obtain a True or False (1 or 0) discrimination result, calculate the discriminator loss based on the discrimination result, pass the loss back to the discriminator, update the discriminator parameters, and optimize the discriminator performance.

[0079] In this embodiment, a constructed discriminator is used for judgment, and the discriminator includes a residual block structure, a fully connected layer and a Sigmod layer;

[0080] The residual block structure includes three parts, each of which is connected by a BN layer, a LeakyReLU layer and a Conv layer. The output of the first part is used as the input of the next part. The difference is that the Conv layer uses 3×3Conv (output channels are 32), 3×3Conv (output channels are 64) and 3×3Conv (output channels are 128) in sequence;

[0081] The discriminator mainly receives two sets of data: one is the predicted high-resolution original surface temperature and the low-resolution surface temperature at the target time from the generator, and the other is the low-resolution surface temperature at the target time and the original high-resolution surface temperature at the target time as conditions to guide the training process. After passing through the convolution layer and the Sigmoid layer, the discriminator maps the two sets of input images to the probability of belonging to the real image (between 0 and 1, 0 means it is not a real image, and 1 means it is a real image). The ideal discriminator can accurately distinguish between the real image and the predicted image, so the ideal output of the first set of images is 0, and the ideal output of the second set is 1. The difference between the discriminator output result and the ideal output is used as the discriminator loss to guide the optimization of the discriminator and improve the discriminator performance. In addition, a multi-scale mechanism is introduced in the discriminator. The conditional input of the discriminator is downsampled to two different scales of 0.5 and 0.25, which is used to train three discriminators with the same architecture at different scales.

[0082] Step 302: After the optimization of the discriminator is completed, continue to optimize the generator model in step 2, and then cycle through the optimization;

[0083] The generator model is optimized through adversarial iterative training of the generator and the discriminator. The generative adversarial network consists of a generator and a discriminator, which are trained in a competitive process. The generator generates fake data similar to real samples, and the discriminator is used to determine the authenticity of the generated data. Through iterative competitive training, the entire system eventually reaches a balanced state. Finally, the generator is used to generate data that is closest to the real sample.

[0084] Common spatiotemporal fusion models require at least three sets of image inputs, and strict time constraints must be applied to the references to ensure fusion accuracy. However, due to bad weather conditions or time inconsistencies between high- and low-resolution data sources, it is not always easy to collect enough data pairs for fine-resolution image sequences in practical applications, which causes some difficulties in practical applications.

[0085] The spatiotemporal fusion method of surface temperature based on conditional generative adversarial network proposed in this invention only needs two sets of images as model input, and there is no restriction on the acquisition time of the reference point. The model only needs this training method to greatly reduce the impact of time inconsistency on the prediction results and improve the robustness of the model to the reference time.

[0086] The specific optimization process includes:

[0087] Step 3021: Input the noise data z (low-resolution surface temperature at the target time) into the generator G, learn the complex mapping function on the real training sample x (real high-resolution surface temperature at the target time), and transform the distribution p of the noise data into z (z) is mapped to the distribution p of the real data data (x); at the same time, the discriminator D is used to distinguish the generated pseudo data G(z) (the high-resolution surface temperature predicted at the target time) from the real training sample x;

[0088] Step 3022: The goal of the generator is to minimize the distribution distance between the pseudo data G(z) and the real training sample x, while the goal of the discriminator is to maximize the distribution between them. The maximum and minimum value function of the generative adversarial network is expressed as follows:

[0089]

[0090] Among them, D(x) is the discriminator's judgment result on the real high-resolution surface temperature at the target time, and D(G(z)) is the discriminator's judgment result on the predicted high-resolution surface temperature at the target time. is the logarithmic expectation of the discriminator’s output for all true samples, is the expectation of the inverse logarithmic value of the discriminator's output for all generated data;

[0091] Among them, its loss function measures the distribution distance through Jensen–Shannon (JS) divergence.

[0092] Without additional constraints on space, the model training stability of generative adversarial networks in practice is weak. The conditional generative adversarial network inputs additional data (the generator additional condition is a high-resolution surface temperature image at the reference time, and the discriminator additional condition is a low-resolution surface temperature image at the target time) as a conditional label into the generator and discriminator, restricting the generated results to drive the network to search for the optimal solution in the desired direction, solving the problem that ordinary generative adversarial networks cannot control the generated data pattern. The restriction can be any type of auxiliary information, such as class labels or data from other modalities. By inputting the conditional data into the discriminator and generator, it is adjusted as an additional input layer.

[0093] Its loss function is expressed as follows:

[0094]

[0095] Where y is the additional condition, D(x|y) is the discriminant result of the real high-resolution surface temperature at the target time under the additional condition, and D(G(z|y)) is the discriminant result of the high-resolution prediction generated under the additional condition;

[0096] The conditional generative adversarial network consists of two parts: the generator and the discriminator. The generator consists of an encoder and a decoder, both of which are built based on the ResNet building block and learn the residual between input and output through skip connections.

[0097] The input of the generator is the FY-4A LST on the predicted date, and another MODIS LST reference image for model input at any time in the same area. The input of the discriminator includes two groups: one group is the predicted high-resolution MODIS LST from the generator and the low-resolution FY-4ALST at the target time, and the other group is the FY-4A LST at the target time and the original MODIS LST at the target time. They are downsampled to different scales and input into the discriminator, followed by a fully connected layer and a Sigmoid layer, and the True or False discrimination result is output. Through iterative adversarial training of the generator and the discriminator, the network reaches the best fitting state, and finally the network can output the most accurate prediction by inputting a set of MODIS-FY LST image pairs. In addition, a multi-scale mechanism is introduced in the discriminator, and the conditional input of the discriminator is downsampled to two different scales of 0.5 and 0.25, which are used to train three discriminators with the same architecture at different scales.

[0098] The loss function of the spatiotemporal fusion method is mainly divided into three parts, namely conditional adversarial loss, content loss and feature loss. The total loss is expressed as:

[0099] L CGAN-LSTFM =L CGAN +L Content +L Feature (3)

[0100] Among them, L CGAN is the conditional adversarial loss, corresponding to formula (2), which is used to measure the performance of the generator and discriminator of the network itself.

[0101] L Content It is the content loss, which represents the pixel error between the predicted result and the actual reference data, and is described as follows:

[0102]

[0103] Among them, δ is a hyperparameter, and here δ is set to 1.0 based on experience; Represents the prediction result of the generator, and M represents the real label image.

[0104] L Feature The feature loss is used to capture perceptual differences and generate high-quality images. The specific approach is to use the well-known pre-trained AutoEncoder model to extract the features of the predicted image and the real image respectively, and calculate the difference loss between the features. The calculation formula is:

[0105]

[0106] Among them, N represents the number of pixels in the feature map, F represents the high-resolution predicted image features at the target moment obtained by the encoder, Represents the features extracted from the real high-resolution image at the target moment.

[0107] Step 303: Use the optimized generator to generate an optimized high-resolution prediction result at the target moment.

[0108] Taking a certain area as the study area, MODIS LST products and FY-4A LST products as experimental data, a spatiotemporal fusion algorithm of land surface temperature without limiting the reference image time is proposed to achieve the spatiotemporal fusion of land surface temperature in the area.

[0109] Obtain MODIS LST products and FY-4A LST products in the test area as experimental data;

[0110] Among them, the MODIS LST product has a temporal resolution of 1 day and a spatial resolution of 1 km; the FY-4A LST product has a temporal resolution of 15 minutes and a spatial resolution of 4 km;

[0111] Perform radiometric calibration, atmospheric correction, cropping and other pre-processing on the image;

[0112] Input the 4km target time FY-4A LST and the 1km reference time MODIS LST into the generator to obtain the 1km target time FY-4A LST result;

[0113] The 1km target time FY-4A LST and 4km target time FY-4A LST are used as one set of data sets, and the 1km target time MODIS LST and 4km target time FY-4A LST are used as another set of data sets to be input into the discriminator for discrimination;

[0114] Continuously optimize the conditions to generate adversarial networks, such as Figure 3 The final prediction result of 1kmFY-4A LST is shown;

[0115] like Figure 4 As shown in the figure, we conducted comparative experiments on different models to verify the influence of reference images on the fusion model and found that the conditional generative adversarial network is the most stable.

[0116] To test the prediction performance of the model based on input data of different scales, the original 1km MODIS LST was upscaled to 4km, 8km and 16km. Three sets of simulated MODIS-FY LST image pairs of different scales were established and input into the network for training and prediction. Figure 5 The prediction results at different scales are presented, and it is shown that as the magnification scale increases, the image clarity and detail retention decrease. However, the LST spatial distribution characteristics of the original MODIS data are well preserved. The conditional generative adversarial network shows good robustness for LST prediction at different resolutions.

[0117] In summary, with the help of the above technical solution of the present invention, the present invention uses conditional generative adversarial networks to achieve spatiotemporal fusion of surface temperature, solving the problem of excessive dependence on the time of reference images in the current spatiotemporal fusion of surface temperature; the present invention only needs a coarse-resolution image of the predicted date and another fine-resolution reference image at any time in the same area for model input, and there is no limit on the acquisition time of the reference. The present invention is robust to reference images at different times and resolutions of different scales, and due to less input and restrictions, the present invention also has more flexibility.

[0118] Embodiment 2

[0119] This embodiment provides a spatiotemporal fusion system of land surface temperature based on a conditional generative adversarial network, including:

[0120] A data acquisition module, which is used to acquire a surface temperature image with a first spatial resolution at a target time and a second spatial resolution at a reference time;

[0121] The conditional generative adversarial network training module is used to train the constructed conditional generative adversarial network based on the acquired surface temperature images of the first spatial resolution at the target moment and the second spatial resolution at the reference moment to obtain the trained conditional generative adversarial network; wherein the training process of the conditional generative adversarial network includes:

[0122] Inputting the surface temperature image of the first spatial resolution at the target time into the generator, and obtaining the prediction result of the second spatial resolution at the initial target time;

[0123] The objective function of the generator is to minimize the distribution distance between the initial second spatial resolution prediction result at the target time and the second spatial resolution surface temperature image at the reference time, and the objective function of the discriminator is to maximize the distribution distance, thereby generating the maximum and minimum objective function of the adversarial network.

[0124] The set conditional labels are input into the generator and discriminator to limit the maximum and minimum objective functions of the adversarial network. The network searches along the desired direction to obtain the optimal solution.

[0125] The prediction module is used to predict the second spatial resolution prediction result at the target moment based on the trained conditional generative adversarial network.

[0126] Embodiment 3

[0127] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the method for spatiotemporal fusion of surface temperature based on a conditional generative adversarial network as described above are implemented.

[0128] Embodiment 4

[0129] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for spatiotemporal fusion of surface temperature based on a conditional generative adversarial network as described above are implemented.

[0130] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0131] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0134] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A spatiotemporal fusion method of surface temperature based on conditional generative adversarial networks, characterized in that: The steps include: Acquire the surface temperature image of the first spatial resolution at the target time and the second spatial resolution at the reference time; The constructed conditional generative adversarial network is trained based on the acquired surface temperature images with the first spatial resolution at the target moment and the second spatial resolution at the reference moment to obtain a trained conditional generative adversarial network; The training process of the conditional generative adversarial network includes: Inputting the surface temperature image of the first spatial resolution at the target time into the generator, and obtaining the prediction result of the second spatial resolution at the initial target time; The objective function of the generator is to minimize the distribution distance between the initial second spatial resolution prediction result at the target time and the second spatial resolution surface temperature image at the reference time, and the objective function of the discriminator is to maximize the distribution distance, thereby generating the maximum and minimum objective function of the adversarial network. The set conditional labels are input into the generator and discriminator to limit the maximum and minimum objective functions of the adversarial network. The network searches along the desired direction to obtain the optimal solution. The second spatial resolution prediction result at the target moment is obtained based on the trained conditional generative adversarial network prediction.

2. The spatiotemporal fusion method of land surface temperature based on conditional generative adversarial network according to claim 1, characterized in that: The training process of the generator includes: Extracting the common features of the surface temperature images of the first spatial resolution at the target time and the second spatial resolution at the reference time based on the encoder, and concatenating the common features along the channel dimension to obtain the first surface temperature features; The first surface temperature feature is input into a decoder for decoding, and restored to the original pixel space through a deconvolution operation to obtain a second surface temperature feature; The content loss function is calculated by combining the second surface temperature feature with the surface temperature of the second spatial resolution at the reference moment, and the generator is optimized based on the content loss function.

3. The spatiotemporal fusion method of land surface temperature based on conditional generative adversarial network according to claim 2, characterized in that: The encoder includes three parts, each of which is connected by a BN layer, a ReLU layer and a Conv layer. The output of the first part is used as the input of the next part. The difference is that the Conv layer uses 3×3Conv 32 layers, 3×3Conv 64 layers and 3×3Conv128 layers in sequence; The decoder includes three parts, each of which is a BN layer, a ReLU layer and a Deconv layer. The output of the first part is used as the input of the next part. The difference is that the Deconv layer uses 3×3Deconv 128 layers, 3×3Deconv 64 layers, and 3×3Deconv 32 layers in sequence.

4. The spatiotemporal fusion method of land surface temperature based on conditional generative adversarial network according to claim 1, characterized in that: The training process of the discriminator includes: The second spatial resolution prediction result at the target time and the first resolution surface temperature at the target time are used as the first set of data, and the first resolution surface temperature at the target time and the second resolution surface temperature at the target time are used as the second set of data; The first set of data and the second set of data are respectively mapped to the probability of belonging to the real image, and the ideal output of the mapping of the first set of data is defined as 0, and the ideal output of the mapping of the second set of data is defined as 1; The difference between the discriminator's output and the ideal output is used as the discriminator loss to update the discriminator's parameters.

5. The spatiotemporal fusion method of land surface temperature based on conditional generative adversarial network according to claim 1, characterized in that: The additional condition of the generator is the second-resolution surface temperature image at the reference time, and the additional condition of the discriminator is the first-resolution surface temperature image at the target time.

6. The spatiotemporal fusion method of land surface temperature based on conditional generative adversarial network according to claim 1, characterized in that: The total loss function is: L CGAN-LSTFM =L CGAN +L Content +L Feature , Among them, L CGAN is the conditional adversarial loss, L Content is the content loss, L Feature represents feature loss, y is the additional condition, D(x|y) is the discriminant result of the second-resolution surface temperature at the target moment under the additional condition, D(G(z|y)) is the discriminant result of the second-resolution prediction generated under the additional condition, D(x) is the discriminant result of the second-resolution surface temperature at the target moment, D(G(z)) is the discriminant result of the second-resolution surface temperature predicted at the target moment, x is the second-resolution surface temperature at the target moment, is the logarithmic expectation of the discriminator’s output for all x, is the expectation of the inverse logarithmic value of the discriminator's output for all generated data; δ is a hyperparameter, represents the generator prediction result, M represents the real label image, N represents the number of pixels in the feature map, and F represents the predicted image features obtained by the encoder. Represents the features extracted from the true labeled images.

7. The spatiotemporal fusion method of land surface temperature based on conditional generative adversarial network according to claim 1, characterized in that: After obtaining the surface temperature images of the first spatial resolution at the target time and the second spatial resolution at the reference time, preprocessing is performed, including radiation calibration, atmospheric correction and cropping.

8. The spatiotemporal fusion system of surface temperature based on conditional generative adversarial network is characterized by: include: A data acquisition module, which is used to acquire a surface temperature image with a first spatial resolution at a target time and a second spatial resolution at a reference time; A conditional generative adversarial network training module is used to train the constructed conditional generative adversarial network based on the acquired surface temperature images with a first spatial resolution at a target moment and a second spatial resolution at a reference moment, to obtain a trained conditional generative adversarial network; The training process of the conditional generative adversarial network includes: Inputting the surface temperature image of the first spatial resolution at the target time into the generator, and obtaining the prediction result of the second spatial resolution at the initial target time; The objective function of the generator is to minimize the distribution distance between the initial second spatial resolution prediction result at the target time and the second spatial resolution surface temperature image at the reference time, and the objective function of the discriminator is to maximize the distribution distance, thereby generating the maximum and minimum objective function of the adversarial network. The set conditional labels are input into the generator and discriminator to limit the maximum and minimum objective functions of the adversarial network. The network searches along the desired direction to obtain the optimal solution. The prediction module is used to predict the second spatial resolution prediction result at the target moment based on the trained conditional generative adversarial network.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the spatiotemporal fusion method of surface temperature based on a conditional generative adversarial network as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the spatiotemporal fusion method of surface temperature based on a conditional generative adversarial network as described in any one of claims 1 to 7 are implemented.

Images