Multi-scale progressive land surface temperature fusion downscaling method and device

Abstract

The present application relates to the technical field of remote sensing image intelligent processing, and particularly relates to a multi-scale progressive land surface temperature fusion downscaling method and device, wherein the method comprises: preprocessing land surface temperature images in a multi-scale image library; constructing a multi-scale training data set containing land surface temperature and at least one auxiliary parameter; constructing a land surface temperature downscaling network based on multi-parameter fusion, and constructing a domain transformation network based on heterogeneous high-frequency information guidance; training the land surface temperature downscaling network and the domain transformation network based on a progressive double-network joint training strategy; fine-tuning a downscaling model meeting a preset low-to-medium resolution condition; and performing low-to-medium-to-high progressive downscaling on a low-resolution land surface temperature image to be processed, so as to generate a multi-scale progressive land surface temperature meeting a target high-resolution condition. The present application fully considers the downscaling difference problem under different resolutions, realizes the physical property retention of a land surface thermal field, and improves the processing efficiency of large-area remote sensing data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent processing of remote sensing images, and particularly relates to a multi-scale progressive land surface temperature fusion downscaling method and device. BACKGROUND

[0002] Satellite remote sensing technology is the most effective means to quickly obtain regional or global scale land surface temperature data, and plays an important role in the research of agricultural drought monitoring, urban heat island effect, global water and heat cycle and climate change. However, due to the hardware limitations of thermal infrared sensors, the existing land surface temperature products are usually of hundreds of meters resolution, and it is challenging to obtain higher spatial resolution land surface temperature data, which cannot meet the needs of fine application. Spatial downscaling technology is an effective way to obtain higher spatial resolution land surface temperature data.

[0003] The existing land surface temperature spatial downscaling technology is mainly based on statistical regression, land surface energy balance, geographic weighted regression and other methods. Among them, the statistical regression method is widely used in land surface temperature spatial downscaling research due to its simple implementation and high precision. Early regression algorithms mainly focus on fitting the linear or simple nonlinear statistical relationship between land surface temperature and remote sensing index, but have insufficient ability to deal with complex nonlinear characteristics and spatial heterogeneity. With the rapid development of deep learning technology, its powerful feature extraction and nonlinear modeling ability is introduced into the land surface temperature downscaling research, which significantly improves the expression ability and precision of the model. However, the existing deep learning downscaling method mostly assumes that the mapping relationship between land surface temperature and auxiliary parameters remains consistent at different scales, and usually learns the mapping relationship between land surface temperature and auxiliary parameters at low resolution scale and directly applies it to high resolution auxiliary parameters to predict high resolution land surface temperature. This assumption ignores the local heterogeneity caused by the scale difference between high and low resolution, which limits the accuracy of the downscaling result. SUMMARY

[0004] The present application provides a multi-scale progressive land surface temperature fusion downscaling method and device to solve the problem that the existing deep learning downscaling method ignores the local heterogeneity caused by the scale difference between high and low resolution, which limits the accuracy of the downscaling result.

[0005] The first aspect embodiment of the present application provides a multi-scale progressive land surface temperature fusion downscaling method, comprising the following steps: obtaining at least one product of a target satellite land surface temperature product satisfying a preset low-resolution condition, a reference satellite land surface temperature product satisfying a preset medium-resolution condition, and an auxiliary parameter product of the reference satellite satisfying a preset high-resolution condition; resampling the at least one product to generate a multi-scale image library corresponding to the resolution of the at least one product, and preprocessing the land surface temperature image in the multi-scale image library to generate a preprocessed image; constructing a multi-scale training data set containing land surface temperature and at least one auxiliary parameter by using the preprocessed image; constructing a land surface temperature downscaling network based on multi-parameter fusion according to the multi-scale training data set, and constructing a domain transformation network based on heterogeneous high-frequency information guidance according to the multi-scale training data set; based on a progressive double-network joint training strategy, training the land surface temperature downscaling network and the domain transformation network by using the training data set in the multi-scale training data set satisfying the preset low-medium resolution condition, to generate a downscaling model satisfying the preset low-medium resolution condition and a downscaling result satisfying the preset medium-resolution condition; fine-tuning the downscaling model satisfying the preset low-medium resolution condition by using the data set in the multi-scale training data set satisfying the preset medium-high resolution condition, to generate a downscaling model satisfying the preset medium-high resolution condition and a downscaling result satisfying the preset high-resolution condition; based on the downscaling model satisfying the preset low-medium resolution condition, the downscaling result satisfying the preset medium-resolution condition, the downscaling model satisfying the preset medium-high resolution condition, and the downscaling result satisfying the preset high-resolution condition, performing low-medium-high progressive downscaling on a to-be-processed low-resolution land surface temperature image to generate a multi-scale progressive land surface temperature satisfying a target high-resolution condition.

[0006] Optionally, in an embodiment of the present application, the preprocessing of the land surface temperature image in the multi-scale image library to generate a preprocessed image comprises: calculating a normalized vegetation index, a normalized water index, and a normalized building index by using the reflectivity image of the target satellite; performing statistical property correction on the land surface temperature data of the reference satellite to generate a reference satellite land surface temperature image satisfying a target resolution condition; preprocessing the land surface temperature image of the target satellite, the land surface temperature image of the reference satellite, and the digital elevation model, the normalized vegetation index, the normalized water index, and the normalized building index image to generate the preprocessed image.

[0007] Optionally, in an embodiment of the present application, the calculation formula of the reference satellite land surface temperature image satisfying the target resolution condition is:

[0008]

[0009] wherein, represents a histogram-matched mid-resolution reference satellite land surface temperature image, represents an original mid-resolution reference satellite land surface temperature image, and respectively represent a standard deviation of a low-resolution target satellite land surface temperature image and an original mid-resolution reference satellite land surface temperature image, and respectively represent a mean value of a low-resolution target satellite land surface temperature image and an original mid-resolution reference satellite land surface temperature image.

[0010] Optionally, in an embodiment of the present application, the constructing a multi-scale training data set containing land surface temperature and at least one auxiliary parameter by using the preprocessed image comprises: cropping the multi-scale image library into at least one group of image blocks; generating a mid-high resolution data set in the multi-scale training data set based on a target satellite land surface temperature image block satisfying a preset low-resolution condition, a reference satellite land surface temperature image block satisfying a preset mid-resolution condition, a digital elevation model image block satisfying a preset mid-resolution condition, a normalized vegetation index image block, a normalized water index image block and a normalized building index image block in the at least one group of image blocks; and generating a high-resolution auxiliary parameter data set in the multi-scale training data set based on a digital elevation model image block satisfying a preset high-resolution condition, a normalized vegetation index image block, a normalized water index image block and a normalized building index image block in the at least one group of image blocks.

[0011] Optionally, in an embodiment of the present application, the constructing a multi-parameter fusion-based land surface temperature downscaling network according to the multi-scale training data set comprises: extracting preliminary features from land surface temperature satisfying a preset coarse resolution condition and auxiliary land surface parameters satisfying a preset fine resolution condition; constructing a multi-scale feature extraction module and extracting multi-scale features of the preliminary features of the auxiliary land surface parameters satisfying the preset fine resolution condition by using the multi-scale feature extraction module; constructing a lightweight CNN module and extracting local features of the land surface temperature satisfying the preset coarse resolution condition and the multi-scale features by using the lightweight CNN module, and performing channel compression on the local features to generate a compression result; constructing a lightweight Transformer module, extracting features of the compression result by using the lightweight Transformer module to generate global features; constructing a downscaling reconstruction module and processing the global features by using the downscaling reconstruction module to generate a downscaled land surface temperature.

[0012] Optionally, in an embodiment of the present application, the extraction formula of the preliminary features is:

[0013]

[0014]

[0015] wherein, and respectively represent the preliminary features of the land surface temperature satisfying the preset coarse resolution condition and the land surface parameter satisfying the preset fine resolution condition, representing the land surface temperature satisfying the preset coarse resolution condition, representing a channel addition operation, representing a convolution operation, , and , respectively being a weight term and a bias term of two convolution operations, being a digital elevation model, being a normalized vegetation index, being a normalized water index, being a normalized building index;

[0016] The construction formula of the multi-scale feature is:

[0017]

[0018] wherein, being an output feature map of the multi-scale feature, representing a convolution attention operation, representing a feature map obtained by combining features with a size of × and × obtained by convolution, representing a feature map obtained by combining features with a size of × and × obtained by convolution;

[0019] The calculation formula of the feature map is respectively:

[0020]

[0021]

[0022] wherein, representing a Relu activation function, , , , and , respectively being a feature map with a size of × 、 × 、 × weight and bias terms in convolution operation;

[0023] The extraction formula of the local feature and the calculation formula of the channel compression are respectively:

[0024]

[0025]

[0026]

[0027] wherein, and are local features obtained by processing the ground surface temperature and multi-scale features satisfying the preset coarse resolution condition by the lightweight CNN module, is a feature map obtained by compressing the output of the two lightweight CNN architectures in the channel, represents a 2-fold up-sampling operation, 、 and 、 respectively represent weight and bias terms of the two lightweight CNN modules, and are weight and bias terms of the channel compression convolution operation;

[0028] The calculation formula of the lightweight Transformer module is:

[0029]

[0030] wherein, is an output feature map of the lightweight Transformer module, is a feature map obtained by compressing the output of the two lightweight CNN architectures in the channel, is a channel segmentation number, represents a scaling factor, , , represents a matrix obtained by linear projection from an input feature map;

[0031] The calculation formula of the matrix is respectively:

[0032]

[0033]

[0034]

[0035] wherein, wherein is a split operation;

[0036] The calculation formula of the down-scaled land surface temperature is:

[0037]

[0038] wherein, is a land surface temperature output by a down-scaled reconstruction module, represents the land surface temperature satisfying a preset coarse resolution condition, represents a 2-fold up-sampling operation, , and , are weight terms and bias terms of single-channel convolution and three-channel convolution, respectively.

[0039] Optionally, in an embodiment of the present application, the constructing a domain transformation network based on heterogeneous high-frequency information guidance according to the multi-scale training data set comprises: extracting land surface temperature high-frequency information from the down-scaled land surface temperature by using Laplace high-pass filtering; extracting local features from the land surface temperature high-frequency information by using the lightweight CNN module; extracting global features from the local features by using the lightweight Transformer module; based on the global features, processing the feature map by using single-channel convolution, and adopting residual connection to construct the domain transformation network based on heterogeneous high-frequency information guidance.

[0040] Optionally, in an embodiment of the present application, the land surface temperature high-frequency information extraction formula is:

[0041]

[0042] wherein, represents a land surface temperature high-frequency information extraction result, represents Laplace high-pass filtering, and the filter kernel is a matrix , is a land surface temperature output by the down-scaled reconstruction module;

[0043] The matrix is defined as:

[0044]

[0045] The calculation formula of the local features is:

[0046]

[0047] wherein, denotes a local feature extracted from the land surface temperature high-frequency information, denotes the lightweight CNN module, denotes a local feature extracted from the land surface temperature high-frequency information;

[0048] The calculation formula of the global feature is:

[0049]

[0050] wherein, denotes a global feature extracted from the local feature, denotes the lightweight Transformer module, denotes a local feature extracted from the land surface temperature high-frequency information;

[0051] The high-frequency information reconstruction formula in the domain transformation network is:

[0052]

[0053] wherein, is a NDVI high-frequency information prediction result obtained after domain transformation, and respectively denote a weight term and a bias term of single-channel convolution, denotes a global feature extracted from the local feature.

[0054] Optionally, in an embodiment of the present application, the calculation formulas of the land surface temperature downscaling network and the domain transformation network are respectively:

[0055]

[0056]

[0057] wherein, and respectively denote a loss function of the land surface temperature downscaling network and the domain transformation network, denotes a land surface temperature fidelity loss meeting a preset coarse resolution condition, denotes a frequency domain consistency loss meeting a preset fine resolution condition, denotes a cross-modal frequency domain alignment loss, is a neural network parameter, are all empirical coefficients for adjusting the weights of each sub-loss function;

[0058] The calculation formula of the land surface temperature fidelity loss meeting the preset coarse resolution condition is:

[0059]

[0060] wherein, denotes the number of training image pairs, denotes the first coarse resolution land surface temperature image in the image pair, denotes the network 1, denotes resampling to the coarse resolution data scale;

[0061] The calculation formula of the frequency domain consistency loss satisfying the preset fine resolution condition is:

[0062]

[0063] wherein, denotes the number of training image pairs, denotes Laplacian high-pass filtering, denotes the first reference fine resolution land surface temperature image in the image pair;

[0064] The calculation formula of the cross-modal frequency domain alignment loss is:

[0065]

[0066] wherein, denotes the network 1, denotes the network 2, denotes the first fine resolution NDVI image in the image pair, denotes the first coarse resolution land surface temperature image in the image pair.

[0067] The second aspect embodiment of the present application provides a multi-scale progressive land surface temperature fusion downscaling device, comprising: an acquisition module configured to acquire at least one product of a target satellite land surface temperature product satisfying a preset low-resolution condition, a reference satellite land surface temperature product satisfying a preset medium-resolution condition, and an auxiliary parameter product of the reference satellite satisfying a preset high-resolution condition; a preprocessing module configured to resample the at least one product to generate a multi-scale image library corresponding to the resolution of the at least one product, and preprocess a land surface temperature image in the multi-scale image library to generate a preprocessed image; a data construction module configured to construct a multi-scale training data set containing land surface temperature and at least one auxiliary parameter by using the preprocessed image; a network construction module configured to construct a land surface temperature downscaling network based on multi-parameter fusion according to the multi-scale training data set, and construct a domain transformation network based on heterogeneous high-frequency information guidance according to the multi-scale training data set; a training module configured to train the land surface temperature downscaling network and the domain transformation network by using a training data set in the multi-scale training data set satisfying a preset low-medium resolution condition based on a progressive dual-network joint training strategy, to generate a downscaling model satisfying a preset low-medium resolution condition and a downscaling result satisfying a preset medium-resolution condition; a generation module configured to fine-tune the downscaling model satisfying the preset low-medium resolution condition by using a data set in the multi-scale training data set satisfying a preset medium-high resolution condition, to generate a downscaling model satisfying a preset medium-high resolution condition and a downscaling result satisfying a preset high-resolution condition; and a downscaling processing module configured to perform low-medium-high progressive downscaling on a to-be-processed low-resolution land surface temperature image based on the downscaling model satisfying the preset low-medium resolution condition, the downscaling result satisfying the preset medium-resolution condition, the downscaling model satisfying the preset medium-high resolution condition, and the downscaling result satisfying the preset high-resolution condition, to generate a multi-scale progressive land surface temperature satisfying a target high-resolution condition.

[0068] Optionally, in an embodiment of the present application, the preprocessing module comprises: a calculation unit configured to calculate a normalized vegetation index, a normalized water index, and a normalized building index by using reflectivity images of the target satellite; a correction unit configured to perform statistical property correction on land surface temperature data of the reference satellite to generate a reference satellite land surface temperature image satisfying a target resolution condition; and a preprocessing unit configured to preprocess the target satellite land surface temperature image, the reference satellite land surface temperature image, and digital elevation model images of the normalized vegetation index, the normalized water index, and the normalized building index to generate the preprocessed image.

[0069] Optionally, in an embodiment of the present application, the calculation formula of the reference satellite land surface temperature image satisfying the target resolution condition is:

[0070]

[0071] wherein, represents a mid-resolution reference satellite land surface temperature image after histogram matching, represents an original mid-resolution reference satellite land surface temperature image, and respectively represent a standard deviation of a low-resolution target satellite land surface temperature image and an original mid-resolution reference satellite land surface temperature image, and respectively represent a mean value of a low-resolution target satellite land surface temperature image and an original mid-resolution reference satellite land surface temperature image.

[0072] Optionally, in an embodiment of the present application, the data construction module comprises: a cropping unit configured to crop the multi-scale image library into at least one group of image blocks; a first generation unit configured to generate a mid-high resolution data set in the multi-scale training data set based on a target satellite land surface temperature image block satisfying a preset low resolution condition, a reference satellite land surface temperature image block satisfying a preset mid resolution condition, a digital elevation model image block satisfying a preset mid resolution condition, a normalized vegetation index image block, a normalized water index image block and a normalized building index image block in the at least one group of image blocks; and a second generation unit configured to generate a high resolution auxiliary parameter data set in the multi-scale training data set based on a digital elevation model image block satisfying a preset high resolution condition, a normalized vegetation index image block, a normalized water index image block and a normalized building index image block in the at least one group of image blocks.

[0073] Optionally, in an embodiment of the present application, the network construction module comprises: a feature extraction unit configured to extract preliminary features from a land surface temperature satisfying a preset coarse resolution condition and an auxiliary land surface parameter satisfying a preset fine resolution condition; a multi-scale feature extraction unit configured to construct a multi-scale feature extraction module and extract multi-scale features of the preliminary features of the auxiliary land surface parameter satisfying the preset fine resolution condition by using the multi-scale feature extraction module; a first local feature extraction unit configured to construct a lightweight CNN module and extract local features in the land surface temperature satisfying the preset coarse resolution condition and the multi-scale features by using the lightweight CNN module, and perform channel compression on the local features to generate a compression result; a first global feature extraction unit configured to construct a lightweight Transformer module and perform feature extraction on the compression result by using the lightweight Transformer module to generate global features; and a first reconstruction unit configured to construct a downscale reconstruction module and process the global features by using the downscale reconstruction module to generate a down-scaled land surface temperature.

[0074] Optionally, in an embodiment of the present application, the extraction formula of the preliminary features is:

[0075]

[0076]

[0077] wherein, and respectively represent the preliminary features of the land surface temperature satisfying the preset coarse resolution condition and the land surface parameter satisfying the preset fine resolution condition, represents the land surface temperature satisfying the preset coarse resolution condition, represents a channel addition operation, represents a convolution operation, , and , are respectively a weight term and a bias term of two convolution operations, is a digital elevation model, is a normalized vegetation index, is a normalized water index, is a normalized building index;

[0078] The construction formula of the multi-scale features is:

[0079]

[0080] wherein, is an output feature map of the multi-scale features, represents a convolution attention operation, represents a feature map obtained by combining features with a size of × and × obtained by convolution, represents a feature map obtained by combining features with a size of × and × obtained by convolution;

[0081] The calculation formula of the feature map is respectively:

[0082]

[0083]

[0084] wherein, represents a Relu activation function, , , , and , are weight terms and bias terms in the convolution operation with a size of × , × , × ;

[0085] The extraction formula of the local feature and the calculation formula of the channel compression are respectively:

[0086]

[0087]

[0088]

[0089] wherein, and are local features obtained by processing the ground surface temperature and the multi-scale feature that meet the preset coarse resolution condition by the lightweight CNN module, is a feature map obtained by compressing the output of the two lightweight CNN architectures in the channel, denotes a 2 times upsampling operation, , and , denote weight terms and bias terms of the two lightweight CNN modules, and are weight terms and bias terms of the channel compression convolution operation;

[0090] The calculation formula of the lightweight Transformer module is:

[0091]

[0092] wherein, is an output feature map of the lightweight Transformer module, is a feature map obtained by compressing the output of the two lightweight CNN architectures in the channel, is a channel segmentation number, denotes a scaling factor, , , denotes a matrix obtained by linear projection from an input feature map;

[0093] The calculation formula of the matrix is respectively:

[0094]

[0095]

[0096]

[0097] wherein, wherein is a split operation;

[0098] The calculation formula of the down-scaled land surface temperature is:

[0099]

[0100] wherein, is the land surface temperature output by the down-scaling reconstruction module, represents the land surface temperature satisfying the preset coarse resolution condition, represents a 2-fold up-sampling operation, , and , are weight terms and bias terms of single-channel convolution and three-channel convolution, respectively.

[0101] Optionally, in an embodiment of the present application, the network construction module comprises: an information extraction unit configured to extract land surface temperature high-frequency information from the down-scaled land surface temperature by using Laplacian high-pass filtering; a second local feature extraction unit configured to extract local features from the land surface temperature high-frequency information by using the lightweight CNN module; a second global feature extraction unit configured to extract global features from the local features by using the lightweight Transformer module; and a second reconstruction unit configured to process the feature map by using single-channel convolution based on the global features and adopt residual connection to construct the domain transformation network based on heterogeneous high-frequency information guidance.

[0102] Optionally, in an embodiment of the present application, the land surface temperature high-frequency information extraction formula is:

[0103]

[0104] wherein, represents a land surface temperature high-frequency information extraction result, represents Laplacian high-pass filtering, and the filter kernel is a matrix , is the land surface temperature output by the down-scaling reconstruction module;

[0105] The matrix is defined as:

[0106]

[0107] The calculation formula of the local feature is:

[0108]

[0109] Wherein, represents the local feature extracted from the surface temperature high-frequency information, represents the lightweight CNN module, represents the local feature extracted from the surface temperature high-frequency information;

[0110] The calculation formula of the global feature is:

[0111]

[0112] Wherein, represents the global feature extracted from the local feature, represents the lightweight Transformer module, represents the local feature extracted from the surface temperature high-frequency information;

[0113] The high-frequency information reconstruction formula in the domain transformation network is:

[0114]

[0115] Wherein, is the NDVI high-frequency information prediction result obtained after domain transformation, and respectively represent the weight term and the bias term of the single-channel convolution, represents the global feature extracted from the local feature.

[0116] Optionally, in an embodiment of the present application, the calculation formulas of the surface temperature downscaling network and the domain transformation network are respectively:

[0117]

[0118]

[0119] Wherein, and respectively represent the loss functions of the surface temperature downscaling network and the domain transformation network, represents the surface temperature fidelity loss meeting the preset coarse resolution condition, represents the frequency domain consistency loss meeting the preset fine resolution condition, represents the cross-modal frequency domain alignment loss, is a neural network parameter, are empirical coefficients used to adjust the weights of each sub-loss function;

[0120] The calculation formula of the ground temperature fidelity loss meeting the preset coarse resolution condition is:

[0121]

[0122] wherein, indicates the number of training image pairs, indicates the i-th image pair, the coarse resolution ground temperature image in the image pair, indicates network 1, indicates resampling to the coarse resolution data scale;

[0123] The calculation formula of the frequency domain consistency loss meeting the preset fine resolution condition is:

[0124]

[0125] wherein, indicates the number of training image pairs, indicates Laplace high-pass filtering, indicates the i-th image pair, the reference fine resolution ground temperature image in the image pair;

[0126] The calculation formula of the cross-modal frequency domain alignment loss is:

[0127]

[0128] wherein, indicates network 1, indicates network 2, indicates the i-th image pair, the fine resolution NDVI image in the image pair, indicates the i-th image pair, the coarse resolution ground temperature image in the image pair.

[0129] The embodiment of the present application can utilize multi-source remote sensing data to construct a multi-scale data set containing surface temperature and multi-source auxiliary parameters, in a low resolution scale, utilize a surface temperature downscaling network to perform heterogeneous feature fusion and downscaling, obtain a medium resolution surface temperature, extract high frequency information from the medium resolution result, perform heterogeneous data high frequency information conversion through a domain transformation network, and enhance spatial detail expression, in a high resolution scale, fine tune the network for scale difference problems, and apply it to the medium resolution downscaling result, obtain a high resolution surface temperature, the present application fully considers the downscaling difference problems in different resolutions, realizes the physical property maintenance of the surface thermal field, and improves the processing efficiency of large area remote sensing data.

[0130] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0131] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings.

[0132] Figure 1 A flow chart of a multi-scale progressive surface temperature fusion downscaling method according to an embodiment of the present application is provided;

[0133] Figure 2 A schematic diagram of the overall process of a multi-scale progressive surface temperature fusion downscaling method according to an embodiment of the present application is provided;

[0134] Figure 3 A schematic diagram of the network framework of a multi-scale progressive surface temperature fusion downscaling method according to an embodiment of the present application is provided;

[0135] Figure 4 A schematic diagram of the results of a multi-scale progressive surface temperature fusion downscaling method according to an embodiment of the present application is provided;

[0136] Figure 5 A structural schematic diagram of a multi-scale progressive surface temperature fusion downscaling device according to an embodiment of the present application is provided.

[0137] Among them, 10 is a multi-scale progressive surface temperature fusion downscaling device; 100 is an acquisition module, 200 is a preprocessing module, 300 is a data construction module, 400 is a network construction module, 500 is a training module, 600 is a generation module, and 700 is a downscaling processing module. DETAILED DESCRIPTION

[0138] Embodiments of the present application are described below in detail, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0139] A multiscale progressive land surface temperature fusion downscaling method and device are described below with reference to the accompanying drawings. In view of the problem that the existing deep learning downscaling method mentioned in the background art ignores the local heterogeneity caused by the scale difference between high and low resolutions, which limits the accuracy of the downscaling result, the present application provides a multiscale progressive land surface temperature fusion downscaling method. In this method, a multiscale data set containing land surface temperature and multi-source auxiliary parameters can be constructed using multi-source remote sensing data. At the low resolution scale, a land surface temperature downscaling network is used for heterogeneous feature fusion downscaling to obtain a medium resolution land surface temperature. High frequency information is extracted from the medium resolution result, and a domain transformation network is used for heterogeneous data high frequency information conversion to enhance spatial detail expression. At the high resolution scale, the network is fine-tuned for the scale difference problem and applied to the medium resolution downscaling result to obtain a high resolution land surface temperature. The present application fully considers the downscaling difference problem at different resolutions, realizes the preservation of the physical properties of the land surface thermal field, and improves the efficiency of processing large-area remote sensing data. Thus, the problem that the existing deep learning downscaling method ignores the local heterogeneity caused by the scale difference between high and low resolutions, which limits the accuracy of the downscaling result, is solved.

[0140] Specifically, Figure 1 A flowchart of a multiscale progressive land surface temperature fusion downscaling method provided by an embodiment of the present application is shown in FIG. 1.

[0141] As Figure 1 shown, the multiscale progressive land surface temperature fusion downscaling method includes the following steps:

[0142] In step S101, at least one product of a target satellite land surface temperature product satisfying a preset low resolution condition, a reference satellite land surface temperature product satisfying a preset medium resolution condition, and a reference satellite auxiliary parameter product satisfying a preset high resolution condition is obtained.

[0143] In actual execution, the embodiment of the present application can acquire a target satellite low-resolution land surface temperature product, a reference satellite medium-resolution land surface temperature product and a high-resolution auxiliary parameter product, including a normalized vegetation index (NDVI), a normalized water body index (NDWI), a normalized building index (NDBI) and a digital elevation model (DEM). The present application can acquire Landsat 9 land surface temperature (100 m) and reflectivity image (30 m), Landsat 7 land surface temperature image (60 m) and SRTM 1 DEM image (30 m), and calculate NDVI, NDWI and NDBI by using the reflectivity image of Landsat 9.

[0144] The embodiment of the present application can fully mine the physical correlation and complementarity between parameters by fusing low-resolution satellite land surface temperature and high-resolution multi-source heterogeneous land surface parameters and combining the characteristic interaction mechanism of the double network, and break through the limitation of single temperature data downscaling.

[0145] In step S102, at least one product is resampled to generate a multi-scale image library corresponding to the resolution of the at least one product, and a land surface temperature image in the multi-scale image library is preprocessed to generate a preprocessed image.

[0146] In the present application, the embodiment can construct a multi-scale image library of low, medium and high resolution based on resampling, correct the statistical characteristics of the land surface temperature image of the reference satellite, calculate the required land surface parameters, and resample the preprocessed data to multiple scales to generate a preprocessed image.

[0147] Optionally, in an embodiment of the present application, the land surface temperature image in the multi-scale image library is preprocessed to generate a preprocessed image, including: calculating a normalized vegetation index, a normalized water body index and a normalized building index by using the reflectivity image of the target satellite; correcting the statistical characteristics of the land surface temperature data of the reference satellite to generate a land surface temperature image of the reference satellite satisfying the target resolution condition; and preprocessing the land surface temperature image of the target satellite, the land surface temperature image of the reference satellite and the images of the digital elevation model, the normalized vegetation index, the normalized water body index and the normalized building index to generate a preprocessed image.

[0148] In actual execution, as shown in Figure 2 , the embodiment of the present application can include the following steps:

[0149] (1) Calculate NDVI, NDWI and NDBI by using the reflectivity image of the target satellite.

[0150] (2) The histogram matching is used to correct the statistical characteristics of the land surface temperature data of the reference satellite, so as to reduce the system deviation. In an embodiment of the present application, the histogram matching is used to correct the statistical characteristics of the Landsat 7 land surface temperature image, and the calculation formula of the land surface temperature image of the reference satellite meeting the target resolution condition is as follows:

[0151]

[0152] wherein, represents the histogram-matched medium-resolution reference satellite land surface temperature image, represents the original medium-resolution reference satellite land surface temperature image, and respectively represent the standard deviation of the low-resolution target satellite land surface temperature image and the original medium-resolution reference satellite land surface temperature image, and respectively represent the mean value of the low-resolution target satellite land surface temperature image and the original medium-resolution reference satellite land surface temperature image.

[0153] (3) The land surface temperature images of the target satellite and the reference satellite, and the DEM, NDVI, NDWI and NDBI images are preprocessed, including geographic registration, terrain correction and cutting, to obtain images covering the same area.

[0154] (4) A multi-scale image library is constructed, and the target satellite image obtained by the above processing is resampled to medium resolution, and the DEM, NDVI, NDWI and NDBI images are resampled to medium resolution and high resolution, that is, the Landsat 9 land surface temperature obtained by the above processing can be resampled to 120 m, and the DEM, NDVI, NDWI and NDBI are resampled to 60 m and 30 m.

[0155] In step S103, a multi-scale training data set containing land surface temperature and at least one auxiliary parameter is constructed by using the preprocessed images.

[0156] In actual execution, the embodiment of the present application can construct a multi-scale training data set containing land surface temperature and multiple auxiliary parameters by using the preprocessed images, including a low-resolution land surface temperature-medium-resolution auxiliary parameter training data set (denoted as a low-medium-resolution data set) and a high-resolution auxiliary parameter data set.

[0157] Optionally, in an embodiment of the present application, a multi-scale training dataset containing the surface temperature and at least one auxiliary parameter is constructed using the preprocessed images, including: cropping the multi-scale image library into at least one group of image blocks; based on the target satellite surface temperature image block in the at least one group of image blocks that meets the preset low-resolution condition, the reference satellite surface temperature image block that meets the preset medium-resolution condition, the digital elevation model image block that meets the preset medium-resolution condition, the normalized vegetation index image block, the normalized water index image block and the normalized building index image block, a medium-high resolution dataset in the multi-scale training dataset is generated; based on the digital elevation model image block, the normalized vegetation index image block, the normalized water index image block and the normalized building index image block in the at least one group of image blocks that meet the preset high-resolution condition, a high-resolution auxiliary parameter dataset in the multi-scale training dataset is generated.

[0158] Specifically, the embodiment of the present application can crop the multi-scale image library into multiple groups of image blocks, and construct a multi-scale dataset according to the multiple groups of image blocks. Among them, the low-resolution target satellite surface temperature image block, the medium-resolution reference satellite surface temperature image block, and the medium-resolution DEM, NDVI, NDWI and NDBI image block constitute a low-medium resolution dataset. The high-resolution DEM, NDVI, NDWI and NDBI image block constitutes a high-resolution auxiliary parameter dataset. Specifically, using the multi-scale image library in the above steps, for 120 m Landsat 9 surface temperature image, 60 m Landsat 7 surface temperature and 60 m (30 m) DEM, NDVI, NDWI and NDBI auxiliary data, image blocks of 32x32 and 64x64 covering the same ground objects are cropped respectively, and 120 m surface temperature-60 m auxiliary parameter dataset (denoted as 120 m-60 m dataset) and 30 m auxiliary parameter dataset are constructed.

[0159] It should be noted that, since the present application involves two stages of downscaling of low-medium resolution and medium-high resolution, in the subsequent description, the relative concepts of "coarse resolution" and "fine resolution" are used to represent the lower resolution and the higher resolution in the two stages of downscaling.

[0160] In step S104, a multi-parameter fusion-based surface temperature downscaling network is constructed according to the multi-scale training dataset, and a domain transformation network based on heterogeneous high-frequency information guidance is constructed according to the multi-scale training dataset.

[0161] In actual execution process, for example, Figure 3As shown, the embodiment of the present application can construct a land surface temperature downscaling network based on multi-parameter fusion according to a multi-scale training data set, and construct a domain transformation network based on heterogeneous high-frequency information guidance according to a multi-scale training data set, adopt a CNN-Transformer hybrid architecture to construct a downscaling network and a domain transformation network for joint training, break through the limitations of a single model, take into account global feature correlation and local texture fidelity, and solve the problem of detail distortion caused by insufficient heterogeneous data fusion.

[0162] Optionally, in an embodiment of the present application, the land surface temperature downscaling network based on multi-parameter fusion is constructed according to a multi-scale training data set, comprising: extracting preliminary features from the land surface temperature satisfying the preset coarse resolution condition and the auxiliary land surface parameter satisfying the preset fine resolution condition; constructing a multi-scale feature extraction module, and extracting multi-scale features of the preliminary features of the auxiliary land surface parameter satisfying the preset fine resolution condition by using the multi-scale feature extraction module; constructing a lightweight CNN module, and extracting local features in the land surface temperature satisfying the preset coarse resolution condition and the multi-scale features by using the lightweight CNN module, and performing channel compression on the local features to generate a compression result; constructing a lightweight Transformer module, and extracting features of the compression result by using the lightweight Transformer module to generate global features; constructing a downscaling reconstruction module, and processing the global features by using the downscaling reconstruction module to generate the downscaling land surface temperature.

[0163] It can be understood that the preset coarse resolution condition and the preset fine resolution condition in the embodiment of the present application correspond to the lower resolution and the higher resolution in the two downscaling stages respectively.

[0164] Specifically, the embodiment of the present application can include the following steps:

[0165] (1) extracting preliminary features from the input coarse resolution land surface temperature and fine resolution auxiliary land surface parameters by convolution respectively, wherein, in an embodiment of the present application, the extraction formula of the preliminary features is:

[0166]

[0167]

[0168] wherein, and represent the preliminary features of the land surface temperature satisfying the preset coarse resolution condition and the land surface parameter satisfying the preset fine resolution condition respectively, represents the land surface temperature satisfying the preset coarse resolution condition, represents a channel addition operation, represents a convolution operation, , and , are weight terms and bias terms of two convolution operations respectively, is a digital elevation model, is a normalized vegetation index, is a normalized water index, is a normalized building index.

[0169] (2) A multi-scale feature extraction module is constructed, which extracts features of the subdivided resolution land surface parameters at three scales by using convolution kernels with sizes of , , , , , , for example, the module extracts features of the subdivided resolution land surface parameters at three scales by using convolution kernels with sizes of , , , , , , combines features of adjacent scales and considers channel and spatial correlation between cross-scale features at the same time, the module is embedded with a convolution attention module to dynamically optimize channel dimension and spatial dimension weights of the feature map, wherein the multi-scale convolution kernel size , , ; the construction formula of the multi-scale feature is:

[0170]

[0171] wherein, is an output feature map of the multi-scale feature, represents a convolution attention operation, represents a feature map obtained by combining features obtained by convolution with sizes of , and , , represents a feature map obtained by combining features obtained by convolution with sizes of , and , ;

[0172] and , that is, the calculation formula of the feature map is:

[0173]

[0174]

[0175] wherein, represents a Relu activation function, , , , and , are weight terms and bias terms in a convolution operation with a size of , , , , , , , , , and , are weight terms and bias terms in a convolution operation with a size of , , , , , .

[0176] (3) Constructing a lightweight CNN module. The high-frequency filtering module is used to extract high-frequency information of the image, and the residual feature block is used to further extract local features. The module effectively retains the image details while reducing the feature map through convolution operation. The convolution attention module is embedded in the module. The lightweight CNN module is used to extract local features from the input coarse resolution land surface temperature and the multi-scale features extracted in step (2) respectively, and the local features are superimposed on the channel. Convolution operation is applied for channel compression to generate compressed convolution results. The extraction formula of local features and the calculation formula of channel compression are as follows:

[0177]

[0178]

[0179]

[0180] wherein, and are local features obtained by the lightweight CNN module processing the land surface temperature and the multi-scale features that meet the preset coarse resolution condition, is the feature map obtained by compressing the output of the two lightweight CNN architectures in the channel, represents a 2 times upsampling operation, , and , are weight terms and bias terms of the two lightweight CNN modules, and are the weight term and the bias term of the channel compression convolution operation, respectively.

[0181] (4) Constructing a lightweight Transformer module. Two efficient Transformer modules are used to capture long-term dependencies in local regions with lower computational cost. The efficient Transformer uses the encoder structure of the standard Transformer and uses an efficient multi-head attention mechanism to block compute Q, K, and V to improve computational efficiency. The lightweight Transformer module extracts features from the output of step (3) and fully excavates the global autocorrelation information. The calculation formula of the lightweight Transformer module is:

[0182]

[0183] wherein, is the output feature map of the lightweight Transformer module, is the feature map compressed in the channel after the output of the two lightweight CNN architectures, is the number of channel partitions, denotes the scaling factor, , , denotes the matrix obtained by linear projection from the input feature map;

[0184] The calculation formula of the matrix is:

[0185]

[0186]

[0187]

[0188] wherein, wherein is the partition operation, in the partition operation ;

[0189] (5) Constructing a downscale reconstruction module. A three-channel convolution and a single-channel convolution are used to process the features output by step (4) to obtain the down-scaled land surface temperature, and a residual connection is used to introduce the up-sampled result of the coarse resolution land surface temperature to prevent information loss. The calculation formula of the down-scaled land surface temperature is:

[0190]

[0191] wherein, is the land surface temperature output by the downscale reconstruction module, represents a ground temperature satisfying a preset coarse resolution condition, represents a 2 times up-sampling operation, , and , respectively represent weight terms and bias terms of single-channel convolution and three-channel convolution.

[0192] Optionally, in an embodiment of the present application, the domain transformation network based on heterogeneous high-frequency information guidance is constructed according to a multi-scale training data set, comprising: extracting ground temperature high-frequency information from the down-scaled ground temperature by using Laplacian high-pass filtering; extracting local features from the ground temperature high-frequency information by using a lightweight CNN module; extracting global features from the local features by using a lightweight Transformer module; based on the global features, processing the feature map by using single-channel convolution, and adopting residual connection to construct the domain transformation network based on heterogeneous high-frequency information guidance.

[0193] wherein the construction of the domain transformation network based on heterogeneous high-frequency information guidance comprises:

[0194] (1) using Laplacian high-pass filtering to extract high-frequency information from the ground temperature down-scaling result obtained in the construction of the down-scaling reconstruction module, wherein in an embodiment of the present application, the ground temperature high-frequency information extraction formula in the domain transformation network is:

[0195]

[0196] wherein, represents the ground temperature high-frequency information extraction result, represents Laplacian high-pass filtering, and the filter kernel is a matrix , is the ground temperature output by the down-scaling reconstruction module;

[0197] the matrix is defined as:

[0198]

[0199] (2) constructing a lightweight CNN module to extract local features from the ground temperature high-frequency information obtained in the above step (1), which can be represented as:

[0200]

[0201] wherein, represents the local features extracted by the module from the ground temperature high-frequency information, represents the lightweight CNN module, represents the local features extracted from the ground temperature high-frequency information;

[0202] (3) Construct a lightweight Transformer module to further extract global features from the local features extracted in step (1) above, which can be represented as:

[0203]

[0204] wherein, represents the global features obtained by the module, represents the lightweight Transformer module, represents the local features extracted from the surface temperature high-frequency information;

[0205] (4) Construct a high-frequency information reconstruction module, use a single-channel convolution to process the feature map obtained in step (2) above, and use residual connection to reduce information loss. The high-frequency information reconstruction formula in the domain transformation network is:

[0206]

[0207] wherein, is the NDVI high-frequency information prediction result obtained after domain transformation, and respectively represent the weight term and the bias term of the single-channel convolution, represents the global features extracted from the local features.

[0208] The embodiment of the present application can use the CNN-Transformer hybrid architecture to construct the joint training of the downscaling network and the domain transformation network, break through the limitation of a single model, consider both global feature correlation and local texture fidelity, and solve the problem of detail distortion caused by insufficient heterogeneous data fusion.

[0209] In step S105, based on the progressive double-network joint training strategy, the surface temperature downscaling network and the domain transformation network are trained using the training data set in the multi-scale training data set that meets the preset low-mid resolution condition, to generate a downscaling model that meets the preset low-mid resolution condition and a downscaling result that meets the preset mid-resolution condition.

[0210] In actual implementation, the embodiment of the present application can adopt a progressive double-network joint training strategy to construct a multi-scale cross-modal loss function to jointly train the downscaling network and the domain transformation network. The downscaling network and the domain transformation network constructed in step S104 are trained to convergence using the low-to-medium resolution training dataset constructed in step S103, to obtain a low-to-medium resolution downscaling model and a medium resolution downscaling result. For example, the network constructed in step S104 is trained to convergence using the 120 m - 60 m dataset constructed in step S103, to obtain a 120 m - 60 m downscaling model and a 60 m downscaling result. The 60 m - 30 m downscaling model and the 30 m downscaling result are obtained by fine-tuning the 120 m - 60 m using the 60 m downscaling result and the 30 m auxiliary parameter dataset constructed in step S103.

[0211] (1) Specifically, a multi-scale cross-modal loss function is constructed to jointly train the downscaling network and the domain transformation network. The multi-scale cross-modal loss function of the downscaling network is composed of three parts, including a coarse resolution land surface temperature fidelity loss for maintaining the accuracy of the background information of the land surface temperature, a fine resolution frequency domain consistency loss for maintaining the richness of the spatial details of the downscaling, and a cross-modal frequency domain alignment loss for further enhancing the expression ability of the spatial detail information, wherein the cross-modal frequency domain alignment loss is also the loss function of the domain transformation network. The loss functions of the downscaling network and the domain transformation network are respectively defined as follows, wherein, in an embodiment of the present application, the calculation formulas of the land surface temperature downscaling network and the domain transformation network are respectively:

[0212]

[0213]

[0214] wherein, and respectively represent the loss functions of the land surface temperature downscaling network and the domain transformation network, represents the land surface temperature fidelity loss meeting the preset coarse resolution condition, represents the frequency domain consistency loss meeting the preset fine resolution condition, represents the cross-modal frequency domain alignment loss, is a neural network parameter, are all empirical coefficients for adjusting the weights of the sub-loss functions, and the empirical coefficients are respectively set to 0.7, 0.3, and 1 in low-to-medium resolution downscaling, and are respectively set to 0.7, 0, and 1 in medium-to-high resolution downscaling;

[0215] The calculation formula of the ground temperature fidelity loss meeting the preset coarse resolution condition is:

[0216]

[0217] wherein, represents the number of training image pairs, represents the first coarse resolution ground temperature image in the image pair, represents network 1, represents resampling to the coarse resolution data scale;

[0218] The calculation formula of the frequency domain consistency loss meeting the preset fine resolution condition is:

[0219]

[0220] wherein, represents the number of training image pairs, represents Laplacian high-pass filtering, represents the first reference fine resolution ground temperature image in the image pair;

[0221] The calculation formula of the cross-modal frequency domain alignment loss is:

[0222]

[0223] wherein, represents network 1, represents network 2, represents the first fine resolution NDVI image in the image pair, represents the first coarse resolution ground temperature image in the image pair.

[0224] (2) The low-mid resolution training data set and the multi-scale cross-modal loss function constructed by the above steps are used to jointly train the downscaling and domain transformation network to convergence, and a trained low-mid resolution downscaling model is obtained, and a mid resolution downscaling result can be obtained by the model.

[0225] The embodiment of the application constructs a resolution adaptive progressive processing link through a phased progressive training strategy, avoids model migration failure caused by distribution difference of different resolution data, and enables the algorithm to maintain downscaling accuracy in a multi-level resolution scene, and is especially suitable for large area multi-scale remote sensing data analysis.

[0226] In step S106, the downscaling model that satisfies the preset low-to-medium resolution condition is fine-tuned using the dataset that satisfies the preset medium-to-high resolution condition in the multi-scale training dataset, so as to generate a downscaling model that satisfies the preset medium-to-high resolution condition and a downscaling result that satisfies the preset high resolution condition.

[0227] In actual implementation, the embodiments of the present invention can use medium-high resolution datasets to fine-tune the low-medium resolution downscaling model to obtain medium-high resolution downscaling model and high resolution downscaling results. That is, the embodiments of the present invention can use the medium resolution downscaling results obtained in the above steps combined with the high resolution auxiliary parameter dataset constructed in the above steps to fine-tune the low-medium resolution downscaling model obtained in step (2) to obtain medium-high resolution downscaling model, so as to avoid the scale inconsistency problem caused by direct model transfer, and thus obtain high resolution downscaling results.

[0228] In this embodiment of the invention, the 120 m - 60 m training dataset constructed in step S103 is used to jointly train the downscaling and domain transformation network constructed in step S104 until convergence using a multi-scale cross-modal loss function, resulting in a trained 120 m - 60 m downscaling model. This model can then yield a 60 m downscaling result. The 60 m downscaling result is then combined with a 30 m auxiliary parameter dataset to fine-tune the 120 m - 60 m downscaling model, resulting in a 60 m - 30 m downscaling model. This avoids the scaling inconsistency problem caused by direct model transfer, thus obtaining a 30 m downscaling result.

[0229] In step S107, based on the downscaling model that satisfies the preset low-to-medium resolution conditions, the downscaling result that satisfies the preset medium resolution conditions, the downscaling model that satisfies the preset medium-to-high resolution conditions, and the downscaling result that satisfies the preset high resolution conditions, the low-resolution land surface temperature image to be processed is downscaled in a low-to-medium-to-high progressive manner to generate a multi-scale progressive land surface temperature that satisfies the target high resolution conditions.

[0230] In practical implementation, embodiments of the present invention can utilize the downscaling network trained and fine-tuned in step S106 to perform low-to-medium-to-high progressive downscaling on the low-resolution land surface temperature image to obtain high-resolution land surface temperature. Embodiments of the present invention can also utilize the low-to-medium resolution downscaling model trained in step S106 to downscale the low-resolution land surface temperature image to obtain medium-resolution land surface temperature, and then use the medium-to-high resolution downscaling model to downscale the medium-resolution land surface temperature to obtain high-resolution land surface temperature.

[0231] Among them, such as Figure 4As shown, in this embodiment of the invention, the 120 m-60 m downscaling model trained in steps S105 and S106 can be used to downscale the 120 m surface temperature image to be processed to obtain a 60 m surface temperature, and then the 60 m-30 m downscaling model can be used to downscale the 60 m surface temperature to obtain a 30 m surface temperature.

[0232] This invention provides a progressive unsupervised downscaling method for multi-scale land surface temperature based on heterogeneous high-frequency constraints to obtain high-resolution land surface temperature images that reflect the true thermal state of the land surface. It also develops an unsupervised downscaling model for the original high-resolution scale to fit the nonlinear relationship between land surface temperature and auxiliary parameters, and uses high-frequency information of high-resolution heterogeneous auxiliary parameters as constraints to achieve fine downscaling.

[0233] The multi-scale progressive land surface temperature fusion and downscaling method proposed in this invention can construct a multi-scale dataset containing land surface temperature and multi-source auxiliary parameters using multi-source remote sensing data. At a low-resolution scale, a land surface temperature downscaling network is used to fuse and downscale heterogeneous features to obtain a medium-resolution land surface temperature. High-frequency information is extracted from the medium-resolution result, and a domain transformation network is used to transform the high-frequency information of the heterogeneous data to enhance the spatial detail representation. At a high-resolution scale, the network is fine-tuned to address the scale difference problem and applied to the medium-resolution downscaling result to obtain a high-resolution land surface temperature. This invention fully considers the downscaling differences between different resolutions, preserving the physical properties of the land surface thermal field and improving the processing efficiency of large-area remote sensing data. Therefore, it solves the problem that existing deep learning downscaling methods ignore the local heterogeneity caused by the scale difference between high and low resolutions, which limits the accuracy of the downscaling results.

[0234] Next, referring to the accompanying drawings, a multi-scale progressive surface temperature fusion downscaling device proposed according to an embodiment of the present invention is described.

[0235] Figure 5 This is a schematic diagram of the structure of the multi-scale progressive surface temperature fusion downscaling device according to an embodiment of the present invention.

[0236] like Figure 5 As shown, the multi-scale progressive surface temperature fusion downscaling device 10 includes: an acquisition module 100, a preprocessing module 200, a data construction module 300, a network construction module 400, a training module 500, a generation module 600, and a downscaling processing module 700.

[0237] Specifically, the acquisition module 100 is used to acquire at least one of the following products: a surface temperature product of the target satellite that meets the preset low-resolution conditions, a surface temperature product of the reference satellite that meets the preset medium-resolution conditions, and an auxiliary parameter product of the reference satellite that meets the preset high-resolution conditions.

[0238] The preprocessing module 200 is configured to resample at least one product to generate a multi-scale image library corresponding to a resolution of the at least one product, and preprocess a land surface temperature image in the multi-scale image library to generate a preprocessed image.

[0239] The data construction module 300 is configured to construct a multi-scale training data set containing land surface temperature and at least one auxiliary parameter by using the preprocessed image.

[0240] The network construction module 400 is configured to construct a land surface temperature downscaling network based on multi-parameter fusion according to the multi-scale training data set, and construct a domain transformation network based on heterogeneous high-frequency information guidance according to the multi-scale training data set.

[0241] The training module 500 is configured to train the land surface temperature downscaling network and the domain transformation network by using training data sets in the multi-scale training data set satisfying a preset low-to-medium resolution condition based on a progressive double-network joint training strategy, to generate a downscaling model satisfying the preset low-to-medium resolution condition and a downscaling result satisfying a preset medium resolution condition.

[0242] The generation module 600 is configured to fine-tune the downscaling model satisfying the preset low-to-medium resolution condition by using data sets in the multi-scale training data set satisfying a preset medium-to-high resolution condition, to generate a downscaling model satisfying the preset medium-to-high resolution condition and a downscaling result satisfying a preset high resolution condition.

[0243] The downscaling processing module 700 is configured to perform low-to-medium-to-high progressive downscaling on a to-be-processed low-resolution land surface temperature image based on the downscaling model satisfying the preset low-to-medium resolution condition, the downscaling result satisfying the preset medium resolution condition, the downscaling model satisfying the preset medium-to-high resolution condition, and the downscaling result satisfying the preset high resolution condition, to generate a multi-scale progressive land surface temperature satisfying a target high resolution condition.

[0244] Optionally, in an embodiment of the present application, the preprocessing module 200 includes a calculation unit, a correction unit, and a preprocessing unit.

[0245] The calculation unit is configured to calculate a normalized vegetation index, a normalized water index, and a normalized building index by using reflectivity images of a target satellite.

[0246] The correction unit is configured to perform statistical property correction on land surface temperature data of a reference satellite to generate a land surface temperature image of the reference satellite satisfying a target resolution condition.

[0247] The preprocessing unit is configured to preprocess the land surface temperature image of the target satellite, the land surface temperature image of the reference satellite, and the digital elevation model image, the normalized vegetation index image, the normalized water index image, and the normalized building index image to generate preprocessed images.

[0248] Optionally, in an embodiment of the present application, the calculation formula of the land surface temperature image of the reference satellite satisfying the target resolution condition is:

[0249]

[0250] wherein, represents the mid-resolution land surface temperature image of the reference satellite after histogram matching, represents the original mid-resolution land surface temperature image of the reference satellite, and respectively represent the standard deviation of the low-resolution land surface temperature image of the target satellite and the original mid-resolution land surface temperature image of the reference satellite, and respectively represent the mean value of the low-resolution land surface temperature image of the target satellite and the original mid-resolution land surface temperature image of the reference satellite.

[0251] Optionally, in an embodiment of the present application, the data construction module 300 comprises a clipping unit, a first generation unit, and a second generation unit.

[0252] The clipping unit is configured to clip the multi-scale image library into at least one group of image blocks.

[0253] The first generation unit is configured to generate a mid-high resolution data set in the multi-scale training data set based on a target satellite land surface temperature image block satisfying a preset low resolution condition, a reference satellite land surface temperature image block satisfying a preset mid resolution condition, a digital elevation model image block satisfying a preset mid resolution condition, a normalized vegetation index image block, a normalized water index image block, and a normalized building index image block in the at least one group of image blocks.

[0254] The second generation unit is configured to generate a high resolution auxiliary parameter data set in the multi-scale training data set based on a digital elevation model image block satisfying a preset high resolution condition, a normalized vegetation index image block, a normalized water index image block, and a normalized building index image block in the at least one group of image blocks.

[0255] Optionally, in an embodiment of the present application, the network construction module 400 comprises a feature extraction unit, a multi-scale feature extraction unit, a first local feature extraction unit, a first global feature extraction unit, and a first reconstruction unit.

[0256] The feature extraction unit is configured to extract preliminary features from the land surface temperature satisfying the preset coarse resolution condition and the auxiliary land surface parameter satisfying the preset fine resolution condition.

[0257] The multi-scale feature extraction unit is configured to construct a multi-scale feature extraction module and extract multi-scale features of the preliminary features of the auxiliary land surface parameter satisfying the preset fine resolution condition by using the multi-scale feature extraction module.

[0258] The first local feature extraction unit is configured to construct a lightweight CNN module, extract local features from the land surface temperature satisfying the preset coarse resolution condition and the multi-scale features by using the lightweight CNN module, and perform channel compression on the local features to generate a compression result.

[0259] The first global feature extraction unit is configured to construct a lightweight Transformer module, perform feature extraction on the compression result by using the lightweight Transformer module, and generate global features.

[0260] The first reconstruction unit is configured to construct a downscale reconstruction module, process the global features by using the downscale reconstruction module, and generate a downscale land surface temperature.

[0261] Optionally, in an embodiment of the present application, the extraction formula of the preliminary features is as follows:

[0262]

[0263]

[0264] wherein, and respectively represent the preliminary features of the land surface temperature satisfying the preset coarse resolution condition and the land surface parameter satisfying the preset fine resolution condition, represents the land surface temperature satisfying the preset coarse resolution condition, represents a channel addition operation, represents a convolution operation, , and , respectively are a weight term and a bias term of two convolution operations, is a digital elevation model, is a normalized vegetation index, is a normalized water index, is a normalized building index.

[0265] The construction formula of the multi-scale features is as follows:

[0266]

[0267] wherein, is an output feature map of the multi-scale feature, represents a convolution attention operation, represents a convolution operation with a size of x and x a feature map obtained by combining features obtained by convolution, represents a convolution operation with a size of x and x a feature map obtained by combining features obtained by convolution;

[0268] The calculation formulae of the feature maps are respectively:

[0269]

[0270]

[0271] wherein, represents a Relu activation function, , , , and , are weight items and bias items in the convolution operations with a size of x , x , x ,

[0272] The extraction formula of the local feature and the calculation formula of the channel compression are respectively:

[0273]

[0274]

[0275]

[0276] wherein, and are local features obtained by processing the ground surface temperature and the multi-scale feature that meet the preset coarse resolution condition by the lightweight CNN module, is a feature map obtained by compressing the outputs of the two lightweight CNN architectures in the channel, represents a 2x upsampling operation, , and , respectively represent weight terms and bias terms of the two lightweight CNN modules, and respectively represent weight terms and bias terms of the channel compression convolution operation;

[0277] The calculation formula of the lightweight Transformer module is:

[0278]

[0279] wherein, is an output feature map of the lightweight Transformer module, is a feature map compressed in channels from the output of the two lightweight CNN architectures, is a channel segmentation number, represents a scaling factor, , , represents a matrix obtained by linear projection from an input feature map;

[0280] The calculation formula of the matrix is:

[0281]

[0282]

[0283]

[0284] wherein, wherein is a segmentation operation;

[0285] The calculation formula of the down-scaled surface temperature is:

[0286]

[0287] wherein, is a surface temperature output by the down-sampling reconstruction module, represents the surface temperature satisfying the preset coarse resolution condition, represents a 2x up-sampling operation, , and , respectively represent weight terms and bias terms of the single-channel convolution and the three-channel convolution.

[0288] Optionally, in an embodiment of the present application, the network construction module 400 comprises an information extraction unit, a local feature extraction unit, a global feature extraction unit and a processing unit.

[0289] The information extraction unit is configured to extract surface temperature high-frequency information from the down-scaled surface temperature by using Laplacian high-pass filtering.

[0290] The second local feature extraction unit is configured to extract local features from the surface temperature high-frequency information by using a lightweight CNN module.

[0291] The second global feature extraction unit is configured to extract global features from the local features by using a lightweight Transformer module.

[0292] The second reconstruction unit is configured to process the feature map by using a single-channel convolution based on the global features, and adopt residual connection to construct a domain transformation network based on heterogeneous high-frequency information guidance.

[0293] Optionally, in an embodiment of the present application, the surface temperature high-frequency information extraction formula is as follows:

[0294]

[0295] wherein, represents the surface temperature high-frequency information extraction result, represents Laplacian high-pass filtering, and the filter kernel is a matrix , is the surface temperature output by the down-scaled reconstruction module;

[0296] The matrix is defined as:

[0297]

[0298] The calculation formula of the local features is as follows:

[0299]

[0300] wherein, represents the local features extracted from the surface temperature high-frequency information, represents the lightweight CNN module, represents the local features extracted from the surface temperature high-frequency information;

[0301] The calculation formula of the global features is as follows:

[0302]

[0303] wherein, represents the global features extracted from the local features, represents the lightweight Transformer module, represents the local features extracted from the surface temperature high-frequency information;

[0304] The high-frequency information reconstruction formula in the domain transformation network is:

[0305]

[0306] wherein, is the NDVI high-frequency information prediction result obtained after domain transformation, and respectively represent the weight term and the bias term of single-channel convolution, represents the global feature extracted from the local feature.

[0307] Optionally, in an embodiment of the present application, the calculation formulas of the land surface temperature downscaling network and the domain transformation network are respectively:

[0308]

[0309]

[0310] wherein, and respectively represent the loss functions of the land surface temperature downscaling network and the domain transformation network, represents the land surface temperature fidelity loss meeting the preset coarse resolution condition, represents the frequency domain consistency loss meeting the preset fine resolution condition, represents the cross-modal frequency domain alignment loss, is a neural network parameter, both are empirical coefficients for adjusting the weights of each sub-loss function;

[0311] The calculation formula of the land surface temperature fidelity loss meeting the preset coarse resolution condition is:

[0312]

[0313] wherein, represents the number of training image pairs, represents the i-th image pair, represents the coarse resolution land surface temperature image in the image pair, represents network 1, represents resampling to the coarse resolution data scale;

[0314] The calculation formula of the frequency domain consistency loss meeting the preset fine resolution condition is:

[0315]

[0316] wherein, represents the number of training image pairs, represents a Laplacian high-pass filter, represents the first a reference fine resolution land surface temperature image in the image pair;

[0317] The formula for calculating the cross-modal frequency domain alignment loss is:

[0318]

[0319] wherein, represents the network 1, represents network 2, represents the first a fine resolution NDVI image in the image pair, represents the first a coarse resolution land surface temperature image in the image pair.

[0320] It should be noted that the foregoing explanation and description of the multi-scale progressive land surface temperature fusion downscaling method embodiment also applies to the multi-scale progressive land surface temperature fusion downscaling device of this embodiment, which will not be described here.

[0321] The multi-scale progressive land surface temperature fusion downscaling device according to the embodiment of the present application can construct a multi-scale data set containing land surface temperature and multi-source auxiliary parameters using multi-source remote sensing data. At a low resolution scale, heterogeneous feature fusion downscaling is performed using a land surface temperature downscaling network to obtain a medium resolution land surface temperature. High-frequency information is extracted from the medium resolution result, and domain transformation network is used to convert the high-frequency information of heterogeneous data to enhance the expression of spatial details. At a high resolution scale, the network is fine-tuned to address the scale difference problem, and is applied to the medium resolution downscaling result to obtain a high resolution land surface temperature. The present application fully considers the downscaling difference problem at different resolutions, realizes the preservation of the physical properties of the land thermal field, and improves the efficiency of processing large-area remote sensing data. Thus, the problem of ignoring the scale difference between high and low resolutions in the existing deep learning downscaling method, which leads to local heterogeneity and limits the accuracy of the downscaling result, is solved.

[0322] In the description of the application, reference can be made to terms such as "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. It is intended that there are at least one embodiment or example where the particular feature, structure, material or characteristic described is included in at least one embodiment or example of the application. These described

[0323] Also, the terms "first", "second" and the like in the description do not necessarily connote an absolute order or quantity, but are used to modify an otherwise identical element. Further, the aforementioned terms "first" and "second" are used anally for description and do not imply or connote any meaning of relative importance or imply a number of technical features indicated. Thus, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise explicitly and specifically limited.

[0324] Any process or method descriptions or descriptions of the flow diagrams in the present application can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for performing specific logic functions or steps in the process, and that the various systems described herein can include one or more circuits, or circuitry, that include the various gates, circuit elements, or other elements in the form of hardware or hardware and software in combination.

Claims

1. A multi-scale progressive surface temperature fusion downscaling method, characterized in that, Includes the following steps: Acquire at least one of the following: a surface temperature product of the target satellite that meets preset low-resolution conditions, a surface temperature product of the reference satellite that meets preset medium-resolution conditions, and an auxiliary parameter product of the reference satellite that meets preset high-resolution conditions. The at least one product is resampled to generate a multi-scale image library corresponding to the resolution of the at least one product, and the land surface temperature images in the multi-scale image library are preprocessed to generate preprocessed images. A multi-scale training dataset containing land surface temperature and at least one auxiliary parameter is constructed using the preprocessed images; A land surface temperature downscaling network based on multi-parameter fusion is constructed based on the multi-scale training dataset, and a domain transformation network guided by heterogeneous high-frequency information is constructed based on the multi-scale training dataset. The construction of the domain transformation network guided by heterogeneous high-frequency information based on the multi-scale training dataset includes: extracting high-frequency land surface temperature information from the downscaled land surface temperature using Laplacian high-pass filtering; extracting local features from the high-frequency land surface temperature information using a lightweight CNN module; extracting global features from the local features using a lightweight Transformer module; and processing the feature map using single-channel convolution based on the global features, and constructing the domain transformation network guided by heterogeneous high-frequency information using residual connections. Based on the progressive dual-network joint training strategy, the surface temperature downscaling network and the domain transformation network are trained using the training dataset in the multi-scale training dataset that meets the preset low-to-medium resolution conditions, so as to generate a downscaling model that meets the preset low-to-medium resolution conditions and a downscaling result that meets the preset medium resolution conditions. The downscaling model that satisfies the preset low-to-medium resolution condition is fine-tuned using the dataset that satisfies the preset medium-to-high resolution condition in the multi-scale training dataset, so as to generate a downscaling model that satisfies the preset medium-to-high resolution condition and a downscaling result that satisfies the preset high resolution condition. Based on the downscaling model that satisfies the preset low-to-medium resolution conditions, the downscaling result that satisfies the preset medium resolution conditions, the downscaling model that satisfies the preset medium-to-high resolution conditions, and the downscaling result that satisfies the preset high resolution conditions, the low-resolution land surface temperature image to be processed is subjected to low-to-medium-to-high progressive downscaling to generate a multi-scale progressive land surface temperature that satisfies the target high resolution conditions.

2. The method according to claim 1, characterized in that, The preprocessing of the surface temperature images in the multi-scale image library to generate preprocessed images includes: The normalized vegetation index, normalized water index, and normalized building index are calculated using the reflectivity imagery of the target satellite. Statistical characteristic corrections are performed on the surface temperature data of the reference satellite to generate a surface temperature image of the reference satellite that meets the target resolution requirements. The surface temperature images of the target satellite, the surface temperature images and digital elevation models of the reference satellite, the normalized vegetation index, the normalized water index, and the normalized building index are preprocessed to generate the preprocessed images.

3. The method according to claim 2, characterized in that, The formula for calculating the surface temperature image of the reference satellite that meets the target resolution condition is as follows: in, This represents a medium-resolution reference satellite surface temperature image after histogram matching. This represents the original medium-resolution reference satellite surface temperature image. and These represent the standard deviations of the low-resolution target satellite surface temperature image and the original medium-resolution reference satellite surface temperature image, respectively. and These represent the average values ​​of the low-resolution target satellite surface temperature image and the original medium-resolution reference satellite surface temperature image, respectively.

4. The method according to claim 2, characterized in that, The construction of a multi-scale training dataset containing land surface temperature and at least one auxiliary parameter using the preprocessed images includes: The multi-scale image library is cropped into at least one set of image blocks; Based on the target satellite surface temperature image block that meets the preset low resolution condition, the reference satellite surface temperature image block that meets the preset medium resolution condition, the digital elevation model image block that meets the preset medium resolution condition, the normalized vegetation index image block, the normalized water index image block, and the normalized building index image block that meet the preset medium resolution condition, a medium-to-high resolution dataset is generated from the multi-scale training dataset. Based on the digital elevation model image blocks, normalized vegetation index image blocks, normalized water index image blocks, and normalized building index image blocks that meet the preset high resolution conditions in the at least one set of image blocks, a high-resolution auxiliary parameter dataset is generated in the multi-scale training dataset.

5. The method according to claim 1, characterized in that, The construction of a surface temperature downscaling network based on multi-parameter fusion using the multi-scale training dataset includes: Preliminary features are extracted from surface temperature that meets the preset coarse resolution conditions and auxiliary surface parameters that meet the preset fine resolution conditions. A multi-scale feature extraction module is constructed, and the multi-scale features of the preliminary features of the auxiliary surface parameters that meet the preset fine resolution conditions are extracted using the multi-scale feature extraction module. The lightweight CNN module is constructed, and the lightweight CNN module is used to extract the local features of the surface temperature and the multi-scale features that meet the preset coarse resolution conditions. The local features are then compressed to generate a compressed result. The lightweight Transformer module is constructed, and the lightweight Transformer module is used to extract features from the compression result to generate global features; A downscaling reconstruction module is constructed, and the global features are processed using the downscaling reconstruction module to generate the downscaled surface temperature.

6. The method according to claim 5, characterized in that, The formula for extracting the preliminary features is as follows: in, and These represent the preliminary characteristics of the surface temperature that meets the preset coarse resolution condition and the surface parameter that meets the preset fine resolution condition, respectively. This represents the surface temperature that meets the preset coarse resolution conditions. This indicates a channel addition operation. This represents the convolution operation. , and , These are the weights and biases for the two convolution operations, respectively. For digital elevation models, Normalized Difference Vegetation Index (NDVI) The normalized water index, The normalized building index; The formula for constructing the multi-scale features is as follows: in, The output feature map of the multi-scale features, This indicates the convolutional attention operation. Indicates size is × and × The feature map obtained by combining the features obtained from convolution. Indicates size is × and × The feature map obtained by combining the features obtained from convolution; The calculation formulas for the feature maps are as follows: in, This represents the ReLU activation function. , , , and , They are respectively the size of × , × , × The weights and biases in the convolution operation; The formulas for extracting local features and calculating channel compression are as follows: in, and These are the local features obtained by the lightweight CNN module through processing the surface temperature and multi-scale features that meet the preset coarse resolution conditions. These are the channel-compressed feature maps of the outputs of two lightweight CNN architectures. This indicates a 2x upsampling operation. , and , These represent the weight and bias terms of the two lightweight CNN modules, respectively. and These are the weight and bias terms for the channel compression convolution operation, respectively. The calculation formula for the lightweight Transformer module is as follows: in, This is the output feature map of the lightweight Transformer module. The outputs of the two lightweight CNN architectures are compressed feature maps across channels. The number of channel segments, Indicates the scaling factor. , , This represents the matrix obtained by linear projection of the input feature map; The formulas for calculating the matrix are as follows: in, ,in For the segmentation operation; The formula for calculating the downscaled surface temperature is as follows: in, The surface temperature output by the downscaling reconstruction module. This represents the surface temperature that meets the preset coarse resolution conditions. This indicates a 2x upsampling operation. , and , These are the weights and biases for single-channel and three-channel convolutions, respectively.

7. The method according to claim 1, characterized in that, The formula for extracting high-frequency surface temperature information in the domain transform network is as follows: in, This indicates the results of high-frequency information extraction of land surface temperature. This represents a Laplace high-pass filter, where the filter kernel is a matrix. , The surface temperature output by the downscaling reconstruction module; The matrix Defined as: The formula for calculating the local features is: in, This represents local features extracted from the high-frequency information of the land surface temperature. This refers to the lightweight CNN module. This represents local features extracted from the high-frequency information of the land surface temperature; The formula for calculating the global features is as follows: in, This represents the global features extracted from the local features. This refers to the lightweight Transformer module. This represents local features extracted from the high-frequency information of the land surface temperature; The formula for reconstructing high-frequency information in the domain transform network is: in, The high-frequency information prediction results of NDVI obtained after domain transformation are shown. and These represent the weight and bias terms of a single-channel convolution, respectively. This represents the global features extracted from the local features.

8. The method according to claim 1, characterized in that, The calculation formulas for the surface temperature downscaling network and the domain transformation network are as follows: in, and Let represent the loss functions of the surface temperature downscaling network and the domain transformation network, respectively. This represents the surface temperature fidelity loss under the preset coarse resolution conditions. This represents the frequency domain consistency loss that satisfies the preset fine-resolution condition. This represents the cross-modal frequency domain alignment loss. For neural network parameters, These are all empirical coefficients used to adjust the weights of each sub-loss function; The formula for calculating the surface temperature fidelity loss that meets the preset coarse resolution condition is as follows: in, This indicates the number of training image pairs. Indicates the first Coarse-resolution surface temperature images from the image pair. Represents network 1, This indicates resampling to a coarse-resolution data scale; The formula for calculating the frequency domain consistency loss that satisfies the preset fine resolution condition is as follows: in, This indicates the number of training image pairs. This indicates a Laplace high-pass filter. Indicates the first Reference fine-resolution surface temperature image in the image pair; The formula for calculating the cross-modal frequency domain alignment loss is as follows: in, This refers to network 1. Represents network 2, Indicates the first For the fine-resolution NDVI image in the image pair, Indicates the first Coarse-resolution surface temperature imagery from the image pair.

9. A multi-scale progressive surface temperature fusion downscaling device, characterized in that, The multi-scale progressive surface temperature fusion downscaling method as described in any one of claims 1-8 includes: The acquisition module is used to acquire at least one of the following products: a surface temperature product of the target satellite that meets the preset low-resolution conditions, a surface temperature product of the reference satellite that meets the preset medium-resolution conditions, and an auxiliary parameter product of the reference satellite that meets the preset high-resolution conditions. A preprocessing module is used to resample the at least one product to generate a multi-scale image library corresponding to the resolution of the at least one product, and preprocess the land surface temperature images in the multi-scale image library to generate preprocessed images. The data construction module is used to construct a multi-scale training dataset containing land surface temperature and at least one auxiliary parameter using the preprocessed images; The network construction module is used to construct a land surface temperature downscaling network based on multi-parameter fusion based on the multi-scale training dataset, and to construct a domain transformation network guided by heterogeneous high-frequency information based on the multi-scale training dataset. The construction of the domain transformation network guided by heterogeneous high-frequency information based on the multi-scale training dataset includes: extracting high-frequency land surface temperature information from the downscaled land surface temperature using a Laplacian high-pass filter; extracting local features from the high-frequency land surface temperature information using a lightweight CNN module; extracting global features from the local features using a lightweight Transformer module; and processing the feature map using single-channel convolution based on the global features, and constructing the domain transformation network guided by heterogeneous high-frequency information using residual connections. The training module is used to train the land surface temperature downscaling network and the domain transformation network based on a progressive dual-network joint training strategy, using the training dataset in the multi-scale training dataset that meets the preset low-to-medium resolution conditions, so as to generate a downscaling model that meets the preset low-to-medium resolution conditions and a downscaling result that meets the preset medium resolution conditions. The generation module is used to fine-tune the downscaling model that satisfies the preset low-to-medium resolution condition using the dataset that satisfies the preset medium-to-high resolution condition in the multi-scale training dataset, so as to generate a downscaling model that satisfies the preset medium-to-high resolution condition and a downscaling result that satisfies the preset high resolution condition. The downscaling module is used to perform low-to-medium-to-high progressive downscaling on the low-resolution land temperature image to be processed based on the downscaling model that meets the preset low-to-medium resolution conditions, the downscaling result that meets the preset medium resolution conditions, the downscaling model that meets the preset medium-to-high resolution conditions, and the downscaling result that meets the preset high resolution conditions, so as to generate a multi-scale progressive land temperature that meets the target high resolution conditions.

Images