End-to-end land surface temperature retrieval method based on multi-source heterogeneous data fusion network

Abstract

This invention discloses an end-to-end land surface temperature retrieval method based on a multi-source heterogeneous data fusion network, comprising: acquiring visible light, near-infrared, and thermal infrared remote sensing images and land surface temperature label data; constructing a dataset by combining NCEP water vapor data with spatiotemporal interpolation resolution matching; building a cross-modal retrieval model that integrates normalization processing, differential feature extraction, a dual-stream encoder, and a U-Net decoder; completing model training and testing, verifying spatial generalization ability using non-overlapping independent region data, and retraining with parameter adjustments and sample additions if the standard is not met; introducing field-observed temperature as a physical constraint to calculate residuals, freezing the main network parameters, and fine-tuning only the linear layer at the end of the decoder to optimize the model; inputting multi-source data of the area to be inverted into the fine-tuned model to achieve high-resolution and high-precision land surface temperature retrieval. This invention achieves automated retrieval without atmospheric parameters, significantly improving the absolute accuracy and spatial detail of the retrieval results, and is suitable for large-scale monitoring of complex surfaces.

Description

Technical Field

[0001] This invention relates to an end-to-end land surface temperature retrieval method based on a multi-source heterogeneous data fusion network, and belongs to the field of land surface temperature retrieval technology. Background Technology

[0002] With the accelerating pace of urbanization, ecological and environmental issues have become increasingly prominent globally, becoming a core topic of common concern for all countries. Land surface temperature (LST) is an extremely important physical quantity, a crucial parameter reflecting the thermal balance between the atmosphere and the land surface, and one of the key indicators for measuring the natural ecological environment of the Earth's surface. Furthermore, LST carries a wealth of environmental information, such as dynamic changes in land cover and water evaporation, which is essential for in-depth research on ecological environment evolution and the formulation of protection strategies. Its acquisition plays a pivotal role in areas such as urban heat island effect monitoring, crop growth assessment, ecological environment assessment, and land use change analysis.

[0003] Traditional methods for acquiring low-temperature sedimentary temperature (LST) data primarily rely on temperature measurement equipment at ground-based stations. While this method can provide some high-precision data and yield good temperature retrieval results, its high cost and data limitations (such as uneven distribution) restrict its application. Currently, with the rapid development of satellite remote sensing technology, acquiring LST data using remote sensing data has become the mainstream approach. Compared to traditional measurement methods, remote sensing technology offers advantages such as wide coverage, low cost, and all-weather observation.

[0004] However, existing inversion methods still face many challenges: First, traditional algorithms based on physical models (such as the split-window algorithm) are limited by the uncertainty of parameters. Real physical models are extremely complex, and in order to simplify calculations, algorithms often need to rely on prior parameters such as surface emissivity and atmospheric water vapor content. These parameters are greatly affected by land cover and atmospheric conditions, which can easily introduce errors, and the linearization approximation of the physical model also leads to a loss of accuracy.

[0005] Secondly, early machine learning inversion methods often employed a "two-stage" or "step-by-step" strategy. These methods typically first use algorithms to invert intermediate physical parameters, such as inverting emissivity first and then using it as input to estimate surface temperature. This step-by-step processing leads to error propagation—small deviations in the parameter inversion of the first stage are propagated and amplified in the final temperature result. Furthermore, with each stage optimized independently, the model cannot capture the global nonlinear mapping relationship between the raw observation data and surface temperature.

[0006] Therefore, leveraging the powerful feature extraction and fitting capabilities of deep learning to construct end-to-end LST retrieval networks has become a current research hotspot. This method can fuse multi-source / multi-modal data information, implicitly learn the complex radiative interaction processes between the atmosphere and the land surface, and achieve higher accuracy and robustness in land surface temperature retrieval without explicit input of physical parameters. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide an end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network, which solves the problems of large cumulative atmospheric correction error and low spatial resolution of thermal infrared data in traditional physical algorithms.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The end-to-end land surface temperature retrieval method based on a multi-source heterogeneous data fusion network includes the following steps: Step 1: Acquire multi-source heterogeneous data of the study area, including visible light, near-infrared and thermal infrared remote sensing image data and surface temperature label data. Based on NCEP atmospheric water vapor content data, perform spatiotemporal interpolation to obtain atmospheric water vapor content data that matches the spatiotemporal resolution of thermal infrared images. After preprocessing the above multi-source heterogeneous data, obtain a multi-source heterogeneous dataset, which is further divided into training set, validation set and test set. Step 2: Construct a multi-resolution cross-modal land surface temperature inversion model, including: a normalization processing module, a differential feature extraction module, a dual-stream encoder, and a U-Net decoder connected in sequence; the land surface temperature inversion model takes visible light, near-infrared and thermal infrared remote sensing image data, as well as atmospheric water vapor content data, as inputs and the inverted land surface temperature as output. Step 3: Use the training set, validation set, and test set to train, validate, and test the multi-resolution cross-modal land surface temperature inversion model constructed in Step 2 to obtain the trained land surface temperature inversion model. Step 4: Obtain multi-source heterogeneous data from another region and input it into the trained land surface temperature inversion model for forward extrapolation to generate the land surface temperature inversion results for that region. Compare the land surface temperature inversion results of that region with the actual temperature labels of that region to verify the spatial generalization ability of the trained land surface temperature inversion model. If the spatial generalization ability does not meet the requirements, return to step 3 to adjust the model parameters or increase the training set samples and retrain; otherwise, proceed to step 5; the other region does not overlap with the study area. Step 5: Introduce measured surface temperature data of the study area as physical constraints, calculate the residual between the surface temperature inversion result of the model and the measured surface temperature of the study area, and fine-tune the trained surface temperature inversion model by freezing the parameters of other parts of the model and only fine-tuning the parameters of the linear layer at the end of the U-Net decoder. Step 6: Input the multi-source heterogeneous data of the area to be inverted into the fine-tuned model to obtain high-resolution surface temperature inversion results.

[0009] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: 1. This invention establishes a direct nonlinear mapping between observation data and surface temperature through end-to-end learning, effectively avoiding the cumulative errors caused by inaccurate atmospheric correction and emissivity estimation in traditional physical algorithms, and significantly improving the inversion accuracy.

[0010] 2. This invention utilizes multi-source data fusion technology to reconstruct low-resolution thermal infrared information into high-resolution surface temperature distribution, solving the problem of low resolution in thermal infrared images and achieving refined monitoring.

[0011] 3. This invention, by deeply exploring the complementary relationship between multispectral texture and thermal information, has stronger robustness in the complex urban land cover domain and improves the inversion accuracy under complex land cover.

[0012] 4. This invention transforms originally blurry, low-resolution thermal data into high-resolution products that can identify ground feature details, thereby improving the usability of surface temperature data in resource surveys and disaster monitoring.

[0013] 5. This invention can be widely applied in various fields such as urban heat island monitoring, underground heat source identification, early warning of geological disasters, and agricultural and forestry ecological monitoring, providing a theoretical basis and technical support for high-precision surface temperature inversion and intelligent diagnosis of thermal anomalies from remote sensing images. Compared with traditional manual interpretation methods, it has advantages such as high precision, high efficiency, and high degree of automation, strong engineering practicality, and broad application and promotion value. Attached Figure Description

[0014] Figure 1 This is a flowchart of the end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to the present invention; Figure 2 This is a diagram of the model architecture proposed in this invention; Figure 3 This is a schematic diagram of the differential feature extraction module and residual module proposed in this invention; Figure 4 This is a schematic diagram of the dual-stream encoder and FFM module proposed in this invention; Figure 5 It is a comparison image between the predicted image and the real image; Figure 6 This is a surface temperature inversion map of a representative region of tropical plateau climate. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] like Figure 1 As shown, this invention proposes an end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network, comprising the following steps: Step 1: Acquire more than 20,000 multi-source remote sensing images of the study area at different resolutions, including multiple spectral channels such as visible light, near infrared, and thermal infrared; the images are from satellites and different sensors, including domestic satellites and Landsat satellites, and there are differences in spatial resolution; perform geometric registration, resampling and cropping operations on the data to obtain the dataset required for model training, and divide it into training set, validation set and test set; Step 1.1: Acquire visible light and near-infrared data (4 bands: red, green, blue and near-infrared bands) with a spatial resolution of 8m from the ZY-1 02D satellite; acquire thermal infrared data (2 bands) and sensor header files with a spatial resolution of 30m from the Landsat satellite; Step 1.2: Obtain NCEP data and calculate atmospheric water vapor content data through temporal and spatial interpolation using the header file; the specific implementation steps are as follows: Atmospheric water vapor content data was downloaded from the NCEP website. Time interpolation was then performed on the data to obtain the water vapor content at the time the data was captured. The formula is as follows: , in, This indicates the water vapor content at the time of Landsat satellite imagery. and These represent the NCEP water vapor content before and after the Landsat image capture time, respectively. Indicates the time when the Landsat image was captured. and These represent the NCEP recording times before and after the Landsat image capture time, respectively; Spatial interpolation was performed on the temporally interpolated atmospheric water vapor content data, and the data was interpolated to the study area to obtain atmospheric water vapor content data corresponding to the Landsat thermal infrared image.

[0017] Step 1.3: Perform geometric registration, upsampling, and other preprocessing on the above remote sensing images, and crop them using a sliding window according to a preset fixed size to generate image patches of a fixed size; among them, the visible light and near-infrared images are cropped to 480×480, the Landsat thermal infrared and atmospheric water vapor images are cropped to 128×128, and the upsampled temperature label data is cropped to 480×480. The combined images are then used to obtain the dataset. Step 1.4: Divide the collected images into a test set using an 8:2 ratio, and then use k-fold cross-validation to divide the dataset into a training set and a validation set. The dataset used during model training is called the training set. After model training, it is necessary to test the model's prediction performance on some new samples. The new samples used at this time are called the test set, and the validation set is used to adjust hyperparameters.

[0018] Step 2, as follows Figure 2 As shown, a multi-resolution cross-modal land surface temperature inversion model (MR-CMF) is constructed, which includes a normalization processing module, a differential feature extraction module, a dual-stream encoder, a multi-scale fusion module, and a U-Net decoder. Step 2.1: For the input visible-near infrared, thermal infrared, and atmospheric water vapor data, construct a differentiated feature extraction module: A residual module employing a fusion channel attention mechanism is used to extract shallow texture and radiation features of each modality at the original resolution while preserving the original feature information; an SE channel attention mechanism is introduced to adaptively weight the feature channels, enabling the model to more accurately focus on effective information; the specific structure is as follows... Figure 3 As shown (the left side is the differential feature extraction module, and the right side is the residual module): In response to the significant heterogeneity of high-resolution optical (including visible and near-infrared bands), thermal infrared, and atmospheric water vapor data in terms of spatial resolution and physical properties, this invention uses the aforementioned residual module to construct a differential feature extraction network.

[0019] The high-resolution optical branch processing consists of a high-resolution image composed of one visible light band and one near-infrared band. It employs a residual module, which utilizes its identity mapping property to preserve high-frequency detail information in the original image to the maximum extent. At the same time, it combines an SE channel attention module to adaptively adjust the feature weights of each spectral channel, thereby achieving accurate perception of complex land cover types. The thermal infrared branch processes thermal infrared images in one band. To address the regional smoothness of the thermal infrared images, dilated convolution is introduced to expand the receptive field. Without reducing resolution, it captures the continuous pattern of thermal radiation within a fixed area; and the SE module is used to filter high signal-to-noise ratio channels to extract pure physical radiation features. The atmospheric water vapor branch processes single-channel atmospheric water vapor data. It employs a residual module with a 7×7 large convolutional kernel. Unlike the sparse dilatation sampling of the thermal infrared branch, the large convolutional kernel performs dense weighted integration on the local neighborhood. By increasing the effective receptive field, it effectively models the global background trend of atmospheric water vapor at a macroscopic level.

[0020] Step 2.2: Construct a parallel dual-stream encoder. Unlike traditional methods that force a uniform resolution, this structure allows two branches to run independently at their respective original resolutions: the high-resolution stream processes visible light and near-infrared data, focusing on capturing details of surface texture; the low-resolution stream targets thermal infrared and atmospheric water vapor data, focusing on extracting macroscopic thermophysical properties. Figure 4 This is a schematic diagram of the dual-stream encoder and FFM module structure.

[0021] The dual-stream encoder includes a high-resolution stream branch, a low-resolution stream branch, a cross-attention module, and first to fourth feature fusion modules. The high-resolution stream branch includes first to fourth encoder modules connected in sequence, and the low-resolution stream branch includes fifth to eighth encoder modules connected in sequence. The input to the first encoder module is the high-resolution feature output from the differential feature extraction module, and the input to the fifth encoder module is the low-resolution feature output from the differential feature extraction module. The outputs of the first and fifth encoder modules serve as the input to the first feature fusion module, the outputs of the second and sixth encoder modules serve as the input to the second feature fusion module, the outputs of the third and seventh encoder modules serve as the input to the third feature fusion module, the outputs of the fourth and eighth encoder modules serve as the input to the fourth feature fusion module, and the output of the fourth feature fusion module and the outputs of the fourth and eighth encoder modules serve as the input to the cross-attention module. The output of the cross-attention module is then used as the output of the dual-stream encoder.

[0022] Step 2.3: To achieve efficient interaction between high-resolution optical features and low-resolution environmental features, a Feature Fusion Module (FFM) is employed in each stage of the dual-stream encoder described above. The core idea of ​​FFM is to capture the global context through horizontal and vertical pooling, while preserving local details using dilated convolutions.

[0023] set up The input feature map has a dimension of . The specific steps are as follows: Step 1: Feature Preprocessing (Dilated Convolution) To reduce computational cost while preserving spatial information, we first use a 3×3 convolution with a dilation rate of 2 to reduce the dimensionality of the features: ; Step 2, Axial Feature Aggregation To capture long-range dependencies, global average pooling and global max pooling are performed in the horizontal and vertical directions, respectively, for each channel. The statistical characteristic formulas in the vertical and horizontal directions are as follows: Vertical direction: , ; Horizontal direction: , .

[0024] Step 3, Spatial Feature Reconstruction: Statistical features in the horizontal and vertical directions are combined using matrix multiplication to reconstruct a feature map containing spatial location information. This is a crucial step in FFM for capturing the global context. , in, This represents matrix multiplication (outer product), which enables the generated feature map to respond to attention in both the horizontal and vertical directions simultaneously; To more clearly illustrate the physical meaning of this operation in spatial coordinates, for the channel The middle is located in the coordinate The pixel value is: .

[0025] Step 4: Residual Connection and Output: Reconstructed Features The number of channels is recovered after 1×1 convolution. The high-resolution branch and the low-resolution branch are denoted as follows: Then, compared with the original input features By performing addition (residual connection), global and local information are fused to obtain enhanced features. and : , , Final FFM output The sum of the two: , Meanwhile, in the deep semantic interaction stage of the encoder, a lightweight cross-attention module (MCFCM) is used for cross-modal interaction; this module accepts three input features: high-resolution streaming output features. Low-resolution stream output characteristics The last FFM output feature is denoted as .

[0026] Step 5, Linear Embedding and Grouping: First, perform independent linear embedding on the three input features. After embedding, it is split into two independent vectors along the channel dimension. ; Generated after embedding : , , Step 6, Channel Shuffling and Segmentation: To more effectively facilitate information exchange between the two modalities, a channel shuffling mechanism is introduced; and After being spliced ​​together, the parts are washed together and then divided into... and This step ensures that the query and key contain mixed information across modalities: , .

[0027] Step 7, Multi-scale Enhancement: Due to memory limitations, the generated... Instead of directly calculating attention, the vectors are first processed through a shared Multiscale Enhancement Module (MEM). The MEM uses convolutional kernels of different sizes (3×3, 5×5, 7×7) to progressively integrate features to capture multiscale contextual information from adjacent scales. , , The specific enhancement process of the MEM module for the input features is as follows: 1) Multi-branch feature extraction: extracting input features The input features are copied along the channel and fed into four parallel branches: the first branch performs a 7×7 convolution operation on the input features; the second branch performs a 5×5 convolution operation on the input features; the third branch performs two consecutive 3×3 convolution operations on the input features; and the fourth branch performs a 1×1 convolution operation on the input features. (After all the above convolution operations, a batch normalization layer GN and a ReLU activation function are cascaded.) 2) Multi-scale feature integration: The output features of the third branch are added element-wise to the output features of the second branch to obtain intermediate fused features; the intermediate fused features are then added element-wise to the output features of the first branch to obtain integrated multi-scale context features; 3) Pixel-level attention enhancement: The integrated multi-scale contextual features are multiplied pixel by pixel with the output features of the fourth branch to obtain the modulated features; 4) Residual connection reconstruction: Introduce skip connections to connect the modulated features with the original input features. Element-wise addition is performed to obtain the final output features of the multi-scale enhancement module (MEM).

[0028] Step 8, Cross-modal attention calculation: After processing the MEM module using the Softmax function... and The dot product is normalized to generate attention weights; and the calculated attention weights are then embedded into the fused features. and In the middle; the final concatenated features are reconstructed through linear and residual connections: , .

[0029] Step 2.4: Construct the U-Net decoder for super-resolution reconstruction; after receiving the fused multi-scale features, gradually restore the spatial size through upsampling layers and convolutional layers, and finally output high-resolution surface temperature inversion results (8m) through regression layers.

[0030] Step 3: Set the algorithm parameters (task objective, number of training iterations, number of iterations per round, batch size, initial learning rate, regularization coefficient, etc.), as shown in Table 1: Table 1

[0031] The multi-source remote sensing dataset of the training set is input into the model; the data first passes through the differential feature extraction module; then it enters the dual-stream encoder, and the multi-scale fusion module injects the texture features in the high-resolution image into the low-resolution thermal infrared image. Finally, the decoder outputs the surface temperature inversion image; the loss function between the predicted temperature and the true label is calculated, and the backpropagation algorithm based on gradient descent is used to update the model parameters according to the loss function; the above process is repeated until the loss function converges or reaches the preset training rounds, and the final MR-CMF model parameters are saved. Step 3.1: Input multi-source remote sensing data into the network and perform end-to-end feature inference. The data is first pre-processed by the differential feature extraction module; then it enters the dual-stream encoder to extract deep features of different modalities; at the same time, the multi-scale fusion module injects texture details from the high-resolution image into the low-resolution thermal infrared feature map; finally, the fused feature stream is upsampled and reconstructed by the U-Net decoder to output the predicted land surface temperature image. Step 3.2: Compare the predicted temperature image output by the model with the corresponding surface temperature label pixel by pixel. Calculate the mean squared error between the two using a preset loss function to quantify the current prediction error of the model; MR-CMF uses the masked Huber loss function to evaluate prediction accuracy. The Huber loss function combines the advantages of MSE and MAE, and its specific mathematical expression is as follows: , in, The total number of pixels in the image. For a mask matrix, when the pixel points When it is a valid observation When pixel When obscured by clouds or the value is invalid ; , in, For residuals, , This is a real temperature label. These are the model's predicted values; This is a hyperparameter used to adjust the sensitivity to outliers.

[0032] Step 3.3: Based on the loss value, the gradient of the parameters of each layer in the network is calculated using the backpropagation algorithm. Combined with the preset learning rate and optimizer, the weights and bias parameters in the encoder, fusion module, and decoder are updated to minimize the prediction error. Step 3.4: Repeat the above process. After each training epoch, evaluate the model performance using the validation set. When the model's loss on the validation set no longer decreases or reaches the preset early stopping condition, stop training, save the weight file of the best-performing MR-CMF model, and obtain a small patch of prediction error map on the test set, such as... Figure 5 As shown in (a)-(b) in the figure.

[0033] Step 4: Model Testing: Select a new regional remote sensing image as input data and use a pre-trained temperature inversion model to predict temperature pixel by pixel. By comparing the model with the measured temperature data of the region, evaluate the prediction accuracy under different terrain and climate conditions, and further verify its spatial generalization ability and robustness. Step 4.1: Obtain multi-source remote sensing data of the new region and input it into the network for forward extrapolation, outputting a surface temperature inversion image; Step 4.2: Align the inversion results with the temperature labels at the pixel level; and select the root mean square error (RMSE) to evaluate the prediction error, and the coefficient of determination to measure the goodness of fit of the model to the temperature change; Step 4.3: Classify and statistically analyze the test results according to land cover type (e.g., water bodies, vegetation, bare soil, built-up areas). Calculate the inversion error for different land cover types and evaluate the robustness of the model in dealing with complex terrain and surface heterogeneity.

[0034] Step 5: Model fine-tuning training based on measured data: After obtaining the optimal model weights, measured surface temperature data (measured with a handheld thermometer) within the study area is introduced as a physical constraint. The residual between the model inversion result and the measured value is calculated. Small sample bias correction is performed by freezing the encoder parameters and fine-tuning only the decoder end. The systematic error of the model is corrected, thereby improving the absolute accuracy of the model in real-world scenarios. Step 5.1: Select 50 key stations within the study area and obtain measured surface temperature values ​​using handheld thermometers. The latitude and longitude coordinates of these measured points Pixel positions of the image after projection onto the metric coordinate system Perform precise matching and extract the initial model prediction values ​​for the corresponding positions; Step 5.2: Using the transfer learning approach, freeze all weight parameters of the two-stream encoder and feature extraction module in the MR-CMF model, preserving the texture and radiation feature extraction capabilities learned by the model on large-scale datasets; only unfreeze the linear layer parameters (weights) at the end of the U-Net decoder. With bias This puts it in a trainable state.

[0035] Step 5.3: Using the measured temperature as the physical truth, calculate the residual between the model inversion result and the measured value. Based on this residual, perform small-sample fine-tuning training, updating only the parameters at the decoder end; define the loss function for the fine-tuning stage. as follows: , in, This represents the high-dimensional deep features obtained after extraction from the linear layer. For the first Measured temperature values ​​at each point The regularization coefficient is used. Step 5.4: Due to the limited number of actual samples, the leave-one-out verification method is used to evaluate the fine-tuning effect. The measured data are divided into Group, conduct Sub-independent fine-tuning experiments: each time selecting One set is used as the training set to update the model parameters, and one set is retained as the validation set to test the corrected accuracy. Finally, the root mean square error of the prediction results on the validation set is calculated. To measure model accuracy, the following definition is used: , in, In order to remove the first The model fine-tuned on the training set of point n, for point n... Predicted values ​​for each point; Step 5.5: After verifying the effectiveness of the strategy, use all measured points to perform final overall fine-tuning of the model, so that the model output approaches the true physical value, thereby correcting the systematic bias of the model.

[0036] Step 6, Model Application: The final MR-CMF model is applied to the entire African region. By inputting multi-source remote sensing images, high spatial resolution land surface temperature products covering the entire country are obtained. Based on the inversion results, specific application analyses are carried out, such as urban heat island effect monitoring and drought assessment in agricultural areas. Step 6.1: Apply the model to representative arid / semi-arid climate regions and representative tropical plateau climate regions; batch input multi-source remote sensing images to generate high spatial resolution land surface temperature (LST) products covering the entire region; obtain the land surface temperature inversion image of the representative tropical plateau climate region on August 4, 2024, as shown below. Figure 6 As shown; Step 6.2: Focus on typical areas and conduct monitoring and analysis in both urban and agricultural scenarios. On the one hand, quantitatively analyze the relationship between urban expansion and the heat island effect to identify areas at risk of thermal environmental damage; on the other hand, for agricultural areas, use the TVDI / CWSI index to assess crop water stress and drought levels.

[0037] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network.

[0038] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0039] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0040] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0041] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0042] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. An end-to-end land surface temperature retrieval method based on a multi-source heterogeneous data fusion network, characterized in that, Includes the following steps: Step 1: Acquire multi-source heterogeneous data of the study area, including visible light, near-infrared and thermal infrared remote sensing image data and surface temperature label data. Based on NCEP atmospheric water vapor content data, perform spatiotemporal interpolation to obtain atmospheric water vapor content data that matches the spatiotemporal resolution of thermal infrared images. After preprocessing the above multi-source heterogeneous data, obtain a multi-source heterogeneous dataset, which is further divided into training set, validation set and test set. Step 2: Construct a multi-resolution cross-modal land surface temperature inversion model, including: a normalization processing module, a differential feature extraction module, a dual-stream encoder, and a U-Net decoder connected in sequence; The surface temperature inversion model takes visible light, near-infrared and thermal infrared remote sensing image data, as well as atmospheric water vapor content data, as inputs and the inverted surface temperature as output. Step 3: Use the training set, validation set, and test set to train, validate, and test the multi-resolution cross-modal land surface temperature inversion model constructed in Step 2 to obtain the trained land surface temperature inversion model. Step 4: Obtain multi-source heterogeneous data from another region and input it into the trained land surface temperature inversion model for forward extrapolation to generate the land surface temperature inversion results for that region. Compare the land surface temperature inversion results of that region with the actual temperature labels of that region to verify the spatial generalization ability of the trained land surface temperature inversion model. If the spatial generalization ability does not meet the requirements, return to step 3 to adjust the model parameters or increase the training set samples and retrain; otherwise, proceed to step 5; the other region does not overlap with the study area. Step 5: Introduce measured surface temperature data of the study area as physical constraints, calculate the residual between the surface temperature inversion result of the model and the measured surface temperature of the study area, and fine-tune the trained surface temperature inversion model by freezing the parameters of other parts of the model and only fine-tuning the parameters of the linear layer at the end of the U-Net decoder. Step 6: Input the multi-source heterogeneous data of the area to be inverted into the fine-tuned model to obtain high-resolution surface temperature inversion results.

2. The end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: For the study area, acquire visible light and near-infrared remote sensing image data with a spatial resolution of 8m from the ZY-1 02D satellite, as well as thermal infrared remote sensing image data, sensor header files, and surface temperature label data with a spatial resolution of 30m from the Landsat satellite. Step 1.2: Obtain the NCEP atmospheric water vapor content data of the study area from the NCEP data, and perform spatial-temporal interpolation calculations in conjunction with the sensor header file to obtain atmospheric water vapor content data consistent with the temporal and spatial resolution of the thermal infrared remote sensing image. Step 1.3: Geometric registration and upsampling preprocessing are performed on the surface temperature label data, atmospheric water vapor content data with the same temporal and spatial resolution as the thermal infrared remote sensing image, and visible light, near-infrared, and thermal infrared remote sensing image data. The preprocessed data are then cropped according to a preset sliding window size to obtain a multi-source heterogeneous dataset. The sliding window size for the visible light and near-infrared remote sensing image data is 480×480, the sliding window size for the surface temperature label data is 480×480, and the sliding window size for the thermal infrared remote sensing image and the atmospheric water vapor content data with the same temporal and spatial resolution as the thermal infrared remote sensing image is 128×128. Step 1.4: Divide the multi-source heterogeneous dataset into the first dataset and the second dataset in an 8:2 ratio. Use k-fold cross-validation to divide the first dataset into a training set and a validation set, and use the second dataset as the test set.

3. The end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to claim 2, characterized in that, The specific process of step 1.2 is as follows: Temporal interpolation was performed on the atmospheric water vapor content data obtained from NCEP data to obtain the atmospheric water vapor content at the time of thermal infrared remote sensing image capture, using the following formula: ， in, This indicates the atmospheric water vapor content at the time the Landsat satellite captured the thermal infrared remote sensing image. This indicates the time when the Landsat satellite captured the thermal infrared remote sensing image. and These represent the NCEP imaging times that are closest to the thermal infrared remote sensing image capture time and are located before and after that capture time, respectively. and They represent and The corresponding NCEP atmospheric water vapor content; Spatial interpolation was performed on the atmospheric water vapor content data at the time of thermal infrared remote sensing image capture to obtain atmospheric water vapor content data with the same temporal and spatial resolution as the thermal infrared remote sensing image.

4. The end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to claim 1, characterized in that, In step 2, the normalization processing module is used to normalize the visible light, near-infrared and thermal infrared remote sensing image data and atmospheric water vapor content data input to the model by performing maximum and minimum value normalization. The differential feature extraction module includes three branches: optical, thermal infrared, and atmospheric water vapor. The optical branch includes a residual module that combines the SE channel attention mechanism to extract high-frequency features from the visible light and near-infrared remote sensing images after the normalization processing module, thereby obtaining high-resolution features. The thermal infrared branch includes a residual module that incorporates the SE channel attention mechanism, used to extract features from the thermal infrared remote sensing image after it has been processed by the normalization module, thus obtaining thermal infrared features. The atmospheric water vapor branch includes a residual module with a 7×7 large convolution kernel, which is used to extract features of atmospheric water vapor content after normalization processing module to obtain atmospheric water vapor features. Low-resolution features are obtained by stitching together thermal infrared features and atmospheric water vapor features using a feature stitching module.

5. The end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to claim 4, characterized in that, In step 2, the dual-stream encoder includes a high-resolution stream branch, a low-resolution stream branch, a cross-attention module, and first to fourth feature fusion modules. The high-resolution stream branch includes first to fourth encoder modules connected in sequence, and the low-resolution stream branch includes fifth to eighth encoder modules connected in sequence. The input of the first encoder module is the high-resolution feature output by the differential feature extraction module, and the input of the fifth encoder module is the low-resolution feature output by the differential feature extraction module. The outputs of the first and fifth encoder modules are used as the input of the first feature fusion module, the outputs of the second and sixth encoder modules are used as the input of the second feature fusion module, the outputs of the third and seventh encoder modules are used as the input of the third feature fusion module, the outputs of the fourth and eighth encoder modules are used as the input of the fourth feature fusion module, and the outputs of the fourth feature fusion module and the fourth and eighth encoder modules are used as the input of the cross-attention module. The output of the cross-attention module is used as the output of the dual-stream encoder.

6. The end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to claim 5, characterized in that, For each feature fusion module, perform the following operation simultaneously on both inputs: Step 1, use a 3×3 dilated convolution with a dilation rate of 2. Dimensionality reduction of the input features: ， in, As input features, Input features After dilated convolution dimensionality reduction, the resulting output feature map is at the [missing information - likely a typo]. Feature representation on each channel; Step 2, for Perform global average pooling and global max pooling in the horizontal and vertical directions respectively: Vertical direction: ， ， Horizontal direction: ， ， in, The input features are respectively at the th On the first channel, along the horizontal direction, for the first... The statistical feature values ​​obtained by performing average pooling and max pooling on all elements of a row; The input features are respectively at the th On the first channel, along the vertical direction, for the first... The statistical feature values ​​obtained by performing average pooling and max pooling on all elements of a row; These are the width and height of the input feature map, respectively; Input features In the On each channel, the spatial coordinates are: Eigenvalues ​​at; Step 3: Combine the statistical feature values ​​in the horizontal and vertical directions using matrix multiplication to obtain the reconstructed features. , will reconstruct features After recovering the number of channels through 1×1 convolution, it is compared with the input features. Add them together to obtain the enhanced features. : ， ， in, Represents matrix multiplication. These are the height and width vectors obtained after performing one-dimensional global average pooling on the input features along the horizontal and vertical directions, respectively. These are the height and width vectors obtained after performing one-dimensional global max pooling on the input features along the horizontal and vertical directions, respectively. After the two inputs of each feature fusion module are processed simultaneously through Step 1 to Step 3, their respective enhanced features are obtained. The two enhanced features are then added together to obtain the output features of each feature fusion module.

7. The end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to claim 5, characterized in that, The specific process of the cross-attention module is as follows: Linear embedding is performed on the three inputs of the cross-attention module, and the output of the fourth feature fusion module is... After embedding, it is split into two independent vectors along the channel dimension. The outputs of the fourth and eighth encoder modules After embedding, the corresponding generation is... : ， ， in, This represents a tensor segmentation operation that uniformly splits the input features into multiple independent sub-features along the channel dimension. This represents a linear mapping operation; They respectively represent the high-frequency characteristics and low frequency characteristics The query and key features obtained after mapping; Will and After concatenation, the data is shuffled and then split into vectors. : ， ， in, This indicates a channel mixing operation. Indicates feature concatenation operation; Utilizing multi-scale enhancement modules To each and Enhancement processing: ， ， Using the Softmax function and Normalize the dot product to generate attention weights. Similarly, generate The calculated attention weights and Embedded into and In the final concatenation, the features are output after linear and residual connections. : ， ， in, The fused features are the final output of the cross-attention module. The total number of channels for the input features; After receiving the fused multi-scale features, the U-Net decoder performs upsampling and reconstruction, and outputs high-resolution surface temperature inversion results.

8. The end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to claim 1, characterized in that, In step 3, during the training of the multi-resolution cross-modal land surface temperature inversion model, the loss function between the model's predicted temperature and the actual temperature label is calculated, and the backpropagation algorithm based on gradient descent is used to update the model parameters according to the loss function. The loss function is the masked Huber loss function. The calculation formula is as follows: ， in, The total number of pixels in the predicted surface temperature image. For a mask matrix, when the pixel points When it is a valid observation When pixel When obscured by clouds or the value is invalid ; ， in, For residuals, , This is a real temperature label. To predict temperature for the model, This is a hyperparameter.

9. The end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network according to claim 1, characterized in that, In step 5, a handheld thermometer is used to obtain the measured value of the ground surface temperature. After projecting the latitude and longitude coordinates of these measured values ​​into a metric coordinate system, the pixel positions of the predicted temperature image are compared. Matching is performed to extract the initial predicted values ​​of the trained land surface temperature inversion model at the corresponding locations; Using the idea of ​​transfer learning, all parameters of other parts of the trained land surface temperature inversion model are frozen, and only the linear layer parameters at the end of the U-Net decoder are unfrozen, so that they are in a trainable state. Using measured surface temperature as the physical truth, the residual between the model inversion result and the measured surface temperature is calculated. Based on the residual, small-sample fine-tuning training is performed, updating only the linear layer parameters at the end of the U-Net decoder. The loss function for the fine-tuning stage is defined. as follows: ， in, This represents the number of measured surface temperature values. These are the weights and biases of the linear layer at the end of the U-Net decoder. This represents the high-dimensional deep features obtained after extraction from the linear layer. For the first Measured surface temperature values The regularization coefficient is used. The leave-one-out verification method was used to evaluate the effect of fine-tuning. The measured values ​​are divided into Group samples, for Sub-independent fine-tuning experiments: each time selecting One set is used as the training set to update the model parameters, and one set is retained as the validation set to test the corrected accuracy. Finally, the root mean square error of the prediction results of all validation sets is calculated to measure the model accuracy, defined as follows: ， in, The root mean square error of the prediction results for all validation sets. In order to remove the first The model fine-tuned on the training set of the first set of samples is used to fine-tune the model for the second set of samples. The predicted value for the group.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the end-to-end land surface temperature inversion method based on a multi-source heterogeneous data fusion network as described in any one of claims 1 to 9.

Images