A method and system for spatiotemporal downscaling of remote sensing images based on multi-scale dynamic weighting

Abstract

The present invention discloses a remote sensing image spatiotemporal downscaling method and system based on multi-scale dynamic weighting, comprising the following steps: obtaining high spatial and low spatial resolution images of the same date within a period of time in a target area, and dividing them into a training set and a test set; constructing a spatiotemporal feature enhancement hierarchical network, the spatiotemporal feature enhancement hierarchical network comprising a feature extraction module, a feature fusion module and a feature reconstruction module; using the training set to train the spatiotemporal feature enhancement hierarchical network, constructing and calculating a multidimensional spatiotemporal loss function, optimizing the internal parameters of the network through back propagation, using the test set to perform quality assessment on the optimized network, and saving the trained network; obtaining a high spatial resolution image of a certain date and a low spatial resolution image of a predicted date, sending them to the trained network for downscaling processing, and obtaining a high spatial resolution predicted image of the predicted date. The present invention can fill in the missing high spatial resolution remote sensing images of some dates.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image spatiotemporal fusion, and in particular relates to a remote sensing image spatiotemporal downscaling method and system based on multi-scale dynamic weighting. Background Art

[0002] High spatiotemporal resolution remote sensing images have a broad market demand and a variety of application scenarios, such as vegetation coverage area estimation, crop yield estimation, atmospheric environment monitoring, field evapotranspiration estimation and many other remote sensing applications. However, due to the contradiction between the sensor pixel size and the scanning width, it is difficult to capture high spatial resolution and high temporal resolution (low revisit period) remote sensing data in a single satellite sensor. For example, the Landsat series of satellites have images with high spatial resolution and low temporal resolution. The spatial resolution of this series of products is 30 meters, which can provide high-resolution surface images, but its revisit period is as high as 16 days. The second category is images with relatively high temporal resolution and low spatial resolution, such as MODIS satellites, which can revisit the earth once a day, but the spatial resolution is relatively low, only 250~1000 meters. In order to solve this problem, the spatiotemporal fusion method guides the high spatial resolution remote sensing data of the existing date through the temporal information of the high temporal resolution image of the predicted date, and then generates a high spatial resolution remote sensing image of the predicted date.

[0003] There are two main methods of existing spatiotemporal fusion: one is the traditional algorithm and the other is the method based on neural networks. Traditional algorithms include methods based on weight functions and methods based on mixed pixel decomposition. Traditional algorithms are simple to calculate, but cannot handle complex terrain or dynamically changing surface features, and are difficult to adapt to nonlinear or complex multi-source remote sensing data features, resulting in poor versatility and poor quality of synthesized images. In the existing spatiotemporal fusion methods based on neural networks, only a single architecture is used and simple fusion processing is performed, resulting in the network being unable to effectively preserve and fuse spatial details and temporal changes in complex terrain. The generated images may have spatiotemporal dislocations, resulting in logical discontinuity of the time series in image representation. For example, some patterns of surface changes may be misjudged, resulting in the inability to accurately identify dynamic processes such as vegetation changes and urban expansion.

[0004] Therefore, it is urgent to solve the above problems. Summary of the invention

[0005] Purpose of the invention: The first purpose of the present invention is to provide a remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting, which downscales the remote sensing image by multi-scale dynamic weighting based on the spatiotemporal feature enhancement hierarchical network to obtain a high spatial resolution image of the predicted date.

[0006] The second object of the present invention is to provide a remote sensing image spatiotemporal downscaling system based on multi-scale dynamic weighting.

[0007] Technical solution: To achieve the above objectives, the present invention discloses a remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting, comprising the following steps:

[0008] (1) Obtain high spatial resolution remote sensing images and low spatial resolution images of the same date in the target area over a period of time. The high spatial resolution image of one date and the low spatial resolution image of another date form a remote sensing image pair. After preprocessing, they are combined with the high spatial resolution real image of another date and divided into a training set and a test set.

[0009] (2) Constructing a spatiotemporal feature enhancement hierarchical network, which includes a feature extraction module, a feature fusion module, and a feature reconstruction module;

[0010] The feature extraction module includes a residual module, a downsampling module and a Transformer module. The remote sensing image is input to the feature extraction module for feature extraction, and the feature extraction is performed using the residual module and the downsampling module step by step. After the minimum scale feature map is obtained through a residual module, it is sent to the Transformer module for feature extraction to obtain a multi-scale feature map. The multi-scale feature map is fused at multiple scales through the feature fusion module.

[0011] The feature reconstruction module includes a residual module, an upsampling module and a Transformer module. The fused multi-scale feature map is input into the feature reconstruction module. The fused minimum-scale feature map is sent to the Transformer module and the residual module for feature reconstruction. The remaining scale feature maps after fusion are reconstructed using the residual module and the upsampling module step by step to obtain a high spatial resolution prediction image corresponding to another date.

[0012] (3) Use the training set to train the spatiotemporal feature enhancement hierarchical network, construct and calculate the multidimensional spatiotemporal loss function, optimize the internal parameters of the spatiotemporal feature enhancement hierarchical network through back propagation, use the test set to evaluate the quality of the optimized spatiotemporal feature enhancement hierarchical network, and save the trained spatiotemporal feature enhancement hierarchical network;

[0013] (4) Obtain a high-spatial-resolution image of a certain date within a period of time in the target area and a low-spatial-resolution image of the predicted date to form a remote sensing image pair and perform preprocessing; send the preprocessed remote sensing image pair to the trained spatiotemporal feature enhancement hierarchical network for downscaling to obtain a high-spatial-resolution predicted image of the predicted date.

[0014] Optionally, the step (1) specifically includes the following steps:

[0015] (1.1) Determine the target area for research, download the remote sensing images of the first satellite and the second satellite on the same date in the target area within the past three years, and form a remote sensing image pair. The remote sensing image pair includes at least the following bands: blue, green, red, and NIR near-infrared. The remote sensing image of the first satellite is a high spatial resolution image, and the remote sensing image of the second satellite is a low spatial resolution image. The spatial resolution of the remote sensing image of the first satellite is higher than that of the remote sensing image of the second satellite;

[0016] (1.2) Perform preliminary screening and preprocessing of remote sensing image pairs, using platform screening and visual judgment to eliminate remote sensing images with cloud cover and streak or band noise problems; perform radiation correction, geometric correction and denoising on remote sensing image pairs, and then resample low spatial resolution images to the same image resolution as high spatial resolution images;

[0017] (1.3) The preprocessed remote sensing image pairs are combined with high spatial resolution real images of another date and then divided into training sets and test sets.

[0018] Optionally, in the step (2), the left branch of the residual module is a convolution kernel, a batch normalization layer BN and a ReLU activation function, followed by another convolution kernel and a batch normalization layer BN; the right branch of the residual module is a convolution kernel; the results of the left and right paths are then summed, and a ReLU activation function is used to output the feature map.

[0019] Optionally, the Transformer module in step (2) includes four Transformer encoder layers, and the input minimum scale feature map is sequentially passed through a linear projection layer and a layer normalization LN, and then linearly mapped into three matrices Q, K and V, which are input into a multi-head self-attention mechanism MSA. Subsequently, the result of the multi-head self-attention mechanism MSA is summed with the result of the linear projection layer for the first time, and the result after the first summation is passed through a layer normalization LN and a multi-layer perceptron MLP, and then summed with the result of the first summation for the second time, so as to output a higher quality minimum scale feature map.

[0020] Optionally, in the feature fusion module in step (2), the original feature map of the input high-resolution image is firstly and the original feature map of the low-resolution image Split evenly according to the number of channels, Represents the width M, height N and number of channels C of the feature map;

[0021] The high-resolution image feature map after splitting becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times to obtain the downsampled feature images ;

[0022] Then, the features of four different dimensions are extracted through deep convolution, and the dimensions are restored to , output multi-scale feature extraction part ;

[0023] The split low-resolution image feature map becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times to obtain the downsampled feature images ;

[0024] Then, the features of four different dimensions are extracted through deep convolution, and the dimensions are restored to , output multi-scale feature extraction part ;

[0025] Afterwards To connect channels, The channels are connected, and then normalized by the GELU activation function, and finally compared with the original feature map and Perform element-by-element multiplication and finally output multi-scale features and ;

[0026] Multi-scale features and Global average pooling and global maximum pooling operations are performed in the spatial dimension and channel dimension respectively, and then the and The pooling results in the spatial dimension and channel dimension are connected, and then the spatial features are processed by two-way 2D convolution. The channel features are processed by a shared MLP, and finally activated by the softmax function to obtain a three-way output result. and , and output the results of two of them and After summing, the original feature map Perform element-by-element multiplication and output the result in another way Directly with the original feature map Element-by-element multiplication is performed, and the final two-way results are summed to obtain the final fusion result.

[0027] Optionally, in step (3), a multidimensional space-time loss function is constructed, the multidimensional space-time loss function includes a time characteristic loss function and a space characteristic loss function, and the expression of the multidimensional space-time loss function is:

[0028] ,

[0029] in, is the spatial characteristic loss function, is the time characteristic loss function, are the weighted parameters of the two major loss functions respectively. The expression of the spatial characteristic loss function is:

[0030] ,

[0031] ,

[0032] in, is the cosine similarity loss function, which determines the directional consistency of the two images by calculating the cosine value of the angle between the two images. The closer the value is to 1, the more similar the overall features of the images are. is the structural similarity index, with a value range of [-1, 1], where 1 means that the two images are exactly the same, and 0 means that the two images are completely different; are weighted parameters, The input i The predicted image and the real image corresponding to the sample image;

[0033] The expression of the time characteristic loss function is:

[0034] ,

[0035] ,

[0036] ,

[0037] in, is the root mean square error loss function, which is used to measure the temporal consistency between the predicted image and the real image; is the consistency loss function, which measures the continuity of the generated image in the time dimension and ensures the smoothness of the time series by calculating the difference between the generated images at adjacent moments; represents the length of the time series, are weighted parameters, The inputi The predicted image and the real image corresponding to the sample image, for Moment i The predicted image corresponding to the sample image, for Moment i The predicted image corresponding to the sample image.

[0038] Optionally, in step (3), the test set is input into the optimized spatiotemporal feature enhancement hierarchical network, and the generated high spatial resolution predicted image is compared with the high spatial resolution real image in terms of indicators in various dimensions until the indicators approach ideal values; the indicators in various dimensions include structural similarity index SSIM, correlation coefficient CC, peak signal-to-noise ratio PSNR, root mean square error RMSE and spectral angle mapping SAM.

[0039] Based on the same inventive concept, the present invention discloses a remote sensing image spatiotemporal downscaling system based on multi-scale dynamic weighting, comprising:

[0040] A data acquisition module is used to acquire high spatial resolution remote sensing images and low spatial resolution images of the same date in a target area within a period of time, wherein a high spatial resolution image of a certain date and a low spatial resolution image of another date form a remote sensing image pair, which is combined with a high spatial resolution real image of another date after preprocessing and divided into a training set and a test set;

[0041] A network construction module is used to construct a spatiotemporal feature enhancement hierarchical network, which includes a feature extraction module, a feature fusion module and a feature reconstruction module;

[0042] The feature extraction module includes a residual module, a downsampling module and a Transformer module. The remote sensing image is input to the feature extraction module for feature extraction, and the feature extraction is performed using the residual module and the downsampling module step by step. After the minimum scale feature map is obtained through a residual module, it is sent to the Transformer module for feature extraction to obtain a multi-scale feature map. The multi-scale feature map is fused at multiple scales through the feature fusion module.

[0043] The feature reconstruction module includes a residual module, an upsampling module and a Transformer module. The fused multi-scale feature map is input into the feature reconstruction module. The fused minimum-scale feature map is sent to the Transformer module and the residual module for feature reconstruction. The remaining scale feature maps after fusion are reconstructed using the residual module and the upsampling module step by step to obtain a high-resolution prediction image corresponding to another date.

[0044] The network training module is used to train the spatiotemporal feature enhancement hierarchical network using the training set, construct and calculate the multidimensional spatiotemporal loss function, optimize the internal parameters of the spatiotemporal feature enhancement hierarchical network through back propagation, evaluate the quality of the optimized spatiotemporal feature enhancement hierarchical network using the test set, and save the trained spatiotemporal feature enhancement hierarchical network;

[0045] The prediction module is used to obtain a high-spatial-resolution image of a certain date within a period of time in the target area and a low-spatial-resolution image of the predicted date to form a remote sensing image pair and perform preprocessing; the preprocessed remote sensing image pair is sent to the trained spatiotemporal feature enhancement hierarchical network for downscaling to obtain a high-spatial-resolution prediction image of the predicted date.

[0046] Optionally, the left branch of the residual module is a convolution kernel, a batch normalization layer BN and a ReLU activation function in sequence, and then passes through another convolution kernel and a batch normalization layer BN; the right branch of the residual module is a convolution kernel; then the results of the left and right paths are summed, and a ReLU activation function is used to output the feature map.

[0047] Optionally, the feature fusion module first converts the input high-resolution image original feature map and the original feature map of the low-resolution image Split evenly according to the number of channels, Represents the width M, height N and number of channels C of the feature map;

[0048] The high-resolution image feature map after splitting becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times to obtain the downsampled feature images ;

[0049] Then, the features of four different dimensions are extracted through deep convolution, and the dimensions are restored to , output multi-scale feature extraction part ;

[0050] The split low-resolution image feature map becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times to obtain the downsampled feature images ;

[0051] Then, the features of four different dimensions are extracted through deep convolution, and the dimensions are restored to , output multi-scale feature extraction part ;

[0052] Afterwards To connect channels, The channels are connected, and then normalized by the GELU activation function, and finally compared with the original feature map and Perform element-by-element multiplication and finally output multi-scale features and ;

[0053] Multi-scale features and Global average pooling and global maximum pooling operations are performed in the spatial dimension and channel dimension respectively, and then the and The pooling results in the spatial dimension and channel dimension are connected, and then the spatial features are processed by two-way 2D convolution. The channel features are processed by a shared MLP, and finally activated by the softmax function to obtain a three-way output result. and , and output the results of two of them and After summing, the original feature map Perform element-by-element multiplication and output the result in another way Directly with the original feature map Element-by-element multiplication is performed, and the final two-way results are summed to obtain the final fusion result.

[0054] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the present invention adopts a spatiotemporal feature enhancement hierarchical network constructed with CNN and Transformer, and downscales the remote sensing image by multi-scale dynamic weighting to obtain a high spatial resolution image of the predicted date; the present invention can be widely used in various types of satellite remote sensing data and various regions for spatiotemporal fusion, and can fill the gap of high-resolution remote sensing images for some dates in the current region with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the process of the present invention;

[0056] Figure 2It is a schematic diagram of the framework of the spatiotemporal feature enhancement hierarchical network in the present invention;

[0057] Figure 3 is a flow chart of the residual module in the present invention;

[0058] Figure 4 It is a flowchart of the Transformer module in the present invention;

[0059] Figure 5 It is a flow chart of the feature fusion module in the present invention;

[0060] Figure 6 is a schematic diagram of a complex area image with high spatial resolution in an embodiment of the present invention;

[0061] Figure 7 It is a schematic diagram of a low spatial resolution complex area image in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] Example 1: This example is recommended to be run on a computer with the following configuration: Intel i913900HX processor, NVIDIA GeForce RTX 4080 graphics processor, main frequency 2.2GHz, memory 32GB, operating system Windows 11. The implementation of the spatiotemporal feature enhancement hierarchical network is based on the Pytorch2.1 deep learning framework toolkit. Figure 1 As shown, a remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting includes the following steps:

[0063] (1) Obtain high spatial resolution remote sensing images and low spatial resolution images of the same date in the target area over a period of time, where the high spatial resolution image of a certain date t1 And another low spatial resolution image at date t2 The remote sensing image pair is composed of preprocessed and compared with the high spatial resolution real image of another date t2 Combine and divide into training set and test set;

[0064] The step (1) specifically includes the following steps:

[0065] (1.1) Determine the target area for research, and use the Earth Explorer platform to download remote sensing images of the target area from Landsat8 and Modis satellites on the same date in the past three years to form a remote sensing image pair. The remote sensing image pair includes at least the following bands: blue, green, red, and NIR. The remote sensing images of Landsat8 satellite are high spatial resolution images, and the remote sensing images of Modis satellite are low spatial resolution images. The spatial resolution of the remote sensing images of Landsat8 satellite is higher than that of the remote sensing images of Modis satellite.

[0066] (1.2) Perform preliminary screening and preprocessing of remote sensing image pairs, using platform screening and visual judgment to eliminate remote sensing images with cloud cover and streaking or band noise problems; perform radiation correction, geometric correction and denoising on the acquired remote sensing image pairs using ENVI software, and then resample the low spatial resolution images to the same image resolution as the high spatial resolution images;

[0067] (1.3) Compare the preprocessed remote sensing image with a high spatial resolution real image at another date t2 Combine and then divide into training set and test set;

[0068] (2) Construct a hierarchical network for spatiotemporal feature enhancement. The hierarchical network for spatiotemporal feature enhancement is a U-shaped network architecture. The backbone of the network adopts the U-shaped architecture of UNet. The hierarchical network for spatiotemporal feature enhancement includes a feature extraction module, a feature fusion module, and a feature reconstruction module. In order to more clearly display the convolution processing results at different levels, a 240x240x3 image is input as an example to clarify the width M, height N, and number of channels C of the feature map in each layer of the network. Figure 2 shown.

[0069] The feature extraction module includes the residual module, the downsampling module and the Transformer module. and The composed remote sensing image pairs are used as the input of the feature extraction module, and enter the feature extraction module for feature extraction. The feature extraction is performed using the residual module and the downsampling module in a step-by-step manner. After the minimum scale feature map is obtained by the last level residual module, it is sent to the Transformer module for feature extraction to obtain a multi-scale feature map.

[0070] The residual module is mainly composed of a convolutional neural network (CNN). The left branch of the residual module is a 3x3 convolution kernel, a batch normalization layer (BN), and a ReLU activation function, followed by a 3x3 convolution kernel and a batch normalization layer (BN); the right branch of the residual module is a 1x1 convolution kernel; the results of the left and right branches are then summed, and a ReLU activation function is used to output the feature map, such as Figure 3 shown.

[0071] The Transformer module contains four Transformer encoder layers. The input minimum scale feature map passes through a linear projection layer and a layer normalization LN in turn, and then is linearly mapped into three matrices Q, K and V and input into a multi-head self-attention mechanism MSA. Then, the result of the multi-head self-attention mechanism MSA is summed with the result of the linear projection layer for the first time. The result after the first summation passes through a layer normalization LN and a multi-layer perceptron MLP, and then is summed with the result of the first summation for the second time to output a higher quality minimum scale feature map, such as Figure 4 shown.

[0072] In the feature fusion module, the original feature map of the input high-resolution image is first and the original feature map of the low-resolution image Divide into four equal parts according to the number of channels;

[0073] The high-resolution image feature map after splitting becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times; the specific expression is:

[0074] ,

[0075] in, is the feature image after downsampling;

[0076] Then, the features of four different dimensions are extracted through 3×3 depth convolution, and the dimensions are restored to ; The specific expression is:

[0077] ,

[0078] in, It is the output of the multi-scale feature extraction part;

[0079] The split low-resolution image feature map becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times; the specific expression is:

[0080] ,

[0081] in, is the feature image after downsampling;

[0082] Then, the features of four different dimensions are extracted through 3×3 depth convolution, and the dimensions are restored to ; The specific expression is:

[0083] ,

[0084] in, It is the output of the multi-scale feature extraction part;

[0085] Afterwards To connect channels, The channels are connected, and then normalized by the GELU activation function, and finally compared with the original feature map and Perform element-by-element multiplication and finally output multi-scale features and ;

[0086] ,

[0087] ,

[0088] Multi-scale features and Global average pooling and global maximum pooling operations are performed in the spatial dimension and channel dimension respectively, and then the and The pooling results in the spatial dimension and channel dimension are connected, and then the spatial features are processed by two-way 2D convolution. The channel features are processed by a shared MLP, and finally activated by the softmax function to obtain a three-way output result. and , and output the results of two of them and After summing, the original feature map Perform element-by-element multiplication and output the result in another way Directly with the original feature map Perform element-by-element multiplication, and finally sum the two results to get the final fusion result, expressed as:

[0089] ,

[0090] ,

[0091] ,

[0092] ,

[0093] ,

[0094] in, is the feature vector in the channel dimension, is the eigenvector in the spatial dimension, To perform average pooling in the channel dimension, To perform the maximum pooling operation in the channel dimension, To perform average pooling in the spatial dimension, To perform the maximum pooling operation in the spatial dimension; They are the three output results after activation by the sotfmax function. is the final fusion result, such as Figure 5 shown.

[0095] The feature reconstruction module includes a residual module, an upsampling module and a Transformer module. The fused multi-scale feature map is input into the feature reconstruction module. The smallest scale feature map after fusion is sent to the Transformer module and the residual module for feature reconstruction. The remaining scale feature maps after fusion are reconstructed using the residual module and the upsampling module step by step. The scale is restored through the upsampling module to obtain a high-resolution prediction image corresponding to date t2. .

[0096] (3) Train the spatiotemporal feature enhancement hierarchical network, calculate the results of the multi-dimensional spatiotemporal loss function, optimize the internal parameters of the spatiotemporal feature enhancement hierarchical network through back propagation, use the test set to evaluate the quality of the optimized spatiotemporal feature enhancement hierarchical network, and save the trained spatiotemporal feature enhancement hierarchical network.

[0097] In step (3), a multidimensional space-time loss function is constructed. The multidimensional space-time loss function includes a time characteristic loss function and a space characteristic loss function. The expression of the multidimensional space-time loss function is:

[0098]

[0099] in, is the spatial characteristic loss function, is the time characteristic loss function, are weighted parameters of the two major loss functions respectively; the spatial characteristic loss function is used to ensure the accuracy of the generated image in terms of spatial resolution and details. The expression of the spatial characteristic loss function is:

[0100] ,

[0101]

[0102] in, is the cosine similarity loss function, which determines the directional consistency of the two images by calculating the cosine value of the angle between the two images. The closer the value is to 1, the more similar the overall features of the images are. is the structural similarity index, with a value range of [-1, 1], where 1 means that the two images are exactly the same and 0 means that the two images are completely different; are weighted parameters, The input i The predicted image and the real image corresponding to the sample image;

[0103] The temporal characteristic loss function is used to ensure the continuity and change trend of the generated image in the time dimension. The expression of the temporal characteristic loss function is:

[0104] ,

[0105] ,

[0106]

[0107] in, is the root mean square error loss function, which is used to measure the temporal consistency between the predicted image and the real image; is the consistency loss function, which measures the continuity of the generated image in the time dimension and ensures the smoothness of the time series by calculating the difference between the generated images at adjacent moments; represents the length of the time series, are weighted parameters, The input i The predicted image and the real image corresponding to the sample image, for Moment i The predicted image corresponding to the sample image, for Moment i The predicted image corresponding to the sample image.

[0108] In step (3), the test set is input into the optimized spatiotemporal feature enhancement hierarchical network, and the generated high-resolution prediction image is With high-resolution real images Compare the indicators in each dimension until they reach the ideal value;

[0109] The indicators of each dimension include structural similarity index SSIM, correlation coefficient CC, peak signal-to-noise ratio PSNR, root mean square error RMSE and spectral angle mapping SAM.

[0110] (4) Obtain a high spatial resolution image of a target area at a certain date t1 within a period of time and low spatial resolution images for forecast date t3 , forming a remote sensing image pair and preprocessing it; the preprocessed remote sensing image pair is sent to the trained spatiotemporal feature enhancement hierarchical network for downscaling to obtain a high spatial resolution prediction image for the prediction date t3 .

[0111] In Example 1, the Beijing Daxing data set is selected, and the experimental example of the data set is shown in FIG. Figure 6 and Figure 7 As shown, the method of this embodiment is used for verification, and the comparison between the indicators of each dimension and the ideal value is obtained, as shown in Table 1.

[0112] Table 1 Comparison of indicators of each dimension with ideal values

[0113]

[0114] Example 2

[0115] The present invention discloses a remote sensing image spatiotemporal downscaling system based on multi-scale dynamic weighting, comprising:

[0116] The data acquisition module is used to obtain high spatial resolution remote sensing images and low spatial resolution images of the same date in the target area over a period of time, among which the high spatial resolution image of a certain date t1 And another low spatial resolution image at date t2 The remote sensing image pair is composed of preprocessed and compared with the high spatial resolution real image of another date t2 Combine and divide into training set and test set.

[0117] The research target area is determined in the data acquisition module, and the remote sensing images of the Landsat8 satellite and the Modis satellite of the same date in the target area in the past three years are downloaded using the Earth Explorer platform to form a remote sensing image pair. The remote sensing image pair includes at least the following bands: blue, green, red, and NIR near infrared. The remote sensing images of the Landsat8 satellite are high spatial resolution images, and the remote sensing images of the Modis satellite are low spatial resolution images. The spatial resolution of the remote sensing images of the Landsat8 satellite is higher than that of the remote sensing images of the Modis satellite.

[0118] Preliminary screening and preprocessing of remote sensing image pairs are performed, and remote sensing images with cloud cover and stripe or band noise problems are eliminated by platform screening and visual judgment; the acquired remote sensing image pairs are subjected to radiation correction, geometric correction and denoising through ENVI software, and then the low spatial resolution images are resampled to the same image resolution as the high spatial resolution images;

[0119] The preprocessed remote sensing image is compared with the high spatial resolution real image at another date t2. Combine and then divide into training set and test set.

[0120] The network construction module is used to construct a spatiotemporal feature enhancement hierarchical network. The spatiotemporal feature enhancement hierarchical network includes a feature extraction module, a feature fusion module and a feature reconstruction module. The spatiotemporal feature enhancement hierarchical network has a U-shaped network architecture, and the network backbone adopts the U-shaped architecture of UNet. In order to more clearly display the convolution processing results at different levels, a 240x240x3 image is input as an example to clarify the width M, height N and number of channels C of the feature map in each layer of the network.

[0121] The feature extraction module includes the residual module, the downsampling module and the Transformer module. and The composed remote sensing image pairs are used as the input of the feature extraction module, and enter the feature extraction module for feature extraction. The feature extraction is performed using the residual module and the downsampling module in a step-by-step manner. After the minimum scale feature map is obtained by the last level residual module, it is sent to the Transformer module for feature extraction to obtain a multi-scale feature map.

[0122] The residual module is mainly composed of a convolutional neural network (CNN). The left branch of the residual module is a 3x3 convolution kernel, a batch normalization layer (BN), and a ReLU activation function, followed by a 3x3 convolution kernel and a batch normalization layer (BN); the right branch of the residual module is a 1x1 convolution kernel; the results of the left and right paths are then summed, and a ReLU activation function is used to output the feature map.

[0123] The Transformer module contains four Transformer encoder layers. The input minimum scale feature map passes through a linear projection layer and a layer normalization LN in turn, and then is linearly mapped into three matrices Q, K and V and input into a multi-head self-attention mechanism MSA. Then, the result of the multi-head self-attention mechanism MSA is summed with the result of the linear projection layer for the first time. The result after the first summation passes through a layer normalization LN and a multi-layer perceptron MLP, and then is summed with the result of the first summation for the second time to output a higher quality minimum scale feature map.

[0124] In the feature fusion module, the original feature map of the input high-resolution image is first and the original feature map of the low-resolution image Divide into four equal parts according to the number of channels;

[0125] The high-resolution image feature map after splitting becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times; the specific expression is:

[0126] ,

[0127] in, is the feature image after downsampling;

[0128] Then, the features of four different dimensions are extracted through 3×3 depth convolution, and the dimensions are restored to ; The specific expression is:

[0129] ,

[0130] in, It is the output of the multi-scale feature extraction part;

[0131] The split low-resolution image feature map becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times; the specific expression is:

[0132] ,

[0133] in, is the feature image after downsampling;

[0134] Then, the features of four different dimensions are extracted through 3×3 depth convolution, and the dimensions are restored to ; The specific expression is:

[0135] ,

[0136] in, It is the output of the multi-scale feature extraction part;

[0137] Afterwards To connect channels, The channels are connected, and then normalized by the GELU activation function, and finally compared with the original feature map and Perform element-by-element multiplication and finally output multi-scale features and ;

[0138] ,

[0139] ,

[0140] Multi-scale features and Global average pooling and global maximum pooling operations are performed in the spatial dimension and channel dimension respectively, and then the and The pooling results in the spatial dimension and channel dimension are connected, and then the spatial features are processed by two-way 2D convolution. The channel features are processed by a shared MLP, and finally activated by the softmax function to obtain a three-way output result. and , and output the results of two of them and After summing, the original feature map Perform element-by-element multiplication and output the result in another way Directly with the original feature map Perform element-by-element multiplication, and finally sum the two results to get the final fusion result, expressed as:

[0141] ,

[0142] ,

[0143] ,

[0144] ,

[0145]

[0146] in, is the feature vector in the channel dimension, is the eigenvector in the spatial dimension, To perform average pooling in the channel dimension, To perform the maximum pooling operation in the channel dimension, To perform average pooling in the spatial dimension, To perform the maximum pooling operation in the spatial dimension; They are the three output results after activation by the sotfmax function. is the final fusion result.

[0147] The feature reconstruction module includes a residual module, an upsampling module and a Transformer module. The fused multi-scale feature map is input into the feature reconstruction module. The smallest scale feature map after fusion is sent to the Transformer module and the residual module for feature reconstruction. The remaining scale feature maps after fusion are reconstructed using the residual module and the upsampling module step by step. The scale is restored through the upsampling module to obtain a high-resolution prediction image corresponding to date t2. .

[0148] The network training module is used to train the spatiotemporal feature enhancement hierarchical network using the training set, construct and calculate the multidimensional spatiotemporal loss function, optimize the internal parameters of the spatiotemporal feature enhancement hierarchical network through back propagation, evaluate the quality of the optimized spatiotemporal feature enhancement hierarchical network using the test set, and save the trained spatiotemporal feature enhancement hierarchical network;

[0149] A multidimensional space-time loss function is constructed in the network training module. The multidimensional space-time loss function includes the time characteristic loss function and the space characteristic loss function. The expression of the multidimensional space-time loss function is:

[0150] ,

[0151] in, is the spatial characteristic loss function, is the time characteristic loss function, are weighted parameters of the two major loss functions respectively; the spatial characteristic loss function is used to ensure the accuracy of the generated image in terms of spatial resolution and details. The expression of the spatial characteristic loss function is:

[0152] ,

[0153]

[0154] in, is the cosine similarity loss function, which determines the directional consistency of the two images by calculating the cosine value of the angle between the two images. The closer the value is to 1, the more similar the overall features of the images are. is the structural similarity index, with a value range of [-1, 1], where 1 means that the two images are exactly the same and 0 means that the two images are completely different; are weighted parameters, The input i The predicted image and the real image corresponding to the sample image;

[0155] The temporal characteristic loss function is used to ensure the continuity and change trend of the generated image in the time dimension. The expression of the temporal characteristic loss function is:

[0156] ,

[0157] ,

[0158] ,

[0159] in, is the root mean square error loss function, which is used to measure the temporal consistency between the predicted image and the real image; is the consistency loss function, which measures the continuity of the generated image in the time dimension and ensures the smoothness of the time series by calculating the difference between the generated images at adjacent moments; represents the length of the time series, are weighted parameters, The input i The predicted image and the real image corresponding to the sample image, for Moment i The predicted image corresponding to the sample image, for Moment i The predicted image corresponding to the sample image.

[0160] In the network training module, the test set is input into the optimized spatiotemporal feature reinforcement hierarchical network, and the generated high-resolution prediction image is With high-resolution real images Compare the indicators in each dimension until they reach the ideal value;

[0161] The indicators of each dimension include structural similarity index SSIM, correlation coefficient CC, peak signal-to-noise ratio PSNR, root mean square error RMSE and spectral angle mapping SAM.

[0162] The prediction module is used to obtain high spatial resolution images of a certain date t1 in the target area within a period of time. and low spatial resolution images for forecast date t3 , form a remote sensing image pair, and perform preprocessing; send the preprocessed remote sensing image pair to the trained spatiotemporal feature enhancement hierarchical network for downscaling processing to obtain a high spatial resolution prediction image for the prediction date t3 .

[0163] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting, characterized in that: The following steps are involved: (1) Obtain high spatial resolution remote sensing images and low spatial resolution images of the same date in the target area over a period of time. The high spatial resolution image of one date and the low spatial resolution image of another date form a remote sensing image pair. After preprocessing, they are combined with the high spatial resolution real image of another date and divided into a training set and a test set. (2) Constructing a spatiotemporal feature enhancement hierarchical network, which includes a feature extraction module, a feature fusion module, and a feature reconstruction module; The feature extraction module includes a residual module, a downsampling module and a Transformer module. The remote sensing image is input into the feature extraction module for feature extraction, and the feature extraction is performed using the residual module and the downsampling module in succession. The minimum scale feature map is obtained by the last level residual module and then sent to the Transformer module for feature extraction to obtain a multi-scale feature map. The multi-scale feature map is fused at multiple scales through the feature fusion module. In the feature fusion module, the original feature map of the input high-resolution image is first and the original feature map of the low-resolution image Split evenly according to the number of channels, Represents the width M, height N and number of channels C of the feature map; The high-resolution image feature map after splitting becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times to obtain the downsampled feature images ; Then, the features of four different dimensions are extracted through deep convolution, and the dimensions are restored to , output multi-scale feature extraction part ; The split low-resolution image feature map becomes , each feature ,in Indicates The feature maps after splitting; After being processed in the same way as the high-resolution image feature map, the multi-scale feature extraction part is output ; Afterwards To connect channels, The channels are connected, and then normalized by the GELU activation function, and finally compared with the original feature map and Perform element-by-element multiplication and finally output multi-scale features and ; Multi-scale features and Global average pooling and global maximum pooling operations are performed in the spatial dimension and channel dimension respectively, and then the and The pooling results in the spatial dimension and channel dimension are connected, and then the spatial features are processed by two-way 2D convolution. The channel features are processed by a shared MLP, and finally activated by the softmax function to obtain a three-way output result. and , and output the results of two of them and After summing, the original feature map Perform element-by-element multiplication and output the result in another way Directly with the original feature map Perform element-by-element multiplication, and finally sum the two results to get the final fusion result; The feature reconstruction module includes a residual module, an upsampling module and a Transformer module. The fused multi-scale feature map is input into the feature reconstruction module. The fused minimum-scale feature map is sent to the Transformer module and the residual module for feature reconstruction. The remaining scale feature maps after fusion are reconstructed using the residual module and the upsampling module step by step to obtain a high spatial resolution prediction image corresponding to another date. (3) Use the training set to train the spatiotemporal feature enhancement hierarchical network, construct and calculate the multidimensional spatiotemporal loss function, optimize the internal parameters of the spatiotemporal feature enhancement hierarchical network through back propagation, use the test set to evaluate the quality of the optimized spatiotemporal feature enhancement hierarchical network, and save the trained spatiotemporal feature enhancement hierarchical network; (4) Obtain a high-spatial-resolution image of a certain date within a period of time in the target area and a low-spatial-resolution image of the predicted date to form a remote sensing image pair and perform preprocessing; send the preprocessed remote sensing image pair to the trained spatiotemporal feature enhancement hierarchical network for downscaling to obtain a high-spatial-resolution predicted image of the predicted date.

2. The remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting according to claim 1, characterized in that: The step (1) specifically includes the following steps: (1.1) Determine the target area for research, download the remote sensing images of the first satellite and the second satellite on the same date in the target area within the past three years, and form a remote sensing image pair. The remote sensing image pair includes at least the following bands: blue, green, red, and NIR near-infrared. The remote sensing image of the first satellite is a high spatial resolution image, and the remote sensing image of the second satellite is a low spatial resolution image. The spatial resolution of the remote sensing image of the first satellite is higher than that of the remote sensing image of the second satellite; (1.2) Perform preliminary screening and preprocessing of remote sensing image pairs, using platform screening and visual judgment to eliminate remote sensing images with cloud cover and streak or band noise problems; perform radiation correction, geometric correction and denoising on remote sensing image pairs, and then resample low spatial resolution images to the same image resolution as high spatial resolution images; (1.3) The preprocessed remote sensing image pairs are combined with high spatial resolution real images of another date and then divided into training sets and test sets.

3. The remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting according to claim 1, characterized in that: The left branch of the residual module in step (2) is a convolution kernel, a batch normalization layer BN and a ReLU activation function, followed by another convolution kernel and a batch normalization layer BN; The right branch of the residual module is a convolution kernel; Then the results of the left and right paths are summed, and a ReLU activation function is used to output the feature map.

4. The remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting according to claim 1, characterized in that: In the step (2), the Transformer module includes four Transformer encoder layers. The input minimum scale feature map passes through a linear projection layer and a layer normalization LN in turn, and then is linearly mapped into three matrices Q, K and V and input into a multi-head self-attention mechanism MSA. Subsequently, the result of the multi-head self-attention mechanism MSA is summed with the result of the linear projection layer for the first time. The result after the first summation passes through a layer normalization LN and a multi-layer perceptron MLP, and then is summed with the result of the first summation for the second time, so as to output a higher quality minimum scale feature map.

5. The remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting according to claim 1, characterized in that: In the step (3), a multidimensional space-time loss function is constructed. The multidimensional space-time loss function includes a time characteristic loss function and a space characteristic loss function. The expression of the multidimensional space-time loss function is: ， in, is the spatial characteristic loss function, is the time characteristic loss function, are the weighted parameters of the two major loss functions respectively. The expression of the spatial characteristic loss function is: ， ， in, is the cosine similarity loss function, which determines the directional consistency of the two images by calculating the cosine value of the angle between the two images. The closer the value is to 1, the more similar the overall features of the images are. is the structural similarity index, with a value range of [-1, 1], where 1 means that the two images are exactly the same, and 0 means that the two images are completely different; are weighted parameters, The input i The predicted image and the real image corresponding to the sample image; The expression of the time characteristic loss function is: ， ， ， in, is the root mean square error loss function, which is used to measure the temporal consistency between the predicted image and the real image; is the consistency loss function, which measures the continuity of the generated image in the time dimension and ensures the smoothness of the time series by calculating the difference between the generated images at adjacent moments; represents the length of the time series, are weighted parameters, The input i The predicted image and the real image corresponding to the sample image, for Moment i The predicted image corresponding to the sample image, for Moment i The predicted image corresponding to the sample image.

6. The remote sensing image spatiotemporal downscaling method based on multi-scale dynamic weighting according to claim 1, characterized in that: In the step (3), the test set is input into the optimized spatiotemporal feature enhancement hierarchical network, and the generated high spatial resolution predicted image is compared with the high spatial resolution real image in terms of indicators in various dimensions until the indicators approach the ideal values; the indicators in various dimensions include the structural similarity index SSIM, the correlation coefficient CC, the peak signal-to-noise ratio PSNR, the root mean square error RMSE and the spectral angle mapping SAM.

7. A remote sensing image spatiotemporal downscaling system based on multi-scale dynamic weighting, characterized in that: include: A data acquisition module is used to acquire high spatial resolution remote sensing images and low spatial resolution images of the same date in a target area within a period of time, wherein a high spatial resolution image of a certain date and a low spatial resolution image of another date form a remote sensing image pair, which is combined with a high spatial resolution real image of another date after preprocessing and divided into a training set and a test set; A network construction module is used to construct a spatiotemporal feature enhancement hierarchical network, which includes a feature extraction module, a feature fusion module and a feature reconstruction module; The feature extraction module includes a residual module, a downsampling module and a Transformer module. The remote sensing image is input into the feature extraction module for feature extraction, and the feature extraction is performed using the residual module and the downsampling module in succession. The minimum scale feature map is obtained by the last level residual module and then sent to the Transformer module for feature extraction to obtain a multi-scale feature map. The multi-scale feature map is fused at multiple scales through the feature fusion module. In the feature fusion module, the original feature map of the input high-resolution image is first and the original feature map of the low-resolution image Split evenly according to the number of channels, Represents the width M, height N and number of channels C of the feature map; The high-resolution image feature map after splitting becomes , each feature ,in Indicates The first feature map retains the original resolution, i.e. , the second feature map is downsampled twice by max pooling, the third is downsampled four times, and the fourth is downsampled eight times to obtain the downsampled feature images ; Then, the features of four different dimensions are extracted through deep convolution, and the dimensions are restored to , output multi-scale feature extraction part ; The split low-resolution image feature map becomes , each feature ,in Indicates The feature maps after splitting; After being processed in the same way as the high-resolution image feature map, the multi-scale feature extraction part is output ; Afterwards To connect channels, The channels are connected, and then normalized by the GELU activation function, and finally compared with the original feature map and Perform element-by-element multiplication and finally output multi-scale features and ; Multi-scale features and Global average pooling and global maximum pooling operations are performed in the spatial dimension and channel dimension respectively, and then the and The pooling results in the spatial dimension and channel dimension are connected, and then the spatial features are processed by two-way 2D convolution. The channel features are processed by a shared MLP, and finally activated by the softmax function to obtain a three-way output result. and , and output the results of two of them and After summing, the original feature map Perform element-by-element multiplication and output the result in another way Directly with the original feature map Perform element-by-element multiplication, and finally sum the two results to get the final fusion result; The feature reconstruction module includes a residual module, an upsampling module and a Transformer module. The fused multi-scale feature map is input into the feature reconstruction module. The fused minimum-scale feature map is sent to the Transformer module and the residual module for feature reconstruction. The remaining scale feature maps after fusion are reconstructed using the residual module and the upsampling module step by step to obtain a high-resolution prediction image corresponding to another date. The network training module is used to train the spatiotemporal feature enhancement hierarchical network using the training set, construct and calculate the multidimensional spatiotemporal loss function, optimize the internal parameters of the spatiotemporal feature enhancement hierarchical network through back propagation, evaluate the quality of the optimized spatiotemporal feature enhancement hierarchical network using the test set, and save the trained spatiotemporal feature enhancement hierarchical network; The prediction module is used to obtain a high-spatial-resolution image of a certain date within a period of time in the target area and a low-spatial-resolution image of the predicted date to form a remote sensing image pair and perform preprocessing; the preprocessed remote sensing image pair is sent to the trained spatiotemporal feature enhancement hierarchical network for downscaling to obtain a high-spatial-resolution prediction image of the predicted date.

8. The remote sensing image spatiotemporal downscaling system based on multi-scale dynamic weighting according to claim 7, characterized in that: The left branch of the residual module is a convolution kernel, a batch normalization layer BN and a ReLU activation function, followed by another convolution kernel and a batch normalization layer BN; The right branch of the residual module is a convolution kernel; Then the results of the left and right paths are summed, and a ReLU activation function is used to output the feature map.

Images