A remote sensing image reconstruction method and device that cooperates spatial channel attention with dynamic frequency domain multi-scale fusion

Abstract

The application discloses a remote sensing image reconstruction method and device combining spatial channel attention and dynamic frequency domain multi-scale fusion, which decomposes, fuses and reconstructs the spectrum of the down-sampling feature in the attention enhancement module, thereby enhancing the expression of high-frequency information by extracting frequency domain features, and can accurately reconstruct the surface detail features. Furthermore, the Landsat-8 image slice sample set with month information is taken as a training sample set, and the month information in the training sample is embedded into the Landsat-8 image data through a time embedding layer to obtain a comprehensive feature map including time and space for training, so that the influence of the dynamic change of the ground surface is considered. Through the above two points, the missing image data caused by factors such as cloud cover can be filled, and the red edge band of Sentinel-2 is reconstructed, thereby providing more accurate and comprehensive data support for the fields of agriculture and ecological monitoring.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing, and particularly relates to a remote sensing image reconstruction method and device based on cooperative spatial channel attention and dynamic frequency domain multi-scale fusion. BACKGROUND

[0002] Satellite remote sensing technology acquires the geometric and physical characteristics of the earth's surface by collecting different bands of electromagnetic spectrum data, and is widely used in land use, agriculture, forestry, water resources and disaster emergency response fields. With the development of global earth observation technology, multiple satellite systems such as Landsat and Sentinel series have been launched, but relying on single satellite data for large-scale and long-time sequence of land monitoring still faces many challenges. Although Landsat-8 / 9 has an 8-day revisit period, the Sentinel-2A / B combination can provide 5-day global coverage, but the working conditions of satellite sensors and atmospheric factors often lead to missing of remote sensing image data.

[0003] Especially in the field of agriculture, the key growth stages of most crops are short and usually coincide with the rainy season. If satellite images are missing due to cloud cover and other reasons, the key observation period will be missed, affecting continuous time series monitoring. Therefore, filling the data gap and ensuring data continuity are crucial for long-term and accurate land change monitoring.

[0004] Current remote sensing image filling techniques mainly include spatial-based methods, spectral-based methods, time-based methods, and spatio-temporal spectral-based methods, but these techniques are still based on a single data source, limiting the demand for high-frequency and continuous monitoring. Another effective solution is to fuse different satellite images to increase the monitoring frequency and the probability of obtaining cloud-free data, but the differences in acquisition geometry, spectral characteristics and spatial resolution often exist in different satellite image data, and simple image fusion will have great uncertainty.

[0005] The Harmonized Landsat Sentinel (HLS) product (HLSL30 and HLSS30) uses Hyperion hyperspectral data sets to simulate Landsat-8 and Sentinel-2 surface reflectance, and corrects the spectral differences between different bands by building a fixed linear regression model, solving the problem of differences between different remote sensing images. However, Landsat-8 lacks the three additional red edge bands provided by Sentinel-2, so HLS images still lack red edge bands when there is no Sentinel-2 data, and cannot completely solve the data missing problem, in addition, the spatial resolution of HLS data set is 30m.

[0006] The patent application with the publication number CN118691496A discloses an image fusion method and system based on a multi-scale smoothing sharpening filter, belonging to the technical field of remote sensing image processing. The method includes: obtaining Landsat-8 images at a target time and Sentinel-2 images at a reference time, and respectively pre-processing; extracting high-frequency components from the pre-processed images through a smoothing sharpening filter; extracting multi-scale detail images from the high-frequency components of the Sentinel-2 images using a multi-scale smoothing sharpening filter; summing the multi-scale detail images of the Sentinel-2 images, and adding them to the pre-processed Landsat-8 images to obtain an image fusion result. The patent application improves the accuracy of Landsat-8 image and Sentinel-2 image fusion by re-extracting and transferring detail information from the high-frequency components of the Sentinel-2 images, and can accurately capture phenology and land cover changes. However, for regions with large differences in full coverage heterogeneity, it is difficult to complete the Sentinel-2 image with high quality.

[0007] Therefore, it is urgent to develop a reconstruction framework for remote sensing images to reconstruct missing images, improve resolution, and ensure the integrity of the time sequence of the images. SUMMARY

[0008] The present application provides a remote sensing image reconstruction method that cooperates spatial channel attention with dynamic frequency domain multi-scale fusion. The method can effectively fill the data gaps of Sentinel-2 images caused by factors such as cloud cover, and reconstruct the red edge band of Sentinel-2.

[0009] The present application provides a remote sensing image reconstruction method that cooperates spatial channel attention with dynamic frequency domain multi-scale fusion. The method can effectively fill the data gaps of Sentinel-2 images caused by factors such as cloud cover, and reconstruct the red edge band of Sentinel-2.

[0010] The filtered Sentinel-2 and Landsat-8 images are pre-processed and sliced to obtain annual monthly time series, time and space matched Landsat-8 and Sentinel-2 image pairs, and the Landsat-8 image slice sample set with month information is used as the training sample set, and the corresponding Sentinel-2 image slice sample is used as the label.

[0011] The training model is constructed, and the training model comprises a time embedding layer, a multi-scale convolution module, a down-sampling module, an enhanced attention module, an up-sampling module and an activation function. The month information in the training sample is embedded into Landsat-8 image data by the time embedding layer to obtain a comprehensive feature map. The comprehensive feature map is sequentially passed through the multi-scale convolution module and the down-sampling module to obtain down-sampling features. The frequency domain reconstruction features and the self-attention features of the down-sampling features are extracted by the enhanced attention module, and then the fusion features are obtained by fusion. The fusion features are sequentially passed through the up-sampling module and the activation function to obtain the predicted Sentinel-2 image.

[0012] A loss function is constructed by the predicted Sentinel-2 image and the label. The training model is trained based on the training sample set by the loss function to obtain a remote sensing image reconstruction model.

[0013] In application, the Landsat-8 image of the adjacent date corresponding to the blocked Sentinel-2 image and the corresponding month information are input into the remote sensing image reconstruction model to obtain the reconstructed Sentinel-2 image.

[0014] Preferably, the frequency domain reconstruction features of the down-sampling features are extracted by the enhanced attention module, and the method comprises:

[0015] The down-sampling features are converted from the spatial domain to the frequency domain by the enhanced attention module, and then the high-frequency components and the low-frequency components are obtained by spectrum decomposition. The high-frequency components and the low-frequency components are converted back to the spatial domain and then fused to obtain the frequency domain reconstruction features.

[0016] Preferably, the method for obtaining the frequency domain reconstruction features comprises:

[0017] The down-sampling features are converted from the spatial domain to the frequency domain by two-dimensional fast Fourier transform to obtain the spectrum of the down-sampling features. The spectrum is decomposed into high-frequency components and low-frequency components by a differentiable band separation algorithm. The high-frequency components and the low-frequency components are converted back to the spatial domain by inverse two-dimensional fast Fourier transform, and then fused to obtain the frequency domain reconstruction features.

[0018] Preferably, the fusion features are obtained by fusing the frequency domain features and the self-attention features of the down-sampling features by the enhanced attention module, and the method comprises:

[0019] The enhanced attention module comprises a first branch, a second branch and a fusion module.

[0020] The first branch is a spatial channel attention module. The spatial self-attention of the down-sampling features is obtained by the spatial channel attention module. The self-attention result is connected with the down-sampling features in residual to obtain the self-attention features.

[0021] The second branch is a frequency domain layer, and a frequency domain reconstruction feature of a down-sampling feature is extracted through the frequency domain layer;

[0022] The fusion module performs pixel-by-pixel weighted fusion on the self-attention feature and the frequency domain reconstruction feature to obtain a fusion feature.

[0023] Preferably, the spatio-temporal comprehensive feature map is obtained by embedding the month information in the Landsat-8 image information through the time embedding layer, and the embedding includes:

[0024] The time embedding layer includes an embedding layer, and the embedding layer is used to map the month information into a high-dimensional feature vector.

[0025] Preferably, the method for obtaining the slice sample of the Landsat-8 and Sentinel-2 image pairs in the time and space matching manner in the whole year and month sequence includes:

[0026] The region of the remote sensing image is divided into a plurality of random points through a grid layout, and then points covered by the Landsat-8 and Sentinel-2 images in each month are screened out.

[0027] Then, after the image pair slice sample with the number of selected null pixels accounting for less than 1% of the total number of pixels is removed, the missing pixels are compensated through bicubic interpolation to obtain the slice sample of the Landsat-8 and Sentinel-2 image pairs in the time and space matching manner in the whole year and month sequence.

[0028] Preferably, the screened Sentinel-2 and Landsat-8 images are preprocessed, and the preprocessing includes:

[0029] The Sentinel-2 images with a cloud coverage lower than 10% are screened out, the Sentinel-2 images with a cloud coverage lower than 15% are screened out, and the screened Sentinel-2 and Landsat-8 images are sequentially subjected to radiation calibration, atmospheric correction, cloud mask, BRDF correction, terrain correction and resampling preprocessing.

[0030] Preferably, the multi-scale convolution module comprises a plurality of convolution layers with different convolution kernel layers and an activation layer, the integrated feature map is subjected to feature extraction by the plurality of convolution layers with different convolution kernel layers to obtain feature maps with different sizes, the feature maps with different sizes are spliced and then activated by the activation layer, and the activation result is input into the down-sampling module.

[0031] The application further provides a remote sensing image reconstruction device with synergic spatial channel attention and dynamic frequency domain multi-scale fusion, comprising a memory and one or more processors, the memory storing executable codes, and the one or more processors executing the executable codes to implement the remote sensing image reconstruction with synergic spatial channel attention and dynamic frequency domain multi-scale fusion.

[0032] Compared with the prior art, the application has the following beneficial effects:

[0033] Since the area of the remote sensing image is an area with full land surface coverage and complex terrain, i.e., an area with large differences in spatial heterogeneity of different land surface coverage environments, the application can accurately reconstruct the missing Sentinel-2 image by enhancing the decomposition, fusion and reconstruction of the frequency spectrum of the down-sampled features in the attention module, extracting the frequency domain features and accurately reconstructing the land surface detail information with different spatial heterogeneity.

[0034] Meanwhile, since the influence of the land surface changes over time, the application uses the Landsat-8 image slice sample set with month information as the training sample set, embeds the month information in the Landsat-8 image data through a time embedding layer to obtain an integrated feature map including time and space for training, so that the influence of time on the Sentinel-2 image is considered.

[0035] Through the above two points, the missing image data caused by cloud cover and other factors can be filled, and the red edge band of the Sentinel-2 is reconstructed, providing more accurate and comprehensive data support for the fields of agriculture and ecological monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flow chart of the remote sensing image reconstruction method with synergic spatial channel attention and dynamic frequency domain multi-scale fusion according to the embodiment of the application is shown in the figure.

[0037] Figure 2 The structure diagram of the remote sensing image reconstruction method with synergic spatial channel attention and dynamic frequency domain multi-scale fusion according to the embodiment of the application is shown in the figure.

[0038] Figure 3 The residual connection structure diagram according to the embodiment of the application is shown in the figure.

[0039] Figure 4A comparison chart of reconstruction results of specific embodiments and other methods of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and technical effect of the present application clearer, the present application is further described in detail below in combination with the drawings of the specification.

[0041] Since the area of full-coverage remote sensing images mostly includes multiple land use types, and different types of ground features have great differences, and the prior art does not consider the influence of dynamic changes of the ground on the quality of Sentinel-2 image completion when performing Sentinel-2 image completion, the specific embodiments of the present application realize the reconstruction of Sentinel-2 images by introducing dynamic frequency to extract frequency domain reconstruction features of down-sampling features.

[0042] The specific embodiments of the present application provide a remote sensing image reconstruction method cooperating spatial channel attention with dynamic frequency domain multi-scale fusion, as shown in Figure 1 , which comprises:

[0043] (1) Constructing a training sample set and a label: pre-processing and slicing the screened Sentinel-2 and Landsat-8 images to obtain the time series, time and space matched Landsat-8 and Sentinel-2 image slice samples, taking the Landsat-8 image slice sample set with month information as the training sample set, and taking the corresponding Sentinel-2 image slice sample as the label.

[0044] In the present embodiment, a typical cloudy and rainy area in southern China, Zhejiang Province, is selected as an example to construct a complete time series of remote sensing images of Zhejiang Province in 2019. Before making the slice sample pair, first obtain multi-source remote sensing data covering Zhejiang Province and pre-process it, which specifically includes the following steps:

[0045] The specific steps for obtaining regional remote sensing images provided by the specific embodiments of the present application are: obtaining Sentinel-2 and Landsat-8 images of Zhejiang Province in 2019 on the Google Earth (GEE) engine, through data screening, selecting Sentinel-2 images with cloud coverage lower than 10% (except that the cloud coverage of the image in June is lower than 50% and the cloud coverage of the image in July is lower than 40%), a total of 760 images. Landsat-8 images with cloud coverage lower than 15% (except that the cloud coverage of the image in January and June is lower than 50% and the cloud coverage of the image in July is lower than 40%), a total of 103 images are obtained.

[0046] After obtaining cloud-screened Sentinel-2 and Landsat-8 imagery, the multi-source remote sensing data undergoes several preprocessing operations, including radiometric calibration, atmospheric correction, cloud masking, BRDF correction, terrain correction, and resampling. First, to reduce residual errors introduced by different atmospheric correction methods, this study applies the Py6S atmospheric correction model to all Landsat-8 and Sentinel-2 top-of-atmosphere (TOA) imagery. In this model, the apparent zenith angle is encoded as "0." The cloud mask for Sentinel-2 data is developed based on the cloud probability data provided by GEE. By selecting a cloud cover threshold, the corresponding cloud mask is obtained and applied to the atmospherically corrected Sentinel-2 imagery, effectively removing the effects of clouds. For Landsat-8 imagery, clouds and cloud shadows are masked using the quality assessment band (QA_PIXEL) provided by GEE. Next, a bidirectional reflectance distribution function (BRDF) model was used to account for reflectance variations in Landsat-8 and Sentinel-2 imagery due to observation geometry and atmospheric effects. The viewing angle was set to an orthographic perspective, and illumination was determined based on the image center latitude. Subsequently, a modified sun-canopy-sensor terrain correction method was used to adjust for reflectance effects due to slope, aspect, and elevation. Finally, the Sentinel-2 imagery was resampled to a 10-meter resolution, reprojected to a Mercator projection, and uniformly converted to the World Geodetic System 1984 (WGS1984). Georeferencing of both data types was performed using the Sentinel-2 global reference image data provided by the European Space Agency.

[0047] The specific steps for obtaining slice sample pairs provided in a specific embodiment of the present invention are as follows: After completing data preprocessing, 1,000 random points were generated throughout Zhejiang Province using a grid pattern. Points covered by both Landsat-8 and Sentinel-2 imagery for each month of 2019 were then screened. A 2.4 km × 2.4 km image block was extracted centered on the point. Image pairs were screened by matching the closest date between the two data points for each point, obtaining a total of 9,238 pairs of slice samples for the monthly Landsat-8 and Sentinel-2 image pairs. Image pairs with null pixels accounting for less than 1% of the total number of slice pixels were screened. Bicubic interpolation was then used to compensate for the missing pixels as much as possible. After several rounds of manual screening, approximately 6,725 pairs of high-quality Landsat-8 and Sentinel-2 image pairs for the entire year of 2019 were retained. The Landsat-8 image slice sample set with monthly information was used as the training sample set, and the corresponding Sentinel-2 image slice samples were used as labels.

[0048] (2) Constructing a training model, that is, constructing a collaborative spatial channel attention and dynamic frequency domain multi-scale fusion network, specifically including the following steps:

[0049] The training model provided by the embodiment of the application includes a time embedding layer, a multi-scale convolution module, a down-sampling module, an enhanced attention module, an up-sampling module and an activation function.

[0050] The time embedding layer provided by the embodiment of the application includes an embedding layer (Embedding Layer) that maps month information (represented in the form of numbers) to a high-dimensional feature vector, normalizes the embedded time feature, and splices the embedded time feature and Landsat-8 image data in the channel dimension to form a comprehensive feature map X1 containing time and space information.

[0051] The interpolation module provided by the embodiment of the application adopts an interpolation technique to improve the spatial resolution of the image to obtain X2.

[0052] The multi-scale convolution module provided by the embodiment of the application has the following specific steps: the module includes three convolution operations, as shown in the following formula: Figure 2 The first convolution operation includes 1 convolution kernel, aims to capture small objects and local changes, and local relationships in the image to obtain X3; the second convolution operation increases the number of convolution kernels to 2, expands the receptive field, and helps to capture larger object structures and global context information to obtain X4; the last convolution operation increases the number of convolution layers to 3, improves the perception ability of the network to larger scale features and the overall scene to obtain X5, and finally splices the three feature maps to obtain X6, that is, X3+X4+X5=X6. The module is composed of a convolution Conv layer and a (ReLU) layer.

[0053] The down-sampling module provided by the embodiment of the application has the following specific steps: the down-sampling process includes three layers, as shown in the following formula: Figure 2 Each layer includes 2 convolution layers, a Pixel-Unshuffle layer and a ReLU activation layer, and this layer-by-layer down-sampling method can extract higher-level and more abstract semantic features, increase the receptive field, enable each feature to capture more context information in a larger range, and reduce the consumption of computing resources and alleviate data redundancy. The first down-sampling layer outputs X7, the second down-sampling layer outputs X8, and the last down-sampling layer outputs X9.

[0054] The enhanced attention mechanism module provided by the embodiment of the application has the following specific steps: the enhanced attention mechanism module introduces a frequency domain layer on the basis of the collaborative spatial channel attention module, as shown in the following formula: Figure 2As shown. X9 is converted from the spatial domain to the frequency domain by two-dimensional fast Fourier transform (FFT2), and the spectrum is decomposed into high-frequency components X using a differentiable frequency band separation algorithm. high With the low frequency component X low ; Secondly, the trainable weight coefficient β∈[0,1] is introduced to establish a dynamic fusion equation to obtain the fused frequency domain feature X fused =β 2 ·X high +X low , where X high 、X low Respectively represent high / low frequency features. The specific embodiment of the present invention can reduce noise by fusing high / low frequency features and retain high frequency information to a large extent. In addition, the frequency domain features are converted to the spatial domain by inverse FFT2 to obtain the frequency domain reconstruction feature X 10 , X9 is processed by collaborative spatial channel attention to obtain X 11 , X 11 The collaborative spatial channel attention feature X is obtained by combining it with X9 through residual connection. 12 ; Adaptive fusion weight α∈R^(1×C×1×1) is used to adjust X 10 and X 12 Perform channel-by-channel weighted fusion to obtain the fusion feature X 13 =α⊙X 10 +(1-α)⊙X 12 , where ⊙ represents the channel-by-channel multiplication operation; finally, a multi-layer perceptron (MLP) containing a gated linear unit (GELU) is used to perform nonlinear transformation on the fusion features. The layers used for nonlinear transformation include linear transformation layer, GELU layer, linear transformation layer and Dropout layer, and the output X processed by the enhanced attention mechanism is obtained. 14 , its mathematical expression is: X 14 =MLP(X 13 )+X 13 , achieving deep optimization of feature space.

[0055] The specific steps of designing the enhanced upsampling module provided by the specific embodiment of the present invention are as follows: the upsampling module corresponds to the downsampling process, such as Figure 2As shown, the up-sampling module protects three layers, the first layer sequentially includes a convolution layer, a Pixel-shuffle layer and a ReLU activation layer, the second layer and the third layer each independently sequentially include a convolution layer, a ReLU activation layer, a residual connection layer, a convolution layer, a Pixel-shuffle layer and a ReLU activation layer, and the last layer sequentially includes a convolution layer and a RE LU activation layer, the outputs of the first three layers of up-sampling are connected with the corresponding inputs of down-sampling through a jump connection, and the spatial resolution of the feature map is restored through three-step up-sampling and convolution operation, and convolution operation and residual block processing are performed after each up-sampling. First, the input feature map X 14 is expanded to twice the original number of channels through a convolution layer; then, through a depth-to-space conversion operation (Pixel shuffle layer), the feature map is rearranged from the depth dimension to the spatial dimension, up-sampling is realized, and the spatial resolution is doubled; then, the up-sampled result is spliced with the down-sampled feature map of the previous stage through a jump connection, and convolution operation is performed, multi-scale information is fused, the feature representation in the reconstruction process is enhanced, the gradient vanishing problem is avoided or reduced, and X 15 , X 16 , and X 17 are obtained through three times of up-sampling in turn. The introduced residual connection mechanism enables the network to learn the residual mapping between the input and the output, thereby accelerating the training process and improving the convergence and generalization ability of the network, and the residual block structure includes Conv, ReLU, residual scaling (ResScale) and element summation operation. In the ResScal operation, a constant scalar is used to deepen the network and slow down the convergence speed, thereby stabilizing the training without increasing the parameters, as shown in Figure 3 . Finally, the processed feature map X 17 is mapped to the final output result, i.e., the predicted Sentinel-2 image, through a convolution layer and a ReLU layer.

[0056] (3) A loss function is constructed by predicting the Sentinel-2 image and the label, and a remote sensing image reconstruction model is obtained by training the training model based on the training sample set:

[0057] The specific steps of the loss function provided in the embodiment of the application are as follows: assuming that L is the input of the network and S is the expected output (or known reference data). The network training is to learn a mapping function S=f(L;w), wherein S is the predicted output, w is a set of all parameters including filter weights and bias, and the mean absolute error MAE is used as the loss function: wherein L represents Landsat-8, S represents Sentinel-2, f(L) is the predicted output, i.e., the reconstructed Sentinel-2, and n represents the number of training samples.

[0058] The specific steps of the network parameter setting provided by the embodiment of the application are as follows: a deep learning framework pytorch1.9.0 is used, epoch is set to 200 rounds, the batch size is 64, the division ratio of the training set, the validation set and the test set is 8:1:1, the initial learning rate is set to 1x10 -4 , the model parameters are updated using the Adam optimizer until the loss no longer decreases.

[0059] (4) The occluded Sentinel-2 image is completed by the remote sensing image reconstruction model, and the completed Sentinel-2 image is evaluated, including:

[0060] In application, the occluded Sentinel-2 image corresponding Landsat-8 and the corresponding month information are input into the remote sensing image reconstruction model to obtain the reconstructed Sentinel-2 image.

[0061] The specific steps of the remote sensing image reconstruction provided by the embodiment of the application are as follows: the Landsat-8 data in the test set is input into the trained remote sensing image reconstruction model to reconstruct the corresponding Sentinel-2 data, and the reconstructed image is obtained.

[0062] The specific steps of the reconstructed image precision evaluation provided by the embodiment of the application are as follows: the pixel values and the band values of the reconstructed image and the real image are compared, and the mean absolute error (MAE) and the peak signal-to-noise ratio (PSNR) are generally used. The formula is as follows:

[0063]

[0064] Wherein, Y and respectively represent the real image and the reconstructed image, and m and n respectively represent the height and width of the image. The lower the MAE value, the better the reconstruction performance.

[0065]

[0066] Wherein, max(Y) represents the maximum pixel value of the image.

[0067] The network established in the embodiment has an evaluation index MAE of 0.0165 and a PSNR of 34.162 in comparison with a reconstructed image and a real image. Compared with a traditional interpolation method Bicubic (MAE: 0.0308, PSNR: 27.232) and other models (such as AUNet: MAE: 0.0185, PSNR: 33.291; DCMNET: MAE: 0.0182, PSNR: 33.424; Transenet: MAE: 0.0189, PSNR: 33.164; RCAN: MAE: 0.0202, PSNR: 32.634), the network provided in the application has a significant improvement in image reconstruction accuracy. In terms of vision, the reconstructed image obtained by the network provided in the application presents clearer and continuous edges, the color tone is closer to the real Sentinel-2, and the visual effect is better than that of other models.

[0068] As shown in Figure 4 the first column is an original Sentinel-2 image, the second, third, fourth, fifth and sixth columns are respectively Sentinel-2 images reconstructed based on Landsat-8 using Bicubic, AUNet, Transenet, RCAN and DCMNE methods, and the last column is a reconstructed image obtained based on the algorithm provided in the application. The fourth row is a local enlarged view of the image in the third row.

[0069] In summary, the embodiment of the application improves the accuracy and robustness of image analysis by extracting feature information at multiple scales and strengthening key features by using a strong attention mechanism. In the specific implementation process, first, multi-scale convolution layers are used to extract multi-level and different scale feature maps from the input image. On this basis, a strong attention mechanism is used to weight and fuse different scale features, so as to more accurately focus on important areas or information in the image.

[0070] Compared with the prior art based on traditional image interpolation methods and using traditional convolutional neural networks or simple multi-scale networks, the framework provided in the application has a significant advantage in reconstruction accuracy. Compared with the reconstruction method using a single image or a global model, the enhanced attention mechanism can effectively suppress the influence of noise, and has higher accuracy and better effect.

[0071] The application further provides a remote sensing image reconstruction device cooperating with spatial channel attention and dynamic frequency domain multi-scale fusion, characterized by comprising a memory and one or more processors, the memory stores executable code, and the one or more processors execute the executable code to implement the remote sensing image reconstruction device cooperating with spatial channel attention and dynamic frequency domain multi-scale fusion.

Claims

1. A remote sensing image reconstruction method based on coordinated spatial channel attention and dynamic frequency domain multi-scale fusion, characterized in that: include: The selected Sentinel-2 and Landsat-8 images were preprocessed and sliced ​​to obtain the Landsat-8 and Sentinel-2 image pair slice samples with matching time and space throughout the year. The Landsat-8 image slice sample set with monthly information was used as the training sample set, and the corresponding Sentinel-2 image slice samples were used as labels. Constructing a training model, the training model including a time embedding layer, a multi-scale convolution module, a downsampling module, an enhanced attention module, an upsampling module, and an activation function, wherein the month information in the training sample is embedded into the Landsat-8 image data through the time embedding layer to obtain a comprehensive feature map, the comprehensive feature map is sequentially passed through the multi-scale convolution module and the downsampling module to obtain downsampling features, the frequency domain reconstruction features and self-attention features of the downsampling features are extracted through the enhanced attention module, and then fused to obtain fused features, and the fused features are sequentially passed through the upsampling module and the activation function to obtain a predicted Sentinel-2 image; By predicting Sentinel-2 images and labels to build a loss function, the remote sensing image reconstruction model is obtained by training the model through the loss function based on the training sample set; When applied, the Landsat-8 image of the nearby date corresponding to the blocked Sentinel-2 image and the corresponding month information are input into the remote sensing image reconstruction model to obtain the reconstructed Sentinel-2 image.

2. The remote sensing image reconstruction method based on collaborative spatial channel attention and dynamic frequency domain multi-scale fusion according to claim 1 is characterized in that: Extracting frequency domain reconstruction features of the downsampled features through the enhanced attention module includes: The enhanced attention module converts the downsampled features from the spatial domain to the frequency domain, and then performs spectral decomposition to obtain high-frequency components and low-frequency components. The high-frequency components and low-frequency components are converted back to the spatial domain and fused to obtain frequency domain reconstruction features.

3. The remote sensing image reconstruction method based on collaborative spatial channel attention and dynamic frequency domain multi-scale fusion according to claim 2 is characterized in that: The method for obtaining frequency domain reconstruction features includes: The downsampled features are subjected to a two-dimensional fast Fourier transform to convert the spatial domain into the frequency domain to obtain the spectrum of the downsampled features. The spectrum is decomposed into high-frequency components and low-frequency components using a differentiable frequency band separation algorithm. The high-frequency components and low-frequency components are subjected to an inverse two-dimensional fast Fourier transform to convert them back to the spatial domain and then fused to obtain the frequency domain reconstructed features.

4. The remote sensing image reconstruction method of collaborative spatial channel attention and dynamic frequency domain multi-scale fusion according to claim 1 is characterized in that: The frequency domain features of the downsampled features and the self-attention features are fused by the enhanced attention module to obtain fused features, including: The enhanced attention module includes a first branch, a second branch and a fusion module; The first branch is a spatial channel attention module, which performs spatial self-attention on the downsampled features through the spatial channel attention module, and performs a residual connection between the spatial self-attention result and the downsampled features to obtain the self-attention feature; The second branch is a frequency domain layer, which extracts frequency domain reconstruction features of down-sampled features through the frequency domain layer; The fusion module performs pixel-by-pixel weighted fusion of the self-attention feature and the frequency domain reconstruction feature to obtain the fusion feature.

5. The remote sensing image reconstruction method of collaborative spatial channel attention and dynamic frequency domain multi-scale fusion according to claim 1 is characterized in that: The time embedding layer embeds the month information in the training sample into the Landsat-8 image information to obtain a spatiotemporal comprehensive feature map, including: The time embedding layer includes an embedding layer, which maps the month information into a high-dimensional feature vector. After the high-dimensional feature vector is normalized, it is added element by element with the corresponding Landsat-8 image data along the channel dimension to obtain a comprehensive feature map containing time and space information.

6. The remote sensing image reconstruction method of collaborative spatial channel attention and dynamic frequency domain multi-scale fusion according to claim 1 is characterized in that: The method for obtaining a full-year monthly time series of Landsat-8 and Sentinel-2 image pair slice samples that match time and space, includes: The remote sensing image area is gridded to generate multiple random points. Then, points covered by both Landsat-8 and Sentinel-2 images are selected for each month. Image blocks of a set area are extracted with these selected points as the center. The image blocks corresponding to the closest date of the Landsat-8 and Sentinel-2 images for each point are extracted to preliminarily screen and obtain Landsat-8 and Sentinel-2 image pair slice samples that form a full-year monthly time series and match time and space. Then, after selecting image pair slice samples in which the number of null pixels accounts for less than 1% of the total number of pixels in the slice, bicubic interpolation is used to fill the missing pixels to obtain Landsat-8 and Sentinel-2 image pair slice samples with a full year of time series and matching time and space.

7. The remote sensing image reconstruction method of collaborative spatial channel attention and dynamic frequency domain multi-scale fusion according to claim 1 is characterized in that: Pre-processing of selected Sentinel-2 and Landsat-8 images, including: Sentinel-2 images with cloud cover less than 10% and Sentinel-2 images with cloud cover less than 15% were screened, and the screened Sentinel-2 and Landsat-8 images were preprocessed in sequence through radiometric calibration, atmospheric correction, cloud mask, BRDF correction, terrain correction, and resampling.

8. The remote sensing image reconstruction method of collaborative spatial channel attention and dynamic frequency domain multi-scale fusion according to claim 1 is characterized in that: The multi-scale convolution module includes multiple convolution layers and activation layers with different numbers of convolution kernel layers. The comprehensive feature map is extracted through multiple convolution layers with different numbers of convolution kernel layers to obtain feature maps of different sizes. The feature maps of different sizes are spliced ​​and activated through the activation layer, and the activation results are input into the downsampling module.

9. A remote sensing image reconstruction device that coordinates spatial channel attention and dynamic frequency domain multi-scale fusion, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to realize the remote sensing image reconstruction of collaborative spatial channel attention and dynamic frequency domain multi-scale fusion as described in any one of claims 1-8.

Images