A Spatiotemporal Fusion Method and System for Remote Sensing Images Based on Parallel Interaction of Swing Transformer and CNN

Abstract

The application provides a remote sensing image space-time fusion method and system based on parallel interaction of Swin Transformer and CNN, and the method comprises the following steps: acquiring historical remote sensing image product data and requirement information, performing cloud removal processing; data preprocessing; constructing a remote sensing image space-time fusion model based on parallel interaction of Swin Transformer and CNN; training a remote sensing image space-time fusion model in a moderate fitting state; and predicting remote sensing images through the obtained space-time fusion model. The application solves the technical problems that the method cannot be applied to a scene with a large information change amplitude in a short time, that image resolution is low, which leads to limited provided information, large pixel classification error, low model robustness, insufficient extracted time change information and spatial features, and complex fusion method introducing noise.

Description

Technical Field

[0001] This invention relates to the field of remote sensing technology, and more specifically to a spatiotemporal fusion method for remote sensing images based on the parallel interaction of Swing Transformer and CNN. Background Technology

[0002] Spatiotemporal fusion technology for remote sensing images aims to combine different types of remote sensing images from multiple satellite sensors to obtain remote sensing images with high spatial and temporal resolution. For example, remote sensing satellite images obtained by the high spatial resolution Landsat series sensors have a spatial resolution of approximately 30m, but a long revisit period. In contrast, the Medium Resolution Imaging Spectroradiometer (MODIS) on the low spatial resolution Aqua and Terra satellites can obtain high temporal resolution images with a revisit period of one day, but the spatial resolution is lower, only 250–1000m. The spatiotemporal fusion method combines these two types of images, resulting in a fused image that simultaneously possesses high temporal and spatial resolution, preserving details and other features of ground features while accommodating rapid changes in the ground.

[0003] In fact, there are already many mature algorithms for spatiotemporal fusion of remote sensing images, and they have achieved certain results. Generally speaking, existing spatiotemporal fusion methods can be subdivided into three categories: decomposition-based methods, weighted methods, and learning-based spatiotemporal fusion methods.

[0004] The essence of weighted function-based algorithms is to calculate the weights of similar adjacent pixels in the input spectral information and sum them. However, this method needs improvement if the information changes too much in a short period of time. The key to decomposition-based fusion methods is to decompose the spectral information at the prediction time point and then use the decomposition results to predict unknown high spatiotemporal resolution images. Decomposition-based methods are relatively simple in principle and easy to implement, but low-resolution images provide limited information due to their low resolution, leading to significant errors in pixel classification, especially in heterogeneous areas with diverse land cover types. Therefore, this method is mainly suitable for areas with fewer land cover types. In recent years, learning-based methods have developed rapidly. These methods mostly do not rely on certain assumptions for fusion; they learn and extract abstract features from acquired historical data and then use these features to reconstruct and generate predicted images. However, they may also suffer from low model robustness, insufficient temporal and spatial information in the coarse images extracted by convolutional neural networks, and overly complex fusion methods may introduce noise.

[0005] The existing invention patent application document CN116363527A, entitled "A Remote Sensing Image Change Detection Method Based on Interactive Feature Perception," includes the following steps: selecting a registered public remote sensing change detection dataset and preprocessing the dataset; building a remote sensing change detection model based on an encoder-decoder, using a change feature interaction layer in the encoder and a symmetric change feature fusion layer in the decoder, with the model ultimately outputting a binary prediction result of the change detection; constructing and calculating a loss function and performing backpropagation; and inputting the training set and validation set into the constructed remote sensing change detection network for training to obtain a trained remote sensing change detection model.

[0006] The existing invention patent application document CN116740415A, entitled "A Remote Sensing Image Classification Method Based on a Bi-branch Graph Convolutional Network Using Spectral Decomposition Functions," describes a bi-branch graph convolutional network (MGCN) that specifically includes a superpixel-based spectral decomposition function module (SSM), a bi-branch graph construction module (MGM), and a multi-branch feature fusion module (FFM). First, the spectral decomposition module (SSM) ensures spectral homogeneity, guaranteeing the spectral uniformity of the initial superpixels, thereby enhancing the discriminative ability of graph nodes. In the bi-branch graph construction module (MGM), spatial adjacency graphs and spectral similarity graphs are constructed from different perspectives based on the representation nodes. Finally, the multi-branch feature fusion module (FFM) is designed for adaptive feature fusion across multiple branches to improve feature discriminative ability.

[0007] The existing invention patent application document CN117237740A, entitled "A SAR Image Classification Method Based on CNN and Transformer," includes the following steps: acquiring synthetic aperture radar (SAR) images; extracting local features of the images through a CNN module and extracting global features of the images through an attention-based Transformer module; performing hierarchical feature fusion through a parallel network structure, wherein the parallel network structure includes multiple layers of CNN modules and Transformer modules that output features at different scales; each layer also includes a CTF module that adaptively fuses the local features, global features, and features fused from the previous layer; and inputting the finally fused features from the parallel network structure into a linear classifier that performs global average pooling and layer normalization for classification result output.

[0008] The existing literature, "Research on Cloud Detection Algorithm for Remote Sensing Images Based on Deep Learning," conducts extensive research on cloud detection algorithms based on convolutional neural networks (CNNs). It finds that when fusing features at different levels, current methods neglect the similarities and differences between features and their value for cloud detection, leading to the introduction of significant noise into the fusion process. Chapter 3 of this paper designs a feature-aware aggregation algorithm for cloud detection in remote sensing images. To better fuse low-level and high-level features of CNNs and alleviate the semantic gap problem in cross-level feature fusion, Chapter 4 designs a feature interaction graph convolution algorithm for cloud detection in remote sensing images. To overcome the limitations of global context modeling in CNNs and fully leverage the advantages of CNNs and visual Transformers, Chapter 5 designs a parallel branching algorithm combining Transformers and CNNs for cloud detection in remote sensing images.

[0009] The existing literature, "Research on Change Detection of Remote Sensing Images Based on Local-Global Feature Representation," designs a dual-branch fusion network based on a parallel structure of CNN and Transformer. Using CNN for downsampling in the initial stage of the Transformer network effectively improves model performance. Therefore, in the Transformer branch, the input features first pass through a CNN-based Stem layer. This layer uses convolution and max pooling to downsample the input image, quickly and effectively capturing shallow information and obtaining a feature map half the size of the original image. Its structural diagram is shown below. Figure 4-2 As shown. Secondly, four consecutive axially cross-attention layers are used to model long-distance dependencies between image patches. Each layer contains 3, 4, 6, and 3 axially cross-attention blocks, with 4, 8, 16, and 32 attention heads, respectively. Given the superior performance of the feature pyramid, this work also considers a similar structure in the Transformer branch. Specifically, from stages D2 to D5, the Transformer branch downsamples the feature maps by a factor of 2 layer to obtain a hierarchical feature representation similar to that of a CNN branch.

[0010] The existing literature, "A Monocular Remote Sensing Image Elevation Estimation Method Based on SwinTransformer-CNN and Its Application in Highway Construction Scenarios," studies this problem and proposes a monocular remote sensing image elevation estimation method based on SwinTransformer and a convolutional neural network (CNN) to address the challenge of large elevation distribution differences in different parts of large-scale remote sensing images and the difficulty of model training. On one hand, SwinTransformer, utilizing a sliding window and hierarchical design, combines the ability of CNNs to process large-size images and extract multi-scale features with the global information interaction capabilities of Transformer. On the other hand, to address the training instability caused by large elevation distribution differences in different parts of large-scale remote sensing images, the proposed method adaptively divides the elevation values ​​for each input image, transforming the elevation estimation problem into a classification-regression problem. Finally, the elevation value of each pixel in the image is obtained from the divided elevation values ​​and their probability distribution.

[0011] Since the spatiotemporal fusion task of remote sensing images differs from image classification tasks, requiring both feature extraction and image reconstruction, the classification-based methods in the aforementioned existing literature and solutions cannot be directly applied to spatiotemporal fusion. The parallel Transformer and CNN structures used in the aforementioned existing solutions for image processing need further optimization. In the aforementioned literature, the SwinTransformer-CNN-based methods use SwinTransformer in the encoder module and CNN in the decoder module for upsampling and other operations; however, the feature extraction and fusion effects also require improvement.

[0012] In summary, existing technologies suffer from several technical problems: they are not applicable to scenarios with large information changes in a short period of time; low image resolution results in limited information provided; large pixel classification errors exist; and the models have low robustness, insufficient extraction of temporal and spatial information and features, and complex fusion methods introduce noise. Summary of the Invention

[0013] The technical problem to be solved by this invention is: how to solve the technical problems in the prior art that are not applicable to scenarios with large information changes in a short period of time, low image resolution leading to limited information, large pixel classification error, low model robustness, insufficient extraction of temporal change information and spatial features, and complex fusion methods that introduce noise.

[0014] This invention solves the above-mentioned technical problems by employing the following technical solution: a remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN, comprising:

[0015] S1. Collect historical remote sensing image product data and requirement information from the area to be fused and predicted, and perform cloud removal processing to obtain coarse-resolution images and fine-resolution images.

[0016] S2. Data preprocessing: Perform resolution alignment and image band alignment operations on coarse-resolution and fine-resolution images to obtain image datasets. Divide the image datasets according to preset division ratios to obtain training sets, validation sets, and test sets.

[0017] S3. Perform dual-branch interaction processing on the Swin Transformer branch and the CNN branch to carry out model building operations to obtain the remote sensing image spatiotemporal fusion model. The model building operations include: network structure construction operations and hyperparameter setting operations. The remote sensing image spatiotemporal fusion model includes: input layer, spatial feature extraction layer, temporal change feature extraction layer, image reconstruction layer and output layer.

[0018] S4. Using the training set, train the remote sensing image spatiotemporal fusion model to obtain a remote sensing image spatiotemporal fusion model with a suitable fit.

[0019] S5. Using a moderately fitted state remote sensing image spatiotemporal fusion model, predictions are made using the remote sensing images from the spatiotemporal fusion model to obtain the predicted MODIS image and predicted Landsat image for the date to be predicted.

[0020] This invention provides a combined model based on the parallel interaction of Swing Transformer and CNN: for low-resolution images, Swing Transformer can effectively extract global features of temporal changes in the data, while CNN can more deeply acquire multi-level local features; to extract deeper features, a bottleneck residual block from the residual network is used in the CNN to make the network sufficiently deep while avoiding problems such as gradient explosion. The trained model is used to predict remote sensing image data, and the prediction model has the advantages of fast prediction speed and high prediction accuracy.

[0021] This invention proposes a spatiotemporal fusion model for remote sensing images based on the parallel interaction of Swing Transformer and CNN. It acquires MODIS and Landsat product data, performs data preprocessing, and constructs a spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN. The model is trained and validated using a dataset to achieve a suitable fit for prediction of high-resolution remote sensing images. Compared to the limitations of single networks in deep learning methods, the model proposed in this chapter improves prediction accuracy. The prediction results of the spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN can be widely applied as a remote sensing data fusion product in various research and daily life fields.

[0022] In a more specific technical solution, S2 includes:

[0023] S21. Use the nearest neighbor interpolation method to upsample the coarse resolution image to align the resolutions of the coarse and fine resolution images.

[0024] S22. For the coarse-resolution image and the fine-resolution image, select no less than two overlapping bands, obtain and arrange them according to the differences in the overlapping band data, and adjust the bands of the coarse-resolution image to align with those of the fine-resolution image.

[0025] In a more specific technical solution, S3 includes:

[0026] S31. Set the input layer, wherein the input information of the input layer includes: coarse resolution image C1 of the reference date, fine resolution image F1 of the reference date, and coarse resolution image C2 of the predicted date;

[0027] S32. Set up a time feature extraction layer, in which preliminary feature extraction of time-varying features is performed. After passing through a CNN layer, the number of channels and feature map size of the time-varying features are adjusted. The coarse features are input into the Swin Transformer branch and the CNN branch respectively. The Swin Transformer branch uses the moving window mechanism to extract global information, and the CNN branch uses the residual network bottleneck structure to supplement local information, thereby adjusting the number of channels and feature map size. Through the interaction of the two branches, the fused time features are obtained.

[0028] S33. Set up a spatial feature extraction layer, and use no less than two convolutional layers and pooling layers to extract spatial features and downsample the input high-resolution remote sensing image to obtain spatial texture detail features.

[0029] S34. Set up an image reconstruction layer to fuse and process temporal variation features and spatial texture detail features to obtain a spatiotemporal fusion model remote sensing image;

[0030] S35. Set the output layer and output high-resolution remote sensing images according to the date to be predicted;

[0031] S36. Perform hyperparameter setting operation;

[0032] S37. Use Dropout regularization to optimize the spatiotemporal fusion model for remote sensing images;

[0033] S38. Evaluate the similarity and image error of high-resolution remote sensing images, and calculate the loss function accordingly.

[0034] Since the features extracted by the SwinT branch and the CNN branch are similar in form, the two branches in this invention can perform feature interaction and fusion at each stage, which can be adapted using a 1*1 convolutional kernel; the main spatial information after fusion comes from high-resolution images, and a feature extraction module is set in the network to ensure the spatial texture information of the predicted image, etc.

[0035] In a more specific technical solution, S31 also includes:

[0036] S311. Based on the coarse-resolution image C2 of the predicted date and the coarse-resolution image C1 of the reference date, the time variation characteristics are obtained;

[0037] S312. Based on the reference date fine-resolution image F1, process to obtain spatial texture detail features.

[0038] In a more specific technical solution, S32 includes:

[0039] S321. Adjust the number of channels and feature map size using the following logic:

[0040]

[0041] In the formula, L0 represents the initial extracted features, which are then input into the SwinT branch and the CNN branch for further processing; W is the weight matrix; and b is the bias. This is a convolution operation;

[0042] In S322 and the Swing Transformer branch, the coarse features are processed using the following logic to obtain the Swing Transformer branch features:

[0043] L1 = Swin(S1 + S) ′ )#(2)

[0044] S2 = Patch Merging(L1)#(3)

[0045] S323. In the CNN branch, the coarse features are processed using the following logic to obtain the CNN branch features:

[0046]

[0047]

[0048] ······#(6)

[0049]

[0050]

[0051] S324. Using the following logic, perform two-branch interactive processing on the Swing Transformer branch and the CNN branch:

[0052]

[0053]

[0054] In the formula, S1 and S2 are the input and output of the SwinT branch, and C1 and C2 are the input h and output of the CNN branch;

[0055] S325. Using the following logic, fuse the Swing Transformer branch features and the CNN branch features to obtain the fused temporal feature F. T :

[0056] F T =S n +BatchNorm(S n )#(11)

[0057] In the formula, S n For the final output of the SwinT branch, C n For the final output of the CNN branch, use BatchNorm to convert S n Align to C n Then, the two features are added together and merged.

[0058] This invention employs a remote sensing image spatiotemporal fusion model that combines the Swing Transformer and CNN in parallel. The dual-branch parallel interactive feature extraction method more effectively extracts global and local information from low-resolution image pairs, and strengthens and fuses features during the interactive process, fully utilizing temporal variation information. The spatial channel attention module enhances important features and improves the fusion effect.

[0059] In a more specific technical solution, S33 utilizes the following logic to perform spatial feature extraction and downsampling to obtain spatial texture detail features F. S :

[0060]

[0061] P s1 =max(C s1 )+b s2 #(13)

[0062] In the formula, C s1 P is the output of the first convolutional layer. s1 This represents the first pooling output, where W is the weight matrix and b is the bias. For convolution operations, max() is the maximum value function.

[0063] In a more specific technical solution, S34 includes:

[0064] S341. Using the following logic, the time-varying feature F T With spatial texture detail features F S Perform a concat connection:

[0065] S342. Using the following logic, feature enhancement is performed using the Spatial Channel Attention Module (CBAM) to obtain enhanced features;

[0066] F c =CBAM(Concat(F T ,F S ))#(14);

[0067] S343. Using the following logic, perform convolution and upsampling operations on the enhanced features to recover the high-resolution remote sensing image:

[0068]

[0069] P n1 =Upsample(C n1 )#(16)

[0070] In the formula, CBAM represents the convolutional channel attention operation.

[0071] This invention enhances the extracted temporal and spatial information through a spatial channel attention module, thereby achieving full feature fusion and enabling image reconstruction and prediction.

[0072] In a more specific technical solution, S38 also includes:

[0073] S381. Using the following logic, calculate the similarity L. ssim :

[0074]

[0075] S382. Calculate the image error using the following logic:

[0076]

[0077] S383, Based on similarity L ssim and image error L pixel Determine the loss function.

[0078] In a more specific technical solution, S383 uses the following logic to determine the loss function:

[0079] L total =L pixel +L ssim #(19)

[0080] In the formula, F2 and The actual image and the predicted image for the date to be predicted are shown below, respectively. ε s and ε p Used to stabilize the training process, where N is the number of pixels. L total It consists of two parts, where λ is the weighting parameter.

[0081] In more specific technical solutions, the remote sensing image spatiotemporal fusion system based on the parallel interaction of Swing Transformer and CNN includes:

[0082] The image acquisition module is used to collect historical remote sensing image product data and requirement information from the area to be fused and predicted, and to perform cloud removal processing to obtain coarse-resolution images and fine-resolution images.

[0083] The image preprocessing module is used to perform data preprocessing, including resolution alignment and image band alignment operations on coarse-resolution and fine-resolution images to obtain an image dataset. The image dataset is divided according to a preset division ratio to obtain a training set, a validation set, and a test set. The image preprocessing module is connected to the image acquisition module.

[0084] The spatiotemporal fusion model construction module is used to perform dual-branch interactive processing on the Swin Transformer branch and the CNN branch to build the model and obtain the spatiotemporal fusion model of remote sensing images. The model construction operation includes network structure construction and hyperparameter setting. The spatiotemporal fusion model of remote sensing images includes an input layer, a spatial feature extraction layer, a temporal change feature extraction layer, an image reconstruction layer, and an output layer. The spatiotemporal fusion model construction module is connected to the image preprocessing module.

[0085] The model fitting training module is used to train the remote sensing image spatiotemporal fusion model using the training set to obtain a remote sensing image spatiotemporal fusion model with a suitable fit. The model fitting training module is connected to the spatiotemporal fusion model construction module.

[0086] The prediction module is used to make predictions using a spatiotemporal fusion model of remote sensing images with appropriate fitting. The prediction module obtains the predicted MODIS image and the predicted Landsat image for the date to be predicted. The prediction module is connected to the model fitting and training module.

[0087] The present invention has the following advantages over the prior art:

[0088] This invention provides a combined model based on the parallel interaction of Swing Transformer and CNN: for low-resolution images, Swing Transformer can effectively extract global features of temporal changes in the data, while CNN can more deeply acquire multi-level local features; to extract deeper features, a bottleneck residual block from the residual network is used in the CNN to make the network sufficiently deep while avoiding problems such as gradient explosion. The trained model is used to predict remote sensing image data, and the prediction model has the advantages of fast prediction speed and high prediction accuracy.

[0089] This invention proposes a spatiotemporal fusion model for remote sensing images based on the parallel interaction of Swing Transformer and CNN. It acquires MODIS and Landsat product data, performs data preprocessing, and constructs a spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN. The model is trained and validated using a dataset to achieve a suitable fit for prediction of high-resolution remote sensing images. Compared to the limitations of single networks in deep learning methods, the model proposed in this chapter improves prediction accuracy. The prediction results of the spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN can be widely applied as a remote sensing data fusion product in various research and daily life fields.

[0090] Since the features extracted by the SwinT branch and the CNN branch are similar in form, the two branches in this invention can perform feature interaction and fusion at each stage, which can be adapted using a 1*1 convolutional kernel; the main spatial information after fusion comes from high-resolution images, and a feature extraction module is set in the network to ensure the spatial texture information of the predicted image, etc.

[0091] This invention employs a remote sensing image spatiotemporal fusion model that combines the Swing Transformer and CNN in parallel. The dual-branch parallel interactive feature extraction method more effectively extracts global and local information from low-resolution image pairs, and strengthens and fuses features during the interactive process, fully utilizing temporal variation information. The spatial channel attention module enhances important features and improves the fusion effect.

[0092] This invention enhances the extracted temporal and spatial information through a spatial channel attention module, thereby achieving full feature fusion and enabling image reconstruction and prediction.

[0093] Compared to existing parallel structures of Transformer and CNN, the SwinTransformer network used in this invention performs better on image tasks. Specifically, SwinTransformer introduces a moving window mechanism, enabling it to learn local information while acquiring global information, giving it an advantage in feature extraction; SwinTransformer also has lower complexity, saving computational resources.

[0094] Existing methods based on SwinTransformer-CNN use SwinTransformer in the encoder module and CNN in the decoder module for upsampling and other operations. This invention, however, uses SwinTransformer and CNN simultaneously for feature extraction, employs an interaction module in the encoder for information communication, and utilizes spatial channel attention for feature enhancement during upsampling. Therefore, the feature extraction and fusion performance of this invention is superior to existing solutions.

[0095] This invention solves the technical problems existing in the prior art, such as its inability to be applied to scenarios with large information changes in a short period of time, low image resolution leading to limited information provided, large pixel classification errors, low model robustness, insufficient extraction of temporal change information and spatial features, and complex fusion methods that introduce noise. Attached Figure Description

[0096] Figure 1 This is a schematic diagram of the basic steps of the remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN in Embodiment 1 of the present invention.

[0097] Figure 2 This is a schematic diagram illustrating the specific steps of data preprocessing in Embodiment 1 of the present invention;

[0098] Figure 3 This is a schematic diagram of the remote sensing image spatiotemporal fusion model architecture of the parallel interaction between Swing Transformer and CNN in Embodiment 1 of the present invention.

[0099] Figure 4 This is a schematic diagram illustrating the specific steps of constructing a spatiotemporal fusion model of remote sensing images in Embodiment 1 of the present invention;

[0100] Figure 5 This is a schematic diagram illustrating the specific steps of training a remote sensing image spatiotemporal fusion model according to Embodiment 1 of the present invention;

[0101] Figure 6 This is a schematic diagram of the module hierarchy of the remote sensing image spatiotemporal fusion system based on the parallel interaction of Swing Transformer and CNN in Embodiment 2 of the present invention;

[0102] Figure 7a This is an actual high-resolution image of Embodiment 3 of the present invention;

[0103] Figure 7b This is a predicted high-resolution image from Embodiment 3 of the present invention. Detailed Implementation

[0104] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0105] Example 1

[0106] like Figure 1 As shown, the remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN provided by the present invention includes:

[0107] S1. Obtain historical remote sensing image product data and requirement information of the area to be fused and predicted from MODIS and Landsat, and perform cloud removal processing.

[0108] S2. Data preprocessing, including adjusting the resolution of the coarse-resolution image provided by MODIS and the fine-resolution image provided by Landsat, aligning the two types of images in terms of bands, and dividing them into training set, validation set and test set according to a certain ratio.

[0109] like Figure 2 As shown, in this embodiment, the data preprocessing step S2 further includes the following specific steps:

[0110] S21, resolution alignment;

[0111] In this embodiment, the resolution of Landsat can be, for example, 30m, and the resolution of MODIS can be, for example, 500m, resulting in different pixel counts in the two images for the same vector region. In this embodiment, nearest neighbor interpolation is used to upsample the MODIS image to adjust the resolution of the two types of images to be consistent.

[0112] S22, Band alignment;

[0113] In this embodiment, the MODIS remote sensing image data and Landsat remote sensing data are not completely identical in band. Specifically, the band ranges of the two types of remote sensing data are not entirely the same, therefore, it is necessary to select the bands for the remote sensing data. Here, six overlapping bands from the two types of data are selected: Red Band (R), Green Band (G), Blue Band (B), Near Infrared Band (NIR), Mid Infrared Band 1 (SWIR1), and Mid Infrared Band 2 (SWIR2). Furthermore, because the order of the overlapping band data is not the same, the MODIS data bands are adjusted to align with the Landsat data.

[0114] S23. Dataset partitioning;

[0115] In this embodiment, the preprocessed dataset is divided in an 8:1:1 ratio, with 80% of the data used as the training set, 10% as the validation set, and 10% as the test set.

[0116] S3. Construct a spatiotemporal fusion model for remote sensing images based on the parallel interaction of Swing Transformer and CNN, including: the construction of the network structure and the setting of relevant hyperparameters;

[0117] like Figure 3 As shown, in this embodiment, the remote sensing image spatiotemporal fusion model includes, but is not limited to: an input layer, a spatial feature extraction layer, a temporal change feature extraction layer, an image reconstruction layer, and an output layer;

[0118] like Figure 4 As shown, in this embodiment, step S3 of constructing the spatiotemporal fusion model of the base remote sensing image further includes the following specific steps:

[0119] S31. Set up the input layer;

[0120] In this embodiment, the input layer includes: a pair of coarse and fine resolution images C1 and F1 for the reference date, and a coarse resolution image C2 for the predicted date. The shape of the aforementioned images is C*H*W, where H and W are the height and width of the image, and C is the number of channels, i.e., the number of bands in the remote sensing image. In the experiment, the number of input channels is fixed at 6. In the remote sensing image spatiotemporal fusion model of this embodiment, the coarse resolution image C2 for the predicted date is subtracted from the coarse resolution image C1 for the reference date as the input of time information, providing time variation features; while the fine resolution image F1 for the reference date is used as the input of spatial information, providing spatial texture detail features for the predicted high-resolution image.

[0121] S32. Set up the time feature extraction layer;

[0122] In this embodiment, preliminary feature extraction is performed on the input time-varying information. After passing through a CNN layer, the number of feature channels and the size of the feature map are adjusted. The process is as follows:

[0123]

[0124] Here, L0 represents the initially extracted features, which are then input into the SwinT branch and the CNN branch for further processing. W is the weight matrix, and b is the bias. This is a convolution operation;

[0125] In this embodiment, coarse features are input into the SwinT branch and the CNN branch respectively. The Swin Transformer continuously extracts global information using its own moving window mechanism, while the CNN branch uses a residual network bottleneck structure to supplement local information. After completing one interaction cycle, the size of the feature map is further reduced while the number of channels is increased; then, it is input into the next interaction cycle, ultimately obtaining fully informative deep features.

[0126] In this embodiment, the processing of the SwinT branch is shown in equations (2) and (3), the processing of the CNN branch is shown in equations (4) to (8), and equations (9) and (10) are the processing of the interaction between the two branches:

[0127] L1 = Swin(S1 + S) ′ )#(2)

[0128] S2 = Patch Merging(L1)#(3)

[0129]

[0130]

[0131] ······#(6)

[0132]

[0133]

[0134]

[0135]

[0136] In this diagram, S1 and S2 are the input and output of the SwinT branch, and C1 and C2 are the input and output of the CNN branch. The feature maps of the SwinT branch are resized and have different channels through Patch Merging, while the feature maps of the CNN branch are resized through their own convolution operations. The outputs of the two branches are aligned. During the interaction, since the two feature sets are similar in shape, information can be exchanged using a 1x1 convolution kernel and BatchNorm / LayerNorm.

[0137] In this embodiment, an interaction process is as shown above. The time feature extraction module consists of multiple interaction stages, and finally, the output features of the two branches are fused to obtain the final time feature F. T The fusion process is shown in equation (11):

[0138] F T =C n +BatchNorm(S n )#(11)

[0139] Among them, S n For the final output of the SwinT branch, C n For the final output of the CNN branch, use BatchNorm to convert S n Align to C n Then, the two features are added together and merged;

[0140] S33. Set up the spatial feature extraction layer;

[0141] In this embodiment, multiple convolutional and pooling layers are used to extract and downsample the spatial features of the input high-resolution remote sensing image to obtain spatial features F. S The process is shown in equations (12) and (13):

[0142]

[0143] P s1 =max(C s1 )+b s2 #(13)

[0144] Among them, C s1 This is the output of the first convolutional layer. P s1This represents the first pooling output; W is the weight matrix, and b is the bias. This is a convolution operation, with `max()` being the maximum value function. After passing through multiple convolutional and pooling layers, the spatial features F are obtained. S ;

[0145] S34. Set the image reconstruction layer;

[0146] In this embodiment, after obtaining the time change information and spatial information, the features of the two parts are fused.

[0147] In this embodiment, the time feature F T Spatial characteristics F S To focus the network on more important features, a concat connection is performed, and the spatial channel attention module CBAM is used for feature enhancement. Convolution and upsampling operations are then performed on the enhanced features to gradually recover the high-resolution remote sensing image. The process is shown in equations (14), (15), and (16):

[0148] F c =CBAM(Concat(F T ,F S ))#(14)

[0149]

[0150] P n1 =Upsample(C n1 )#(16)

[0151] In equation (14) above, CBAM is the convolutional channel attention operation, and equations (15) and (16) above are the convolution and upsampling operations, respectively. After multiple convolutional and upsampling layers, the final output is gradually obtained;

[0152] S35, Set the output layer;

[0153] In this embodiment, the output image is a high-resolution image F2 of the date to be predicted, with the same resolution and band as the input F1;

[0154] S36. After completing the model structure construction, set the hyperparameters related to model training; in this embodiment, the hyperparameters include, but are not limited to: initial learning rate and number of iterations.

[0155] S37. Use Dropout regularization to compensate for the loss of some neurons, avoid overfitting, and improve the generalization ability of the model.

[0156] S38. Select a loss function to evaluate image similarity;

[0157] In this embodiment, the loss function selected is Multi-Scale Structural Similarity (MS-SSIM), which is widely used in current image reconstruction models. This loss function evaluates the similarity between two images at multiple scales, thus exhibiting better robustness. Similarity L ssim The calculation method is shown in the following formula (17):

[0158]

[0159] In this embodiment, for image prediction, the image error can be calculated at the pixel level, and the calculation formula is shown in the following formula (18):

[0160]

[0161] In this embodiment, the loss function is composed of these two parts, as shown in equation (19):

[0162] L total =L pixel +L ssim #(19)

[0163] Among them, F2 and The actual image and the predicted image for the date to be predicted are shown below, respectively. ε s and ε p Used to stabilize the training process, where N is the number of pixels. L totak It consists of two parts, where λ is the weighting parameter;

[0164] S4. Use the training set obtained in step S2 above to train the remote sensing image spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN constructed in step 3, and obtain a remote sensing image spatiotemporal fusion model with a moderate fit.

[0165] like Figure 5 As shown, in this embodiment, step S4 of training the spatiotemporal fusion model of remote sensing images further includes the following specific steps:

[0166] S41. The training set is used to train a remote sensing image spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN, and the loss is calculated by the loss function, and the calculation formula is shown in the above formulas (15)-(17).

[0167] S42. Determine whether the loss of the current model has reached the expected loss;

[0168] S43. If so, stop training, save the model, and perform the following step S48 to avoid model overfitting.

[0169] S44. If not, it means the current model is underfitting; continue training.

[0170] S45. Determine whether the model training rounds have reached the preset number of iterations;

[0171] S46. If not, perform backpropagation and gradient descent on the model, update the parameters of each layer of the model, and repeat step S41.

[0172] S47. If yes, stop training and save the model;

[0173] S48. Use the partitioned validation set to validate the already trained remote sensing image spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN.

[0174] S49. Determine whether the current model has reached a fit on the validation set;

[0175] S410. If the model performs significantly differently in the training and testing sets, it indicates that the model has reached an overfitting state. The parameters of the neural network model need to be adjusted, and the aforementioned steps S41 to S49 are repeated.

[0176] S411. If the model performs similarly in the training and test sets, it indicates that the model has reached a state of adequate fit.

[0177] S412. Use the divided test set to verify the reliability of the remote sensing image spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN that has been trained in step S49, and save the model.

[0178] S5. Predict the Landsat image using the obtained spatiotemporal fusion model remote sensing image. Use a pair of MODIS and Landsat images of a date close to the prediction date, and a MODIS image of the prediction date, to predict the Landsat image.

[0179] Example 2

[0180] like Figure 6 As shown, in this embodiment, a remote sensing image spatiotemporal fusion system based on parallel interaction of Swin Transformer and CNN is applied to the generation of high-resolution remote sensing images. This system includes the following basic modules:

[0181] User interaction module 1 includes: data acquisition module 11, hyperparameter setting module 12 and data visualization module 13;

[0182] Model module 2 includes: data preprocessing module 21, model training module 22, and spatiotemporal fusion remote sensing image prediction module 23.

[0183] In this embodiment, the data acquisition module 11 is used to acquire historical remote sensing image data; the hyperparameter setting module 12 is used to set the parameters in the remote sensing image spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN; the data visualization module 13 is used to display the data in each step of the operation for easy observation; the data preprocessing module 21 is used to align the acquired dataset in terms of resolution and bands; the model training module 22 is used to train the model; and the spatiotemporal fusion remote sensing image prediction module 23 is used to predict remote sensing images and write the predicted remote sensing image data into the database.

[0184] Example 3

[0185] In this embodiment, public datasets from two locations are used: the Colambatally Irrigated Area (CIA) and the Gwydir Downstream Basin (LGC), which represent phenological and land cover changes, respectively. The CIA dataset covers southern New South Wales, a region dominated by rice and equipped with modern irrigation systems. It contains 17 pairs of cloud-free MODIS Landsat data from 2001 to 2002, each pair measuring 6×2040×1720 pixels. During the dataset collection period, the CIA region experienced virtually no land cover change, only temporal differences, which can be used to test the performance of spatiotemporal fusion methods in phenological change prediction. The LGC study area is located in northern New South Wales and contains 14 pairs of cloud-free data (6×2720×3200 pixels) from 2004 to 2005. A flood occurred in the region in mid-December 2004, causing significant land cover change, making the LGC dataset ideal for testing the performance of spatiotemporal fusion algorithms in land cover change prediction. The dataset is divided into training, validation, and test sets according to a ratio of 80%, 10%, and 10%, respectively.

[0186] In this embodiment, the remote sensing image spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN provided in Example 1 is trained using a training set, and the model's fit is tested using a test set, ultimately bringing the model to a state of adequate fit. Then, the trained spatiotemporal fusion model is used to test the test set.

[0187] like Figure 7a and Figure 7b As shown in this embodiment, to illustrate the effectiveness of the remote sensing image spatiotemporal fusion method based on parallel interaction of SwinTransformer and CNN provided by the present invention, several commonly used image evaluation metrics in the field of spatiotemporal fusion are used to evaluate the spatiotemporal fusion results: Root Mean Square Error (RMSE), Structural Similarity (SSIM), Correlation Coefficient (CC), and Spectral Angle Mapping (SAM). See also Figure 7a , Figure 7b Table 1 shows the fusion results and evaluation indicators, in which all indicators achieved good results. Compared to real images, Figure 7a , Figure 7b The predicted images exhibit high homogeneity. The predicted images possess rich spectral information, demonstrate good fusion of detailed textures, and exhibit relatively low noise.

[0188] In this embodiment, the reliability of the spatiotemporal fusion method provided by the present invention is verified by experimental results from both subjective visual and objective indicators.

[0189] Table 1 Evaluation parameters for predicted images

[0190]

[0191]

[0192] In summary, this invention provides a combined model based on the parallel interaction of Swing Transformer and CNN: for low-resolution images, Swing Transformer can effectively extract global features of temporal changes in the data, while CNN can more deeply acquire multi-level local features; to extract deeper features, a bottleneck residual block from the residual network is used in the CNN to make the network sufficiently deep while avoiding problems such as gradient explosion. The trained model is then used to predict remote sensing image data, and the prediction model has the advantages of fast prediction speed and high prediction accuracy.

[0193] This invention proposes a spatiotemporal fusion model for remote sensing images based on the parallel interaction of Swing Transformer and CNN. It acquires MODIS and Landsat product data, performs data preprocessing, and constructs a spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN. The model is trained and validated using a dataset to achieve a suitable fit for prediction of high-resolution remote sensing images. Compared to the limitations of single networks in deep learning methods, the model proposed in this chapter improves prediction accuracy. The prediction results of the spatiotemporal fusion model based on the parallel interaction of Swing Transformer and CNN can be widely applied as a remote sensing data fusion product in various research and daily life fields.

[0194] Since the features extracted by the SwinT branch and the CNN branch are similar in form, the two branches in this invention can perform feature interaction and fusion at each stage, which can be adapted using a 1*1 convolutional kernel; the main spatial information after fusion comes from high-resolution images, and a feature extraction module is set in the network to ensure the spatial texture information of the predicted image, etc.

[0195] This invention employs a remote sensing image spatiotemporal fusion model that combines the Swing Transformer and CNN in parallel. The dual-branch parallel interactive feature extraction method more effectively extracts global and local information from low-resolution image pairs, and strengthens and fuses features during the interactive process, fully utilizing temporal variation information. The spatial channel attention module enhances important features and improves the fusion effect.

[0196] This invention enhances the extracted temporal and spatial information through a spatial channel attention module, thereby achieving full feature fusion and enabling image reconstruction and prediction.

[0197] This invention solves the technical problems existing in the prior art, such as its inability to be applied to scenarios with large information changes in a short period of time, low image resolution leading to limited information provided, large pixel classification errors, low model robustness, insufficient extraction of temporal change information and spatial features, and complex fusion methods that introduce noise.

[0198] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spatiotemporal fusion method for remote sensing images based on parallel interaction of Swing Transformer and CNN, characterized in that, The method includes: S1. Collect historical remote sensing image product data and requirement information from the area to be fused and predicted, and perform cloud removal processing to obtain coarse-resolution images and fine-resolution images. S2. Data preprocessing: Perform resolution alignment and image band alignment operations on the coarse resolution image and the fine resolution image to obtain an image dataset. Divide the image dataset according to a preset division ratio to obtain a training set, a validation set, and a test set. S3. Perform dual-branch interaction processing on the Swin Transformer branch and the CNN branch to carry out model building operations to obtain a remote sensing image spatiotemporal fusion model. The model building operations include: network structure construction operations and hyperparameter setting operations. The remote sensing image spatiotemporal fusion model includes: an input layer, a spatial feature extraction layer, a temporal change feature extraction layer, an image reconstruction layer, and an output layer. S3 includes: S31. Set an input layer, wherein the input information of the input layer includes: a coarse resolution image C1 of the reference date, a fine resolution image F1 of the reference date, and a coarse resolution image C2 of the predicted date; S32. A time feature extraction layer is set up, wherein preliminary feature extraction is performed on the time-varying features. After passing through a CNN layer, the number of channels and the feature map size of the time-varying features are adjusted. The coarse features are input into the SwinTransformer branch and the CNN branch respectively. The SwinTransformer branch uses a moving window mechanism to extract global information. The CNN branch uses a residual network bottleneck structure to supplement local information, thereby adjusting the number of channels and the feature map size. Through dual-branch interactive processing, the fused time features are obtained. S33. Set up a spatial feature extraction layer, and use no less than two convolutional layers and pooling layers to extract spatial features and downsample the input high-resolution remote sensing image to obtain the spatial texture detail features; S34. Set up an image reconstruction layer to fuse the temporal variation features and the spatial texture detail features to obtain a spatiotemporal fusion model remote sensing image; S35. Set the output layer and output the high-resolution remote sensing image according to the date to be predicted; S36. Perform the hyperparameter setting operation; S37. Use Dropout regularization to optimize the spatiotemporal fusion model of the remote sensing image; S38. Evaluate the similarity and image error of the high-resolution remote sensing images, and calculate the loss function accordingly; S4. Using the training set, the remote sensing image spatiotemporal fusion model is trained to obtain a remote sensing image spatiotemporal fusion model with a suitable fit. S5. Using the aforementioned appropriately fitted state remote sensing image spatiotemporal fusion model, prediction is made through the remote sensing image of the spatiotemporal fusion model to obtain the predicted MODIS image and predicted Landsat image for the date to be predicted.

2. The remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN according to claim 1, wherein step S2 includes: S21. Upsample the coarse resolution image using the nearest neighbor interpolation method to align the resolutions of the coarse resolution image and the fine resolution image. S22. For the coarse resolution image and the fine resolution image, select no less than two overlapping bands, obtain and arrange them according to the difference in the overlapping band data, and adjust the bands of the coarse resolution image to align with the fine resolution image.

3. The remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN according to claim 1, step S31 further includes: S311. Based on the predicted date coarse resolution image C2 and the reference date coarse resolution image C1, time change features are obtained through processing. S312. Based on the reference date fine-resolution image F1, spatial texture detail features are obtained through processing.

4. The remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN as described in claim 1, characterized in that, S32 includes: S321. Adjust the number of channels and the feature map size using the following logic: In the formula, This involves initial feature extraction, followed by input into the SwinT branch and the CNN branch for further processing. This is the weight matrix. For bias, This is a convolution operation; S322. In the Swing Transformer branch, the coarse features are processed using the following logic to obtain the Swing Transformer branch features: S323. In the CNN branch, the coarse features are processed using the following logic to obtain the CNN branch features: S324. Using the following logic, perform the dual-branch interaction processing on the Swing Transformer branch and the CNN branch: In the formula, and For the input and output of the SwinT branch, and Let h be the input and output of the CNN branch; S325. Using the following logic, fuse the Swin Transformer branch features and the CNN branch features to obtain the fused temporal features. : In the formula, This is the final output of the SwinT branch. For the final output of the CNN branch, use BatchNorm to... Align to Then, the two features are added together and merged.

5. In the remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN as described in claim 1, in step S33, spatial feature extraction and downsampling are performed using the following logic to obtain spatial texture detail features. : In the formula, This is the output of the first convolutional layer. This indicates the first pooled output. The weight matrix, For deviation, For convolution operations, It is a function for maximizing the value.

6. The remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN as described in claim 1, characterized in that, S34 includes: S341. Using the following logic, the time change characteristics are... With the spatial texture detail features Perform a concat connection: S342. Using the following logic, feature enhancement is performed using the Spatial Channel Attention Module (CBAM) to obtain enhanced features; ； S343. Using the following logic, perform convolution and upsampling operations on the enhanced features to recover the high-resolution remote sensing image: In the formula, CBAM represents the convolutional channel attention operation.

7. The remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN according to claim 1, step S38 further includes: S381. Calculate the similarity using the following logic. : S382. Calculate the image error using the following logic: S383, Based on the aforementioned similarity and the image error Determine the loss function.

8. The remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN as described in claim 7, characterized in that, In step S383, the loss function is determined using the following logic: In the formula, and These are the actual image and the predicted image for the date to be predicted, respectively. and Used to stabilize the training process. This represents the number of pixels. It consists of two parts. These are the weight parameters.

9. A remote sensing image spatiotemporal fusion system based on parallel interaction of Swing Transformer and CNN, used to execute the remote sensing image spatiotemporal fusion method based on parallel interaction of Swing Transformer and CNN as described in any one of claims 1 to 8, characterized in that, The system includes: The image acquisition module is used to collect historical remote sensing image product data and requirement information from the area to be fused and predicted, and to perform cloud removal processing to obtain coarse-resolution images and fine-resolution images. The image preprocessing module is used to perform data preprocessing, performing resolution alignment and image band alignment operations on the coarse resolution image and the fine resolution image to obtain an image dataset. The image dataset is divided according to a preset division ratio to obtain a training set, a validation set, and a test set. The image preprocessing module is connected to the image acquisition module. The spatiotemporal fusion model construction module is used to perform the dual-branch interactive processing on the Swin Transformer branch and the CNN branch to perform model construction operations and obtain a spatiotemporal fusion model of remote sensing images. The model construction operations include: network structure construction operations and hyperparameter setting operations. The spatiotemporal fusion model of remote sensing images includes: an input layer, a spatial feature extraction layer, a temporal change feature extraction layer, an image reconstruction layer, and an output layer. The spatiotemporal fusion model construction module is connected to the image preprocessing module. The model fitting training module is used to train the remote sensing image spatiotemporal fusion model using the training set to obtain a moderately fitted remote sensing image spatiotemporal fusion model. The model fitting training module is connected to the spatiotemporal fusion model construction module. The prediction module is used to make predictions using the spatiotemporal fusion model of the moderately fitted remote sensing image, thereby obtaining the predicted MODIS image and the predicted Landsat image for the date to be predicted. The prediction module is connected to the model fitting training module.

Images