Remote sensing image spatio-temporal fusion method and device based on wavelet transform convolution and multi-scale feature fusion

Abstract

The application provides a remote sensing image space-time fusion method and device based on wavelet transform convolution and multi-scale feature fusion. The method comprises the following steps: acquiring high and low resolution remote sensing images at a reference time and a prediction time; constructing a feature extraction network based on wavelet transform convolution, performing hierarchical feature extraction through a double-branch input structure, and outputting three groups of features of a shallow layer, a middle layer and a deep layer; performing multi-scale pooling processing on the middle layer features to capture global context information; applying multi-dilation rate hole convolution to the deep layer features in parallel to expand the receptive field; performing adaptive weighted fusion on the three enhanced features through a channel attention mechanism; and finally, reconstructing the high resolution image at the prediction time through feature refinement and band mapping. The application realizes large receptive field feature extraction with a small number of parameter increments, effectively retains high frequency texture details, improves the spatial accuracy of the space-time fused image, and can be used in the fields of agricultural monitoring, vegetation dynamic analysis and the like.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of remote sensing technology and computer vision technology, and in particular to a method and apparatus for spatiotemporal fusion of remote sensing images based on wavelet transform convolution and multi-scale feature fusion. Background Technology

[0002] Spatiotemporal fusion technology for remote sensing images aims to establish a mapping relationship between high temporal resolution, low spatial resolution images and low temporal resolution, high spatial resolution images, generating fused images that combine both high temporal and spatial resolution to meet the needs of fields such as agricultural monitoring and vegetation dynamics analysis for achieving a balance between temporal and spatial resolution in remote sensing data. Existing spatiotemporal fusion methods are mainly divided into three categories:

[0003] The first category is based on weighting functions, represented by STARFM and ESTARFM, which achieve pixel-level fusion by constructing a weighting function of spatial distance and spectral similarity. The second type is based on sparse representation, which fuses images by learning dictionary mapping relationships between high- and low-resolution images. The third category is based on deep learning methods, which utilize convolutional neural networks to learn end-to-end spatiotemporal fusion mapping.

[0004] Among these methods, deep learning-based approaches have achieved good results in terms of fusion accuracy. However, their feature extraction networks generally employ standard two-dimensional convolution operations, which are limited by the limited receptive field of standard convolutions and the difficulty in distinguishing multi-frequency information in images. This results in insufficient accuracy of the fusion results in areas such as land cover edges and texture details. Furthermore, existing methods typically employ only simple feature stitching operations for multi-scale feature fusion, lacking effective feature selection and enhancement mechanisms, thus limiting fusion accuracy in areas with complex land cover types.

[0005] Therefore, there is an urgent need for a method and apparatus for spatiotemporal fusion of remote sensing images based on wavelet transform convolution and multi-scale feature fusion to solve the problems existing in the current technology. Summary of the Invention

[0006] This invention provides a method and apparatus for spatiotemporal fusion of remote sensing images based on wavelet transform convolution and multi-scale feature fusion. It addresses the problems in existing spatiotemporal fusion methods, such as the difficulty of standard convolution in fully capturing multi-frequency information of remote sensing images and the limited receptive field, as well as the simple multi-scale feature fusion method and the lack of an effective channel feature selection mechanism, which leads to insufficient spatial accuracy of the fused image in terms of ground object edges, texture details and complex areas.

[0007] The core technology of this invention is to construct a dual-branch feature extraction network based on wavelet transform convolution to output three sets of hierarchical features. The middle-level features are processed by multi-scale pooling, and the deep-level features are processed by multi-dilation rate dilated convolution. Then, the three enhanced features are adaptively weighted and fused through a channel attention mechanism to reconstruct the predicted image.

[0008] In a first aspect, the present invention provides a spatiotemporal fusion method for remote sensing images based on wavelet transform convolution and multi-scale feature fusion, the method comprising the following steps: Acquire high-resolution images of the same region at the reference time, low-resolution images at the reference time, and low-resolution images at the predicted time. A feature extraction network based on wavelet transform convolution is constructed. The high-resolution image at the reference time is used as the input of the main branch, and the channel stitching result of the low-resolution image at the reference time and the low-resolution image at the prediction time is used as the input of the auxiliary branch. Hierarchical feature extraction is performed through wavelet transform convolution layer, and the output is shallow features containing spatial detail information, medium features containing structural semantic information, and deep features containing global semantic information. Multi-scale pooling is performed on mid-level features to capture global spatial context information at different spatial scales and generate pooled enhanced features. Dilated convolutions with different dilation rates are applied in parallel to deep features to expand the receptive field without changing the spatial resolution of the feature map, thereby generating dilated and enhanced features. Shallow features, pooling enhancement features, and dilation enhancement features are fused along the channel dimension, and the fused features are adaptively weighted using a channel attention mechanism to obtain attention-enhanced features. The attention-enhanced features are refined and mapped using a wavelet transform convolution module to reconstruct a high-resolution image at the predicted time.

[0009] Furthermore, the wavelet transform convolutional layer performs discrete wavelet transform decomposition on the input feature map to obtain low-frequency sub-bands and multiple high-frequency sub-bands. Depth-separable convolution operations are performed on each sub-band, and features are reconstructed through inverse wavelet transform. The features are then superimposed with the features of the basic convolutional branch for output.

[0010] Furthermore, the feature extraction network integrates spatial features of the high-resolution images input from the main branch through a wavelet transform convolution module, and extracts temporal difference features of the stitched low-resolution images input from the auxiliary branch through a wavelet transform convolution module. After the two features are concatenated in the channel dimension, they are processed by a combination of convolution module and wavelet transform convolution module, and with residual connection, to output shallow features, middle features and deep features in sequence. The mid-level features are obtained by downsampling the shallow features, and the deep features are obtained by downsampling the mid-level features.

[0011] Furthermore, multi-scale pooling processing includes: The mid-level features are input into multiple parallel branches. Each branch downsamples the feature map to different scales through an adaptive average pooling layer. After the channel dimension is widened by a convolutional layer, it is upsampled to restore the original spatial size. The output features of each branch are concatenated in the channel dimension, then dimensionality is reduced by a convolutional layer, and then upsampled to the same spatial size as the shallow features to obtain pooling-enhanced features.

[0012] Furthermore, the number of parallel branches is three, and the adaptive average pooling downsampling factors of the three branches are 2x, 4x and 8x, respectively; Each branch widens the channel dimension through 1×1 convolution, and after batch normalization and activation function processing, it is restored to the original spatial size through bilinear interpolation upsampling.

[0013] Furthermore, the dilated convolutions with different dilation rates include at least three parallel dilated convolution branches, each branch using dilated convolution layers with different dilation rates and maintaining the feature map size unchanged through edge padding, and the dilation rates of the at least three parallel dilated convolution branches are 2, 4 and 8, respectively. The outputs of each branch are concatenated along the channel dimension and then dimensionality reduced by a convolutional layer. They are then upsampled to restore the same spatial size as the shallow features, resulting in dilated and enhanced features.

[0014] Furthermore, channel attention mechanisms include: Global average pooling is performed on the features fused from the channel dimensions to obtain the global description vector; The global description vector is passed sequentially through a dimensionality reduction convolutional layer, a non-linear activation function, and an dimensionality increase convolutional layer to establish the dependencies between channels; The channel attention weight vector is obtained by normalization using the Sigmoid function. The weight vector is then multiplied with the fused features channel by channel to achieve adaptive enhancement of effective feature channels and adaptive suppression of redundant feature channels.

[0015] Furthermore, the method also includes a model training step: The feature extraction network, multi-scale pooling, multi-scale dilated convolution, and channel attention mechanism are jointly trained end-to-end using a combination of mean squared error loss function and structural similarity loss function as the total loss function.

[0016] Secondly, the present invention provides a remote sensing image spatiotemporal fusion device based on wavelet transform convolution and multi-scale feature fusion, comprising: The image acquisition module is used to acquire high-resolution images of the same area at the reference time, low-resolution images at the reference time, and low-resolution images at the prediction time. The feature extraction module is used to construct a feature extraction network based on wavelet transform convolution. It takes the high-resolution image at the reference time as the main branch input and the channel stitching result of the low-resolution image at the reference time and the low-resolution image at the prediction time as the auxiliary branch input. It performs hierarchical feature extraction through wavelet transform convolution layers and outputs shallow features containing spatial detail information, medium features containing structural semantic information, and deep features containing global semantic information. The multi-scale pooling enhancement module is used to perform multi-scale pooling processing on mid-layer features to generate pooled enhanced features. The multi-scale dilated convolution enhancement module is used to apply dilated convolutions with different dilation rates to deep features in parallel to generate dilated enhanced features; The channel attention fusion module is used to fuse shallow features, pooling enhancement features, and dilation enhancement features in the channel dimension, and to perform adaptive weighting using the channel attention mechanism to obtain attention-enhanced features; The image reconstruction module is used to refine the attention-enhanced features and perform band mapping through the wavelet transform convolution module to reconstruct a high-resolution image at the predicted time.

[0017] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to execute the above-described method for spatiotemporal fusion of remote sensing images based on wavelet transform convolution and multi-scale feature fusion.

[0018] The main contributions and innovations of this invention are as follows: 1. This invention replaces standard convolution with wavelet transform convolution. Through discrete wavelet transform, the input features are decomposed into low-frequency and high-frequency sub-bands, and features are learned separately for each. This allows the network to simultaneously capture the low-frequency spatial structure and high-frequency texture details of remote sensing images throughout the feature extraction process, effectively improving the reconstruction accuracy of fused images in areas of ground object edges and texture details. Simultaneously, wavelet transform convolution trades a linearly increasing number of parameters for an exponentially increasing effective receptive field, achieving a larger sensing range and improving spatial coherence with a lower parameter count than standard convolution.

[0019] 2. This invention designs a dual-branch input structure and a hierarchical feature extraction backbone network. It extracts and fuses the spatial detail information of the high-resolution reference image and the temporal change information of the low-resolution image pair, respectively, and outputs three sets of hierarchical features from shallow to deep and from weak to strong semantics, providing a sufficient and non-redundant feature foundation for subsequent multi-scale enhancement.

[0020] 3. This invention uses a multi-scale pooling feature extraction module and a multi-scale dilated convolution feature extraction module to asymmetrically enhance mid-level and deep features respectively. The former integrates local texture, medium structure and global context information through multi-scale adaptive average pooling, while the latter expands the receptive field of deep features without reducing resolution through multi-diffraction dilated convolution. Together, they make up for the shortcomings of standard convolution in capturing multi-scale information.

[0021] 4. This invention uses a channel attention mechanism to adaptively weight and fuse shallow features, pooling enhancement features, and dilation enhancement features. It automatically enhances feature channels that contribute significantly to the prediction task and suppresses redundant channels, solving the problem of interference with effective features caused by simple stitching methods and significantly improving the spatial accuracy of the fused image.

[0022] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a remote sensing image spatiotemporal fusion method based on wavelet transform convolution and multi-scale feature fusion according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a network structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a feature extraction network module according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a multi-scale pooling feature extraction network module according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a multi-scale dilated convolutional feature extraction network module according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the attention mechanism module according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a wavelet transform convolution module, a normal convolution module, a dilated convolution module, a depth wavelet transform convolution module, and an output module according to embodiments of the present invention. Figure 8 This is a schematic diagram comparing the spatiotemporal fusion results according to an embodiment of the present invention with the prior art; Figure 9 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0025] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0026] Example 1 like Figures 1-7 As shown in the figure, this embodiment provides a remote sensing image spatiotemporal fusion method based on wavelet transform convolution and multi-scale feature fusion. This method constructs an end-to-end deep learning network to fuse high spatial resolution low temporal resolution images at the reference time with low spatial resolution high temporal resolution images at both the reference time and the prediction time to generate a high spatial resolution remote sensing image at the prediction time.

[0027] This embodiment uses a typical agricultural area in Jiangsu Province, China as the experimental area, selecting Landsat-8 (i.e., 2015-2021) as the experimental zone. Figure 1 Landsat data) and GF1 (i.e. Figure 1 The GF1 imagery (GF1 data) was used as the experimental dataset. After preprocessing including radiometric calibration, atmospheric correction (FLAASH method), and geometric fine correction, the spatial resolution of the GF1 imagery was 2m; the spatial resolution of the Landsat data was 30m after the same preprocessing. Using the high-resolution GF1 imagery as a reference, geometric registration and spatial resampling were performed on the Landsat imagery to ensure strict alignment of the two in a unified coordinate system.

[0028] In this embodiment, the reference time is denoted as The predicted time is denoted as Acquire the registered remote sensing image data. During the model training phase, the required data includes: low spatial resolution imagery at the reference time. High spatial resolution images at reference time Low spatial resolution images of the predicted time And high spatial resolution images of the predicted time as training supervision labels, and During the model inference phase, only input is required. , and The prediction results can then be output.

[0029] The spatiotemporal fusion method in this embodiment includes the following steps: Step 1: Construct a training sample library for the spatiotemporal fusion model Step 1.1, Data Download and Screening. Download time-series satellite data of the target area from a publicly available remote sensing data management platform, prioritizing map sheets with cloud coverage below 10% to ensure data validity. Within the same study area, construct a ternary input combination of "reference time pair + low-resolution predicted time" based on the principle of temporal adjacency. The corresponding high-resolution image of the true predicted time. As a supervisory label during network training.

[0030] Step 1.2, Data Processing. Radiometric calibration, atmospheric correction, orthorectification, and geometric fine-tuning are performed on the multi-source remote sensing images. Using the high-resolution image as the spatial reference, spatial resampling and geometric fine-tuning are performed on the low-resolution image to ensure strict sub-pixel spatial alignment of the multi-source images within a unified geographic coordinate system.

[0031] Step 1.3, Sample cropping and screening. The registered full image is cropped and screened using a size of [size missing]. Block cropping is performed using a sliding window with a pixel size and a sliding step size of 32 pixels. The quality assessment mask built into the remote sensing data is used to automatically check whether each data block contains cloud pixels, and data blocks with clouds or cloud shadow coverage exceeding 5% are removed.

[0032] Step 1.4, Data Augmentation and Sample Base Partitioning. Data augmentation operations such as random horizontal and vertical flipping are performed on the selected valid data blocks to improve the model's geometric invariance to spatial orientation and its generalization ability. The final constructed valid temporal combination sample pairs are divided into training, validation, and test sets in an 8:1:1 ratio. In this embodiment, 15 pairs of Landsat-GF imagery with cloud cover below 5% are selected, resulting in a training set of 38,400 pairs, a validation set of 9,600 pairs, and a test set of 16,000 pairs.

[0033] Step 2: Construct a feature extraction network based on wavelet transform convolution like Figure 3As shown, the feature extraction network designed in this step is the core feature extraction backbone of the entire spatiotemporal fusion network (model). Its design goal is to fully explore the spatial detail information in the high-resolution reference image, while effectively fusing the temporal change information carried by the low-resolution image, and output three sets of hierarchical feature representations from shallow to deep and from weak to strong semantics.

[0034] Step 2.1: Design wavelet transform convolutional layers Wavelet transform convolutional layers decompose the input feature map into low-frequency and high-frequency sub-bands by performing multi-level discrete wavelet transform decomposition. Depth-separable convolution operations are then performed in the sub-band spaces at different scales. Finally, the features are reconstructed through inverse wavelet transform and superimposed with the basic convolution results for output.

[0035] Given input features (shape is) ,in For batch size, For the number of channels, and (These represent the height and width of the feature map, respectively). The feature map is decomposed into low-frequency and high-frequency components through successive wavelet decomposition, with the spatial dimensions halved at each level. First, the feature map is decomposed into low-frequency subbands using discrete wavelet transform. and three high-frequency sub-bands: horizontal, vertical, and diagonal. , , :

[0036] in The discrete wavelet transform operation decomposes the input feature map into a low-frequency subband. and three high-frequency subbands , , .

[0037] Subsequently, in each frequency sub-band space, depthwise separable convolutional kernels, which are independently learned from each other's parameters (the number of groups is set to the number of channels multiplied by 4), are used. Spatial filtering is performed in parallel on the low-frequency subband and each high-frequency subband to extract the high- and low-frequency response features of edge texture and structural contours:

[0038] in For depthwise separable convolution operations, each operation is performed independently on its respective frequency subband.

[0039] Furthermore, starting from the deepest frequency component, the features of different frequency bands are reconstructed step by step through inverse discrete wavelet transform, and the spatial resolution of the original feature map is reconstructed by inverse mapping:

[0040] in The inverse wavelet transform operation reconstructs the features of each sub-band into a spatial domain feature map. For input features; The output features are then calculated. Finally, the reconstructed result of the wavelet branch is added element-wise to the base convolutional branch, which also has learnable scaling parameters, and the result is output.

[0041] This embodiment uses Haar wavelets, with downsampling by a factor of 2 at each level. Level ( The equivalent receptive field (for kernel size) of the convolution mapping back to the original resolution is approximately [value missing]. , The effective receptive field of the cascaded modules can reach approximately With series Exponential growth. The number of parameters in a single-layer wavelet transform convolution is the sum of the parameters of the base branch and each level of sub-band convolution, i.e. With series Linear growth. The actual configuration in this embodiment is as follows: , The number of parameters for a single-layer wavelet transform convolution is . and standards Fully convolutional (number of parameters) Compared to the number of channels, When the parameters are 24, 64, and 128, the number of parameters in the wavelet transform convolution is 1224, 3264, and 6528, respectively, while the standard... The number of parameters for convolution are 5208, 36928, and 147584, respectively. Wavelet transform convolution has a significant advantage in terms of parameter efficiency.

[0042] Step 2.2: Construct a feature extraction network based on wavelet transform convolution like Figure 3 As shown, the feature extraction network (i.e. Figure 2 The feature extraction network module adopts a dual-branch input structure. The main branch input is... High-resolution images of the moment (dimension is) ,in For batch size, This represents the number of spectral bands. and (where the feature map has height and width), the auxiliary branch input is , Low-resolution images at all times and The splicing result along the channel dimension (dimension is) (This refers to low-resolution stitched images.)

[0043] Step 2.2.1, Extract shallow features The high-resolution image input to the main branch is sequentially processed through two layers of wavelet transform convolution (WTC) modules (each module contains one convolution kernel). A wavelet transform convolutional layer with a stride of 1, a batch normalization layer, and a ReLU activation layer are used to complete the initial spatial feature integration. For the stitched low-resolution image input from the auxiliary branch, a wavelet transform convolutional module is used for feature integration to extract temporal difference features. The two shallow feature paths are concatenated along the channel dimension to achieve the initial fusion of shallow information from the high- and low-resolution images. After stitching the two feature paths, the number of channels is merged. The dimension is obtained as The initial features are then fused together. These initial features are sequentially input into one ordinary convolutional module (containing one set of ordinary convolutional layers, a batch normalization layer, and ReLU activation), one deep concatenated wavelet transform convolutional module (this module is composed of a first ordinary convolutional submodule, a wavelet transform convolutional module, and a second ordinary convolutional submodule connected in series; the first ordinary convolutional submodule contains one set of ordinary convolutional layers, a batch normalization layer, and a ReLU activation layer; the wavelet transform convolutional module contains one wavelet transform convolutional layer, a batch normalization layer, and a ReLU activation layer; the second ordinary convolutional submodule contains one set of ordinary convolutional layers, a batch normalization layer, and a ReLU activation layer), and one ordinary convolutional module. Finally, these inputs are concatenated with the initial features via residual connections to obtain the shallow features. The space dimensions are .

[0044] Step 2.2.2, Extracting mid-layer features The shallow features are input into one dilated convolutional module (containing one dilated convolutional layer with a stride of 2, one batch normalization layer, and one ReLU activation layer), one regular convolutional module, one wavelet transform convolutional module, and one regular convolutional module. These are then added to the features before fusion using residual addition to obtain the mid-level semantic features. The space dimensions are .

[0045] Step 2.2.3, Extract deep features For the mid-level features output in step 2.2.2, after inputting one dilated convolutional module (dilation rate of 2, stride of 2), one ordinary convolutional module, one wavelet transform convolutional module, and one ordinary convolutional module, they are added to the features before fusion using residual addition to obtain the deep semantic features. The space dimensions are .

[0046] The spatial dimensions of the three sets of output features are respectively , , These correspond to the shallow spatial detail information, mid-level structural semantic information, and deep global semantic information of the network, respectively. Shallow features retain high-frequency textures and object edges, offering the highest resolution, and are not downsampled further to avoid detail loss. Mid-level features represent medium-scale structures and regional semantics, and are fed into subsequent multi-scale pooling modules to complete global context perception. Deep features, after two downsampling operations, have a large receptive field and strong global semantics but weak spatial detail; they are fed into subsequent multi-scale dilated convolution modules to expand the receptive field and restore multi-scale context without reducing resolution.

[0047] Step 3: Construct a multi-scale pooling feature extraction network like Figure 4 As shown, the multi-scale pooling feature extraction network module mainly targets the semantic features of the mid-level structure. By constructing a spatial pyramid pooling structure, contextual information within different receptive fields at multiple scales is captured, thus compensating for the shortcomings of mid-layer features in global perception.

[0048] Step 3.1: Extract the semantic features of the mid-level results output in Step 2.2.2. The input consists of three parallel processing branches. Each branch sequentially passes the feature map through an adaptive average pooling layer to downsample it by a factor of 2, 4, and 8, and then each branch is further processed by a convolutional kernel. A convolutional layer with a stride of 1 widens the channel dimension to twice its original size, followed by a batch normalization layer and a ReLU activation layer. Finally, bilinear interpolation upsampling restores the original spatial dimensions. Compared to max pooling with a fixed step size, adaptive average pooling can automatically calculate pooling parameters based on the actual size of the input feature map, ensuring the consistency of output size under different input resolutions. At the same time, average pooling can smooth local noise and enhance the spatial stability of features.

[0049] Step 3.2: The output features of the three branches are concatenated along the channel dimension to obtain multi-scale fused features, which are then processed by a convolutional kernel. A convolutional layer with a stride of 1 reduces the feature dimension to the original input size, effectively integrating multi-scale contextual information. Bilinear interpolation upsamples the feature map to the same spatial size as the shallow features. This yields pooling-enhanced features.

[0050] The pooling enhancement features obtained after processing by the multi-scale pooling feature extraction network module effectively integrate multi-level spatial context information, from local texture to medium-sized regional structure to large-scale land cover distribution, while maintaining the original spatial resolution.

[0051] Step 4: Construct a multi-scale dilated convolutional feature extraction network like Figure 5 As shown, the multi-scale dilated convolutional feature extraction network module targets deep features. (space dimensions) The processing involves using a parallel, multi-diffraction dilated convolutional structure to expand the receptive field without altering the spatial resolution of the feature map, thereby capturing a wide range of spatial dependencies and compensating for deep features. Spatial detail loss due to multiple downsampling.

[0052] Step 4.1, process the deep features output in step 2.2.3. Simultaneously input three parallel dilated convolution branches. Each branch uses dilated convolutional layers with dilation rates of 2, 4, and 8 (kernel size is...). The step size is 1), and edge padding is performed using ReplicationPad (a PyTorch layer that pads boundaries by copying the edge values ​​of the input tensor to expand the data size and maintain the continuity of boundary features, avoiding artifacts introduced by zero padding) to keep the feature map size constant. The receptive field of the branch with an expansion rate of 2 is... Suitable for capturing details of small-area terrain boundaries; the receptive field of a branch with an expansion rate of 4 is... Suitable for capturing the outline structure of medium-scale features; the receptive field of a branch with an expansion rate of 8 is... It is suitable for capturing large-scale land feature distribution patterns and spatial relationships.

[0053] Step 4.2: The three dilated convolution outputs are concatenated along the channel dimension and then processed by the convolution kernel. A convolutional layer with a stride of 1 is used to reduce the dimensionality to the input size, and then bilinear interpolation is used to upsample the feature map to the same spatial size as the shallow layer features. This achieves the mapping of deep semantic features to a high-resolution space, resulting in dilated and enhanced features. .

[0054] Step 5: Channel attention mechanism and multi-scale feature fusion like Figure 6As shown, after multi-scale feature enhancement, shallow features, pooling enhancement features, and dilation enhancement features are concatenated along the channel dimension to obtain a high-dimensional fused feature. This high-dimensional fused feature contains rich information from different levels and receptive fields, but the feature importance varies significantly between different channels. To address this, a channel attention mechanism module is designed to adaptively select channels for the high-dimensional fused feature.

[0055] Step 5.1, Global Context Compression. Global average pooling is performed sequentially along the spatial height and width dimensions on the high-dimensional fused features, ultimately compressing them into... The global description vector.

[0056] Step 5.2, Modeling inter-channel dependencies. The global description vector is sequentially input into two convolutional kernels. A convolutional layer with a stride of 1 and a ReLU activation function are used to introduce interactions between features through non-linear activation. The number of channels is compressed to 1 / 4 of the original after the first convolutional layer, restored to the original number after the second convolutional layer, and then processed by a third convolutional kernel. Convolutions with a stride of 1 maintain the same dimension, forming a channel attention modeling structure of "compression-activation-restoration".

[0057] Step 5.3, Attention Weighting and Feature Enhancement. The output of the channel dependency model is normalized using the Sigmoid function. The range is determined to obtain the channel attention weight vector. The channel attention weight vector is then multiplied channel by channel with the high-dimensional fusion feature (broadcasting mechanism) to achieve adaptive enhancement of important channel features and adaptive suppression of redundant channel features, outputting attention-enhanced features with the same dimension as the input.

[0058] Step 6: Model Training and Target Temporal Image Prediction Step 6.1, Multi-level Feature Refinement. The attention-enhanced fused features output from Step 5 are sequentially refined through two layers of wavelet transform convolutional modules. The first wavelet transform convolutional module compresses the feature dimension to 1 / 4 of its original size; the second wavelet transform convolutional module further reduces the feature dimension to 1 / 8. Because the wavelet transform convolutional module can learn features simultaneously on both low-frequency and high-frequency subbands, it can better preserve the high-frequency texture details of the remote sensing image during the feature refinement process compared to standard convolution.

[0059] Step 6.2, Band Dimension Mapping and Output Normalization. The refined features output in Step 6.1 are passed through the final convolution kernel to... A convolutional layer with a stride of 1 compresses the feature dimension to the same channel dimension as the input data, achieving a mapping from the high-dimensional feature representation within the network to the band space of the target remote sensing image. Then, the output is element-wise normalized using the Sigmoid function, limiting the predicted values ​​to a specific range. Within the range, the final high spatial resolution remote sensing image predicted at the target time is obtained. .

[0060] Step 6.3, Network Training Strategy. The entire network employs an end-to-end joint training strategy to compare predicted images with real high-resolution labeled images. The pixel-level mean squared error loss function (MSE Loss) is the primary optimization objective, combined with the structural similarity loss function (SSIM Loss) to constrain the structural consistency of the reconstructed image, forming the total loss function:

[0061] in The mean squared error loss is defined as follows: , and The predicted image and the labeled image are respectively in the first... The radiance value at each pixel location. Total number of pixels; The structural similarity loss is defined as follows: ; The preset balance weight coefficient.

[0062] This embodiment uses the PyTorch deep learning framework to construct and train the aforementioned network model. In the network input... 4 channels ( High-resolution GF1 reference image, and Each has 4 channels, spliced ​​together to form 8 channels. The network output is 4 channels. High-resolution GF1 prediction imagery. Training configuration as follows: batch size set to 2, using Adam optimizer (initial learning rate...). , , Weight decay The learning rate decay strategy employs cosine annealing scheduling (Cosine AnnealingLR). The training rounds are set to 100. In each round, the MSE and SSIM metrics are evaluated on the validation set, and the weights of the best-performing model are recorded and saved.

[0063] Step 7: Accuracy Evaluation and Experimental Results The fusion effect of this method is verified using test set data. The optimal model weights are loaded, and the predicted high-resolution GF1 image is obtained relative to the input network at the test time. For large-format images, the following methods are applied: The sliding window (step size 32 pixels) is used for block inference, and the overlapping areas are averaged and stitched together to obtain a complete fused image.

[0064] Peak signal-to-noise ratio (PSNR), structural similarity (SSIM), root mean square error (RMSE), mean absolute error (MAE), relative global dimensionless error (ERGAS), and universal image quality index (UIQI) were selected as quantitative evaluation indicators. The method of this invention was compared with existing mainstream spatiotemporal fusion methods (including deep learning-based ECPW-STFN, MFSTFN, MLFF_GAN, etc.), and the results are shown in Table 1. Table 1: Comparison of quantitative indicators between the method of the present invention and existing methods

[0065] As shown in Table 1, the method of this invention achieves a PSNR of 27.0952, reduces RMSE and MAE to 0.0507 and 0.0392 respectively, and reduces ERGAS to 34.3522, comprehensively outperforming existing mainstream spatiotemporal fusion methods. Especially... Figure 8 As shown in the visual comparison of typical experimental areas, in areas where crops undergo rapid phenological transitions such as rice transplanting and harvesting, and in areas with highly heterogeneous and fragmented farmland patches, conventional methods generally suffer from broken field ridge boundaries and radiometric deviations (exhibiting severe pseudo-color anomalies) due to the blurring of multi-frequency information caused by the receptive field and single full convolution. However, the Normalized Difference Vegetation Index (NDVI) predicted by this invention and the Pearson correlation coefficient R between the multispectral fusion image and the measured true value reach over 0.94, and the SSIM is consistently higher than 0.92, achieving a perfect balance between high temporal resolution and high spatial resolution.

[0066] To verify the necessity of the technical features of this invention, this embodiment further conducted ablation experiments, constructing four ablation derivative models based on this scheme. The results are shown in Table 2. Here, My_conv replaces all wavelet transform convolutional layers in the network with standard... Variants of fully convolutional; My_wo_pool is a variant that removes the mid-level multi-scale pyramid pooling enhancement module; My_wo_dilated is a variant that removes the deep multi-dilation rate parallel dilated convolution module; My_wo_attention is a variant that removes the channel attention selection module.

[0067] Table 2: Comparison Results of Ablation Experiments

[0068] Analysis of Table 2 shows that when wavelet transform convolution is replaced with standard full convolution (My_conv), the number of model parameters surges from 4,684,884 to 5,457,582, while SSIM deteriorates from 0.7768 to 0.7214, and RMSE increases by 20.5%. This indicates that wavelet transform convolution makes a decisive contribution to the fidelity of ground feature edges while maintaining parameter efficiency. Removing mid-level multi-scale pooling (My_wo_pool) or deep dilated convolution (My_wo_dilated) significantly degrades all accuracy indicators, especially the PSNR, which drops sharply by more than 1.11 dB after the deep dilated convolution is removed, confirming the necessity of this module for expanding the receptive field. Specifically, My_wo_dilated, due to the removal of the deep dilated convolution module and its subsequent upsampling and channel fusion pathways, has a significantly reduced network depth, resulting in a substantial decrease in the number of parameters. After removing channel attention (My_wo_attention), MAE and RMSE increased significantly, indicating the key role of this module in eliminating redundant channel interference and ensuring the rationality of reconstructed radiation.

[0069] Example 2 This embodiment is basically the same as Embodiment 1, except that the experimental area is selected and the data source is adjusted.

[0070] This embodiment uses a typical agricultural region in Northeast China as the experimental area, selecting Sentinel-2 and GF2 product data covering 2018 to 2023 as the experimental dataset. After preprocessing including radiometric calibration, atmospheric correction, and geometric fine correction, the GF2 imagery has a spatial resolution of 1m; the Sentinel-2 data, after the same preprocessing, has a spatial resolution of 10m. Using the high-resolution GF2 imagery as a reference, geometric registration and spatial resampling are performed on the Sentinel-2 imagery to ensure strict alignment of the two in a unified coordinate system.

[0071] In this embodiment, the reference time is denoted as The predicted time is denoted as Acquire four sets of registered remote sensing image data: low spatial resolution image at the reference time. (Sentinel-2, 10m resolution, 4 bands), high spatial resolution imagery at the reference time (GF2, 1m resolution, 4 bands) Low spatial resolution imagery at the predicted time (Sentinel-2, 10m resolution, 4 bands), and high spatial resolution labeled images for prediction time-to-time used for model training supervision. (GF2, 1m resolution, 4 bands). The same four spectral bands (green, red, red-edge, and near-infrared) as in Example 1 were selected for the experiment.

[0072] During the construction of the sample library, 20 pairs of Sentinel-2 and GF2 images with cloud cover below 10% were selected. A pixel-sliding window (32-pixel stride) is used for cropping, combined with data augmentation operations such as random horizontal flipping, random vertical flipping, and random rotation, to generate 48,000 pairs of training samples, 12,000 pairs of validation samples, and 20,000 pairs of test samples. The network structure and training strategy remain consistent with Example 1.

[0073] Example 3 This embodiment provides a remote sensing image spatiotemporal fusion device based on wavelet transform convolution and multi-scale feature fusion, used to implement the method of Embodiment 1, including: The image acquisition module is used to acquire high-resolution images of the same area at the reference time, low-resolution images at the reference time, and low-resolution images at the prediction time. Feature extraction module (i.e.) Figure 2 The feature extraction network module is used to construct a feature extraction network based on wavelet transform convolution. The high-resolution image at the reference time is used as the main branch input, and the channel stitching result of the low-resolution image at the reference time and the low-resolution image at the prediction time is used as the auxiliary branch input. Hierarchical feature extraction is performed through wavelet transform convolution layer, and the output is shallow features containing spatial detail information, medium features containing structural semantic information, and deep features containing global semantic information. The multi-scale pooling enhancement module is used to perform multi-scale pooling processing on mid-level features, capturing global spatial context information at different spatial scales and generating pooled enhanced features. The multi-scale dilated convolution enhancement module is used to apply dilated convolutions with different dilation rates to deep features in parallel, expanding the receptive field and generating dilated enhanced features without changing the spatial resolution of the feature map. The channel attention fusion module is used to fuse shallow features, pooling enhancement features and dilation enhancement features in the channel dimension, and to adaptively weight the fused features using the channel attention mechanism to obtain attention-enhanced features; The image reconstruction module is used to refine the attention-enhanced features and perform band mapping through the wavelet transform convolution module to reconstruct a high-resolution image at the predicted time.

[0074] The specific implementation details of each of the above modules are completely consistent with the corresponding steps in Implementation Example 1.

[0075] Example 4 This embodiment also provides an electronic device, see reference. Figure 9It includes a memory 402 and a processor 401, the memory 402 storing a computer program and the processor 401 being configured to run the computer program to perform the steps in any of the above method embodiments.

[0076] Specifically, the processor 401 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0077] The memory 402 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 402 may include removable or non-removable (or fixed) media. Where appropriate, the memory 402 may be internal or external to a data processing device. In a particular embodiment, the memory 402 is non-volatile memory. In a particular embodiment, the memory 402 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random Access Memory (FPMDRAM), Extended Data Out Dynamic Random Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0078] The memory 402 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 401.

[0079] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any of the remote sensing image spatiotemporal fusion methods based on wavelet transform convolution and multi-scale feature fusion in the above embodiments.

[0080] Optionally, the electronic device may further include a transmission device 403 and an input / output device 404, wherein the transmission device 403 is connected to the processor 401 and the input / output device 404 is connected to the processor 401.

[0081] The transmission device 403 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 403 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0082] Input / output device 404 is used for inputting or outputting information. It can be a speaker, microphone, monitor, or keyboard.

[0083] Example 5 This embodiment also provides a readable storage medium storing a computer program, which includes program code for controlling and executing a process, the process including the remote sensing image spatiotemporal fusion method based on wavelet transform convolution and multi-scale feature fusion according to Embodiment 1.

[0084] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0085] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0086] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 1 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0087] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] The above embodiments are merely illustrative of several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A spatiotemporal fusion method for remote sensing images based on wavelet transform convolution and multi-scale feature fusion, characterized in that, Includes the following steps: Acquire high-resolution images of the same region at the reference time, low-resolution images at the reference time, and low-resolution images at the predicted time. A feature extraction network based on wavelet transform convolution is constructed. The high-resolution image at the reference time is used as the input of the main branch, and the channel stitching result of the low-resolution image at the reference time and the low-resolution image at the prediction time is used as the input of the auxiliary branch. Hierarchical feature extraction is performed through wavelet transform convolution layer, and the output is shallow features containing spatial detail information, medium features containing structural semantic information, and deep features containing global semantic information. Multi-scale pooling is performed on the mid-layer features to capture global spatial context information at different spatial scales and generate pooled enhanced features. Dilated convolutions with different dilation rates are applied in parallel to the deep features to expand the receptive field without changing the spatial resolution of the feature map, thereby generating dilated and enhanced features. The shallow features, the pooling enhancement features, and the dilation enhancement features are fused along the channel dimension, and the fused features are adaptively weighted using a channel attention mechanism to obtain attention-enhanced features. The attention-enhanced features are refined and mapped using a wavelet transform convolution module to reconstruct a high-resolution image at the predicted time.

2. The remote sensing image spatiotemporal fusion method as described in claim 1, characterized in that, The wavelet transform convolutional layer performs discrete wavelet transform decomposition on the input feature map to obtain low-frequency sub-bands and multiple high-frequency sub-bands. Depth-separable convolution operations are performed on each sub-band, and features are reconstructed through inverse wavelet transform and superimposed with the features of the basic convolutional branch for output.

3. The remote sensing image spatiotemporal fusion method as described in claim 1, characterized in that, The feature extraction network integrates spatial features of the high-resolution image input from the main branch using a wavelet transform convolution module, and extracts temporal difference features of the stitched low-resolution image input from the auxiliary branch using a wavelet transform convolution module. After the two features are concatenated in the channel dimension, they are processed by a combination of a convolution module and a wavelet transform convolution module, and with residual connection, the shallow features, middle features and deep features are output sequentially. The mid-layer features are obtained by downsampling the shallow-layer features, and the deep-layer features are obtained by downsampling the mid-layer features.

4. The remote sensing image spatiotemporal fusion method as described in claim 1, characterized in that, The multi-scale pooling process includes: The mid-layer features are input into multiple parallel branches. Each branch downsamples the feature map to different scales through an adaptive average pooling layer, widens the channel dimension through a convolutional layer, and then restores it to the original spatial size through upsampling. The output features of each branch are concatenated in the channel dimension, reduced in dimension by a convolutional layer, and then upsampled to the same spatial size as the shallow features to obtain the pooling-enhanced features.

5. The remote sensing image spatiotemporal fusion method as described in claim 4, characterized in that, The number of parallel branches is three, and the adaptive average pooling downsampling factors of the three branches are 2x, 4x and 8x respectively; Each branch widens the channel dimension through 1×1 convolution, and after batch normalization and activation function processing, it is restored to the original spatial size through bilinear interpolation upsampling.

6. The remote sensing image spatiotemporal fusion method as described in claim 1, characterized in that, The dilated convolution with different dilation rates includes at least three parallel dilated convolution branches, each branch using dilated convolution layers with different dilation rates and maintaining the feature map size unchanged by edge padding. The dilation rates of the at least three parallel dilated convolution branches are 2, 4 and 8, respectively. The outputs of each branch are concatenated along the channel dimension, then dimensionality is reduced by a convolutional layer, and finally upsampled to restore the same spatial size as the shallow features to obtain the dilated enhancement features.

7. The remote sensing image spatiotemporal fusion method as described in claim 1, characterized in that, The channel attention mechanism includes: Global average pooling is performed on the features fused from the channel dimensions to obtain the global description vector; The global description vector is sequentially passed through a dimension reduction convolutional layer, a non-linear activation function, and a dimension increase convolutional layer to establish the dependency relationship between channels; The channel attention weight vector is obtained by normalization using the Sigmoid function. The weight vector is then multiplied with the fused features channel by channel to achieve adaptive enhancement of effective feature channels and adaptive suppression of redundant feature channels.

8. The remote sensing image spatiotemporal fusion method as described in claim 1, characterized in that, The method also includes a model training step: The feature extraction network, the multi-scale pooling process, the multi-scale dilated convolution process, and the channel attention mechanism are jointly trained end-to-end using a combination of mean squared error loss function and structural similarity loss function as the total loss function.

9. A spatiotemporal fusion device for remote sensing images based on wavelet transform convolution and multi-scale feature fusion, characterized in that, include: The image acquisition module is used to acquire high-resolution images of the same area at the reference time, low-resolution images at the reference time, and low-resolution images at the prediction time. The feature extraction module is used to construct a feature extraction network based on wavelet transform convolution. It takes the high-resolution image at the reference time as the main branch input and the channel stitching result of the low-resolution image at the reference time and the low-resolution image at the prediction time as the auxiliary branch input. It performs hierarchical feature extraction through wavelet transform convolution layers and outputs shallow features containing spatial detail information, medium features containing structural semantic information, and deep features containing global semantic information. A multi-scale pooling enhancement module is used to perform multi-scale pooling processing on the mid-layer features to generate pooled enhanced features. A multi-scale dilated convolution enhancement module is used to apply dilated convolutions with different dilation rates in parallel to the deep features to generate dilated enhanced features; The channel attention fusion module is used to fuse the shallow features, the pooling enhancement features, and the dilation enhancement features in the channel dimension, and to perform adaptive weighting using the channel attention mechanism to obtain attention enhancement features; The image reconstruction module is used to refine the attention enhancement features and perform band mapping through the wavelet transform convolution module to reconstruct a high-resolution image at the predicted time.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the remote sensing image spatiotemporal fusion method according to any one of claims 1 to 8.

Images