A Spatiotemporal Fusion Method for Remote Sensing Images Based on Physical Mechanism Guidance and Structure Texture Guidance Masks
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-01-26
- Publication Date
- 2026-05-26
Smart Images
Figure CN122089581A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of remote sensing image processing technology, and relates to a spatiotemporal fusion method for remote sensing images based on physical mechanism guidance and structure texture guidance mask. Background Technology
[0002] Remote sensing technology, as a crucial means of acquiring information about the Earth's surface, plays an irreplaceable role in fields such as agricultural monitoring, ecological environment assessment, natural disaster emergency response, and urbanization process analysis. In these practical applications, surface processes often exhibit highly dynamic characteristics, making the simultaneous acquisition of remote sensing data with both high spatial and temporal resolution particularly important. However, limited by the physical capabilities of satellite sensors, a single remote sensing satellite system can hardly simultaneously meet the demands for high spatiotemporal resolution.
[0003] High temporal resolution satellites, such as the Moderate Resolution Imaging Spectroradiometer (MODIS) and Sentinel-3, typically have extremely short revisit periods, enabling them to capture rapid changes in ground features. However, their spatial resolution is relatively low, with a large number of mixed pixels. High spatial resolution satellites, such as Landsat-8 / 9 and Sentinel-2, provide clear details of surface textures, but their long revisit periods and susceptibility to cloud and rain conditions can lead to missing valuable observational data.
[0004] To address the aforementioned contradictions, remote sensing image spatiotemporal fusion technology has emerged. This technology aims to fuse high-frequency temporal information from low-resolution images and high-frequency spatial information from high-resolution images using algorithmic methods, generating a synthetic image that possesses both high temporal frequency and high spatial detail. Existing spatiotemporal fusion methods are mainly divided into three categories based on their core principles.
[0005] Traditional methods based on transformation and weighting, exemplified by the spatiotemporal adaptive reflectance fusion model STARFM, rely on spectral similarity and spatial distance to search for similar pixels within a sliding window. The subsequently developed enhanced spatiotemporal adaptive reflectance fusion model ESTARFM introduces transformation coefficients to improve prediction accuracy for heterogeneous landscapes. While these methods are logically intuitive, they are highly dependent on the assumption of pixel spectral similarity.
[0006] Methods based on mixed pixel decomposition, such as the Flexible Spatiotemporal Data Fusion Model (FSDAF), are based on the theory of linear spectral mixing models and obtain endmember abundance changes by unmixing low-resolution images. These methods have good physical interpretability, but the linear assumptions are overly idealized, and the unmixing process is highly unstable.
[0007] Deep learning-based methods have gradually become mainstream in recent years. For example, the dual-stream convolutional spatiotemporal fusion network DCSTFN uses a dual-stream convolutional network to extract features separately, and the SwinFusion fusion model based on the Swin Transformer introduces a long-range dependency capture mechanism. These methods learn complex nonlinear mapping relationships through training on large-scale data, and generally outperform traditional methods in terms of metrics.
[0008] However, existing deep learning methods still have significant limitations when dealing with complex real-world environments. First, the models lack guidance from physical mechanisms, typically treating all bands as homogeneous channels, leading to low prediction accuracy and spectral distortion in the infrared band, which is sensitive to spectral changes. Second, because temporal variation information originates from blurry, low-resolution images, the boundaries of directly extracted change features are blurred, resulting in jagged edges and texture loss in dynamic regions. Finally, the receptive field of feature extraction is limited, making it difficult to balance local details with global contextual information, causing the models to either lose details or make inaccurate predictions of large-scale changes when dealing with complex terrain.
[0009] Therefore, there is an urgent need for a new method for spatiotemporal fusion of remote sensing images that can effectively solve the above problems. Summary of the Invention
[0010] In view of this, the purpose of this invention is to provide a spatiotemporal fusion method for remote sensing images based on physical mechanism-guided and structure texture-guided masks, so as to overcome the key defects existing in the prior art. Specifically, the objectives include the following three aspects:
[0011] By decoupling spectral physical characteristics, the prediction error in the infrared band is significantly reduced, ensuring the accuracy of remote sensing index calculations such as the normalized vegetation index, and solving the problem of spectral distortion caused by neglecting the physical differences of the bands in existing methods.
[0012] By introducing high-resolution texture constraints, ghosting artifacts and jagged edges in changing regions are eliminated, generating high spatiotemporal resolution images with sharp boundaries and rich details, thus improving the problem of blurred boundaries in dynamic regions.
[0013] By employing a multi-scale adaptive perception mechanism, the coherence of global landform tones and phenological trends is maintained while restoring local details, achieving an effective balance between local texture and global context, and enhancing the visual consistency and practicality of the fused image.
[0014] To achieve the above objectives, the present invention provides the following technical solution: A spatiotemporal fusion method for remote sensing images based on physical mechanism-guided and structure-texture-guided masks includes the following steps: Receive a high-resolution reference image at the initial time, a low-resolution image at the initial time, and a low-resolution image at the predicted time; The difference between the low-resolution image at the predicted time and the low-resolution image at the initial time is calculated to obtain a low-resolution difference map, and edge features are extracted from the high-resolution reference image. The low-resolution difference map is subjected to physical mechanism-guided spectral group coding to obtain variation features, and the high-resolution reference image is subjected to spectral group coding to obtain detail features; A prediction mask is obtained by generating a structure-texture guided mask using the detailed features and the variation features. The detailed features, the variation features, and the edge features are fused together and a high-resolution residual map is generated through a reconstruction network; The spatiotemporal fusion image is obtained by adding the high-resolution residual image to the high-resolution reference image.
[0015] Furthermore, the physical mechanism-guided spectral grouping coding includes: dividing the multi-band remote sensing image into a visible light band group, a near-infrared band group, and a short-wave infrared band group, and then extracting features from each group and splicing them together to form a fused feature.
[0016] Furthermore, after the spectral grouping and encoding, the fused features are further reweighted by a coordinate spectral attention module, which generates attention weights through horizontal and vertical aggregation.
[0017] Furthermore, the structure-texture guided mask generation includes: using the detail features as a conditional flow to generate a scaling factor map and an offset factor map, and performing affine modulation on the variation features as a content flow, wherein the affine modulation includes multiplying the variation features element-wise with the scaling factor map and then adding them element-wise with the offset factor map.
[0018] Furthermore, the extraction of edge features is implemented using the Sobel operator.
[0019] Furthermore, the reconstruction network comprises multiple stacked modern residual blocks, each containing a large-size convolutional layer and an inverted bottleneck layer.
[0020] Furthermore, the fused features also include spatial consistency calibration through a selective pixel attention module, which dynamically weights gating weights generated through local and global perception paths.
[0021] Furthermore, the local perception path uses standard convolution to extract features, while the global perception path uses dilated convolution to extract features.
[0022] Furthermore, the high-resolution residual image is weighted by the prediction mask and then added to the high-resolution reference image, the weighting coefficients including a bias constant. α .
[0023] Furthermore, the bias constant α The value range is 0.2.
[0024] The beneficial effects of this invention are as follows: (1) Existing deep learning methods typically treat all bands as homogeneous channels, ignoring the differences in the physical mechanisms of the responses of visible, near-infrared, and short-wave infrared bands to ground objects. This invention employs a spectral grouping coding strategy guided by physical mechanisms to divide the input data into three physical subspaces—visible, near-infrared, and short-wave infrared—for feature extraction, thus avoiding mutual interference between features of different physical attributes. Combined with a coordinate-spectral attention mechanism, the model can explicitly learn the importance weights between bands, significantly reducing the prediction error of infrared bands that are sensitive to spectral changes. This design makes the fused images more accurate in calculating remote sensing indices such as the Normalized Difference Vegetation Index (NDVI), meeting the stringent requirements for spectral fidelity in agricultural monitoring and ecological environment assessment.
[0025] (2) To address the challenges of jagged edges and blurring in change regions caused by low-resolution difference images, this invention designs a structure-texture-guided mask generation module. This module utilizes the clear texture of a high-resolution reference image as a priori condition and uses an affine modulation mechanism to spatially constrain and inject texture into the blurred change features. Specifically, the change features are shaped by scaling and offset factors, forcing the blurred features to conform to the texture boundaries of the high-resolution image. This enables the model to maintain sharp boundaries and rich details consistent with the high-resolution reference image in areas of significant change such as crop harvesting and flooding, effectively eliminating ghosting artifacts and jagged effects common in traditional methods.
[0026] (3) In the feature reconstruction stage, this invention combines modern residual blocks and selective pixel attention modules to form an efficient multi-scale perception mechanism. Modern residual blocks utilize large-size convolutional kernels to expand the receptive field, enabling them to perceive a wide range of phenological changes; the selective pixel attention module uses a dual-stream gating mechanism to adaptively decide whether to focus on local texture restoration or global background consistency for each pixel. This design allows the model to simultaneously consider subtle ground textures and macroscopic phenological trends in complex terrain environments, significantly improving the spatiotemporal coherence and visual realism of the fused image.
[0027] (4) Through the synergistic effect of physical mechanism guidance and structural texture constraints, this invention achieves synergistic optimization in multiple dimensions such as amplitude error, spectral angle error, and multi-band normalization error. Experimental results show that on the CIA and LGC datasets, this method outperforms existing mainstream methods in key indicators such as root mean square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), spectral angle mapping (SAM), and relative global dimension comprehensive error (ERGAS), confirming its technical superiority and practical value.
[0028] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 For the overall network architecture; Figure 2 Designed for coordinate spectral attention module; Figure 3 Designed for selective pixel attention modules; Figure 4 Design for a structure-texture guided mask generation module; Figure 5 Landsat image for reference time; Figure 6 Landsat image for the predicted time; Figure 7 This is the predicted image for this invention. Detailed Implementation
[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0031] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0032] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0033] This invention proposes a spatiotemporal fusion network for remote sensing images based on physical mechanism guidance and structure texture guidance masks. This method aims to address the problems of low prediction accuracy in the infrared band, blurred boundaries of changing regions, and difficulty in simultaneously considering local and global features in existing technologies.
[0034] I. Overall Network Architecture like Figure 1 As shown, the network architecture described in this invention is an end-to-end deep learning model, and its overall workflow is as follows: 1. The network receives three input data streams: a high-resolution reference image at the initial time (referred to as "high-resolution reference image"), a low-resolution image at the initial time, and a low-resolution image at the prediction time.
[0035] 2. First, calculate the difference between the low-resolution image at the predicted time and the initial time to obtain a "low-resolution difference map" that represents the changes in the basic ground features; at the same time, use the edge extraction operator (Sobel operator) to extract "edge features" from the high-resolution reference map to preserve the high-frequency boundary information of the ground features.
[0036] 3. Two-stream feature encoding: The network contains two parallel encoding paths, one for spectral variations and the other for spatial details. a. Variation Flow (Spectral Guided): Processes low-resolution difference maps. First, visible light and infrared features are decoupled through a physically guided spectral group encoder, and then the coordinate-spectral attention module is used to capture cross-channel spectral variation features.
[0037] b. Detail Flow (Spatial Guided): Processes the high-resolution reference image. It first passes through a spectral group encoder to extract basic features, then enters a selective pixel attention module to adaptively extract fine spatial structure and texture features.
[0038] 4. Structure-Guided Mask Generation: The encoded features are fed into the structure-texture guided mask generator. This module uses the clear texture of the high-resolution reference image as a "mold" and reshapes the blurred and changing features through an affine modulation mechanism, outputting a "predictive mask" that indicates the probability of change and has clear boundaries.
[0039] 5. Feature Fusion and Reconstruction: Detail features, variation features, and edge features extracted in step 2 are concatenated along the channel dimension and then input into a backbone network composed of multiple stacked modern residual blocks for deep fusion. At the output, spatial focusing is performed again through a selective pixel attention mechanism to finally generate a high-resolution residual map.
[0040] 6. Output Results: The high-resolution reference image is added to the weighted high-resolution residual image to obtain the final high spatiotemporal resolution fused image.
[0041] To achieve the above functions, the present invention designs the following three core sub-modules: Coordinate Spectral Attention Module 1. Module Design Motivation In remote sensing image fusion tasks, the spatial distribution of spectral responses from different land features such as water bodies, vegetation, and buildings is non-uniform. Traditional channel attention mechanisms typically use global average pooling to compress two-dimensional spatial information into a single value, resulting in the loss of crucial location information. To enhance the expressive power of spectral features while accurately preserving the spatial location information of land features, this embodiment designs a coordinate spectral attention module.
[0042] 2. Structure and Working Principle This module captures long-range dependencies by decomposing two-dimensional spatial features into two mutually orthogonal one-dimensional directional features. The specific processing flow includes the following three steps: Step 1: Coordinate Information Aggregation First, the input feature maps are fed into two parallel adaptive average pooling layers for processing: Horizontal aggregation: Features are aggregated along the height direction of the image to generate a vertical feature vector with the original height and a width of one. This vector captures long-distance dependencies along the height direction.
[0043] Vertical aggregation: Features are aggregated along the width of the image to generate a horizontal feature vector with a height of one and a width equal to the original width. This vector captures long-distance dependencies along the width direction. This decomposition strategy allows the network to perceive the global view in two directions separately, while preserving the precise positional coordinates in the other direction.
[0044] Step 2: Coordinate-Spectral Joint Encoding To facilitate the interaction between spectral and spatial location information, the generated vertical and horizontal feature vectors are concatenated spatially to obtain a hybrid feature map containing information from both directions. Subsequently, a shared 1x1 convolutional layer is used to reduce the dimensionality of the hybrid feature map, followed by a group normalization layer and a non-linear activation layer (such as LeakyReLU). This step not only reduces the number of computational parameters in the model, but more importantly, by compressing the channel dimension, it enables cross-channel information interaction, generating an intermediate feature map rich in spatial orientation information.
[0045] Step 3: Attention Generation and Feature Reweighting The encoded intermediate feature map is then divided into two independent branches along the spatial dimension, corresponding to the height and width directions, respectively.
[0046] The feature channel number is restored to the original dimension using two independent 1x1 convolutional layers.
[0047] The sigmoid activation function is used to generate vertical and horizontal attention weight maps, respectively. Finally, these two attention weight maps are element-wise multiplied with the original input feature map. In this way, the network can adaptively highlight key spectral features and spatial locations while suppressing background noise, based on the generated weights.
[0048] 3. Beneficial effects Compared to conventional attention mechanisms, this module has the following significant advantages: Preservation of location information: By using an orthogonal decomposition strategy, the spatial information collapse caused by traditional global pooling is avoided, enabling the model to accurately locate small features (such as field ridges and roads).
[0049] Enhanced spectral discrimination: By compressing and restoring the channel dimension, the network explicitly learns the importance weights between bands, effectively improving the expressive ability of infrared band features.
[0050] Computationally efficient: Using only one-dimensional pooling and one-to-one convolution, computational complexity is significantly reduced. Modular design as follows... Figure 2 As shown.
[0051] Selective pixel attention module 1. Module Design Motivation In the process of remote sensing image fusion, the feature scales that need to be focused on vary significantly across different regions. For example, detailed textures such as farmland boundaries and roads require fine local features for reconstruction, while areas covered by clouds and large areas of forest require broad global contextual information to maintain consistency. To enable the network to adaptively adjust its focus on features at different scales, this embodiment designs a selective pixel attention module.
[0052] 2. Structure and Working Principle This module extracts local details and global context separately through a dual-stream sensing path, and achieves dynamic fusion using a gating mechanism. The specific processing flow includes the following four steps: Step 1: Feature Dimensionality Reduction First, a 1-to-1 convolutional layer is used to compress the channels of the input high-dimensional feature map. This step aims to reduce the complexity of subsequent calculations while extracting the most representative basic features.
[0053] Step 2: Dual-stream feature perception The dimensionality-reduced feature maps are then input into two parallel perceptual paths: Local perceptual path: Features are extracted using a 3x3 standard convolution. This path has a small receptive field and focuses on capturing high-frequency details such as edges and corners in the image.
[0054] Global Perception Path: This path employs dilated convolutions with a dilation rate of four to extract features. While maintaining the feature map resolution, it significantly expands the receptive field, focusing on capturing a wide range of background context and low-frequency structural information. Both paths are equipped with group normalization layers and non-linear activation layers to ensure the stability of feature extraction.
[0055] Step 3: Adaptive Gating Generation To determine whether each pixel should rely more on local or global features, the feature maps output from the two paths are concatenated along the channel dimension. The concatenated feature map is then input into a gated generator containing a 1x1 convolutional layer and a sigmoid activation function. This generator outputs a gated weight map with values ranging from zero to one. Weight values close to one indicate that the pixel is biased towards local details, while weight values close to zero indicate that the pixel is biased towards global context.
[0056] Step 4: Dynamic Weighting and Output The generated gated weight map is used to perform weighted fusion of the two-stream features: the final fused feature equals (local features multiplied by gated weights) plus (global features multiplied by (one minus the gated weights)). Finally, the fused features are passed through a 1-to-1 convolutional layer and a sigmoid activation function to generate the final spatial attention map, which is then multiplied element-wise with the original input features of the module to complete feature reweighting.
[0057] 3. Beneficial effects Compared to traditional spatial attention mechanisms, this module has the following advantages: Multi-scale adaptive: It can automatically switch the focus of attention according to the image content, which preserves clear texture boundaries and ensures the consistency of the overall tone.
[0058] Lightweight design: It abandons complex fully connected layers and Softmax operations, and only uses convolution and sigmoid gating, which greatly reduces the amount of computation and is suitable for processing large-size remote sensing images.
[0059] Avoiding the checkerboard effect: Through a carefully designed combination of dilated convolution and local convolution, the grid-like artifacts that may be caused by single-scale convolution are effectively eliminated. Modular design such as... Figure 3 As shown.
[0060] Structure-Texture Guided Mask Generation Module 1. Module Design Motivation In the spatiotemporal fusion of remote sensing images, since the input temporal variation information mainly comes from low-resolution images, the directly extracted variation features often suffer from problems such as blurred boundaries and missing textures. This leads to jagged edges or ghosting artifacts in the generated fused image in areas of significant change (such as crop harvesting or flooding). To address this challenge, this embodiment designs a structure-texture guided mask generation module based on affine modulation, aiming to utilize the clear texture of a high-resolution reference image as a "mold" to spatially constrain and inject textures into blurred variation features.
[0061] 2. Structure and Working Principle This module achieves modulation of low-frequency variation information by high-frequency structural information through the collaborative work of structural and content branches. The specific processing flow includes the following four steps: Step 1: Dual-stream feature extraction Structural branch (conditional flow): Detailed features from the input high-resolution reference image. This branch extracts high-frequency structural information such as sharp edges and textures from the image through a multi-layer convolutional network, serving as the "conditions" for subsequent modulation.
[0062] Content Branch (Content Stream): Input the difference features of the low-resolution image. This branch extracts coarse information on the intensity of ground feature changes as the "content" to be modulated.
[0063] Step 2: Modulation Parameter Generation Using the high-frequency structural information extracted from the structural branches, two sets of spatial modulation parameters are predicted through two parallel 1x1 convolutional layers: Scaling factor map: Used to control the spatial distribution range of variation features. This factor is processed by the Sigmoid activation function, which normalizes its value to between zero and one, acting as a kind of "gating".
[0064] Offset factor map: Used to inject detailed information into changing features. This factor does not undergo an activation function, preserving the original numerical distribution and containing rich texture details.
[0065] Step 3: Affine Modulation Operation This is the core step of this module. The following modulation operation is performed: the variation features of the content branch are multiplied element-wise with the scaling factor map. This step uses high-resolution structural boundaries (scaling factors) to trim the diffusion range of the variation features and suppress noise in non-variable areas. Subsequently, the product result is added element-wise with the offset factor map. This step directly injects high-resolution texture details (offset factors) into the variation features, thereby restoring the fine textures within the variation areas.
[0066] Step 4: Mask Refinement and Output The modulated feature map combines accurate intensity of change with clear structural texture. This feature map is further processed through refined convolutional layers, ultimately outputting a single-channel prediction mask. This mask indicates the probability intensity of change at each pixel and possesses sharp edge structure.
[0067] 3. Beneficial effects Through the affine modulation mechanism described above, this module achieves the following technical effects: Boundary sharpening: Forces the variation features of the blur to conform to the texture boundaries of the high-resolution image, effectively eliminating jagged edges and smudging in the fused image.
[0068] Texture fidelity: By injecting an offset factor, the changed areas not only have the correct color, but also have realistic texture details of ground features.
[0069] Strong noise resistance: By utilizing the gating effect of the scaling factor, it automatically suppresses background noise commonly found in low-resolution difference maps.
[0070] This mask is used to spatially selectively enhance high-resolution residuals during the image reconstruction stage, thereby providing stronger correction to changed areas and keeping unchanged areas stable.
[0071] Module design such as Figure 4 As shown.
[0072] This embodiment, combining data flow and feature dimension changes, details the specific working process of the network described in this invention. The core of this invention lies in designing a dual-stream heterogeneous coding structure, which differentiates the extraction of spectral attributes of changing features and spatial attributes of detail features, respectively.
[0073] 1. Data Input and Preprocessing Stage Specifically, the network first receives three tensors as input: a high-resolution reference image at the initial time step, a low-resolution image at the initial time step, and a low-resolution image at the prediction time step, each with 6 input channels. The data first passes through an interpolation unit to obtain a low-resolution interpolation image (6 channels). Simultaneously, the high-resolution reference image is fed into a learnable edge extractor (initializing the Sobel operator), outputting an edge feature map (6 channels). Subsequently, the low-resolution interpolation image is fed into a preprocessing module containing a 7×7 large kernel convolution and group normalization layers, outputting preprocessed variation features (6 channels) to eliminate the mosaic effect.
[0074] 2. Dual-stream hybrid feature coding stage Specifically, the preprocessed variation features and the high-resolution reference image are respectively entered into two coding branches with different design philosophies: Branch 1: Variation Feature Coding (Spectral Guided) a. The preprocessed variation characteristics (6 channels) first pass through a physical grouping layer and are divided into a visible light group (3 channels), a near-infrared group (1 channel), and a short-wave infrared group (2 channels). The spectral grouping strategy divides the multi-channel data into several subspaces with similar physical response mechanisms based on the physical band characteristics of the sensor.
[0075] b. The three sets of sub-features are processed by independent convolutional layers to extract shallow features, and then concatenated along the channel dimension to output a fused feature map with 96 channels.
[0076] c. The fused feature map is then fed into the Coordinate-Spectral Attention (CSA) module. The CSA module focuses on the interaction of the spectral dimensions, generating cross-channel attention weights through horizontal and vertical pooling, thereby enhancing the network's ability to perceive "in which band the change occurred". The final output is the encoded change feature (96 channels).
[0077] Branch 2: Detail Feature Encoding (Spatial Oriented) a. The high-resolution reference image (6 channels) is also first extracted through a physical grouping layer and shallow convolution to generate a basic feature map with a dimension of 96 channels.
[0078] b. The base feature map is then fed into the Selective Pixel Attention (PSA) module.
[0079] c. The PSA module utilizes a dual-stream path (3×3 local convolution and dilated global convolution) for multi-scale perception, generating a pixel-level spatial attention mask to identify high-frequency textures and low-frequency backgrounds.
[0080] d. The final output is encoded detailed features (96 channels), rich in discriminative spatial structural information. Furthermore, the edge feature map is passed through a projection layer to output projected edge features (48 channels). Additionally, the edge feature map is passed through a projection layer to output projected edge features with a dimension of 48 channels.
[0081] 3. Structure-Texture Guided Mask Generation Stage Specifically, the encoded detail features (96 channels, conditional flow) and the encoded variation features (96 channels, content flow) are simultaneously fed into the structure-texture guided mask generator. After extracting structural information through convolution, the detail features predict a scale map and a shift map, both with 96 channels. Then, an affine modulation operation is performed: the variation features are element-wise multiplied by the scale factor and then element-wise added to the shift factor. The modulated features (96 channels) are output. This feature is then refined through a refining layer (convolutional dimensionality reduction + sigmoid), finally outputting a prediction mask (1 channel).
[0082] 4. Multi-scale feature fusion and reconstruction stage Specifically, encoded detail features (96 channels), encoded variation features (96 channels), and projected edge features (48 channels) are concatenated along the channel dimension to generate a joint feature map (240 channels). This map is then reduced to 96 channels via fusion convolution and fed into a backbone network consisting of three stacked large-kernel inverted residual blocks. Each block sequentially contains a 7×7 depthwise convolution, group normalization, an inverted bottleneck layer, and layer scaling, maintaining 96 channels in the output. Subsequently, the feature map undergoes spatial consistency calibration again through a selective pixel attention module.
[0083] 5. Image reconstruction and output stage Attention-weighted feature maps are fed into the reconstruction layer to generate high-resolution residual maps. Finally, the high-resolution residual map is multiplied by (prediction mask + bias constant). ),in The optimal value is 0.2 to ensure minimal correction capability in areas of slight variation. Finally, it is added to the original high-resolution reference image to output the final high spatiotemporal resolution fused image.
[0084] This invention was compared with some classic models on the CIA and LGC datasets. The results are as follows (all indicators are statistically analyzed on effective pixels (excluding NoData), and the average of 6 bands is taken; PSNR is calculated by normalizing the pixel value to [0,1]).
[0085] Table 1 CIA Dataset
[0086] Table 2 LGC Dataset
[0087] Figures 5-7 The fusion effect on the CIA dataset is shown. The experiment uses data from 20011017 to 20011102 for visualization (20011017 represents the reference time t0, and 20011102 represents the predicted time t1).
[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A spatio-temporal fusion method of remote sensing images based on physical mechanism guidance and structural texture oriented masks, characterized in that: The method comprises the following steps: receiving a high-resolution reference image at an initial time, a low-resolution image at the initial time, and a low-resolution image at a prediction time; calculating a low-resolution difference image by calculating the difference between the low-resolution image at the prediction time and the low-resolution image at the initial time, and extracting edge features from the high-resolution reference image; performing physical mechanism guided spectral grouping coding on the low-resolution difference image to obtain change features, and performing spectral grouping coding on the high-resolution reference image to obtain detail features; generating a prediction mask by using the detail features to perform structure-texture oriented mask generation on the change features; fusing the detail features, the change features, and the edge features, and generating a high-resolution residual image by using a reconstruction network; adding the high-resolution residual image and the high-resolution reference image to obtain a spatio-temporal fusion image.
2. The method of claim 1, wherein the method is characterized by: The physical mechanism guided spectral grouping coding comprises: dividing a multi-band remote sensing image into a visible light band group, a near-infrared band group, and a short-wave infrared band group, and respectively performing feature extraction on each group and then splicing the groups into fused features.
3. The method of claim 2, wherein the method is characterized by: The spectral grouping coding further comprises re-weighting the fused features by using a coordinate spectral attention module, and the coordinate spectral attention module generates attention weights by horizontal direction aggregation and vertical direction aggregation.
4. The method of claim 1, wherein the method is characterized by: The structure-texture oriented mask generation comprises: using the detail features as a condition stream to generate a scaling factor map and an offset factor map, and performing affine modulation on the change features as a content stream, and the affine modulation comprises multiplying the change features and the scaling factor map element by element and then adding the offset factor map element by element.
5. The method of claim 1, wherein the method is characterized by: The edge features are extracted by using a Sobel operator.
6. The method of claim 1, wherein the method is characterized by: The reconstruction network comprises a plurality of stacked modern residual blocks, and each modern residual block comprises a large-size convolution layer and an inverted bottleneck layer.
7. The method of claim 1, wherein the method is characterized by: The fused features further comprise spatial consistency calibration by using a selective pixel attention module, and the selective pixel attention module generates gating weights by using a local perception path and a global perception path to perform dynamic weighting.
8. The method of claim 7, wherein the method is characterized by: The local perception path uses a standard convolution to extract features, and the global perception path uses a dilated convolution to extract features.
9. The method of claim 1, wherein the method is characterized by: The high resolution residual map is added to the prediction mask weighted high resolution reference image, with a weighting factor including a bias constant α .
10. The method of claim 9, wherein the method is characterized by: the bias constant α is in the range of 0.2.