A panchromatic sharpening method based on frequency domain phase adaptive correction
Patent Information
- Application Number
- CN202611363377.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-04
- Publication Date
- 2026-10-09
AI Technical Summary
[0007]本发明的主要目的是为了解决现有的遥感图像处理中所用的全色锐化方法在使用过程中存在的诸多问题，而提供的一种基于频域相位自适应校正的全色锐化方法
[0046]本发明提供的基于频域相位自适应校正的全色锐化方法采用单步流匹配生成框架直接预测图像流形状态，避免了传统生成式模型的高耗时迭代采样过程，在仅需单次网络函数评估的情况下完成生成，显著降低了计算资源消耗和推理延迟；本发明创造性地从频域角度揭示了深层网络空间退化的本质为“相位偏移”问题，并提出利用全色图像天然的高频几何结构构建频域锚点。通过结构能量掩码与Von-Mises相干门限进行自适应相位修正，有效遏制了深层特征在网络传播过程中的几何畸变；在频域自适应校正过程中，本发明巧妙地采用了“相位插值、振幅保留”的策略。由于图像的光谱信息主要由频域振幅决定，该方法在大幅修正空间结构细节的同时，完美保留了网络学习到的多光谱振幅特征，避免了传统融合方法中常见的光谱失真问题。
Smart Images
Figure CN122887731A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image processing and computer vision, specifically to a panchromatic sharpening method based on frequency domain phase adaptive correction. Background Technology
[0002] Currently, panchromatic sharpening is a fundamental and crucial technique in remote sensing image processing. Its main purpose is to fuse a panchromatic image with high spatial resolution with a multispectral image with high spectral resolution but lower spatial resolution, thereby reconstructing a multispectral image that simultaneously possesses high spatial and spectral resolution. High-quality panchromatic sharpening results are an important prerequisite for many subsequent remote sensing applications (such as land cover classification, target recognition, and precise mapping).
[0003] Early panchromatic sharpening methods mainly included traditional approaches such as component substitution, multi-resolution analysis, and variational optimization. While these methods achieved some success, they relied on hand-designed prior assumptions or physical models, making it difficult to fully represent the complex nonlinear mapping relationship between panchromatic and multispectral images. This often led to spectral distortion or loss of spatial detail in the fusion results. In recent years, deep learning technology has made significant progress in the field of multimodal image fusion. Among them, generative models, with their powerful data distribution modeling capabilities and excellent detail generation performance, have become a research hotspot in panchromatic sharpening. However, existing generative techniques still have the following significant drawbacks:
[0004] First, conventional diffusion or flow matching models heavily rely on iterative numerical sampling during the inference phase. This iterative generation mechanism, requiring dozens or even hundreds of network function evaluations, leads to extremely high computational resource consumption and inference latency, making it difficult to meet the demands of fast, real-time processing of large-scale images in practical remote sensing applications. Second, to address the inference efficiency issue, recent research has begun exploring single-step generation frameworks. However, while reducing the number of generation steps, it often requires relying on large Transformer networks with a large number of parameters to maintain generation quality, which further increases the computational burden. If lightweight networks are used for single-step generation, the spatial structure of intermediate feature representations will gradually degrade as the network deepens due to the inevitable repetitive convolutions, nonlinear activation functions, and hierarchical downsampling operations in deep neural networks, resulting in phenomena such as spatial smoothing, blurred boundaries, and structural shifts.
[0005] More importantly, analyzing the aforementioned spatial degradation phenomenon from a frequency domain perspective reveals it to be essentially a "phase shift" problem accumulated during the propagation of deep features. According to Fourier theory, the spatial geometric positioning and structural details of an image are primarily determined by the phase components in the frequency domain. Existing panchromatic sharpening methods typically perform feature fusion directly in the spatial or channel domains, lacking explicit modeling and constraint mechanisms for frequency domain phase distortion within deep networks. They cannot fully utilize the inherent high-resolution geometry of panchromatic images as reliable "anchor points" to correct this phase shift, and they also struggle to perfectly maintain the amplitude that determines spectral features when correcting spatial structure.
[0006] In summary, there is an urgent need in the field for a method that can operate within a single-step generation framework and explicitly overcome the structural degradation problem in the feature propagation of deep networks, so as to achieve panchromatic sharpened image reconstruction with both high spatial structure fidelity and spectral consistency while ensuring extremely high inference efficiency. Summary of the Invention
[0007] The main objective of this invention is to address the numerous problems existing in the current pancolor sharpening methods used in remote sensing image processing, and to provide a pancolor sharpening method based on frequency domain phase adaptive correction.
[0008] The panchromatic sharpening method based on frequency domain phase adaptive correction provided by this invention includes the following steps:
[0009] Step 1: Obtain the upsampled low-resolution multispectral image (LRMS) and its corresponding high-resolution panchromatic image (PAN) from the multi-source remote sensing data;
[0010] Step 2: Extract the conditional features of the multispectral and panchromatic images using a preset conditional encoder, and construct multi-scale panchromatic frequency domain anchors by performing frequency domain decomposition based on the panchromatic image using an anchor encoder.
[0011] Step 3: The initial state containing noise is used as the starting point of the generation process. Combined with time step embedding and conditional features, it is input into the pre-trained frequency domain adaptive generation network for multi-scale feature propagation.
[0012] Step 4: During the multi-scale feature propagation process, the frequency domain correction module is applied to adaptively correct the phase of the intermediate feature representation in the frequency domain using multi-scale panchromatic frequency domain anchor points, while retaining the amplitude characteristics of the intermediate feature representation.
[0013] Step 5: Based on the features after adaptive correction in the frequency domain at each scale, directly predict the manifold state of the target image and output a high-resolution multispectral image after pancolor sharpening.
[0014] In step 3, the frequency domain adaptive generation network is a U-Net network architecture based on an encoder-decoder structure. It includes residual blocks and frequency domain modules inserted before each downsampling and upsampling operation. Frequency domain processing uses a frequency domain correction module to correct the phase of intermediate features. Conditional features and time step embedding are injected into the network through adaptive layer normalization to modulate the generation process. The frequency domain adaptive generation network is based on a single-step flow matching average velocity generation framework, and its training loss function is the 1-norm error loss in the instantaneous velocity space.
[0015] The specific steps for adaptively correcting the phase of the intermediate feature representation in the frequency domain using the multi-scale panchromatic frequency domain anchor points in step 4 are as follows:
[0016] (i) The intermediate feature F and its corresponding multi-scale panchromatic frequency domain anchor P in the feature propagation process are transformed to the frequency domain by two-dimensional real-valued fast Fourier transform; the amplitude and phase components are extracted in the frequency domain respectively, and its frequency domain representation is as follows:
[0017] ;
[0018] The index i corresponds to the intermediate feature and the full-color frequency domain anchor point, respectively, where (u,v) are the frequency domain coordinates, A is the amplitude, and φ is the phase.
[0019] (II) Calculate the structure energy mask, specifically including: using the discrete Laplace transfer function to calculate the contribution of each frequency component to the spatial variation, and extracting the frequency domain structure energy E corresponding to the multi-scale panchromatic frequency domain anchor point P, as follows:
[0020] ;
[0021] in, The discrete Laplace transfer function is as follows:
[0022] H and W are the feature map dimensions;
[0023] The frequency domain structure energy is logarithmically transformed and normalized to a global mean. Then, a parameterized activation function is used to map the normalized energy to the target interval to generate a structure energy mask, as follows:
[0024] ;
[0025] Where γ and σ are the scaling factor and the Sigmoid function, respectively.
[0026] (III) Calculate the coherence threshold, specifically including: calculating the cyclic phase difference between the intermediate feature and the multi-scale panchromatic frequency domain anchor point, as shown in the following formula:
[0027] ;
[0028] Phase coherence is evaluated on the cyclic manifold using the nonnormalized Von-Mises kernel function, generating the coherence threshold C:
[0029] ;
[0030] Where K is a learnable parameter that controls the coherent response;
[0031] (iv) Complex domain phase transfer and image reconstruction, specifically including: calculating the adaptive frequency domain correction weight W by combining the Hadamard product of the structure energy mask and the Von-Mises coherence threshold:
[0032] ;
[0033] By using adaptive frequency domain correction weights to map the frequency domain phase onto the complex unit circle and performing weighted interpolation, the following composite variable is obtained:
[0034] ;
[0035] The principal complex angles of the composite variable are further extracted as the corrected characteristic phases as follows:
[0036] ;
[0037] Arg() is used to return the principal phase angle; combined with the amplitude component extracted from the intermediate features. With the corrected characteristic phase The feature map after reconstruction and correction using inverse Fourier transform is as follows:
[0038] .
[0039] The pancolor sharpening system used in the above-mentioned pancolor sharpening method based on frequency domain phase adaptive correction includes:
[0040] The data acquisition module is used to acquire the multispectral image to be processed and its corresponding panchromatic image;
[0041] The feature extraction module is used to extract conditional features of multispectral and panchromatic images through a preset conditional encoder, and to construct multi-scale panchromatic frequency domain anchors by performing frequency domain decomposition based on the panchromatic image through an anchor encoder.
[0042] The feature propagation module is used to take the initial state containing noise as the starting point of the generation process, and combine the time step embedding with the conditional features, and input them into the pre-trained frequency domain adaptive generation network for multi-scale feature propagation.
[0043] The frequency domain correction module is used in the process of multi-scale feature propagation. It uses multi-scale panchromatic frequency domain anchors to adaptively correct the phase of the intermediate feature representation in the frequency domain, while preserving the amplitude characteristics of the intermediate feature representation.
[0044] The manifold prediction module is used to directly predict the manifold state of the target image based on the features after adaptive correction in the frequency domain at each scale, and outputs a high-resolution multispectral image after pancolor sharpening.
[0045] The beneficial effects of this invention are:
[0046] The panchromatic sharpening method based on frequency domain phase adaptive correction provided by this invention directly predicts the image manifold state using a single-step flow matching generation framework, avoiding the time-consuming iterative sampling process of traditional generative models. Generation is completed with only a single network function evaluation, significantly reducing computational resource consumption and inference latency. This invention creatively reveals from a frequency domain perspective that the essence of spatial degradation in deep networks is a "phase shift" problem, and proposes using the natural high-frequency geometric structure of panchromatic images to construct frequency domain anchors. Adaptive phase correction is performed using structural energy masks and Von-Mises coherence thresholds, effectively curbing geometric distortion of deep features during network propagation. In the frequency domain adaptive correction process, this invention cleverly employs a "phase interpolation, amplitude preservation" strategy. Since the spectral information of an image is mainly determined by frequency domain amplitude, this method perfectly preserves the multispectral amplitude features learned by the network while significantly correcting spatial structural details, avoiding the spectral distortion problem common in traditional fusion methods. Attached Figure Description
[0047] Figure 1 This is a flowchart illustrating the workflow and dual encoder structure design of the full-color sharpening method described in this invention.
[0048] Figure 2 This is a schematic diagram of the frequency domain adaptive generation network described in this invention.
[0049] Figure 3 This is a schematic diagram of the frequency domain correction module described in this invention. Detailed Implementation
[0050] Please see Figures 1 to 3 As shown:
[0051] The panchromatic sharpening method based on frequency domain phase adaptive correction provided by this invention includes the following steps:
[0052] Step 1: Obtain multi-source image data.
[0053] Acquire the low-resolution multispectral (LRMS) image to be processed and upsample it to the same spatial size as the corresponding high-resolution panchromatic (PAN) image.
[0054] The upsampled multispectral image is designed as I. MS The panchromatic image is I PAN .
[0055] Step 2: Multi-scale feature extraction and full-color frequency domain anchor point construction.
[0056] See Figure 1 This invention deploys a dual encoder structure to provide supplementary prior information.
[0057] On the one hand, a conditional encoder is used to extract the stitched input of multispectral and panchromatic images to obtain multi-scale conditional features C. l (l represents the network layer).
[0058] On the other hand, an anchor encoder is used to construct a spatial pyramid from the panchromatic image, and absolute geometric references are extracted through frequency domain decomposition to output multi-scale panchromatic frequency domain anchor points P. l These anchor points have strict spatial alignment with intermediate features at each level.
[0059] Step 3: Multi-scale feature propagation under the single-step flow matching framework.
[0060] The initial state containing noise is used as the starting point of the generation process (let the state be y). t In the frequency domain adaptive generative network (based on the symmetric U-Net architecture design, see reference...), Figure 2 In the process, the current time step t and the target time step r are combined, and the time step is embedded with the conditional features and injected into the network residual block through adaptive layer normalization to modulate the forward generation process of the network.
[0061] Step 4: Perform adaptive frequency domain correction.
[0062] Deep networks can experience accumulated phase drift during downsampling and multi-layer nonlinear operations. This embodiment inserts a frequency domain module (see reference) before each encoder and decoder in the frequency domain adaptive generation network performs spatial resolution transformation operations (such as pooling or upsampling). Figure 2 ).
[0063] See Figure 3 The processing flow of the frequency domain correction module is as follows:
[0064] First, the intermediate feature F and its corresponding multi-scale panchromatic frequency domain anchor point P in the feature propagation process are transformed to the frequency domain using a two-dimensional fast Fourier transform (FFT2D):
[0065] ,
[0066] Where (u,v) are frequency domain coordinates, A is amplitude, and φ is phase.
[0067] Next, the structural energy mask M is calculated to guide the network to focus on structurally significant regions. The contribution of each frequency component to spatial variation (i.e., Dirichlet energy) is calculated using the discrete Laplace transfer function, and the frequency domain structural energy corresponding to the multi-scale panchromatic frequency domain anchor point P is extracted.
[0068] ,
[0069] in, The discrete Laplace transfer function is, i.e.
[0070] H and W are the spatial dimensions of the feature image;
[0071] The frequency domain structure energy E is logarithmically transformed and normalized to a global mean to obtain the normalized energy: ,
[0072] Where μ is the global expected mean of the structure energy after logarithmic transformation; the normalized energy is mapped to the target interval through a parameterized activation function to generate a structure energy mask:
[0073] ,
[0074] Where γ is a learnable scaling factor and σ is the Sigmoid function.
[0075] Then, the Von-Mises coherence threshold C is calculated to ensure that correction is performed only in regions where the phase exhibits a consistent trend. The cyclic phase difference between the intermediate feature and the panchromatic frequency domain anchor point is then calculated.
[0076] ,
[0077] Phase consistency is evaluated on the cyclic manifold:
[0078] ,
[0079] Where K is a learnable parameter that controls the coherent response.
[0080] Finally, complex domain phase transfer and image reconstruction are performed. Adaptive frequency domain correction weights are obtained using the Hadamard product of the structure energy mask and the coherence threshold.
[0081] .
[0082] To avoid phase boundary discontinuities caused by direct interpolation in Euclidean space, this embodiment maps the frequency domain phase onto the complex unit circle for weighted interpolation:
[0083] ,
[0084] Extract the principal complex angles of the composite variable as the corrected characteristic phase.
[0085] .
[0086] Finally, by combining the initially saved intermediate feature amplitude components, the spatial feature map is reconstructed using inverse fast Fourier transform (IFFT2D):
[0087] ,
[0088] This mechanism perfectly preserves the original spectral features (amplitude) while aligning the spatial structure (phase).
[0089] Step 5: Directly predict the manifold state of the target image.
[0090] After all levels of frequency domain adaptive correction, the output of the frequency domain adaptive generator network directly provides the manifold state of the image based on single-step prediction (assuming the inference time step t and r are 1 and 0 respectively), and accumulates it with the input, finally outputting a panchromatic sharpened image with both high spatial resolution and high spectral resolution.
[0091] During the model training phase, this embodiment establishes a goal-oriented approach based on a single-step flow matching framework. The manifold state predicted by the network is converted into predicted instantaneous velocity, and the 1-norm loss in the instantaneous velocity space is used as the loss function for parameter optimization, constraining the model's motion trajectory during single-step generation, enabling the model to converge quickly and stably.
[0092] Based on the same inventive concept, this embodiment also provides a pancolor sharpening system based on frequency domain adaptive correction, which includes virtual entity functional modules, namely: a data acquisition module, a feature extraction module, a feature propagation module, a frequency domain correction module, and a manifold prediction module. The specific functions and workflows of the above modules correspond one-to-one with the steps in the above method embodiments, and will not be repeated here.
Claims
1. A panchromatic sharpening method based on frequency domain phase adaptive correction, characterized in that: The method includes the following steps: Step 1: Obtain the upsampled low-resolution multispectral image (LRMS) and its corresponding high-resolution panchromatic image (PAN) from the multi-source remote sensing data; Step 2: Extract the conditional features of the multispectral and panchromatic images using a preset conditional encoder, and construct multi-scale panchromatic frequency domain anchors by performing frequency domain decomposition based on the panchromatic image using an anchor encoder. Step 3: The initial state containing noise is used as the starting point of the generation process. Combined with time step embedding and conditional features, it is input into the pre-trained frequency domain adaptive generation network for multi-scale feature propagation. Step 4: During the multi-scale feature propagation process, the frequency domain correction module is applied to adaptively correct the phase of the intermediate feature representation in the frequency domain using multi-scale panchromatic frequency domain anchor points, while retaining the amplitude characteristics of the intermediate feature representation. Step 5: Based on the features after adaptive correction in the frequency domain at each scale, directly predict the manifold state of the target image and output a high-resolution multispectral image after pancolor sharpening.
2. The panchromatic sharpening method based on frequency domain phase adaptive correction according to claim 1, characterized in that: In step 3, the frequency domain adaptive generation network is a U-Net network architecture based on an encoder-decoder structure, which includes residual blocks and frequency domain modules inserted before each downsampling and upsampling operation; the frequency domain correction block uses the frequency domain correction module to perform structural correction on the intermediate features; Conditional features and time-step embeddings are injected into the network through adaptive layer normalization to modulate the generation process; The frequency domain adaptive generative network is based on the single-step flow matching average velocity generation framework, and its training loss function is the 1-norm error loss in the instantaneous velocity space.
3. The panchromatic sharpening method based on frequency domain phase adaptive correction according to claim 1, characterized in that: The specific steps for adaptively correcting the phase of the intermediate feature representation in the frequency domain using the multi-scale panchromatic frequency domain anchor points in step 4 are as follows: (i) The intermediate feature F and its corresponding multi-scale panchromatic frequency domain anchor point P in the feature propagation process are transformed to the frequency domain by two-dimensional real-valued fast Fourier transform; the amplitude and phase components are extracted in the frequency domain respectively, and its frequency domain representation is as follows: ； Where index i corresponds to the intermediate feature and the full-color frequency domain anchor point respectively, where (u,v) are the frequency domain coordinates, A is the amplitude, and φ is the phase; (II) Calculate the structure energy mask, specifically including: using the discrete Laplace transfer function to calculate the contribution of each frequency component to the spatial variation, and extracting the frequency domain structure energy E corresponding to the multi-scale panchromatic frequency domain anchor point P, as follows: ； in, The discrete Laplace transfer function is as follows: H and W are the feature map dimensions; The frequency domain structure energy is logarithmically transformed and normalized to a global mean. Then, a parameterized activation function is used to map the normalized energy to the target interval to generate a structure energy mask, as follows: ； Where γ and σ are the scaling factor and the Sigmoid function, respectively; (III) Calculate the coherence threshold, specifically including: calculating the cyclic phase difference between the intermediate feature and the multi-scale panchromatic frequency domain anchor point, as shown in the following formula: ； Phase coherence is evaluated on the cyclic manifold using the nonnormalized Von-Mises kernel function, generating the coherence threshold C: ； Where K is a learnable parameter that controls the coherent response; (iv) Complex domain phase transfer and image reconstruction, specifically including: calculating the adaptive frequency domain correction weight W by combining the Hadamard product of the structure energy mask and the Von-Mises coherence threshold: ； By using adaptive frequency domain correction weights to map the frequency domain phase onto the complex unit circle and performing weighted interpolation, the following composite variable is obtained: ； The principal complex angles of the composite variable are further extracted as the corrected characteristic phases as follows: ； Arg() is used to return the principal phase angle; combined with the amplitude component extracted from the intermediate features. With the corrected characteristic phase The feature map after reconstruction and correction using inverse Fourier transform is as follows: 。 4. The panchromatic sharpening method based on frequency domain phase adaptive correction according to claim 1, characterized in that: The full-color sharpening system includes: The data acquisition module is used to acquire the multispectral image to be processed and its corresponding panchromatic image; The feature extraction module is used to extract conditional features of multispectral and panchromatic images through a preset conditional encoder, and to construct multi-scale panchromatic frequency domain anchors by performing frequency domain decomposition based on the panchromatic image through an anchor encoder. The feature propagation module is used to take the initial state containing noise as the starting point of the generation process, and combine the time step embedding with the conditional features, and input them into the pre-trained frequency domain adaptive generation network for multi-scale feature propagation. The frequency domain correction module is used in the process of multi-scale feature propagation. It uses multi-scale panchromatic frequency domain anchors to adaptively correct the phase of the intermediate feature representation in the frequency domain, while preserving the amplitude characteristics of the intermediate feature representation. The manifold prediction module is used to directly predict the manifold state of the target image based on the features after adaptive correction in the frequency domain at each scale, and outputs a high-resolution multispectral image after pancolor sharpening.