A multispectral and panchromatic image fusion method based on structure-to-detail asynchronous diffusion refinement

Abstract

The application relates to the technical field of remote sensing image processing, in particular to a multispectral and panchromatic image fusion method based on structure-to-detail asynchronous diffusion refinement, which comprises the following steps: acquiring a low-resolution multispectral image and a high-resolution panchromatic image, and performing up-sampling on the low-resolution multispectral image to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image; in a first noise interval, a guide model is used to recover the overall spatial structure of a fusion image and complete structure reconstruction; in a second noise interval, through an asynchronous transmission mechanism, the leading information of a generation process is gradually transferred from a spatial structure stream based on the high-resolution panchromatic image to a spectral residual stream based on multispectral residuals; in a third noise interval, spectral compensation and local detail correction are performed on the fusion image to obtain a high-resolution multispectral fusion image. The method can realize a more coordinated, stable and efficient image fusion effect in spatial structure recovery and spectral detail refinement.

Description

Technical Field

[0001] The embodiments of this application relate to the field of remote sensing image processing technology, and in particular to a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail. Background Technology

[0002] Multispectral and panchromatic image fusion, also known as panchromatic sharpening, is a key task in remote sensing image processing. Multispectral images possess rich spectral information, but their spatial resolution is typically low due to limitations in sensor imaging conditions and energy allocation. Panchromatic images, on the other hand, offer higher spatial resolution, providing clear edges, textures, and geometric structures, but only contain broad-band grayscale observations, lacking complete multispectral information. Therefore, effectively fusing the spectral information from low-resolution multispectral images with the spatial structure information from high-resolution panchromatic images to generate high-resolution multispectral images with both high spatial resolution and high spectral fidelity is a crucial foundation for tasks such as remote sensing image interpretation, target recognition, land cover classification, change detection, and refined mapping.

[0003] Existing panchromatic sharpening methods mainly include traditional methods and deep learning methods. Traditional methods, such as component substitution, multi-resolution analysis, and variational optimization, while each with its own focus, generally suffer from problems such as reliance on manual design and susceptibility to spectral distortion or spatial artifacts. Among deep learning methods, convolutional neural network-based methods are limited by local receptive fields and struggle to model long-range dependencies; Transformer-based methods, while possessing global modeling capabilities, suffer from high computational costs and require large amounts of data; generative adversarial network-based methods exhibit unstable training and are prone to producing pseudo-textures; and existing diffusion model-based methods typically employ simultaneous conditional fusion, using both panchromatic and multispectral images as conditions in the same denoising stage. This fails to effectively distinguish the different roles of the two in spatial structure reconstruction and spectral detail refinement, leading to mutual interference between spatial detail enhancement and spectral preservation, making it difficult to achieve an ideal balance. Summary of the Invention

[0004] In view of this, embodiments of this application propose a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail. This method aims to solve the technical problem in existing synchronous diffusion fusion methods where the spatial structure constraints of panchromatic images and the spectral information constraints of multispectral images are difficult to coordinate, leading to a conflict between spatial detail enhancement and spectral fidelity. The goal is to achieve a more coordinated, stable, and efficient image fusion effect in terms of spatial structure restoration and spectral detail refinement.

[0005] To achieve the above objectives, embodiments of this application propose a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail, the method comprising the following steps: Acquire low-resolution multispectral images and high-resolution panchromatic images of the same region, and upsample the low-resolution multispectral image to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image; In the first noise interval of the reverse denoising process of the diffusion model, the spatial structure information in the high-resolution panchromatic image is used to guide the model to recover the overall spatial structure of the fused image and complete the structure reconstruction. In the second noise interval of the reverse denoising process, the dominant information of the generation process is gradually transferred from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual through an asynchronous transmission mechanism, thus completing a smooth transition of information dominance. In the third noise region of the reverse denoising process, based on the spectral residual flow and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, spectral compensation and local detail correction are performed on the fused image to complete the spectral refinement and thus obtain the high-resolution multispectral fused image. Output high-resolution multispectral fusion images.

[0006] To achieve the above objectives, embodiments of this application also propose a multispectral and panchromatic image fusion system based on asynchronous diffusion refinement from structure to detail, the system comprising: The input preprocessing module is used to acquire low-resolution multispectral images and high-resolution panchromatic images of the same region, and to upsample the low-resolution multispectral images to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image. The structure reconstruction module is used in the first noise interval of the reverse denoising process of the diffusion model to guide the model to recover the overall spatial structure of the fused image by utilizing the spatial structure information in the high-resolution panchromatic image, thus completing the structure reconstruction. The asynchronous transmission module is used in the second noise interval of the reverse denoising process to gradually transfer the dominant information in the generation process from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual through the asynchronous transmission mechanism, thus completing a smooth transition of information dominance. The spectral refinement and fusion module is used in the third noise region of the reverse denoising process. Based on the spectral residual flow, and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, it performs spectral compensation and local detail correction on the fused image to complete the spectral refinement and obtain the high-resolution multispectral fused image. Image output module, used to output high-resolution multispectral fused images.

[0007] In one implementation, the spectral refinement and fusion module is specifically used for: A residual prediction method is adopted, in which a denoising network predicts the supplementary residual of the high-resolution multispectral fusion image relative to the initial multispectral estimated image; The supplemented residual is added to the initial multispectral estimated image to achieve spectral compensation and local detail correction of the fused image, resulting in the final high-resolution multispectral fused image.

[0008] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement the multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail as described above.

[0009] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail as described above.

[0010] This application proposes a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail. It acquires a low-resolution multispectral image and a high-resolution panchromatic image of the same region, and upsamples the low-resolution multispectral image to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image. In the first noise interval of the inverse denoising process of the diffusion model, the spatial structure information in the high-resolution panchromatic image is used to guide the model to recover the overall spatial structure of the fused image, completing the structure reconstruction. In the second noise interval of the inverse denoising process, an asynchronous transmission mechanism is used to gradually transfer the dominant information in the generation process from the spatial structure flow based on the high-resolution panchromatic image to the multispectral residual. The spectral residual flow smoothly transitions information dominance. In the third noise region of the reverse denoising process, based on the spectral residual flow and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, spectral compensation and local detail correction are performed on the fused image to complete spectral refinement and obtain a high-resolution multispectral fused image. The high-resolution multispectral fused image is then output. This scheme can solve the technical problem in existing synchronous diffusion fusion methods where the spatial structure constraints of the panchromatic image and the spectral information constraints of the multispectral image are difficult to coordinate, leading to a conflict between spatial detail enhancement and spectral fidelity. This achieves a more coordinated, stable, and efficient image fusion effect in terms of spatial structure restoration and spectral detail refinement.

[0011] Optionally, in the first noise interval of the reverse denoising process of the diffusion model, the spatial structure information in the high-resolution panchromatic image is used to guide the model to recover the overall spatial structure of the fused image and complete the structure reconstruction. This includes: performing low-pass and high-pass decomposition on the high-resolution panchromatic image to extract its high-frequency spatial detail components; calculating the correlation information between the high-resolution panchromatic image and the initial multispectral estimated image to generate correlation weights for controlling the injection intensity of the high-frequency spatial detail components into different spectral bands; and using the high-frequency spatial detail components and correlation weights as structural prior inputs to the diffusion network to guide the model to generate an initial high-resolution multispectral representation that conforms to the spatial structure of the high-resolution panchromatic image in the first noise interval.

[0012] Optionally, in the second noise interval of the reverse denoising process, an asynchronous transmission mechanism is used to gradually transfer the dominant information of the generation process from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual, completing a smooth transition of information dominance, including: Define spatial structure flow and spectral residual flow; where, spatial structure flow... and spectral residual current The calculation formulas include: ; ; in, Represents a high-resolution panchromatic image; This represents the initial multispectral estimated image; Represents an ideal high-resolution multispectral image; This represents the low-pass filter function; Represents the correlation function; This indicates missing details and spectral correction terms; Through an asynchronous transmission mechanism, within the second noise interval, the smooth propagation of the spatial structure flow and spectral residual flow in the time dimension is achieved using recursive coefficients and control. Among these, the recursive coefficients... and satisfy: ; ; in, The noise scheduling parameters are preset for the diffusion model.

[0013] Alternatively, the forward process of the asynchronous transmission mechanism can be represented as: ; in, For the first The intermediate state of a time step. For cumulative noise scheduling parameters, and These are the start and end time steps of the second noise interval, respectively. This is the noise term.

[0014] Alternatively, the reverse process of the asynchronous transmission mechanism can be represented as: ; ; in, Indicates the first The intermediate state of a time step. This represents the residual information predicted by the denoising network, used to predict the spectral residual flux. .

[0015] Optionally, in the third noise interval of the reverse denoising process, based on the spectral residual flow and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, spectral compensation and local detail correction are performed on the fused image to complete spectral refinement and obtain a high-resolution multispectral fused image. This includes: using a residual prediction method, the denoising network predicts the supplementary residual of the high-resolution multispectral fused image relative to the initial multispectral estimated image; adding the supplementary residual to the initial multispectral estimated image to achieve spectral compensation and local detail correction of the fused image, thereby obtaining the final high-resolution multispectral fused image.

[0016] Optionally, the denoising network of the diffusion model includes: a time-text adaptive feature modulation module and a Top-K sparse cross-attention module; the time-text adaptive feature modulation module is used to modulate the network features according to the current time step and sensor text prompts; the Top-K sparse cross-attention module is used to selectively inject high-frequency spatial information of the panchromatic image through a Top-K filtering mechanism, using multispectral features as queries and panchromatic features as keys and values. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0018] Figure 1 This is a flowchart of a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail, provided in one embodiment of this application; Figure 2This is a schematic diagram of a three-stage asynchronous diffusion process provided in one embodiment of this application; Figure 3 This is a schematic diagram of a denoising network structure provided in one embodiment of this application; Figure 4 This is a schematic diagram of Top-K sparse cross-attention provided in one embodiment of this application; Figure 5 This is a schematic diagram of the overall framework of the asynchronous diffusion panchromatic sharpening method provided in one embodiment of this application; Figure 6 This is a schematic diagram of the structure of a multispectral and panchromatic image fusion system based on asynchronous diffusion refinement from structure to detail, provided in another embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0020] Multispectral and panchromatic image fusion, also known as panchromatic sharpening, is a key task in remote sensing image processing. Multispectral images possess rich spectral information, but their spatial resolution is typically low due to limitations in sensor imaging conditions and energy allocation. Panchromatic images, on the other hand, offer higher spatial resolution, providing clear edges, textures, and geometric structures, but only contain broad-band grayscale observations, lacking complete multispectral information. Therefore, effectively fusing the spectral information from low-resolution multispectral images with the spatial structure information from high-resolution panchromatic images to generate high-resolution multispectral images with both high spatial resolution and high spectral fidelity is a crucial foundation for tasks such as remote sensing image interpretation, target recognition, land cover classification, change detection, and refined mapping.

[0021] Existing panchromatic sharpening methods mainly include traditional methods and deep learning methods. Traditional methods, such as component substitution, multi-resolution analysis, and variational optimization, while each with its own focus, generally suffer from problems such as reliance on manual design and susceptibility to spectral distortion or spatial artifacts. Among deep learning methods, convolutional neural network-based methods are limited by local receptive fields and struggle to model long-range dependencies; Transformer-based methods, while possessing global modeling capabilities, suffer from high computational costs and require large amounts of data; generative adversarial network-based methods exhibit unstable training and are prone to producing pseudo-textures; and existing diffusion model-based methods typically employ simultaneous conditional fusion, using both panchromatic and multispectral images as conditions in the same denoising stage. This fails to effectively distinguish the different roles of the two in spatial structure reconstruction and spectral detail refinement, leading to mutual interference between spatial detail enhancement and spectral preservation, making it difficult to achieve an ideal balance.

[0022] In view of this, embodiments of this application propose a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail. This method aims to solve the technical problem in existing synchronous diffusion fusion methods where the spatial structure constraints of panchromatic images and the spectral information constraints of multispectral images are difficult to coordinate, leading to a conflict between spatial detail enhancement and spectral fidelity. The goal is to achieve a more coordinated, stable, and efficient image fusion effect in terms of spatial structure restoration and spectral detail refinement.

[0023] Specifically, embodiments of this application provide an asynchronous diffusion panchromatic sharpening method, including an input preprocessing module, a structure reconstruction stage, an asynchronous transmission stage, a spectral refinement stage, a denoising network module, and a diffusion sampling module. The input preprocessing module upsamples a low-resolution multispectral (LRMS) image and aligns it to a high-resolution panchromatic (PAN) image in size. The structure reconstruction stage recovers the overall spatial contour of the fused image using the high-frequency structure of the panchromatic image in the early diffusion stage. The asynchronous transmission stage bridges the spatial structure flow and the spectral residual flow within an intermediate time interval. The spectral refinement stage performs spectral compensation and detail correction in the later diffusion stage. The denoising network module predicts features based on noise features, the high-resolution panchromatic image, the upsampled LRMS (Multispectral) image, temporal embedding, and textual prompts when residual or clean features need to be predicted. The diffusion sampling module performs inverse denoising at predetermined time steps and outputs a high-resolution multispectral fused image.

[0024] It should be noted that the upsampled MS band specifically refers to the various spectral bands that are obtained by upsampling a low-resolution multispectral image and have the same size as the panchromatic image.

[0025] By leveraging the characteristics of different image restoration targets at different time stages of the diffusion model, the spatial structure restoration, cross-modal residual transformation, and detail spectral refinement in pancolor sharpening are modeled in stages. At the same time, the asynchronous dual-stream mechanism explicitly distinguishes between PAN high-frequency structural information and MS spectral residual information, avoiding the problem of mutual interference between spatial details and spectral information in traditional synchronous fusion methods.

[0026] One embodiment of this application proposes a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail, applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for description. The implementation details of the multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail proposed in this embodiment will be described in detail below. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0027] The specific process of the multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Acquire a low-resolution multispectral image and a high-resolution panchromatic image of the same region, and upsample the low-resolution multispectral image to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image.

[0028] Specifically, firstly, accurately registered low-resolution multispectral images and PAN images for the same geographic region are acquired. Since the spatial resolution of the LRMS image is lower than that of the PAN image, an upsampling operation is required on the LRMS image. For example, bicubic interpolation or nearest-neighbor interpolation methods are used to ensure that its spatial dimensions are consistent with the PAN image, thus obtaining an initial multispectral estimation image. This initial image mainly contains reliable spectral information and a relatively coarse spatial structure, which can serve as the basis for subsequent structure reconstruction and residual estimation.

[0029] Simultaneously, to extract spatial structure information from the PAN image, low-pass and high-pass decomposition can be performed. For example, the low-frequency components of the PAN image can be obtained through Gaussian filtering or mean filtering, and then the low-frequency components can be subtracted from the original PAN image to obtain its high-frequency spatial detail components. These high-frequency spatial detail components are used to reflect geometric structure information such as edges, textures, and contours.

[0030] Therefore, the correlation information between the PAN and the upsampled LRMS is further calculated to estimate the strength of transferable high-frequency details in each region and band. Inspired by traditional multi-resolution analysis, the residual between the ideal high-resolution multispectral image and the upsampled low-resolution multispectral image is considered to be approximately represented by the PAN high-frequency information modulated by the correlation coefficient. Unlike traditional methods, instead of directly using a fixed high-frequency injection formula to output the results, this residual relationship is embedded in the diffusion generation process, allowing the diffusion network to gradually learn the residual mapping during multi-stage denoising.

[0031] Step 102: In the first noise interval of the reverse denoising process of the diffusion model, the spatial structure information in the high-resolution panchromatic image is used to guide the model to restore the overall spatial structure of the fused image and complete the structure reconstruction.

[0032] For example, the denoising network of the diffusion model includes: a time-text adaptive feature modulation module and a Top-K sparse cross-attention module; the time-text adaptive feature modulation module is used to modulate the network features according to the current time step and sensor text prompts; the Top-K sparse cross-attention module is used to selectively inject high-frequency spatial information of the panchromatic image with multispectral features as queries and panchromatic features as keys and values ​​through a Top-K filtering mechanism.

[0033] In one possible embodiment, step 102 includes: performing low-pass and high-pass decomposition on the high-resolution panchromatic image to extract its high-frequency spatial detail components; calculating the correlation information between the high-resolution panchromatic image and the initial multispectral estimated image to generate correlation weights for controlling the injection intensity of the high-frequency spatial detail components into different spectral bands; and using the high-frequency spatial detail components and correlation weights as structural prior inputs to the diffusion network to guide the model to generate an initial high-resolution multispectral representation that conforms to the spatial structure of the high-resolution panchromatic image within a first noise interval.

[0034] For example, correlation weights are used to reflect the response relationship between PAN images and different MS bands to avoid using the same injection strategy for all bands, thereby reducing spectral shifts caused by high-frequency structure mismatches.

[0035] See Figure 2 In the reverse denoising process of the diffusion model (i.e., reverting to diffusion), the time step is... from arrive The interval is divided into three stages. The first stage is the structural reconstruction stage, corresponding to a time step interval of [missing information]. At this stage, the diffusion model is at a high noise level, and the main information of the image has been overwhelmed by noise. Therefore, the primary task of the model is to recover the global spatial structure and large-scale contours of the image.

[0036] Understandably, the core of this stage is to use the high-frequency spatial detail information of the PAN image as a strong structural prior to guide the denoising process.

[0037] Specifically, the PAN high-frequency features extracted in the above embodiments are used as conditional inputs into the denoising network. Meanwhile, considering the potential differences in spectral responses between different MS bands, to avoid spectral distortion caused by directly injecting PAN high-frequency information into all bands, this embodiment further calculates the correlation weights between the PAN and the upsampled LRMS image.

[0038] In a preferred embodiment, the relevance weight is calculated using the following formula: ; in, Represents a high-resolution panchromatic image Compared with the initial multispectral estimated image Local correlation weights between them; The first multispectral image represents the first multispectral image. One band, namely the upsampled MS band; Indicates in a local window Internal, high-resolution panchromatic image Compared with the initial multispectral estimated image No. Covariance of each band; Indicates in a local window Internal, high-resolution panchromatic image The variance.

[0039] Understandably, this correlation weight reflects the degree of correlation between the PAN and a specific MS band in a local region. The higher the correlation, the more high-frequency details the band can safely inherit from the PAN; conversely, it is necessary to inject with caution to avoid introducing irrelevant textures that could lead to spectral shifts.

[0040] In the structural reconstruction stage, the denoising network utilizes PAN high-frequency features and correlation weights to prioritize the recovery of the main structure, major edges, and large-scale geometry of the fused image, forming an intermediate result with a stable spatial skeleton. This stage does not pursue fine spectral details, laying a solid structural foundation for subsequent stages.

[0041] Step 103: In the second noise interval of the reverse denoising process, the dominant information of the generation process is gradually transferred from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual through an asynchronous transmission mechanism, thus completing a smooth transition of information dominance.

[0042] See Figure 2The second stage is the asynchronous transmission stage, with a corresponding time step interval of [missing information]. This stage, situated between structural reconstruction and spectral refinement, is a crucial step in realizing the innovation of this invention. Its core function is to resolve the conflict between spatial structural information and spectral residual information during the fusion process, achieving a smooth and gradual handover of dominance between the two types of information.

[0043] To achieve this goal, this method introduces two data streams: a spatial structure stream and a spectral residual stream. The spatial structure stream represents the high-frequency structural information extracted from the PAN image that can be used to enhance spatial resolution. The spectral residual stream represents the residual between the true high-resolution multispectral image (HRMS) and the upsampled LRMS image, i.e., the details and spectral corrections that need to be recovered.

[0044] In one possible embodiment, step 103 includes: in the second noise interval of the reverse denoising process, through an asynchronous transmission mechanism, gradually transferring the dominant information of the generation process from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual, completing a smooth transition of information dominance, including: Define spatial structure flow and spectral residual flow; where, spatial structure flow... and spectral residual current The calculation formulas include: ; ; in, Represents a high-resolution panchromatic image; This represents the initial multispectral estimated image; Represents an ideal high-resolution multispectral image; This represents the low-pass filter function; Represents the correlation function; It needs to be explained that spatial structure flow and spectral residual current superscript and It is used to distinguish between spatial structure flow and spectral residual flow.

[0045] Because the two data streams have different objectives, distributions, and convergence speeds, they cannot be simply replaced or superimposed at the same time step. Therefore, this method designs an asynchronous transmission mechanism, using recursive coefficients... and Controlling the propagation of spatially deterministic components of the spectrum.

[0046] Specifically, through an asynchronous transmission mechanism, within the second noise interval, the smooth propagation of the spatial structure flow and the spectral residual flow in the time dimension is achieved by utilizing recursive coefficients and controlling the spatial structure flow and spectral residual flow. ; ; The corresponding recursive formula is: ; ; in, The noise scheduling parameters are preset for the diffusion model.

[0047] For example, recursion coefficients and From time step Start moving towards time step Recursion, with initial values ​​for recursion. , .

[0048] The forward process in this stage can be summarized as follows: Within the time frame, the initial state of the spectral residual, the initial state of the PAN high-frequency structure, and the noise term are combined into an intermediate state. This allows the intermediate state to simultaneously contain information from both the spatial structure flow and the spectral residual flow. During the reverse process, the denoising network predicts the spectral residual information based on the current noise characteristics, the PAN image, the upsampled MS image, and temporal conditions, and combines this with the PAN high-frequency structure information to complete the stepwise transition.

[0049] At time step Within, the initial state of the spectral residual can be... Initial state of PAN high-frequency structure Together with the noise term, they form an intermediate state. This state contains information from both streams, and its proportion varies with... It changes with the changes. The forward process of the asynchronous transmission mechanism is represented as: ; in, For the first The intermediate state of a time step. For cumulative noise scheduling parameters, and These are the start and end time steps of the second noise interval, respectively. This is the noise term.

[0050] During the inference phase, a deterministic sampling strategy based on DDIM can be employed. The reverse process of the asynchronous transmission mechanism (i.e., used for inference) is represented as follows: ; ; in, Indicates the first The intermediate state of a time step. This represents the residual information predicted by the denoising network, used to predict the spectral residual flux. .

[0051] Through the above reverse process, the model can achieve the following at time step: from Decrease to In the process, the dominant information for noise reduction is gradually shifted from relying on... The spatial structure reconstruction, smoothly transferred to a system dependent on The spectral residual recovery enables a seamless transition of information dominance.

[0052] Step 104: In the third noise interval of the reverse denoising process, based on the spectral residual flow and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, spectral compensation and local detail correction are performed on the fused image to complete the spectral refinement and thus obtain the high-resolution multispectral fused image.

[0053] In one possible embodiment, step 104 includes: using a residual prediction method, a denoising network predicts the supplementary residual of the high-resolution multispectral fusion image relative to the initial multispectral estimated image; adding the supplementary residual to the initial multispectral estimated image to achieve spectral compensation and local detail correction of the fusion image, thereby obtaining the final high-resolution multispectral fusion image.

[0054] See Figure 2 The third stage is the spectral refinement stage, corresponding to a time step interval of [missing information]. After the first two stages, the overall spatial structure of the fused image has been stabilized, and the dominance of the generation process has shifted to the spectral residual stream. At this point, the model enters the low-noise range, focusing on restoring fine-grained texture, edge details, and local spectral consistency.

[0055] At this stage, the denoising network continues to use the current noise characteristics. PAN image Upsampled MS image and time step Given the condition, predict the spectral residual. This residual represents the recovery from the current state to a clean state. The required correction amount. Finally, the predicted residuals are added to the upsampled LRMS image to obtain the final fusion result: ; Understandably, this residual prediction method focuses the network's learning on missing spatial details and spectral correction terms, rather than generating the entire image from scratch, thus preserving as much detail as possible. The original spectral information in the PAN image improves spectral fidelity. Meanwhile, the high-frequency structural information of the PAN image can still be used as an auxiliary input to the network to guide the fine reconstruction of building edges, road textures, and other terrain features.

[0056] Step 105: Output a high-resolution multispectral fusion image.

[0057] For example, ultimately, after a complete backdiffusion sampling process, the model outputs a high-resolution multispectral fusion image that simultaneously has high spatial resolution (inherited from the PAN image) and high spectral fidelity (preserved from the MS image).

[0058] In one possible embodiment, the denoising network of the diffusion model includes: a time-text adaptive feature modulation module and a Top-K sparse cross-attention module; Among them, the time-text adaptive feature modulation module is used to modulate network features according to the current time step and sensor text prompts; the Top-K sparse cross-attention module is used to selectively inject high-frequency spatial information of the panchromatic image through a Top-K filtering mechanism, using multispectral features as queries and panchromatic features as keys and values.

[0059] For example, to support the three-stage process described in the above embodiments, such as Figure 3 As shown, this embodiment designs a dedicated denoising network module. The core components of this network include: a time-text adaptive feature modulation module and a Top-K sparse cross-attention module.

[0060] The time-text adaptive feature modulation module is used to receive time steps. The embedding vectors and optional sensor text cues (such as "WorldView-3") modulate the feature maps in the network. This allows the network to perceive the current diffusion stage and the imaging characteristics of different satellite sensors, thereby dynamically adjusting feature extraction and fusion strategies.

[0061] like Figure 4 As shown, the Top-K sparse cross-attention module 8 in this embodiment is used to filter panchromatic image spatial information related to multispectral reconstruction. This module uses MS features as the query Q and PAN features as the key K and value V. First, it calculates the similarity matrix between MS and PAN features. Then, according to the Top-K filtering strategy, it retains only attention responses that contribute highly and suppresses responses that contribute less or are irrelevant to the task. Finally, it obtains the modulated PAN features through weighted summation. Through this design, the model can focus on utilizing spatial edge, texture, and structural information related to multispectral image reconstruction, reducing the impact of irrelevant high-frequency noise, pseudo-textures, and erroneous spatial details on the fusion result, thereby reducing the risk of spectral distortion and spatial artifacts.

[0062] like Figure 5 As shown, Figure 5 This is a schematic diagram of the overall framework of the asynchronous diffusion panchromatic sharpening system provided in this embodiment. The system includes: a low-resolution multispectral image 1, a high-resolution panchromatic image 2, a high-pass extraction and correlation estimation module 3, a structure reconstruction stage 4, an asynchronous transmission stage 5, a spectral refinement stage 6, a denoising network module 7, and a high-resolution multispectral fusion result 9.

[0063] Low-resolution multispectral image 1 and high-resolution panchromatic image 2 are used as input data in this embodiment. Low-resolution multispectral image 1 primarily provides the spectral information and multi-band observation basis required for the fusion process, while high-resolution panchromatic image 2 primarily provides spatial structure, edge contours, and high-frequency texture information. High-resolution panchromatic image 2 is input to high-pass extraction and correlation estimation module 3. The low-frequency component of the panchromatic image is obtained through low-pass filtering, and the high-frequency structural information is obtained by subtracting the low-frequency component from the original panchromatic image. Simultaneously, based on the correlation between low-resolution multispectral image 1 and high-resolution panchromatic image 2, correlation weights used to constrain the high-frequency structure injection intensity are estimated, thereby constructing the spatial structure flow.

[0064] The structural reconstruction stage 4, asynchronous transmission stage 5, and spectral refinement stage 6 sequentially act on the diffusion reverse denoising process. Structural reconstruction stage 4 is used to recover the overall spatial contour and main edge structures of the fused image in the early stages of diffusion; asynchronous transmission stage 5 is used to coordinate the conversion relationship between the spatial structure flow and the spectral residual flow in the intermediate stages; and spectral refinement stage 6 is used to perform spectral compensation, local detail correction, and texture consistency enhancement in the later stages. The denoising network module 7 is used to predict the residual information required for the current denoising state in the asynchronous diffusion transition stage and the spectral detail refinement stage. Finally, after a complete reverse diffusion sampling process, a high-resolution multispectral fusion result is output.

[0065] This application proposes a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail. It acquires a low-resolution multispectral image and a high-resolution panchromatic image of the same region, and upsamples the low-resolution multispectral image to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image. In the first noise interval of the inverse denoising process of the diffusion model, the spatial structure information in the high-resolution panchromatic image is used to guide the model to recover the overall spatial structure of the fused image, completing the structure reconstruction. In the second noise interval of the inverse denoising process, an asynchronous transmission mechanism is used to gradually transfer the dominant information in the generation process from the spatial structure flow based on the high-resolution panchromatic image to the multispectral residual. The spectral residual flow smoothly transitions information dominance. In the third noise region of the reverse denoising process, based on the spectral residual flow and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, spectral compensation and local detail correction are performed on the fused image to complete spectral refinement and obtain a high-resolution multispectral fused image. The high-resolution multispectral fused image is then output. This scheme can solve the technical problem in existing synchronous diffusion fusion methods where the spatial structure constraints of the panchromatic image and the spectral information constraints of the multispectral image are difficult to coordinate, leading to a conflict between spatial detail enhancement and spectral fidelity. This achieves a more coordinated, stable, and efficient image fusion effect in terms of spatial structure restoration and spectral detail refinement.

[0066] Taking a remote sensing image of an urban area as an example, the input consists of an 8-band multispectral image downsampled by 4x and the corresponding high-resolution panchromatic image. Assuming a total of 100 steps, the method used in this embodiment is implemented as follows: In the structural reconstruction phase, during the first 60 steps of reverse denoising, the model successfully recovered macroscopic structures such as road networks, outlines of large buildings, and main channels of rivers using PAN images, forming a clear geometric skeleton.

[0067] In the asynchronous transmission phase, over the next 20 steps, the model gradually shifts its focus from structural information such as road edges to spectral residual information such as spectral differences in vegetation areas and subtle variations in the reflectance of roof materials. Because of the asynchronous transmission, the spectral characteristics of adjacent green spaces are not erroneously altered when restoring the road structure.

[0068] In the spectral refinement stage, in the final 20 steps, the model meticulously restored the detailed textures of vehicles, tree shadows, building windows, etc., and ensured the spectral proportions between different bands through residual learning, such as maintaining the high reflectivity of vegetation in the near-infrared band.

[0069] The resulting fused image achieves spatial resolution comparable to the PAN image, clearly showing the edges of ground features. In terms of spectral fidelity, its spectral curves across all bands are highly consistent with the original low-resolution multispectral image, avoiding the color shift and spectral distortion problems common in traditional methods. Experiments show that, compared to the diffusion model baseline method using synchronous conditional fusion, this method significantly improves peak signal-to-noise ratio, structural similarity, and spectral angle mapping, demonstrating a superior balance between spatial detail enhancement and spectral information preservation.

[0070] Compared with the prior art, the method provided by the embodiments of this application can achieve the following technical effects: First, this application divides the pancolor sharpening process into three stages: structural reconstruction, asynchronous diffusion transition, and spectral detail refinement. This makes the denoising process of the diffusion model match the reconstruction pattern from overall structure to local details in remote sensing image fusion, thereby improving the rationality and stability of the fusion process.

[0071] Second, this application sets an asynchronous diffusion transition stage, which allows the spatial structure flow and the spectral residual flow to interact gradually at different time steps. This can alleviate the conflict between the PAN spatial enhancement constraint and the MS spectral preservation constraint, thereby achieving a better balance between spatial detail enhancement and spectral fidelity.

[0072] Third, this application introduces a correlation-weighted or time-step modulation mechanism, enabling the contribution of PAN high-frequency information to different MS bands to adaptively adjust based on band correlation and the current diffusion stage. Compared to fixed injection weights, this approach is more suitable for different sensors, different spectral response ranges, and complex terrain scenarios.

[0073] Fourth, this application uses the Top-K sparse cross-attention module to filter PAN features with MS features, reducing the interference of irrelevant PAN high-frequency information on the fusion result, which helps to reduce the risk of false textures, false edges and local over-enhancement.

[0074] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0075] Another embodiment of this application proposes a multispectral and panchromatic image fusion system based on asynchronous diffusion refinement from structure to detail. The details of this multispectral and panchromatic image fusion system based on asynchronous diffusion refinement from structure to detail are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this example. Figure 6 This is a schematic diagram of the structure of a multispectral and panchromatic image fusion system based on asynchronous diffusion refinement from structure to detail proposed in this embodiment, including: The input preprocessing module 210 is used to acquire a low-resolution multispectral image and a high-resolution panchromatic image of the same region, and to upsample the low-resolution multispectral image to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image. The structure reconstruction module 220 is used to guide the model to recover the overall spatial structure of the fused image by utilizing the spatial structure information in the high-resolution panchromatic image in the first noise interval of the reverse denoising process of the diffusion model, thereby completing the structure reconstruction. The asynchronous transmission module 230 is used in the second noise interval of the reverse denoising process to gradually transfer the dominant information of the generation process from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual through the asynchronous transmission mechanism, thus completing a smooth transition of information dominance. The spectral refinement and fusion module 240 is used in the third noise region of the reverse denoising process. Based on the spectral residual flow, and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, it performs spectral compensation and local detail correction on the fused image to complete the spectral refinement and obtain the high-resolution multispectral fused image. Image output module 250 is used to output high-resolution multispectral fusion images.

[0076] In one implementation, the spectral refinement and fusion module 240 is specifically used for: A residual prediction method is adopted, in which a denoising network predicts the supplementary residual of the high-resolution multispectral fusion image relative to the initial multispectral estimated image; The supplemented residual is added to the initial multispectral estimated image to achieve spectral compensation and local detail correction of the fused image, resulting in the final high-resolution multispectral fused image.

[0077] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0078] It is worth mentioning that all modules and units involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units do not exist in this embodiment.

[0079] Another embodiment of this application provides an electronic device, such as Figure 7 As shown, it includes: a memory 31 and a processor 32. The memory 31 stores instructions that the processor 32 can execute. When the processor 32 is configured to execute the instructions, the electronic device can realize a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail as described in the above method embodiment.

[0080] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0081] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0082] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail as described in the above method embodiments.

[0083] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0084] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail, characterized in that, The method includes: Acquire low-resolution multispectral images and high-resolution panchromatic images of the same region, and upsample the low-resolution multispectral image to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image; In the first noise interval of the reverse denoising process of the diffusion model, the spatial structure information in the high-resolution panchromatic image is used to guide the model to recover the overall spatial structure of the fused image and complete the structure reconstruction. In the second noise interval of the reverse denoising process, the dominant information of the generation process is gradually transferred from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual through an asynchronous transmission mechanism, thus completing a smooth transition of information dominance. In the third noise region of the reverse denoising process, based on the spectral residual flow and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, spectral compensation and local detail correction are performed on the fused image to complete the spectral refinement and thus obtain the high-resolution multispectral fused image. Output high-resolution multispectral fusion images.

2. The method according to claim 1, characterized in that, In the first noise interval of the inverse denoising process of the diffusion model, spatial structure information in the high-resolution panchromatic image is used to guide the model to recover the overall spatial structure of the fused image, thus completing the structure reconstruction, including: Low-pass and high-pass decomposition is performed on high-resolution panchromatic images to extract their high-frequency spatial detail components. The correlation information between the high-resolution panchromatic image and the initial multispectral estimated image is calculated, and correlation weights are generated to control the intensity of high-frequency spatial detail components injected into different spectral bands. The high-frequency spatial detail components and correlation weights are used as structural prior inputs to the diffusion network to guide the model in generating an initial high-resolution multispectral representation that conforms to the spatial structure of a high-resolution panchromatic image within the first noise interval.

3. The method according to claim 1 or 2, characterized in that, In the second noise interval of the reverse denoising process, through an asynchronous transmission mechanism, the dominant information of the generation process is gradually transferred from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual, completing a smooth transition of information dominance, including: Define spatial structure flow and spectral residual flow; where, spatial structure flow... and spectral residual current The calculation formulas include: ； ； in, Represents a high-resolution panchromatic image; This represents the initial multispectral estimated image; Represents an ideal high-resolution multispectral image; This represents the low-pass filter function; Represents the correlation function; Through an asynchronous transmission mechanism, within the second noise interval, the smooth propagation of the spatial structure flow and spectral residual flow in the time dimension is achieved using recursive coefficients and control. Among these, the recursive coefficients... and satisfy: ； ； in, The noise scheduling parameters are preset for the diffusion model.

4. The method according to claim 3, characterized in that, The forward process of the asynchronous transmission mechanism is represented as follows: ； in, For the first The intermediate state of a time step. For cumulative noise scheduling parameters, and These are the start and end time steps of the second noise interval, respectively. This is the noise term.

5. The method according to claim 4, characterized in that, The reverse process of the asynchronous transmission mechanism is represented as follows: ； ； in, Indicates the first The intermediate state of a time step. This represents the residual information predicted by the denoising network, used to predict the spectral residual flux. .

6. The method according to claim 1, characterized in that, The third noise interval in the reverse denoising process, based on the spectral residual flow and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, performs spectral compensation and local detail correction on the fused image to complete spectral refinement, thereby obtaining a high-resolution multispectral fused image, including: A residual prediction method is adopted, in which a denoising network predicts the supplementary residual of the high-resolution multispectral fusion image relative to the initial multispectral estimated image; The supplemented residual is added to the initial multispectral estimated image to achieve spectral compensation and local detail correction of the fused image, resulting in the final high-resolution multispectral fused image.

7. The method according to claim 1, characterized in that, The denoising network of the diffusion model includes: a time-text adaptive feature modulation module and a Top-K sparse cross-attention module; The time-text adaptive feature modulation module is used to modulate network features based on the current time step and sensor text cues; The Top-K sparse cross-attention module is used to selectively inject high-frequency spatial information from the panchromatic image by using multispectral features as queries and panchromatic features as keys and values ​​through a Top-K filtering mechanism.

8. A multispectral and panchromatic image fusion system based on asynchronous diffusion refinement from structure to detail, characterized in that, The system includes: The input preprocessing module is used to acquire low-resolution multispectral images and high-resolution panchromatic images of the same region, and to upsample the low-resolution multispectral images to obtain an initial multispectral estimation image with the same size as the high-resolution panchromatic image. The structure reconstruction module is used in the first noise interval of the reverse denoising process of the diffusion model to guide the model to recover the overall spatial structure of the fused image by utilizing the spatial structure information in the high-resolution panchromatic image, thus completing the structure reconstruction. The asynchronous transmission module is used in the second noise interval of the reverse denoising process to gradually transfer the dominant information in the generation process from the spatial structure stream based on the high-resolution panchromatic image to the spectral residual stream based on the multispectral residual through the asynchronous transmission mechanism, thus completing a smooth transition of information dominance. The spectral refinement and fusion module is used in the third noise region of the reverse denoising process. Based on the spectral residual flow, and combined with the spatial structure information of the high-resolution panchromatic image and the spectral information of the upsampled multispectral image, it performs spectral compensation and local detail correction on the fused image to complete the spectral refinement and obtain the high-resolution multispectral fused image. Image output module, used to output high-resolution multispectral fused images.

9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement the multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement the multispectral and panchromatic image fusion method based on asynchronous diffusion refinement from structure to detail as described in any one of claims 1 to 7.

Images