A high-quality HRHS image generation method based on twin network structure

Abstract

The present application relates to the field of remote sensing image processing, and discloses a high-quality HRHS image generation method based on a dual network structure, comprising the following steps: receiving input low-resolution hyperspectral images and panchromatic images, respectively for providing spectral information and spatial information; filtering the low-resolution hyperspectral images through a noise-resistant pre-fusion sub-network, and generating a pre-fused hyperspectral image using the high spatial resolution information of the panchromatic image and the spectral feature correlation of the low-resolution hyperspectral image; reconstructing and enhancing the high-frequency details of the pre-fused image based on a conditional diffusion model through a high-frequency enhancement sub-network; fusing the low-frequency information of the pre-fused image with the high-frequency detail information generated by the high-frequency enhancement sub-network through an inverse wavelet transform, and outputting the final high-resolution hyperspectral image. The present invention can effectively fuse spectral and spatial information in a variety of noise environments, and generate a high-resolution hyperspectral image with high spectral fidelity and spatial details.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a high-quality HRHS image generation method based on a twin network structure. Background Art

[0002] Hyperspectral imaging technology can perform high-precision spectral analysis of ground targets by recording continuous narrow-band spectral information of electromagnetic waves. It is widely used in remote sensing scene classification, mineral resource exploration, environmental monitoring and other fields. However, due to the physical limitations of satellite sensors, current hyperspectral images usually have the problem of low spatial resolution, which significantly restricts their application in high-precision remote sensing tasks. In contrast, although panchromatic images have low spectral resolution, they have high spatial resolution. Therefore, a method of fusing hyperspectral and panchromatic images (i.e., hyperpansharpening) was proposed to generate hyperspectral images (HRHS) with both high spectral fidelity and high spatial resolution.

[0003] Existing hyperspectral and panchromatic image fusion methods mostly rely on traditional optimization algorithms or deep learning models. However, these methods have obvious limitations in practical applications. On the one hand, due to the poor robustness to noise, traditional methods are prone to introduce spectral distortion or spatial blur in the fusion process, especially when there is noise in low-resolution hyperspectral images (LRHS). On the other hand, although deep learning-based fusion methods can improve the fusion effect by learning nonlinear mapping relationships, most methods do not consider interference such as Gaussian noise, stripe noise and mixed noise that are common in actual hyperspectral images, which makes the hyperspectral images generated under complex noise conditions often have serious detail loss or noise residue. In addition, when dealing with the association between spectral and spatial features, most existing methods adopt a simple feature stacking strategy, ignoring the complex interaction between the two, thereby limiting the quality of the fused image.

[0004] In this context, in order to address the problems of insufficient noise resistance, loss of detail information and low spectral fidelity in the fusion of hyperspectral and panchromatic images in the existing technologies, the present invention proposes a high-quality and high-resolution hyperspectral image generation method based on a twin network structure. By introducing a noise-resistant pre-fusion sub-network and a high-frequency enhancement sub-network based on a conditional diffusion model, the present invention solves the technical problems of traditional methods such as sensitivity to noise, insufficient detail processing and insufficient modeling of the correlation between spectral and spatial features. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a high-quality HRHS image generation method based on a twin network structure, which solves the problems of spatial blur, spectral distortion and detail loss in the existing hyperspectral and panchromatic image fusion methods in complex noise environments, and realizes the generation of noise-resistant high-quality and high-resolution hyperspectral images.

[0006] To achieve the above objectives, the present invention is implemented by the following technical scheme: a high-quality HRHS image generation method based on a twin network structure, comprising the following steps:

[0007] Receiving input low-resolution hyperspectral images and panchromatic images, and providing spectral information and spatial information respectively;

[0008] The low-resolution hyperspectral image is subjected to noise filtering by means of an anti-noise pre-fusion sub-network, and a pre-fused high-resolution hyperspectral image is generated by utilizing the high spatial resolution information of the panchromatic image and the correlation between the spectral features of the low-resolution hyperspectral image;

[0009] Reconstructing and enhancing the high-frequency details of the pre-fused image based on a diffusion model through a high-frequency enhancement sub-network to generate an enhanced high-resolution hyperspectral image;

[0010] The low-frequency information of the pre-fused image is fused with the high-frequency detail information generated by the high-frequency enhancement sub-network through inverse wavelet transform to output the final high-resolution hyperspectral image.

[0011] The present invention also provides a high-quality HRHS image generation device based on a twin network structure, comprising:

[0012] A receiving module, used for receiving a low-resolution hyperspectral image and a panchromatic image as inputs of spectral information and spatial information, respectively;

[0013] The noise-resistant pre-fusion sub-network module is used to perform two upsampling and convolution operations on the low-resolution hyperspectral image to remove noise and extract spectral features. At the same time, the panchromatic image is used to extract spatial features. The spectral features and spatial features are fused through the spatial-spectral fusion module to generate a pre-fused hyperspectral image.

[0014] A high-frequency enhancement sub-network module is used to process the pre-fused hyperspectral image based on a diffusion model, gradually add noise to the high-frequency features to generate a noisy image through a forward process, and gradually remove noise and reconstruct high-frequency details through a reverse process;

[0015] The output module is used to fuse the low-frequency information generated by the anti-noise pre-fusion sub-network module with the high-frequency details generated by the high-frequency enhancement sub-network module through inverse wavelet transform, and output the final high-resolution hyperspectral image.

[0016] The present invention provides a high-quality HRHS image generation method based on a twin network structure, which has the following beneficial effects:

[0017] 1. The present invention uses a dual network architecture to efficiently fuse the spectral information of low-resolution hyperspectral images with the spatial information of panchromatic images. The anti-noise pre-fusion subnetwork (APS) uses large kernel convolution and spatial-spectral fusion module (S2FM) to achieve the initial integration of spectral and spatial features, effectively reducing the interference of noise on the fusion process and improving the stability of fusion and spectral fidelity.

[0018] 2. Aiming at the problem that low-resolution hyperspectral images are often interfered by various noises in real applications, the present invention effectively suppresses the influence of various complex noises such as Gaussian noise and mixed noise through an anti-noise pre-fusion sub-network and a high-frequency enhancement sub-network (HES) based on a conditional diffusion model. Experiments show that the present invention can still generate high-quality HRHS images in high-intensity and complex noise environments.

[0019] 3. The high frequency enhancement subnetwork (HES) of the present invention uses a conditional diffusion model to gradually generate high frequency details, and combines wavelet transform to accurately reconstruct spectral and spatial features. By enhancing spectral and spatial details in the frequency domain, the present invention is significantly superior to existing methods in detail restoration and fidelity, solving the problem that traditional methods easily lead to blurred details or texture loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0021] Figure 2 It is a schematic diagram of the overall framework of the method of the present invention;

[0022] Figure 3 It is a schematic diagram of the structure of the noise-resistant pre-fusion sub-network of the present invention;

[0023] Figure 4 It is a schematic diagram of the structure of the high-frequency enhancement sub-network of the present invention;

[0024] Figure 5 It is a schematic diagram of the device structure of the present invention.

[0025] Among them, 10, receiving module; 20, anti-noise pre-fusion sub-network module; 30, high-frequency enhancement sub-network module; 40, output module. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Please refer to the attached Figure 1 -Attached Figure 4 The present invention provides a high-quality HRHS image generation method based on a dual-subnetwork structure. By receiving a low-resolution hyperspectral image (LRHS) and a panchromatic image (PAN), a HRHS image with both high spectral resolution and high spatial resolution is generated by using the collaborative processing of an anti-noise pre-fusion subnetwork (APS) and a high-frequency enhancement subnetwork (HES).

[0028] like Figure 1 As shown, the high-quality HRHS image generation method based on the twin network structure may include the following steps:

[0029] S1. Receiving and preprocessing input data;

[0030] S2, processing of noise-resistant pre-fusion sub-network;

[0031] S3, high-frequency detail reconstruction and enhancement based on diffusion model;

[0032] S4. Generation of the final image.

[0033] Each step of the method of the present invention is described in detail below.

[0034] For step S1, in this embodiment, step S1 receives input low-resolution hyperspectral image (LRHS) and panchromatic image (PAN), and provides necessary spectral information and spatial information support for subsequent steps. Specifically, the low-resolution hyperspectral image and panchromatic image are used to provide high spectral information and high spatial resolution information respectively in this method.

[0035] In one possible implementation, a low-resolution hyperspectral image It can be expressed as ,in is the number of bands of the hyperspectral image, and the spatial resolution of each band is . Full color image It is a single-band high-resolution image with a spatial resolution of , which is expressed as .

[0036] It should be noted that low-resolution hyperspectral images The spectral information contained in the image is rich, but due to its low resolution and limited by the sensor performance, its band information may be interfered by various noises, including Gaussian noise, stripe noise, impulse noise, etc. These noises may interfere with the extraction and fusion of spectral information in subsequent processing steps.

[0037] As an option, full-color images Provides high spatial resolution information. It is understandable that since panchromatic images are usually acquired over a wide spectral range, their spectral resolution is low, but they have strong noise resistance, which provides important spatial reference information for the generation of high-resolution images.

[0038] In some embodiments, the low resolution hyperspectral image and full color images It can be collected by satellite sensors or other remote sensing equipment. For example, low-resolution hyperspectral images can be derived from hyperspectral imaging sensors such as AVIRIS or Hyperion, while panchromatic images can be derived from optical imaging devices. Exemplarily, the output of these sensors can be provided in the form of digital image files and used as input for subsequent processing.

[0039] In order to ensure that the low-resolution hyperspectral image and the panchromatic image can be correctly processed by the anti-noise pre-fusion subnetwork (APS) of the present invention, in this embodiment, the input images are required to be consistent in data form. For example, all input images need to be normalized so that the pixel value range is unified in the range of [0,1] to avoid the influence of different data ranges on the feature extraction results.

[0040] It should be noted that, in practical applications, the number of bands and resolution of the input image can be adjusted according to specific task requirements. For example, in one possible implementation, for some tasks that only focus on specific bands, only hyperspectral images of some bands can be used as input without affecting the implementation effect of the present invention.

[0041] As an implementation method, the valid area of ​​the input image can be cropped during the input receiving process. For example, if there is a large amount of background or invalid area in the image, the input image can be cropped using target detection technology or a preset region of interest (ROI) to further reduce the computational overhead of subsequent processing.

[0042] In another possible implementation, the input image may have partial data loss or artifacts during transmission. The present invention can be combined with image preprocessing technology (such as interpolation reconstruction or spatial domain filtering) to complete the missing area, thereby ensuring the integrity of the input data.

[0043] It should be noted that the input form of the low-resolution hyperspectral image and the panchromatic image is not limited to file input. In some embodiments, the image can be received in a real-time data stream, for example, by directly acquiring the hyperspectral and panchromatic image data through satellite communication or drone transmission.

[0044] In summary, the key to step S1 is to receive and prepare low-resolution hyperspectral images and full color images , laying the foundation for subsequent anti-noise processing and high-resolution image generation.

[0045] For step S2, in this embodiment, step S2 uses an anti-noise pre-fusion sub-network (APS) to fusion the low-resolution hyperspectral image. Noise filtering and full color image High spatial resolution information and Generate pre-fused high-resolution hyperspectral images based on spectral feature correlation This step mainly includes noise removal, spectral and spatial feature extraction, and spectral-spatial feature fusion.

[0046] Specifically, in order to reduce the low-resolution hyperspectral image To reduce the impact of the noise on the subsequent fusion process and extract clean spectral features, this embodiment designs an upsampling noise filter (UNF) module. As an implementation method, the UNF module Perform two step-by-step upsampling operations and combine large kernel convolution to filter the noise to obtain clean spectral features. The specific calculation formula is:

[0047]

[0048] in, Indicates a two-fold upsampling operation, converting the low-resolution Spatial resolution increased to that of full-color images Consistent ; and The convolution operations with sizes of 5×5 and 7×7 are used to gradually remove noise and retain valid spectral information.

[0049] It should be noted that the convolution operation is performed after each upsampling, which can effectively smooth the artifacts introduced by interpolation and further suppress noise, making Express clearer spectral features.

[0050] In another implementation, in order to extract spatial features with high spatial resolution, the full-color image It is input into a 3×3 convolutional layer for processing and outputs enhanced spatial features The calculation formula is:

[0051]

[0052] It should be understood that the convolution operation can enhance the feature expression ability of the full-color image by expanding the number of channels, providing efficient spatial information representation for subsequent fusion with spectral features.

[0053] In this embodiment, in order to further improve the expression ability of spectral and spatial features, the spectral features are respectively and spatial characteristics Learning is done through the residual block (ResBlock). The core formula of the residual block is:

[0054]

[0055] in, represents the input features, represents the output features, It is a combination of group normalization (GN), Swish activation function and 3×3 convolution, defined as follows:

[0056]

[0057] It should be noted that the introduction of residual blocks can effectively alleviate the problem of gradient vanishing in deep networks through skip connections, while retaining the original feature information and improving the expressiveness of features.

[0058] As an option, the present invention designs a spatial-spectral fusion module (S2FM) for fusion and Specifically, the two are first concatenated in the channel dimension as fusion features :

[0059]

[0060] in, Represents a channel dimension concatenation operation.

[0061] Fusion Features Features are further extracted through the GSC block, and local and global spatial-spectral correlations are captured through the sliding window self-attention module (SWSA). The design of SWSA can achieve efficient modeling of global features through cross-window information exchange, while being computationally economical.

[0062] In order to improve the network's perception of spectral information, the present invention further uses the spectral enhancement block (SEB) to dynamically adjust the channel weights of the fusion features. By learning the weights of different channels, SEB can effectively enhance the expression ability of spectral features. Its processing formula is:

[0063]

[0064] In one possible implementation, multiple S2FM modules are used in series to gradually enhance the spectral-spatial correlation of the fused features. Expressed as:

[0065]

[0066] It should be noted that Represents a 1×1 convolution operation, which is used to adjust the channel dimension to match the input requirements of the subsequent module.

[0067] Finally, the fusion features are constructed through a 3×3 convolutional layer. Processing to generate pre-fused hyperspectral images :

[0068]

[0069] It is important to understand that the pre-fused hyperspectral image It contains the spectral information of low-resolution hyperspectral images and the high spatial resolution information of panchromatic images, laying a solid foundation for the subsequent enhancement of high-frequency details.

[0070] In summary, this embodiment realizes the denoising of low-resolution hyperspectral images and the efficient fusion of spectral and spatial features through the anti-noise pre-fusion sub-network, which provides support for the generation of high-quality high-resolution hyperspectral images.

[0071] For step S3, in this embodiment, step S3 uses a high frequency enhancement sub-network (HES) to perform the pre-fused hyperspectral image generated in step S2 based on the diffusion model. Reconstruct and enhance high-frequency details to generate high-resolution hyperspectral images. Specifically, HES gradually generates high-frequency details of the image through the forward and reverse processes of the diffusion model, and combines spectral and spatial information to achieve high-quality detail enhancement.

[0072] In one possible implementation, the processing of HES is divided into three stages: forward diffusion process, reverse generation process and high-frequency feature reconstruction based on encoder-decoder architecture.

[0073] Forward diffusion process

[0074] In this embodiment, in the forward diffusion process of the high-frequency enhancement subnetwork (HES), Gaussian noise is gradually added to the high-frequency information to generate a process in which the noise-free high-frequency features gradually degenerate into noise features with a Gaussian distribution, providing a basis for subsequent reverse denoising and high-frequency information reconstruction.

[0075] Specifically, we firstly Perform wavelet transform and decompose it into a low-frequency subband and three high frequency sub-bands , , These high-frequency subbands contain image detail information, such as edges and textures.

[0076] Then, the three high-frequency sub-bands are concatenated in the channel dimension to obtain the initial high-frequency information. :

[0077]

[0078] in, represents the initial high-frequency information, It represents the splicing operation in the channel dimension. It can be understood that the purpose of this step is to centrally represent the high-frequency detail information of the image for subsequent processing.

[0079] Then, gradually Adding Gaussian noise gradually transforms it into a Gaussian noise image with a standard normal distribution. This gradual noise addition process is based on the Markov chain model and is defined as follows:

[0080]

[0081]

[0082] in:

[0083] represents the time step;

[0084] represents Gaussian distribution;

[0085] and Respectively and High frequency characteristics after step noise addition;

[0086] is a predefined noise variance hyperparameter, and its value range is (0,1).

[0087] It should be noted that through the above Markov chain, the forward diffusion process is from noise-free high-frequency features Start by gradually transforming it into a completely random Gaussian noise feature This gradual process of adding noise can smoothly model the degradation of high-frequency features.

[0088] Based on the reparameterization technique, at any time step , Can be directly expressed as the initial high-frequency information and standard Gaussian noise The functional form of is:

[0089]

[0090] in:

[0091] ;

[0092] ;

[0093] is a randomly sampled Gaussian noise.

[0094] The above formula shows that any time step in the forward diffusion process All are original high-frequency information With Gaussian noise A linear combination of and control.

[0095] To further characterize any time step of , its distribution can be expressed as:

[0096]

[0097] This formal description further reveals that in the forward diffusion process, The generation of high frequency information The weight between the noise and the noise is dynamically adjusted.

[0098] In summary, the forward diffusion process gradually adds noise to the initial high-frequency features. The noise features converted into Gaussian distribution lay the necessary modeling foundation for the denoising operation and high-frequency information reconstruction in the reverse generation process.

[0099] Reverse generation process

[0100] In this embodiment, the reverse generation process of the diffusion model aims to generate a Gaussian noise image from a completely noisy image. The noise is gradually removed to reconstruct the high-frequency information Through the step-by-step denoising process of back diffusion, the high-frequency details of the image can be gradually restored, providing detailed support for the final generation of high-resolution hyperspectral images.

[0101] Specifically, the reverse generation process follows the inverse process definition of the Markov chain, and generates the target high-frequency information by gradually denoising from Gaussian noise. The process can be expressed as:

[0102]

[0103] in:

[0104] represents the probability of generating complete high-frequency information by the back-diffusion process;

[0105] represents the initial Gaussian noise Distribution of

[0106] Indicated in Step from Denoising Generation The conditional probability of

[0107] Represents the network parameters of the high-frequency enhancement subnetwork (HES).

[0108] It should be noted that each step of the reverse denoising is modeled as a conditional Gaussian distribution, which is defined as:

[0109]

[0110] in:

[0111] Indicates The mean prediction of the step;

[0112] Indicates Step variance prediction;

[0113] Represents conditional information, which serves as input conditions for constraint generation during the reverse generation process.

[0114] In this embodiment, the condition information Combined with full color image and pre-fused hyperspectral images The multi-scale feature information of the two is extracted through wavelet decomposition, and the low-frequency and high-frequency sub-bands are respectively spliced ​​in the channel dimension. The specific definition is:

[0115]

[0116] in:

[0117] Represents a full-color image The wavelet decomposition characteristic coefficients of

[0118] Represents pre-fused hyperspectral image The wavelet decomposition characteristic coefficients of .

[0119] Full color image After wavelet decomposition, a low-frequency subband is generated and three high frequency sub-bands . These subbands are concatenated into wavelet decomposition features :

[0120]

[0121] in:

[0122] Represents the concatenation operation in the channel dimension;

[0123] It is the complete wavelet decomposition feature of the full-color image.

[0124] Pre-fusion of hyperspectral images After wavelet decomposition, a low-frequency subband is generated and three high frequency sub-bands . These subbands are concatenated into wavelet decomposition features :

[0125]

[0126] in:

[0127] Represents the complete wavelet decomposition features of the pre-fused hyperspectral image.

[0128] In the reverse generation process, the HES network uses conditional information , combined with the input of the current step and time step , stepwise prediction mean and variance , thus from Denoising Generation . Through gradual iteration, high-frequency information is finally generated .

[0129] Specifically, the reverse process starts with a randomly initialized Gaussian noise At the beginning, the denoised intermediate features are gradually generated through the above conditional probability formula until the time step The final high-frequency features are generated when The core of this process lies in accurately modeling the association between conditional information and high-frequency information, thereby generating high-quality results that are consistent with the true high-frequency features.

[0130] In summary, this embodiment uses the reverse diffusion generation process to gradually denoise the target high-frequency features from completely random Gaussian noise, and combines the wavelet feature information of the full-color image and the pre-fused image to ensure that the generated high-frequency features have high fidelity and rich detail information, providing support for the final high-resolution hyperspectral image.

[0131] High-frequency feature reconstruction based on encoder-decoder architecture

[0132] In this embodiment, the architecture of the high-frequency enhancement subnetwork (HES) is based on an encoder-decoder structure. The encoder is composed of a spectral fusion module (SFM) for extracting multi-scale spectral features; the decoder is composed of a spatial fusion reconstruction module (SFRM) for reconstructing high-dimensional, low-resolution spectral features into high-resolution features containing high-frequency details. Through this architecture, HES can fully combine spectral and spatial information to generate high-quality high-frequency features. .

[0133] The main goal of the encoder is to extract multi-scale spectral features through downsampling operations and integrate the spectral features so that the subsequent decoder can reconstruct the details.

[0134] 1. Splitting of input features

[0135] First, the pre-fused image Extracted wavelet features Split into low-frequency subbands along the channel dimension and high frequency subband

[0136]

[0137] in:

[0138] Represents clean low-frequency spectral information;

[0139] Indicates the high-frequency features to be enhanced.

[0140] Will The input image generated by forward diffusion Concatenate and process through a 3×3 convolutional layer to generate the initial encoding features .

[0141] 2. Extraction of multi-scale spectral features

[0142] In order to make the encoded features contain more accurate spectral information, a 3×3 convolution layer is used to Processing and combined with a two-fold downsampling operation to extract multi-scale spectral features At the same time, the spectral fusion module (SFM) is used to integrate the multi-scale spectral features and the previous layer encoding features:

[0143]

[0144] in:

[0145] is the concatenated input feature;

[0146] Indicates The encoding features of the layer;

[0147] Represents the time step.

[0148] 3. Structure of spectral fusion module

[0149] The main function of SFM is to integrate input features and time step , capturing the correlation of spectral features while avoiding information loss. The specific formula is:

[0150]

[0151] in:

[0152] Represents the GN-Swish-Con combination block;

[0153] It is a global spectral self-attention module, which is used to capture the global correlation between spectral features;

[0154] Represent query, key, and value matrices respectively;

[0155] is a multi-layer perceptron that generates implicit embeddings of time steps.

[0156] 4. Multi-layer coding and skip connections

[0157] The encoder contains multiple encoding layers, which extract spectral features of different resolutions by stepwise downsampling. At the same time, to avoid gradient vanishing and feature loss, skip connections are used to pass features of the same resolution directly to the decoder.

[0158] The main goal of the decoder is to restore the multi-scale spectral features extracted by the encoder into high-resolution high-frequency features through upsampling operations, while combining spatial information for detail enhancement.

[0159] 1. Extraction of spatial detail features

[0160] From the full-color image Wavelet decomposition characteristics Extracting multi-scale spatial detail features , the specific formula is:

[0161]

[0162] 2. Integration and recovery of decoding features

[0163] The decoder effectively integrates the coding features, spatial detail features and upper-layer decoding features through the spatial fusion reconstruction module (SFRM). The formula of SFRM is:

[0164]

[0165] in:

[0166] ;

[0167] Indicates Decoded features of the layer;

[0168] Represents the encoding features of the corresponding layer.

[0169] 3. Structure of the spatial fusion reconstruction module

[0170] SFRM captures the local and global spatial feature correlation through the sliding window self-attention module (SWSA), and combines time step embedding and residual connection for feature integration. The specific formula is:

[0171]

[0172] in:

[0173] represents the sliding window self-attention module;

[0174] Used for channel adjustment.

[0175] 4. Stepwise upsampling and reconstruction

[0176] The decoder restores low-resolution features to the target resolution by upsampling layer by layer, and gradually generates high-frequency details with high resolution by combining the upper-layer decoded features and the encoder’s skip connection features.

[0177] Through the learning of multiple layers of SFM and SFRM, the high-frequency enhancement sub-network finally generates the reconstructed high-frequency sub-band Specifically, this high-frequency subband is output through the decoder and fused with the low-frequency information by inverse wavelet transform in the subsequent step to form the final high-resolution hyperspectral image.

[0178] In summary, the high-frequency feature reconstruction process based on the encoder-decoder architecture realizes the deep fusion of spectral and spatial information, generates high-quality high-frequency details through gradual up- and down-sampling operations, and provides support for the final generation of high-resolution hyperspectral images.

[0179] For step S4, in this embodiment, step S4 fuses the low-frequency information of the pre-fused image with the high-frequency detail information generated by the high-frequency enhancement sub-network through inverse wavelet transform to output the final high-resolution hyperspectral (HRHS) image. This step aims to integrate low-frequency and high-frequency features, retain the spectral information of the pre-fused image, and combine the detail information after high-frequency enhancement to generate a high-quality image with both spectral fidelity and spatial details.

[0180] In one possible implementation, the pre-fused image and the high-frequency sub-band generated by the high-frequency enhancement sub-network is the input of this step. Specifically, the pre-fused image Provides low-frequency information, represented as a low-frequency subband , while high-frequency details are represented by high-frequency sub-bands These subbands retain the overall contour and local detail information of the image in the wavelet domain.

[0181] It should be noted that the pre-fused image and full color images The wavelet features of have been extracted in the previous step. In this step, the wavelet decomposition of the pre-fused image obtains the low-frequency subband , which is expressed as:

[0182]

[0183] The high-frequency sub-band generated by the high-frequency enhancement sub-network Contains the spliced Subband, expressed as:

[0184]

[0185] As an option, the channel information of the high-frequency sub-bands is kept consistent with the spectral bands, ensuring that the edge and detail features of different spectral bands are accurately modeled.

[0186] Specifically, this step uses inverse wavelet transform to reintegrate the low-frequency subband and the high-frequency subband into an image in the spatial domain. The mathematical representation of inverse wavelet transform is as follows:

[0187]

[0188] in:

[0189] Represents a two-dimensional discrete wavelet inverse transform operation;

[0190] represents the low-frequency subband;

[0191] Represents the high-frequency sub-band, generated by the high-frequency enhancement sub-network.

[0192] It should be understood that the inverse wavelet transform operation reconstructs the low-frequency subband and the high-frequency subband in the spatial domain, wherein the low-frequency subband provides a framework for global spectral information and the high-frequency subband supplements the details and texture of the image.

[0193] It should be noted that the high-resolution hyperspectral image reconstructed by inverse wavelet transform Features include:

[0194] Low frequency subband It provides global consistency of the spectrum and ensures the spectral fidelity of the image;

[0195] High frequency sub-band It provides local details such as edges and textures, and improves the spatial resolution of the image.

[0196] In some embodiments, in order to further improve the quality of the image, the output Perform post-processing operations, such as artifact removal or color smoothing, to eliminate possible reconstruction errors.

[0197] This embodiment integrates low-frequency and high-frequency features through inverse wavelet transform, efficiently combines the spectral information of the pre-fused image with the detail information after high-frequency enhancement, and generates the final high-resolution hyperspectral image. . It achieves efficient fusion of spectral and spatial features, providing the final output image with both high spectral fidelity and detail clarity.

[0198] As a preferred embodiment of the present invention, in order to further optimize the quality of the generated high-resolution hyperspectral (HRHS) image, the present invention designs a comprehensive loss function, which is composed of a pre-fusion loss , high frequency wavelet loss and reconstruction losses The total loss function is defined as:

[0199]

[0200] Through the combined optimization of the above three parts of loss, it can ensure that the generated image meets the high quality requirements in terms of spectral fidelity and spatial details.

[0201] Pre-fusion loss

[0202] As a preferred embodiment of the present invention, the pre-fusion loss Pre-fused hyperspectral images for constrained noise-resistant pre-fusion subnetwork (APS) generation Real high-resolution hyperspectral images The difference between , thereby ensuring the spectral fidelity of the preliminary fusion result. The specific definition of pre-fusion loss is:

[0203]

[0204] in:

[0205] Represents the hyperspectral image generated by the noise-resistant pre-fusion sub-network;

[0206] Represents a true high-resolution hyperspectral image;

[0207] Represents the L1 norm, which is used to measure the pixel-level differences of images.

[0208] Through this loss function, the pre-fused image can accurately retain the spectral information while laying the foundation for subsequent high-frequency detail enhancement.

[0209] High frequency wavelet loss

[0210] As a preferred embodiment of the present invention, high frequency wavelet loss High-frequency features generated by the High-frequency Enhancement Subnetwork (HES) With the real high frequency characteristics The difference between , thus ensuring the quality of high-frequency details. Specifically defined as:

[0211]

[0212] in:

[0213] is the real high frequency sub-band;

[0214] is the high-frequency sub-band generated by the high-frequency enhancement sub-network;

[0215] represents the high-frequency enhancement subnetwork, Its network parameters;

[0216] is random noise sampled from a standard Gaussian distribution;

[0217] is the time step;

[0218] Represents condition information, including pre-fused images and full color images Wavelet characteristics of

[0219] Indicates the time step The cumulative noise factor.

[0220] It should be noted that the high-frequency wavelet loss is directly optimized to generate high-frequency features , which can ensure the fineness of the generated image in high-frequency details and avoid the loss or distortion of detail information.

[0221] Reconstruction losses

[0222] As a preferred embodiment of the present invention, the reconstruction loss Used to constrain the final generated high-resolution hyperspectral image Real high-resolution hyperspectral images The difference between , thereby ensuring the overall quality of the output image. The specific definition of reconstruction loss is:

[0223]

[0224] in:

[0225] To generate the high-resolution hyperspectral image,

[0226] It is a real high-resolution hyperspectral image.

[0227] Through this loss function, it can be ensured that the final generated hyperspectral image is consistent with the real image in terms of overall spectral information and spatial details.

[0228] Through the combined optimization of the above three parts of loss, the pre-fusion loss The spectral consistency of the initial fusion image is guaranteed, and the high-frequency wavelet loss The authenticity of detail generation is ensured, while the reconstruction loss The spectral and spatial characteristics of the final output image are balanced. This comprehensive loss design can comprehensively improve the quality of the generated image and provide high-fidelity and high-resolution hyperspectral image support for a variety of remote sensing and monitoring tasks.

[0229] The high-quality HRHS image generation device based on the twin network structure described below and the high-quality HRHS image generation method based on the twin network structure described above can refer to each other.

[0230] Please see attached Figure 5 The present invention also provides a high-quality HRHS image generation device based on a twin network structure, comprising:

[0231] The receiving module 10 is used to receive the low-resolution high-spectral image and the panchromatic image as inputs of spectral information and spatial information respectively;

[0232] The anti-noise pre-fusion sub-network module 20 is used to perform two upsampling and convolution operations on the low-resolution hyperspectral image to remove noise and extract spectral features, and at the same time use the full-color image to extract spatial features, and fuse the spectral features and spatial features through the spatial-spectral fusion module to generate a pre-fused hyperspectral image;

[0233] A high-frequency enhancement sub-network module 30 is used to process the pre-fused hyperspectral image based on a diffusion model, gradually add noise to the high-frequency features to generate a noisy image through a forward process, and gradually remove noise and reconstruct high-frequency details through a reverse process;

[0234] The output module 40 is used to fuse the low-frequency information generated by the anti-noise pre-fusion sub-network module with the high-frequency details generated by the high-frequency enhancement sub-network module through inverse wavelet transform, and output a final high-resolution hyperspectral image.

[0235] The device of this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0236] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A high-quality HRHS image generation method based on a twin network structure, characterized in that: The following steps are involved: Receiving input low-resolution hyperspectral images and panchromatic images, and providing spectral information and spatial information respectively; The low-resolution hyperspectral image is subjected to noise filtering by means of an anti-noise pre-fusion sub-network, and a pre-fused high-resolution hyperspectral image is generated by utilizing the high spatial resolution information of the panchromatic image and the correlation between the spectral features of the low-resolution hyperspectral image; Reconstructing and enhancing the high-frequency details of the pre-fused image based on a diffusion model through a high-frequency enhancement sub-network to generate an enhanced high-resolution hyperspectral image; The low-frequency information of the pre-fused image is fused with the high-frequency detail information generated by the high-frequency enhancement sub-network through inverse wavelet transform to output the final high-resolution hyperspectral image. The anti-noise pre-fusion sub-network includes: The low-resolution hyperspectral image is upsampled twice, and the noise is filtered out using large kernel convolution to obtain clean spectral features; The clean spectral features and the full-color image are convolved respectively, and the spectral features and the spatial features are extracted using a residual block; The spectral features are fused with the spatial features through the spatial-spectral fusion module to establish the correlation between the spectrum and space; Outputting a pre-fused hyperspectral image containing high-fidelity subject information through a convolution operation; The high frequency enhancement sub-network includes: Based on the diffusion model, the high-frequency sub-bands are gradually denoised through the forward process to generate a Gaussian noise image, and the high-frequency details are gradually denoised through the reverse process. The encoder module combines the input features with the denoised spectral features through multi-layer fusion operations to obtain encoded features containing rich spectral information; The decoder module combines the encoded features with the high-frequency details of the panchromatic image through a multi-layer fusion operation to reconstruct a high-resolution hyperspectral image.

2. According to the method of claim 1, the high-quality HRHS image generation method based on the twin network structure is characterized in that: The spatial-spectral fusion module includes: The spectral features and spatial features are spliced ​​into fusion features using channel splicing operations; Capturing local and global spatial similarities through a sliding window self-attention module; Dynamically adjust channel weights through spectral enhancement blocks to improve the perception of spectral information; The fused features are combined through multiple nonlinear transformations and residual connections to generate enhanced fused features.

3. The high-quality HRHS image generation method based on a twin network structure according to claim 1 is characterized in that: The forward process in the high-frequency enhancement subnetwork includes: Perform wavelet transform on the reference image and decompose it into one low-frequency sub-band and three high-frequency sub-bands; Concatenate the high-frequency subbands along the channel dimension into the initial high-frequency information , and gradually add noise through the following Markov chain process: in, is the time step The noise result is: is the noise variance, is the identity matrix.

4. The high-quality HRHS image generation method based on a twin network structure according to claim 3 is characterized in that: The reverse process in the high-frequency enhancement sub-network includes: Stepwise denoising of random Gaussian noise image and reconstruction of high-frequency detail information through the following conditional distribution: in, and are the mean and variance of the predicted distribution, respectively. It is the conditional information, including the wavelet coefficients of the full-color image and the pre-fused image.

5. The high-quality HRHS image generation method based on a twin network structure according to claim 1 is characterized in that: The encoder of the high-frequency enhancement sub-network adopts a spectral fusion module to extract multi-scale spectral features, and the spectral fusion module includes: Generate implicit embedding features of time steps through multi-layer perceptron; Use the global spectral self-attention mechanism to capture the correlation between spectral features; The enhanced encoding features are output by combining multiple layers of nonlinear transformations and residual connections.

6. The high-quality HRHS image generation method based on a twin network structure according to claim 1 is characterized in that: The decoder of the high-frequency enhancement sub-network adopts a spatial fusion reconstruction module to realize the reconstruction of high-frequency features, and the spatial fusion reconstruction module includes: Use sliding window self-attention to capture local and global correlations between decoded features; Generate time step embedding features and perform feature fusion through multi-layer perceptron; High-resolution high-spectral detail features are output through residual connection and multi-layer convolution.

7. The high-quality HRHS image generation method based on a twin network structure according to claim 1 is characterized in that: The method is optimized by the following loss function: Pre-fusion loss : Pre-fused image used to constrain the output of the noise-resistant pre-fusion subnetwork Compared with real hyperspectral images The differences between them are specifically expressed as: in, is the pre-fused image, For real high-resolution hyperspectral images, represents the L1 norm; High frequency wavelet loss : High-frequency subband used to constrain the output of the high-frequency enhancement sub-network With the real high frequency subband The differences between them are specifically expressed as: in, is the high-frequency enhancement subnetwork, Its network parameters; ,in , is the time step The noise variance under ; represents noise sampled from a standard normal distribution; It is the conditional information of the reverse diffusion process, including the wavelet coefficients of the full-color image and the pre-fused image; Reconstruction losses : Used to constrain the final generated high-resolution hyperspectral image Compared with real hyperspectral images The differences between them are specifically expressed as: in, The final hyperspectral image is generated; The total loss function is expressed as: .

Images