Hyperspectral image panchromatic sharpening method based on space-spectrum modulation condition diffusion model

Abstract

The invention discloses a hyperspectral image panchromatic sharpening method based on a space-spectrum modulation condition diffusion model, and the method comprises the steps: obtaining a hyperspectral image and a panchromatic image which are paired in a database, and carrying out the cutting and data enhancement processing, thereby obtaining a training set, a verification set and a test set; performing noise addition operation on the initial data to construct data required by a prediction network; constructing a space-spectrum modulation noise prediction network, wherein the space-spectrum modulation noise prediction network comprises a time step encoder, two condition encoders, an intermediate layer and a decoder; training the space-spectrum modulation noise prediction network to obtain a trained space-spectrum modulation noise prediction network; and performing panchromatic sharpening on the hyperspectral image by using the trained space-spectrum adjustment noise prediction network. Through the space-spectrum noise prediction network, condition information of different modals can be utilized more effectively, so that an adverse effect of a final sharpening result caused by information confusion is avoided.

Description

Technical Field

[0001] The invention relates to the field of remote sensing image processing, and in particular to a hyperspectral image panchromatic sharpening method based on a spatial-spectral modulation conditional diffusion model. Background Art

[0002] Satellite remote sensing aims to provide sufficient and accurate observation information of the earth's surface. However, due to the hardware constraints of satellite sensors, the spatial resolution of hyperspectral images is affected by their spectral resolution, resulting in the loss of information such as tiny spatial textures and boundary details. Considering that most existing practical applications need to be based on high-spatial-resolution hyperspectral images, it is necessary to use the hyperspectral image panchromatic sharpening method that has received much attention in recent years. The paired single-band high-spatial-resolution panchromatic image is fused with the original observed low-spatial-resolution hyperspectral image to obtain an ideal high-spatial-resolution hyperspectral image.

[0003] The original hyperspectral image pan-sharpening technology was developed based on the traditional multispectral image sharpening method, which mainly includes three categories: component replacement, multi-resolution analysis, and variational optimization. However, these methods all use fixed data priors. The component replacement and multi-resolution analysis methods generally have spectral and spatial distortion, while the variational optimization method requires a long iterative optimization process. With the rapid rise of deep learning, with the help of the powerful feature learning ability of neural networks, the pan-sharpening method based on deep learning is superior to the traditional method in terms of fusion accuracy and efficiency. However, most of the sharpening networks based on convolutional neural networks use a pixel-by-pixel mean square error loss function to optimize the network parameters, which will cause the final fusion result to be too average, resulting in a blurry effect and unnecessary lack of details.

[0004] In recent years, with the rapid development of generative models in the field of image generation, the denoising diffusion model has performed the best. It gradually transfers a Gaussian distribution sample to the target generation sample through iterative learning of the prediction process of the posterior distribution. In the application of panchromatic sharpening, the panchromatic image and the hyperspectral image are injected into the denoising prediction network as conditional information to guide the model to generate an ideal hyperspectral image with high spatial resolution. Compared with the panchromatic sharpening method based on the generative adversarial model, it has greatly alleviated the problems of model crash and insufficient effect fidelity. However, most of the existing panchromatic sharpening methods based on the diffusion model do not consider the characteristics of the two different modal images of panchromatic and hyperspectral images. Directly inputting them into the denoising network as conditional information will cause confusion of spectral and spatial information, and ultimately lead to problems such as local black shadows and insufficient fidelity of spatial details in the generated results. Summary of the invention

[0005] In order to overcome the shortcomings of the panchromatic sharpening technology based on the diffusion model in the prior art, the object of the present invention is to provide a hyperspectral image panchromatic sharpening method based on the spatial-spectral modulation conditional diffusion model.

[0006] The present invention includes two processes, namely the forward denoising process and the reverse denoising process. The forward denoising process is to gradually add noise to the training data into a random normally distributed sampled Gaussian noise by using a Markov chain process. This process is also to create the corresponding label data for model training, which is used as the optimization target for training the noise prediction network. In order to make more effective use of the conditional information of the two different states of full-color and hyperspectral images, we used two encoders with similar structures as the injection networks of these two conditional information in the design of the noise prediction network. The feature information of each level of the two encoders is simultaneously connected to the decoder level of the corresponding scale. In the decoder, this method designs a spectral-spatial modulation module to emphasize the important information of different conditional inputs and fuse them to avoid unnecessary information confusion leading to poor final generation effect. In the reverse denoising process, the trained noise prediction network is iteratively denoised according to the set time step to generate the target output.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A hyperspectral image panchromatic sharpening method based on a spatial-spectral modulation conditional diffusion model, comprising:

[0009] Obtain paired hyperspectral images and panchromatic images from the database, and obtain training sets, validation sets, and test sets after cropping and data enhancement.

[0010] Perform noise operation on the initial data to construct the data required for the prediction network;

[0011] Constructing a spatial-spectral modulation noise prediction network, wherein the spatial-spectral modulation noise prediction network includes a time step encoder, two conditional encoders, an intermediate layer and a decoder;

[0012] Training a spatial-spectral modulation noise prediction network to obtain a trained spatial-spectral modulation noise prediction network;

[0013] Hyperspectral image pan-sharpening is performed using the trained spatial-spectral adjusted noise prediction network.

[0014] Furthermore, the original hyperspectral images in the database are blurred and downsampled at a set ratio to obtain simulated low-resolution hyperspectral images. The low-resolution hyperspectral images are bilinearly interpolated to the same size as the full-color images to form paired large-scale image pairs, which are cropped according to the set size and generated by flipping and rotating data augmentation methods to generate a specified number of training sets, validation sets, and test sets.

[0015] Furthermore, the noise addition calculation formula is specifically:

[0016]

[0017] Among them, ε t represents randomly sampled Gaussian noise, x t represents the noise image corresponding to time step t, α i is a preset weight parameter, which increases linearly with the increase of time steps. x0 represents the difference between the reference high-resolution hyperspectral image and the interpolated low-resolution hyperspectral image. The reference refers to the high-resolution hyperspectral reference corresponding to the low-resolution hyperspectral image.

[0018] Furthermore, the noise addition operation is specifically as follows: setting the total time step T of the noise addition, randomly sampling a time step t in the interval 0-T, and obtaining the noise image x corresponding to the time step t according to the noise addition calculation formula t .

[0019] further,

[0020] The time step encoder inputs an integer t∈(0,T), obtains a one-dimensional vector through the time step encoder, and embeds it into each level of the noise prediction network through the Linear layer;

[0021] The two conditional encoders are respectively a full-color conditional injection encoder and a hyperspectral image injection encoder, the input of the full-color conditional injection encoder is a noise image and a full-color image, and the input of the hyperspectral image injection encoder is a noise image and a hyperspectral image;

[0022] In the intermediate layer, the outputs of the two conditional encoders are concatenated through the channel dimension and then input into the intermediate layer, and then feature fused and then input into the decoder;

[0023] The decoder includes four decoding layers and an output layer. The decoding layer includes a spatial-spectral modulation module, two convolutional layers and a two-fold upsampling layer. The output layer is used to adjust the output channel and output the correct data format.

[0024] Furthermore, the spatial-spectral modulation module includes a spectral enhancement attention layer and a local spatial high-frequency enhancement attention layer.

[0025] Furthermore, the intermediate layer includes two residual convolution layers and a global self-attention layer.

[0026] Furthermore, in the training of the spatial-spectral modulated noise prediction network, in each update training, a batch of data is selected from the training set and the validation set, and the data is denoised to obtain noise and noise images. The noise image, time step encoder, full-color image and hyperspectral image are input into the noise prediction network, and then the loss of the noise and the output of the noise prediction network is calculated, and the network parameters are iteratively updated through the back-propagation algorithm until the loss value converges stably.

[0027] Furthermore, the trained spatial-spectral adjustment noise prediction network is used to perform panchromatic sharpening of the hyperspectral image, specifically:

[0028] The trained noise prediction network is used to take a random noise as input x from the normal distribution based on the Markov chain process according to the set total time step T. T Then, in each iteration, the current time step t encoding and test condition information [pan, hs], as well as the noise image x are input. t , where the calculation process is as follows, the generation ends at t = 0, and the final sharpening result is obtained;

[0029]

[0030] where α t is the weight parameter, and z is random Gaussian noise.

[0031] Furthermore, α i The value is (0.0001, 0.02).

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] The present invention proposes a hyperspectral panchromatic sharpening method based on a spatial-spectral modulation conditional diffusion model. The method designs a spatial-spectral modulation noise prediction network, in which two conditional injection encoders with the same structure are used to respectively receive conditional information of two different modes, and then the information from the two encoders is input into a spatial-spectral modulation module, and the important features in the conditional information of the two different states are respectively enhanced, and then the two are fused and decoded to obtain the corresponding output. In the powerful generation process of the diffusion model, the conditional information of different modes can be more effectively utilized through this spatial-spectral noise prediction network to avoid information confusion, which causes adverse effects on the final sharpening result. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart for implementing the present invention.

[0035] Figure 2 The figure is a schematic diagram of the reverse denoising generation process of the present invention.

[0036] Figure 3 Structural diagram of the spatial-spectral modulation noise prediction network designed for the present invention.

[0037] Figure 4 Structural diagram of the coding layer and the middle layer in the noise prediction network designed for the present invention.

[0038] Figure 5 Structural diagram of the decoding layer in the noise prediction network designed for the present invention.

[0039] Figure 6 Generate an effect comparison diagram for the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0041] Example 1

[0042] like Figure 1-Figure 6 As shown, a method for panchromatic sharpening of a hyperspectral image based on a spatial-spectral modulation conditional diffusion model comprises the following steps:

[0043] Step 1: Select the corresponding area of ​​the paired large-scale hyperspectral image and panchromatic image, crop it according to the set size, and obtain the training set, validation set and test set through data enhancement processing;

[0044] Specifically: This embodiment selects the Pavia Center data obtained by the ROSIS sensor, which has a size of 1096*715. After processing, there are 102 bands in total. In this example, the original hyperspectral image is used as a high-resolution hyperspectral reference image (GT), and the full-color image is obtained by performing spectral linear combination of the original hyperspectral image.

[0045] Then, the original hyperspectral image was blurred and downsampled at a ratio of 4 to obtain a simulated low-resolution hyperspectral image. The low-resolution hyperspectral image was then bilinearly interpolated to the same size as the full-color image to form a paired large image pair. Then, image blocks of 846*700 and 250*400 were selected for the production of training and validation sets. Specifically, the training and validation sets were cropped at a size of 32 and the specified number of training and validation sets were generated by flipping and rotating data augmentation methods. In addition, image blocks of 128*128 were selected as test sets for model performance testing.

[0046] Step 2 adds noise to the initial data to construct the data required for the prediction network;

[0047] Specifically, set the total time step of noise addition T, in this example T = 2000, randomly sample a time step t in the interval 0-T, and obtain the noise image x corresponding to time step t according to the following noise addition calculation formula t ,This noise and noise image will serve as the training label and input of the noise prediction network.

[0048] The noise calculation formula is:

[0049]

[0050] where ε t represents randomly sampled Gaussian noise, x t represents the noise image corresponding to time step t. α i is a preset weight parameter, whose value range in this example is (0.0001, 0.02), and it increases linearly with the increase of time steps, that is, if the time is infinite tends to 1, then x t Approximate Gaussian noise. x0 represents the difference between the reference high-resolution hyperspectral image and the interpolated low-resolution hyperspectral image, which helps to approximate the noisy hyperspectral image with rich spectral and spatial information to random noise within a limited time step.

[0051] Step 3, constructing a spatial-spectral modulation noise prediction network, wherein the spatial-spectral modulation noise prediction network includes a time step encoder, two conditional encoders, an intermediate layer and a decoder;

[0052] Specifically:

[0053] The temporal encoder (T embedding) inputs an integer t∈(0,T), which is passed through the time step encoder to obtain a one-dimensional vector, which is then embedded into each level of the noise prediction network through a linear layer. Specifically, before being embedded into each level, the vector length is modulated by a linear layer to facilitate addition and fusion with the input features of the current level.

[0054] The various layers refer to the conditional encoder, the middle layer and the decoder.

[0055] The two conditional encoders are the panchromatic conditional injection encoder (encodeP) and the hyperspectral image injection encoder (encodeH), specifically:

[0056] The input of the panchromatic conditional injection encoder is the noise image and the panchromatic image, and the input of the hyperspectral image injection encoder is the noise image and the hyperspectral image. The conditional information of these two different modalities is channel-joined and then injected into each encoder. The specific structure of the encoder is as follows: Figure 4 As shown in the figure, it is mainly composed of two residual convolutional networks. In the forward operation process, it is fused with the time step encoder, and a double downsampling is performed at the end of each encoding layer, which is consistent with the U-net network. The specific operation process is as follows:

[0057] {P i |i=1,2,3,4}=encodeP[cat(x t ,pan)]

[0058] {h i |i=1,2,3,4}=encodeH[cat(x t ,hs)]

[0059] Where P i represents the intermediate layer features of different scales obtained by injecting full color condition into the encoder, h i Represents the intermediate layer features of different scales obtained by injecting hyperspectral conditions into the encoder.

[0060] In the intermediate layer, the outputs of the two conditional encoders are concatenated through the channel dimension and then input into the intermediate layer, and then feature fused and then input into the decoder;

[0061] Among them, the specific structure of the middle layer consists of two residual convolution layers and a global self-attention layer added after the two residual convolution layers. The global self-attention layer is used to strengthen the important information in the features. The specific calculation process is as follows:

[0062] f1=mid(cat(mid(h4),mid(p4)))

[0063] Where mid represents the middle layer, h4 and p4 represent the final outputs of the two encoders respectively.

[0064] The decoder includes four decoding layers and an output layer, wherein the decoding layer includes a spatial-spectral modulation module, two convolution residual layers and a double upsampling layer, and the output layer is used to adjust the output channel to output the correct data format. Specifically, the spatial-spectral modulation module is as follows Figure 5 As shown in the figure, it includes a spectral enhancement attention layer and a local spatial high-frequency enhancement attention layer. The same scale features from two different encoding layers are input into the corresponding enhancement attention layer, and then the modulated conditional features are obtained through a fusion MLP layer, and then input into the decoding convolution layer for feature recovery. The specific operation process is as follows:

[0065] {Q s ,K s ,V s}=local token embedding(p i )

[0066] {Q c ,K c ,V c}=token embedding(h i )

[0067] f i =UP{Resnet(cat(f i-1 ,MLP(cat(V s *σ(Q s *K s T ),V c *σ(Q c T *K c )))),t)}

[0068] Where K s T , R c T Indicates the transpose of the corresponding matrix. Local token embedding means that the features are first segmented without overlap using a fixed window. In this example, when the input p i ∈R 1*1*16*16 ,through

[0069] Through local token embedding, we get Q,K,V∈R 1*64*1*4*1,Compared with normal token embedding, it not only performs global attention calculation, but also local attention calculation, which is conducive to enhancing some local high-frequency information in the full-color image. Combined with the global context mapping, it can more comprehensively use the spatial information of the full-color image to guide the feature decoding process. At the same time, through a spectral attention layer, the high-spectral condition features with rich spectral information are enhanced to avoid excessive redundant spectral information interfering with the feature encoding process. In order to achieve the spatial-spectral modulation effect.

[0070] Step 4, training a spatial-spectral modulation noise prediction network to obtain a trained spatial-spectral modulation noise prediction network;

[0071] Training of the noise prediction network. Each time the training is updated, a batch of data is selected from the training set and the validation set, and the input data is constructed according to step 2. The noise data, the time encoder, and the conditional information (full-color image and hyperspectral image) are input into the noise prediction network. The noise added in step 2 is then compared with the output of the noise prediction network for loss calculation, and the network parameters are iteratively updated through the back propagation algorithm until the loss value converges stably. The calculation process is as follows:

[0072] min||∈ θ (x t ,t,[pan,hs]),ε t ||2

[0073] where ∈ θ Represents the noise predicted by the noise prediction network. ||2 represents the calculation of the second norm.

[0074] Step 5: Use the trained noise prediction network according to the set total time step T, as follows Figure 2 As shown in the Markov chain-based process, a random noise from a normal distribution is used as input x T Then, in each iteration, the current time step t encoding and test condition information [pan, hs], as well as the noise image x are input. t , where the calculation process is as follows, the generation ends at t = 0, and the final sharpening result is obtained;

[0075]

[0076] The setting of α is the same as described in step 2, and z is random Gaussian noise.

[0077] This example uses three convolutional neural network methods and a diffusion model method to compare with this method. The index comparison results and visual comparison results are shown in Table 1 and Figure 6As shown in the figure, this method can improve the hyperspectral panchromatic sharpening effect to a certain extent.

[0078] Table 1 Comparison results of the effect indicators of this method and other classic methods

[0079]

[0080]

[0081] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A hyperspectral image panchromatic sharpening method based on a spatial-spectral modulation conditional diffusion model, characterized in that: include: Obtain paired hyperspectral images and panchromatic images from the database, and obtain training sets, validation sets, and test sets after cropping and data enhancement. Perform noise operation on the initial data to construct the data required for the prediction network; Constructing a spatial-spectral modulation noise prediction network, wherein the spatial-spectral modulation noise prediction network includes a time step encoder, two conditional encoders, an intermediate layer and a decoder; Training a spatial-spectral modulation noise prediction network to obtain a trained spatial-spectral modulation noise prediction network; Hyperspectral image pan-sharpening is performed using the trained spatial-spectral adjusted noise prediction network.

2. The method for pan-sharpening a hyperspectral image according to claim 1, characterized in that: The original hyperspectral images in the database are blurred and downsampled according to the set ratio to obtain simulated low-resolution hyperspectral images. The low-resolution hyperspectral images are bilinearly interpolated to the same size as the full-color images to form paired large-scale image pairs, which are cropped according to the set size and generated by flipping and rotating data augmentation methods to generate a specified number of training sets, validation sets, and test sets.

3. The method for pan-sharpening a hyperspectral image according to claim 1, characterized in that: The noise addition calculation formula is specifically: Among them, ε t represents randomly sampled Gaussian noise, x t represents the noise image corresponding to time step t, α i is a preset weight parameter, which increases linearly with the increase of time steps, and x0 represents the difference between the reference high-resolution hyperspectral image and the interpolated low-resolution hyperspectral image.

4. The method for pan-sharpening a hyperspectral image according to claim 1, characterized in that: The noise addition operation is specifically as follows: setting the total time step T of the noise addition, randomly sampling a time step t in the interval 0-T, and obtaining the noise image x corresponding to the time step t according to the noise addition calculation formula t .

5. The method for pan-sharpening a hyperspectral image according to claim 1, characterized in that: The time step encoder inputs an integer t∈(0,T), obtains a one-dimensional vector through the time step encoder, and embeds it into each level of the noise prediction network through the Linear layer; The two conditional encoders are respectively a full-color conditional injection encoder and a hyperspectral image injection encoder, the input of the full-color conditional injection encoder is a noise image and a full-color image, and the input of the hyperspectral image injection encoder is a noise image and a hyperspectral image; In the intermediate layer, the outputs of the two conditional encoders are concatenated through the channel dimension and then input into the intermediate layer, and then feature fused and then input into the decoder; The decoder includes four decoding layers and an output layer. The decoding layer includes a spatial-spectral modulation module, two convolutional layers and a two-fold upsampling layer. The output layer is used to adjust the output channel and output the correct data format.

6. The method for pan-sharpening a hyperspectral image according to claim 5, characterized in that: The spatial-spectral modulation module includes a spectral enhancement attention layer and a local spatial high-frequency enhancement attention layer.

7. The method for pan-sharpening a hyperspectral image according to any one of claims 1 to 6, characterized in that: The intermediate layer includes two residual convolution layers and a global self-attention layer.

8. The method for pan-sharpening a hyperspectral image according to claim 1, characterized in that: In the training of the spatial-spectral modulated noise prediction network, in each update training, a batch of data is selected from the training set and the validation set, and the data is denoised to obtain noise and noise images. The noise image, time step encoder, full-color image and hyperspectral image are input into the noise prediction network, and then the loss of the noise and the output of the noise prediction network is calculated, and the network parameters are iteratively updated through the back-propagation algorithm until the loss value converges stably.

9. The method for pan-sharpening a hyperspectral image according to claim 1, characterized in that: The method of using the trained spatial-spectral adjustment noise prediction network to perform panchromatic sharpening of the hyperspectral image is specifically as follows: The trained noise prediction network is used to take a random noise as input x from the normal distribution based on the Markov chain process according to the set total time step T. T Then, in each iteration, the current time step t encoding and test condition information [pan, hs], as well as the noise image x are input. t , where the calculation process is as follows, the generation ends at t = 0, and the final sharpening result is obtained; where α t is the weight parameter, and z is random Gaussian noise.

10. The method for pan-sharpening of hyperspectral images according to claim 3, characterized in that: α i The value is (0.0001, 0.02).

Images