Unsupervised spectrum super-resolution reconstruction method and device

Abstract

The invention relates to an unsupervised spectral super-resolution reconstruction method and device. The method comprises the following steps: establishing a spectral super-resolution database with a guide hyperspectral image and an RGB image; point spectrum prior is extracted from the guide hyperspectral image, the RGB image and the point spectrum prior are fused, and a point diagram fusion result is obtained; performing spectral information feature mining on the point diagram fusion result to obtain a preliminary spectral super-resolution result; and performing multi-level fusion on the RGB image and the initial spectral super-resolution result to obtain a target spectral super-resolution image. The beneficial effects of the invention are that the invention provides a spectrum super-resolution model guided by point spectrum prior. The model can overcome the defects that accurate spectral priori is difficult to introduce and spatial information is difficult to inject accurately in an existing method, and high-fidelity high-space and high-spectral-resolution image data are rapidly produced through RGB images.

Description

Technical Field

[0001] The invention relates to the field of remote sensing image processing, and more specifically, to an unsupervised spectral super-resolution reconstruction method and device. Background Art

[0002] Hyperspectral images are valuable in various applications because of their rich spectral information. For example, hyperspectral images not only have irreplaceable advantages in satellite / UAV remote sensing target recognition, crop monitoring, environmental monitoring, mineral exploration and other fields, but also have important application prospects in daily work and life such as drinking water testing, fruit and vegetable freshness testing, and cultural relics identification. However, the acquisition process of hyperspectral images is complex and costly, making it difficult for ordinary people to get them. Spectral super-resolution reconstruction aims to reconstruct RGB images into high-spectral resolution images by "soft supplementing hard".

[0003] At present, a large number of spectral super-resolution reconstruction methods have been proposed, mainly including traditional methods and deep learning-based methods. Traditional methods include methods based on sparse dictionaries, spectral component deviation functions, and maximum a posteriori probability. Although these methods have achieved certain results, they are usually unable to effectively reconstruct high-fidelity images of the spatial spectrum. The spectral super-resolution reconstruction method based on deep learning trains a large amount of sample data and continuously learns the mapping relationship therein in a data-driven way, and finally obtains a high-spectral resolution image, which has certain advantages over traditional methods. In general, in order to ensure the accuracy of spectral super-resolution reconstruction, most existing methods use the whole scene hyperspectral image as prior information and perform spectral super-resolution reconstruction in a supervised form, resulting in the established spectral super-resolution reconstruction model being too dependent on training data, poor generalization performance, and difficult to integrate into imaging devices. In addition. Existing spectral super-resolution reconstruction methods often focus more on the extraction and recovery of spectral information, ignoring the importance of spatial information to the resulting image, resulting in a certain blurring effect on the image. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and to propose an unsupervised spectral super-resolution reconstruction method and device.

[0005] In a first aspect, an unsupervised spectral super-resolution reconstruction method is provided, comprising:

[0006] Step 1: Establish a spectral super-resolution database with guided hyperspectral images and RGB images;

[0007] Step 2: extracting point spectrum priors from the guided hyperspectral image, fusing the RGB image and the point spectrum priors to obtain a point image fusion result;

[0008] Step 3: mining the spectral information features of the point image fusion result to obtain a preliminary spectral super-resolution result;

[0009] Step 4: Perform multi-level fusion on the RGB image and the preliminary spectral super-resolution result to obtain a target spectral super-resolution image.

[0010] Preferably, step 2 comprises:

[0011] Step 2.1: Use K-means clustering to cluster the guided hyperspectral image into n categories, and perform global average pooling on each category separately. The generated sequence is compared with each sequence in the category for spectral loss measurement. This operation is performed n times at a time to extract the n most spectrally representative points as point spectral priors.

[0012] Step 2.2: extracting point spectrum prior features and RGB image features, wherein the point spectrum prior features and the RGB image features have the same dimension, and merging the point spectrum prior features and the RGB image features to obtain a point image fusion result.

[0013] Preferably, in step 3, the features are decomposed into an abundance matrix and an endmember matrix, and the endmember data are used to perform spectral super-resolution operations.

[0014] Preferably, in step 4, the RGB image and the preliminary spectral super-resolution result are input into a multi-stage shuffle-reshuffle Transformer model to obtain a target spectral super-resolution image; the multi-stage shuffle-reshuffle Transformer model includes three Swin Transformers with moving windows and mask operations removed, and PSU operations are embedded in the middle of the three Swin Transformers to rearrange image rows and columns.

[0015] As a preference, it also includes:

[0016] Step 5: Optimize through hybrid loss function.

[0017] Preferably, the hybrid loss function includes a KL divergence loss function, an L1 loss function and a TV loss function.

[0018] In a second aspect, an unsupervised spectral super-resolution reconstruction device is provided, which is used to perform any method described in the first aspect, including:

[0019] Establishing a module for establishing a spectral super-resolution database with guided hyperspectral images and RGB images;

[0020] A first fusion module is used to extract a point spectrum prior from the guide hyperspectral image, fuse the RGB image with the point spectrum prior, and obtain a point image fusion result;

[0021] A mining module is used to mine the spectral information features of the point image fusion result to obtain a preliminary spectral super-resolution result;

[0022] The second fusion module is used to perform multi-level fusion on the RGB image and the preliminary spectral super-resolution result to obtain a target spectral super-resolution image.

[0023] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0024] In a fourth aspect, an electronic device is provided, including:

[0025] Memory, used to store computer programs;

[0026] A processor is used to execute the computer program to implement any method as described in the first aspect.

[0027] The beneficial effects of the present invention are:

[0028] 1. The present invention constructs an unsupervised spectral super-resolution reconstruction framework and proposes a spectral super-resolution model guided by point spectral priors. This model can overcome the defects of existing methods that are difficult to introduce accurate spectral priors and difficult to accurately inject spatial information, and quickly produce high-fidelity high-spatial and high-spectral resolution image data with RGB images.

[0029] 2. The unsupervised spectral super-resolution framework constructed by the present invention can inject accurate spectral priors and spatial information features to accurately guide the super-resolution of RGB images to high-fidelity hyperspectral images. In addition, in order to ensure the credibility of the fusion results generated by the network, a hybrid loss function is introduced to optimize the network, focusing on constraining spectral and spatial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the unsupervised spectral super-resolution method based on point spectrum prior guidance;

[0031] Figure 2 It is a schematic diagram of the method for obtaining a priori point spectrum;

[0032] Figure 3 It is a schematic diagram of the structure of the Cross Attention module and the SWMS module;

[0033] Figure 4 It is a structural diagram of the PSU module;

[0034] Figure 5This is a comparison chart of spectral errors on the ICVL dataset;

[0035] Figure 6 This is a comparison chart of spectral errors on the DFC2018 Houston dataset;

[0036] Figure 7 This is a comparison chart of spectral errors on the TG1HRSSC dataset. DETAILED DESCRIPTION

[0037] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0038] Embodiment 1:

[0039] In order to achieve high-fidelity spectral super-resolution imaging of remote sensing images, the present application provides an unsupervised spectral super-resolution reconstruction method. First, the RGB image and the hyperspectral image are preprocessed and data augmented. Secondly, the point spectral information of the guided hyperspectral image is extracted to generate a point spectral prior, and the extracted point spectral prior and the RGB image are fused to obtain a feature map with accurate spectral information. Next, a super-resolution module based on spectral decoupling is used to perform spectral super-resolution on the feature map to obtain a preliminary super-resolution result. Finally, the generated preliminary super-resolution result is fused with the RGB image to achieve accurate spatial information injection.

[0040] Specifically, Figure 1 As shown, the method includes:

[0041] Step 1: Establish a spectral super-resolution database with guided hyperspectral images and RGB images.

[0042] Step 2: extracting a point spectrum prior from the guided hyperspectral image, fusing the RGB image with the point spectrum prior, and obtaining a point image fusion result.

[0043] Step 2 includes:

[0044] Step 2.1: Use K-means clustering to cluster the guided hyperspectral image into n categories, and perform global average pooling on each category separately. The generated sequence is compared with each sequence in the category for spectral loss measurement. This operation is performed n times at a time to extract the n most spectrally representative points as point spectral priors, which can be expressed as:

[0045] S n =SAM min (K n (Y g),GAP(K n (Y g )))

[0046] S=Concate(S1,S2,…,S N )

[0047] Among them, K n represents the pixel area of ​​the nth class after the K-means operation, GAP represents the global average pooling, SAM min It means calculating the spectral angle loss of each pixel pair and returning the pixel with the minimum loss, S n Represents the spectral point of the n-th pixel area, Y g Represents the guided hyperspectral image, Concat represents the concatenation operation of the input features in the channel dimension, and S represents the complete point spectrum prior.

[0048] Step 2.2: extracting point spectrum prior features and RGB image features, wherein the point spectrum prior features and the RGB image features have the same dimension, and merging the point spectrum prior features and the RGB image features to obtain a point image fusion result.

[0049] Specifically, for the extracted point spectrum prior, since it is similar to a text sequence, the text-image fusion strategy is referred to, and the point spectrum prior and RGB image are converted into features of the same dimension through operations such as feature extraction, and then fused through the CrossAttention module. The CrossAttention module is as follows: Figure 3 As shown in (a), the RGB image at this time fully absorbs the spectral information in the spectral prior. Among them, the Linear layer and Relu activation function are used to extract features for the point spectral prior, and 3×3 convolution, Relu activation function and Maxpool layer are used to extract features for the RGB image, which is expressed as:

[0050] F p =M l (M r (M l (S)))

[0051] F RGB =M CMR (M l (X)

[0052] F attention =M A (Concate(F p ,F RGB ))

[0053] Among them, X represents the RGB image, M l , M r and MCMR Represents the Linear layer, Relu activation function and CMR layer respectively. The CMR module is composed of a convolutional layer, a maximum pooling layer and a Relu activation function in sequence. A represents the Cross Attention layer, F p ,F RGB are the point spectrum prior and the features of the RGB image after feature extraction, F Attention Yes F p and F RGB After Concat, pass M A characteristics.

[0054] Step 3: perform spectral information feature mining on the point map fusion result to obtain a preliminary spectral super-resolution result.

[0055] like Figure 2 As shown in the spectral super-resolution module based on decoupling, a spectral super-resolution module is established. The spectral super-resolution module based on decoupling decomposes the features into an abundance matrix and an endmember matrix. The abundance matrix represents the weight of each spectral component, and the endmember matrix contains the basic spectral information of the feature. This application uses endmember data to perform spectral super-resolution operations, which can effectively utilize the spectral information in the feature and has strong interpretability, expressed as:

[0056]

[0057] in, and Represent the 1st, 2nd, 3rd and 4th Linear layers respectively, M s represents the Softmax layer; A represents the abundance matrix, represents the rough hyperspectral image generated initially, and the decoder is M l The weight is the end metadata.

[0058] This method not only improves the spectral resolution, but also has strong interpretability, which helps to understand the spatial distribution and interrelationships of spectral components.

[0059] Step 4: Perform multi-level fusion on the RGB image and the preliminary spectral super-resolution result to obtain a target spectral super-resolution image.

[0060] In step 4, the moving window of the Swin Transformer will reduce the boundary window, and the mask operation will inhibit the interaction of the boundary, resulting in weakened self-attention. Therefore, a multi-stage shuffle-reshuffle Transformer is proposed. First, SWMS is introduced, that is, the Swin Transformer without the moving window and mask operation, as shown in Figure 3(b) and embedding PSU operation in the middle of the three SWMS (e.g. Figure 4 As shown in the figure, the image rows and columns are rearranged to achieve efficient global attention and retain boundary information. The window sizes of the three SWMS are 8, 4, and 2, respectively, which imitate the feature pyramid structure to enhance the extraction of spatial information, thereby processing spatial details more finely while maintaining global information, as shown in the figure. Figure 2 The multi-stage shuffle-reshuffle Transformer is shown in Figure 1, which is represented as follows:

[0061]

[0062] F2=F Unshuffle (SWMS4(F1))

[0063] Z=SWMS2(F2)

[0064] Among them, F1 and F2 are the feature maps generated in the intermediate stage of step 4, and Z is the final high-fidelity hyperspectral image; SWMS w represents the SWMS with a window size of w, where SWMS is the Swin Transformer without window rearrangement and masking operations; F Shuffle and F Unshuffle There are two steps of PSU, where PSU is a pixel rearrangement operation.

[0065] Embodiment 2:

[0066] Based on Example 1, Example 2 of the present application provides a more specific unsupervised spectral super-resolution reconstruction method, including:

[0067] Step 1: Establish a spectral super-resolution database with guided hyperspectral images and RGB images.

[0068] Specifically, the basic data sets used in Example 2 of the present application are four existing publicly available data sets: the Salinas data set is an image of the Salinas Valley in California, USA. The corrected image contains 204 channels, covering the spectral range of 0.4-2.5μm. The ICVL data set consists of 201 images, each of which contains 31 spectral channels. We randomly selected 184 images as training sets and 6 images as test sets. Before the experiment, each image was cropped to (31,256,256). The DFC2018 Houston data set contains 50 spectral channels. In this experiment, 92 images were randomly selected as training sets and 6 images were used as test sets. The TG1HRSSC data set consists of three types of data: panchromatic images, visible and near-infrared images of 54 effective bands, and short-wave infrared images of 52 effective bands. In this experiment, we selected visible and near-infrared images, randomly selected 40 images as training sets, and 6 images as test sets. As an embodiment, the present invention uses computer software Pycharm and Pytorch framework based on NVIDIA GeForce RTX 3090 GPU to implement automatic operation process.

[0069] In addition, this embodiment also preprocesses the data set, including: cropping the spatial dimension of the RGB image to 256×256, and performing normalization and other processing to achieve data set expansion, so as to avoid overfitting and gradient explosion problems caused by too large original data and small data volume during deep learning training. The guided hyperspectral shadow is spectrally downsampled to 31, 50 and 54 respectively to match the number of channels of the spectral super-resolution RGB images of the three data sets.

[0070] Step 2: extracting point spectrum priors from the guided hyperspectral image, fusing the RGB image with the point spectrum priors, and obtaining a point image fusion result.

[0071] Step 3: perform spectral information feature mining on the point map fusion result to obtain a preliminary spectral super-resolution result.

[0072] Step 4: Perform multi-level fusion on the RGB image and the preliminary spectral super-resolution result to obtain a target spectral super-resolution image.

[0073] Step 5: Optimize through the hybrid loss function.

[0074] Specifically, in unsupervised training, multiple loss functions are used to constrain, thereby ensuring the generation of high-fidelity hyperspectral images. The KL divergence loss function is used to ensure that the abundance map is discrete, making the abundance distribution of each pixel clearer and more specific, in line with physical properties, and helpful for parsing and restoring the distribution of material components in real scenes. The reconstruction loss ensures that the generated hyperspectral image is similar to the RGB image after the spectral response function, ensuring that the reconstructed image maintains authenticity and detail fidelity within the visible light range. The TV loss can ensure that adjacent bands of the generated hyperspectral image are similar, thereby reducing mutations between bands and improving the continuity and stability of the hyperspectral image. The synergistic effect of these losses not only helps the model obtain more accurate hyperspectral reconstruction, but also effectively improves the physical rationality and detail fidelity of the generated image, making the generated hyperspectral image closer to reality in both spatial and spectral dimensions.

[0075] The expression of the mixed loss function is:

[0076]

[0077] L R =||XZ·R||2

[0078]

[0079] L total =α·L K +β·L R +γ·L TV

[0080] Among them, L K , L R and L TV represents KL loss, reconstruction loss and TV loss, α, β and γ represent adjustable parameters, ∈ represents a small constant, R represents the spectral response function, C and HW represent the number of channels and the height-width product of the image, and L total Represents the total loss function.

[0081] In addition, Example 2 of the present application experimentally verifies the above method:

[0082] exist Figure 5 , Figure 6 and Figure 7The experimental results on the ICVL, DFC2018 Houston and TG1HRSSC datasets are shown respectively. The first column shows the pseudo-color image generated by HSI, and the remaining five columns show the error map between the image generated by the network and the label. In the error map, the bluer color indicates a smaller difference from the true situation. The spectral super-resolution images of the method proposed in the present invention are compared with those of five state-of-the-art methods. Previous methods have limitations in recovering spectral and spatial details, resulting in artifacts or spots, and cannot accurately reconstruct image edges. In addition, their overall restoration quality is insufficient, lacking global consistency and continuity of space and spectrum. In contrast, the method proposed in the present invention overcomes these shortcomings and achieves high-quality super-resolution with consistent spectrum and spatial continuity, benefiting from more accurate spectral priors and effective supplementation of spatial information.

[0083] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0084] Embodiment 3:

[0085] On the basis of Example 1, Example 3 of the present application provides an unsupervised spectral super-resolution reconstruction device, comprising:

[0086] Establishing a module for establishing a spectral super-resolution database with guided hyperspectral images and RGB images;

[0087] A first fusion module is used to extract a point spectrum prior from the guide hyperspectral image, fuse the RGB image with the point spectrum prior, and obtain a point image fusion result;

[0088] A mining module is used to mine the spectral information features of the point image fusion result to obtain a preliminary spectral super-resolution result;

[0089] The second fusion module is used to perform multi-level fusion on the RGB image and the preliminary spectral super-resolution result to obtain a target spectral super-resolution image.

[0090] Specifically, the device provided in this embodiment is a device corresponding to the method provided in Example 1. Therefore, the parts in this embodiment that are the same or similar to those in Example 1 can be referenced to each other and will not be repeated in this application.

Claims

1. An unsupervised spectral super-resolution reconstruction method, characterized in that: include: Step 1: Establish a spectral super-resolution database with guided hyperspectral images and RGB images; Step 2: extracting point spectrum priors from the guided hyperspectral image, fusing the RGB image and the point spectrum priors to obtain a point image fusion result; Step 3: mining the spectral information features of the point image fusion result to obtain a preliminary spectral super-resolution result; Step 4: Perform multi-level fusion on the RGB image and the preliminary spectral super-resolution result to obtain a target spectral super-resolution image.

2. The unsupervised spectral super-resolution reconstruction method according to claim 1, characterized in that: Step 2 includes: Step 2.1: Use K-means clustering to cluster the guided hyperspectral image into n categories, and perform global average pooling on each category separately. The generated sequence is compared with each sequence in the category for spectral loss measurement. This operation is performed n times at a time to extract the n most spectrally representative points as point spectral priors. Step 2.2: extracting point spectrum prior features and RGB image features, wherein the point spectrum prior features and the RGB image features have the same dimension, and merging the point spectrum prior features and the RGB image features to obtain a point image fusion result.

3. The unsupervised spectral super-resolution reconstruction method according to claim 2, characterized in that: In step 3, the features are decomposed into an abundance matrix and an endmember matrix, and the endmember data are used to perform spectral super-resolution operations.

4. The unsupervised spectral super-resolution reconstruction method according to claim 3, characterized in that: In step 4, the RGB image and the preliminary spectral super-resolution result are input into a multi-stage shuffle-reshuffle Transformer model to obtain a target spectral super-resolution image; the multi-stage shuffle-reshuffle Transformer model includes three Swin Transformers with moving windows and mask operations removed, and a PSU operation is embedded in the middle of the three Swin Transformers to rearrange image rows and columns.

5. The unsupervised spectral super-resolution reconstruction method according to claim 4, characterized in that: Also includes: Step 5: Optimize through hybrid loss function.

6. The unsupervised spectral super-resolution reconstruction method according to claim 5, characterized in that: In step 5, the hybrid loss function includes a KL divergence loss function, an L1 loss function, and a TV loss function.

7. An unsupervised spectral super-resolution reconstruction device, characterized in that: Used to perform the method according to any one of claims 1 to 6, comprising: Establishing a module for establishing a spectral super-resolution database with guided hyperspectral images and RGB images; A first fusion module is used to extract a point spectrum prior from the guide hyperspectral image, fuse the RGB image with the point spectrum prior, and obtain a point image fusion result; A mining module is used to mine the spectral information features of the point image fusion result to obtain a preliminary spectral super-resolution result; The second fusion module is used to perform multi-level fusion on the RGB image and the preliminary spectral super-resolution result to obtain a target spectral super-resolution image.

8. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 6.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 6.

Images