An unsupervised spectral super-resolution reconstruction method and apparatus

Abstract

This invention relates to an unsupervised spectral super-resolution reconstruction method and apparatus, comprising: establishing a spectral super-resolution database with a guiding hyperspectral image and an RGB image; extracting point spectral priors from the guiding hyperspectral image; fusing the RGB image and the point spectral priors to obtain a point-map fusion result; mining spectral information features from the point-map fusion result to obtain a preliminary spectral super-resolution result; and performing multi-level fusion of the RGB image and the preliminary spectral super-resolution result to obtain a target spectral super-resolution image. The beneficial effects of this invention are: it proposes a spectral super-resolution model guided by point spectral priors. This model overcomes the shortcomings of existing methods in introducing accurate spectral priors and accurately injecting spatial information, enabling the rapid generation of high-fidelity, high-spatial- and high-spectral-resolution image data from RGB images.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing, and more specifically, to an unsupervised spectral super-resolution reconstruction method and apparatus. Background Technology

[0002] Hyperspectral images are invaluable in various applications due to their rich spectral information. They offer irreplaceable advantages in satellite / UAV remote sensing target identification, crop monitoring, environmental monitoring, and mineral exploration. Furthermore, hyperspectral images hold significant promise for everyday applications such as drinking water testing, fruit and vegetable freshness detection, and artifact identification. However, acquiring hyperspectral images is complex and costly, making them difficult for ordinary users to access. Spectral super-resolution reconstruction aims to reconstruct RGB images into hyperspectral resolution images through a "software-to-hardware" approach.

[0003] Currently, numerous spectral super-resolution reconstruction methods have been proposed, mainly including traditional methods and deep learning-based methods. Traditional methods include those based on sparse dictionaries, spectral component bias functions, and maximum a posteriori probability. While these methods have achieved some success, they typically cannot effectively reconstruct high-fidelity images of the spatial spectrum. Deep learning-based spectral super-resolution reconstruction methods train on a large amount of sample data, using data-driven learning to continuously learn the mapping relationships, ultimately obtaining a high-spectral-resolution image, which has certain advantages over traditional methods. Overall, to ensure the accuracy of spectral super-resolution reconstruction, most existing methods use the entire hyperspectral image as prior information, performing spectral super-resolution reconstruction in a supervised manner. This results in spectral super-resolution reconstruction models that are overly reliant on training data, exhibiting poor generalization performance and being difficult to integrate into imaging devices. Furthermore, existing spectral super-resolution reconstruction methods often focus more on the extraction and recovery of spectral information, neglecting the importance of spatial information in the resulting image, leading to certain blurring effects. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an unsupervised spectral super-resolution reconstruction method and apparatus.

[0005] Firstly, an unsupervised spectral super-resolution reconstruction method is provided, including:

[0006] Step 1: Establish a spectral super-resolution database with guided hyperspectral and RGB images;

[0007] Step 2: Extract point spectral priors from the guided hyperspectral image, fuse the RGB image and the point spectral priors, and obtain the point image fusion result;

[0008] Step 3: Perform spectral information feature mining on the point map fusion result to obtain preliminary spectral super-resolution results;

[0009] Step 4: Perform multi-level fusion of the RGB image and the preliminary spectral super-resolution result to obtain the target spectral super-resolution image.

[0010] Preferably, step 2 includes:

[0011] Step 2.1: Use K-means clustering to cluster the guided hyperspectral image into n classes, and perform global average pooling on each class separately. Measure the spectral loss of the generated sequence with each sequence in the class. Perform this operation n times to extract the n most spectrally representative points as point spectral priors.

[0012] Step 2.2: Extract the point spectral prior features and RGB image features. The point spectral prior features and RGB image features have the same dimension. Then, fuse the point spectral prior features and RGB image features to obtain the point image fusion result.

[0013] Preferably, in step 3, the features are decomposed into an abundance matrix and an endmember matrix, and the endmember data is used to perform spectral super-resolution operations.

[0014] Preferably, in step 4, the RGB image and the preliminary spectral super-resolution result are input into a multi-level shuffle-reshuffle Transformer model to obtain the target spectral super-resolution image; the multi-level shuffle-reshuffle Transformer model includes three Swing Transformers with the moving window and mask operations removed, and a PSU operation is embedded between the three Swing Transformers to rearrange the image rows and columns.

[0015] As a preferred option, it also includes:

[0016] Step 5: Optimize using a hybrid loss function.

[0017] Preferably, the hybrid loss function includes the KL divergence loss function, the L1 loss function, and the TV loss function.

[0018] In a second aspect, an unsupervised spectral super-resolution reconstruction apparatus is provided for performing any of the methods described in the first aspect, including:

[0019] A module is established to create a spectral super-resolution database with guided hyperspectral and RGB images;

[0020] The first fusion module is used to extract point spectral priors from the guided hyperspectral image, fuse the RGB image and the point spectral priors, and obtain a point image fusion result.

[0021] The mining module is used to mine spectral information features from the point map fusion results to obtain preliminary spectral super-resolution results;

[0022] The second fusion module is used to perform multi-level fusion of the RGB image and the preliminary spectral super-resolution result to obtain the target spectral super-resolution image.

[0023] Thirdly, a computer storage medium is provided, wherein a computer program is stored therein; when the computer program is run on a computer, the computer causes the computer to perform any of the methods described in the first aspect.

[0024] Fourthly, an electronic device is provided, comprising:

[0025] Memory, used to store computer programs;

[0026] A processor for executing the computer program to implement the method as described in any of the first aspects.

[0027] The beneficial effects of this invention are:

[0028] 1. This invention constructs an unsupervised spectral super-resolution reconstruction framework and proposes a spectral super-resolution model guided by point spectral priors. This model overcomes the shortcomings of existing methods that struggle to introduce accurate spectral priors and accurately inject spatial information, enabling the rapid generation of high-fidelity, high-spatial-resolution image data from RGB images.

[0029] 2. The unsupervised spectral super-resolution framework constructed in this invention can inject accurate spectral priors and spatial information features, accurately guiding RGB images to be super-resolution into high-fidelity hyperspectral images. Furthermore, to ensure the credibility of the fusion results generated by the network, a hybrid loss function is introduced to optimize the network, focusing on constraining spectral and spatial information. Attached Figure Description

[0030] Figure 1 The flowchart shows an unsupervised super-resolution method based on point spectral priors.

[0031] Figure 2 A schematic diagram of a method for obtaining point spectrum priors;

[0032] Figure 3 A schematic diagram of the Cross Attention module and the SWMS module;

[0033] Figure 4 This is a schematic diagram of the PSU module.

[0034] Figure 5A comparison of spectral errors on the ICVL dataset;

[0035] Figure 6 A comparison of spectral errors on the DFC2018 Houston dataset;

[0036] Figure 7 This is a comparison chart of spectral errors on the TG1HRSSC dataset. Detailed Implementation

[0037] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0038] Example 1:

[0039] To achieve high-fidelity spectral super-resolution imaging of remote sensing images, this application provides an unsupervised spectral super-resolution reconstruction method. First, preprocessing and data augmentation operations are performed on the RGB and hyperspectral images. Second, point spectral information is extracted from the guiding hyperspectral image to generate point spectral priors. The extracted point spectral priors are then fused with the RGB image to obtain a feature map with accurate spectral information. Next, a spectral decoupling-based super-resolution module is used to perform spectral super-resolution on the feature map to obtain preliminary super-resolution results. Finally, the generated preliminary super-resolution results are fused with the RGB image to achieve accurate spatial information injection.

[0040] Specifically, such as Figure 1 As shown, the method includes:

[0041] Step 1: Establish a spectral super-resolution database with guided hyperspectral and RGB images.

[0042] Step 2: Extract point spectral priors from the guided hyperspectral image, fuse the RGB image and the point spectral priors, and obtain the point image fusion result.

[0043] Step 2 includes:

[0044] Step 2.1: Use K-means clustering to cluster the guided hyperspectral image into n classes, and perform global average pooling on each class individually. Measure the spectral loss of the generated sequences with each sequence within the same class. This operation is performed n times to extract the n most spectrally representative points as point spectral priors, expressed as:

[0045] S n =SAM min (K n (Y g),GAP(K n (Y g )))

[0046] S = Concate(S1, S2, ..., S N )

[0047] Among them, K n This represents the pixel region of class n after K-means operation, GAP represents global average pooling, and SAM... min This represents calculating the spectral angle loss for each pixel pair and returning the pixel with the minimum loss, S n Y represents the spectral points of the nth pixel region. g This represents the guided hyperspectral image, Concat represents the concatenation operation of input features in the channel dimension, and S represents the complete point spectral prior.

[0048] Step 2.2: Extract the point spectral prior features and RGB image features. The point spectral prior features and RGB image features have the same dimension. Then, fuse the point spectral prior features and RGB image features to obtain the point image fusion result.

[0049] Specifically, for the extracted point spectral prior, since it resembles a text sequence, a text-image fusion strategy is referenced. The point spectral prior and the RGB image are transformed into features of the same dimension through feature extraction and other operations, and then fused using the CrossAttention module. The CrossAttention module is as follows: Figure 3 As shown in (a), the RGB image at this time fully absorbs the spectral information from the spectral prior. Specifically, for the point-based spectral prior, a Linear layer and a ReLU activation function are used to extract features. For the RGB image, a 3×3 convolution, a ReLU activation function, and a Maxpool layer are used for feature extraction, as shown below:

[0050] F p =M l (M r (M l (S)))

[0051] F RGB =M CMR (M l (X))

[0052] F attention =M A (Concate(F p ,F RGB ))

[0053] Where X represents the RGB image, M l M r and MCMR These represent the Linear layer, ReLU activation function, and CMR layer, respectively. The CMR module consists of a convolutional layer, a max-pooling layer, and a ReLU activation function in sequence. M A F represents the Cross Attention layer. p ,F RGB These are the point spectral prior and the features extracted from the RGB image, respectively, F. Attention It is F p and F RGB After Concat, then through M A Its characteristics.

[0054] Step 3: Perform spectral information feature mining on the point map fusion result to obtain preliminary spectral super-resolution results.

[0055] like Figure 2 As shown in the decoupled spectral super-resolution module, a spectral super-resolution module is established. The decoupled spectral super-resolution module decomposes features into an abundance matrix and an endmember matrix. The abundance matrix represents the weights of each spectral component, while the endmember matrix contains the basic spectral information of the features. This application uses endmember data for spectral super-resolution operations, which can effectively utilize the spectral information in the features and has strong interpretability, as shown below:

[0056]

[0057] in, and M represents the 1st, 2nd, 3rd, and 4th Linear layers, respectively. s A represents the Softmax layer; A represents the abundance matrix. This represents the initially generated coarse hyperspectral image; the decoder is M. l The weight is the end metadata.

[0058] This method not only improves spectral resolution but also has strong interpretability, helping to understand the spatial distribution and interrelationships of spectral components.

[0059] Step 4: Perform multi-level fusion of the RGB image and the preliminary spectral super-resolution result to obtain the target spectral super-resolution image.

[0060] In step 4, the moving window of the Swing Transformer shrinks the bounding window, and the masking operation suppresses boundary interactions, leading to weakened self-attention. Therefore, a multi-level shuffle-reshuffle Transformer is proposed. First, SWMS is introduced, which is a Swing Transformer without the moving window and masking operations, such as... Figure 3As shown in (b), and embedding PSU operations between the three SWMS (such as...). Figure 4 As shown, the image rows and columns are rearranged to achieve efficient global attention while preserving boundary information. The window sizes of the three SWMS are 8, 4, and 2, respectively, mimicking the feature pyramid structure to enhance spatial information extraction. This allows for more detailed processing of spatial details while maintaining global information. Figure 2 The multi-level shuffle-reshuffle Transformer is shown as follows:

[0061]

[0062] F2 = F Unshuffle (SWMS4(F1))

[0063] Z = SWMS2(F2)

[0064] Where F1 and F2 are feature maps generated in the intermediate stage of step 4, and Z is the final high-fidelity hyperspectral image; SWMS w This represents a SWMS with a window size of w, where SWMS is a Swing Transformer that removes window reflow and masking operations; F Shuffle and F Unshuffle This consists of two steps in PSU, where PSU is the pixel rearrangement operation.

[0065] Example 2:

[0066] Based on Example 1, Example 2 of this application provides a more specific unsupervised spectral super-resolution reconstruction method, including:

[0067] Step 1: Establish a spectral super-resolution database with guided hyperspectral and RGB images.

[0068] Specifically, the base datasets used in Example 2 of this application are four publicly available datasets: the Salinas dataset, which contains images of the Salinas Valley in California, USA. The corrected images contain 204 channels, covering the 0.4-2.5μm spectral range. The ICVL dataset consists of 201 images, each containing 31 spectral channels. We randomly selected 184 images as the training set and 6 images as the test set. Before the experiment, each image was cropped to (31, 256, 256). The DFC2018 Houston dataset contains 50 spectral channels. In this experiment, 92 images were randomly selected as the training set and 6 images as the test set. The TG1HRSSC dataset consists of three classes of data: panchromatic images, visible and near-infrared images with 54 effective bands, and short-wave infrared images with 52 effective bands. In this experiment, we selected visible and near-infrared images, randomly selecting 40 images as the training set and 6 images as the test set. As one embodiment, the present invention uses the computer software PyCharm and the PyTorch framework based on the NVIDIA GeForce RTX 3090 GPU to implement an automated running process.

[0069] Furthermore, this embodiment preprocesses the dataset, including cropping the spatial dimension of the RGB images to 256×256 and performing normalization to expand the dataset and avoid overfitting and gradient explosion problems that may occur during deep learning training due to the original data being too large or the data volume being too small. The guiding hyperspectral images were downsampled to 31, 50, and 54 channels respectively to match the number of channels in the spectral super-resolution RGB images of the three datasets.

[0070] Step 2: Extract point spectral priors from the guided hyperspectral image, fuse the RGB image and the point spectral priors, and obtain the point image fusion result.

[0071] Step 3: Perform spectral information feature mining on the point map fusion result to obtain preliminary spectral super-resolution results.

[0072] Step 4: Perform multi-level fusion of the RGB image and the preliminary spectral super-resolution result to obtain the target spectral super-resolution image.

[0073] Step 5: Optimize using a hybrid loss function.

[0074] Specifically, in unsupervised training, multiple loss functions are employed to constrain and ensure the generation of high-fidelity hyperspectral images. The KL divergence loss function ensures the abundance map is discrete, making the abundance distribution of each pixel clearer and more definite, conforming to physical properties, and aiding in the analysis and reconstruction of the material composition distribution in real-world scenes. The reconstruction loss ensures that the generated hyperspectral image is similar to the RGB image processed by the spectral response function, ensuring the reconstructed image maintains realism and detail fidelity within the visible light range. The TV loss ensures that adjacent bands of the generated hyperspectral image are similar, thereby reducing abrupt changes between bands and improving the continuity and stability of the hyperspectral image. The synergistic effect of these losses not only helps the model obtain more accurate hyperspectral reconstructions but also effectively improves the physical plausibility and detail fidelity of the generated image, making the generated hyperspectral image closer to reality in both spatial and spectral dimensions.

[0075] The expression for the hybrid loss function is:

[0076]

[0077] L R =||XZ·R||2

[0078]

[0079] L total =α·L K +β·L R +γ·L TV

[0080] Among them, L K L R and L TV Represents KL loss, reconstruction loss, and TV loss; α, β, and γ represent adjustable parameters; ∈ represents a small constant; R represents the spectral response function; C and HW represent the number of image channels and the product of height and width; L total This represents the total loss function.

[0081] Furthermore, Embodiment 2 of this application experimentally verified the above method:

[0082] exist Figure 5 , Figure 6 and Figure 7Experimental results were presented on the ICVL, DFC2018 Houston, and TG1HRSSC datasets. The first column represents the pseudo-color image generated by HSI, and the remaining five columns show the error map between the network-generated image and the label. In the error map, bluer colors indicate smaller differences from the real image. The proposed method was compared with five state-of-the-art methods for spectral super-resolution images. Previous methods have limitations in restoring spectral and spatial details, resulting in artifacts or blotches and failing to accurately reconstruct image edges. Furthermore, their overall restoration quality is insufficient, lacking global consistency and continuity in both space and spectral dimensions. In contrast, the proposed method overcomes these shortcomings, achieving high-quality super-resolution with consistent spectral and spatial continuity, benefiting from more accurate spectral priors and effective supplementation of spatial information.

[0083] It should be noted that the parts in this embodiment that are the same as or similar to those in Embodiment 1 can be referred to each other, and will not be repeated in this application.

[0084] Example 3:

[0085] Based on Example 1, Example 3 of this application provides an unsupervised spectral super-resolution reconstruction device, comprising:

[0086] A module is established to create a spectral super-resolution database with guided hyperspectral and RGB images;

[0087] The first fusion module is used to extract point spectral priors from the guided hyperspectral image, fuse the RGB image and the point spectral priors, and obtain a point image fusion result.

[0088] The mining module is used to mine spectral information features from the point map fusion results to obtain preliminary spectral super-resolution results;

[0089] The second fusion module is used to perform multi-level fusion of the RGB image and the preliminary spectral super-resolution result to obtain the target spectral super-resolution image.

[0090] Specifically, the device provided in this embodiment is the same as the device corresponding to the method provided in embodiment 1. Therefore, the parts in this embodiment that are the same as or similar to those in embodiment 1 can be referred to each other, and will not be described again in this application.

Claims

1. An unsupervised spectral super-resolution reconstruction method, characterized in that, include: Step 1: Establish a spectral super-resolution database with guided hyperspectral and RGB images; Step 2: Extract point spectral priors from the guided hyperspectral image, fuse the RGB image and the point spectral priors to obtain the point image fusion result; Step 2 includes: Step 2.1: Use K-means clustering to cluster the guided hyperspectral image into n classes, and perform global average pooling on each class separately. Measure the spectral loss of the generated sequence with each sequence in the class. Perform this operation n times to extract the n most spectrally representative points as point spectral priors. Step 2.2: Extract the point spectral prior features and RGB image features. The point spectral prior features and RGB image features have the same dimension. Then, fuse the point spectral prior features and RGB image features to obtain the point image fusion result. Step 3: Perform spectral information feature mining on the point map fusion result to obtain preliminary spectral super-resolution results; in Step 3, the features are decomposed into abundance matrix and endmember matrix, and the endmember data is used to perform spectral super-resolution operation; Step 4: Perform multi-level fusion on the RGB image and the preliminary spectral super-resolution result to obtain the target spectral super-resolution image; In step 4, the RGB image and the preliminary spectral super-resolution result are input into a multi-level shuffle-reshuffle Transformer model to obtain the target spectral super-resolution image; The multi-level shuffle-reshuffle Transformer model includes three Swing Transformers with the moving window and mask operations removed, and a PSU operation is embedded between the three Swing Transformers to rearrange the image rows and columns; Step 5: Optimize using a hybrid loss function; in Step 5, the hybrid loss function includes the KL divergence loss function, the L1 loss function, and the TV loss function.

2. An unsupervised spectral super-resolution reconstruction device, characterized in that, To perform the method of claim 1, comprising: A module is established to create a spectral super-resolution database with guided hyperspectral and RGB images; The first fusion module is used to extract point spectral priors from the guided hyperspectral image, fuse the RGB image and the point spectral priors, and obtain a point image fusion result. The mining module is used to mine spectral information features from the point map fusion results to obtain preliminary spectral super-resolution results; The second fusion module is used to perform multi-level fusion of the RGB image and the preliminary spectral super-resolution result to obtain the target spectral super-resolution image.

3. A computer storage medium, characterized in that, The computer storage medium stores a computer program; when the computer program is run on the computer, it causes the computer to perform the method of claim 1.

4. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as claimed in claim 1.

Images