A hyperspectral image denoising method based on a sparse TRANSFORMER structure

Abstract

The application provides a hyperspectral image denoising method based on a sparse TRANSFORMER structure, which utilizes a sparse spatial-spectral TRANSFORMER denoising network to jointly learn the spatial-spectral features of a hyperspectral image, thereby helping with denoising. In the spatial dimension of the hyperspectral image, in order to deal with the problem of excessive smoothing, a local convolution module is introduced to enhance the spatial high-frequency feature details; meanwhile, a Top-k sparse self-attention mechanism is proposed in the spectral dimension, which adaptively selects the most relevant spectral channel information for feature interaction, thereby effectively capturing the correlation of spectral features between different spectral bands. Through the joint learning of the spatial-spectral features of the hyperspectral image, the application can perform efficient and high-quality denoising and repairing operations on the input noisy hyperspectral image according to the trained network parameter weights.

Description

Technical Field

[0001] This invention relates to a hyperspectral image denoising method based on a sparse TRANSFORMER structure, belonging to the field of image processing technology. Background Technology

[0002] Hyperspectral images offer finer spectral resolution compared to color images, providing approximate spectral curves of reflectance from objects at various spatial locations within a scene, which can be used to identify land cover types. Therefore, hyperspectral images have wide applications in remote sensing, material identification, agriculture, medical diagnosis, and classification. However, during spectrometer acquisition, hyperspectral images are often affected by various types of noise due to low light levels, photon effects, atmospheric interference, and sensor circuitry, negatively impacting subsequent analysis. Therefore, denoising hyperspectral images is a crucial preprocessing step for further analysis and applications.

[0003] Traditional hyperspectral denoising methods primarily rely on a priori regularized denoising models derived from the inherent physical properties of hyperspectral images. These models mainly utilize low-rank priors, nonlocal similarity, spatial-spectral correlation, and global spectral correlation. While these methods effectively leverage the latent spatial and spectral features of hyperspectral images and achieve good denoising results, their final denoising performance still depends on the degree to which manually set prior information matches the inherent physical properties of the hyperspectral image. Furthermore, this prior-model-based hyperspectral image denoising approach requires multiple iterative optimization processes, resulting in high time complexity.

[0004] To address the issue of excessively long denoising times in traditional methods, existing technologies utilize learned neural networks to rapidly denoise noisy hyperspectral images while maintaining good denoising performance. Furthermore, by capturing the correlation between spatial and spectral features of hyperspectral images through neural networks, the denoising effect on hyperspectral images is further enhanced.

[0005] However, convolutional neural network-based methods are insufficient for extracting global features from hyperspectral images. The spatial-spectral coupling correlation of hyperspectral images is a crucial prior property. However, the existing self-attention mechanism structure is still insufficient for the joint processing of spatial-spectral features, resulting in insufficient utilization of the spatial-spectral coupling correlation of hyperspectral images.

[0006] Therefore, those skilled in the art urgently need to improve existing methods for hyperspectral image denoising. Summary of the Invention

[0007] Objective: To overcome the shortcomings of existing technologies, this invention provides a hyperspectral image denoising method based on sparse TRANSFORMER structures.

[0008] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0009] A hyperspectral image denoising method based on sparse TRANSFORMER structure includes the following steps:

[0010] The sparse spatial spectrum Transformer denoising network is trained using training samples. When the error loss is minimized, the trained sparse spatial spectrum Transformer denoising network is obtained.

[0011] The noisy image is input into the trained sparse spatial spectrum Transformer denoising network to obtain the repaired hyperspectral image.

[0012] As a preferred embodiment, the sparse spatial spectrum Transformer denoising network includes: a first convolutional layer, multiple sparse spatial spectrum denoising modules, a second convolutional layer, and a third convolutional layer.

[0013] The output of the first convolutional layer is connected in series with multiple sparse spatial spectrum denoising modules. The output of the last sparse spatial spectrum denoising module is connected to the input of the second convolutional layer. The output of the second convolutional layer and the output of the first convolutional layer are respectively connected to the input of the first residual connection. The output of the first residual connection is connected to the input of the third convolutional layer. The output of the third convolutional layer and the input of the first convolutional layer are respectively connected to the input of the second residual connection.

[0014] The sparse spatial spectrum denoising module includes: multiple sparse spatial spectrum Transformer modules and a fourth convolutional layer.

[0015] The multiple sparse spatial spectrum Transformer modules are connected in series. The output of the last sparse spatial spectrum Transformer module is connected to the input of the fourth convolutional layer. The output of the fourth convolutional layer and the input of the first sparse spatial spectrum Transformer module are respectively connected to the input of the third residual connection.

[0016] The sparse spatial spectrum Transformer module includes: a first normalization layer, a sparse spatial spectrum self-attention module, a second normalization layer, and a multilayer perceptron.

[0017] The output of the first normalization layer is connected to the input of the sparse spatial spectrum self-attention module. The output of the sparse spatial spectrum self-attention module and the input of the first normalization layer are respectively connected to the input of the fourth residual connection. The output of the fourth residual connection is connected to the input of the second normalization layer. The output of the second normalization layer is connected to the input of the multilayer perceptron. The output of the multilayer perceptron and the input of the second normalization layer are respectively connected to the input of the fifth residual connection.

[0018] The sparse spatial spectrum self-attention module includes a spatial dimension Transformer module and a spectral dimension Transformer module, which are connected in series.

[0019] As a preferred embodiment, the number of sparse spatial spectrum denoising modules is set to 6.

[0020] As a preferred embodiment, the number of sparse spatial spectrum Transformer modules is set to 6.

[0021] As a preferred embodiment, the spatial dimension Transformer module includes: the output of a window partitioning module connected to the input of a first linear projection operation module; the first and second outputs of the first linear projection operation module respectively connected to the input of a first matrix multiplication module; the output of the first matrix multiplication module and the output of a position encoding module respectively connected to the input of a sixth residual connection; the output of the sixth residual connection connected to the input of a first activation function; the third output of the first linear projection operation module connected in series with a fifth convolutional layer and a second activation function; the output of the first activation function and the third output of the first linear projection operation module respectively connected to the input of a second matrix multiplication module; the output of the second matrix multiplication module and the output of the second activation function respectively connected to the input of a seventh residual connection; the output of the seventh residual connection connected to the input of a second linear projection operation module; and the output of the second linear projection operation module connected to the input of a first window restoration module.

[0022] As a preferred embodiment, the spectral dimension Transformer module includes: the output of a first window recovery module connected to the input of a spectral channel mixing module; the output of the spectral channel mixing module connected to the input of a third linear projection operation module; the first and second outputs of the third linear projection operation module connected to the input of a third matrix multiplication module; and the output of the third matrix multiplication module connected sequentially to a recommendation module, a tensor operation function, and a third activation function. The output of the mask module is also connected to the input of the tensor operation function. The output of the third activation function and the third output of the third linear projection operation module are connected to the input of a fourth matrix multiplication module; the output of the fourth matrix multiplication module is connected to the input of the fourth linear projection operation module; and the output of the fourth linear projection operation module is connected to the input of the dimension transformation module.

[0023] As a preferred embodiment, the spectral channel mixing module includes: the output of the first window recovery module is connected to the input of the fifth linear projection operation module and the input of the eighth residual connection, respectively. The output of the fifth linear projection operation module and the output of the first window recovery module are connected to the input of the matrix multiplication module, the output of the matrix multiplication module is also connected to the input of the eighth residual connection, the output of the eighth residual connection and the output of the first static attention module are connected to the input of the fifth matrix multiplication module, the output of the fifth matrix multiplication module is connected to the input of the fourth activation function, and the output of the fourth activation function and the output of the second static attention module are connected to the input of the sixth matrix multiplication module, respectively.

[0024] As a preferred option, the first activation function is the SoftMax function. The second activation function is the GELU function. The third activation function is the SoftMax function. The fourth activation function is the ReLU function.

[0025] As a preferred embodiment, the error loss is the mean square error loss.

[0026] Beneficial Effects: This invention provides a hyperspectral image denoising method based on a sparse TRANSFORMER structure. It utilizes a sparse spatial-spectral Transformer denoising network to jointly learn the spatial-spectral features of hyperspectral images, thereby aiding in denoising. In the spatial dimension of the hyperspectral image, to address the issue of excessive smoothness, a local convolutional module is introduced to enhance high-frequency spatial feature details. Simultaneously, in the spectral dimension, a Top-k sparse self-attention mechanism is proposed, adaptively selecting the most relevant spectral channel information for feature interaction, thus effectively capturing the correlation of spectral features between different spectral bands. Through the joint learning of the spatial-spectral features of the hyperspectral image, this invention can perform efficient and high-quality denoising and restoration operations on input noisy hyperspectral images based on the trained network parameter weights. Attached Figure Description

[0027] Figure 1 This is a diagram of the sparse spatial spectrum Transformer denoising network framework.

[0028] Figure 2 This is the algorithm flowchart for the spatial dimension Transformer module.

[0029] Figure 3 This is the algorithm flowchart for the Transformer module in the spectral dimension.

[0030] Figure 4 The algorithm flowchart is shown for the spectral channel mixing module.

[0031] Figure 5 This is a demonstration of some of the noise reduction effects. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0033] The present invention will be further described below with reference to specific embodiments.

[0034] The sparse spatial spectrum Transformer denoising network proposed in this invention is as follows: Figure 1 As shown, previous methods have insufficiently extracted the joint spatial-spectral features of hyperspectral images, resulting in distorted spectral features of the denoised hyperspectral images and consequently, unsatisfactory performance in subsequent applications such as hyperspectral image classification. In contrast, the sparse spatial-spectral Transformer denoising network of this invention can effectively extract joint spatial-spectral features, thereby improving denoising performance while preserving the most complete spectral features.

[0035] The sparse spatial spectrum Transformer denoising network proposed in this invention includes: a first convolutional layer (Conv), multiple sparse spatial spectrum denoising modules (SSSDM), a second convolutional layer, and a third convolutional layer.

[0036] The output of the first convolutional layer (Conv) is connected in series with multiple sparse spatial spectrum denoising modules (SSSDM). The output of the last sparse spatial spectrum denoising module is connected to the input of the second convolutional layer. The output of the second convolutional layer and the output of the first convolutional layer are respectively connected to the input of the first residual connection. The output of the first residual connection is connected to the input of the third convolutional layer. The output of the third convolutional layer and the input of the first convolutional layer are respectively connected to the input of the second residual connection.

[0037] In one embodiment, the number of sparse spatial spectrum denoising modules (SSSDM) is set to 6.

[0038] The Sparse Spatial Spectrum Denoising Module (SSSDM) includes: multiple Sparse Spatial Spectrum Transformer Modules (SSST) and a fourth convolutional layer.

[0039] The multiple sparse spatial spectrum Transformer modules (SSST) are connected in series. The output of the last sparse spatial spectrum Transformer module is connected to the input of the fourth convolutional layer. The output of the fourth convolutional layer and the input of the first sparse spatial spectrum Transformer module are respectively connected to the input of the third residual connection.

[0040] In one embodiment, the number of Sparse Spatial Spectrum Transformer (SSST) modules is set to 6.

[0041] The Sparse Spatial Spectrum Transformer (SSST) module includes: a first normalization layer (LN), a sparse spatial spectrum self-attention module (SSSMA), a second normalization layer, and a multilayer perceptron (MLP).

[0042] The output of the first normalization layer (LN) is connected to the input of the sparse spatial spectrum self-attention module (SSSMA). The output of the SSSMA and the input of the first normalization layer are respectively connected to the input of the fourth residual connection. The output of the fourth residual connection is connected to the input of the second normalization layer. The output of the second normalization layer is connected to the input of the multilayer perceptron (MLP). The output of the multilayer perceptron (MLP) and the input of the second normalization layer are respectively connected to the input of the fifth residual connection.

[0043] The Sparse Spatial Spectral Self-Attention Module (SSSMA) includes a spatial dimension Transformer module and a spectral dimension Transformer module, which are connected in series.

[0044] like Figure 2 As shown, the Spatial Dimension Transformer module (SpaT) includes: the output of a window partitioning module connected to the input of a first linear projection operation module (Linear); the first output (Q) and second output (K) of the first linear projection operation module connected to the input of a first matrix multiplication module; the output of the first matrix multiplication module and the output of the position encoding module connected to the input of a sixth residual connection; the output of the sixth residual connection connected to the input of a first activation function; the third output (V) of the first linear projection operation module connected in series with a fifth convolutional layer and a second activation function; the output of the first activation function and the third output of the first linear projection operation module connected to the input of a second matrix multiplication module; the output of the second matrix multiplication module and the output of the second activation function connected to the input of a seventh residual connection; the output of the seventh residual connection connected to the input of a second linear projection operation module; and the output of the second linear projection operation module connected to the input of a first window restoration module.

[0045] like Figure 3 As shown, the spectral dimension Transformer module (SpeT) includes: the output of the first window recovery module is connected to the input of the spectral channel mixing module (SCM); the output of the spectral channel mixing module (SCM) is connected to the input of the third linear projection operation module (Linear); the first output (Q) and the second output (K) of the third linear projection operation module (Linear) are respectively connected to the input of the third matrix multiplication module; the output of the third matrix multiplication module is sequentially connected to the recommendation module (Top-k), the tensor operation function (Scatter), and the third activation function. The output of the mask module (Mask) is also connected to the input of the tensor operation function (Scatter). The output of the third activation function and the third output (V) of the third linear projection operation module (Linear) are respectively connected to the input of the fourth matrix multiplication module; the output of the fourth matrix multiplication module is connected to the input of the fourth linear projection operation module (Linear); and the output of the fourth linear projection operation module (Linear) is connected to the input of the dimension transformation module.

[0046] like Figure 4As shown, the spectral channel mixing module (SCM) includes: the output of the first window recovery module is connected to the input of the fifth linear projection operation module (Linear) and the input of the eighth residual connection. The output of the fifth linear projection operation module (Linear) and the output of the first window recovery module are connected to the input of the matrix multiplication module. The output of the matrix multiplication module is also connected to the input of the eighth residual connection. The output of the eighth residual connection and the output of the first static attention module are connected to the input of the fifth matrix multiplication module. The output of the fifth matrix multiplication module is connected to the input of the fourth activation function. The output of the fourth activation function and the output of the second static attention module are connected to the input of the sixth matrix multiplication module.

[0047] The first activation function is the SoftMax function. The second activation function is the GELU function. The third activation function is the SoftMax function. The fourth activation function is the ReLU function.

[0048] This invention introduces a hyperspectral image denoising method based on a sparse transformer structure, comprising the following steps:

[0049] Step 1: Train the sparse spatial spectrum Transformer denoising network using training samples. When the error loss is minimized, the trained sparse spatial spectrum Transformer denoising network is obtained.

[0050] Step 2: Input the noisy image into the trained sparse spatial spectrum Transformer denoising network to obtain the repaired hyperspectral image.

[0051] Furthermore, the error loss is calculated using mean squared error (MSE) loss. During network training, the error between the network output image and the clean image is calculated by setting the mean squared error (MSE loss) as the loss function. Network training is achieved by minimizing the MSE loss, ensuring that the network can learn the optimal output for denoising hyperspectral images.

[0052] Furthermore, the method for denoising noisy images using a pre-trained sparse spatial spectrum Transformer denoising network includes the following steps:

[0053] The input noisy image is first passed through a convolutional layer to obtain shallow features of the input noisy image.

[0054] Shallow features are extracted through multiple cascaded sparse spatial-spectral denoising modules to jointly extract spatial and spectral features from the image. This allows the network to utilize the non-local spatial similarity and global spectral correlation of the hyperspectral image for targeted denoising. Each sparse spatial-spectral denoising module performs a residual connection to utilize the original image features and prevent overfitting during network training.

[0055] After multiple cascaded sparse spectral denoising modules, shallow features and the original noisy image are fused through two residual connections, thereby recovering the denoised and repaired clean hyperspectral image from the deep image features.

[0056] This invention mainly addresses the problems in the current field of hyperspectral image denoising, such as the Transformer model's insufficient utilization of the spatial-spectral coupling correlation of hyperspectral images, the overly smooth processing of spatial features, and the excessive focus on all channel features in the spectral dimension.

[0057] The Spatial Dimension Transformer module (SpaT) of this invention introduces a local convolution module to enhance the details of local spatial features, solving the problem of the Transformer structure being too smooth in processing spatial features. Simultaneously, a Top-k sparse self-attention mechanism is proposed in the Spectral Dimension Transformer module (SpeT), which adaptively selects the most relevant spectral channel information for feature interaction, thereby effectively capturing the most relevant spectral feature information for auxiliary denoising.

[0058] Sparse Spatial Spectrum Transformer Module (SSST): Assumptions Let L represent the input features of the Lth sparse spatial spectrum Transformer module. The specific processing procedure of the sparse spatial spectrum Transformer module can be represented as follows:

[0059] Z l =SSSMA(LN(Z) l-1 ))+Z l-1 (1)

[0060] Z l =MLP(LN(Z′) l ))+Z′ l (2)

[0061] Where: Z′ l and Z l These represent the output features of the sparse spatial spectrum multi-head self-attention module and the sparse spatial spectrum Transformer module, respectively. Therefore, Z lThis is the input feature of the next sparse spatial spectrum Transformer. Here, SSSMA represents the sparse spatial spectrum self-attention module, MLP represents the multilayer perceptron, and LN represents the normalization layer.

[0062] The Sparse Spatial Spectral Self-Attention Module (SSSMA) mainly consists of two structures: Spatial Transformer (SpaT) and Spectral Transformer (SpeT).

[0063] Input feature Z l-1 After layer normalization, the input features of the sparse spatial spectrum multi-head self-attention module are obtained as follows: First, a windowing operation is performed in the spatial dimension, resulting in a small window of size M×M (set to 8 in this invention), thus dividing the entire input feature in the spatial dimension. A set of non-overlapping small windows can be represented as The dimensions of each small window are: After segmentation, each feature block is processed using the self-attention operation of a spatial dimension Transformer to extract the non-local spatial feature similarity of the hyperspectral image. This process can be described as follows:

[0064]

[0065]

[0066]

[0067] Here, WinPartition represents the window partitioning operation, and WinReverse represents the reverse window restoration operation. The window restoration operation restores the small window feature blocks output from the Transformer's self-attention operation to the same size as the input features, thus obtaining the final output feature dimension. The invention then passes the obtained output features to the subsequent spectral dimension Transformer. The spectral dimension Transformer mainly consists of a spectral channel mixing module and a Top-k sparse self-attention module. This process can be simply represented as:

[0068] Z scm =SpeT(SCM(Z) spa (6)

[0069] The Spatial Dimension Transformer Module Self-Attention Operation (SpaT): This invention introduces a local enhancement module into the original Transformer self-attention mechanism. This module incorporates image edge details into the self-attention operation, enhancing the network's ability to model image edge details and allowing the network to capture more image detail information. Finally, the feature information of each small window is sequentially restored, and the resulting output features are... Pass it backwards.

[0070] Specifically, the feature blocks of each small window of the input image The query is obtained through linear projection. key value Three vectors can be specifically represented as:

[0071]

[0072] in: These are weights of size C×C. A multi-head mechanism is then used to combine information from different window subspaces. Therefore... Divided into N heads, represented as and Therefore, each head The spatial self-attention matrix is ​​calculated as follows:

[0073]

[0074]

[0075] Where: d is The dimension of is specifically C / N. B represents the relative deviation.

[0076] Meanwhile, in order to supplement the local feature information of the image, the local enhancement module will use the values ​​obtained by linear projection. As input, a convolutional layer and activation function are passed through. Unlike typical Transformer self-attention structures, this design helps to incorporate local information from each small window, supplementing the insufficient local information extraction of the Transformer. This process can be represented as follows:

[0077]

[0078] After spatial self-attention computation, all multi-head outputs are processed. After embedding them together, a linear operation is performed to obtain the output of the Transformer self-attention operation in the spatial dimension. Then perform the window restoration operation to obtain the final output.

[0079] Furthermore, this paper obtains non-local spatial information by performing a spatial shift operation between each sparse spatial spectrum multi-head self-attention module. Specifically, the shift operation involves shifting the input features before partitioning. It is implemented using pixels of size.

[0080] The spectral dimension Transformer module (SpeT): Specifically, the spectral dimension Transformer first passes through a spectral channel mixing module (SCM), such as... Figure 4 As shown. This improves the expressive power of spectral features across different bands, ensuring that the Top-k algorithm can select the most relevant bands. The proposed spectral channel mixing module mainly consists of two parts: dynamic mixing (DM) and static attention mapping (SM). The entire process can be simply represented as follows:

[0081] Z scm =SM(GELU(DM(Z) spa (11)

[0082] Where: is Z scm Z spa Let represent the input and output feature maps, respectively, and GELU represent the activation function. This paper proposes a method for obtaining inter-band correlation using element-wise dynamic weighted channel mixing. Specifically, firstly, scaling weights for each input feature map are calculated through linear projection. Then, element-wise multiplication is performed on the position of each feature information, followed by residual concatenation with the input features, and finally, the feature information is scaled using the scaling weights. The dynamic mixing process can be represented as:

[0083] Z dm =(W2(W1Z) spa ⊙Z spa +Z spa (12)

[0084] Unlike traditional channel-based gating modules that only control individual channels to pass information to subsequent channels, the static attention mapping proposed in this paper uses a static attention mapping graph. Information from all channels is mixed. Because hyperspectral images have global spectral correlations, all band information needs to be processed. Attention maps W1, W2, and W3 are jointly learned with the entire network during training. The static attention mapping operation is represented as follows:

[0085] Z scm =Z dm W3 (13)

[0086] After passing through the spectral channel mixing module, the resulting hyperspectral image features are extracted using a Top-k sparse self-attention module. Specifically, the input image is first processed... Transpose Then the query is obtained through linear projection. key Sum Three vectors. Specifically, this can be represented as:

[0087]

[0088] in These are weights of size C×C. The specific implementation of the Top-k algorithm in the Top-k sparse self-attention module can then be represented as:

[0089]

[0090] Where Γ κ A learnable selection operator for selection operators is implemented as follows:

[0091]

[0092] Specifically, for the query Q obtained by linear projection spe and key K spe First, a self-attention matrix is ​​calculated. However, unlike typical self-attention modules, directly applying SoftMax processing weakens high-frequency information in the attention matrix, resulting in an overly smoothed outcome with insufficient detail processing. This paper utilizes the Top-k method to mask irrelevant redundant features in the self-attention matrix before applying SoftMax processing, effectively avoiding overly smoothed feature extraction. It's important to note that, unlike dropout, the Top-k algorithm used in this invention includes a learnable Top-k operator that adaptively selects the value of K, which is manually set. This adaptive K selection dynamically controls the number of most relevant features selected for different bands. This K value is also learnable. Furthermore, the spectral dimension Transformer performs similar operations to the spatial dimension Transformer, employing a multi-head mechanism. Finally, this paper concatenates all outputs of the multi-head attention and obtains the final output through linear projection. This is the final output of each sparse spatial spectrum multi-head self-attention module. It's worth emphasizing that, compared to Z... spa In comparison, Z spe It has more spectral details and also retains key spatial information.

[0093] This invention analyzes existing methods for hyperspectral image denoising, focusing on deep learning-based denoising approaches. It proposes a spatial Transformer module and a spectral Transformer module within the network, enabling the network to acquire joint spatial and spectral features of the hyperspectral image. In the spatial Transformer, a moving window-based self-attention module captures the non-local spatial similarity of the hyperspectral image, and a local enhancement module supplements and fuses local spatial features. Secondly, in the spectral Transformer, Top-k sparse self-attention is used to select spectral bands, utilizing the K most relevant bands for auxiliary denoising, and a spectral channel mixing module enhances the ability to explore global spectral correlations. Compared to previous deep learning denoising algorithms, this method achieves better extraction of spectral features and superior denoising performance in spatial details.

[0094] This invention employs a sparse spatial-spectral Transformer denoising network to jointly extract spatial-spectral features from hyperspectral images, improving the model's ability to extract both spatial detail features and global spectral features, ultimately achieving excellent denoising results. It enables fast and efficient denoising operations on noisy hyperspectral image datasets. Using the proposed scheme, such as... Figure 5 As shown, this invention utilizes a neural network to learn hyperspectral image features. Compared to traditional methods that manually set prior feature information, this invention significantly improves the extraction and utilization of hyperspectral image features, allowing model parameters to better fit the intrinsic features of the hyperspectral image, thereby achieving better denoising results.

[0095] Example:

[0096] All experimental data in this invention were implemented on the ICVL hyperspectral image dataset, which consists of 201 images with a spatial resolution of 1392×1300 and 31 spectral bands ranging from 400 to 700 nm. 100 hyperspectral images were randomly selected as the training set, 50 as the validation set, and the remaining 50 as the test set.

[0097] The deep learning-based methods QRNN3D and TRQ3D compared in this paper, along with the method proposed in this paper, are all implemented in PyTorch and run on a GeForce RTX 3090. Traditional methods based on manually setting prior information, including BM4D and LLRT, are implemented in MATLAB and run on an Intel Core i5-12400 CPU.

[0098] To better evaluate our proposed method, this invention uses three performance metrics: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM). Higher PSNR and SSIM values ​​indicate better denoising performance, while lower SAM values ​​indicate higher spectral structure fidelity in the denoised image.

[0099] Table 1 shows the denoising comparison of hyperspectral images containing Gaussian noise.

[0100]

[0101] Gaussian noise is zero-mean, independent and identically distributed (iid) Gaussian noise with different variances. We evaluated Gaussian noise under four conditions: variances of 30, 50, and 70, as well as Blind noise (variance ranging from 10 to 70).

[0102] Table 2 shows the denoising comparison of hyperspectral images containing complex noise.

[0103]

[0104] Complex noise can be mainly divided into five types: non-iid Gaussian noise, a mixture of non-iid Gaussian noise and cutoff noise, a mixture of non-iid Gaussian noise and stripe noise, a mixture of non-iid Gaussian noise and impulse noise, and a mixture of all the above complex noises.

[0105] The performance metrics of the proposed method and its comparative methods on the ICVL dataset are shown in Tables 1 and 2. The proposed method outperforms the other comparative methods. Under Gaussian noise conditions, the proposed method achieves a maximum improvement of 0.84 dB in PSNR. Under mixed noise conditions, the proposed method achieves a maximum improvement of 1.03 dB in PSNR. This demonstrates that the sparse spatial spectrum Transformer module significantly improves denoising performance for mixed noise with uneven noise intensity.

[0106] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A hyperspectral image denoising method based on sparse transformer structure, characterized in that: Includes the following steps: The sparse spatial spectrum Transformer denoising network is trained using training samples. When the error loss is minimized, the trained sparse spatial spectrum Transformer denoising network is obtained. The noisy image is input into the trained sparse spatial spectrum Transformer denoising network to obtain the repaired hyperspectral image. The sparse spatial spectrum Transformer denoising network includes: a first convolutional layer, multiple sparse spatial spectrum denoising modules, a second convolutional layer, and a third convolutional layer. The output of the first convolutional layer is connected in series with multiple sparse spatial spectrum denoising modules. The output of the last sparse spatial spectrum denoising module is connected to the input of the second convolutional layer. The output of the second convolutional layer and the output of the first convolutional layer are respectively connected to the input of the first residual connection. The output of the first residual connection is connected to the input of the third convolutional layer. The output of the third convolutional layer and the input of the first convolutional layer are respectively connected to the input of the second residual connection. The sparse spatial spectrum denoising module includes: multiple sparse spatial spectrum Transformer modules and a fourth convolutional layer; The multiple sparse spatial spectrum Transformer modules are connected in series. The output of the last sparse spatial spectrum Transformer module is connected to the input of the fourth convolutional layer. The output of the fourth convolutional layer and the input of the first sparse spatial spectrum Transformer module are respectively connected to the input of the third residual connection. The sparse spatial spectrum Transformer module includes: a first normalization layer, a sparse spatial spectrum self-attention module, a second normalization layer, and a multilayer perceptron. The output of the first normalization layer is connected to the input of the sparse spatial spectrum self-attention module. The output of the sparse spatial spectrum self-attention module and the input of the first normalization layer are respectively connected to the input of the fourth residual connection. The output of the fourth residual connection is connected to the input of the second normalization layer. The output of the second normalization layer is connected to the input of the multilayer perceptron. The output of the multilayer perceptron and the input of the second normalization layer are respectively connected to the input of the fifth residual connection. The sparse spatial spectrum self-attention module includes a spatial dimension Transformer module and a spectral dimension Transformer module, which are connected in series.

2. The hyperspectral image denoising method based on a sparse TRANSFORMER structure according to claim 1, characterized in that: The number of sparse spatial spectrum denoising modules is set to 6.

3. The hyperspectral image denoising method based on a sparse TRANSFORMER structure according to claim 1, characterized in that: The number of sparse spatial spectrum Transformer modules is set to 6.

4. The hyperspectral image denoising method based on a sparse TRANSFORMER structure according to claim 1, characterized in that: The spatial dimension Transformer module includes: the output of the window partitioning module is connected to the input of the first linear projection operation module; the first and second outputs of the first linear projection operation module are respectively connected to the input of the first matrix multiplication module; the output of the first matrix multiplication module and the output of the position encoding module are respectively connected to the input of the sixth residual connection; the output of the sixth residual connection is connected to the input of the first activation function; the third output of the first linear projection operation module is connected in series with the fifth convolutional layer and the second activation function; the output of the first activation function and the third output of the first linear projection operation module are respectively connected to the input of the second matrix multiplication module; the output of the second matrix multiplication module and the output of the second activation function are respectively connected to the input of the seventh residual connection; the output of the seventh residual connection is connected to the input of the second linear projection operation module; and the output of the second linear projection operation module is connected to the input of the first window restoration module.

5. The hyperspectral image denoising method based on a sparse TRANSFORMER structure according to claim 1, characterized in that: The spectral dimension Transformer module includes: the output of the first window recovery module is connected to the input of the spectral channel mixing module; the output of the spectral channel mixing module is connected to the input of the third linear projection operation module; the first and second outputs of the third linear projection operation module are respectively connected to the input of the third matrix multiplication module; the output of the third matrix multiplication module is sequentially connected to the Top-k sparse self-attention module, the tensor operation function, and the third activation function; the output of the mask module is also connected to the input of the tensor operation function; the output of the third activation function and the third output of the third linear projection operation module are respectively connected to the input of the fourth matrix multiplication module; the output of the fourth matrix multiplication module is connected to the input of the fourth linear projection operation module; and the output of the fourth linear projection operation module is connected to the input of the dimension transformation module.

6. The hyperspectral image denoising method based on a sparse transformer structure according to claim 5, characterized in that: The spectral channel mixing module includes: the output of the first window recovery module is connected to the input of the fifth linear projection operation module and the input of the eighth residual connection, respectively; the output of the fifth linear projection operation module and the output of the first window recovery module are connected to the input of the matrix multiplication module, the output of the matrix multiplication module is also connected to the input of the eighth residual connection, the output of the eighth residual connection and the output of the first static attention module are connected to the input of the fifth matrix multiplication module, the output of the fifth matrix multiplication module is connected to the input of the fourth activation function, and the output of the fourth activation function and the output of the second static attention module are connected to the input of the sixth matrix multiplication module, respectively.

7. The hyperspectral image denoising method based on a sparse TRANSFORMER structure according to claim 6, characterized in that: The first activation function is the SoftMax function; the second activation function is the GELU function; the third activation function is the SoftMax function; and the fourth activation function is the RELU function.

8. The hyperspectral image denoising method based on a sparse transformer structure according to claim 1, characterized in that: The error loss is the mean square error loss.

Images