A Hyperspectral Image Denoising Method and System Based on Spatial-Spectral Joint Self-Attention Mechanism

Abstract

This invention discloses a method and system for hyperspectral image denoising based on a spatial-spectral joint self-attention mechanism, belonging to the field of image processing technology. First, based on the characteristics of hyperspectral images, a spatial-spectral joint self-attention mechanism network is constructed. The noisy hyperspectral image is used as input to the network to extract spatial-spectral features. Then, a global spectral self-attention mechanism is used to extract the band correlations of the hyperspectral image. Finally, the extracted spatial-spectral features are reconstructed using a multiple perceptron and residual connections to reconstruct a clean, noise-free hyperspectral image. The system includes a feature extraction subsystem, a non-local spatial self-attention subsystem, a global spectral self-attention subsystem, and an image reconstruction subsystem. This invention can effectively restore noisy hyperspectral images to obtain noise-free hyperspectral images. Compared to convolutional networks, it can better model long-range dependency information and has better adaptability to target hyperspectral images.

Description

TECHNICAL FIELD

[0001] The application relates to a hyperspectral image denoising method and system based on a spectral joint self-attention mechanism, and belongs to the technical field of spectral image processing. BACKGROUND

[0002] Hyperspectral images provide rich information in the spectral dimension and are widely used in remote sensing, material identification, agricultural production, medical diagnosis, etc. However, during the sensing imaging process, due to limited light, photon effects and atmospheric interference, etc., the hyperspectral image is often affected by noise, which is not conducive to the subsequent application of the spectral image. As an important processing means, hyperspectral image denoising can improve the quality of the spectral image and help the promotion of subsequent tasks.

[0003] Compared with color images, hyperspectral images provide spectral features at the pixel level. The hyperspectral imaging system continuously images between narrow bands, which means that there is a statistical correlation between the frequency bands of the hyperspectral image. Early denoising work, such as dictionary-based learning methods and BM3D, focuses on the non-local similarity of the image in the spatial dimension, without considering the spectral features. Therefore, more hyperspectral image denoising work simultaneously utilizes spatial similarity and spectral correlation. Among them, the multilinear tool extracts spectral components and spatial information for denoising. In addition, a parallel factor analysis model is adopted to utilize the decomposition uniqueness and the low-rank characteristics of hyperspectral images. Sparse representation and total variation regularization are also widely used in hyperspectral image denoising. Although there are various handcrafted priors, traditional hyperspectral image denoising methods based on optimization iteration are still difficult to optimize and time-consuming.

[0004] With the development of deep learning, hyperspectral image denoising technology based on convolutional neural networks shows obvious advantages compared with traditional denoising methods. This method relies on convolution filters to extract features of data in the spatial dimension and the spectral dimension, and faces the trade-off between computational efficiency and the modeling ability of non-local similarity. In addition, the learned convolution filters are static weights, which means that the filters used for feature extraction are fixed in the test phase. These methods use the knowledge learned from the training data set for testing, while the intrinsic characteristics of the target hyperspectral image are not fully utilized.

[0005] In recent years, the self-attention mechanism (Transformer) has been applied to visual tasks, which can well capture the internal similarity of the target image. However, existing Transformers are mainly designed for natural images, and due to the strong spectral correlation of hyperspectral images, the neglect of spectral features will have a negative impact on the denoising result. SUMMARY

[0006] The purpose of the present application is to creatively propose a hyperspectral image denoising method and system based on a space-spectrum joint self-attention mechanism to overcome the defects and deficiencies of the prior art, and to extract the non-local spatial similarity and global spectral correlation of the hyperspectral image by using the self-attention mechanism (Transformer) to obtain better denoising effect.

[0007] The present application adopts the following technical solutions.

[0008] A hyperspectral image denoising method based on a space-spectrum joint self-attention mechanism, comprising the following steps:

[0009] Step 1: Constructing a space-spectrum joint Transformer network according to the characteristics of the hyperspectral image.

[0010] Specifically, the expression of the noise model of the hyperspectral image is:

[0011] Y = X + η

[0012] Where Y is the noisy hyperspectral image, X is the original clean hyperspectral image, and η is the noise.

[0013] The noisy hyperspectral image is taken as the input of the network to extract the image space-spectrum joint features.

[0014] Specifically, the feature extraction process is as follows:

[0015] F k =H k (F k-1 ),k=1,2,…,K

[0016] Where F k is the feature extracted by different layers of the network, H k represents the kth space-spectrum self-attention mechanism residual network; F0 = conv3(Y), the network first extracts the shallow layer feature F0 from the noisy hyperspectral Y through 3x3 convolution, and then extracts the deep features of the noisy image through the stacked K residual modules.

[0017] Step 2: Extracting the spatial similarity features of the hyperspectral image through the spatial self-attention mechanism of the offset window. For the space-spectrum joint Transformer network, the input feature Z l-1 Through the self-attention mechanism network, the multi-dimensional characteristics of the hyperspectral image are used for denoising.

[0018] Specifically, the self-attention mechanism network includes a space-spectrum self-attention module SSMA, a multilayer perception MLP and a regularization layer LN. The feature extraction of the self-attention mechanism network is as follows:

[0019] Z l′ = SSMA(LN(Z l-1 ))+ Z l-1

[0020] Z l = MLP(LN(Z l ′ ))+ Z l ′

[0021] wherein, Z l ′ and Z l represent the output of the l-th layer SSMA module and MLP layer, respectively. The SSMA module includes a non-local spatial self-attention module NLSA and a global spectral self-attention module GSA. For the input Z in , the extraction process of the non-local spatial self-attention feature is represented as:

[0022]

[0023]

[0024]

[0025] wherein, WinPartition represents a window partition operation on the feature, represents the i-th non-local feature, and WinReverse represents a merging operation on the features partitioned in the form of windows;

[0026] represents the feature block obtained from Z in by window partitioning, H represents the height of the feature, W represents the width of the feature, and M represents the window size.

[0027] Step 3: Extract the band correlation of the hyperspectral image through the global spectral self-attention mechanism.

[0028] Specifically, the obtained non-local feature Z nonlocal is extracted and utilized by the global spectral self-attention module GSA for the high correlation between the spectra:

[0029] Z global = GSA(Z nonlocal )

[0030] wherein, Z global represents the global spectral feature.

[0031] Step 4: The extracted spatial-spectral feature is passed through a multi-layer perceptron and a residual connection to obtain the feature F kA clean and noise-free hyperspectral image is recovered.

[0032] Specifically, two 3*3 convolution layers are used to map and recover the features, and the obtained features are connected with the shallow layer features through residual connection.

[0033] Based on the spatial similarity of the hyperspectral image, the non-local features in the spatial dimension are obtained through the self-attention mechanism of the offset window. Further, a multi-layer perception and a residual connection are introduced to obtain a smoother denoising effect.

[0034] In the above manner, the characteristics of the hyperspectral image are effectively utilized to recover a clean hyperspectral image.

[0035] The second object of the application is to provide a hyperspectral image denoising system based on a spatial-spectral joint self-attention mechanism, comprising a feature extraction subsystem, a non-local spatial self-attention subsystem, a global spectral self-attention subsystem and an image reconstruction subsystem.

[0036] The feature extraction subsystem is used to obtain the shallow features of the image from the noisy hyperspectral image.

[0037] The non-local spatial self-attention subsystem is used to extract the non-local spatial self-similarity of the hyperspectral image and remove the noise in a coarse-grained manner.

[0038] The global spectral self-attention subsystem is used to extract the high correlation between the bands of the hyperspectral image and remove the noise in a fine-grained manner.

[0039] The image reconstruction subsystem is used to reconstruct a clean and noise-free hyperspectral image.

[0040] The connection relationship between the above subsystems is that the output end of the feature extraction subsystem is connected with the input end of the non-local spatial self-attention subsystem, the output end of the non-local spatial self-attention subsystem is connected with the input end of the global spectral self-attention subsystem, and the output end of the global spectral self-attention subsystem is connected with the input end of the image reconstruction subsystem.

[0041] Advantages

[0042] Compared with the prior art, the application has the following advantages:

[0043] 1. The application is based on the characteristics of the hyperspectral image, fully utilizes the spatial self-similarity and spectral high correlation of the target image itself, and can effectively recover the noisy hyperspectral image to obtain a noise-free hyperspectral image.

[0044] 2. This invention extracts the intrinsic similarity of images through a self-attention mechanism. Compared with convolutional networks, it can better model long-distance dependency information and has better adaptability to target hyperspectral images. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the method of the present invention.

[0046] Figure 2 This is a network framework diagram of the method of the present invention.

[0047] Figure 3 This is a structural diagram of the practical application framework of the spatial-spectral joint Transformer of the present invention.

[0048] Figure 4 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0049] To better illustrate the purpose and advantages of the present invention, the method of the invention will be further described below in conjunction with the accompanying drawings and examples.

[0050] Example

[0051] like Figure 1 As shown, a hyperspectral image denoising method based on a spatial-spectral joint self-attention mechanism includes the following steps:

[0052] Step 1: Construct a spatial-spectral joint Transformer network based on the characteristics of hyperspectral images;

[0053] Step 2: Use the noisy hyperspectral image as input to the network to extract joint spatial-spectral features of the image;

[0054] Step 3: For the residual module, the network input features are denoised using the multi-dimensional characteristics of the hyperspectral image through the Transformer network;

[0055] Step 4: Recover a clean hyperspectral image from deep features;

[0056] The hyperspectral image denoising method based on a joint spatial-spectral self-attention mechanism in this application constructs a joint spatial-spectral Transformer network. Utilizing the spatial similarity of hyperspectral images, it acquires non-local features in the spatial dimension through a self-attention mechanism with an offset window. The spectral self-attention mechanism leverages the high correlation between hyperspectral image bands for denoising. Furthermore, the introduction of a multilayer perceptron and residual connections achieves a smoother denoising effect.

[0057] Furthermore, in one embodiment of this application, the expression for the noise model of the hyperspectral image is specifically:

[0058] Y = X + η

[0059] where Y is the noisy hyperspectral image, X is the original hyperspectral image, and η is the noise.

[0060] Further, in an embodiment of the present application, the noisy hyperspectral image is taken as the input of the network, and the image space-spectrum joint feature is extracted. The feature extraction process is as follows:

[0061] F k = H k (F k-1 ), k = 1, 2, …, K

[0062] where F k is the feature extracted by different layers of the network, H k represents the kth space-spectrum Transformer residual network, and F0 = conv3(Y). The network first extracts the shallow feature F0 from the noisy hyperspectral Y through 3x3 convolution, and then extracts the deep feature of the noisy image through the stacked K residual modules.

[0063] Further, for the residual module, the network input feature Z l-1 is denoised by the Transformer module by utilizing the multi-dimensional characteristics of the hyperspectral image.

[0064] The Transformer network includes a space-spectrum self-attention module (SSMA), a multi-layer perceptron (MLP), and a regularization layer (LN). The feature extraction formula of the Transformer module is as follows:

[0065] Z l ′ = SSMA (LN (Z l-1 ))+ Z l-1 ,

[0066] Z l = MLP (LN (Z l ′ ))+ Z l ′ ,

[0067] where Z l ′ and Z l are the outputs of the SSMA module and the Transformer network, respectively.

[0068] Further, in an embodiment of the present application, the SSMA layer includes a non-local spatial self-attention module (NLSA) and a global spectral self-attention module (GSA). For the input Z inThe feature extraction process can be represented as:

[0069]

[0070]

[0071]

[0072] wherein, represents a feature block obtained from Z in according to window division, The obtained non-local features are extracted and utilized by a global spectral self-attention module to extract and utilize high correlation between spectrums:

[0073] Z global =GSA(Z nonlocal ).

[0074] Further, in an embodiment of the present application, a clean hyperspectral image is recovered from deep features F k .

[0075] Specifically, two 3x3 convolution layers are used to map and recover the features, wherein the obtained features are connected with shallow features through residual connection.

[0076] Figure 2 is a method framework diagram of the present application, which is composed of a feature extraction layer, a residual space-spectrum joint Transformer layer, and an image reconstruction layer.

[0077] Figure 3 is a framework diagram of the space-spectrum joint Transformer layer of the embodiment of the present application, which includes a non-local spatial self-attention module and a global spectral self-attention module. Compared with other hyperspectral image denoising methods, the space-spectrum joint Transformer layer can better model long-distance dependencies in the image and effectively utilize spatial self-similarity and spectral strong correlation of the hyperspectral image.

[0078] Figure 4 is a composition diagram of a hyperspectral image denoising system based on a space-spectrum joint self-attention mechanism provided in an embodiment of the present application. The system includes a feature extraction subsystem 10, a non-local spatial self-attention subsystem 20, a global spectral self-attention subsystem 30, and an image reconstruction subsystem 40.

[0079] The feature extraction subsystem 10 is configured to perform preliminary feature extraction on the image.

[0080] The non-local spatial self-attention subsystem 20 is configured to extract spatial self-similarity information of the hyperspectral image and perform coarse-grained denoising on the noisy image.

[0081] A global spectral self-attention subsystem 20 is configured to extract spectral strong correlation information of the hyperspectral image and perform fine-grained denoising on the noisy image.

[0082] An image reconstruction subsystem 40 is configured to reconstruct a clean hyperspectral image from the depth features.

[0083] The foregoing embodiments are illustrative of the hyperspectral image denoising method based on the spatial-spectral joint self-attention mechanism, and are also applicable to the hyperspectral image denoising system based on the spatial-spectral joint self-attention mechanism of the present embodiment, which will not be described herein again.

Claims

1. A hyperspectral image denoising method based on a joint spectral and spatial self-attention mechanism, characterized in that, The method comprises the following steps: Step 1: constructing a space-spectrum joint Transformer network according to the characteristics of a hyperspectral image; Taking a noisy hyperspectral image as the input of the network, the space-spectrum joint features of the image are extracted; The expression of the noise model of the hyperspectral image is as follows: Y=X+η Wherein, Y is a noisy hyperspectral image, X is an original clean hyperspectral image, and η is noise; The feature extraction process is as follows: F k = H k (F k-1 ), k = 1, 2,..., K where F k are features extracted from different layers of the network, H k represents the kth spatial-spectral self-attention mechanism residual network; F0=conv3(Y), the network first extracts shallow features F0from the noisy hyperspectrum Y through a 3×3 convolution, and then extracts deep features of the noisy image through a stack of K residual modules; Step 2: extracting the spatial similarity features of the hyperspectral image through the spatial self-attention mechanism of the offset window; For the spectral-spatial joint Transformer network, the input feature Z of the lth layer l-1 Through the self-attention mechanism network, the multi-dimensional characteristics of the hyperspectral image are used for denoising; The self-attention mechanism network comprises a space-spectrum self-attention module SSMA, a multilayer perceptron MLP and a regularization layer LN; The feature extraction of the self-attention mechanism network is as follows: Z' l = SSMA(LN(Z l-1 ))+ Z l-1 Z l = MLP(LN(Z' + 1)) + Z' l l ​ wherein Z' and Z l and Z l respectively represent the output of the l-th layer SSMA module and the MLP layer; the SSMA module comprises a non-local spatial self-attention module NLSA and a global spectral self-attention module GSA; For input Z in The extraction process of non-layout space self-attention features is represented as: wherein WinPartition represents a window partition operation on the feature, denotes the ith non-local feature, and WinReverse represents a merging operation on the features partitioned in a window form. denotes the ith non-local feature, and WinReverse represents a merging operation on the features partitioned in a window form. in denotes the ith non-local feature, and WinReverse represents a merging operation on the features partitioned in a window form. H represents the height of the feature, W represents the width of the feature, and M represents the window size. Step 3: extracting the band correlation of the hyperspectral image through the global spectrum self-attention mechanism; The obtained non-local feature Z nonlocal The high correlation between the spectra is extracted and utilized by the global spectral self-attention module GSA: Z global = GSA(Z nonlocal ) wherein Z global represents global spectral features; Step 4: the extracted space-spectrum features are mapped and recovered through two 3*3 convolution layers, and the features obtained are connected with the shallow layer features through residual connection; 2. The hyperspectral image denoising method based on the joint spectral and spatial self-attention mechanism according to claim 1, wherein, In step 4, two 3*3 convolution layers are used to map and recover the features, and the features obtained are connected with the shallow layer features through residual connection; Based on the spatial similarity of the hyperspectral image, the non-local features in the spatial dimension are obtained through the self-attention mechanism of the offset window, and the noise is removed through the spectral self-attention mechanism and the high correlation between the bands of the hyperspectral image.

3. The hyperspectral image denoising method based on joint spectral and spatial self-attention mechanism according to claim 2, characterized in that, The multilayer perceptron and the residual connection are introduced to obtain a smoother denoising effect.

4. A hyperspectral image denoising system based on a joint spectral and spatial self-attention mechanism for implementing the method of claim 1, characterized in that, The method comprises a feature extraction subsystem, a non-local spatial self-attention subsystem, a global spectral self-attention subsystem and an image reconstruction subsystem; The feature extraction subsystem is used to obtain the shallow layer features of the image from the noisy hyperspectral image; The non-local spatial self-attention subsystem is used to extract the non-local spatial self-similarity of the hyperspectral image and remove the noise in a coarse-grained manner; The global spectral self-attention subsystem is used to extract the high correlation between the bands of the hyperspectral image and remove the noise in a fine-grained manner; The image reconstruction subsystem is used to reconstruct a clean noise-free hyperspectral image; The connection relationship between the above subsystems is that the output end of the feature extraction subsystem is connected with the input end of the non-local spatial self-attention subsystem, the output end of the non-local spatial self-attention subsystem is connected with the input end of the global spectral self-attention subsystem, and the output end of the global spectral self-attention subsystem is connected with the input end of the image reconstruction subsystem.

Images