Hyperspectral denoising enhancement method and system based on noise decoupling

Abstract

The invention relates to a hyperspectral image denoising enhancement method and system based on noise decoupling, and belongs to the technical field of computer camera shooting. The method comprises the following steps: modeling according to a noise model, decoupling noise into modeling noise and non-modeling noise based on a physical link process, and constructing a hyperspectral image denoising data set with supervised accurate modeling noise according to a real pairwise data set; constructing a hyperspectral denoising network of accurate modeling noise, and pre-training the network by using a supervised synthetic data set; constructing a hyperspectral denoising network of inaccurate modeling noise, and training the network by using a real paired data set in combination with an accurate modeling noise removal network; establishing a multi-stage supervision constraint, and optimizing network parameters; and storing training parameters, generating a denoised hyperspectral image, and completing reasoning and index evaluation. According to the invention, noise decoupling can be carried out based on the real physical link and the deep learning network and real noise is removed in a combined manner, so that the hyperspectral imaging quality is improved.

Description

Technical Field

[0001] The present invention relates to a hyperspectral image denoising and enhancement method and system, and in particular to a hyperspectral image denoising and enhancement method and system based on noise decoupling, belonging to the field of computer imaging technology. Background Art

[0002] Hyperspectral images provide rich spectral information, making them indispensable in a variety of applications, such as remote sensing, classification, and identification. Despite their potential, hyperspectral images are often compromised by various real-world noises due to the limitations of imaging technology and the complexity of the captured environment. Therefore, a robust and effective denoising solution is crucial.

[0003] Recently, deep learning has emerged as a powerful alternative for hyperspectral denoising, capable of directly learning the mapping from noisy to clean images. Despite significant progress in learning-based methods, they still face a learning bottleneck. Due to the complex noise distribution and scarcity of real data, the complex mapping between paired real data remains difficult to learn, limiting their denoising performance.

[0004] From a data perspective, noise modeling methods analyze physical noise and model various noise components to synthesize realistic hyperspectral images for training, rather than using limited real data. However, real noise is often complex and difficult to model accurately, resulting in deviations between synthetic and real data, which weakens their effectiveness in real scenarios. Even well-known noises, such as readout noise and streak noise, are difficult to model accurately due to inaccurate fitting or the presence of some unknown noise. In addition, the difference in peak signal-to-noise ratio between real and synthetic hyperspectra further demonstrates the impact of inaccurate modeling and even the presence of some unknown noise. These problems severely degrade the quality of synthetic data, and the deviation from real data makes it continuously challenging to accurately and effectively learn complex mappings in real-world scenarios. Summary of the Invention

[0005] This paper aims to overcome the shortcomings of existing technologies by creatively proposing a method and system for denoising and enhancing hyperspectral images based on noise decoupling. By introducing a noise decoupling mechanism and combining it with a deep learning network, this paper can effectively separate and remove complex noise, and enhance the spectral and spatial information of hyperspectral images.

[0006] The present invention is implemented by adopting the following technical solutions.

[0007] A hyperspectral image denoising and enhancement method based on noise decoupling is characterized by specifically comprising the following steps:

[0008] Step 1: Model the noise model and decouple the noise into accurately modeled noise and inaccurately modeled noise. A supervised hyperspectral image denoising dataset with accurately modeled noise is constructed based on a real paired dataset.

[0009] Specifically, the present invention can construct a training data set by modeling based on noise analysis and calibrating an actual spectral camera to obtain camera parameters.

[0010] The complete noise modeling can be expressed as:

[0011] Y = X + N (1)

[0012] N=N a +N i (2)

[0013] Where X represents the clean hyperspectral image, N represents the noise component, and Y represents the noisy image. The noise N can be accurately modeled by the component N a and the inaccurate modeling component N i combination.

[0014] The training dataset includes: pairs of accurately modeled noisy hyperspectral images (obtained based on the collected high-resolution images and the degradation model) and clean hyperspectral images. The degradation model is uniformly expressed as:

[0015] Y a (λ)=X(λ)+N a (λ) (3)

[0016] N a (λ)~g(k,m(λ),v(λ)) (4)

[0017] Where X(λ) represents the λ channel of the clean hyperspectral image, N a (λ) is the accurately modeled noise component of the λ channel, Y a (λ) represents the noisy image synthesized in the λ channel. g(k,m(λ),v(λ)) represents the Gaussian noise distribution, k represents the system gain, m(λ) means the value of m(λ), and v(λ) means the variance of v(λ) in the λ channel.

[0018] Step 2: Construct a hyperspectral denoising network that accurately models noise and pre-train the network using a supervised synthetic dataset.

[0019] Specifically, the present invention uses a synthetic, accurately modeled, noisy paired hyperspectral image dataset to train a network. The deep learning network primarily consists of an encoder and a decoder. The encoder extracts key multi-scale features, while the decoder recovers clean hyperspectral images based on these features. Both the encoder and decoder use a basic convolution-attention module as their fundamental unit:

[0020] F′=SpatialConv(F)+SpectralConv(F) (5)

[0021] Attention(Q,K,V)=V·Softmax(K·Q) (6)

[0022] F1=W·Attention(Q,K,V)+F′ (7)

[0023] Here, F represents the image features extracted from the previous layer, SpatialConv represents 3D spatial convolution, i.e., convolution is performed only in the spatial dimension with a kernel size of (1, K, K), SpectralConv represents 3D spectral convolution, i.e., convolution is performed only in the spectral dimension with a kernel size of (K, 1, 1), and F′ represents the intermediate output features. Q, K, and V are obtained through linear processing, W represents the parameters that can be learned and optimized, and F1 represents the output features. The entire network processing can be expressed as:

[0024]

[0025] Among them, f AMNet represents the network that processes accurate modeling noise at this stage, θ represents the parameter to be optimized, Represents the result after denoising.

[0026] Step 3: Construct a hyperspectral denoising network with inaccurately modeled noise, and train the network using a real paired dataset in conjunction with the accurately modeled noise removal network.

[0027] Specifically, considering the high-frequency characteristics of inaccurate modeling noise (i.e., the difference between real noise and accurate modeling noise, which is regarded as "pseudo" inaccurate modeling noise), a wavelet-guided convolutional network is constructed. First, the high-frequency information of the noise is extracted through the constructed wavelet filter:

[0028] Y L ,Y H =DWT(Y), (9)

[0029]

[0030] Where Y represents the real noisy image, DWT represents Haar discrete wavelet transform, and Y L ,Y H Respectively represent the decomposed low-frequency information and high-frequency information. abs represents the absolute value operation, Interpolate represents the interpolation operation, It means that the high-frequency noise guidance information with the same spatial resolution as the original image is obtained after processing the extracted high-frequency information.

[0031] Considering the extraction of multi-scale features to guide the removal of inaccurate modeling noise, the extracted high-frequency noise guidance information is processed in multiple stages:

[0032]

[0033] Conv3d represents 3D convolution, and IWT represents inverse wavelet discrete transform. This step-by-step process obtains multi-level features G. i , i = 1, 2, 3, and insert these features into the decoder of the basic 3D convolutional network for denoising:

[0034]

[0035] where f IMNet represents the network that processes inaccurate modeling noise at this stage, θ represents the parameter to be optimized, Represents the result after denoising.

[0036] Step 4: Establish multi-stage supervision constraints and optimize network parameters.

[0037] Specifically, three stages of training constraints are performed for the training of the accurate modeling noise removal network and the inaccurate modeling noise removal network. Step 4 includes the following steps:

[0038] Step 4.1: Pre-training of the noise removal network for accurate modeling.

[0039] Specifically, the loss function of the supervised constraint in the first stage of pre-training is based on the input and output hyperspectral images, using L c loss:

[0040]

[0041] Where X represents the clean image, Represents the predicted image, ∈=10 -3 Represents a constant.

[0042] Step 4.2: Training of the inaccurate modeling noise removal network and joint fixed accurate modeling noise removal network.

[0043] Specifically, in the second stage of training, the parameters of the accurate modeling noise removal network are fixed, and the parameters of the inaccurate modeling noise removal network are optimized and updated. In addition to the loss used in the previous stage, the Kullback-Leibler (KL) divergence loss is added:

[0044]

[0045] Among them, p(Y i ) represents the true distribution, Represents the predicted distribution, and i represents the number of pixels.

[0046] Step 4.3: Joint fine-tuning training of the accurate modeling noise removal network and the inaccurate modeling noise removal network.

[0047] Specifically, the accumulated errors of each stage are eliminated. In this stage, two noise removal networks are fine-tuned jointly, and the losses of the previous two stages are removed. In order to ensure the spectral fidelity, the loss:

[0048]

[0049] Where N is the number of pixels. Comprehensive use loss:

[0050]

[0051] where λ k and λ s represents a hyperparameter.

[0052] Step 5: Save the training parameters, generate the denoised hyperspectral image, and complete the inference and indicator evaluation.

[0053] Specifically, in order to objectively evaluate the effect of the generated hyperspectral image, objective evaluation indicators can be generated based on peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM).

[0054] Based on the above method, the present invention proposes a hyperspectral image denoising and enhancement system based on noise decoupling, which includes: a data preprocessing subsystem, an accurate modeling noise removal subsystem, an inaccurate modeling noise removal subsystem and a supervised optimization and result evaluation subsystem.

[0055] The connection relationship between the above-mentioned component systems is:

[0056] The output end of the data preprocessing subsystem is connected to the input end of the accurate modeling noise removal subsystem and the inaccurate modeling noise removal subsystem, the output end of the inaccurate modeling noise removal subsystem is connected to the input end of the accurate modeling noise removal subsystem, and finally the output end of the accurate modeling noise removal subsystem is connected to the supervised optimization and result evaluation subsystem.

[0057] Beneficial effects

[0058] Compared with the prior art, the method and system of the present invention have the following advantages:

[0059] 1. The present invention is based on a multi-stage noise decoupling framework to effectively separate and remove complex noise components, including accurate and inaccurately modeled noise based on the real noise model, effectively solving the noise problem.

[0060] 2. The present invention uses a high-frequency wavelet-guided network to adaptively extract high-frequency features to effectively suppress the noise problem of inaccurate modeling.

[0061] 3. The present invention adopts a multi-stage learning strategy to eliminate noise components through separate pre-training and joint fine-tuning in each stage, effectively alleviating the problem of noise accumulation error. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flow chart of the method of the present invention.

[0063] Figure 2 Schematic diagram of the neural network structure described in the method of the present invention.

[0064] Figure 3 It is a schematic diagram of the composition of the system of the present invention. DETAILED DESCRIPTION

[0065] In order to better illustrate the purpose and advantages of the present invention, the inventive method is further described below with reference to the accompanying drawings and examples.

[0066] Example

[0067] like Figure 1 As shown, a hyperspectral image denoising and enhancement method based on noise decoupling includes the following steps:

[0068] Step 1: Model the noise model and decouple the noise into accurately modeled noise and inaccurately modeled noise. A supervised hyperspectral image denoising dataset with accurately modeled noise is constructed based on a real paired dataset.

[0069] Step 2: Construct a hyperspectral denoising network that accurately models noise and pre-train the network using a supervised synthetic dataset.

[0070] Step 3: Construct a hyperspectral denoising network with inaccurately modeled noise, and train the network using a real paired dataset in conjunction with the accurately modeled noise removal network.

[0071] Step 4: Establish multi-stage supervision constraints and optimize network parameters.

[0072] Step 4.1: Pre-training of the noise removal network for accurate modeling.

[0073] Step 4.2: Training of the inaccurate modeling noise removal network and joint fixed accurate modeling noise removal network.

[0074] Step 4.3: Joint fine-tuning training of the accurate modeling noise removal network and the inaccurate modeling noise removal network.

[0075] Step 5: Save the training parameters, generate the denoised hyperspectral image, and complete the inference and indicator evaluation.

[0076] Among them, the hyperspectral image denoising and enhancement method based on noise decoupling of the embodiment of the present application is modeled through a noise model, and the decoupled noise is divided into accurately modeled noise and inaccurately modeled noise. A supervised hyperspectral image denoising dataset with accurately modeled noise is constructed based on a real paired dataset; a hyperspectral denoising network with accurately modeled noise is constructed, and the network is pre-trained using a supervised synthetic dataset; a hyperspectral denoising network with inaccurately modeled noise is constructed, and the network is trained using a real paired dataset in conjunction with an accurately modeled noise removal network; multi-stage supervision constraints are established (including: accurately modeled noise removal network pre-training, inaccurately modeled noise removal network pre-training and joint fine-tuning training), and network parameters are optimized; training parameters are saved, denoised hyperspectral images are generated, and reasoning and index evaluation are completed. In this way, the problem of existing real complex noise can be solved, and hyperspectral data can be effectively denoised.

[0077] Furthermore, in one embodiment of the present application, a training dataset is constructed based on the noise model and by calibrating an actual spectral camera to obtain camera parameters. The complete noise modeling can be expressed as:

[0078] Y=X+N (18)

[0079] N=N a +N i (19)

[0080] Where X represents the clean hyperspectral image, N represents the noise component, and Y represents the noisy image. The noise N can be accurately modeled by the component N a and the inaccurate modeling component N i combination.

[0081] The training dataset includes: pairs of accurately modeled noisy hyperspectral images (obtained based on the collected high-resolution images and the degradation model) and clean hyperspectral images. The degradation model is uniformly expressed as:

[0082] Y a (λ)=X(λ)+N a (λ) (20)

[0083] N a (λ)~g(k,m(λ),v(λ)) (21)

[0084] Where X(λ) represents the λ channel of the clean hyperspectral image, N a(λ) is the accurately modeled noise component of the λ channel, Y a (λ) represents the noisy image synthesized in the λ channel. g(k,m(λ),v(λ)) represents the Gaussian noise distribution, k represents the system gain, m(λ) means the value of m(λ), and v(λ) means the variance of v(λ) in the λ channel.

[0085] Furthermore, in one embodiment of the present application, a hyperspectral denoising network that accurately models noise is constructed and pre-trained using a supervised synthetic dataset. The deep learning network is mainly composed of an encoder and a decoder, both of which use a basic convolution-attention module as the basic unit:

[0086] F′=SpatialConv(F)+SpectralConv(F) (22)

[0087] Attention(Q,K,V)=V·Softmax(K·Q) (23)

[0088] F1=W·Attention(Q,K,V)+F′ (24)

[0089] Among them, F represents the image features extracted by the previous layer, SpatialConv represents 3D spatial convolution, that is, the convolution kernel size is (1, K, K) and convolution is performed only in the spatial dimension, SpectralConv represents 3D spectral convolution, that is, the convolution kernel size is (K, 1, 1) and convolution is performed only in the spectral dimension, and F′ represents the intermediate output feature. Q, K, and V represent the learnable mapping matrices obtained through linear processing, W represents the parameters that can be learned and optimized, and F1 represents the output feature. The entire network processing can be expressed as:

[0090]

[0091] Among them, f AMNet represents the network that processes accurate modeling noise at this stage, θ represents the parameter to be optimized, Represents the result after denoising, Y a A noisy image representing accurately modeled noise.

[0092] Furthermore, in one embodiment of the present application, a hyperspectral denoising network for inaccurately modeled noise is constructed, and the network is trained using a real paired dataset in conjunction with an accurately modeled noise removal network. For the hyperspectral denoising network for inaccurately modeled noise, a wavelet-guided convolutional network is constructed. First, the high-frequency information of the noise is extracted through the constructed wavelet filter:

[0093] Y L ,Y H =DWT(Y), (26)

[0094]

[0095] Where Y represents the real noisy image, DWT represents Haar discrete wavelet transform, and Y L ,Y H Respectively represent the decomposed low-frequency information and high-frequency information. abs represents the absolute value operation, Interpolate represents the interpolation operation, It means that after processing the extracted high-frequency information, high-frequency noise guidance information with the same spatial resolution as the original image is obtained. Multi-scale processing:

[0096]

[0097] Conv3d represents 3D convolution, and IWT represents inverse wavelet discrete transform. This step-by-step process obtains multi-level features G. i , i = 1, 2, 3, and insert these features into the decoder of the basic 3D convolutional network for denoising:

[0098]

[0099] where f IMNet represents the network that processes inaccurate modeling noise at this stage, θ represents the parameter to be optimized, Represents the result after denoising.

[0100] Furthermore, in one embodiment of the present application, multi-stage supervision constraints are established to optimize network parameters.

[0101] Specifically, the first stage accurately models the pre-training of the noise removal network. Based on the input and output hyperspectral images, L c loss:

[0102]

[0103] Where X represents the clean image, Represents the predicted image, ∈=10 -3 Represents a constant.

[0104] Specifically, in the second stage of training, the parameters of the accurate modeling noise removal network are fixed, and the parameters of the inaccurate modeling noise removal network are optimized and updated. In addition to the loss used in the previous stage, the Kullback-Leibler (KL) divergence loss is added:

[0105]

[0106] Among them, p(Y i ) represents the true distribution, Represents the predicted distribution, i represents the maximum number of pixels to be the total number of pixels.

[0107] Specifically, the third stage jointly fine-tunes the accurate modeling noise removal network and the inaccurate modeling noise removal network to eliminate the accumulated errors of each stage. In addition to the losses of the previous two stages, the loss:

[0108]

[0109] Where N is the number of pixels. Comprehensive use loss:

[0110]

[0111] where λ k and λ s represents a hyperparameter.

[0112] Furthermore, in one embodiment of the present application, the training parameters are saved, a denoised hyperspectral image is generated based on the input data, and inference and metric evaluation are completed. To objectively evaluate the effectiveness of the generated hyperspectral image, objective evaluation metrics can be generated based on peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM).

[0113] Figure 2 This is a schematic diagram of the neural network architecture used in this embodiment. It constructs a paired hyperspectral image dataset with accurate noise from real paired datasets and a camera noise model. Wavelet high-frequency filters extract multi-scale high-frequency features of the hyperspectral image noise. An inaccurate modeling noise removal network removes inaccurate modeling noise, combining the prior knowledge of the accurate modeling noise removal network for removing modeling noise. Compared to other denoising methods, this method can handle complex real-world noise scenarios and produce high-quality, clean hyperspectral images.

[0114] Figure 3 A schematic diagram of the composition of a hyperspectral image denoising and enhancement system based on noise decoupling provided in an embodiment of the present application includes a data preprocessing subsystem 10, an inaccurate modeling noise removal subsystem 20, an accurate modeling noise removal subsystem 30, and a supervised optimization and result evaluation subsystem 40:

[0115] The data preprocessing subsystem 10 is used to process the collected hyperspectral images and generate hyperspectral images with accurate modeling noise according to a specific camera degradation model, which are paired with clean hyperspectral images to form a training data set.

[0116] Furthermore, the inaccurate modeling noise removal subsystem 20 includes a wavelet high frequency filter and a removal network, which combines the prior knowledge of accurate modeling noise removal to remove the inaccurate modeling noise.

[0117] Furthermore, the accurate modeling noise removal subsystem 30 removes the accurate modeling noise.

[0118] Furthermore, the supervised optimization and result evaluation subsystem 40 is used to establish a loss function to optimize the network of the aforementioned system, further retain the trained network parameters and generate results, and use built-in evaluation indicators to evaluate the results.

[0119] The connection relationship between the above-mentioned component systems is:

[0120] The output end of the data preprocessing subsystem is connected to the input end of the accurate modeling noise removal subsystem and the inaccurate modeling noise removal subsystem, the output end of the inaccurate modeling noise removal subsystem is connected to the input end of the accurate modeling noise removal subsystem, and finally the output end of the accurate modeling noise removal subsystem is connected to the supervised optimization and result evaluation subsystem.

[0121] The explanation of the hyperspectral image denoising and enhancement method based on noise decoupling in the aforementioned embodiment is also applicable to the hyperspectral image denoising and enhancement system based on noise decoupling in this embodiment, and will not be repeated here.

[0122] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hyperspectral denoising and enhancement method based on noise decoupling, characterized in that: The following steps are involved: Step 1: Modeling is performed based on the noise model, and decoupling noise is divided into noise that can be accurately modeled and noise that cannot be accurately modeled. A supervised hyperspectral image denoising dataset with accurate modeling noise is constructed based on a real paired dataset. Step 2: Construct a hyperspectral denoising network that accurately models noise and pre-train the network using a supervised synthetic dataset. Step 3: Construct a hyperspectral denoising network with inaccurately modeled noise, and train the network using a real paired dataset in conjunction with the accurately modeled noise removal network. Step 4: Establish multi-stage supervision constraints and optimize network parameters. Step 4.1: Pre-training of the noise removal network for accurate modeling. Step 4.2: Training of the inaccurate modeling noise removal network and joint fixed accurate modeling noise removal network. Step 4.3: Joint fine-tuning training of the accurate modeling noise removal network and the inaccurate modeling noise removal network. Step 5: Save the training parameters, generate the denoised hyperspectral image, and complete the inference and indicator evaluation.

2. The method according to claim 1, wherein The noise decoupling and dataset construction in step 1 include: Noise modeling: Y=X+N N=Na+Ni Where X represents the clean hyperspectral image, N represents the noise component, and Y represents the noisy image. The noise N can be accurately modeled by the component N a and the inaccurate modeling component N i combination. Paired training dataset construction: Y a (λ)=X(λ)+N a (l) N a (λ)~g(k,m(λ),v(λ)) Where X(λ) represents the λ channel of the clean hyperspectral image, N a (λ) is the accurately modeled noise component of the λ channel, Y a (λ) represents the noisy image synthesized in the λ channel. g(k,m(λ),v(λ)) represents the Gaussian noise distribution, k represents the system gain, m(λ) means the value of m(λ), and v(λ) means the variance of v(λ) in the λ channel.

3. The method according to claim 1, wherein The accurate noise removal network in step 2 includes a convolution-attention based encoder and decoder, where the encoder is used to extract key multi-scale features, and the decoder restores a clean hyperspectral image based on these features.

4. The method according to claim 1, wherein The inaccurate noise removal network in step 3 extracts high-frequency information based on a wavelet filter, and the multi-scale feature extraction module fuses the high-frequency information to assist in noise removal.

5. The method according to claim 3, wherein The wavelet filter representation is: Y L ,Y H =DWT(Y), Where Y represents the real noisy image, DWT represents Haar discrete wavelet transform, and Y L ,Y H Respectively represent the decomposed low-frequency information and high-frequency information. abs represents the absolute value operation, Interpolate represents the interpolation operation, It means that the high-frequency noise guidance information with the same spatial resolution as the original image is obtained after processing the extracted high-frequency information.

6. The method according to claim 1, wherein The multi-stage supervision constraint in step 4 includes: pre-training of the accurate modeling noise removal network; training of the inaccurate modeling noise removal network; and joint fine-tuning training of the accurate modeling noise removal network and the inaccurate modeling noise removal network.

7. The method according to claim 6, wherein The accurate modeling noise removal network is pre-trained using L c loss: Where X represents the clean image, Represents the predicted image, ∈=10 -3 Represents a constant. The training of the inaccurate modeling noise removal network is augmented with the Kullback-Leibler (KL) divergence loss: Among them, p(Y i ) represents the true distribution, Represents the predicted distribution, and i represents the number of pixels. The joint fine-tuning training of the accurate modeling noise removal network and the inaccurate modeling noise removal network increases the use of loss:

8. The method according to claim 1, wherein The indicator evaluation in step 5 includes: objectively evaluating the generated hyperspectral image using peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and spectral angle mapping (SAM).

9. A hyperspectral denoising and enhancement system based on noise decoupling, characterized in that: include: Data preprocessing subsystem: used for data collection and processing to generate paired data. Inaccurate modeling noise removal subsystem: A deep network based on wavelet high-frequency filters removes inaccurate modeling noise. Accurate modeling noise removal subsystem: A deep network with attention mechanism is used to remove accurate modeling noise. Supervised optimization and result evaluation subsystem: saves training parameters and generates noise-removed hyperspectral images to complete performance evaluation.

Images