A hyperspectral image haze removal method and system
By extracting long-wavelength and shallow features from hyperspectral images and combining feature fusion attention and spatial scene attention calculations, the problem of poor haze removal effect in hyperspectral images was solved, achieving better haze removal and scene restoration results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNIV OF PETROLEUM (EAST CHINA)
- Filing Date
- 2023-10-20
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies are ineffective at removing haze from hyperspectral images, resulting in low spatial visibility and contrast, distorted spectral curves, and poor performance of traditional methods under complex haze conditions, with poor local contrast, color distortion, and loss of detail information.
A hyperspectral image haze removal method is adopted. By extracting long-wavelength fog-free land cover features and shallow features, and combining feature fusion attention calculation, spatial scene attention calculation and residual calculation, spectral spatial feature extraction and haze removal are achieved. Finally, the dehazed features are converted into dehazed images.
It effectively restored scene information under haze, improved the dehazing effect of hyperspectral images, and enhanced the reconstruction performance and dehazing effect of the network, especially in hyperspectral images of different scenes and frequency bands.
Smart Images
Figure CN117670718B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image processing technology, specifically to a method and system for removing haze from hyperspectral images. Background Technology
[0002] The rapid development of new Earth observation technologies has provided a wealth of optical remote sensing (RS) images with high spatial and spectral resolution for many fields, including environmental monitoring, resource exploration, and land management. However, due to scattering and absorption by atmospheric particles, optical RS images, including hyperspectral images, are inevitably contaminated by haze and clouds. Haze can cause the loss of some, but not all, scene information, resulting in lower visibility and contrast in the spatial domain and distorted curves in the spectral domain for high-resolution HSI. Therefore, it is crucial to recover scenes under haze conditions and improve the quality of haze images to maintain the advantage of accurately identifying HSI in various RS applications.
[0003] To address the haze problem in RS images, researchers have proposed numerous dehazing methods, which can be broadly categorized into two types: model-driven and data-driven methods. Model-driven methods typically possess robust physical mechanisms, but model derivation is often a challenging task. Existing model-driven methods can be broadly classified into two categories: methods based on atmospheric scattering models and methods based on dark matter subtraction models. However, atmospheric scattering models are only applicable to horizontal visual imaging; because these model-based methods are usually designed for a specific region or scene, they perform poorly in complex hazy conditions, resulting in problems such as poor local contrast, color distortion, and loss of detail.
[0004] In recent years, with the rapid development of deep learning in computer vision and image processing, data-driven dehazing methods have demonstrated state-of-the-art performance. Compared with model-driven methods, data-driven methods have stronger applicability and generalization ability. Current data-driven methods are all based on multispectral image design and are applicable to a limited number of bands. Recently, researchers have proposed a hyperspectral dehazing method based on spectral grouping. However, this method processes highly correlated bands separately, ignoring the spectral characteristics of the global bands. Therefore, the dehazing effect varies in hyperspectral images of different scenes and with different numbers of frequency bands. Summary of the Invention
[0005] In view of the above-mentioned shortcomings in the prior art, the present invention provides a method and system for removing haze from hyperspectral images, which solves the problem of poor haze removal effect of the prior art.
[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:
[0007] A method for removing haze from hyperspectral images is provided, which includes the following steps:
[0008] S1. Extract fog-free land cover features from long-wavelength hyperspectral images;
[0009] S2. Extract shallow features from hyperspectral images;
[0010] S3. The shallow features are fused with the fog-free land cover features to obtain the fused features;
[0011] S4. Extract spectral spatial features and remove fog from the fused features to obtain dehazed features;
[0012] S5. Convert the defogging features into defogging images to complete the removal of haze from hyperspectral images.
[0013] Furthermore, the specific method of step S1 includes the following sub-steps:
[0014] S1-1. After performing two convolutions on the long bands in the hyperspectral image, the first spatial scene attention calculation is performed to obtain the first fog-free land cover feature.
[0015] S1-2. After performing the first residual calculation on the first fog-free land cover feature, perform the second spatial scene attention calculation to obtain the second fog-free land cover feature.
[0016] S1-3. After performing a second residual calculation on the second fog-free land cover feature, a third spatial scene attention calculation is performed to obtain the third fog-free land cover feature.
[0017] Furthermore, the specific method of step S2 includes:
[0018] Perform a convolution on the hyperspectral image once to obtain the first shallow feature layer; perform a series of convolutions on the first shallow feature layer to obtain the second shallow feature layer.
[0019] Furthermore, the specific method of step S3 includes the following sub-steps:
[0020] S3-1. After splicing the second shallow layer feature with the first fog-free land cover feature, perform feature fusion attention calculation to obtain the first fused feature;
[0021] S3-2. Perform residual calculation on the first fused feature to obtain the first residual result;
[0022] S3-3. After splicing the first residual result with the second fog-free land cover feature, perform feature fusion attention calculation to obtain the second fused feature;
[0023] S3-4. Perform residual calculation on the second fusion feature to obtain the second residual result;
[0024] S3-5. After splicing the second residual result with the third fog-free land cover feature, perform feature fusion attention calculation to obtain the third fused feature.
[0025] Furthermore, the specific method of step S4 includes the following sub-steps:
[0026] S4-1. Perform residual calculation on the third fusion feature to obtain the third residual result;
[0027] S4-2. Perform a jump connection on the first residual result, the second residual result, and the third residual result to obtain the first connection result;
[0028] S4-3. Convolve the first connection result and then perform a jump connection between the convolution result of the first connection result and the first fused feature to obtain the second connection result, which is the dehazing feature.
[0029] Furthermore, the specific method for step S5 is as follows:
[0030] By performing two convolutions on the dehazing features, the dehazing features are converted into dehazing images, thus completing the removal of haze from hyperspectral images.
[0031] Furthermore, the specific method for feature fusion attention computation includes the following steps:
[0032] A1. Input the input features into a 1×1 convolutional layer and a 3×3 convolutional layer in sequence to obtain the first 3×3 convolution result;
[0033] A2. Input the first 3×3 convolution result into a global average pooling layer, a 1×1 convolution layer, a ReLU layer, a 1×1 convolution layer and a Sigmoid activation layer in sequence to obtain the first Sigmoid activation result;
[0034] A3. Multiply the first Sigmoid activation result and the first 3×3 convolution result to obtain the first multiplication result;
[0035] A4. Input the first multiplication result into a 3×3 depth convolutional layer, a 3×3 depth dilated convolutional layer and a 1×1 convolutional layer in sequence to obtain the first 1×1 convolution result;
[0036] A5. Multiply the first 1×1 convolution result with the first multiplication result to obtain the feature fusion attention calculation result.
[0037] Furthermore, the specific method for spatial scene attention calculation includes the following steps:
[0038] B1. Input the input features sequentially into a 1×1 convolutional layer, a ReLU layer, a 1×1 convolutional layer, and a Sigmoid activation layer to obtain the second Sigmoid activation result;
[0039] B2. Multiply the second Sigmoid activation result with the input feature to obtain the second multiplication result;
[0040] B3. Input the second multiplication result into a 3×3 convolutional layer, and input the output of the 3×3 convolutional layer into parallel 3×3 convolutional layers and 5×5 convolutional layers respectively. Concatenate the outputs of the parallel 3×3 convolutional layers and 5×5 convolutional layers to obtain the first concatenation result.
[0041] B4. Input the first concatenation result into a Sigmoid activation layer and a 3×3 convolutional layer in sequence to obtain the second 3×3 convolution result;
[0042] B5. Multiply the second 3×3 convolution result with the second multiplication result and input the result into a 3×3 convolution layer to obtain the spatial scene attention calculation result.
[0043] Furthermore, the specific method for residual calculation is as follows:
[0044] The input features are sequentially fed into a 3×3 convolutional layer, a 1×1 convolutional layer, and a 3×3 convolutional layer to obtain the third 3×3 convolution result; the input features are fed into a 1×1 convolutional layer to obtain the second 1×1 convolution result; the third 3×3 convolution result and the second 1×1 convolution result are added together to obtain the residual calculation result.
[0045] A hyperspectral image haze removal system is provided, comprising a haze-free land cover feature extraction module, a shallow feature extraction module, a feature fusion module, a haze removal module, and a dehazed image conversion module; wherein:
[0046] The fog-free land cover feature extraction module is used to extract fog-free land cover features in the long band of hyperspectral images;
[0047] The shallow feature extraction module is used to extract shallow features from hyperspectral images;
[0048] The feature fusion module is used to fuse shallow features with fog-free land cover features to obtain fused features;
[0049] The fog removal module is used to extract spectral spatial features and remove fog from the fused features to obtain dehazed features;
[0050] The dehazing image conversion module is used to convert dehazing features into dehazing images, thereby completing the removal of haze from hyperspectral images.
[0051] The beneficial effects of this invention are as follows:
[0052] 1. This invention can effectively learn the full-band mapping relationship from foggy to fog-free, and inject long-wavelength information into the defogging network to help restore fog-free scenes, thus solving the problem of insufficient utilization of spectral and spatial information in traditional methods.
[0053] 2. This invention utilizes the cooperation of a main branch and an auxiliary branch. The main branch fully learns the mapping relationship from foggy to fog-free by using attention mechanism and global residual learning, while the auxiliary branch injects ground feature information in long-wavelength images into the defogging process to improve the reconstruction performance of the network, that is, to improve the defogging effect of hyperspectral images.
[0054] 3. The Feature Fusion Attention Calculation Method (FFAB) proposed in this invention allows the fusion of features from two branches, focusing more attention on more important spectral channels and spatial information; the Spatial Scene Attention Calculation Method (SSAB) proposed in this invention allows for the further extraction of fog-free and useful scene features from long-wavelength information, thereby effectively improving the dehazing effect of hyperspectral images. Attached Figure Description
[0055] Figure 1 This is a schematic diagram of the overall process of this method;
[0056] Figure 2 This is a schematic diagram of the overall structure of the system;
[0057] Figure 3 Flowchart for feature fusion attention calculation;
[0058] Figure 4 Flowchart for spatial scene attention calculation;
[0059] Figure 5 Here is a flowchart of the residual calculation process;
[0060] Figure 6 This is a true-color (after grayscale processing) image after dehazing, as shown in the example. Detailed Implementation
[0061] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0062] like Figure 1 As shown, the hyperspectral image haze removal method includes the following steps:
[0063] S1. Extract fog-free land cover features from long-wavelength hyperspectral images;
[0064] S2. Extract shallow features from hyperspectral images;
[0065] S3. The shallow features are fused with the fog-free land cover features to obtain the fused features;
[0066] S4. Extract spectral spatial features and remove fog from the fused features to obtain dehazed features;
[0067] S5. Convert the defogging features into defogging images to complete the removal of haze from hyperspectral images.
[0068] In one embodiment of the present invention, the Gaofen-5 AHSI hyperspectral sensor has a total of 330 bands, with spectral resolutions of 5 nm (visible and near-infrared band) and 10 nm (mid-infrared band), a spatial resolution of 30 m, a radiometric resolution of 12 bits, and a swath width of 60 km. Bands [193-200] and [246-262] of the AHSI image are affected by water and atmospheric absorption, so these bands are also removed in this embodiment. Therefore, the fog-containing Gaofen-5 image used in this embodiment has a total of 305 bands, and the three images are 512 x 512 pixels in size, covering wetlands, vegetation, and water-land interface areas, respectively. Specifically, the long band in step S1 is the [246-287] band.
[0069] like Figure 2 As shown, the overall structure of this invention consists of two main branches: the main branch is an end-to-end full-band attention network, and the auxiliary branch is a long-wavelength scene attention network. In the main branch, fog imagery is used as input. First, two convolutional layers are used for shallow feature extraction. Then, three sets of feature fusion attention blocks (FFAB) and residual blocks (RB) are used for spectral spatial feature extraction and fog removal. Finally, three convolutional layers are used to convert the extracted features into a de-fog image. Three jump connections are used for the fusion of deep and shallow features and global residual learning. In the auxiliary branch, long-wavelength bands that are not sensitive to fog are extracted and fed into two convolutional layers to extract shallow features. Then, they are fed into a combination of three spatial scene attention blocks (SSAB) and residual blocks (RB) to extract fog-free land cover features. These features help with scene de-fog removal in the main branch.
[0070] Feature Fusion Attention Block (FFAB) is a two-attention mechanism used to fuse features from the main branch and auxiliary branch, and adds weights to important denoised spectral spatial information. For example... Figure 3As shown, firstly, a 1×1 convolutional layer and a 3×3 convolutional layer are connected to reduce dimensionality and shallowly fuse the superimposed features from the main and auxiliary branches. Secondly, channel attention is employed, where global average pooling (GAP) is used for feature extraction, two fully connected layers (a 1×1 convolution, a rectified linear unit (ReLU), and another 1×1 convolution) perform self-attention on the spectral channel features, and a sigmoid activation function weights the original features, which helps in the spectral reconstruction of pixels in hazy regions. Finally, 3D convolutional attention is used to further extract joint spectral and spatial features beneficial for dehazing. Specifically, a 3×3 depthwise convolution (DW-Conv) is used to extract local spatial structure information, a 3×3 depthwise dilated convolution (DW-D-Conv) is used to extract long-range dependencies in the spatial domain, and a 1×1 pointwise convolution is used to extract spectral adaptability. Compared with traditional 3D convolution, it can extract key 3D features of hyperspectral images with fewer parameters.
[0071] The Spatial Scene Attention Block (SSAB) integrates pixel attention and spatial attention mechanisms to extract fog-free spatial scene features from long-wavelength bands, such as... Figure 4 As shown, long-wavelength bands are not sensitive to haze, but haze still exists and its spatial distribution is uneven. A pixel attention module is first used to focus on informational features, such as haze-free land cover areas, and to reduce the haze intensity of certain haze pixels. Then, a spatial attention module consisting of convolutional layers with different filter sizes is used to extract multi-scale semantic and contextual features, which is beneficial for spatial reconstruction of different land covers. For spatial attention, parallel 3×3 and 5×5 convolutional layers are used to replace channel-dimensional pooling operations, followed by feature concatenation, 3×3 convolution, and a sigmoid activation function to assign weights to the spatial scene. The final 3×3 convolution is used to optimize the weighted features. The resulting spatial scene features are added to FFAB to help recover the haze-free scene.
[0072] Residual blocks (RBs) are used to improve feature extraction performance and the network's learning ability, such as... Figure 5 As shown, residual learning allows convolutional layers with different filter sizes to learn the rich spatial semantics and spectral information of hyperspectral images, thereby improving the training stability of deep haze-free feature extraction. Specifically, three convolutional layers with filter sizes of 3×3, 1×1, and 3×3 are used in the main data stream, while a 1×1 convolutional layer is used in the residual data stream. It is important to note that the batch normalization (BN) layer in the traditional residual block is not used, as it might reduce the flexibility of the feature range through normalization.
[0073] In the specific implementation process, considering the complexity of spectral and spatial information in hyperspectral images, we designed a new loss function for network training. To make the reconstructed image as close as possible to a clean image, we first adopted a simple and efficient mean squared error loss (MSEloss) function, the expression of which is:
[0074] L MSE =||H pre -H hazefree ||
[0075] Where H pre and H hazefree These represent the recovered hyperspectral image and the corresponding haze-free image, respectively.
[0076] Due to the vast spectral bands of hyperspectral images and the varying sensitivities of different wavelengths to haze, effectively reconstructing the spectral curve is one of the key objectives in haze removal. Based on the difference between hazy and haze-free spectra, we propose a novel spectral loss function, defined as follows:
[0077] L Spec =||sp pre -sp hazefree ||
[0078] Among them, sp pre sp represents the average spectral pixels of the restored image. hazefree This represents the average spectral pixels of the corresponding haze-free image. The total loss function is defined as:
[0079] L Total =L MSE +L Spec
[0080] To verify the dehazing effect of this system (HyD-Net), this embodiment selected 1600 foggy-fog image pairs for network training, and 3 scene images for testing. The entire network was trained with 5000 epoachs, using the Adam optimization operator to accelerate training, with momentum parameters set to 0.9, 0.999, and 10, respectively. -8 The initial learning rate is set to 1×10. -4The learning rate was adjusted using a cosine strategy. The entire network was implemented on the PyTorch framework and an NVIDIA GeForce RTX 3060 GPU. Four quantitative evaluation metrics were selected for evaluating dehazed images: structural similarity (SSIM), peak signal-to-noise ratio (PSNR), universal image quality index (UQI), and spectral angle mapping (SAM).
[0081] Table 1 compares the results of this invention with those of the classic DCP, the model-driven method Defog, and the data-driven method SG-Net in terms of quantitative evaluation metrics. As shown in Table 1, compared to existing dehazing methods, this invention achieves the optimal quality evaluation metric value and exhibits good dehazing robustness in different scenarios. The dehazed true-color (after grayscale processing) image is shown below. Figure 6 As shown, the DCP method cannot remove haze interference from hyperspectral images; the Defog method exhibits excessive dehazing, resulting in severe distortion of the ground object spectra; SG-Net suffers from scene distortion; and the scene restored by this invention is closest to the real scene. Overall, this invention has superior dehazing and scene restoration capabilities.
[0082] Table 1: Comparison of Quality Evaluation After Dehazing of Hyperspectral Images
[0083]
[0084]
[0085] In summary, this invention proposes novel feature fusion attention, spatial scene attention, and spectral loss. The novel feature fusion attention combines spectral attention with large-kernel convolution to fully integrate long-wavelength information from auxiliary branches, obtaining spectral and 3D depth features beneficial for image dehazing. Spatial scene attention combines spatial attention with pixel attention to learn important ground feature scene information from long-wavelength bands, which plays a crucial role in haze removal in the main branch. Spectral loss is constructed based on the difference in average spectral vectors. Compared to variance loss, which constrains the overall spectral and spatial pixel value reconstruction, it focuses on the recovery and reconstruction of spectral curves for different ground features, improving haze removal performance.
Claims
1. A method for removing haze from hyperspectral images, characterized in that, Includes the following steps: S1. Extract fog-free land cover features from long-wavelength hyperspectral images; S2. Extract shallow features from hyperspectral images; S3. The shallow features are fused with the fog-free land cover features to obtain the fused features; S4. Extract spectral spatial features and remove fog from the fused features to obtain dehazed features; S5. Convert the defogging features into a defogging image to complete the haze removal from the hyperspectral image. The specific method of step S1 includes the following sub-steps: S1-1. After performing two convolutions on the long bands in the hyperspectral image, the first spatial scene attention calculation is performed to obtain the first fog-free land cover feature. S1-2. After performing the first residual calculation on the first fog-free land cover feature, perform the second spatial scene attention calculation to obtain the second fog-free land cover feature. S1-3. After performing a second residual calculation on the second fog-free land cover feature, a third spatial scene attention calculation is performed to obtain the third fog-free land cover feature. The specific methods for step S2 include: Perform a convolution on the hyperspectral image once to obtain the first shallow layer feature; perform a convolution on the first shallow layer feature in turn to obtain the second shallow layer feature; The specific method of step S3 includes the following sub-steps: S3-1. After splicing the second shallow layer feature with the first fog-free land cover feature, perform feature fusion attention calculation to obtain the first fused feature; S3-2. Perform residual calculation on the first fused feature to obtain the first residual result; S3-3. After splicing the first residual result with the second fog-free land cover feature, perform feature fusion attention calculation to obtain the second fused feature; S3-4. Perform residual calculation on the second fusion feature to obtain the second residual result; S3-5. After splicing the second residual result with the third fog-free land cover feature, perform feature fusion attention calculation to obtain the third fused feature.
2. The method for removing haze from hyperspectral images according to claim 1, characterized in that, The specific method of step S4 includes the following sub-steps: S4-1. Perform residual calculation on the third fusion feature to obtain the third residual result; S4-2. Perform a jump connection on the first residual result, the second residual result, and the third residual result to obtain the first connection result; S4-3. Convolve the first connection result and then perform a jump connection between the convolution result of the first connection result and the first fused feature to obtain the second connection result, which is the dehazing feature.
3. The method for removing haze from hyperspectral images according to claim 2, characterized in that, The specific method for step S5 is as follows: By performing two convolutions on the dehazing features, the dehazing features are converted into dehazing images, thus completing the removal of haze from hyperspectral images.
4. The method for removing haze from hyperspectral images according to claim 1, characterized in that, The specific method for feature fusion attention calculation includes the following steps: A1. Input the input features into a 1×1 convolutional layer and a 3×3 convolutional layer in sequence to obtain the first 3×3 convolution result; A2. Input the first 3×3 convolution result into a global average pooling layer, a 1×1 convolution layer, a ReLU layer, a 1×1 convolution layer and a Sigmoid activation layer in sequence to obtain the first Sigmoid activation result; A3. Multiply the first Sigmoid activation result and the first 3×3 convolution result to obtain the first multiplication result; A4. Input the first multiplication result into a 3×3 depth convolutional layer, a 3×3 depth dilated convolutional layer and a 1×1 convolutional layer in sequence to obtain the first 1×1 convolution result; A5. Multiply the first 1×1 convolution result with the first multiplication result to obtain the feature fusion attention calculation result.
5. The method for removing haze from hyperspectral images according to claim 1, characterized in that, The specific method for spatial scene attention calculation includes the following steps: B1. Input the input features sequentially into a 1×1 convolutional layer, a ReLU layer, a 1×1 convolutional layer, and a Sigmoid activation layer to obtain the second Sigmoid activation result; B2. Multiply the second Sigmoid activation result with the input feature to obtain the second multiplication result; B3. Input the second multiplication result into a 3×3 convolutional layer, and input the output of the 3×3 convolutional layer into parallel 3×3 convolutional layers and 5×5 convolutional layers respectively. Concatenate the outputs of the parallel 3×3 convolutional layers and 5×5 convolutional layers to obtain the first concatenation result. B4. Input the first concatenation result into a Sigmoid activation layer and a 3×3 convolutional layer in sequence to obtain the second 3×3 convolution result; B5. Multiply the second 3×3 convolution result with the second multiplication result and input the result into a 3×3 convolution layer to obtain the spatial scene attention calculation result.
6. The method for removing haze from hyperspectral images according to claim 1, characterized in that, The specific method for residual calculation is as follows: The input features are sequentially fed into a 3×3 convolutional layer, a 1×1 convolutional layer, and a 3×3 convolutional layer to obtain the third 3×3 convolution result; the input features are fed into a 1×1 convolutional layer to obtain the second 1×1 convolution result; the third 3×3 convolution result and the second 1×1 convolution result are added together to obtain the residual calculation result.
7. A system based on the hyperspectral image haze removal method according to any one of claims 1 to 6, characterized in that, It includes a fog-free land cover feature extraction module, a shallow feature extraction module, a feature fusion module, a fog removal module, and a dehazed image conversion module; among which: The fog-free land cover feature extraction module is used to extract fog-free land cover features in the long band of hyperspectral images; The shallow feature extraction module is used to extract shallow features from hyperspectral images; The feature fusion module is used to fuse shallow features with fog-free land cover features to obtain fused features; The fog removal module is used to extract spectral spatial features and remove fog from the fused features to obtain dehazed features; The dehazing image conversion module is used to convert dehazing features into dehazing images, thereby completing the removal of haze from hyperspectral images.