A hyperspectral image restoration method

Abstract

The application discloses a hyperspectral image restoration method, belonging to the technical field of hyperspectral imaging, which comprises inputting a single hyperspectral image to be restored into a hyperspectral image restoration network for denoising and super-resolution reconstruction. The denoising method comprises grouping a noisy hyperspectral image by spectral overlap, inputting the grouped image into a multi-scale space-spectrum fusion module to obtain respective branch features, splicing and dimensioning the branch features to obtain dimensioned local multi-scale space-spectrum features, and inputting the local multi-scale space-spectrum features into an adaptive space feature aggregation group to obtain global space-spectrum features. The noisy hyperspectral image is fused with the global space-spectrum features after being processed by a convolution layer to obtain a final denoising image. The super-resolution reconstruction method can be obtained by extending the denoising method, which comprises upsampling the global space-spectrum features, and upsampling the low-resolution hyperspectral image to be restored before the image is processed by the convolution layer. The application solves the problems of noise introduction and spatial resolution reduction in the process of hyperspectral imaging in the prior art.

Description

Technical Field

[0001] This invention relates to a hyperspectral image restoration method, belonging to the field of hyperspectral imaging technology. Background Technology

[0002] In recent years, with the continuous deepening of remote sensing technology research, the application of hyperspectral imaging has also flourished. Hyperspectral imaging captures rich spectral information by capturing hundreds of continuous bands from the visible spectrum, near-infrared to far-infrared. Benefiting from its higher spectral resolution, hyperspectral images produce more distinctive feature representations than traditional RGB images. Hyperspectral image restoration encompasses two aspects: high-resolution super-resolution reconstruction (HSI-SR) and denoising (HSID), aiming to improve the quality of hyperspectral images (HSI). Due to physical characteristics and cost limitations, hyperspectral imaging equipment cannot simultaneously acquire images with high spectral and high spatial resolution. Furthermore, influenced by physical limitations and environmental factors, images are often accompanied by atmospheric scattering and sensor noise, affecting the accuracy of subsequent analysis. Therefore, researching hyperspectral image restoration techniques can compensate for equipment limitations, improve the image signal-to-noise ratio, reduce data processing difficulty, and provide a high-quality data foundation for tasks such as land cover classification and anomaly detection.

[0003] Convolutional neural networks (CNNs) are widely used in hyperspectral image restoration due to their automatic feature learning capabilities and ability to process high-dimensional spatial-spectral information. However, their dependence on large amounts of labeled data, their emphasis on spatial features while neglecting spectral information, and the need for high-resolution auxiliary images still pose challenges to their processing. Therefore, research focuses on how to effectively extract spatial-spectral information without auxiliary images and utilize global and local spatial-spectral correlations to improve the quality of hyperspectral image restoration.

[0004] In summary, existing technologies have significant shortcomings in hyperspectral image restoration. Noise is often introduced during hyperspectral imaging, leading to a decrease in spatial resolution. Furthermore, the prevalent issues of insufficient and imbalanced samples in hyperspectral images make it difficult for existing technologies to reliably restore the quality of degraded hyperspectral images. These problems collectively limit the accuracy and effectiveness of hyperspectral image restoration. Summary of the Invention

[0005] The purpose of this invention is to provide a hyperspectral image restoration method that improves the accuracy of hyperspectral image restoration, solves the problems of noise and spatial resolution reduction introduced in the hyperspectral imaging process by existing technologies, and stably completes the task of image quality restoration of degraded hyperspectral images under the conditions of insufficient and unbalanced samples that are common in hyperspectral images.

[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0007] This invention provides a hyperspectral image restoration method, comprising:

[0008] Acquire the single hyperspectral image (HSI) to be restored;

[0009] The single hyperspectral image HSI to be restored is input into the hyperspectral image restoration network SSFSA for denoising / super-resolution reconstruction to obtain the restored hyperspectral image.

[0010] The restored hyperspectral image includes the denoised hyperspectral image / the hyperspectral image after super-resolution reconstruction;

[0011] The method for denoising using the hyperspectral image restoration network SSFSA includes:

[0012] Using a spectral overlap grouping and parameter sharing strategy, the number of overlapping spectral channels is set, and the HSI of a single hyperspectral image is overlapped and grouped along the spectral dimension. The HSI of each group of hyperspectral images is input into the corresponding multi-scale spatial-spectral fusion module MSSF to extract the mapping relationship in the multi-scale space, obtain all branch features, concatenate all branch features, and increase the dimensionality through a 1×1 convolutional layer to obtain the dimensionality-increased local multi-scale spatial-spectral features. Among them, all groups share the network parameters of each multi-scale spatial-spectral fusion module MSSF.

[0013] The upgraded local multi-scale spatial-spectral features are input into the adaptive spatial feature aggregation group ASAG to obtain global spatial-spectral features.

[0014] The single hyperspectral image is passed through a 1×1 convolutional layer, fused with global spatial-spectral features, and then passed through another 1×1 convolutional layer to obtain a denoised hyperspectral image.

[0015] Among them, the method of denoising by extending the hyperspectral image restoration network SSFSA, and the method of super-resolution reconstruction by the hyperspectral image restoration network SSFSA, include:

[0016] After obtaining the global spatial-spectral features, an upsampling operation is performed;

[0017] Before passing the single hyperspectral image through a 1×1 convolutional layer, a bicubic upsampling operation is performed.

[0018] Furthermore, single hyperspectral images are grouped along the spectral dimension and input into the Multi-Scale Spatial-Spectral Fusion (MSSF) module to extract the mapping relationships in the multi-scale space, obtaining all branch features. All branch features are then concatenated and upscaled using a 1×1 convolutional layer to obtain the upscaled local multi-scale spatial-spectral features, including:

[0019] After grouping a single hyperspectral image along the spectral dimension, the images are input into two half-channel branches and one full-channel branch of the Multi-Scale Spatial-Spectral Fusion Module (MSSF) to obtain two half-channel branch features and one full-channel branch feature.

[0020] After concatenating the two half-channel branch features, the ReLU activation function is applied, and then the full-channel branch features are added pixel by pixel to obtain the fused multi-scale features.

[0021] The fused multi-scale features are then passed through a spatial residual module to obtain multi-scale spatial features.

[0022] The multi-scale spatial features are fed into the channel attention module CAB of the residual connection to obtain multi-scale spatial-spectral features;

[0023] After pixel-by-pixel addition of multi-scale spatial-spectral features and multi-scale spatial features, branch features are obtained;

[0024] All branch features are concatenated and then increased in dimensionality through a 1×1 convolutional layer to obtain the increased local multi-scale spatial-spectral features.

[0025] Furthermore, the local multi-scale spatial-spectral features are represented as follows:

[0026] ;

[0027] In the formula, This represents the local multi-scale spatial-spectral features after dimensionality enhancement. This indicates the number of spectral channels after dimensionality upscaling. Represents the height in the image spatial resolution. Represents the width in the image spatial resolution. This represents a 1×1 convolution operation. Indicates a cascading operation. This represents the local multi-scale spatial-spectral features obtained by cascading all branch features. This represents the feature of the first branch. This represents the second branch feature. Indicates the first Branch features, Indicates the first Branch features, This indicates the number of spectral channels in the single hyperspectral image (HSI) to be restored.

[0028] Furthermore, after grouping the single hyperspectral image along the spectral dimension, it is input into two half-channel branches and one full-channel branch of the multi-scale spatial-spectral fusion module MSSF, respectively, to obtain two half-channel branch features and one full-channel branch feature, including:

[0029] After grouping a single hyperspectral image along the spectral dimension, the number of spectral channels in each group is divided into two and input into the two half-channel branches of the Multi-Scale Spatial-Spectral Fusion Module (MSSF). After two 3×3 convolution operations, each 3×3 convolution operation is processed with the ReLU activation function, followed by another 3×3 convolution operation, and finally through a layer of dilated convolution to obtain the features of the two half-channel branches.

[0030] After grouping a single hyperspectral image along the spectral dimension, all spectral channels of each group are input into the full-channel branch of the Multi-Scale Spatial-Spectral Fusion Module (MSSF). After processing through three cascaded 3×3 convolutions with ReLU activation functions, and then through a layer of dilated convolution, the full-channel branch features are obtained.

[0031] Furthermore, the multi-scale spatial features are represented as follows:

[0032] ;

[0033] In the formula, Represents multi-scale spatial features. This represents the multi-scale feature resulting from the fusion of half-channel branch output features and full-channel branch output features. This represents the first grouping of a single hyperspectral image along its spectral dimension. Groups, This represents a 3×3 convolution operation.

[0034] Furthermore, the step of feeding the multi-scale spatial features into the channel attention module (CAB) of the residual connection to obtain multi-scale spatial-spectral features includes:

[0035] The first channel attention feature is obtained by performing two 3×3 convolution operations on the multi-scale spatial features. The GELU activation function is used after the first 3×3 convolution operation.

[0036] The first channel attention feature is passed through a global average pooling layer and then through two 3×3 convolution operations to obtain the second channel attention feature. The ReLU activation function is used after the first 3×3 convolution operation.

[0037] The second channel attention feature is processed by the Sigmoid function and then multiplied pixel-by-pixel with the first channel attention feature to obtain the multi-scale spatial-spectral feature.

[0038] Furthermore, the adaptive spatial feature aggregation group ASAG includes A cascaded adaptive spatial aggregation module SAB, the A series of cascaded adaptive spatial aggregation modules form residuals through long and short skip connections, where... .

[0039] Furthermore, the upgraded local multi-scale spatial-spectral features are input into the adaptive spatial feature aggregation group ASAG to obtain global spatial-spectral features. Each cascaded adaptive spatial aggregation module SAB in the adaptive spatial feature aggregation group ASAG includes:

[0040] For each adaptive spatial aggregation module (SAB), after layer normalization, it goes through a parallel three-branch structure including a channel attention module (CAB), a 3×3 depthwise convolution (DWConv), and a learnable convolutional kernel attention (LCK) to obtain the output features of the three branches.

[0041] The output features of the three branches are aggregated using pixel-wise multiplication to perform spatial and spectral feature aggregation. Then, the features are passed through a 1×1 convolutional layer and residual connections to obtain the global deep features output by the adaptive spatial aggregation module SAB.

[0042] Specifically, the learnable convolutional kernel attention (LCK) is used to adaptively process the spatial features of the local multi-scale spatial-spectral features after layer normalization; the 3×3 depthwise convolution (DWConv) is used to capture the spatially invariant features of the local multi-scale spatial-spectral features after layer normalization; and the channel attention module (CAB) is used to further extract and refine the spectral features of the local multi-scale spatial-spectral features after layer normalization.

[0043] Furthermore, the upgraded local multi-scale spatial-spectral features are processed through N cascaded SABs to obtain global deep features, which are represented as follows:

[0044] ;

[0045] In the formula, Indicates the first A global deep feature, Indicates the first One SAB module, Indicates the first A global deep feature, Indicates the first One SAB module, This indicates the first SAB module. This represents the local multi-scale spatial-spectral features after dimensionality increase;

[0046] The global spatial-spectral features are represented as follows:

[0047] ;

[0048] In the formula, Represents global spatial-spectral characteristics, "" indicates pixel-by-pixel addition.

[0049] Furthermore, the output features of the three branches include LCK output features, wherein the method for obtaining the LCK output features includes:

[0050] We learn a set of convolutional kernels shared across all spatial locations and all images by utilizing the local multi-scale spatial-spectral features after layer normalization;

[0051] A lightweight convolutional branch is used to predict each spatial location of the local multi-scale spatial-spectral features after the prediction layer is normalized, to obtain a fusion coefficient map. The lightweight convolutional branch contains two 3×3 grouped convolutional layers and a SimpleGate activation function.

[0052] The fusion coefficient feature map and the convolution kernel are linearly fused, and the fusion weight of each spatial location of the normalized local multi-scale spatial-spectral features is calculated by linear combination of the convolution kernels.

[0053] The local multi-scale spatial-spectral features after layer normalization are transformed by 1×1 convolution to obtain the enhanced feature map;

[0054] The enhanced feature map and the fusion weights of the layer-normalized local multi-scale spatial-spectral features at each spatial location are subjected to group convolution operation. Through group convolution calculation, the LCK output feature at each spatial location is obtained.

[0055] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0056] 1. This invention, by employing spectral grouping and parameter sharing strategies, fully leverages the strong correlation between adjacent bands, effectively improving computational efficiency and feature extraction accuracy during the restoration process. Simultaneously, it combines a multi-scale spatial-spectral fusion module (MSSF) to capture spatial and spectral information of the image at different scales, achieving richer and more refined image restoration results. Furthermore, it applies an adaptive spatial aggregation group (ASAG), which adaptively optimizes the spatial information aggregation process using learnable convolutional kernels and automatically adjusts the aggregation strategy based on image content, thereby significantly restoring image details and structure. In summary, the hyperspectral image restoration method provided by this invention can significantly improve restoration accuracy, effectively addressing the noise and spatial resolution degradation problems introduced during hyperspectral imaging, and achieving stable image quality restoration even under conditions of insufficient and imbalanced samples.

[0057] 2. This invention designs a multi-scale spatial-spectral fusion module (MSSF). First, it utilizes a parallel branch-group convolutional structure, including half-channel and full-channel branches, to effectively fuse spatial and spectral information at different scales. This allows for the parallel enhancement of internal and external relationships between different spectral bands, facilitating the capture of smooth image features related to low-frequency information. The half-channel branch reduces computation while maintaining feature richness, while the full-channel branch further extracts global features. Second, each branch uses a layer of dilated convolution to increase the receptive field without increasing network parameters, extracting multi-scale spatial information from different receptive fields. Finally, a channel attention module with residual connections is added, utilizing spectral similarity and complementarity to mine the correlation of spectral features. This not only improves computational efficiency but also significantly enhances the precision and accuracy of image restoration, particularly when processing complex hyperspectral images.

[0058] 3. The adaptive spatial feature aggregation group ASAG provided by this invention, through cascaded adaptive spatial aggregation modules SAB and combined with long and short skip connections to form residuals, achieves dynamic adjustment and adaptive aggregation of local multi-scale spatial-spectral features. In particular, the three-branch structure within the adaptive spatial aggregation module SAB can finely extract and fuse spatial and spectral features, further improving the detail recovery capability and structural preservation of image restoration. Furthermore, the learnable convolutional kernel attention (LCK) achieves adaptive selection and fusion of learnable convolutional kernels at each location by predicting the fusion coefficient map, thereby enhancing the expressive power of global spatial-spectral features. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the structure of the hyperspectral image restoration network SSFSA provided in an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of the structure of the multi-scale spatial spectrum fusion module MSSF provided in an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of the adaptive spatial aggregation module SAB provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0063] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0064] Example 1

[0065] like Figure 1 As shown in the figure, this embodiment introduces a hyperspectral image restoration method, including:

[0066] Acquire the single hyperspectral image (HSI) to be restored;

[0067] Hyperspectral images contain rich spectral information. However, the quality of hyperspectral images to be restored is poor due to problems such as noise and low resolution. The purpose of this embodiment is to improve the accuracy of hyperspectral image restoration, solve the problems of noise and spatial resolution reduction introduced by existing technologies in the hyperspectral imaging process, and stably complete the task of image quality restoration of degraded hyperspectral images under the common conditions of insufficient and unbalanced samples in hyperspectral images.

[0068] The single hyperspectral image to be restored is input into the hyperspectral image restoration network SSFSA for denoising / super-resolution reconstruction to obtain the restored hyperspectral image.

[0069] The hyperspectral image restoration network SSFSA framework provided in this embodiment combines spectral and spatial information and restores a single hyperspectral image through a specific joint encoding method, which can more effectively utilize the information redundancy and differences in hyperspectral images.

[0070] The method for denoising using the hyperspectral image restoration network SSFSA includes:

[0071] Step 1: Using a spectral overlap grouping and parameter sharing strategy, set the number of overlapping spectral channels, overlap and group the HSI of a single hyperspectral image along the spectral dimension, input each group of HSI into the multi-scale spatial-spectral fusion module MSSF, extract the mapping relationship in the multi-scale space, obtain all branch features, concatenate all branch features, and increase the dimensionality through a 1×1 convolutional layer to obtain the increased local multi-scale spatial-spectral features.

[0072] In this invention, all groups share the network parameters of each multi-scale spatial-spectral fusion module (MSSF), avoiding high computational costs and complex optimization processes. The structure of the MSSF is as follows: Figure 2 As shown.

[0073] This invention employs spectral grouping and parameter sharing strategies. Spectral grouping reduces the difficulty of feature extraction, while parameter sharing significantly reduces the number of model parameters. Setting the number of overlapping spectra ensures the continuity of information between adjacent spectral bands, preventing information loss. Therefore, this invention, through spectral grouping and parameter sharing, reduces computational costs while maintaining performance, thus improving the training efficiency of the hyperspectral image restoration network SSFSA.

[0074] This invention groups a single hyperspectral image along the spectral dimension and inputs it into the multi-scale spatial-spectral fusion module MSSF. By grouping along the spectral dimension, the hyperspectral image restoration network SSFSA can process different spectral bands independently. The multi-scale spatial-spectral fusion module MSSF can also extract the mapping relationship in multi-scale space and capture spatial features at different scales, providing strong support for subsequent feature fusion and denoising.

[0075] This invention extracts mapping relationships in multi-scale space to obtain all branch features, concatenates all branch features, and then uses a 1×1 convolutional layer to increase the dimensionality. The aim is to fuse the multi-scale spatial features extracted by the MSSF (Multi-Scale Spatial-Spectral Fusion) module. Furthermore, the concatenation operation stitches together features from different branches to form a more comprehensive feature representation. The 1×1 convolutional layer is used to increase the dimensionality of the fused features. Through concatenation and 1×1 convolutional layer dimensionality increase processing, the hyperspectral image restoration network SSFSA can generate local multi-scale spatial-spectral features. These multi-scale spatial-spectral features contain both spatial and spectral information, providing crucial information for subsequent denoising operations.

[0076] Step 2: Input the upgraded local multi-scale spatial-spectral features into the adaptive spatial feature aggregation group ASAG to obtain global spatial-spectral features.

[0077] This invention inputs the upscaled local multi-scale spatial-spectral features into the adaptive spatial feature aggregation group (ASAG) to obtain global spatial-spectral features. It adaptively aggregates local multi-scale spatial-spectral features to generate global spatial-spectral features. These global spatial-spectral features reflect the overall structure and spectral characteristics of the hyperspectral image. Through processing with the adaptive spatial feature aggregation group (ASAG), the hyperspectral image restoration network SSFSA can capture the global information of the hyperspectral image, providing a more comprehensive feature representation for subsequent denoising and reconstruction operations.

[0078] Step 3: After passing the single hyperspectral image through a 1×1 convolutional layer, it is fused with global spatial-spectral features, and then passed through another 1×1 convolutional layer to obtain the denoised hyperspectral image.

[0079] This invention fuses a single hyperspectral image with global spatial-spectral features after passing it through a 1×1 convolutional layer, and then passes it through another 1×1 convolutional layer to obtain a denoised hyperspectral image. The aim is to fuse the original hyperspectral image with global spatial-spectral features, and then perform final integration and processing through 1×1 convolutional layers to generate the final denoised result. By fusing the original image and global features, and combining the processing of 1×1 convolutional layers, this invention enables the Hyperspectral Image Restoration Network (SSFSA) to generate denoised hyperspectral images. These hyperspectral images effectively reduce noise interference while preserving the original spectral information.

[0080] Among them, the method of denoising by extending the hyperspectral image restoration network SSFSA, and the method of super-resolution reconstruction by the hyperspectral image restoration network SSFSA, include:

[0081] After obtaining the global spatial-spectral features, an upsampling operation is performed.

[0082] This invention upscales low-resolution hyperspectral images to high resolution through an upsampling operation. This is one of the key steps in super-resolution reconstruction tasks. Through the upsampling operation, the hyperspectral image restoration network SSFSA can generate hyperspectral images with higher spatial resolution, thereby meeting the demand for high-resolution images in practical applications.

[0083] Before passing the single hyperspectral image through a 1×1 convolutional layer, a bicubic upsampling operation is performed.

[0084] Bicubic upsampling uses information from the surrounding 16 pixels to estimate the value of the target pixel, which can generate smoother image edges and details. In the super-resolution reconstruction task of this invention, bicubic upsampling is used as a preprocessing step to provide a higher resolution input image for subsequent network processing.

[0085] Example 2

[0086] Based on the same inventive concept as Embodiment 1, this embodiment describes the specific implementation steps of a hyperspectral image restoration method, including:

[0087] Step 1: Obtain the single hyperspectral image (HSI) to be restored.

[0088] Step 2: Input the single hyperspectral image to be restored into the hyperspectral image restoration network SSFSA for denoising / super-resolution reconstruction to obtain the restored hyperspectral image;

[0089] In this embodiment, the restored hyperspectral image includes the denoised hyperspectral image / the hyperspectral image after super-resolution reconstruction.

[0090] In some embodiments, the training parameters of the hyperspectral image restoration network SSFSA are set as follows:

[0091] The number of spectra in each group was set to 8, and the number of overlapping spectra between adjacent groups was set to 2. The dilation rates of the dilated convolutions in the multi-scale spatial-spectral fusion module MSSF were set to 1, 3, and 5, respectively. The hyperspectral image restoration network SSFSA was implemented based on the PyTorch framework, using the Adam optimizer to optimize network parameters, with an initial learning rate set to 5×10⁻⁶. -5 After 30 iterations, the learning rate decays to 1 / 10 of its original value, the batch size is set to 16, and the number of training iterations is set to 100.

[0092] The method for denoising using the hyperspectral image restoration network SSFSA includes:

[0093] Step 2.1: Based on the spectral grouping and parameter sharing strategy, set the number of overlapping spectra.

[0094] In this embodiment, the number of overlapping spectra in the spectral grouping and parameter sharing strategy is set to... That is, to ensure that there is a relationship between adjacent groups after grouping. The same spectrum.

[0095] Step 2.2: Group the single hyperspectral image along the spectral dimension, input it into the multi-scale spatial-spectral fusion module MSSF, extract the mapping relationship in the multi-scale space, obtain all branch features, concatenate all branch features, and increase the dimensionality through a 1×1 convolutional layer to obtain the increased local multi-scale spatial-spectral features.

[0096] In this embodiment, an overlapping grouping method is used, based on the number of overlapping spectra. The input hyperspectral image to be restored Divided into G groups along the spectral dimension, denoted as , of which Each group is denoted as Hyperspectral image to be restored have spectral channels and Spatial resolution, Represents the height in the image spatial resolution. Represents the width in the image spatial resolution, the first grouped... Grouping of hyperspectral images have Spectral channels, of which This indicates rounding up. The hyperspectral images of each group are input into the branch network MSSF for encoding.

[0097] This embodiment uses a spectral grouping strategy to add each hyperspectral image group to the branch network as input, so as to effectively utilize the correlation between adjacent spectra. By sharing parameters in the branch network, the difficulty of parameter and feature extraction of the hyperspectral image restoration network SSFSA is reduced.

[0098] In this embodiment, a single hyperspectral image is grouped along the spectral dimension and input into the Multi-Scale Spatial-Spectral Fusion (MSSF) module. Utilizing the information redundancy and spectral differences between adjacent spectra, the mapping relationship in the multi-scale space is extracted to obtain all branch features. All branch features are concatenated and then upscaled using a 1×1 convolutional layer to obtain the upscaled local multi-scale spatial-spectral features, including:

[0099] Step 2.2.1: After grouping a single hyperspectral image along the spectral dimension, input the two half-channel branches and one full-channel branch of the Multi-Scale Spatial-Spectral Fusion Module (MSSF) respectively to obtain the features of the two half-channel branches and the features of the full-channel branch.

[0100] In this embodiment, the first branch output characteristics and the second branch output characteristics of the two half-channel branches HCB are respectively expressed as:

[0101] ;

[0102] In the formula, These represent the first branch output characteristics and the second branch output characteristics of the half-channel branch HCB, respectively. This represents a 3×3 convolution operation. This represents a 3×3 convolution with ReLU operation. Indicates the expansion coefficient is The dilated convolution plus ReLU operation, where the first branch input features and the second branch input features of the two half-channel branches HCB are respectively .

[0103] In this embodiment, the full-channel branch output characteristic is represented as:

[0104] ;

[0105] In the formula, This represents the output characteristics of the full-channel branch. This represents the first hyperspectral image grouped along the spectral dimension. Group input features, where the first group of features is... The number of spectral channels for the group input features is half-channel input features The number of spectral channels is Two half-channel input features Combined, these constitute the full-channel input features. .

[0106] Step 2.2.2: After concatenating the two half-channel branch features, the ReLU activation function is applied, and then the full-channel branch features are added pixel by pixel to obtain the fused multi-scale features;

[0107] In this embodiment, the half-channel branch output characteristic is represented as:

[0108] ;

[0109] In the formula, This indicates the output characteristics of a half-channel branch. Represents the ReLU activation function. This represents the first branch output feature of HCB, which combines two half-channel branches. Second branch output features .

[0110] The fused multi-scale features are represented as follows:

[0111]

[0112] In the formula, This represents the multi-scale feature resulting from the fusion of half-channel branch output features and full-channel branch output features.

[0113] Step 2.2.3: After passing the fused multi-scale features through the spatial residual module, multi-scale spatial features are obtained;

[0114] In this embodiment, the spatial residual module consists of two 3×3 convolution functions:

[0115] ;

[0116] In the formula, This represents the multi-scale spatial features after passing through the spatial residual module.

[0117] Step 2.2.4: Feed the multi-scale spatial features into the channel attention module CAB of the residual connection to obtain multi-scale spatial-spectral features. After adding the multi-scale spatial-spectral features to the multi-scale spatial features pixel by pixel, the branch features are obtained.

[0118] In this embodiment, the first The output features of each branch are represented as follows:

[0119] ;

[0120] In the formula, Indicates the first Each branch outputs features. This represents the spectral attention module CAB.

[0121] Step 2.2.5: Concatenate all branch features and increase the dimensionality through a 1×1 convolutional layer to obtain the increased local multi-scale spatial-spectral features.

[0122] In this embodiment, the local multi-scale spatial-spectral features are represented as follows:

[0123] ;

[0124] In the formula, This represents the local multi-scale spatial-spectral features after dimensionality enhancement. This indicates the number of spectral channels after dimensionality enhancement; in this embodiment, it is set to 240. This represents a 1×1 convolution operation. Indicates a cascading operation. This represents the local multi-scale spatial-spectral features obtained by cascading all branch features. This represents the feature of the first branch. This represents the second branch feature. Indicates the first Branch features, Indicates the first When concatenating the output features of G branch MSSF, due to the overlap between adjacent groups, it is necessary to average the feature values ​​within the overlapping bands based on their original spectral band positions to generate a spectral channel number of... The comprehensive feature map.

[0125] In this embodiment, after grouping a single hyperspectral image along the spectral dimension, the data are input into two half-channel branches and one full-channel branch of the Multi-Scale Spatial-Spectral Fusion Module (MSSF) to obtain two half-channel branch features and one full-channel branch feature, including:

[0126] After grouping a single hyperspectral image along the spectral dimension, it is input into the two half-channel branches of the multi-scale spatial-spectral fusion module MSSF. After two 3×3 convolution operations, each 3×3 convolution operation is processed by the ReLU activation function, followed by another 3×3 convolution operation, and finally processed by a layer of dilated convolution to obtain the features of the two half-channel branches.

[0127] After grouping a single hyperspectral image along the spectral dimension, it is input into the full-channel branch of the multi-scale spatial-spectral fusion module MSSF. After processing through three cascaded 3×3 convolutions with ReLU activation functions, and then through one layer of dilated convolution, the full-channel branch features are obtained.

[0128] In this embodiment, the step of feeding multi-scale spatial features into the channel attention module (CAB) of the residual connection to obtain multi-scale spatial-spectral features includes:

[0129] The first channel attention feature is obtained by performing two 3×3 convolution operations on the multi-scale spatial features. The GELU activation function is used after the first 3×3 convolution operation.

[0130] The first channel attention feature is passed through a global average pooling layer and then through two 3×3 convolution operations to obtain the second channel attention feature. The ReLU activation function is used after the first 3×3 convolution operation.

[0131] The second channel attention feature is processed by the Sigmoid function and then multiplied pixel-by-pixel with the first channel attention feature to obtain the multi-scale spatial-spectral feature.

[0132] Step 2.3: Input the upgraded local multi-scale spatial-spectral features into the adaptive spatial feature aggregation group ASAG to obtain global spatial-spectral features.

[0133] In some embodiments, the adaptive spatial feature aggregation group (ASAG) includes A cascaded adaptive spatial aggregation module SAB, the A series of cascaded adaptive spatial aggregation modules form residuals through long and short skip connections, where... The structure of the adaptive spatial aggregation module SAB is as follows: Figure 3 As shown.

[0134] In some embodiments, the upscaled local multi-scale spatial-spectral features are input into the adaptive spatial feature aggregation group ASAG to obtain global spatial-spectral features. Each cascaded adaptive spatial aggregation module SAB in the adaptive spatial feature aggregation group ASAG includes:

[0135] For each adaptive spatial aggregation module (SAB), after layer normalization, it goes through a parallel three-branch structure including a channel attention module (CAB), a 3×3 depthwise convolution (DWConv), and a learnable convolutional kernel attention (LCK) to obtain the output features of the three branches.

[0136] The output features of the three branches are aggregated using pixel-wise multiplication to perform spatial and spectral feature aggregation. Then, the aggregated features are passed through a 1×1 convolutional layer and residual connections to obtain the global deep features output by the adaptive spatial aggregation module SAB.

[0137] Specifically, the learnable convolutional kernel attention (LCK) is used to adaptively process the spatial features of the local multi-scale spatial-spectral features after layer normalization; the 3×3 depthwise convolution (DWConv) is used to capture the spatially invariant features of the local multi-scale spatial-spectral features after layer normalization; and the channel attention module (CAB) is used to further extract and refine the spectral features of the local multi-scale spatial-spectral features after layer normalization.

[0138] In this embodiment, the layer normalization LN operation is represented as:

[0139] ;

[0140] In the formula, Indicates layer normalization LN operation, Indicates the first The input feature map of each spatial aggregation module Indicates the first The output features of a spatial aggregation module after passing through an LN layer.

[0141] After the layer-normalized LN operation, a parallel three-branch structure is used, with each branch employing a different operator, including a 3×3 depthwise convolution (DWConv), a channel attention module (CAB), and a learnable convolutional kernel attention (LCK) operation, represented as follows:

[0142] ;

[0143] In the formula, This represents a depthwise convolution operation. This represents the channel attention module CAB. This represents the convolution kernel attention module.

[0144] In some embodiments, the channel attention module (CAB) includes two standard convolutional layers, a GELU activation function, and a spectral attention mechanism, wherein the spectral attention mechanism includes a global average pooling layer, a dimensionality reduction layer, a dimensionality increase layer, and a sigmoid function.

[0145] In this embodiment, the first The output feature map formed by the global deep features output by each SAB module is represented as follows:

[0146] ;

[0147] in, It is the first The output feature maps of each SAB module This represents pixel-by-pixel multiplication.

[0148] In this embodiment, the global deep features are represented as:

[0149] ;

[0150] In the formula, Indicates the first A global deep feature, Indicates the first One SAB module, Indicates the first A global deep feature, Indicates the first One SAB module, This shows the first SAB module. It represents local multi-scale spatial-spectral characteristics.

[0151] The global spatial-spectral features are represented as follows:

[0152] ;

[0153] In the formula, Represents global spatial-spectral characteristics, "" indicates pixel-by-pixel addition.

[0154] In some embodiments, the output features of the three branches include LCK output features, wherein the method for obtaining the LCK output features includes:

[0155] We learn a set of convolutional kernels shared across all spatial locations and all images by utilizing the local multi-scale spatial-spectral features after layer normalization;

[0156] A lightweight convolutional branch is used to predict each spatial location of the local multi-scale spatial-spectral features after the prediction layer is normalized, to obtain a fusion coefficient map. The lightweight convolutional branch contains two 3×3 grouped convolutional layers and a SimpleGate activation function.

[0157] The fusion coefficient feature map and the convolution kernel are linearly fused, and the fusion weight of each spatial location of the normalized local multi-scale spatial-spectral features is calculated through the linear combination of the convolution kernels.

[0158] In this embodiment, the spatial location is located The fusion weight is expressed as:

[0159] ;

[0160] In the formula, Indicates spatial location The fusion weight, Indicates the first The fusion coefficients of a learnable convolutional kernel. Indicates the first A learnable convolutional kernel.

[0161] The local multi-scale spatial-spectral features after layer normalization are transformed by 1×1 convolution to obtain the enhanced feature map;

[0162] The enhanced feature map and the fusion weights of the layer-normalized local multi-scale spatial-spectral features at each spatial location are subjected to group convolution operation. The fusion feature map at each spatial location is obtained through group convolution calculation and is used as the LCK output feature.

[0163] In this embodiment, the LCK output feature map is represented as follows:

[0164] ;

[0165] In the formula, Indicates spatial location LCK output characteristics at the location, This indicates group convolution calculation. Indicates spatial location Enhanced feature map at the location.

[0166] In this embodiment, the enhanced feature map is represented as:

[0167] ;

[0168] In the formula, This represents the enhanced feature map.

[0169] Step 2.4: After passing the single hyperspectral image through a 1×1 convolutional layer, it is fused with global spatial-spectral features, and then passed through another 1×1 convolutional layer to obtain the denoised hyperspectral image.

[0170] Among them, the method of denoising by extending the hyperspectral image restoration network SSFSA, and the method of super-resolution reconstruction by the hyperspectral image restoration network SSFSA, include:

[0171] After obtaining the global spatial-spectral features, an upsampling operation is performed, which is represented as:

[0172] ;

[0173] In the formula, This represents the feature map after upsampling of global spatial-spectral features. This represents the upsampling function performed by the PixelShuffle operator.

[0174] Before passing the single hyperspectral image through a 1×1 convolutional layer, a bicubic upsampling operation is performed.

[0175] In this embodiment, the super-resolution restored hyperspectral image is represented as:

[0176] ;

[0177] In the formula, This refers to the restored hyperspectral image, i.e., the hyperspectral image after super-resolution reconstruction. This represents the restored features obtained by upsampling a single hyperspectral image and passing it through a 1×1 convolutional layer, then fusing it with the upsampled global spatial-spectral features. This indicates a single hyperspectral image. Through bicubic upsampling operation.

[0178] Upsampling is only required for hyperspectral super-resolution reconstruction tasks. For hyperspectral denoising tasks, upsampling is not necessary. In this case, the restored hyperspectral image is represented as:

[0179] ;

[0180] In the formula, This indicates the restored hyperspectral image, i.e., the hyperspectral image after denoising.

[0181] In some embodiments, a weighted average method is used to... The loss function is obtained by weighting the loss function and the spatial-spectral total variation (SSTV) loss function. This loss function measures the difference between the restored hyperspectral image and the sharp ground truth image of the single hyperspectral image. In this embodiment, the loss function is expressed as:

[0182] ;

[0183] In the formula, Indicates the total loss. This represents the parameter set of the hyperspectral image restoration network SSFSA. express loss function The spatial-spectral total variation (SSTV) loss function is represented. These represent the weighting coefficients used to balance the two losses, where, .

[0184] Among them, the The loss function is expressed as:

[0185] ;

[0186] In the formula, express loss, The number of images in a training batch. and They represent the first The sharp ground truth image and the corresponding restored hyperspectral image of each single hyperspectral image. The spatial-spectral total variation (SSTV) loss function is expressed as:

[0187] ;

[0188] In the formula, This represents the total spatial-spectral variation (SSTV) loss. , and They represent calculations respectively. Functions of horizontal, vertical, and spectral gradients.

[0189] Example 3

[0190] Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium storing computer instructions thereon, characterized in that the computer instructions, when executed by a processor, implement the steps of the methods of Embodiment 1 or 2 described above.

[0191] Example 4

[0192] Based on the same inventive concept as other embodiments, the present invention also provides a computer program product, including computer instructions, characterized in that, when the computer instructions are executed by a processor, they implement the steps of the methods described in Embodiment 1 or 2 above.

[0193] In summary, this invention, by employing spectral grouping and parameter sharing strategies, fully leverages the strong correlation between adjacent bands, effectively improving computational efficiency and feature extraction accuracy during the restoration process. Simultaneously, by combining a multi-scale spatial-spectral fusion module (MSSF), it captures spatial and spectral information of the image at different scales, achieving richer and more refined image restoration results. Furthermore, it applies an adaptive spatial aggregation group (ASAG), which adaptively optimizes the spatial information aggregation process using learnable convolutional kernels and automatically adjusts the aggregation strategy based on image content, thereby significantly restoring image details and structure. In conclusion, the hyperspectral image restoration method provided by this invention can significantly improve restoration accuracy, effectively addressing the noise and spatial resolution degradation problems introduced during hyperspectral imaging, and achieving stable image quality restoration even under conditions of insufficient and imbalanced samples.

[0194] This invention designs a multi-scale spatial-spectral fusion module (MSSF). First, it utilizes a parallel branch-group convolutional structure, including half-channel and full-channel branches, to effectively fuse spatial and spectral information at different scales. This allows for the parallel enhancement of internal and external relationships between different spectral bands, facilitating the capture of smooth image features related to low-frequency information. The half-channel branch reduces computation while maintaining feature richness, while the full-channel branch further extracts global features. Second, each branch uses a layer of dilated convolution to increase the receptive field without increasing network parameters, extracting multi-scale spatial information from different receptive fields. Finally, a channel attention module with residual connections is added, leveraging spectral similarity and complementarity to mine the correlation of spectral features. This not only improves computational efficiency but also significantly enhances the precision and accuracy of image restoration, particularly when processing complex hyperspectral images.

[0195] The adaptive spatial feature aggregation group (ASAG) provided by this invention achieves dynamic adjustment and adaptive aggregation of local multi-scale spatial-spectral features through cascaded adaptive spatial aggregation modules (SAB) and residuals formed by long and short skip connections. Specifically, the three-branch structure within the adaptive spatial aggregation module (SAB), including a channel attention module (CAB), a 3×3 depthwise convolution (DWConv), and a learnable convolutional kernel attention module (LCK), can precisely extract and fuse spatial and spectral features, further improving the detail recovery capability and structural preservation of image restoration. Furthermore, the learnable convolutional kernel attention module (LCK) achieves adaptive selection and fusion of learnable convolutional kernels at each location by predicting fusion coefficient maps, thereby enhancing the expressive power of global spatial-spectral features.

[0196] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0199] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0200] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A hyperspectral image restoration method, characterized in that, include: Acquire the single hyperspectral image (HSI) to be restored; The single hyperspectral image HSI to be restored is input into the hyperspectral image restoration network SSFSA for denoising or super-resolution reconstruction to obtain the restored hyperspectral image. The restored hyperspectral image includes the denoised hyperspectral image / the hyperspectral image after super-resolution reconstruction; The method for denoising using the hyperspectral image restoration network SSFSA includes: Using a spectral overlap grouping and parameter sharing strategy, the number of overlapping spectral channels is set. After grouping a single hyperspectral image along the spectral dimension, the two half-channel branches and one full-channel branch of the multi-scale spatial-spectral fusion module MSSF are respectively input to obtain two half-channel branch features and one full-channel branch feature. The number of spectral channels for each group is divided into two, which are then input into the two half-channel branches of the Multiscale Spatial-Spectral Fusion Module (MSSF). All spectral channels of each group are input to the full-channel branch of the Multiscale Spatial-Spectral Fusion Module (MSSF); After concatenating the two half-channel branch features, the ReLU activation function is applied, and then the full-channel branch features are added pixel by pixel to obtain the fused multi-scale features. The fused multi-scale features are then passed through a spatial residual module to obtain multi-scale spatial features. The multi-scale spatial features are fed into the channel attention module CAB of the residual connection to obtain multi-scale spatial-spectral features; After pixel-by-pixel addition of multi-scale spatial-spectral features and multi-scale spatial features, branch features are obtained; All branch features are concatenated and then increased in dimension through a 1×1 convolutional layer to obtain the increased local multi-scale spatial-spectral features; In this context, all groups share the network parameters of each multi-scale spatial spectrum fusion module (MSSF); The upgraded local multi-scale spatial-spectral features are input into the adaptive spatial feature aggregation group ASAG to obtain global spatial-spectral features. The single hyperspectral image is passed through a 1×1 convolutional layer, fused with global spatial-spectral features, and then passed through another 1×1 convolutional layer to obtain a denoised hyperspectral image. Among them, the method for super-resolution reconstruction using the hyperspectral image restoration network SSFSA includes: After obtaining the global spatial-spectral features, an upsampling operation is performed; After performing bicubic upsampling on a single hyperspectral image, it is fused with the global spatial-spectral features after the upsampling operation through a 1×1 convolutional layer, and then passed through another 1×1 convolutional layer to obtain the super-resolution reconstructed hyperspectral image.

2. The hyperspectral image restoration method according to claim 1, characterized in that, The local multi-scale spatial-spectral features are represented as follows: ； In the formula, This represents the local multi-scale spatial-spectral features after dimensionality enhancement. This indicates the number of spectral channels after dimensionality upscaling. Represents the height in the image spatial resolution. Represents the width in the image spatial resolution. This represents a 1×1 convolution operation. Indicates a cascading operation. This represents the local multi-scale spatial-spectral features obtained by cascading all branch features. This represents the feature of the first branch. This represents the second branch feature. Indicates the first Branch features, Indicates the first Branch features, This indicates the number of spectral channels in the single hyperspectral image (HSI) to be restored.

3. The hyperspectral image restoration method according to claim 1, characterized in that, After grouping the single hyperspectral image along the spectral dimension, it is input into two half-channel branches and one full-channel branch of the multi-scale spatial-spectral fusion module MSSF, respectively, to obtain two half-channel branch features and one full-channel branch feature, including: After two 3×3 convolution operations, each 3×3 convolution operation is processed by the ReLU activation function, followed by another 3×3 convolution operation, and finally a dilated convolution layer is applied to obtain two half-channel branch features. After processing with three cascaded 3×3 convolutions and ReLU activation functions, and then through a layer of dilated convolution, full-channel branch features are obtained.

4. The hyperspectral image restoration method according to claim 1, characterized in that, The multi-scale spatial features are represented as follows: ； In the formula, Represents multi-scale spatial features. This represents the multi-scale feature resulting from the fusion of half-channel branch output features and full-channel branch output features. This represents the first grouping of a single hyperspectral image along its spectral dimension. Groups, This represents a 3×3 convolution operation.

5. The hyperspectral image restoration method according to claim 1, characterized in that, The process of feeding multi-scale spatial features into the channel attention module (CAB) of the residual connection to obtain multi-scale spatial-spectral features includes: The first channel attention feature is obtained by performing two 3×3 convolution operations on the multi-scale spatial features. The GELU activation function is used after the first 3×3 convolution operation. The first channel attention feature is passed through a global average pooling layer and then through two 3×3 convolution operations to obtain the second channel attention feature. The ReLU activation function is used after the first 3×3 convolution operation. The second channel attention feature is processed by the Sigmoid function and then multiplied pixel-by-pixel with the first channel attention feature to obtain the multi-scale spatial-spectral feature.

6. The hyperspectral image restoration method according to claim 1, characterized in that, The adaptive spatial feature aggregation group ASAG includes A cascaded adaptive spatial aggregation module SAB, the A series of cascaded adaptive spatial aggregation modules form residuals through long and short skip connections, where... .

7. The hyperspectral image restoration method according to claim 6, characterized in that, The upgraded local multi-scale spatial-spectral features are input into the adaptive spatial feature aggregation group ASAG to obtain global spatial-spectral features. Each cascaded adaptive spatial aggregation module SAB in the adaptive spatial feature aggregation group ASAG includes: For each adaptive spatial aggregation module (SAB), after layer normalization, it goes through a parallel three-branch structure including a channel attention module (CAB), a 3×3 depthwise convolution (DWConv), and a learnable convolutional kernel attention (LCK) to obtain the output features of the three branches. The output features of the three branches are aggregated using pixel-wise multiplication to perform spatial and spectral feature aggregation. Then, the features are passed through a 1×1 convolutional layer and residual connections to obtain the global deep features output by the adaptive spatial aggregation module SAB. Specifically, the learnable convolutional kernel attention (LCK) is used to adaptively process the spatial features of the local multi-scale spatial-spectral features after layer normalization; the 3×3 depthwise convolution (DWConv) is used to capture the spatially invariant features of the local multi-scale spatial-spectral features after layer normalization; and the channel attention module (CAB) is used to further extract and refine the spectral features of the local multi-scale spatial-spectral features after layer normalization.

8. The hyperspectral image restoration method according to claim 1 or 7, characterized in that, The upgraded local multi-scale spatial-spectral features are processed through N cascaded SABs to obtain global deep features, which are represented as follows: ； In the formula, Indicates the first A global deep feature, Indicates the first One SAB module, Indicates the first A global deep feature, Indicates the first One SAB module, This indicates the first SAB module. This represents the local multi-scale spatial-spectral features after dimensionality increase; The global spatial-spectral features are represented as follows: ； In the formula, Represents global spatial-spectral characteristics, "" indicates pixel-by-pixel addition.

9. The hyperspectral image restoration method according to claim 7, characterized in that, The output features of the three branches include LCK output features, wherein the methods for obtaining LCK output features include: We learn a set of convolutional kernels shared across all spatial locations and all images by utilizing the local multi-scale spatial-spectral features after layer normalization; A lightweight convolutional branch is used to predict each spatial location of the local multi-scale spatial-spectral features after the prediction layer is normalized, to obtain a fusion coefficient map. The lightweight convolutional branch contains two 3×3 grouped convolutional layers and a SimpleGate activation function. The fusion coefficient feature map and the convolution kernel are linearly fused, and the fusion weight of each spatial location of the normalized local multi-scale spatial-spectral features is calculated by linear combination of the convolution kernels. The local multi-scale spatial-spectral features after layer normalization are transformed by 1×1 convolution to obtain the enhanced feature map; The enhanced feature map and the fusion weights of the layer-normalized local multi-scale spatial-spectral features at each spatial location are subjected to group convolution operation. Through group convolution calculation, the LCK output feature at each spatial location is obtained.

Images