A hyperspectral image super-resolution reconstruction method based on recursive dilated self-attention

Abstract

The present invention discloses a hyperspectral image super-resolution reconstruction method based on recursive hole self-attention, comprising the following steps: obtaining an original low-resolution image, preprocessing the low-resolution image, and obtaining a grouped feature image; based on a local feature extraction module, performing a channel-level local feature extraction on the grouped feature image; inputting the channel-level local features into a global feature extraction module to extract the global features of the image; performing upsampling and convolution operations on the global features of the image to obtain a hyperspectral super-resolution image. The present invention uses a recursive hole self-attention mechanism for a single hyperspectral image as input, which can simultaneously feed back local features and global features, enhance cross-dimensional information interaction, thereby improving the ability to obtain image edge textures, and can improve the quality of hyperspectral image super-resolution reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of hyperspectral image processing, and in particular relates to a hyperspectral image super-resolution reconstruction method based on recursive hole self-attention. Background Art

[0002] Hyperspectral images, unlike the visible light visible to the human eye, capture all cross-spectral information. Compared to traditional multispectral images, hyperspectral images contain reflectance information from several continuous bands of a scene. Therefore, compared to traditional RGB images and multispectral images (MSIs), they possess richer spectral information and spectral bands, enabling them to capture intrinsic properties and differences that are difficult to identify with multispectral images. However, the spatial resolution of hyperspectral images acquired by existing technologies is low, making their development and application more difficult. Therefore, improving the resolution of hyperspectral images has become a critical task.

[0003] Hyperspectral image super-resolution reconstruction involves reconstructing a high-resolution image corresponding to a scene from one or more consecutively observed low-resolution images. This approach facilitates the practical application of hyperspectral imaging. Depending on the number of input images, methods for improving the spatial resolution of input images can be categorized as either fusion-based hyperspectral reconstruction or single-hyperspectral image-based reconstruction. While fusion-based hyperspectral imaging is widely popular, obtaining the input low-resolution and auxiliary images is challenging. Summary of the Invention

[0004] In response to the problem of lack of cross-channel information interaction in the existing technology, the present invention proposes a hyperspectral image super-resolution reconstruction method based on recursive void self-attention. Taking a single hyperspectral image as input, the recursive void self-attention mechanism is referenced, which can simultaneously feed back local features and global features, enhance cross-dimensional information interaction, thereby improving the ability to obtain image edge texture and the quality of hyperspectral image super-resolution reconstruction.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A hyperspectral image super-resolution reconstruction method based on recursive dilated self-attention includes the following steps:

[0007] Acquire an original low-resolution image, preprocess the low-resolution image, and obtain a grouped feature image;

[0008] Based on the local feature extraction module, performing a channel-level local feature extraction on the grouped feature image;

[0009] Inputting the local features at the channel level into a global feature extraction module to extract global features of the image;

[0010] Upsampling and convolution operations are performed on the global features of the image to obtain a hyperspectral super-resolution image.

[0011] Preferably, the local feature extraction module is a MSAM-multi-scale spectral attention module, which is composed of a deformable convolution module DCONV and a residual mean-convolution extraction block AC;

[0012] The deformable convolution module DCONV is used to extract spatial information of different scales by using three dilated convolutions with different dilation rates;

[0013] The residual mean-convolution extraction block AC is used to learn the importance of each channel based on the spatial information of different scales, extract the features of the key channels that meet the preset requirements, and obtain information between channels.

[0014] Preferably, the method for learning the importance of each channel based on the spatial information of different scales includes:

[0015] Compress the channel information through the average pooling operation;

[0016] The weights of the compressed channel information are learned through several fully connected layers to obtain the weight information between channels;

[0017] Based on the weight information between the channels, the channels of the image are weighted and recalibrated to extract the features of the key channels that meet the preset requirements and obtain the information between the channels.

[0018] Preferably, the residual mean-convolution extraction block AC consists of global pooling, convolution and Sigmoid activation functions and residual connections.

[0019] Preferably, the global feature extraction module is composed of a LiteViT model, which uses a recursive void attention mechanism RASA and a void attention mechanism ASA.

[0020] Preferably, the method of inputting the local features at one channel level into a global feature extraction module to extract the global features of the image includes:

[0021] Firstly perform a layer normalization operation on the local features at the channel level;

[0022] Perform a transposed convolution on the normalized result;

[0023] The convolution result is fed into LiteViT’s attention mechanism to extract the global features of the image.

[0024] Preferably, the convolution result is fed into the attention mechanism of LiteViT to extract the global features of the image. The method includes:

[0025] The convolution result is fed into the atrous attention mechanism ASA, and multi-scale queries are calculated through three depthwise convolutions after linear projection.

[0026] Add the calculated result to the weight calculated by the Sigmoid function to capture multi-scale information;

[0027] The captured multi-scale information is fed into the query (Query, Q), key (Key, K), and value (Value, V) to perform a CSA attention mechanism operation;

[0028] The result after the CSA attention mechanism is fed into a multi-layer perceptron (MLP) after a Norm operation, and the output is downsampled. The downsampled output is then fed into the recursive atrous self-attention mechanism (RASA) to obtain the global features of the image.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The hyperspectral image super-resolution reconstruction method proposed in the present invention has broad prospects and can be used to improve low-resolution hyperspectral images. The spatial resolution of hyperspectral images can be improved through hyperspectral image reconstruction technology. In agriculture, hyperspectral images can help people prevent and detect pests and diseases. In environmental surveys, hyperspectral images can help experts evaluate the properties of the soil, vegetation coverage, etc. by analyzing the spectral characteristics of the surface. In the military field, hyperspectral images are also often used for target identification, etc. In short, hyperspectral image super-resolution reconstruction technology helps people gain more information and knowledge in related fields by improving the spatial resolution of images, and has high practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 It is an overall framework diagram of an embodiment of the present invention;

[0033] Figure 2 Schematic diagram of a local feature extraction module MSAM according to an embodiment of the present invention;

[0034] Figure 3Schematic diagram of a global feature extraction module according to an embodiment of the present invention;

[0035] Figure 4 The present invention is a flowchart of a hyperspectral image super-resolution reconstruction method based on recursive hole self-attention. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Example 1

[0039] like Figure 4 As shown, the present invention provides a hyperspectral image super-resolution reconstruction method based on recursive hole self-attention, comprising the following steps:

[0040] Obtain the original low-resolution image, preprocess the low-resolution image, and obtain the grouped feature image;

[0041] Based on the local feature extraction module, a channel-level local feature extraction is performed on the grouped feature images;

[0042] Input a channel-level local feature into the global feature extraction module to extract the global features of the image;

[0043] The global features of the image are upsampled and convolved to obtain a high-spectral super-resolution image.

[0044] In this embodiment, the present invention mainly proposes a method for super-resolution reconstruction of hyperspectral images that combines CNN with Transformer architecture. It consists of two modules, one is the local feature extraction module MSAM-multi-scale spectral attention module, which mainly performs local feature extraction at a channel level (a channel level refers to this part, the model mainly extracts the information of the feature map in each channel. It can be understood as operating and analyzing channels with higher weights) on the blocked feature image. The other is composed of a global feature extraction module LiteViT. It mainly uses a recursive void attention mechanism, focusing on the global information of the image. The local feature extraction module is mainly composed of several deformable convolution modules, and then its output result will be sent to a global feature extraction module mainly composed of Transformer. Finally, it is output after upsampling and convolution operations. The processed feature map integrates local features and global features that interact across space. The entire framework is as follows Figure 1 As shown, where B refers to the number of spectral bands.

[0045] In this embodiment, the local feature extraction module is the MSAM-Multi-Scale Spectral Attention Module. It mainly consists of a deformable convolution (Dilated Convolutiona, DCONV) and a residual mean-convolution extraction (Avgpooling Convolutiona, AC) block. Three dilated convolutions with different dilation rates are used to extract spatial information at different scales. Compared with traditional convolution, dilated convolution obtains different receptive fields by using different dilation rates, so that the feature information of the image will not be lost due to the pooling operation. This allows the receptive field to focus on different areas at different times and to explore the spatial similarity between different scales. The AC module allows the network to automatically learn the importance of each channel. First, the channel information is compressed through the average pooling operation, and then its weight is learned through several fully connected layers. The learned weight information between channels is used to perform weight analysis on the image channels and perform feature recalibration, that is, focusing on channels with higher weights, thereby extracting the features of the key channels to obtain information between channels. Among them, when performing deformable convolution and two convolution operations (the two convolution layers in the residual mean convolution module, which are used to further extract image features), the parameter sharing strategy is used to reduce the number of parameters and reduce the complexity of the model. The AC module is composed of global pooling, convolution and Sigmoid activation functions and residual connections. The role of the convolution layer is to perform dimensionality reduction operations on the channel dimension and dimensionality enhancement on the channel dimension respectively. The output result of the previous layer is used as input to the AC block to explore the correlation across the channel space. By introducing the multi-scale spectral attention module, it is possible to focus on important channels while keeping the existing number of parameters unchanged, thereby improving the extraction of local features. The framework of the local feature extraction module is as follows: Figure 2 shown.

[0046] In this embodiment, the global feature extraction module is composed of Lite Vision Transformer (LiteViT), which focuses on the global information of the image. LiteViT is a model that uses Transformer for visual tasks. It uses Recursive Atrous Self-Attention (RASA) and Atrous Self-Attention (ASA). In ASA, xσ(x) represents different dilation rate functions. Multi-scale queries are calculated by three depth convolutions after linear projection. These convolutions share kernel weights but have different dilation rates: 1, 3, and 5. Their outputs are added to the weights calculated by the Sigmoid function to achieve self-calibration. This can be achieved using the SiLU function. Multi-scale information is used to calculate the similarity graph, which weights the sum of the values. It is used to capture multi-scale information. At the same time, the convolutional self-attention mechanism (CSA) is used. The convolutional attention mechanism is a method to improve model performance by focusing on different parts of the input data. It combines the advantages of convolutional networks and Transformers to achieve better generalization capabilities and larger model capacity. Therefore, by effectively combining the two, the convolutional attention mechanism can help us achieve a better balance between accuracy and efficiency. Its attention mechanism obtains the mutual information between the global image through the query (Query, Q) of the key (Key, K) value (Value, V) pair, thereby realizing the extraction of global features of the image. The global feature extraction module framework is shown in the figure below. Figure 3 shown.

[0047] Example 2

[0048] The present invention is a hyperspectral image super-resolution reconstruction method based on recursive hole self-attention mechanism, and its overall network framework is as follows Figure 1 The specific implementation steps of this method are as follows:

[0049] 1. Preprocess the original low-resolution image and divide it into G groups, each containing several overlapping spectra. The size of the image is H*W*B / G, where H is the height of the image, W is the width of the image, and B is the number of spectral bands (channels).

[0050] 2. The processed image X i as input.

[0051] The first stage: First, each X iAfter a convolution with expansion rates of 1, 3, and 5, the output is used as the input of the PReLU activation function to obtain the activated result. The outputs of the convolution with different expansion rates are added together to obtain a new feature map that integrates different features, and the results are residually connected. Finally, the output result Y of size H*W*B / G is obtained. i . All Y i At the same time, in order to simultaneously utilize the similarity and complementarity of the spectrum, the residual attention mechanism is used to further extract its features. In the residual attention mechanism, the input Y i Batch normalization (Batch_Norm) is performed, and then the Batch_Norm image is sent to the convolution layer, and the number of channels is reduced according to a certain dimensionality reduction ratio. The residual AC block is used to extract features between different spectral bands and obtain the output result Z. This is shown in formula (1).

[0052]

[0053] The second stage: The output result Z of the upper layer is used as the input of the global feature extraction module. After passing through a convolutional neural module, it is sent to the global feature extraction module. First, Z sent to the global feature extraction module needs to undergo a layer normalization (LayerNorm) operation, and then a transposed convolution is performed on the normalized result. The convolutional result is then sent to the attention mechanism of LiteViT. After being sent in, a CSA attention mechanism (Convolutional Self-Attention) operation is performed on it as a query (Query, Q), key (Key, K), and value (Value, V). Before attention is performed, a Norm operation is first performed on it. By using the CSA attention mechanism, the network can simultaneously focus on multiple positions in the sequence and learn information from different subspaces in the sequence. The principle is shown in formula (2). By using the CSA attention mechanism, its computational efficiency is improved. The result of the CSA attention mechanism, after undergoing a Norm, is fed into a multi-layer perceptron (MLP). The output is then downsampled and fed into a recursive atrous self-attention module, which consists of two weight-sharing components. This parameter sharing approach results in a more lightweight model with significantly fewer parameters than other models. The RASA attention mechanism first applies RASA to the input data, then performs SiLu activation on the result. Based on the result, a corresponding residual connection is then made. The result of the residual connection is then again RASA-activated and activated. The result is then fed back into the RASA module, repeating this cycle several times. Finally, the output is denoted as a new X.

[0054]

[0055] Where, d k It refers to the dimension of the key (K) vector. V represents the value (Value) vector.

[0056] The third stage: First, the output X from the second stage is upsampled. This upsampling uses a transposed convolution. Transposed convolution expands the input feature map by inserting a specified number of zeros between each pixel, then pads the expanded feature map with zeros. After padding, the same convolution kernel as in traditional convolution is used to calculate the weighted sum, and the result is output. Using transposed convolution increases the spatial size of the input feature map. The resulting output is of size C × sH × sW. The original input image is then upsampled using bicubic interpolation. Bicubic interpolation calculates the value of the target pixel by applying a cubic polynomial linear interpolation to the 16 pixels near the target pixel, resulting in a smoother upsampled image. The original low-resolution image is upsampled using bicubic interpolation to produce an output of size B × sH × sW. This is then subjected to a 1 × 1 convolution, resulting in a C × sH × sW output. Finally, it is fused with the up-sampled result after transposed convolution. Then, a 1×1 convolution is performed on the fused result to obtain a high-spectral super-resolution image with the same dimensions B×H×W as the original image.

[0057] During the model training implementation phase, 3465 groups of overlapping images were constructed as training samples. For the loss function, the improved L1 loss and Charbonnier loss are used, which not only have good robustness to outliers in the data, but also can well produce sparse solutions and have non-differentiability. Compared with the square error loss function (MSE, L2), the Charbonnier loss function performs well in the task of super-resolution reconstruction of hyperspectral images, and its error distribution is more uniform and has better convergence effect; Charbonnier has excellent performance in natural images, but unlike natural images, hyperspectral images contain a large amount of spectral distortion, while Charbonnier ignores these characteristics. Therefore, the present invention introduces a spatial angle similarity maximization (Spectral Angle Mapper, SAM) loss, which is a method for processing data based on spectral angles to maintain its spectral consistency and spatial details. At the same time, in order to improve local clarity, a gradient loss (Gradient Loss) is also introduced.

[0058] L CharbonnierIt is a smooth L1 loss function commonly used in computer vision and image processing, especially in the fields of optical flow, stereo vision and image denoising. It is a variant of L1 loss (absolute error loss) and can be regarded as a compromise between squared error and absolute error, providing better numerical stability and optimization properties. Charbonnier , is a commonly used regression loss function used to improve the performance of the model. Compared with L2, it can better produce clear edges. Charbonnier The loss function is shown in formula (3). Where x is the difference between the predicted value and the true value. When ||x|| is large, the Charbonnier loss behaves like the L1 loss, making it less sensitive to large errors, thereby improving robustness. When ||x|| is small, its behavior is closer to the L2 loss, which can accurately adjust small errors and improve the precision of the model. ε is a small constant that ensures that the loss function is differentiable everywhere, so that the loss function still maintains good behavior near zero. L Charbonnier Compared with L1, the curve of this loss function is smoother. The gradient of the value close to zero will not be too small due to the existence of constant ε, thus avoiding gradient vanishing. The value far from zero will not cause gradient explosion due to too large gradient.

[0059]

[0060] SAM loss is commonly used in the analysis and processing of remote sensing data and is a relatively new deep learning loss function. It is particularly suitable for similarity learning in high-dimensional feature spaces, emphasizing the angular similarity between feature vectors rather than the traditional Euclidean distance or dot product similarity. It mainly compares the difference between the provided pixel spectrum and the reference pixel spectrum. By focusing on angle rather than distance, SAM loss is insensitive to the scale and size of the features, which helps to improve the model's robustness to input changes. At the same time, SAM loss helps to learn more discriminative feature representations by maximizing the angular difference of intra-class similarity and minimizing the angular difference of inter-class similarity. By optimizing angular similarity, it enables the model to better recognize and distinguish subtle differences in visual content, and can help optimize the model to better preserve the spectral characteristics of the target image. The SAM loss is shown in formula (4).

[0061]

[0062] Where, L spe (θ) represents the SAM loss function value. θ is the angle between the two vectors. N is the total number of images. Represents the super-resolution result output by the model. This represents a real picture. It is a picture that exists without any processing.

[0063] Gradient loss is mainly used when the structural details of an image need to be maintained or the image needs to be reconstructed. Its main function is to minimize the gradient difference between the target image and the predicted image, thereby improving the details of the edge and texture of the image and thus improving the quality of the image. It focuses on the gradient difference between the predicted image and the target image, not just the difference in pixel values. It is shown in formula (6). M is an operator representing the gradient of H. The gradient map of the gradient H of the hyperspectral image is shown in formula (5). M(*) is the gradient operator used to obtain H. Represents a function that calculates gradient values ​​in the horizontal, vertical, and spectral dimensions.

[0064]

[0065] Where, L gra (θ) is the value of the gradient loss function.

[0066] In the present invention, the weighted average of LCharbonnier, SAM loss and gradient loss function is used as the total loss function of the model, which is shown in formula (7).

[0067] L total (θ)=L Charbonnier +0.5L spe +0.1L gra (7)

[0068] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A hyperspectral image super-resolution reconstruction method based on recursive hole self-attention, characterized in that: The following steps are involved: Acquire an original low-resolution image, preprocess the low-resolution image, and obtain a grouped feature image; Based on the local feature extraction module, a channel-level local feature extraction is performed on the grouped feature image; Inputting the local features at the one channel level into a global feature extraction module to extract global features of the image; Performing upsampling and convolution operations on the global features of the image to obtain a hyperspectral super-resolution image; The local feature extraction module is a MSAM-multi-scale spectral attention module, which consists of a deformable convolution module DCONV and a residual mean-convolution extraction block AC; The deformable convolution module DCONV is used to extract spatial information of different scales by using three dilated convolutions with different dilation rates; The residual mean-convolution extraction block AC is used to learn the importance of each channel based on the spatial information of different scales, extract the features of the key channels that meet the preset requirements, and obtain the information between the channels; The global feature extraction module consists of the LiteViT model, which uses the recursive void attention mechanism RASA and the void attention mechanism ASA.

2. The hyperspectral image super-resolution reconstruction method based on recursive hole self-attention according to claim 1 is characterized in that: Based on the spatial information of different scales, the method of learning the importance of each channel includes: Compress the channel information through average pooling operation; The weights of the compressed channel information are learned through several fully connected layers to obtain the weight information between channels; Based on the weight information between the channels, the channels of the image are weighted and recalibrated to extract the features of the key channels that meet the preset requirements and obtain the information between the channels.

3. The hyperspectral image super-resolution reconstruction method based on recursive hole self-attention according to claim 1 is characterized in that: The residual mean-convolution extraction block AC consists of global pooling, convolution and sigmoid activation functions and residual connections.

4. The hyperspectral image super-resolution reconstruction method based on recursive hole self-attention according to claim 1 is characterized in that: Inputting the local features at the one channel level into the global feature extraction module, the method for extracting the global features of the image includes: Firstly perform a layer normalization operation on the local features at the channel level; Perform a transposed convolution on the normalized result; The convolution result is sent to the attention mechanism of LiteViT to extract the global features of the image.

5. The hyperspectral image super-resolution reconstruction method based on recursive hole self-attention according to claim 4 is characterized in that: The convolution result is fed into LiteViT's attention mechanism. The methods for extracting the global features of the image include: The convolution result is sent to the void attention mechanism ASA, and multi-scale queries are calculated through three deep convolutions after linear projection; Add the calculated result to the weight calculated by the Sigmoid function to capture multi-scale information; The captured multi-scale information is sent as query (Query, Q), key (Key, K), and value (Value, V) to perform a CSA attention mechanism operation; The result after the CSA attention mechanism is sent to a multi-layer perceptron MLP after a Norm operation, the output is downsampled, and the downsampled output is sent to the recursive void self-attention mechanism RASA to obtain the global features of the image.

Images