An image enhancement method based on spatial spectrum aggregation and interaction

Abstract

This invention provides an image enhancement method based on spatial spectral aggregation and interaction, comprising: acquiring a first hyperspectral image to be processed; processing the first hyperspectral image using a trained two-dimensional aggregation interaction network to obtain a second hyperspectral image, wherein the resolution of the second hyperspectral image is higher than that of the first hyperspectral image; wherein the two-dimensional aggregation interaction network includes a group feature extraction module, a global feature extraction module, a bicubic interpolation upsampling module, and a high-resolution image reconstruction module. The group feature extraction module and the global feature extraction module systematically utilize intra-dimensional and inter-dimensional correlations through multi-scale aggregation within dimensions and inter-dimensional interaction, and finally use a high-resolution image reconstruction unit to perform image reconstruction, achieving excellent performance in hyperspectral image enhancement tasks and exhibiting superior performance in various super-resolution tasks.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and in particular to an image enhancement method based on spatial spectral aggregation and interaction. Background Technology

[0002] Hyperspectral images (HSI) possess dozens to hundreds of spectral bands, providing rich spectral information for various substances and targets, and have been widely used in fields such as medical diagnosis, mineral exploration, and ground object detection. However, due to limitations in imaging equipment, HSI often has low spatial resolution, leading to the loss of detailed information, including minute spatial textures and details. Hyperspectral Image Super-Resolution (HISR) refers to generating high-resolution hyperspectral images (HRHSI) from low-resolution hyperspectral images (LRHSI). Therefore, HISR technology has significant research value in various fields.

[0003] Recently, many HISR methods have been proposed; however, existing methods neglect the intra-dimensional and inter-dimensional correlations of spatial and spectral information, resulting in poor super-resolution quality. How to effectively and comprehensively utilize the similarity and complementarity of spatial and spectral information remains a challenging problem. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the present invention provides an image enhancement method based on spatial spectral aggregation and interaction to solve the technical problem of poor super-resolution quality of hyperspectral images in the prior art.

[0005] To achieve the above and other related objectives, this invention provides an image enhancement method based on spatial spectral aggregation and interaction, comprising: acquiring a first hyperspectral image to be processed; processing the first hyperspectral image using a trained two-dimensional aggregation interaction network to obtain a second hyperspectral image, wherein the resolution of the second hyperspectral image is higher than that of the first hyperspectral image; wherein the expression of the two-dimensional aggregation interaction network is as follows:

[0006] I SR =H RU (H GU (H SU (I LR )),I LR ↑),

[0007] In the formula, I LRFor the first hyperspectral image, I SR The second hyperspectral image; H SU and H GU These are a group feature extraction module and a global feature extraction module, respectively, for multi-scale aggregation within dimensions and interaction between dimensions; I LR ↑ indicates that the first hyperspectral image is upsampled using bicubic interpolation; H RU This is a high-resolution image reconstruction module.

[0008] In one embodiment of the present invention, the group feature extraction module comprises G dual extraction interaction units and a convolutional layer; the group feature extraction module processes the first hyperspectral image according to the following steps: dividing the first hyperspectral image into G groups along the channel dimension; extracting features from the G groups using the G dual extraction interaction units respectively; concatenating the features extracted by all G dual extraction interaction units; and mapping the concatenated features to a higher dimension using the convolutional layer to obtain the output features of the group feature extraction module.

[0009] In one embodiment of the present invention, the expression of the dual extraction interaction unit is as follows:

[0010] F spa =SPM(X) DEIB-in )·SpeIM(SWM(X DEIB-in )),

[0011] F spe =SWM(X DEIB-in )·SpaIM(SPM(X DEIB-in )),

[0012] X DEIB-out =Conv(F spa +F spe ),

[0013] In the formula, X DEIB-in and X DEIB-out These are the input and output features of the dual extraction interaction unit, respectively; SPM is the spatial pyramid block, SWM is the spectral weight block, SpeIM is the spectral interaction block, SpaIM is the spatial interaction block, and F... spa and F spe It is an intermediate feature.

[0014] In one embodiment of the present invention, the expression for the spatial pyramid block is as follows:

[0015] X1=ReLU(DConv(X SPM-in )),

[0016] X2=ReLU(DConv(Downsampling(X1))),

[0017] X3 = ReLU(Conv(Downsampling(X2))),

[0018] X SPM-out =Conv(Upsampling(Conv(Upsampling(X3)+X2))+X1),

[0019] In the formula, X SPM-in and X SPM-out These are the input and output features of the spatial pyramid block, respectively; DConv is depthwise separable convolution, ReLU is the activation function, Conv is convolution, Downsampling is downsampling, Upsampling is upsampling, and X1 to X3 are intermediate features.

[0020] In one embodiment of the present invention, the spectral weight block weights the input feature X according to the following steps. SWM-in Processing: The input feature X of size C×H×W SWM-in Perform a convolution operation to obtain features with unchanged size, then perform a reshaping operation to obtain a first feature with size C×HW; for the input feature X SWM-in Perform a convolution operation to compress the number of channels to 1, then perform a reshaping operation to obtain a second feature of size HW×1; perform matrix multiplication on the first and second features, and perform another reshaping operation to obtain a third feature X4 of size C×1×1; process the input feature X according to the following formula. SWM-in Processing with the third feature X4:

[0021] X SWM-out =(ReLU(X) SWM-in )·ReLU(Sigmoid(X4)),

[0022] In the formula, X SWM-out This refers to the output characteristics of the spectral weight block.

[0023] In one embodiment of the present invention, the expression of the spatial interaction block is as follows:

[0024] X5=GeLU(Conv(X SpaIM-in )),

[0025] X6 = AxialShift(X5) + X5,

[0026] X SpaIM-out =Sigmoid(Conv(X6)),

[0027] In the formula, X SpaIM-in and X SpaIM-outThese are the input and output features of the spatial interaction block, respectively. GeLU is the activation function, AxialShift is the axial displacement, and X5 and X6 are intermediate features.

[0028] In one embodiment of the present invention, the expression of the spectral interaction block is as follows:

[0029] X7 = AvgPool(X SpeIM-in ),

[0030] X8=GeLU(Conv(X7))+ChannelMix(GeLU(GConv(X7))),

[0031] X SpeIM-out =Sigmoid(Conv(X8)),

[0032] In the formula, X SpeIM-in and X SpeIM-out These are the input and output features of the spectral interaction block, respectively. AvgPool is average pooling, ChannelMix is ​​channel shuffling, GConv is grouped convolution, and X7 and X8 are intermediate features.

[0033] In one embodiment of the present invention, the expression of the global feature extraction module is as follows:

[0034] X GU-out =H DITB (X GU-in )+X GU-in ,

[0035] In the formula, X GU-in and X GU-out H represents the input and output features of the global feature extraction module, respectively. DITB It is a dual-interaction Transformer unit.

[0036] In one embodiment of the present invention, the expression of the dual-interaction Transformer unit is as follows:

[0037] Z spa =MSSA(LN(X) DITB-in ))·SpeIM(LN(X DITB-in )),

[0038] Z spe =WSSA(LN(X) DITB-in ))·SpaIM(LN(X DITB-in )),

[0039] Z'=(Z spa +Z spe )+X DITB-in ,

[0040] X DITB-out =MLP(LN(Z'))+Z',

[0041] In the formula, X DITB-in and X DITB-out These are the input and output features of the dual-interaction Transformer unit, respectively; MSSA is the masked spatial self-attention block, WSSA is the windowed spectral self-attention block, SpeIM is the spectral interaction block, SpaIM is the spatial interaction block, LN is layer normalization, and MLP is the multilayer perceptron.

[0042] In one embodiment of the present invention, the mask space self-attention block applies the following steps to the input feature X. MSSA-in Processing: The input feature X MSSA-in Perform convolution and depthwise separable convolution sequentially to generate Q1, K1, and V1; calculate the mask space self-attention using the following formula:

[0043]

[0044] In the formula, λ i Let λ be a pre-defined set of sparsity rates, and 0 ≤ λ i ≤1; for the Attention Mask Reshape to obtain the input feature X MSSA-in Features of the same size are the output features X of the self-attention block in the mask space. MSSA-out .

[0045] In one embodiment of the present invention, the windowed spectral self-attention block applies the following steps to the input feature X. WSSA-in Processing: The input feature X WSSA-in The tensor is projected onto a 3C-channel tensor by a 1×1 convolution, and then divided into Q2, K2, and V2 according to the channels. Q2, K2, and V2 are then spatially divided into M×M windows to obtain Q. 2,j K 2,j V 2,j Where j∈{1,2,3,…,M×M}, M=2s, and s is a scaling factor; calculate the self-attention of each window using the following formula:

[0046]

[0047] The windows are merged in their original arrangement and then convolved and projected to obtain the output feature X of the window's spectral self-attention block. WSSA-out .

[0048] In one embodiment of the present invention, the expression of the high-resolution image reconstruction module is as follows:

[0049] I SR =Conv(Upsampling(X) GU-out )+Conv(I LR ↑)),

[0050] In the formula, X GU-out The output features of the global feature extraction module are defined as Upsampling and Conv.

[0051] The beneficial effects of this invention are as follows: The image enhancement method based on spatial spectral aggregation and interaction proposed in this invention systematically utilizes intra-dimensional and inter-dimensional correlations through multi-scale aggregation within dimensions and inter-dimensional interaction of the group feature extraction module and the global feature extraction module. Finally, the high-resolution image reconstruction unit is used to perform image reconstruction, which achieves excellent performance in hyperspectral image enhancement tasks and has excellent performance in various super-resolution tasks. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The accompanying drawings are incorporated in and constitute a part of this specification, illustrating embodiments consistent with this application, and are used together with the description to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0053] Figure 1 A flowchart of an image enhancement method provided in an embodiment of the present invention;

[0054] Figure 2 This is a general architecture diagram of a two-dimensional aggregation interaction network provided in an embodiment of the present invention;

[0055] Figure 3 This is an architectural diagram of a dual extraction interaction unit provided in an embodiment of the present invention;

[0056] Figure 4 This is a structural diagram of a spatial pyramid block provided in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the architecture of a spectral weighting block provided in an embodiment of the present invention;

[0058] Figure 6 This is an architectural diagram of a spatial interaction block provided in an embodiment of the present invention;

[0059] Figure 7This is a schematic diagram of an axial displacement operation provided in an embodiment of the present invention;

[0060] Figure 8 This is an architectural diagram of a spectral interaction block provided in an embodiment of the present invention;

[0061] Figure 9 This is a schematic diagram of a channel rinsing operation provided in an embodiment of the present invention;

[0062] Figure 10 This is an architecture diagram of a dual-interaction Transformer unit provided in an embodiment of the present invention;

[0063] Figure 11 This is an architectural diagram of a mask space self-attention block provided in an embodiment of the present invention;

[0064] Figure 12 This is a schematic diagram of the architecture of a windowed spectral self-attention block provided in an embodiment of the present invention. Detailed Implementation

[0065] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other. In addition to the specific methods, equipment, and materials used in the embodiments, based on the knowledge of the prior art and the description of the present invention by those skilled in the art, any prior art methods, equipment, and materials similar to or equivalent to the methods, equipment, and materials in the embodiments of the present invention can be used to implement the present invention.

[0066] It should be understood that the terminology used in the embodiments of this invention is for describing specific particular implementations and not for limiting the scope of protection of this invention. Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art.

[0067] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In some embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0068] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions, and operations that may be implemented in the methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0069] To facilitate understanding of this invention, the following is a summary of some concepts, abbreviations, and their hierarchical relationships used in the text:

[0070] Two-dimensional convergent interactive network;

[0071] 1. Group feature extraction module;

[0072] 1.1 Dual Extraction Interaction Unit (DEIB);

[0073] 1.1.1 Spatial Pyramid Block (SPM);

[0074] 1.1.2 Spectral Weighted Block (SWM);

[0075] 1.1.3 Spatial Interaction Block (SpaIM);

[0076] 1.1.4, Spectral Interaction Block (SpeIM);

[0077] 2. Global Feature Extraction Module;

[0078] 2.1 Dual Interactive Transformer Unit (DITB);

[0079] 2.1.1 Mask Space Self-Attention Block (MSSA);

[0080] 2.1.2 Windowed Spectral Self-Attention Block (WSSA);

[0081] 2.1.3 Spatial Interaction Block (SpaIM);

[0082] 3.1.4, Spectral Interaction Block (SpeIM);

[0083] 3. High-resolution image reconstruction module.

[0084] 4. Bicubic interpolation upsampling.

[0085] Please see Figure 1 , Figure 1 An image enhancement method based on spatial spectral aggregation and interaction, provided in an embodiment of the present invention, includes: acquiring a first hyperspectral image to be processed (i.e., Figure 1 The first hyperspectral image (LR image in the image) is processed using a trained two-dimensional aggregation interaction network to obtain the second hyperspectral image (i.e., the LR image in the image); Figure 1 (HR image in the image), the second hyperspectral image has a higher resolution than the first hyperspectral image.

[0086] In this embodiment, the architecture diagram of the two-dimensional aggregation interaction network is as follows: Figure 2 As shown, its expression is as follows:

[0087] I SR =H RU (H GU (H SU (I LR )),I LR ↑),

[0088] In the formula, I LR For the first hyperspectral image, I SR This is the second hyperspectral image; H SU and H GU These are a group feature extraction module and a global feature extraction module, respectively, for multi-scale aggregation within dimensions and interaction between dimensions; I LR ↑ indicates that the first hyperspectral image is upsampled using bicubic interpolation; H RU This module is for high-resolution image reconstruction. The group feature extraction module and the global feature extraction module systematically utilize intra-dimensional and inter-dimensional correlations through multi-scale aggregation within dimensions and inter-dimensional interactions. Finally, the high-resolution image reconstruction unit is used for image reconstruction, achieving excellent performance in hyperspectral image enhancement tasks and demonstrating superior performance in various super-resolution tasks.

[0089] Understandably, the first hyperspectral image to be processed is a preprocessed image to fit the input size of the two-dimensional aggregation interaction network. The trained two-dimensional aggregation interaction network can generally be obtained through the following steps: (1) first obtain LRHSI-HRHSI image pairs (i.e., LR-HR image pairs) as training samples; (2) build the two-dimensional aggregation interaction network; (3) use the samples to train the two-dimensional aggregation interaction network to obtain the trained two-dimensional aggregation interaction network.

[0090] During training, the loss function is used to calculate the super-resolution result I. SR With real data IHR The error between the two is calculated, and the network parameters are optimized using backpropagation to complete the training of the two-dimensional aggregation interaction network. Specifically, this involves combining image I from the LRHSI-HRHSI image pair above. LR Substituting it into the two-dimensional convergent interactive network, we obtain image I. SR Then, based on image I SR Image I in the LRHSI-HRHSI image pair HR Calculate the loss using the following loss function formula:

[0091] L total =L1+0.5L spe ,

[0092]

[0093] In the formula, The real image corresponding to the input data. The super-resolution result of the network is shown below, with the superscript 'i' indicating the correspondence between the super-resolution result and the real image, and 'N' representing the total number of samples in each batch. The above steps facilitate the training of the network parameters. The trained two-dimensional aggregation interactive network can then be directly used to perform super-resolution on the input first hyperspectral image to obtain its corresponding second hyperspectral image.

[0094] Please see Figure 2 In a specific embodiment of the present invention, the group feature extraction module consists of G dual extraction interaction units and one convolutional layer, wherein the dual extraction interaction unit is... Figure 2 The DEIB group feature extraction module processes the first hyperspectral image according to the following steps.

[0095] (1) Divide the first hyperspectral image into G groups along the channel dimension. Each group has the same number of spectral bands, which can be expressed by the formula:

[0096]

[0097] (2) Features of the G groups are extracted using G dual extraction interaction units. The G dual extraction interaction units share parameters, which can be expressed by the formula:

[0098]

[0099] In the formula, H DEIB It is a dual-extraction interactive unit.

[0100] (3) The features extracted by all G dual-extraction interaction units are concatenated, which can be expressed by the formula:

[0101]

[0102] (4) The concatenated features are mapped to a higher dimension using a convolutional layer to obtain the output features of the group feature extraction module, which can be expressed by the formula:

[0103]

[0104] In the formula, This refers to the output features of the group feature extraction module.

[0105] Please see Figure 3 In a specific embodiment of the present invention, the expression of the dual extraction interaction unit is as follows:

[0106] F spa =SPM(X) DEIB-in )·SpeIM(SWM(X DEIB-in )),

[0107] F spe =SWM(X DEIB-in )·SpaIM(SPM(X DEIB-in )),

[0108] X DEIB-out =Conv(F spa +F spe ),

[0109] In the formula, X DEIB-in and X DEIB-out These represent the input and output features of the dual extraction interaction unit, respectively; SPM is the spatial pyramid block, SWM is the spectral weight block, SpeIM is the spectral interaction block, SpaIM is the spatial interaction block, and F... spa and F spe This serves as an intermediate feature. To facilitate effective cross-dimensional interaction, spatial interaction blocks and spectral interaction blocks enhance information through neighborhood expansion in the spatial and channel dimensions, respectively.

[0110] Please see Figure 4 In a specific embodiment of the present invention, the expression for the spatial pyramid block is as follows:

[0111] X1=ReLU(DConv(X SPM-in )),

[0112] X2=ReLU(DConv(Downsampling(X1))),

[0113] X3 = ReLU(Conv(Downsampling(X2))),

[0114] X SPM-out=Conv(Upsampling(Conv(Upsampling(X3)+X2))+X1),

[0115] In the formula, X SPM-in and X SPM-out These represent the input and output features of the spatial pyramid block, respectively; DConv is depthwise separable convolution, ReLU is the activation function, Conv is convolution, Downsampling is downsampling, Upsampling is upsampling, and X1 to X3 are intermediate features. The spatial pyramid block is a module for multi-scale feature extraction. In each layer, dilated convolutions with different dilation rates are used to achieve receptive fields of different scales, enhancing the model's ability to perceive object size, position, and context.

[0116] Please see Figure 5 In a specific embodiment of the present invention, the spectral weight block weights the input feature X according to the following steps. SWM-in Processing: (1) Processing the input feature X of size C×H×W SWM-in (2) Perform convolution operation to obtain features with unchanged size, and then perform reshaping operation to obtain the first feature with size C×HW; SWM-in (2) Perform convolution operation to compress the number of channels to 1 and then perform reshaping operation to obtain the second feature with size HW×1; (3) Perform matrix multiplication on the first and second features to obtain a feature map of size C×1, and perform reshaping operation again to obtain the third feature X4 with size C×1×1; (4) Perform the input feature X according to the following formula. SWM-in Processing with the third feature X4:

[0117] X SWM-out =(ReLU(X) SWM-in )·ReLU(Sigmoid(X4)),

[0118] In the formula, X SWM-out This represents the output feature of the spectral weight block. The spectral weight block obtains a vector representing the spectrum through linear projection and acquires channel weights through vector calculation, thus enabling the analysis of continuous spectral variation trends.

[0119] It should be noted that, in order to illustrate the dimensions of the input, output, and intermediate features of each module, we define the dimensions of the input features as C×H×W, and use this dimension as a basis to illustrate the dimensions of other features. For different modules, the dimensions of their input features are all defined as C×H×W, but this does not mean that the input feature dimensions of these modules are the same; they can be the same or different. This dimension is only used to illustrate the size variations of subsequent features within that module.

[0120] Please see Figure 6 In a specific embodiment of the present invention, the expression of the spatial interaction block is as follows:

[0121] X5=GeLU(Conv(X SpaIM-in )),

[0122] X6 = AxialShift(X5) + X5,

[0123] X SpaIM-out =Sigmoid(Conv(X6)),

[0124] In the formula, X SpaIM-in and X SpaIM-out X1 and X2 are the input and output features of the spatial interaction block, respectively. GeLU is the activation function, AxialShift is the axial displacement, and X5 and X6 are intermediate features.

[0125] In this embodiment, the input feature X of the spatial interaction block is recorded. SpaIM-in The size of the feature is C×H×W. First, it is mapped to d channels through convolution and GeLU function, and the resulting feature X5 has a size of d×H×W.

[0126] Axial displacement such as Figure 7 As shown, it divides feature X5 into six equal groups along the channel dimension. The feature maps of the first three groups are shifted to the left along the horizontal axis (shifts of H / 4, H / 2, and 3H / 4, respectively), and the feature maps of the last three groups are shifted to the top along the vertical axis (shifts of W / 4, W / 2, and 3W / 4, respectively). Regions exceeding the boundary are filled to the opposite side in a cyclic manner. Finally, the six shifted equal groups are combined along the channel dimension, and residual connections are used to preserve the original spatial information, resulting in feature X6. The axial displacement operation establishes connections between each feature point and its multiple axial points, enabling the spatial interaction block to accumulate comprehensive spatial context knowledge. Finally, the channel dimension is modulated through convolution and the sigmoid activation function to generate spatial interaction features. Figure X SpaIM-out Its dimensions are 1×H×W.

[0127] Please see Figure 8 In a specific embodiment of the present invention, the expression of the spectral interaction block is as follows:

[0128] X7 = AvgPool(X SpeIM-in ),

[0129] X8=GeLU(Conv(X7))+ChannelMix(GeLU(GConv(X7))),

[0130] X SpeIM-out =Sigmoid(Conv(X8)),

[0131] In the formula, X SpeIM-in and X SpeIM-out These are the input and output features of the spectral interaction block, respectively. AvgPool is average pooling, ChannelMix is ​​channel shuffling, GConv is grouped convolution, and X7 and X8 are intermediate features.

[0132] In this embodiment, the input feature X of the spectral interaction block is denoted as... SpeIM-in The size is C×H×W. First, the spatial dimensionality information is compressed through average pooling, resulting in a feature X7 with a size of C×1×1. Then, it is processed through two parallel branches. One branch uses 1×1 convolutions and the GeLU function to adjust the channels, followed by channel shuffling to obtain supplementary information. Channel shuffling is as follows: Figure 9 As shown, it divides the input into multiple equal segments along the channel dimension and reverses the channel order within each segment. Another branch preserves the intrinsic spectral features through a combination of 1×1 convolution and the GeLU function. The outputs of the two branches establish communication between adjacent spectral channels through element-wise summation, enabling the spectral interaction blocks to accumulate rich spectral neighborhood knowledge. Finally, the channel dimension is modulated through 1×1 convolution and the Sigmoid activation function to obtain the spectral interaction features. Figure X SpeIM-out Its dimensions are C×1×1.

[0133] Please see Figure 2 In a specific embodiment of the present invention, the expression of the global feature extraction module is as follows:

[0134] X GU-out =H DITB (X GU-in )+X GU-in ,

[0135] In the formula, X GU-in and X GU-out These are the input and output features of the global feature extraction module, H. DITB It is a dual-interaction Transformer unit, that is Figure 2 DITB in.

[0136] Please see Figure 10 In a specific embodiment of the present invention, the expression of the dual-interaction Transformer unit is as follows:

[0137] Z spa =MSSA(LN(X) DITB-in ))·SpeIM(LN(X DITB-in )),

[0138] Z spe=WSSA(LN(X) DITB-in ))·SpaIM(LN(X DITB-in )),

[0139] Z'=(Z spa +Z spe )+X DITB-in ,

[0140] X DITB-out =MLP(LN(Z'))+Z',

[0141] In the formula, X DITB-in and X DITB-out These represent the input and output features of the dual-interaction Transformer unit, respectively; MSSA is the masked spatial self-attention block, WSSA is the windowed spectral self-attention block, SpeIM is the spectral interaction block, SpaIM is the spatial interaction block, LN is layer normalization, and MLP is a multilayer perceptron. In this embodiment, the masked spatial self-attention block and the windowed spectral self-attention block compute self-attention along the spatial and spectral dimensions, respectively, while the spatial and spectral interaction blocks are used to provide cross-dimensional consistency.

[0142] The architecture diagrams of the spatial interaction block and the spectral interaction block have been described in detail in the previous embodiments, and will not be repeated here. The mask spatial self-attention block and the window spectral self-attention block will be described in detail below.

[0143] Please see Figure 11 In a specific embodiment of the present invention, the mask space self-attention block applies the following steps to the input feature X. MSSA-in Processing: (1) Processing the input feature X MSSA-in Perform convolution and depthwise separable convolution sequentially to generate Q1, K1, and V1. (2) Calculate the mask space self-attention according to the following formula:

[0144]

[0145] In the formula, λ i Let λ be a pre-defined set of sparsity rates, and 0 ≤ λ i ≤1; In this step, the mask calculates the pairwise similarity between Q1 and K1, and ensures that attention focuses only on highly relevant regions by selectively pruning query-key pairs with low similarity in the attention matrix. This process is driven by a set of sparsity ratios λ. i Control, where each λ i Each branch corresponds uniquely to a single parallel computation branch. The final attention is the sum of the outputs of all branches.

[0146] (3) Attention Mask Reshape to obtain the input feature X MSSA-inFeatures of the same size are the output features X of the self-attention blocks in the mask space. MSSA-out .

[0147] In a specific embodiment of the present invention, a set of sparsity rates may have, for example, four values: 4 / 5, 3 / 4, 2 / 3, and 1 / 2.

[0148] Please see Figure 12 In a specific embodiment of the present invention, the windowed spectral self-attention block applies the following steps to the input feature X. WSSA-in Processing: (1) Input feature X WSSA-in (1) Project the tensor onto a 3C channel by a 1×1 convolution, and divide the tensor into Q2, K2, and V2 according to the channel; (2) Divide Q2, K2, and V2 into M×M windows according to the space to obtain Q 2,j K 2,j V 2,j Where j∈{1,2,3,…,M×M}, M=2s, and s is a scaling factor; let the input feature X be... WSSA-in The dimensions are C×H×W, so the size of each window is C×H / M×W / M. Dividing into multiple windows can avoid the neglect of local spectral differences caused by the multi-head partitioning of channels in the classic spectral Transformer. During the partitioning process, the overlapping areas between adjacent windows are preserved to achieve a smoother region representation and promote information sharing between windows.

[0149] (3) The window is reshaped to a shape of C×HW / M 2 The size of the token is used to calculate the self-attention of each window using the following formula:

[0150]

[0151] In the formula, j represents the j-th window. The output of the attention is reshaped to a size of C×H / M×W / M. Then, the windows are merged in their original arrangement and convolutional projection is performed to obtain the output feature X of the window spectral self-attention block. WSSA-out .

[0152] Please see Figure 2 In a specific embodiment of the present invention, the expression of the high-resolution image reconstruction module is as follows:

[0153] I SR =Conv(Upsampling(X) GU-out )+Conv(I LR ↑)),

[0154] In the formula, X GU-outX represents the output feature of the global feature extraction module. Upsampling is used for upsampling, and Conv is for convolution. Upsampling is applied to the output feature X of the global feature extraction module. GU-out Spatial dimensions were restored to match the SR image, with bicubic feature I LR The ↑ values ​​are directly added to the end of the network to better match SR features. Then, the convolutional layer adjusts the number of channels to C to obtain the final output SR image I. SR .

[0155] It should be noted that the steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0156] To evaluate the super-resolution performance of the super-resolution method of this invention, three evaluation metrics are used: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Spectral Angle Mapper (SAM). The comparisons are performed with six other hyperspectral image enhancement models (MCNet, SSPSR, RFSR, GELIN, MSDformer, and SRDNet) on different datasets and at different super-resolution magnifications.

[0157] Table 1: Comparison of ×4 super-resolution dimensions on the Cave dataset.

[0158] method scale PSNR↑ SSIM↑ SAM↓ MCNet ×4 38.4765 0.9747 3.0438 SSPSR ×4 38.5009 0.9748 3.0951 RFSR ×4 38.6181 0.9752 2.9548 GELIN ×4 38.5629 0.9751 3.0329 MSDformer ×4 38.4123 0.9744 3.1036 SRDNet ×4 38.7071 0.9753 2.9856 This invention ×4 38.8008 0.9761 2.9497 MCNet ×8 32.4231 0.9289 4.3065 SSPSR ×8 32.4634 0.9278 4.4105 RFSR ×8 32.5879 0.9315 4.2496 GELIN ×8 32.4635 0.9279 4.4362 MSDformer ×8 32.4055 0.9245 4.3685 SRDNet ×8 32.7278 0.9321 4.1046 This invention ×8 32.7899 0.9324 4.0876

[0159] Table 2: Comparison of various methods on the Chikusei dataset.

[0160]

[0161]

[0162] As can be seen from Tables 1 and 2, regardless of the dataset or whether the super-resolution size is ×4 or ×8, the performance of the present invention is superior to the six existing hyperspectral image enhancement models, which demonstrates the advantages of our model in large-channel hyperspectral image super-resolution tasks.

[0163] In summary, the two-dimensional aggregation interaction network in this invention combines spectral interaction blocks with masked spatial self-attention blocks, and spatial interaction blocks with windowed spectral self-attention blocks, achieving cross-scale feature aggregation within both spatial and spectral dimensions. Furthermore, this invention proposes spatial and spectral interaction blocks, promoting spatial-spectral dependent interactions through a neighborhood expansion mechanism and cross-dimensional collaborative guidance. Finally, image reconstruction is performed using a high-resolution image reconstruction module, achieving excellent performance across various super-resolution tasks.

[0164] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An image enhancement method based on spatial spectral aggregation and interaction, characterized in that, include: Acquire the first hyperspectral image to be processed; The first hyperspectral image is processed using a trained two-dimensional aggregation interaction network to obtain a second hyperspectral image, the resolution of which is higher than that of the first hyperspectral image. The expression for the two-dimensional aggregation interaction network is as follows: I SR =H RU (H GU (H SU (I LR )),I LR ↑), In the formula, I LR For the first hyperspectral image, I SR The second hyperspectral image; H SU and H GU These are a group feature extraction module and a global feature extraction module, respectively, for multi-scale aggregation within dimensions and interaction between dimensions; I LR ↑ indicates that the first hyperspectral image is upsampled using bicubic interpolation; H RU For high-resolution image reconstruction module; The group feature extraction module consists of G dual extraction interaction units and one convolutional layer; The feature extraction module processes the first hyperspectral image according to the following steps: The first hyperspectral image is divided into G groups along the channel dimension; Features of G groups are extracted using G dual-extraction interaction units respectively; The features extracted by all G dual-extraction interaction units are concatenated; The concatenated features are mapped to a higher dimension using a convolutional layer to obtain the output features of the group feature extraction module; The expression for the global feature extraction module is as follows: X GU-out =H DITB (X GU-in )+X GU-in , In the formula, X GU-in and X GU-out H represents the input and output features of the global feature extraction module, respectively. DITB It is a dual-interaction Transformer unit; The expression for the high-resolution image reconstruction module is as follows: I SR =Conv(Upsampling(X GU-out )+Conv(I LR ↑)), In the formula, X GU-out The output features of the global feature extraction module are defined as Upsampling and Conv.

2. The image enhancement method based on spatial spectral aggregation and interaction according to claim 1, characterized in that, The expression for the dual extraction interaction unit is as follows: F spa =SPM(X DEIB-in )·SpeIM(SWM(X DEIB-in )), F spe =SWM(X DEIB-in )·SpaIM(SPM(X DEIB-in )), X DEIB-out =Conv(F spa +F spe ), In the formula, X DEIB-in and X DEIB-out These are the input and output features of the dual extraction interaction unit, respectively. SPM stands for Spatial Pyramid Block, SWM for Spectral Weight Block, SpeIM for Spectral Interaction Block, SpaIM for Spatial Interaction Block, and F... spa and F spe It is an intermediate feature.

3. The image enhancement method based on spatial spectral aggregation and interaction according to claim 2, characterized in that, The expression for the spatial pyramid block is as follows: X1=ReLU(DConv(X SPM-in )), X2=ReLU(DConv(Downsampling(X1))), X3 = ReLU(Conv(Downsampling(X2))), X SPM-out =Conv(Upsampling(Conv(Upsampling(X3)+X2))+X1), In the formula, X SPM-in and X SPM-out These are the input and output features of the spatial pyramid block, respectively; DConv is depthwise separable convolution, ReLU is the activation function, Conv is convolution, Downsampling is downsampling, Upsampling is upsampling, and X1~X3 are intermediate features.

4. The image enhancement method based on spatial spectral aggregation and interaction according to claim 2, characterized in that, The spectral weighting block applies the input feature X according to the following steps. SWM-in Processing: For the input feature X of size C×H×W SWM-in After performing a convolution operation to obtain features with unchanged size, a reshaping operation is performed to obtain the first feature with size C×HW. For the input feature X SWM-in Perform a convolution operation to compress its number of channels to 1, and then perform a reshaping operation to obtain a second feature with a size of HW×1. Perform matrix multiplication on the first feature and the second feature, and then perform a reshaping operation to obtain a third feature X4 with a size of C×1×1; The input feature X is processed according to the following formula. SWM-in Processing with the third feature X4: X SWM-out =(ReLU(X SWM-in )·ReLU(Sigmoid(X4)), In the formula, X SWM-out This refers to the output characteristics of the spectral weight block.

5. The image enhancement method based on spatial spectral aggregation and interaction according to claim 2, characterized in that, The expression for the spatial interaction block is as follows: X5=GeLU(Conv(X SpaIM-in )), X6 = AxialShift(X5) + X5 X SpaIM-out =Sigmoid(Conv(X6)), In the formula, X SpaIM-in and X SpaIM-out These are the input and output features of the spatial interaction block, respectively. GeLU is the activation function, AxialShift is the axial displacement, and X5 and X6 are intermediate features.

6. The image enhancement method based on spatial spectral aggregation and interaction according to claim 2, characterized in that, The expression for the spectral interaction block is as follows: X7=AvgPool(X SpeIM-in ), X8=GeLU(Conv(X7))+ChannelMix(GeLU(GConv(X7))), X SpeIM-out =Sigmoid(Conv(X8)), In the formula, X SpeIM-in and X SpeIM-out These are the input and output features of the spectral interaction block, respectively. AvgPool is average pooling, ChannelMix is ​​channel shuffling, GConv is grouped convolution, and X7 and X8 are intermediate features.

7. The image enhancement method based on spatial spectral aggregation and interaction according to claim 1, characterized in that, The expression for the dual-interaction Transformer unit is as follows: Z spa =MSSA(LN(X DITB-in ))·SpeIM(LN(X DITB-in )), Z spe =WSSA(LN(X DITB-in ))·SpaIM(LN(X DITB-in )), Z’=(Z spa +Z spe )+X DITB-in , X DITB-out =MLP(LN(Z'))+Z', In the formula, X DITB-in and X DITB-out These are the input and output features of the dual-interaction Transformer unit, respectively. MSSA stands for Mask Spatial Self-Attention Block, WSSA stands for Window Spectral Self-Attention Block, SpeIM stands for Spectral Interaction Block, SpaIM stands for Spatial Interaction Block, LN stands for Layer Normalization, and MLP stands for Multilayer Perceptron.

Images