Image enhancement method based on spatial spectrum aggregation and interaction

Abstract

The invention provides an image enhancement method based on spatial spectrum aggregation and interaction. The method comprises the following steps: acquiring a to-be-processed first hyperspectral image; processing the first hyperspectral image by using a trained two-dimensional aggregation interaction network to obtain a second hyperspectral image, the resolution of the second hyperspectral image being higher than that of the first hyperspectral image; wherein the two-dimensional aggregation interaction network comprises a group feature extraction module, a global feature extraction module and a bicubic interpolation up-sampling and high-resolution image reconstruction module. The group feature extraction module and the global feature extraction module systematically utilize intra-dimension and inter-dimension correlation through intra-dimension multi-scale aggregation and inter-dimension interaction, and finally, image reconstruction is carried out by utilizing a high-resolution image reconstruction unit, so that excellent hyperspectral image enhancement task performance is realized; the method has excellent performance under various super-resolution tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image enhancement, in particular to an image enhancement method based on spatial-spectral aggregation and interaction. BACKGROUND

[0002] Hyperspectral image (HSI) has dozens to hundreds of bands, providing rich spectral information for different substances and targets, and has been widely used in medical diagnosis, mineral exploration, ground object detection and other fields. However, due to the limitation of imaging equipment, the spatial resolution of HSI is often low, resulting in the loss of detailed information, including small spatial textures and details. Hyperspectral image super-resolution (HISR) refers to generating high spatial resolution hyperspectral image (HRHSI) from low spatial resolution hyperspectral image (LRHSI). Therefore, HISR technology has important research significance in various fields.

[0003] Recently, many HISR methods have been proposed, however, existing such methods ignore the correlation within and between spatial and spectral dimensions, resulting in poor quality of super-resolution. How to effectively and comprehensively utilize the similarity and complementarity of spatial and spectral information is still a challenging problem. SUMMARY

[0004] In view of the defects of the above prior art, the present application provides an image enhancement method based on spatial-spectral aggregation and interaction to solve the technical problem of poor quality of hyperspectral image super-resolution in the prior art.

[0005] To achieve the above object and other related objects, the present application provides an image enhancement method based on spatial-spectral aggregation and interaction, comprising: acquiring a first hyperspectral image to be processed; processing the first hyperspectral image by using a trained two-dimensional aggregation interaction network to obtain a second hyperspectral image, the resolution of the second hyperspectral image being higher than that of the first hyperspectral image; wherein the expression of the two-dimensional aggregation interaction network is as follows:

[0006] I SR =H RU (H GU (H SU (I LR )),I LR ↑),

[0007] In the formula, I LRis the first hyperspectral image, I SR is the second hyperspectral image; H SU and H GU are a group feature extraction module and a global feature extraction module, respectively, to perform multi-scale aggregation within dimensions and inter-dimensional interaction; I LR ↑ represents bicubic interpolation up-sampling on the first hyperspectral image; H RU is a high-resolution image reconstruction module.

[0008] In an embodiment of the present application, the group feature extraction module comprises G double-extraction interaction units and a convolution layer; the group feature extraction module processes the first hyperspectral image in the following steps: dividing the first hyperspectral image into G groups along the channel dimension; extracting features of the G groups using the G double-extraction interaction units, respectively; splicing the features extracted by all G double-extraction interaction units; mapping the spliced features to a higher dimension using the convolution layer to obtain output features of the group feature extraction module.

[0009] In an embodiment of the present application, the expression of the double-extraction interaction unit is as follows:

[0010] F spa = SPM(X DEIB-in )·SpeIM(SWM(X DEIB-in )),

[0011] F spe =SWM(X DEIB-in )·SpaIM(SPM(X DEIB-in )),

[0012] X DEIB-out =Conv(F spa +F spe ),

[0013] wherein X DEIB-in and X DEIB-out are input features and output features of the double-extraction interaction unit, respectively; SPM is a spatial pyramid block, SWM is a spectral weight block, SpeIM is a spectral interaction block, SpaIM is a spatial interaction block, F spa and F spe are intermediate features.

[0014] In an embodiment of the present application, the expression of the spatial pyramid block is as follows:

[0015] X1 = ReLU(DConv(X SPM-in )),

[0016] X2 = ReLU(DConv(Downsampling(X1))),

[0017] X3 = ReLU(Conv(Downsampling(X2))),

[0018] X SPM-out =Conv(Upsampling(Conv(Upsampling(X3)+X2))+X1),

[0019] In the formula, X SPM-in and X SPM-out These are the input and output features of the spatial pyramid block, respectively; DConv is depthwise separable convolution, ReLU is the activation function, Conv is convolution, Downsampling is downsampling, Upsampling is upsampling, and X1 to X3 are intermediate features.

[0020] In one embodiment of the present invention, the spectral weight block weights the input feature X according to the following steps. SWM-in Processing: The input feature X of size C×H×W SWM-in Perform a convolution operation to obtain features with unchanged size, then perform a reshaping operation to obtain a first feature with size C×HW; for the input feature X SWM-in Perform a convolution operation to compress the number of channels to 1, then perform a reshaping operation to obtain a second feature of size HW×1; perform matrix multiplication on the first and second features, and perform another reshaping operation to obtain a third feature X4 of size C×1×1; process the input feature X according to the following formula. SWM-in Processing with the third feature X4:

[0021] X SWM-out =(ReLU(X) SWM-in )·ReLU(Sigmoid(X4)),

[0022] In the formula, X SWM-out This refers to the output characteristics of the spectral weight block.

[0023] In one embodiment of the present invention, the expression of the spatial interaction block is as follows:

[0024] X5=GeLU(Conv(X SpaIM-in )),

[0025] X6 = AxialShift(X5) + X5,

[0026] X SpaIM-out =Sigmoid(Conv(X6)),

[0027] In the formula, X SpaIM-in and X SpaIM-outX5 and X6 are intermediate features, GeLU is an activation function, AxialShift is axial shift, and X7 and X8 are intermediate features.

[0028] In an embodiment of the present application, the expression of the spectral interaction block is as follows:

[0029] X7 = AvgPool (X SpeIM-in ),

[0030] X8 = GeLU (Conv (X7)) + ChannelMix (GeLU (GConv (X7))),

[0031] X SpeIM-out = Sigmoid (Conv (X8)),

[0032] wherein X SpeIM-in and X SpeIM-out are input and output features of the spectral interaction block respectively, AvgPool is average pooling, ChannelMix is channel mixing, GConv is group convolution, and X7 and X8 are intermediate features.

[0033] In an embodiment of the present application, the expression of the global feature extraction module is as follows:

[0034] X GU-out = H DITB (X GU-in ) + X GU-in ,

[0035] wherein X GU-in and X GU-out are input and output features of the global feature extraction module respectively, and H DITB is a double interaction Transformer unit.

[0036] In an embodiment of the present application, the expression of the double interaction Transformer unit is as follows:

[0037] Z spa = MSSA (LN (X DITB-in )) · SpeIM (LN (X DITB-in )),

[0038] Z spe = WSSA (LN (X DITB-in )) · SpaIM (LN (X DITB-in )),

[0039] Z' = (Z spa + Z spe ) + X DITB-in ,

[0040] X DITB-out =MLP(LN(Z’))+Z’,

[0041] wherein, X DITB-in and X DITB-out are input features and output features of the double interaction Transformer unit respectively; MSSA is a mask space self-attention block, WSSA is a window spectral self-attention block, SpeIM is a spectral interaction block, SpaIM is a spatial interaction block, LN is layer normalization, and MLP is a multi-layer perceptron.

[0042] In an embodiment of the present application, the mask space self-attention block processes the input features X MSSA-in in the following steps: the input features X MSSA-in are sequentially subjected to convolution and depth separable convolution processing to generate Q1, K1, and V1; and the mask space self-attention is calculated according to the following formula:

[0043]

[0044] wherein, λ i is a preset set of sparsity, and 0≤λ i ≤1; the Attention Mask is reshaped to obtain a feature with the same size as the input features X MSSA-in , that is, the output feature X MSSA-out of the mask space self-attention block.

[0045] In an embodiment of the present application, the window spectral self-attention block processes the input features X WSSA-in in the following steps: the input features X WSSA-in are projected to a 3C channel tensor by 1x1 convolution, and the tensor is divided into Q2, K2, and V2 by channel; the Q2, the K2, and the V2 are divided into MxM windows to obtain Q 2,j , K 2,j , and V 2,j , where j∈{1,2,3,…,MxM}, M=2s, and s is a scale factor; and the self-attention of each window is calculated according to the following formula:

[0046]

[0047] The windows are combined in the original arrangement mode and subjected to convolution projection to obtain the output feature X WSSA-out of the window spectral self-attention block.

[0048] In an embodiment of the present application, the expression of the high-resolution image reconstruction module is as follows:

[0049] I SR =Conv(Upsampling(X GU-out )+Conv(I LR ↑)),

[0050] In the formula, X GU-out is the output feature of the global feature extraction module, Upsampling is up-sampling, and Conv is convolution.

[0051] The image enhancement method provided by the application has the advantages that the group feature extraction module and the global feature extraction module systematically utilize the intra-dimension and inter-dimension correlations through multi-scale aggregation in the dimension and inter-dimension interaction, and finally, the high-resolution image reconstruction unit is utilized to perform image reconstruction, thereby realizing excellent hyperspectral image enhancement task performance and excellent performance in various super-resolution tasks. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. The drawings herein are incorporated into the specification and form a part of the specification, show the embodiments consistent with the application, and are used together with the specification to explain the principles of the application. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0053] Figure 1 The flowchart of the image enhancement method provided by an embodiment of the application is shown in FIG. 1.

[0054] Figure 2 The total architecture diagram of the dual-dimension aggregation interaction network provided by an embodiment of the application is shown in FIG. 2.

[0055] Figure 3 The architecture diagram of the dual-extraction interaction unit provided by an embodiment of the application is shown in FIG. 3.

[0056] Figure 4 The architecture diagram of the spatial pyramid block provided by an embodiment of the application is shown in FIG. 4.

[0057] Figure 5 The architecture diagram of the spectral weight block provided by an embodiment of the application is shown in FIG. 5.

[0058] Figure 6 The architecture diagram of the spatial interaction block provided by an embodiment of the application is shown in FIG. 6.

[0059] Figure 7A schematic diagram of axial shift operation is provided for an embodiment of the present application.

[0060] Figure 8 An architecture diagram of a spectral interaction block is provided for an embodiment of the present application.

[0061] Figure 9 A schematic diagram of channel shuffle operation is provided for an embodiment of the present application.

[0062] Figure 10 An architecture diagram of a dual interaction Transformer unit is provided for an embodiment of the present application.

[0063] Figure 11 An architecture diagram of a mask space self-attention block is provided for an embodiment of the present application.

[0064] Figure 12 An architecture diagram of a window spectral self-attention block is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0065] The above and other advantages and features of the present application will become apparent from the following description of the embodiments, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the application. It is to be understood that the terminology used herein is for the purpose of describing the particular embodiments only and is not intended to be limiting of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains.

[0066] It is to be understood that the terminology used herein is for the purpose of describing the particular embodiments only and is not intended to be limiting of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains.

[0067] In the following description, numerous specific details are discussed to provide a thorough understanding of the embodiments of the application. However, one of ordinary skill in the art will recognize that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In some instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the application. Furthermore, the descriptions of the embodiments of the application are not meant to limit the scope of the application.

[0068] The computer program product of the present application can be implemented by a variety of means. For example, and these are just a few of the implementations, the computer program product can be implemented on the above-described computer system 1000 of the present application by a software program 1020 that is stored in the computer-readable medium 1010 and executable by the processor 1002. As another example, the computer program product can be implemented as a stand-alone application on the computer system 1000. As yet another example, the computer program product can be implemented as a plug-in or add-on to the computer system 1000. As yet another example, the computer program product can be implemented on a different computer system and then communicated to the computer system 1000 via a computer-readable medium 1010. As yet another example, the computer program product can be implemented on a different computer system and then communicated to the computer system 1000 via a computer network.

[0069] For the convenience of understanding the present application, some concepts, abbreviations and their hierarchical relationships involved in the text are sorted out as follows:

[0070] Dual-dimensional aggregated interaction network;

[0071] 1. Group feature extraction module;

[0072] 1.1. Dual extraction interaction unit (DEIB);

[0073] 1.1.1. Spatial pyramid block (SPM);

[0074] 1.1.2. Spectral weight block (SWM);

[0075] 1.1.3. Spatial interaction block (SpaIM);

[0076] 1.1.4. Spectral interaction block (SpeIM);

[0077] 2. Global feature extraction module;

[0078] 2.1. Dual interaction Transformer unit (DITB);

[0079] 2.1.1. Masked spatial self-attention block (MSSA);

[0080] 2.1.2. Windowed spectral self-attention block (WSSA);

[0081] 2.1.3. Spatial interaction block (SpaIM);

[0082] 3.1.4. Spectral interaction block (SpeIM);

[0083] 3. High-resolution image reconstruction module.

[0084] 4. bicubic up-sampling.

[0085] See Figure 1 , Figure 1 An image enhancement method based on spatial spectrum aggregation and interaction is provided for an embodiment of the present application, comprising: acquiring a first hyperspectral image to be processed (i.e. the LR image in Figure 1 ); processing the first hyperspectral image by using a trained dual-dimensional aggregation interaction network to obtain a second hyperspectral image (i.e. the HR image in Figure 1 ), the resolution of the second hyperspectral image being higher than that of the first hyperspectral image.

[0086] In this embodiment, the architecture diagram of the dual-dimensional aggregation interaction network is as shown in Figure 2 , and its expression is as follows:

[0087] I SR = H RU (H GU (H SU (I LR )), I LR ↑),

[0088] In the formula, I LR is the first hyperspectral image, I SR is the second hyperspectral image; H SU and H GU are a group feature extraction module and a global feature extraction module respectively, for multi-scale aggregation within the dimension and interaction between the dimensions; I LR ↑ represents bicubic up-sampling of the first hyperspectral image; H RU is a high-resolution image reconstruction module. The group feature extraction module and the global feature extraction module systematically utilize the intra-dimension and inter-dimension correlation through multi-scale aggregation within the dimension and interaction between the dimensions, and finally utilize the high-resolution image reconstruction unit to perform image reconstruction, thereby realizing excellent hyperspectral image enhancement task performance and having excellent performance under various super-resolution tasks.

[0089] Understandably, the first hyperspectral image to be processed is a pre-processed image to adapt to the input size of the dual-dimensional aggregation interaction network. The trained dual-dimensional aggregation interaction network can generally be obtained through the following steps: (1) first acquire a pair of LR HSI-HR HSI images (i.e. a pair of LR-HR images) as training samples; (2) build a dual-dimensional aggregation interaction network; (3) train the dual-dimensional aggregation interaction network by using the samples to obtain the trained dual-dimensional aggregation interaction network.

[0090] In the training process, a loss function is used to calculate the measure of the super-resolution result I SR and the real data IHR error between them, and the network parameters are optimized using the back propagation method, thus completing the training of the dual-dimensional aggregation interaction network. Specifically, the image I LR in the above LRHSI-HRHSI image pair is substituted into the dual-dimensional aggregation interaction network to obtain the image I SR ; then, the loss loss is calculated according to the image I SR , the image I HR in the LRHSI-HRHSI image pair, and the following loss function calculation formula:

[0091] L total = L1+ 0.5L spe ,

[0092]

[0093] In the formula, I is a real image corresponding to the input data, I is a super-resolution result of the network, the superscript i is used to indicate the correspondence between the super-resolution result and the real image, and N is the total number of samples in each batch. The above steps can conveniently help us to train the network parameters, and the trained dual-dimensional aggregation interaction network can be directly used to perform super-resolution on the input first hyperspectral image to obtain the corresponding second hyperspectral image.

[0094] Please refer to Figure 2 In a specific embodiment of the present application, the group feature extraction module comprises G double-extraction interaction units and a convolution layer, and the double-extraction interaction unit is DEIB in Figure 2 . The group feature extraction module processes the first hyperspectral image according to the following steps.

[0095] (1) The first hyperspectral image is divided into G groups along the channel dimension, and the number of spectral bands in each group is consistent, which can be represented by the formula:

[0096]

[0097] (2) G double-extraction interaction units are used to extract the features of the G groups, and the G double-extraction interaction units have shared parameters, which can be represented by the formula:

[0098]

[0099] In the formula, H DEIB is the double-extraction interaction unit.

[0100] (3) The features extracted by all the G double-extraction interaction units are spliced, which can be represented by the formula:

[0101]

[0102] (4) The spliced features are mapped to a higher dimension by using a convolution layer to obtain output features of the group feature extraction module, which can be expressed by a formula as follows:

[0103]

[0104] In the formula, The output features of the group feature extraction module are obtained.

[0105] Please refer to Figure 3 In an embodiment of the present application, the expression of the double-extraction interaction unit is as follows:

[0106] F spa = SPM(X DEIB-in )·SpeIM(SWM(X DEIB-in )),

[0107] F spe = SWM(X DEIB-in )·SpaIM(SPM(X DEIB-in )),

[0108] X DEIB-out = Conv(F spa +F spe ),

[0109] In the formula, X DEIB-in and X DEIB-out are input features and output features of the double-extraction interaction unit respectively; SPM is a spatial pyramid block, SWM is a spectral weight block, SpeIM is a spectral interaction block, SpaIM is a spatial interaction block, F spa and F spe are intermediate features. In order to promote effective cross-dimension interaction, the spatial interaction block and the spectral interaction block respectively perform neighborhood expansion information enhancement from the spatial and channel dimensions.

[0110] Please refer to Figure 4 In an embodiment of the present application, the expression of the spatial pyramid block is as follows:

[0111] X1 = ReLU(DConv(X SPM-in )),

[0112] X2 = ReLU(DConv(Downsampling(X1))),

[0113] X3 = ReLU(Conv(Downsampling(X2))),

[0114] X SPM-out=Conv(Upsampling(Conv(Upsampling(X3)+X2))+X1),

[0115] wherein X SPM-in and X SPM-out are input and output features of the spatial pyramid block respectively; DConv is a depth separable convolution, ReLU is an activation function, Conv is a convolution, Downsampling is downsampling, Upsampling is upsampling, and X1-X3 are intermediate features. The spatial pyramid block is a module for multi-scale feature extraction, which realizes receptive fields of different scales in each layer by using dilated convolutions with different dilation rates, and enhances the model's perception of object size, position and context.

[0116] See Figure 5 In a specific embodiment of the present application, the spectral weight block processes the input feature X SWM-in in the following steps: (1) performing convolution operation on the input feature X SWM-in with a size of CxHxW to obtain a feature with an unchanged size, and then performing reshaping operation to obtain a first feature with a size of CxHW; (2) performing convolution operation on the input feature X SWM-in to compress its channel number to 1, and then performing reshaping operation to obtain a second feature with a size of HWx1; (3) performing matrix multiplication on the first feature and the second feature to obtain a feature map with a size of Cx1, and then performing reshaping operation to obtain a third feature X4 with a size of Cx1x1; and (4) processing the input feature X SWM-in and the third feature X4 according to the following formula:

[0117] X SWM-out =(ReLU(X SWM-in )·ReLU(Sigmoid(X4)),

[0118] wherein X SWM-out is the output feature of the spectral weight block. The spectral weight block obtains a vector representing the spectrum through linear projection, and obtains channel weights through vector calculation, thereby realizing analysis of the trend of continuous spectrum change.

[0119] It should be noted that, in order to explain the sizes of the input features, output features and intermediate features of each module, we define the size of the input feature as CxHxW, and based on this size, we explain the sizes of other features. For different modules, the size of their input features is defined as CxHxW, which does not mean that the sizes of the input features of these modules are consistent. It can be consistent or inconsistent, and this size is only used to introduce the size changes of the subsequent features in the module.

[0120] See Figure 6 In an embodiment of the present application, the expression of the spatial interaction block is as follows:

[0121] X5=GeLU(Conv(X SpaIM-in )),

[0122] X6=AxialShift(X5)+X5,

[0123] X SpaIM-out =Sigmoid(Conv(X6)),

[0124] where X SpaIM-in and X SpaIM-out are the input and output features of the spatial interaction block respectively, GeLU is an activation function, AxialShift is axial shift, X5 and X6 are intermediate features.

[0125] In this embodiment, the size of the input feature X SpaIM-in of the spatial interaction block is C×H×W. First, it is mapped to d channels through convolution and GeLU function, and the obtained feature X5 has a size of d×H×W.

[0126] The axial shift is shown in Figure 7 . It divides the feature X5 into six equal groups along the channel dimension, and the feature maps of the first three groups are shifted left along the horizontal axis (the shift amounts are H / 4, H / 2 and 3H / 4 respectively), and the feature maps of the last three groups are shifted up along the vertical axis (the shift amounts are W / 4, W / 2 and 3W / 4 respectively). The regions beyond the boundary are filled to the opposite side in a circular manner. Finally, the six equal groups after the shift are combined along the channel dimension, and the residual connection is reserved to retain the original spatial information to obtain the feature X6. The axial shift operation establishes the connection between each feature point and its multiple axial points, so that the spatial interaction block can accumulate comprehensive spatial context knowledge. Finally, the channel dimension is modulated through convolution and Sigmoid activation function to generate the spatial interaction feature Figure X SpaIM-out with a size of 1×H×W.

[0127] See Figure 8 In an embodiment of the present application, the expression of the spectral interaction block is as follows:

[0128] X7=AvgPool(X SpeIM-in ),

[0129] X8=GeLU(Conv(X7))+ChannelMix(GeLU(GConv(X7))),

[0130] X SpeIM-out =Sigmoid(Conv(X8)),

[0131] In the formula, X SpeIM-in and X SpeIM-out These are the input and output features of the spectral interaction block, respectively. AvgPool is average pooling, ChannelMix is ​​channel shuffling, GConv is grouped convolution, and X7 and X8 are intermediate features.

[0132] In this embodiment, the input feature X of the spectral interaction block is denoted as... SpeIM-in The size is C×H×W. First, the spatial dimensionality information is compressed through average pooling, resulting in a feature X7 with a size of C×1×1. Then, it is processed through two parallel branches. One branch uses 1×1 convolutions and the GeLU function to adjust the channels, followed by channel shuffling to obtain supplementary information. Channel shuffling is as follows: Figure 9 As shown, it divides the input into multiple equal segments along the channel dimension and reverses the channel order within each segment. Another branch preserves the intrinsic spectral features through a combination of 1×1 convolution and the GeLU function. The outputs of the two branches establish communication between adjacent spectral channels through element-wise summation, enabling the spectral interaction blocks to accumulate rich spectral neighborhood knowledge. Finally, the channel dimension is modulated through 1×1 convolution and the Sigmoid activation function to obtain the spectral interaction features. Figure X SpeIM-out Its dimensions are C×1×1.

[0133] Please see Figure 2 In a specific embodiment of the present invention, the expression of the global feature extraction module is as follows:

[0134] X GU-out =H DITB (X GU-in )+X GU-in ,

[0135] In the formula, X GU-in and X GU-out These are the input and output features of the global feature extraction module, H. DITB It is a dual-interaction Transformer unit, that is Figure 2 DITB in.

[0136] Please see Figure 10 In a specific embodiment of the present invention, the expression of the dual-interaction Transformer unit is as follows:

[0137] Z spa =MSSA(LN(X) DITB-in ))·SpeIM(LN(X DITB-in )),

[0138] Z speWSSA(LN(X DITB-in ))·SpaIM(LN(X DITB-in )),

[0139] Z’=(Z spa +Z spe )+X DITB-in ,

[0140] X DITB-out =MLP(LN(Z’))+Z’,

[0141] where X DITB-in and X DITB-out are the input and output features of the dual-interaction Transformer unit, respectively; MSSA is the mask spatial self-attention block, WSSA is the window spectral self-attention block, SpeIM is the spectral interaction block, SpaIM is the spatial interaction block, LN is the layer normalization, and MLP is the multi-layer perceptron. In this embodiment, the mask spatial self-attention block and the window spectral self-attention block calculate self-attention along the spatial dimension and the spectral dimension, respectively, and the spatial interaction block and the spectral interaction block are used to provide cross-dimension consistency.

[0142] In the foregoing embodiments, the architecture diagrams of the spatial interaction block and the spectral interaction block have been described in detail, and thus will not be described again here. The mask spatial self-attention block and the window spectral self-attention block will be described in detail below.

[0143] Referring to Figure 11 , in a specific embodiment of the present application, the mask spatial self-attention block processes the input feature X MSSA-in in the following steps: (1) the input feature X MSSA-in is subjected to convolution and depthwise separable convolution in sequence to generate Q1, K1, and V1. (2) the mask spatial self-attention is calculated according to the following formula:

[0144]

[0145] where λ i is a preset set of sparsity rates, and 0≤λ i ≤1; in this step, the pair-wise similarity between Q1 and K1 is calculated, and by selectively pruning the query-key pairs with low similarity in the attention matrix, it is ensured that the attention is focused only on the highly relevant areas, and this process is controlled by a set of sparsity rates λ i , where each λ i corresponds to one parallel computing branch. The final attention is the sum of the outputs of all branches.

[0146] (3) Attention Mask is reshaped to obtain the output feature X MSSA-inFeatures of the same size are the output features X of the self-attention blocks in the mask space. MSSA-out .

[0147] In a specific embodiment of the present invention, a set of sparsity rates may have, for example, four values: 4 / 5, 3 / 4, 2 / 3, and 1 / 2.

[0148] Please see Figure 12 In a specific embodiment of the present invention, the windowed spectral self-attention block applies the following steps to the input feature X. WSSA-in Processing: (1) Input feature X WSSA-in (1) Project the tensor onto a 3C channel by a 1×1 convolution, and divide the tensor into Q2, K2, and V2 according to the channel; (2) Divide Q2, K2, and V2 into M×M windows according to the space to obtain Q 2,j K 2,j V 2,j Where j∈{1,2,3,…,M×M}, M=2s, and s is a scaling factor; let the input feature X be... WSSA-in The dimensions are C×H×W, so the size of each window is C×H / M×W / M. Dividing into multiple windows can avoid the neglect of local spectral differences caused by the multi-head partitioning of channels in the classic spectral Transformer. During the partitioning process, the overlapping areas between adjacent windows are preserved to achieve a smoother region representation and promote information sharing between windows.

[0149] (3) The window is reshaped to a shape of C×HW / M 2 The size of the token is used to calculate the self-attention of each window using the following formula:

[0150]

[0151] In the formula, j represents the j-th window. The output of the attention is reshaped to a size of C×H / M×W / M. Then, the windows are merged in their original arrangement and convolutional projection is performed to obtain the output feature X of the window spectral self-attention block. WSSA-out .

[0152] Please see Figure 2 In a specific embodiment of the present invention, the expression of the high-resolution image reconstruction module is as follows:

[0153] I SR =Conv(Upsampling(X) GU-out )+Conv(I LR ↑)),

[0154] In the formula, X GU-outUpsampling is up-sampling, and Conv is convolution. The spatial size of the output feature X of the global feature extraction module is recovered to match the SR image through up-sampling, and the bicubic feature I GU-out is directly added to the tail of the network to better match the SR feature, and then the convolution layer adjusts the number of channels to C to obtain the final output SR image I LR . SR .

[0155] It should be noted that the step division of the above various methods is only for the purpose of clear description, and can be combined into one step or some steps can be split and decomposed into multiple steps as long as the same logical relationship is included, and all are within the protection scope of the present application. Adding irrelevant modifications or introducing irrelevant designs in the algorithm or process does not change the core design of the algorithm and process, and is within the protection scope of the patent.

[0156] To evaluate the super-resolution performance of the super-resolution method of the present application, the present application uses three evaluation indexes of Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM) and Spectral Angle Mapper (SAM) to evaluate the super-resolution results. Among them, the comparison is carried out with other six hyperspectral image enhancement models (MCNet, SSPSR, RFSR, GELIN, MSDformer and SRDNet) on different data sets and different super-resolution multiples.

[0157] Table 1: Comparison on ×4 super-resolution size on Cave data set.

[0158] Method Scale PSNR↑ SSIM↑ SAM↓ MCNet ×4 38.4765 0.9747 3.0438 SSPSR ×4 38.5009 0.9748 3.0951 RFSR ×4 38.6181 0.9752 2.9548 GELIN ×4 38.5629 0.9751 3.0329 MSDformer ×4 38.4123 0.9744 3.1036 SRDNet ×4 38.7071 0.9753 2.9856 Invention ×4 38.8008 0.9761 2.9497 MCNet ×8 32.4231 0.9289 4.3065 SSPSR ×8 32.4634 0.9278 4.4105 RFSR ×8 32.5879 0.9315 4.2496 GELIN ×8 32.4635 0.9279 4.4362 MSDformer ×8 32.4055 0.9245 4.3685 SRDNet ×8 32.7278 0.9321 4.1046 Invention ×8 32.7899 0.9324 4.0876

[0159] Table 2: Comparison of various methods on Chikusei data set.

[0160]

[0161]

[0162] From Table 1 and Table 2, it can be seen that no matter which data set, whether it is ×4 or ×8 super-resolution size, the effect of the present application is better than the six hyperspectral image enhancement models in the prior art, which can reflect the advantage of our model in the large channel hyperspectral image super-resolution task.

[0163] In summary, the dual-dimensional aggregation interaction network in the application combines the spectral interaction block with the mask space self-attention block, and simultaneously combines the spatial interaction block with the window spectral self-attention block, to realize cross-scale feature aggregation within the spatial dimension and within the spectral dimension. In addition, the application also proposes the spatial interaction block and the spectral interaction block, to promote the interaction of spatial-spectral dependence through the neighborhood expansion mechanism and with the aid of cross-dimension collaborative guidance. Finally, image reconstruction is performed through the high-resolution image reconstruction module, and excellent performance is achieved in various super-resolution tasks.

[0164] The above embodiments only exemplarily illustrate the principles and effects of the application, and are not intended to limit the application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical ideas disclosed by the application should be covered by the claims of the application.

Claims

1. A method for image enhancement based on spatial spectral aggregation and interaction, characterized in that, The method comprises the following steps: acquiring a first hyperspectral image to be processed; processing the first hyperspectral image by using a trained dual-dimensional aggregated interactive network to obtain a second hyperspectral image, the resolution of the second hyperspectral image being higher than that of the first hyperspectral image; wherein the expression of the dual-dimensional aggregated interactive network is as follows: I SR = H RU (H GU (H SU (I LR )), I LR ↑), In the formula, I LR is the first hyperspectral image, I SR is the second hyperspectral image; H SU and H GU are a group feature extraction module and a global feature extraction module, respectively, to perform multi-scale aggregation within dimensions and inter-dimensional interaction; I LR ↑ indicates bicubic interpolation up-sampling on the first hyperspectral image; H RU is a high-resolution image reconstruction module.

2. The image enhancement method based on spatial-spectral aggregation and interaction according to claim 1, characterized in that, the group feature extraction module comprises G dual-extraction interactive units and a convolution layer; the group feature extraction module processes the first hyperspectral image according to the following steps: dividing the first hyperspectral image into G groups along the channel dimension; extracting the features of the G groups by using the G dual-extraction interactive units respectively; splicing the features extracted by all the G dual-extraction interactive units; mapping the spliced features to a higher dimension by using the convolution layer to obtain the output features of the group feature extraction module.

3. The image enhancement method based on spatial-spectral aggregation and interaction according to claim 2, characterized in that, the expression of the dual-extraction interactive unit is as follows: F spa = SPM(X DEIB-in ) · SpeIM(SWM(X DEIB-in )) F spe = SWM(X DEIB-in ) · SpaIM(SPM(X DEIB-in )), X DEIB-out = Conv(F spa + F spe ), In the formula, X DEIB-in and X DEIB-out are input features and output features of the double-extraction interaction unit, respectively. SPM is a spatial pyramid block, SWM is a spectral weight block, SpeIM is a spectral interaction block, SpaIM is a spatial interaction block, F spa and F spe is an intermediate feature.

4. The image enhancement method based on spatial-spectral aggregation and interaction according to claim 3, characterized in that, the expression of the spatial pyramid block is as follows: X1 = ReLU(DConv(X SPM-in )), X2 = ReLU(DConv(X1 SPM-in )) X2 = ReLU(DConv(Downsampling(X1))), X3 = ReLU(Conv(Downsampling(X2))), X SPM-out = Conv(Upsampling(Conv(Upsampling(X3) + X2) + X1), wherein X SPM-in and X SPM-out are input and output features of the spatial pyramid block, respectively; DConv is a depth separable convolution, ReLU is an activation function, Conv is a convolution, Downsampling is down-sampling, Upsampling is up-sampling, and X1-X3 are intermediate features.

5. The method of image enhancement based on spatial-spectral aggregation and interaction according to claim 3, characterized in that, The spectral weight block processes the input feature X according to the following steps: SWM-in X = X - Xmean For the input feature X of size C×H×W SWM-in After performing a convolution operation to obtain features with unchanged size, a reshaping operation is performed to obtain the first feature with size C×HW. Convolution operation is performed on the input feature X SWM-in , and after the channel number is compressed to 1, reshaping operation is performed to obtain a second feature with a size of HWx1. performing matrix multiplication on the first feature and the second feature, and performing a reshaping operation again to obtain a third feature X4 with a size of C×1×1; The input features X are processed according to the following equation: SWM-in and the third feature X4: X SWM-out = (ReLU(X SWM-in ) · ReLU(Sigmoid(X4)), In the formula, X SWM-out is the output feature of the spectral weight block.

6. The image enhancement method based on spatial-spectral aggregation and interaction according to claim 3, characterized in that, the expression of the spatial interaction block is as follows: X5 = GeLU(Conv(X SpaIM-in )), X5 = GeLU(Conv(X SpaIM-in )), X5 X6 = AxialShift(X5) + X5, X SpaIM-out = Sigmoid(Conv(X6)), wherein X SpaIM-in and X SpaIM-out are input and output features of the spatial interaction block, respectively, GeLU is an activation function, AxialShift is an axial shift, and X5 and X6 are intermediate features.

7. The image enhancement method based on spatial-spectral aggregation and interaction according to claim 3, characterized in that, the expression of the spectral interaction block is as follows: X7 = AvgPool(X SpeIM-in ), X8 = GeLU(Conv(X7)) + ChannelMix(GeLU(GConv(X7))), X SpeIM-out = Sigmoid(Conv(X8)), wherein X SpeIM-in and X SpeIM-out are input and output features of the spectral interaction block, respectively, AvgPool is an average pooling, ChannelMix is a channel shuffle, GConv is a group convolution, and X7 and X8 are intermediate features.

8. The image enhancement method based on spatial-spectral aggregation and interaction according to claim 1, characterized in that, the expression of the global feature extraction module is as follows: X GU-out = H DITB (X GU-in )+ X GU-in , In the formula, X GU-in and X GU-out are input features and output features of the global feature extraction module respectively, H DITB is a double-interaction Transformer unit.

9. The image enhancement method based on spatial-spectral aggregation and interaction according to claim 8, characterized in that, the expression of the dual-interactive Transformer unit is as follows: Z spa = MSSA(LN(X DITB-in )) · SpeIM(LN(X DITB-in )), Z spe = WSSA(LN(X DITB-in )) · SpaIM(LN(X DITB-in )), Z' = (Z spa + Z spe + X DITB-in , X DITB-out = MLP(LN(Z')) + Z', In the formula, X DITB-in and X DITB-out are input features and output features of the double interaction Transformer unit, respectively. MSSA is a mask spatial self-attention block, WSSA is a window spectral self-attention block, SpeIM is a spectral interaction block, SpaIM is a spatial interaction block, LN is a layer normalization, and MLP is a multi-layer perception.

10. The image enhancement method based on spatial-spectral aggregation and interaction according to claim 1, characterized in that, the expression of the high-resolution image reconstruction module is as follows: I SR = Conv(Upsampling(X GU-out )+ Conv(I LR ↑)), In the formula, X GU-out is the output feature of the global feature extraction module, Upsampling is up sampling, and Conv is convolution.

Images