Remote-sensing hyperspectral image super-resolution reconstruction method based on hypergraph neural network

Abstract

The present invention relates to the technical field of image restoration, and in particular to a remote-sensing hyperspectral image super-resolution reconstruction method based on a hypergraph neural network, comprising: S1, acquiring a hyperspectral image, and preprocessing the hyperspectral image to obtain a training set and a verification set; S2, constructing a hypergraph neural network; S3, using training images in the training set to construct a three-layer hypergraph, and on the basis of the three-layer hypergraph, constructing hypergraph operators corresponding to the training images; S4, using the training images and the hypergraph operators corresponding to the training images to train the hypergraph neural network, and using a loss function to iterate network parameters of the hypergraph neural network, to obtain a trained hypergraph neural network; and S5, inputting, to the trained hypergraph neural network, low-resolution hyperspectral images to be reconstructed for reconstruction to obtain a high-resolution hyperspectral image. The present invention has an excellent reconstruction result.

Description

Remote sensing hyperspectral image super-resolution reconstruction method based on hypergraph neural network

[0001] The present application claims priority to the Chinese patent application No.CN202410527023.1, filed on April 29, 2024, and entitled "Remote sensing hyperspectral image super-resolution reconstruction method based on hypergraph neural network", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of image restoration, and in particular to a remote sensing hyperspectral image super-resolution reconstruction method based on hypergraph neural network. BACKGROUND

[0003] The research of hyperspectral image super-resolution technology can be traced back to the 20th century. Early work was mainly achieved through traditional algorithms, including wavelet transformation, maximum a posteriori estimation and spectral mixture analysis. The weakness of traditional algorithms is that they cannot introduce external prior information and often involve difficult problems to solve. After that, the emergence of convolutional neural networks gave the model the ability to learn independently. Although the development of natural image super-resolution technology based on deep learning has entered a relatively mature stage, the methods for hyperspectral images are still very limited. The existing super-resolution learning algorithms do not consider semantic information when extracting features and reconstructing, resulting in reconstructed images that cannot be well used for subsequent spectral interpretation work. SUMMARY

[0004] To solve the problem that the existing super-resolution learning algorithms do not consider semantic information when extracting features and reconstructing, resulting in reconstructed images that cannot be well used for subsequent spectral interpretation work, the present application provides a remote sensing hyperspectral image super-resolution reconstruction method based on hypergraph neural network, which has excellent reconstruction results.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] A hypergraph neural network system, the hypergraph neural network system comprising a first residual block, a first hypergraph attention module, a first reconstruction module, a first 3x3 convolution layer, a ReLU module, a second 3x3 convolution layer, a first processing unit, a second processing unit, a third processing unit, a fourth processing unit and a fifth processing unit, each processing unit in the first processing unit to the fifth processing unit comprising a first output, a second output and a third output;

[0007] The low-resolution remote sensing hyperspectral image A input to the hypergraph neural network system is processed by the first processing unit to the fifth processing unit in sequence, the first output of the fifth processing unit is input to the first residual block for residual processing, and the feature map A1 is obtained, the second output of the fifth processing unit is processed by the first hypergraph attention module and the first 3*3 convolution layer in sequence, and the feature map A2 is obtained, the feature map A1 and the feature map A2 are added to obtain the feature map A3, the feature map A3 is input to the first reconstruction module for image reconstruction to obtain the feature map A4, the low-resolution remote sensing hyperspectral image A is subjected to bicubic interpolation operation to obtain the feature map A5, the feature map A4, the third output of each of the first processing unit to the fifth processing unit, and the feature map A5 are added to obtain the feature map A6, and the feature map A6 is processed by the second 3*3 convolution layer, the ReLU module and the second 3*3 convolution layer in sequence to realize the reconstruction of the low-resolution remote sensing hyperspectral image A.

[0008] Further, the second output of the fifth processing unit is processed by the first hypergraph attention module and the first 3*3 convolution layer in sequence, which includes first hypergraph learning processing by the first hypergraph attention module, and then convolution operation by the first 3*3 convolution layer on the result of the hypergraph learning processing to obtain the feature map A2.

[0009] Further, each processing unit in the first processing unit to the fifth processing unit includes a second residual block, a second hypergraph attention module, a multi-order spatial interaction module and a second reconstruction module, in each processing unit, the output features of the second residual block and the second hypergraph attention module are input to the multi-order spatial interaction module for feature fusion, and the feature fusion result is input to the second reconstruction module for feature reconstruction to obtain the third output of each processing unit.

[0010] Further, the second residual block includes a third 3*3 convolution layer and a ReLU module, the feature map B input to the second residual block is processed by the third 3*3 convolution layer, the ReLU module and the third 3*3 convolution layer in sequence to obtain the feature map B1, the feature map B1 and the feature map B are added to obtain the output feature of the second residual block, and the output feature of the second residual block is taken as the first output of each processing unit.

[0011] Further, the second hypergraph attention module includes an average pooling layer, a maximum pooling layer, a 1x1 convolution layer, a ReLU module and a Sigmoid module, performs a size resetting operation on the feature map B input to the second hypergraph attention module to obtain a feature map B2, copies the feature map B2 twice to obtain three layers of feature maps B2, inputs the three layers of feature maps B2 to the parallel average pooling layer and maximum pooling layer respectively for pooling processing to obtain an average pooling result and a maximum pooling result, performs an addition operation on the average pooling result and the maximum pooling result, and sequentially passes through the 1x1 convolution layer, the ReLU module, the 1x1 convolution layer and the Sigmoid module to sequentially perform 1x1 convolution and corresponding ReLU activation operation and 1x1 convolution and corresponding Sigmoid activation operation to obtain a distribution weight value corresponding to the three layers of feature maps B2, multiplies the distribution weight value and the hypergraph operator corresponding to the low-resolution remote sensing hyperspectral image A to obtain a weighted hypergraph operator, multiplies the weighted hypergraph operator, the three layers of feature maps B2 and the network parameters of the hypergraph neural network system to obtain three layers of feature maps respectively containing detail texture features, spectral features and position features, and takes the output feature of the second hypergraph attention module as the second output of each processing unit.

[0012] Further, the second hypergraph attention module includes an average pooling layer, a maximum pooling layer, a 1x1 convolution layer, a ReLU module and a Sigmoid module, performs a size resetting operation on the feature map B input to the second hypergraph attention module to obtain a feature map B2, copies the feature map B2 twice to obtain three layers of feature maps B2, inputs the three layers of feature maps B2 to the parallel average pooling layer and maximum pooling layer respectively for pooling processing to obtain an average pooling result and a maximum pooling result, performs an addition operation on the average pooling result and the maximum pooling result, and sequentially passes through the 1x1 convolution layer, the ReLU module, the 1x1 convolution layer and the Sigmoid module to sequentially perform 1x1 convolution and corresponding ReLU activation operation and 1x1 convolution and corresponding Sigmoid activation operation to obtain a distribution weight value corresponding to the three layers of feature maps B2, multiplies the distribution weight value and the hypergraph operator corresponding to the low-resolution remote sensing hyperspectral image A to obtain a weighted hypergraph operator, multiplies the weighted hypergraph operator, the three layers of feature maps B2 and the network parameters of the hypergraph neural network system to obtain three layers of feature maps respectively containing detail texture features, spectral features and position features, and takes the output feature of the second hypergraph attention module as the second output of each processing unit.

[0013] Further, the multi-order spatial interaction module includes a fourth 3x3 convolution layer and a fifth 3x3 convolution layer, inputs the three layers of feature maps respectively containing detail texture features, spectral features and position features to the multi-order spatial interaction module, and performs Hadamard product operation on the two layers of feature maps containing detail texture features and spectral features to obtain a feature map C1, inputs the feature map C1 to the fourth 3x3 convolution layer for convolution operation to obtain a feature map C2, performs Hadamard product operation on the feature map C2 and the feature map containing position features to obtain a feature map C3, inputs the feature map C3 to the fifth 3x3 convolution layer for convolution operation to obtain a feature map C4, and performs Hadamard product operation on the feature map C4 and the output feature of the second residual block to obtain a fusion feature map.

[0014] Further, the second reconstruction module includes a sixth 3x3 convolution layer, a ReLU module and a transposed convolution layer, sequentially processes the fusion feature map through the sixth 3x3 convolution layer, the ReLU module and the transposed convolution layer, and obtains the third output of each processing unit through convolution operation, ReLU activation operation and transposed convolution operation.

[0015] A remote sensing hyperspectral image super-resolution reconstruction method based on a hypergraph neural network system is realized by using a hypergraph neural network system, and specifically includes the following steps:

[0016] S1: Obtain a remote sensing hyperspectral image, and pre-process the remote sensing hyperspectral image to obtain a training set and a verification set;

[0017] S2: Construct a hypergraph neural network system, construct a three-layer hypergraph using a training image in the training set, and construct a hypergraph operator corresponding to the training image based on the three-layer hypergraph;

[0018] S3: Train the hypergraph neural network system using the training image and the hypergraph operator corresponding to the training image to obtain a trained hypergraph neural network system;

[0019] S4: input a low-resolution remote sensing hyperspectral image to be reconstructed into the trained hypergraph neural network system for reconstruction to obtain a high-resolution remote sensing hyperspectral image.

[0020] Further, step S1 specifically includes the following steps:

[0021] S11: Divide the hyperspectral image into a training set and a verification set according to a ratio of 7:3;

[0022] S12: Randomly select n image blocks in the training set and the verification set respectively, and sequentially perform enlargement, random horizontal flipping, rotation at different angles, and scaling at different magnifications on the image blocks;

[0023] S13: Perform bicubic downsampling processing on the processed image blocks so that all the image blocks are 32x32 low-resolution remote sensing hyperspectral images.

[0024] Further, in step S2, the specific steps of constructing a three-layer hypergraph using a training image in the training set include:

[0025] S21: Select target nodes on the training image in sequence, and calculate the LBP values of each target node and eight adjacent pixels, select ten nodes with the closest LBP value Hamming distance to the target node to construct a hyperedge;

[0026] S22: According to the Hamming distance of the LBP values of the ten nodes and the target node, assign weights to the ten nodes by the following formula to obtain a hypergraph containing LBP texture features:

[0027] (1) ;

[0028] wherein, is the weight of the i-th node of the hypergraph containing LBP texture features, and are the Hamming distance and the average Hamming distance of the LBP values of the target node and the i-th node, respectively;

[0029] S23: calculate the Euclidean distance of the spectral signal of the target node and other nodes, and select the ten nodes with the closest Euclidean distance to the target node to construct a hyperedge;

[0030] S24: according to the Euclidean distance of the ten nodes and the target node, assign weights to the ten nodes by the following formula to obtain a hypergraph containing spectral clustering:

[0031] (2) ;

[0032] wherein, is the weight of the i-th node of the hypergraph containing spectral clustering, and are the Euclidean distance and the average Euclidean distance between the target node and the i-th node, respectively;

[0033] S25: construct a hypergraph containing position relationship by the following formula:

[0034] (3) ;

[0035] (4) ;

[0036] (5) ;

[0037] wherein, and are three types of hyperedges in the hypergraph containing position relationship;

[0038] S26: splice the hypergraph containing LBP texture features, the hypergraph containing spectral clustering and the hypergraph containing position relationship into a three-layer hypergraph containing LBP texture features, spectral clustering and position relationship.

[0039] Further, the network parameters of the hypergraph neural network system are iterated by using a loss function.

[0040] Further, in step S2, the three-layer hypergraph is converted into a hypergraph operator G corresponding to the training image by the following formula:

[0041] (6) ;

[0042] wherein, and are the diagonal matrix of the degree of the hyperedge and the target node, respectively, W is the initialized unit matrix, is the three-layer hypergraph.

[0043] Further, step S3 specifically includes the following steps:

[0044] S31: input the training image and the hypergraph operator corresponding to the training image into the hypergraph neural network system;

[0045] S32: calculate the training image input into the second residual block by the following formula to obtain the first output of the first processing unit :

[0046] (7) ;

[0047] wherein, is a residual operation, is the training image;

[0048] S33: calculate the training image input into the second hypergraph attention module by the following formula to obtain the second output of the first processing unit :

[0049] (8) ;

[0050] wherein, is a hypergraph learning operation, is a hypergraph operator;

[0051] S34: perform feature fusion on the first output and the second output input into the multi-order spatial interaction module by the following formula to obtain the fusion feature map :

[0052] (9) ;

[0053] wherein, is a feature fusion operation;

[0054] S35: process the fusion feature map input into the second reconstruction module by the following formula to obtain the third output of the first processing unit :

[0055] (10) ;

[0056] wherein, is a reconstruction operation;

[0057] S36: repeat steps S32-S35, and perform bicubic interpolation operation on the training image to obtain the reconstructed image by the following formula :

[0058] (11) ;

[0059] wherein, For the convolution operation, BIC is a bicubic interpolation operation, I 1 -I 5 are the third outputs of the five processing units, respectively, I 6 is a feature map A4;

[0060] S37: update the network parameters of the hypergraph neural network system using the loss function, and after completing at least 25000 times of network parameter updates, test the network performance of the hypergraph neural network system using the validation set until the results of the objective image quality evaluation indicators of the reconstructed image no longer rise or reach the preset training round.

[0061] Further, the loss function is composed of L1 loss, SAM loss and gradient loss, and the weight distribution of L1 loss, SAM loss and gradient loss is 1:0.3:0.1; the evaluation indicators include PSNR, SSIM and SAM, and the preset training round is not less than 50 times.

[0062] Further, step S33 specifically includes the following steps:

[0063] S331: performing a size resetting operation on the training image and copying it twice to obtain three layers of feature maps;

[0064] S332: adding the pooling results obtained by respectively performing maximum pooling and average pooling processing on the three layers of feature maps through the following formula:

[0065] (12);

[0066] wherein, is a weight value, is the (n-1)th layer of three-layer feature maps in the hypergraph neural network system, MP and AP are average pooling processing and maximum pooling processing, respectively;

[0067] S333: calculating the distribution weight value obtained by sequentially performing convolution operation on the weight value through a 1x1 convolution layer, a ReLU module and a 1x1 convolution layer through the following formula:

[0068] (13);

[0069] wherein, is a convolution operation;

[0070] S334: multiplying the distribution weight value and the hypergraph operator corresponding to the training image to obtain a weighted hypergraph operator :

[0071] (14);​

[0072] wherein, is a real set, N is the number of all nodes, G is a hypergraph operator corresponding to the training image;

[0073] S335: multiplying the weighted hypergraph operator, the three-layer feature map and the network parameter by the following formula to obtain a three-layer feature map containing detailed texture features, spectral features and position features:

[0074] (15);

[0075] wherein, is a three-layer feature map containing detailed texture features, spectral features and position features, and θ is a network parameter.

[0076] Further, the step S35 specifically comprises the following steps:

[0077] S351: performing feature fusion on the two-layer feature map containing detailed texture features and spectral features by the following formula, and feeding the fusion result into a fourth 3x3 convolution layer for convolution operation to obtain a feature fusion map :

[0078] (16);

[0079] wherein, is a Hadamard product operation, is a feature map containing detailed texture features, is a feature map containing spectral features, C is the number of channels of the hyperspectral image, W is the width of the hyperspectral image, and H is the height of the hyperspectral image;

[0080] S352: performing feature fusion on the feature fusion map and a feature map containing position features, and feeding the fusion result into a fifth 3x3 convolution layer for convolution operation to obtain a feature fusion map :

[0081] (17);

[0082] wherein, is a feature map containing position features;

[0083] S353: performing cross-domain interaction on the feature fusion map and the output feature of the residual block to obtain a fusion feature map containing local information and global information :

[0084] (18);

[0085] wherein, is the feature map output by the residual block.

[0086] Compared with the prior art, the above technical solution has the following advantages:

[0087] The application constructs a three-layer supergraph based on LBP texture features, spectral clustering and position features, and designs a supergraph neural network with attention mechanism, realizes semantic information transmission in the true sense in the super-resolution field, and provides strong support for subsequent spectral interpretation work.

[0088] The application constructs a multi-order spatial interaction module, realizes cross guidance and fusion of different types of features in the supergraph neural network by calculating the Hadamard product between different source features, and the cross-domain feature fusion can provide a more complete and convincing basis for image reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0089] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0090] Fig. 1 is a flowchart of a remote sensing hyperspectral image super-resolution reconstruction method based on a supergraph neural network according to an embodiment of the application;

[0091] Fig. 2 is a network structure diagram of a supergraph neural network according to an embodiment of the application;

[0092] Fig. 3 is a network structure diagram of a second supergraph attention module according to an embodiment of the application;

[0093] Fig. 4(a) is a supergraph based on LBP texture features according to an embodiment of the application;

[0094] Fig. 4(b) is a supergraph based on spectral clustering according to an embodiment of the application;

[0095] Fig. 4(c) is a supergraph based on position relationship according to an embodiment of the application;

[0096] Fig. 5(a) shows a super-resolution reconstruction diagram of a remote sensing hyperspectral image super-resolution reconstruction method based on a supergraph neural network on a MDAS data set according to an embodiment of the application;

[0097] FIG. 5(b) shows a super-resolution reconstruction diagram of the remote sensing hyperspectral image super-resolution reconstruction method based on the supergraph neural network according to the embodiment of the present application on the Chikusei dataset;

[0098] FIG. 5(c) shows a super-resolution reconstruction diagram of the remote sensing hyperspectral image super-resolution reconstruction method based on the supergraph neural network according to the embodiment of the present application on the Pavia Centre dataset. DETAILED DESCRIPTION

[0099] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0100] In the following description, specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in a variety of ways other than those described herein without departing from the spirit and scope of the present application, which can be practiced by persons skilled in the art without departing from the spirit and scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0101] FIG. 1 shows the flow of the remote sensing hyperspectral image super-resolution reconstruction method based on the supergraph neural network according to the embodiment of the present application.

[0102] As shown in FIG. 1, the remote sensing hyperspectral image super-resolution reconstruction method based on the supergraph neural network according to the present application specifically includes the following steps:

[0103] S1: Obtain a hyperspectral image and pre-process the hyperspectral image to obtain a training set and a validation set.

[0104] Step S1 specifically includes the following steps:

[0105] S11: Divide the hyperspectral image into a training set and a validation set according to a ratio of 7:3;

[0106] S12: Randomly select n image blocks in the training set and the validation set, respectively, and sequentially perform enlargement, random horizontal flipping, rotation at different angles, and scaling at different magnifications on the image blocks;

[0107] S13: Perform bicubic down-sampling processing on the processed image blocks so that all the image blocks are low-resolution hyperspectral images of 32x32.

[0108] S2: Construct a supergraph neural network.

[0109] S3: Construct a three-layer supergraph using the training images in the training set, and construct a supergraph operator corresponding to the training images based on the three-layer supergraph.

[0110] Step S3 specifically includes the following steps:

[0111] S31: Select the target node on the training image in turn, and calculate the LBP value of each target node and the adjacent eight pixels. Ten nodes with the closest Hamming distance to the LBP value of the target node are selected to construct a hyperedge;

[0112] S32: According to the Hamming distance of the LBP value of the ten nodes and the target node, the weight of the ten nodes is assigned by the following formula, and the hypergraph containing the LBP texture feature is obtained:

[0113] (1);

[0114] wherein, is the weight of the i-th node of the hypergraph containing the LBP texture feature, and are the Hamming distance and the average Hamming distance of the LBP value of the target node and the i-th node, respectively;

[0115] S33: Calculate the Euclidean distance of the spectral signal of the target node and other nodes, and select ten nodes with the closest Euclidean distance to the target node to construct a hyperedge;

[0116] S34: According to the Euclidean distance of the ten nodes and the target node, the weight of the ten nodes is assigned by the following formula, and the hypergraph containing the spectral clustering is obtained:

[0117] (2);

[0118] wherein, is the weight of the i-th node of the hypergraph containing the spectral clustering, and are the Euclidean distance and the average Euclidean distance of the target node and the i-th node, respectively;

[0119] S35: The hypergraph containing the position relationship is constructed by the following formula:

[0120] (3);

[0121] (4);

[0122] (5);

[0123] wherein, and are three types of hyperedges in the hypergraph containing the position relationship;

[0124] S36: The hypergraph containing the LBP texture feature, the hypergraph containing the spectral clustering, and the hypergraph containing the position relationship are spliced into a three-layer hypergraph containing the LBP texture feature, the spectral clustering, and the position relationship;

[0125] S37: convert the three-layer hypergraph into a hypergraph operator G corresponding to the training image by the following formula:

[0126] (6);

[0127] wherein, and are diagonal matrices of the degrees of the hyperedges and the target nodes respectively, W is an initialized unit matrix, is a three-layer hypergraph.

[0128] S4: train the hypergraph neural network using the training image and the hypergraph operator corresponding to the training image, and iteratively update the network parameters of the hypergraph neural network using a loss function to obtain a trained hypergraph neural network.

[0129] Step S4 specifically includes the following steps:

[0130] S41: input the training image and the hypergraph operator corresponding to the training image into the hypergraph neural network.

[0131] S42: calculate the training image input into the second residual block by the following formula to obtain the first output of the processing unit :

[0132] (7);

[0133] wherein, is a residual operation, is the training image.

[0134] S43: calculate the training image input into the second hypergraph attention module by the following formula to obtain the second output of the processing unit :

[0135] (8);

[0136] wherein, is a hypergraph learning operation, is the hypergraph operator.

[0137] Step S43 specifically includes the following steps:

[0138] S431: perform a size resetting operation on the training image and copy it twice to obtain a three-layer feature map;

[0139] S432: add the pooling results obtained by performing maximum pooling and average pooling on the three-layer feature map by the following formula:

[0140] (9);

[0141] wherein, is a weight value, is a three-layer feature map of the (n-1)-th layer in the hypergraph neural network, MP and AP are average pooling processing and maximum pooling processing, respectively;

[0142] S433: the assigned weight value is obtained by performing convolution operation on the weight value through two cascaded 1×1 convolution layers :

[0143] (10);

[0144] wherein, is a convolution operation;

[0145] S434: the weighted hypergraph operator is obtained by multiplying the assigned weight value and the hypergraph operator corresponding to the training image through the following formula :

[0146] (11);

[0147] wherein, is a real set, and N is the number of all nodes;

[0148] S435: the three-layer feature map containing the detailed texture feature, the spectral feature and the position feature is obtained by multiplying the weighted hypergraph operator, the three-layer feature map and the network parameter through the following formula:

[0149] (12);

[0150] wherein, is a three-layer feature map containing the detailed texture feature, the spectral feature and the position feature, and θ is a network parameter.

[0151] S44: the first output and the second output input into the multi-order spatial interaction module are fused through a feature fusion operation to obtain a fused feature map :

[0152] (13);

[0153] wherein, is a feature fusion operation.

[0154] The step S44 specifically comprises the following steps:

[0155] S441: the two-layer feature map containing the detailed texture feature and the spectral feature is fused through the following formula, and the fusion result is fed into a fourth 3×3 convolution layer to perform convolution operation to obtain a feature fusion map :

[0156] (15);

[0157] in, For the Hadamard product calculation, This is a feature map that contains detailed texture features. The hyperspectral image contains spectral features, where C is the number of channels, W is the width, and H is the height.

[0158] S442: Fusing feature maps The feature map containing location features is fused, and the fused result is fed into the fifth 3×3 convolutional layer for convolution to obtain the feature fusion map. :

[0159] (16);

[0160] in, A feature map containing location features;

[0161] S443: Fusing feature maps Cross-domain interaction is performed with the output features of the residual block to obtain a fused feature map containing both local and global information. :

[0162] (17);

[0163] in, This is the feature map output by the residual block, where 2C×H×W is the dimension of the feature map.

[0164] S45: The fused feature map input to the second reconstruction module is processed by the following formula. Processing is performed to obtain the third output of the processing unit. :

[0165] (14);

[0166] in, For reconstruction operation.

[0167] S46: Repeat steps S42-S45, and perform bicubic interpolation on the training image to obtain the reconstructed image using the following formula. :

[0168] (18);

[0169] in, BIC is a bicubic interpolation operation, I 1 -I 5 is the third output of the five processing units, I 6 is the feature map A4.

[0170] S47: update the network parameters of the hypergraph neural network using the loss function, and after completing at least 25,000 updates of the network parameters, test the network performance of the hypergraph neural network using the validation set until the result of the objective image quality evaluation index of the reconstructed image no longer rises or reaches the preset training round.

[0171] The loss function is composed of L1 loss, SAM loss and gradient loss, and the weight distribution of L1 loss, SAM loss and gradient loss is 1:0.3:0.1.

[0172] The evaluation index includes peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and spectral angle mapping (SAM), and the preset training round is not less than 50 times.

[0173] L1 loss, SAM loss and gradient loss are all prior art.

[0174] During the training process, the optimizer selects Adam( and ), the batch size is 4, the initial learning rate is 0.0001, and the training round is set to 50.

[0175] S5: input the low-resolution hyperspectral image to be reconstructed into the trained hypergraph neural network for reconstruction to obtain a high-resolution hyperspectral image.

[0176] Fig. 2 shows a network structure diagram of a hypergraph neural network according to the present embodiment.

[0177] As shown in FIG. 2, the supergraph neural network comprises five cascaded processing units, a first residual block, a first supergraph attention module, a first reconstruction module, a first 3x3 convolution layer and a second 3x3 convolution layer, the processing unit comprises a first output, a second output and a third output, wherein the low-resolution hyperspectral image A input to the supergraph neural network is processed by the image processing of the five cascaded processing units, the first output of the fifth processing unit is input to the first residual block for residual processing to obtain a feature map A1, the second output of the fifth processing unit is sequentially input to the first supergraph attention module and the first 3x3 convolution layer for supergraph learning and convolution operation to obtain a feature map A2, the feature map A1 and the feature map A2 are added to obtain a feature map A3, the feature map A3 is input to the first reconstruction module for image reconstruction to obtain a feature map A4, the feature map A4, the third output of each processing unit and a feature map A5 obtained by bicubic interpolation operation on the low-resolution hyperspectral image A are added to obtain a feature map A6, and the feature map A6 is input to the two cascaded second 3x3 convolution layers for twice convolution and once ReLU activation operation, thereby realizing the reconstruction of the low-resolution hyperspectral image A.

[0178] The processing unit comprises a second residual block, a second supergraph attention module, a multi-order spatial interaction module and a second reconstruction module, the output features of the second residual block and the second supergraph attention module are both input to the multi-order spatial interaction module for feature fusion, and the feature fusion result is input to the second reconstruction module for feature reconstruction to obtain the third output of the processing unit.

[0179] The second residual block is composed of two cascaded third 3x3 convolution layers, the feature map B input to the second residual block is processed by twice convolution and once ReLU activation operation of the two cascaded third 3x3 convolution layers to obtain a feature map B1, and the feature map B1 and the feature map B are added to obtain the output feature of the second residual block.

[0180] FIG. 3 shows a network structure schematic diagram of the second supergraph attention module according to the present embodiment.

[0181] As shown in FIG. 3, the second supergraph attention module includes an average pooling layer, a maximum pooling layer, and two cascaded 1x1 convolution layers, performs a size resetting operation on the feature map B input to the second supergraph attention module to obtain a feature map B2, copies the feature map B2 twice to obtain three layers of feature maps B2, inputs the three layers of feature maps B2 to the average pooling layer and the maximum pooling layer in parallel for pooling processing to obtain an average pooling result and a maximum pooling result, performs an addition operation on the average pooling result and the maximum pooling result, and then sequentially performs a 1x1 convolution and a corresponding ReLU activation operation and a 1x1 convolution and a corresponding Sigmoid activation operation on the two cascaded 1x1 convolution layers to obtain a distribution weight value corresponding to the three layers of feature maps B2, multiplies the distribution weight value and the supergraph operator corresponding to the low-resolution hyperspectral image A to obtain a weighted supergraph operator, and multiplies the weighted supergraph operator, the three layers of feature maps B2, and network parameters to obtain three layers of feature maps respectively containing detail texture features, spectral features, and position features.

[0182] The multi-order spatial interaction module includes a fourth 3x3 convolution layer and a fifth 3x3 convolution layer, inputs the three layers of feature maps to the multi-order spatial interaction module, and performs a Hadamard product operation on the two layers of feature maps containing the detail texture features and the spectral features to obtain a feature map C1, inputs the feature map C1 to the fourth 3x3 convolution layer for convolution operation to obtain a feature map C2, performs a Hadamard product operation on the feature map C2 and the feature map containing the position features to obtain a feature map C3, inputs the feature map C3 to the fifth 3x3 convolution layer for convolution operation to obtain a feature map C4, and performs a Hadamard product operation on the feature map C4 and the output feature of the second residual block to obtain a fusion feature map.

[0183] The second reconstruction module includes a cascaded sixth 3x3 convolution layer and a transposed convolution layer, inputs the fusion feature map to the cascaded sixth 3x3 convolution layer and the transposed convolution layer, and performs convolution operation, transposed convolution operation, and a corresponding ReLU activation operation on the fusion feature map to obtain a third output of the processing unit.

[0184] In the processing unit, the output feature of the second residual block is taken as a first output of the processing unit, and the output feature of the second supergraph attention module is taken as a second output of the processing unit.

[0185] The bicubic interpolation operation, the ReLU activation operation, the Sigmoid activation operation, and the Hadamard product operation are all prior art.

[0186] The construction of the hypergraph is a key step that affects the performance of the hypergraph neural network. The foundation of the hypergraph construction is the hyperedge. The relationship between the nodes in the hyperedge is the prior information applied to the hypergraph neural network. The application of what kind of information and the intensity of the application will greatly affect the learning process of the network. FIG. 4(a)-(c) respectively show three-layer hypergraphs based on LBP texture features, spectral clustering, and location relationships according to the hypergraph neural network provided in the present embodiment. The hypergraph based on LBP texture features, spectral clustering, and location relationships in the present application is spliced into a three-layer hypergraph.

[0187] FIG. 5(a)-(c) shows the hyper-resolution reconstruction of the remote sensing hyperspectral image super-resolution reconstruction method based on the hypergraph neural network according to the present embodiment on the MDAS, Chikusei, and Pavia Centre data sets.

[0188] As shown in FIG. 5(a)-(c), the remote sensing hyperspectral image super-resolution reconstruction method based on the hypergraph neural network provided in the present embodiment can achieve good super-resolution reconstruction results on the three data sets.

[0189] It should also be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0190] The various embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be mutually referred to.

[0191] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hypergraph neural network system, comprising: The hypergraph neural network system comprises a first residual block, a first hypergraph attention module, a first reconstruction module, a first 3*3 convolution layer, a ReLU module, a second 3*3 convolution layer, a first processing unit, a second processing unit, a third processing unit, a fourth processing unit and a fifth processing unit, each processing unit in the first processing unit to the fifth processing unit comprises a first output, a second output and a third output; After the low-resolution remote sensing hyperspectral image A input into the hypergraph neural network system is processed by the first processing unit to the fifth processing unit in turn, the first output of the fifth processing unit is input into the first residual block for residual processing to obtain a feature map A1, the second output of the fifth processing unit is processed by the first hypergraph attention module and the first 3*3 convolution layer in turn to obtain a feature map A2, the feature map A1 and the feature map A2 are added to obtain a feature map A3, the feature map A3 is input into the first reconstruction module for image reconstruction to obtain a feature map A4, the low-resolution remote sensing hyperspectral image A is subjected to a bicubic interpolation operation to obtain a feature map A5, the feature map A4, the third output of each of the first processing unit to the fifth processing unit and the feature map A5 are added to obtain a feature map A6, and the feature map A6 is subjected to twice convolution and once ReLU activation operation by the second 3*3 convolution layer, the ReLU module and the second 3*3 convolution layer in turn to realize reconstruction of the low-resolution remote sensing hyperspectral image A.

2. The hypergraph neural network system of claim 1, wherein, The second output of the fifth processing unit is processed by the first hypergraph attention module and the first 3*3 convolution layer in turn, which comprises hypergraph learning processing by the first hypergraph attention module first, and then convolution operation by the first 3*3 convolution layer on the result of the hypergraph learning processing to obtain the feature map A2.

3. The hypergraph neural network system of claim 1, wherein, Each processing unit in the first processing unit to the fifth processing unit comprises a second residual block, a second hypergraph attention module, a multi-order spatial interaction module and a second reconstruction module, in each processing unit, the output features of the second residual block and the second hypergraph attention module are input into the multi-order spatial interaction module for feature fusion, and the feature fusion result is input into the second reconstruction module for feature reconstruction to obtain the third output of each processing unit.

4. The hypergraph neural network system of claim 3, wherein, The second residual block comprises a third 3*3 convolution layer and a ReLU module, a feature map B input into the second residual block is subjected to twice convolution and once ReLU activation operation by the third 3*3 convolution layer, the ReLU module and the third 3*3 convolution layer in turn to obtain a feature map B1, the feature map B1 and the feature map B are added to obtain the output feature of the second residual block, and the output feature of the second residual block is taken as the first output of each processing unit.

5. The hypergraph neural network system of claim 4, wherein, The second supergraph attention module includes an average pooling layer, a maximum pooling layer, a 1*1 convolution layer, a ReLU module and a Sigmoid module, performs a size resetting operation on a feature map B input to the second supergraph attention module, obtains a feature map B2, copies the feature map B2 twice to obtain three layers of feature maps B2, inputs the three layers of feature maps B2 into the average pooling layer and the maximum pooling layer in parallel respectively for pooling processing, obtains an average pooling result and a maximum pooling result, performs an adding operation on the average pooling result and the maximum pooling result, sequentially passes through a 1*1 convolution layer, a ReLU module, a 1*1 convolution layer and a Sigmoid module, sequentially performs 1*1 convolution and corresponding ReLU activation operation and 1*1 convolution and corresponding Sigmoid activation operation, obtains a distribution weight value corresponding to the three layers of feature maps B2, multiplies the distribution weight value and a supergraph operator corresponding to the low-resolution remote sensing hyperspectral image A, obtains a weighted supergraph operator, multiplies the weighted supergraph operator, the three layers of feature maps B2 and network parameters of a supergraph neural network system, obtains three layers of feature maps respectively containing detail texture features, spectral features and position features, and takes the output feature of the second supergraph attention module as a second output of each processing unit.

6. The hypergraph neural network system of claim 5, wherein, The multi-order spatial interaction module includes a fourth 3*3 convolution layer and a fifth 3*3 convolution layer, inputs the three layers of feature maps respectively containing detail texture features, spectral features and position features into the multi-order spatial interaction module, and performs Hadamard product operation on two layers of feature maps containing detail texture features and spectral features, to obtain a feature map C1, inputs the feature map C1 into the fourth 3*3 convolution layer for convolution operation, to obtain a feature map C2, performs Hadamard product operation on the feature map C2 and a feature map containing position features, to obtain a feature map C3, inputs the feature map C3 into the fifth 3*3 convolution layer for convolution operation, to obtain a feature map C4, and performs Hadamard product operation on the feature map C4 and the output feature of the second residual block, to obtain a fusion feature map.

7. The hypergraph neural network system of claim 6, wherein, The second reconstruction module includes a sixth 3*3 convolution layer, a ReLU module and a transposed convolution layer, sequentially processes the fusion feature map through the sixth 3*3 convolution layer, the ReLU module and the transposed convolution layer, and obtains a third output of each processing unit through convolution operation, ReLU activation operation and transposed convolution operation.

8. A remote sensing hyperspectral image super-resolution reconstruction method based on a hypergraph neural network system, implemented by using the hypergraph neural network system of any one of claims 1-7, characterized in that, Specifically includes the following steps: S1: obtaining a remote sensing hyperspectral image, and pre-processing the remote sensing hyperspectral image to obtain a training set and a validation set; S2: constructing a supergraph neural network system, constructing three layers of supergraphs by using training images in the training set, and constructing a supergraph operator corresponding to the training images based on the three layers of supergraphs; S3: training the supergraph neural network system by using the training images and the supergraph operator corresponding to the training images, to obtain a trained supergraph neural network system; S4: inputting the low-resolution remote sensing hyperspectral image to be reconstructed into the trained hypergraph neural network system for reconstruction to obtain a high-resolution remote sensing hyperspectral image.

9. The hypergraph neural network system-based remote sensing hyperspectral image super-resolution reconstruction method according to claim 8, characterized in that, The step S1 specifically comprises the following steps: S11: dividing the hyperspectral image into a training set and a validation set according to a ratio of 7:3; S12: randomly selecting n image blocks in the training set and the validation set respectively, and sequentially performing enlargement, random horizontal flipping, rotation at different angles, and scaling at different magnifications on the image blocks; S13: performing bicubic downsampling processing on the processed image blocks, so that all the image blocks are low-resolution remote sensing hyperspectral images with a size of 32x32.

10. The hypergraph neural network system-based remote sensing hyperspectral image super-resolution reconstruction method according to claim 8, characterized in that, In step S2, the specific steps for constructing a three-layer hypergraph using the training images in the training set include: S21: sequentially selecting a target node on the training image, and calculating the LBP values of the target node and eight adjacent pixels, and selecting ten nodes with the closest LBP value Hamming distance to the target node to construct a hyperedge; S22: assigning weights to the ten nodes according to the Hamming distance between the LBP values of the ten nodes and the target node by the following formula to obtain a hypergraph containing LBP texture features: （1）； wherein, weight of the i-th node of the supergraph comprising the LBP texture features, and respectively, the Hamming distance and the average Hamming distance between the LBP values of the target node and the i-th node; S23: calculating the Euclidean distance of the spectral signal between the target node and other nodes, and selecting ten nodes with the closest Euclidean distance to the target node to construct a hyperedge; S24: assigning weights to the ten nodes according to the Euclidean distance between the ten nodes and the target node by the following formula to obtain a hypergraph containing spectral clustering: （2）； wherein weight of the i-th node of the hypergraph comprising the spectral clustering, and respectively, the Euclidean distance and the average Euclidean distance between the target node and the i-th node; S25: constructing a hypergraph containing position relationships by the following formula: （3）； （4）； （5）； wherein and for the three types of hyperedges in the hypergraph containing position relationships; S26: splicing the hypergraph containing LBP texture features, the hypergraph containing spectral clustering, and the hypergraph containing position relationships into a three-layer hypergraph containing LBP texture features, spectral clustering, and position relationships.

11. The hypergraph neural network system-based remote sensing hyperspectral image super-resolution reconstruction method according to claim 8, characterized in that, Iterate the network parameters of the hypergraph neural network system using a loss function.

12. The hypergraph neural network system-based remote sensing hyperspectral image super-resolution reconstruction method according to claim 8, characterized in that, In step S2, convert the three-layer hypergraph into a hypergraph operator G corresponding to the training image by the following formula: （6）； wherein, and D1and D2are diagonal matrices of the degrees of the super-edge and target nodes, respectively, and W is an initialized identity matrix, for the three-layer hypergraph.

13. The hypergraph neural network system-based remote sensing hyperspectral image super-resolution reconstruction method according to claim 8, characterized in that, The step S3 specifically comprises the following steps: S31: inputting the training image and the hypergraph operator corresponding to the training image into the hypergraph neural network system; S32: calculating the training image input to the second residual block by the following formula to obtain the first output of the first processing unit ： （7）； wherein For residual operation, for the training image; S33: calculating the training image input to the second hypergraph attention module by the following formula to obtain the second output of the first processing unit ： （8）； wherein for hypergraph learning operations, for the hypergraph operator; S34: passing the first output input to the multi-stage spatial interaction module through the following formula and a second output perform feature fusion to obtain a fused feature map ： （9）； wherein, for the feature fusion operation; S35: fusing the feature maps input to the second reconstruction module by the following formula processing to obtain a third output of the first processing unit ： （10）； wherein for the reconstruction operation; S36: repeating steps S32-S35 and performing bicubic interpolation operation on the training image, a reconstructed image is obtained by ： （11）； wherein, for the convolution operation, BIC is a bicubic interpolation operation, I 1 -I 5 are third outputs of the five processing units, respectively, I 6 is the feature map A4; S37: update the network parameters of the hypergraph neural network system using the loss function, and after completing at least 25000 times of network parameter updates, test the network performance of the hypergraph neural network system using the validation set until the reconstructed image the result of the objective image quality evaluation index of does not rise or reaches a preset training round.

14. The hypergraph neural network system-based remote sensing hyperspectral image super-resolution reconstruction method according to claim 13, characterized in that, The loss function is composed of L1 loss, SAM loss, and gradient loss, and the weight distribution of the L1 loss, the SAM loss, and the gradient loss is 1:0.3:0.1; the evaluation index includes PSNR, SSIM, and SAM, and the preset training round is not less than 50 times.

15. The hypergraph neural network system-based remote sensing hyperspectral image super-resolution reconstruction method according to claim 13, characterized in that, The step S33 specifically comprises the following steps: S331: performing a size resetting operation on the training image and copying it twice to obtain three-layer feature maps; S332: add the pooling results obtained by respectively performing maximum pooling and average pooling on the three-layer feature map through the following formula: （12）； wherein, for the weight values, MP and AP are average pooling and maximum pooling, respectively, for the three-layer feature map of the n-1 layer in the hypergraph neural network system; S333: Calculate the assigned weight value obtained by sequentially performing convolution operations on the weight value through a 1x1 convolution layer, a ReLU module, and a 1x1 convolution layer ： （13）； wherein, is a convolution operation; S334: obtain a weighted hypergraph operator by multiplying the assigned weight value and the hypergraph operator corresponding to the training image ： （14）； wherein is a real set, N is the number of all nodes, and G is a hypergraph operator corresponding to the training image; S335: multiply the weighted hypergraph operator, the three-layer feature map, and the network parameters to obtain a three-layer feature map containing detailed texture features, spectral features, and position features through the following formula: （15）； wherein is a three-layer feature map containing detailed texture features, spectral features, and position features, and θ is a network parameter.

16. The hypergraph neural network system-based remote sensing hyperspectral image super-resolution reconstruction method according to claim 13, characterized in that, The step S34 specifically includes the following steps: S341: The two-layer feature maps containing the detail texture features and the spectral features are fused by the following formula, and the fusion result is fed into a fourth 3x3 convolution layer for convolution operation to obtain a feature fusion map ： （16）； wherein, for the Hadamard product operation, for a feature map containing fine texture features, is a feature map containing spectral features, C is the number of channels of the hyperspectral image, W is the width of the hyperspectral image, and H is the height of the hyperspectral image; S342: fusing the features The feature fusion is performed with the feature map containing the position feature, and a convolution operation is performed on the fusion result by feeding the fusion result into a fifth 3x3 convolution layer to obtain a feature fusion map ： （17）； wherein is a feature map containing position features; S343: fusing the features Interact with the output features of the residual block across domains to obtain a fusion feature map containing local information and global information ： （18）； wherein is a feature map output by the residual block.

Images