Remote sensing hyperspectral image super-resolution reconstruction method based on hybrid neural network

Abstract

The invention relates to the technical field of remote sensing hyperspectral images, in particular to a remote sensing hyperspectral image super-resolution reconstruction method based on a hybrid neural network, and the method comprises the steps: firstly carrying out the preprocessing of a remote sensing hyperspectral image data set, and obtaining a training set; constructing a hybrid neural network with a spectrum hypergraph branch, a space hypergraph branch, a semantic hypergraph branch, a sensitive wave band extraction branch and a recovery and reconstruction branch; training the hybrid neural network by using a training set to obtain a hybrid network model; and finally, inputting a to-be-reconstructed low-resolution remote sensing hyperspectral image into the hybrid network model to obtain a corresponding reconstructed remote sensing hyperspectral image. According to the method, space, spectrum and semantic feature information are fully combined, and a main body structure and detail change characteristics in an image are captured; the sensitive wave band extraction branch is provided to enhance cross guidance and fusion among multi-domain features, and the effect of super-resolution reconstruction is fully improved.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing hyperspectral images, and in particular relates to a remote sensing hyperspectral image super-resolution reconstruction method based on a hybrid neural network. Background Art

[0002] Thanks to the ability to capture rich spectral signals in the observed scene, hyperspectral images can provide more accurate guidance for image interpretation. With the upgrade of hardware technology, hyperspectral images have gradually become a very important source of information in remote sensing information processing technology. However, due to the physical limitations of imaging spectrometers, hyperspectral images usually face the problem of low spatial resolution during the acquisition process. The low spatial resolution greatly limits its application potential, especially for tasks such as refined species classification and anomaly detection. Image super-resolution technology can improve the spatial resolution of hyperspectral images at a lower cost, effectively broadening the application scope of hyperspectral images.

[0003] Hyperspectral image super-resolution technology aims to reconstruct high-quality images with higher spatial resolution using existing low-resolution images through algorithmic means. This technology pursues finer spatial details while retaining spectral information as much as possible. Compared with simple and direct interpolation algorithms, hyperspectral image super-resolution reconstruction needs to consider both spatial information and spectral signals during implementation, which increases the challenge of the task. In the early research in this field, scholars focused on traditional algorithms. Wavelet transform, maximum a posteriori estimation and mixed pixel analysis are the hot technical routes. With the development and innovation of machine learning in the field of image processing, hyperspectral image super-resolution reconstruction methods based on deep learning networks have become a research hotspot in this field.

[0004] Compared with traditional methods, the advantage of deep learning is that it can automatically learn complex patterns in images with the help of a large amount of training data, and usually has strong performance and generalization. In the reconstruction process, the rich spectral information carried in the image must be considered. The computational mode of 3D convolution has unique advantages in extracting spectral features, so it has been introduced into this field by some scholars. However, 3D convolution also brings a high computational burden, which brings certain restrictions to the design of the model. The local features in remote sensing images have obvious cross-regional and cross-scale reproducibility, so the extraction of global information seems to be very critical. As Transformer has emerged in multiple visual tasks, the excellent performance of the self-attention mechanism has gradually attracted attention. The self-attention mechanism associates pixels at different positions in the global range and autonomously assigns weights. However, the learning process of Transformer is completed autonomously by the network, lacking the guidance of prior information and learning focus, which limits the learning efficiency and task orientation of the network.

[0005] In recent years, hypergraph learning has been gradually introduced into the field of image processing as a learning model with great potential. Hypergraph is a more flexible and complex data expression format that can connect globally dependent feature points based on specific prior information. This means that guidance and bias can be applied to the network's learning process by mapping node information into a more explicit and task-oriented feature space. Super-resolution reconstruction of remote sensing images usually corresponds to a large observation area, and the complex components in a single pixel are highly coupled, which makes the representation of internal image information very challenging. The complete representation and full utilization of features in multiple dimensions are the key to achieving accurate image reconstruction. The advantage of hypergraph feature space transformation is particularly suitable for processing high-dimensional and complex data such as hyperspectral images, and can make up for the disadvantages of the above-mentioned convolutional neural networks and Transformer architectures. Summary of the invention

[0006] In view of this, the present invention aims to provide a remote sensing hyperspectral image super-resolution reconstruction method based on a hybrid neural network, which uses hypergraph learning as a benchmark learning model, and wavelet transform and spectral mixing analysis as data analysis methods. While characterizing and utilizing spectral information and spatial information, hypergraph learning realizes the capture of long-range correlations between pixels, thereby improving the richness and breadth of the network learning process.

[0007] To achieve the above object, the technical solution created by the present invention is implemented as follows: A remote sensing hyperspectral image super-resolution reconstruction method based on a hybrid neural network, comprising: S1: Obtain remote sensing hyperspectral image dataset and preprocess the dataset to obtain a training set; S2: construct a hybrid neural network, and train the hybrid neural network using the training set obtained in step S1 to obtain a hybrid network model; The hybrid neural network includes: a spectral hypergraph branch, which extracts spectral features from the input image; a spatial hypergraph branch, which extracts spatial features from the input image; a semantic hypergraph branch, which extracts semantic features from the input image; a sensitive band extraction branch, which fuses spectral features, spatial features and semantic features, and uses the fused features to extract high-quality semantic features of the input image; a restoration and reconstruction branch, which takes the initial super-resolution image of the input image as the basis, combines spectral features, spatial features, semantic features and high-quality semantic features, and obtains a reconstructed remote sensing hyperspectral image; S3: Input the low-resolution remote sensing hyperspectral image to be reconstructed into the hybrid network model obtained in step S2 to obtain the corresponding reconstructed remote sensing hyperspectral image.

[0008] Furthermore, in step S1, the preprocessing process of the data set includes: segmenting, amplifying and degrading the data set to obtain a low-resolution input image.

[0009] Furthermore, in the spectral hypergraph branch in step S2: a one-dimensional wavelet transform is performed on the input image to obtain an approximation coefficient and a detail coefficient; a low-frequency feature hypergraph and a high-frequency feature hypergraph are constructed according to the approximation coefficient and the detail coefficient, and then the low-frequency feature hypergraph and the high-frequency feature hypergraph are respectively converted by the following formula to obtain a low-frequency feature hypergraph operator and a high-frequency feature hypergraph operator: ; ; in, represents the low-frequency feature hypergraph operator, represents the high-frequency feature hypergraph operator, represents the low-frequency feature hypergraph, represents the high-frequency feature hypergraph, and The diagonal matrices representing the hyperedge degree and vertex degree respectively, E is initialized to the identity matrix; The input image is used as the input feature and is input into the spectral hypergraph mutual attention module together with the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator. The output feature is then processed with the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator by no less than one spectral hypergraph mutual attention module to obtain the spectral feature.

[0010] Furthermore, in the spectral hypergraph mutual attention module: The input features are multiplied by the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator to obtain the low-frequency spectral feature map and the high-frequency spectral feature map, namely: ; ; in, , n represents the number of spectral hypergraph mutual attention modules, N represents the total number of spectral hypergraph mutual attention modules, represents the spectral features output by the previous spectral hypergraph mutual attention module, The input feature of the spectral hypergraph mutual attention module is the input image, represents the low-frequency spectral feature map, Represents the high-frequency spectral feature map, and Represents the weight coefficient for calculating the low-frequency spectral feature map and the high-frequency spectral feature map; Extract the query matrix from the low-frequency spectral feature map, extract the key matrix and value matrix from the high-frequency spectral feature map, and then perform a multi-head mutual attention mechanism operation on the query matrix, key matrix, and value matrix to obtain the spectral features output by the current spectral hypergraph mutual attention module, namely: ; in, represents the spectral features output by the current spectral hypergraph mutual attention module, represents the query matrix, represents the key matrix, represents the value matrix, Represents the operation of the multi-head mutual attention mechanism.

[0011] Furthermore, in the spatial hypergraph branch in step S2: The input image is reduced in dimension, and then the reduced image is subjected to a two-dimensional wavelet transform to obtain the low-frequency component and high-frequency component of the input image, namely: ; in, represents the low-frequency component, Represents the high-frequency components in the horizontal, vertical, and diagonal directions of the input image. represents the two-dimensional wavelet transform, Represents the image retained after the input image is reduced in dimension; The low-frequency components and high-frequency components are interpolated twice bicubicly, the extracted frequency features are extended, and the low-frequency spatial information hypergraph and high-frequency spatial information hypergraph of the input image are constructed by the following formula: ; in, represents the low-frequency spatial information hypergraph, represents a high-frequency spatial information hypergraph, Represents the complete function of constructing low-frequency spatial information hypergraph and high-frequency spatial information hypergraph; The corresponding elements of the low-frequency spatial information hypergraph and the high-frequency spatial information hypergraph are added to obtain the spatial information hypergraph of the input image, and the spatial information hypergraph is converted by the following formula to obtain the spatial information hypergraph operator: ; in, represents the spatial information hypergraph operator, Represents a spatial information hypergraph, and The diagonal matrices representing the hyperedge degree and vertex degree respectively, E is initialized to the identity matrix; The input image is used as the input feature and is input into the spatial hypergraph self-attention module together with the spatial information hypergraph operator. The output feature is then processed together with the spatial information hypergraph operator by at least one spatial hypergraph self-attention module to obtain the spatial feature.

[0012] Furthermore, in the spatial hypergraph self-attention module: Multiply the input features by the spatial information hypergraph operator to obtain the spatial information feature graph, namely: ; in, , m represents the number of spatial hypergraph self-attention modules, M represents the total number of spatial hypergraph self-attention modules, represents the spatial features output by the previous spatial hypergraph self-attention module, The feature representing the input space hypergraph self-attention module is the input image, Represents the spatial information feature map, Represents the weight coefficient for calculating the spatial information feature map; Extract the query matrix, key matrix and value matrix from the spatial information feature graph, and then perform multi-head self-attention mechanism operation on the query matrix, key matrix and value matrix to obtain the spatial features output by the current spatial hypergraph self-attention module, namely: ; in, represents the spatial features output by the current spatial hypergraph self-attention module, represents the query matrix, represents the key matrix, represents the value matrix, Represents the operation of the multi-head self-attention mechanism.

[0013] Furthermore, in the semantic hypergraph branch in step S2: The abundance matrix is ​​parsed from the input image using a non-negative matrix factorization algorithm; According to the abundance matrix, the semantic hypergraph of the input image is constructed by the following formula: ; in, represents a semantic hypergraph, represents the abundance matrix, Represents the complete function for constructing a semantic hypergraph; The semantic hypergraph operator is obtained by converting the semantic hypergraph into a semantic hypergraph operator through the following formula: ; in, represents a semantic hypergraph operator, and The diagonal matrices representing the hyperedge degree and vertex degree respectively, E is initialized to the identity matrix; The input image is used as the input feature and the semantic hypergraph operator is input into the semantic hypergraph. Figure 3 In the D module, the output features are then combined with the semantic hypergraph operator through at least one semantic hypergraph. Figure 3 The D module processes and obtains semantic features.

[0014] Furthermore, in the semantic Figure 3 In module D: Multiply the input features with the semantic hypergraph operator to obtain the semantic information feature graph, namely: ; in, , o indicates semantic super Figure 3 The number of D modules, Expressing semantics Figure 3 The total number of D modules, Indicates the previous semantic super Figure 3 The semantic features output by the D module, Indicates that the input semantics is super Figure 3 The feature of D module is the input image, Represents the semantic information feature map, Represents the weight coefficient for calculating the semantic information feature map; Then perform three-dimensional convolution on the semantic information feature map to obtain the current semantic hyperparameter. Figure 3 Semantic features output by the D module.

[0015] Furthermore, the sensitive band extraction branch in step S2 includes a weight acquisition module and a reference residual module; wherein: In the weight acquisition module, the pixel value statistics of each band in the spectral feature, spatial feature and semantic feature are performed to estimate the probability distribution of the pixel value of each band; the entropy value of each band in each feature is calculated by the maximum entropy formula, and the band with the largest entropy is selected as the sensitive band of each feature; The baseline residual module includes no less than 3 cascaded residual blocks, among which the sensitive bands of the three features are used as weights between two adjacent residual blocks, which are multiplied with the corresponding elements of the input features of the previous residual block, and the multiplied features are added with the input features of the next residual block to obtain the output features; each residual block includes no less than 2 cascaded convolutional layers, and the input features of the current residual block are convolved by the convolutional layer and then added with the corresponding elements of the input features of the current residual block to obtain the output features; the output features obtained by the last residual block are high-quality semantic features.

[0016] Furthermore, in the restoration and reconstruction branch in step S2, The corresponding elements of the spectral features, spatial features, semantic features and high-quality semantic features are added, and the added features are sequentially subjected to at least one convolution operation and a transposed convolution operation, and the obtained features are added to the corresponding elements of the preliminary super-resolution image obtained by upsampling the input image, and the added features are sequentially subjected to at least one convolution operation to obtain a reconstructed remote sensing hyperspectral image.

[0017] Compared with the prior art, the invention can achieve the following beneficial effects: (1) The invention provides a remote sensing hyperspectral image super-resolution reconstruction method based on a hybrid neural network, which uses hypergraph learning as a benchmark learning mode and wavelet transform and spectral hybrid analysis as data analysis methods. The model captures the main structure and detail variation characteristics of the image based on high and low frequency features in space and spectrum. The sensitive band extraction branch is combined to fuse the spectral features, spatial features and semantic features, strengthen the propagation and reconstruction of semantic information, and improve the reconstruction effect. (2) In the remote sensing hyperspectral image super-resolution reconstruction method based on hybrid neural network described in the invention, the spectral hypergraph branch combines one-dimensional wavelet transform, hypergraph learning and mutual attention mechanism to better characterize and utilize spectral information; in the spectral hypergraph mutual attention module, the frequency characteristics of each signal node are considered and processed, which can effectively improve the fidelity of the reconstructed spectrum; in the same scene, since the target has strong cross-regional reproducibility and the hypergraph represents the dependency relationship between nodes in the global range, hypergraph learning realizes the capture of long-range correlation between pixels, thereby improving the richness and breadth of the network learning process; (3) In the remote sensing hyperspectral image super-resolution reconstruction method based on hybrid neural network described in the invention, the spatial hypergraph branch combines two-dimensional wavelet transform, hypergraph learning and self-attention mechanism to achieve accurate reconstruction of image structure information and detail texture; in the spatial hypergraph self-attention module, hypergraph learning based on the main structure and spatial features of the input image can efficiently utilize similar features that appear repeatedly in the image, achieve mutual enhancement between feature points and accurate reconstruction of spatial information; (4) In the remote sensing hyperspectral image super-resolution reconstruction method based on hybrid neural network described in the invention, the semantic learning module combining unmixing, hypergraph learning and 3D convolution in the semantic hypergraph branch effectively reduces the difficulty of information propagation by mapping highly coupled complex information into a more clear high-level semantic space, which is conducive to more accurate detail reconstruction; (5) In the hybrid neural network-based remote sensing hyperspectral image super-resolution reconstruction method described in the present invention, the intervention of sensitive band attention in the sensitive band extraction branch dynamically adjusts the weight of each feature during information fusion to more accurately guide the interaction between spectral, spatial and high-level semantic information. This approach ensures the collaborative work of different feature domains and ultimately improves the super-resolution effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of a process for super-resolution reconstruction of remote sensing hyperspectral images based on a hybrid neural network according to an embodiment of the present invention; Figure 2 A schematic diagram of the network structure of a hybrid neural network according to an embodiment of the present invention; Figure 3 A schematic diagram of a spectral hypergraph mutual attention module according to an embodiment of the present invention; Figure 4 A schematic diagram of a spatial hypergraph self-attention module according to an embodiment of the present invention; Figure 5 The semantic hyperparameter described in the embodiment of the present invention is created Figure 3 Schematic diagram of D module; Figure 6 A visualization comparison result diagram of the reconstruction method described in the embodiment of the present invention and other reconstruction methods on the MDAS data set; Figure 7 A visualization comparison result diagram of the reconstruction method described in the embodiment of the present invention and other reconstruction methods on the Pavia Centre dataset; Figure 8 This is a visualization comparison result diagram of the reconstruction method described in the embodiment of the present invention and other reconstruction methods on the Houston dataset. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0020] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0021] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0022] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0023] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0024] like Figure 1 to Figure 2 As shown, the remote sensing hyperspectral image super-resolution reconstruction method based on hybrid neural network described in the embodiment of the present invention includes: S1: Obtain a remote sensing hyperspectral image dataset and preprocess the dataset to obtain a training set.

[0025] In step S1, the preprocessing process of the data set includes: segmenting, amplifying and degrading the data set to obtain a low-resolution input image. In some embodiments, 70% of the image area in the remote sensing hyperspectral image data set is selected as a training set, and 30% of the area is selected as a validation set. Randomly select patches from each area, and set different patch sizes according to the magnification factor. Among them, 2, 3 and 4 times magnification correspond to patch sizes of 64×64, 96×96 and 128×128, respectively, and each patch is randomly horizontally flipped, rotated at different angles, and scaled at different magnifications. Finally, these patches are downsampled to 32×32 low-resolution hyperspectral images according to different scale factors through bicubic downsampling, and the low-resolution hyperspectral images are used as the input images of the hybrid neural network, and the corresponding real remote sensing hyperspectral images are used as the output images of the hybrid neural network, and the hybrid neural network is subsequently trained. In a certain embodiment, remote sensing hyperspectral images are obtained from the MDAS data set, the PaviaCentre data set and the Houston data set to form a remote sensing hyperspectral image data set.

[0026] S2: Construct a hybrid neural network, and use the training set obtained in step S1 to train the hybrid neural network to obtain a hybrid network model.

[0027] like Figure 2 As shown, the hybrid neural network includes parallel spectral hypergraph branches, spatial hypergraph branches, semantic hypergraph branches and sensitive band extraction branches, as well as a restoration and reconstruction branch that receives the output features of the three branches. Among them, the spectral hypergraph branch extracts spectral features in the input image, the spatial hypergraph branch extracts spatial features in the input image, the semantic hypergraph branch extracts semantic features in the input image, the sensitive band extraction branch fuses spectral features, spatial features and semantic features, and uses the fused features to extract high-quality semantic features of the input image. The restoration and reconstruction branch is based on the preliminary super-resolution image of the input image, and combines spectral features, spatial features, semantic features and high-quality semantic features to obtain a reconstructed remote sensing hyperspectral image.

[0028] Specifically, in the spectral hypergraph branch in step S2: a one-dimensional wavelet transform is performed on the input image to obtain an approximation coefficient and a detail coefficient, which can be expressed by the following formula: ; in, represents the approximation coefficient, represents the detail coefficient, represents the input image, Represents one-dimensional wavelet transform, H, W and C represent the height, width and number of channels of the input image respectively. Indicates the size A real matrix of .

[0029] According to the similarity of low-frequency and high-frequency characteristics between each spectrum, the low-frequency feature hypergraph and high-frequency feature hypergraph are constructed using the approximation coefficient and detail coefficient. The nodes of the hypergraph are the approximation coefficients or detail coefficients of each spectrum, and each hyperedge represents the similarity between the nodes. The higher the similarity, the higher the weight assigned to the hyperedge. Specifically, each hyperedge contains ten nodes selected by the K nearest neighbor algorithm based on Euclidean distance, and the weight of each node is obtained by a negative exponential function. The hypergraph construction process can be expressed as: ; in, represents the low-frequency feature hypergraph, represents the high-frequency feature hypergraph, Represents the complete function of constructing low-frequency feature hypergraph and high-frequency feature hypergraph, , T represents the number of nodes, Indicates the size A real matrix of .

[0030] The low-frequency feature hypergraph and the high-frequency feature hypergraph are converted into the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator by the following formulas: ; ; in, represents the low-frequency feature hypergraph operator, represents the high-frequency feature hypergraph operator, and are the diagonal matrices representing the hyperedge degree and vertex degree respectively, and E is initialized to the identity matrix.

[0031] The input features and the input image are subjected to hypergraph learning, specifically: the input image is used as the input feature, and is input into the spectral hypergraph mutual attention module with the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator, and the output feature is then processed with the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator by at least one spectral hypergraph mutual attention module to obtain the spectral feature. In one embodiment, the number of spectral hypergraph mutual attention modules is 6.

[0032] The structure of the spectral hypergraph mutual attention module is as follows Figure 3 As shown, where: the input features are multiplied by the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator respectively to obtain the low-frequency spectral feature map and the high-frequency spectral feature map, that is: ; ; in, , n represents the number of spectral hypergraph mutual attention modules, N represents the total number of spectral hypergraph mutual attention modules, represents the spectral features output by the previous spectral hypergraph mutual attention module, The input feature of the spectral hypergraph mutual attention module is the input image, represents the low-frequency spectral feature map, Represents the high-frequency spectral feature map, and Represents the weight coefficient for calculating the low-frequency spectral feature map and the high-frequency spectral feature map, Indicates the size A real matrix of .

[0033] The mutual attention algorithm is performed on the low-frequency spectral feature map and the high-frequency spectral feature map to better utilize the dependency between the high-frequency features and the low-frequency structure, that is, the query matrix is ​​extracted from the low-frequency spectral feature map, and the key matrix and value matrix are extracted from the high-frequency spectral feature map. Then, the query matrix, the key matrix and the value matrix are operated by the multi-head mutual attention mechanism to obtain the spectral features output by the current spectral hypergraph mutual attention module, that is: ; in, represents the spectral features output by the current spectral hypergraph mutual attention module, represents the query matrix, represents the key matrix, represents the value matrix, Represents the operation of the multi-head mutual attention mechanism.

[0034] In summary, the feature extraction process of the complete spectral hypergraph mutual attention module can be expressed as: ; in, Represents the processing operation of the nth spectral hypergraph mutual attention module, in which the frequency characteristics of each signal node are considered and processed, which can effectively improve the fidelity of the reconstructed spectrum. In addition, within the same scene, the target has strong cross-regional reproducibility, and the hypergraph represents the dependencies between nodes in the global scope. Hypergraph learning captures the long-range correlation between pixels, thereby improving the richness and breadth of the network learning process.

[0035] In addition to spectral information, for super-resolution tasks aimed at improving spatial resolution, capturing and utilizing spatial information is equally important. Low-frequency signals in an image represent features that change slowly in space. This portion of pixel values ​​changes more slowly, and usually contains the main structural information in the observed scene, such as the brightness distribution of the background; high-frequency signals in an image represent features that change rapidly in space, and this portion of pixel values ​​changes more rapidly, and usually contains detailed structural information in the observed scene, such as the edge and texture of the target. Therefore, in the spatial hypergraph branch of step S2 of the present invention, a spatial feature extraction module combining two-dimensional wavelet transform, hypergraph learning, and self-attention mechanism is used to achieve accurate reconstruction of image structural information and detailed texture.

[0036] Specifically, hyperspectral images usually have hundreds of bands, the quality of each band varies greatly, and contains a lot of redundant information. In order to simplify the processing process and reduce the computational overhead, in the spatial hypergraph branch, the input image is first reduced in dimension, and then the reduced image is subjected to a two-dimensional wavelet transform to obtain the low-frequency and high-frequency components of the input image, namely: ; ; in, represents the low-frequency component, Represents the high-frequency components in the horizontal, vertical, and diagonal directions of the input image. represents a two-dimensional wavelet transform. In the embodiment of the present invention, principal component analysis (PCA) is preferably used to reduce the dimension of the image. k represents the number of channels retained after the image dimension reduction. represents the image retained after the input image is reduced in dimension, Indicates the size The low-frequency and high-frequency components are doubled by bicubic interpolation, the extracted frequency features are extended, and the low-frequency and high-frequency spatial information hypergraphs of the input image are constructed by the following formula: ; ; in, represents the low-frequency spatial information hypergraph, represents a high-frequency spatial information hypergraph, Represents the complete function of constructing low-frequency spatial information hypergraph and high-frequency spatial information hypergraph; Add the corresponding elements of the low-frequency spatial information hypergraph and the high-frequency spatial information hypergraph to obtain the spatial information hypergraph of the input image, that is: ; in, Represents a spatial information hypergraph; The spatial information hypergraph is converted into the spatial information hypergraph operator by the following formula: ; in, Represents a spatial information hypergraph operator.

[0037] Input features or input images for hypergraph learning, specifically, input the input image as input features, and input it into the spatial hypergraph self-attention module with the spatial information hypergraph operator, and then the output features and the spatial information hypergraph operator are processed by at least one spatial hypergraph self-attention module to obtain spatial features. In one embodiment, the number of spatial hypergraph self-attention modules is 6.

[0038] The structure of the spatial hypergraph self-attention module is as follows Figure 4 As shown, specifically including: Multiply the input features by the spatial information hypergraph operator to obtain the spatial information feature graph, namely: ; in, , m represents the number of spatial hypergraph self-attention modules, M represents the total number of spatial hypergraph self-attention modules, represents the spatial features output by the previous spatial hypergraph self-attention module, The feature representing the input space hypergraph self-attention module is the input image, Represents the spatial information feature map, Represents the weight coefficient for calculating the spatial information feature map.

[0039] Extract the query matrix, key matrix and value matrix from the spatial information feature graph, and then perform multi-head attention mechanism operation on the query matrix, key matrix and value matrix to obtain the spatial features output by the current spatial hypergraph self-attention module, namely: ; in, represents the spatial features output by the current spatial hypergraph self-attention module, represents the query matrix, represents the key matrix, represents the value matrix, Represents the operation of the multi-head self-attention mechanism.

[0040] To sum up, the feature extraction process of the complete spatial hypergraph self-attention module can be expressed as: ; in, Represents the feature extraction operation of the spatial hypergraph self-attention module.

[0041] In the spatial hypergraph self-attention module, hypergraph learning based on main structure and detail texture can efficiently utilize similar features that appear repeatedly in the image to achieve mutual enhancement between feature points and accurate reconstruction of spatial information.

[0042] A pixel in a hyperspectral image often corresponds to a larger observation area, which means that the image contains a large number of mixed pixels. Different targets usually present complex interspersed releases. This uneven distribution and multi-category coupling affect the expression of their respective spatial, spectral and frequency domain information. Information decoupling for mixed pixels can more accurately perform feature decomposition and reconstruction, and reduce the mutual interference of multi-dimensional information in complex environments. Therefore, the present invention proposes a semantic hypergraph branch based on unmixing, hypergraph learning and 3D convolution in step S2 to complete high-quality semantic information extraction. Specifically, in the semantic hypergraph branch, it is necessary to first parse out the end members representing pure substances and the abundance ratio of each end member in each pixel from the input image. In the embodiment of the present invention, the non-negative matrix decomposition (NMF) algorithm based on the gradient descent method is preferably used to estimate the end member matrix and the abundance matrix, and its basic form can be expressed as: ; in represents the original input image, Represents the abundance matrix of each end member content in the pixel, The end member matrix represents the spectral characteristics of pure substances, q represents the number of end members, Indicates the size The matrix of Indicates the size The objective function of NMF is: ; in represents the Frobenius norm, represents the elements in the input image X, Represents the abundance matrix A and the end member matrix In addition, the abundance matrix A and the end member matrix obtained by analysis The non-negative constraint and the sum-to-one constraint must be satisfied, namely:

[0043] in, Represents the abundance of the jth endmember contained in the i-th node.

[0044] According to the abundance matrix, the semantic hypergraph of the input image is constructed by the following formula: ; in, represents a semantic hypergraph, Represents the complete function for building a semantic hypergraph.

[0045] The semantic hypergraph operator is obtained by converting the semantic hypergraph into a semantic hypergraph operator through the following formula: ; in, Represents a semantic hypergraph operator.

[0046] Input images or input features for hypergraph learning, that is, input images as input features and input semantic hypergraph operators into semantic hypergraphs. Figure 3 In the D module, the output features are then combined with the semantic hypergraph operator through at least one semantic hypergraph. Figure 3 D module processing, obtain semantic features. In one embodiment, semantic hyper Figure 3 The number of D modules is 6.

[0047] Semantic Super Figure 3 The structure of the D module is as follows Figure 5 As shown, the input feature is multiplied by the semantic hypergraph operator to obtain the semantic information feature map, that is: ; in, , o indicates semantic super Figure 3 The number of D modules, Expressing semantics Figure 3 The total number of D modules, Indicates the previous semantic super Figure 3 The semantic features output by the D module, Indicates that the input semantics is super Figure 3 The feature of D module is the input image, Represents the semantic information feature map, Represents the weight coefficient for calculating the semantic information feature map; In order to better extract the spectral features, the embodiment of the present invention further performs three-dimensional convolution processing on the semantic information feature map to obtain the current semantic hyperspectral feature map. Figure 3 The semantic features output by the D module are:

[0048] in, Indicates the current semantic super Figure 3 The semantic features output by the D module, represents the three-dimensional convolution processing, Indicates the size In the embodiment of the present invention, the three-dimensional convolution process includes the feature to be processed Convolution processing is performed on three consecutive three-dimensional convolutional layers, and the features after convolution in each three-dimensional convolutional layer are activated by the ReLU activation function.

[0049] In summary, the complete semantic Figure 3 The feature extraction process of the D module can be expressed as:

[0050] in, Expressing semantics Figure 3 Feature extraction operation of D module.

[0051] An important feature of hypergraph learning is that it can support cross-pixel node collaborative optimization. Pixels with similar abundance distributions are grouped together in the same hyperedge, and the model can consider the collaborative relationship between different pixels when optimizing. Figure 3 The D module effectively reduces the difficulty of information dissemination by mapping highly coupled complex information into a more clear high-level semantic space, which is conducive to more accurate reconstruction of details.

[0052] The above three branches respectively extract the spectral, spatial and high-level semantic information in the input image, which can provide a more comprehensive guidance for the reconstruction process. However, there are domain differences between the features extracted by each branch, and independent learning paths often lead to poor feature compatibility for reconstruction. The sensitive band extraction branch in step S2 of the present invention can achieve cross-domain information fusion.

[0053] Specifically, the sensitive band extraction branch in step S2 includes a weight acquisition module and a reference residual module; wherein: In the weight acquisition module, the pixel value statistics of each band in the spectral features, spatial features and semantic features are performed to estimate the probability distribution of the pixel value of each band. , the entropy value of each band in each feature is calculated by the maximum entropy formula, that is: ; in Represents the set of pixel values ​​of the i-th band, and selects the band with the largest entropy as the sensitive band for each feature.

[0054] The reference residual module includes no less than 3 cascaded residual blocks, in which the sensitive bands of the three features are used as weights between two adjacent residual blocks, and the corresponding elements of the input features of the previous residual block are multiplied, and the multiplied features are added to the input features of the next residual block to obtain the output features. In the embodiment of the present invention, the reference residual module has a total of six residual blocks, and the output of the second residual block is fused with the sensitive bands of other branches and sent to the third residual block. At this time, the above content can be expressed as: ; ; in, represents the weight, , and Respectively represent the sensitive bands in spectral features, spatial features, and semantic features, represents the output feature of the second residual block, that is, the input feature of the third residual block, represents the output feature of the third residual block, Represents the residual block pair feature operation; the remaining residual blocks include no less than 2 cascaded convolutional layers, the input features of the current residual block are convolved by the convolutional layer, and then added to the corresponding elements of the input features of the current residual block to obtain the output features; the output features obtained by the last residual block are high-quality semantic features. In the embodiment of the present invention, each residual block includes two cascaded convolutional layers, and the activation function between the two convolutional layers is ReLU.

[0055] The intervention of sensitive band attention in the sensitive band extraction branch dynamically adjusts the weight of each feature during information fusion to more accurately guide the interaction between spectral, spatial and high-level semantic information. This approach ensures the collaborative work of different feature domains and ultimately improves the super-resolution effect.

[0056] In the restoration and reconstruction branch in step S2, the corresponding elements of the spectral features, spatial features, semantic features and high-quality semantic features are added, and the added features are sent to a convolution operation and a transposed convolution operation for at least one time in sequence, and the obtained features are added to the corresponding elements of the preliminary super-resolution image obtained by upsampling the input image, and then the added features are sequentially subjected to a convolution operation for at least one time, and a reconstructed remote sensing hyperspectral image is obtained. In the embodiment of the present invention, the added features are sent to a convolution operation and a transposed convolution operation for one time in sequence, and the activation function between the convolution operation and the transposed convolution operation is a ReLU activation function; the input image is subjected to bicubic interpolation to obtain a preliminary super-resolution image with a resolution consistent with the reconstructed remote sensing hyperspectral image; the features after the addition of the preliminary super-resolution image and the features output by the transposed convolution operation are subjected to two convolution operations, and the activation function between the two convolution operations is a ReLU activation function, and the final reconstructed remote sensing hyperspectral image is obtained.

[0057] In the training process of step S2, the embodiment of the present invention preferably uses a loss function constructed by cooperating between the L1 loss function, the SAM loss function and the gradient loss function to train the hybrid neural network. The constructed loss function is as follows: ; in, represents the loss function constructed in the embodiment of the present invention, y represents the real remote sensing hyperspectral image corresponding to the input image, represents the reconstructed remote sensing hyperspectral image corresponding to the input image, represents the L1 loss function, represents the SAM loss function, represents the gradient loss function, , and Respectively represent the weight coefficients of the L1 loss function, the SAM loss function, and the gradient loss function. In the embodiment of the present invention, . During the training process, the optimizer selects Adam (the first-order momentum decay rate is 0.9, the second-order momentum decay rate is 0.999), the batchsize is 4, and the initial learning rate is 0.0001. The number of training generations is set to 50. The parameters of the hybrid neural network are updated according to the loss function. As the loss function gradually decreases, the super-resolution reconstruction effect of the hybrid neural network on the low-resolution input image becomes better and better. After completing several rounds of iterative updates of the network parameters, the performance of the hybrid network model is tested using the validation set. When the objective image quality evaluation index result of the reconstructed image of the validation set no longer increases or reaches the preset upper limit of the training rounds, the training is terminated. In a certain embodiment, when the floating amplitude of the image evaluation index of the validation set is lower than 0.1 within ten rounds, the training is terminated.

[0058] S3: Input the low-resolution remote sensing hyperspectral image to be reconstructed into the hybrid network model obtained in step S2 to obtain the corresponding reconstructed remote sensing hyperspectral image.

[0059] In order to verify the effectiveness of the remote sensing hyperspectral image super-resolution reconstruction method based on hybrid neural network provided by the present invention, the embodiment of the present invention adopts peak signal-to-noise ratio (PSNR), structural similarity (SSIM), spectral angle similarity (SAM) and relative global error separation metric (ERGAS) to objectively evaluate the quality of reconstructed remote sensing hyperspectral images, and uses bicubic interpolation (Bicubic), 3D-FCNN (published in the paper "Hyperspectral ImageSpatial Super-Resolution via 3D Full Convolutional Neural Network" in the journal "Remote Sensing"), GDRRN (published in the paper "Single Hyperspectral Image Super-resolution with GroupedDeep Recursive Residual Network" in the journal "2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM)"), SSPSR (published in the paper "Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery" in the journal "IEEE TRANSACTIONS ONCOMPUTATIONAL IMAGING"), MCNet (published in the journal "Remote Sensing, “Mixed 2D / 3D Convolutional Network for Hyperspectral Image Super-Resolution”, ERCSR (published in IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, “Exploring the Relationship Between 2D / 3D Convolution forHyperspectral Image Super-Resolution”), GELIN (published in IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, “A Group-Based Embedding Learning and Integration Network for Hyperspectral Image Super-Resolution”)The method provided by the present invention is compared with the method provided by the present invention on the MDAS dataset, the Pavia Centre dataset and the Houston dataset respectively. The objective comparison results of the method provided by the present invention and the comparison method on the MDAS dataset are shown in Table 1: Table 1:

[0060] From the experimental results, the method provided by the present invention has achieved the best numerical results under the three magnification factors of 2×, 3× and 4×. Specifically, in the 2× super-resolution reconstruction, compared with the suboptimal results, the four indicators were optimized by 0.042dB, 0.0002, 0.13 and 0.077 respectively. Compared with other indicators, the SAM of the reconstructed image of the present invention has a more obvious lead. The SAM under three multiples was optimized by 0.13, 0.185 and 0.155 respectively, which proves the effectiveness of the method provided by the present invention in improving spectral fidelity. The hypergraph constructed based on the high-frequency information and low-frequency information in the spectral signal effectively establishes the spectral similarity relationship across regions. The proposed spectral hypergraph mutual attention module integrates the low-frequency features and high-frequency features in the spectrum in the global range, and the mutual attention mechanism further captures and strengthens the long-range correlation between signals. The experimental results show that the above strategy for spectral information has a very positive effect on improving spectral fidelity. The comparison results are shown in the figure below. Figure 6 As shown, from Figure 6 It can be observed that the edges and textures reconstructed by the method provided by the present invention are clearer, while other methods usually have more obvious blur and deformation. Whether it is numerical value or visual effect, the method provided by the present invention has more obvious advantages.

[0061] The objective comparison results of the method provided by the present invention and the comparison method on the Pavia Centre dataset are shown in Table 2. It can be seen from Table 2 that in the 3× super-resolution reconstruction task, compared with the suboptimal results, the method provided by the present invention has improved the four indicators by 0.171dB, 0.0042, 0.097 and 0.107 respectively; in the 4× super-resolution reconstruction task, the method provided by the present invention achieved two optimal indicators and two suboptimal indicators. The method provided by the present invention constructs hypergraphs with different characteristics based on spectral high / low frequency information, spatial high / low frequency information and high-level semantic space, and achieves more accurate reconstruction by comprehensively characterizing the pixel correlation in a global range. The comparison results are shown in the figure below. Figure 7 As shown, from Figure 7 It can be seen that in the red frame marked area, the method provided by the present invention can better restore tiny details and alleviate spatial distortion.

[0062] Table 2:

[0063] The objective comparison results of the method provided by the present invention and the comparative method on the Houston dataset are shown in Table 3. It can be seen from Table 3 that in the 2× and 3× super-resolution reconstruction tasks, all indicators of the reconstructed image obtained by the method provided by the present invention are optimal, among which the PSNR reaches 38.119dB (+0.032dB) and 35.004dB (+0.066dB), respectively. This proves that hypergraph learning based on high and low frequency features can well capture the main structure and subtle changes in the image. The hypergraph branch of the method provided by the present invention is used to capture global information, and the residual branch is used to mine local information. Feature extraction at multiple scales takes into account the neighborhood details and long-range correlations of information nodes in the network learning process. In summary, our method has excellent performance on all three datasets, proving the superiority and robustness of the model. The comparison results are shown in the figure below. Figure 8 As shown, from Figure 8 It can be seen that in the area marked in the red frame, the method provided by the present invention reconstructs more realistic road details.

[0064] Table 3:

[0065] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0066] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing hyperspectral image super-resolution reconstruction method based on hybrid neural network, characterized in that: include: S1: Obtain remote sensing hyperspectral image dataset and preprocess the dataset to obtain a training set; S2: Construct a hybrid neural network, and train the hybrid neural network using the training set obtained in step S1 to obtain a hybrid network model; the hybrid neural network includes: The spectral hypergraph branch extracts spectral features from the input image; The spatial hypergraph branch extracts spatial features from the input image; The semantic hypergraph branch extracts semantic features from the input image; A sensitive band extraction branch is used to fuse the spectral feature, the spatial feature and the semantic feature, and to extract high-quality semantic features of the input image using the fused features; The restoration and reconstruction branch is based on the preliminary super-resolution image of the input image and combines the spectral feature, the spatial feature, the semantic feature and the high-quality semantic feature to obtain a reconstructed remote sensing hyperspectral image; S3: Input the low-resolution remote sensing hyperspectral image to be reconstructed into the hybrid network model obtained in step S2 to obtain the corresponding reconstructed remote sensing hyperspectral image.

2. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 1, characterized in that: In step S1, the preprocessing process of the data set includes: The data set is segmented, amplified and degraded to obtain a low-resolution input image, which is used as an input of the hybrid neural network.

3. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 1, characterized in that: In the spectral hypergraph branch in step S2: Perform one-dimensional wavelet transform on the input image to obtain approximation coefficients and detail coefficients; According to the approximation coefficient and the detail coefficient, a low-frequency feature hypergraph and a high-frequency feature hypergraph are constructed, and then the low-frequency feature hypergraph and the high-frequency feature hypergraph are respectively converted by the following formula to obtain a low-frequency feature hypergraph operator and a high-frequency feature hypergraph operator: ； ； in, represents the low-frequency feature hypergraph operator, represents the high-frequency feature hypergraph operator, represents the low-frequency feature hypergraph, represents the high-frequency feature hypergraph, and The diagonal matrices representing the hyperedge degree and vertex degree respectively, E is initialized to the identity matrix; The input image is used as the input feature and is input into the spectral hypergraph mutual attention module together with the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator. The output feature is then processed together with the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator by at least one spectral hypergraph mutual attention module to obtain the spectral feature.

4. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 3 is characterized in that: In the spectral hypergraph mutual attention module: The input features are multiplied by the low-frequency feature hypergraph operator and the high-frequency feature hypergraph operator respectively to obtain a low-frequency spectral feature map and a high-frequency spectral feature map, that is: ； ； in, , n represents the number of the spectral hypergraph mutual attention modules, N represents the total number of the spectral hypergraph mutual attention modules, represents the spectral features output by the previous spectral hypergraph mutual attention module, Indicates that the feature input to the spectral hypergraph mutual attention module is the input image, represents the low-frequency spectral characteristic graph, represents the high-frequency spectral characteristic graph, and Indicates the weight coefficient for calculating the low-frequency spectral feature graph and the high-frequency spectral feature graph; Extract the query matrix from the low-frequency spectral feature map, extract the key matrix and the value matrix from the high-frequency spectral feature map, and then perform a multi-head mutual attention mechanism operation on the query matrix, the key matrix and the value matrix to obtain the spectral features output by the current spectral hypergraph mutual attention module, that is: ； in, represents the spectral features output by the current spectral hypergraph mutual attention module, represents the query matrix, represents the bond matrix, represents the value matrix, Represents the operation of the multi-head mutual attention mechanism.

5. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 1, characterized in that: In the spatial hypergraph branch in step S2: The input image is reduced in dimension, and then the reduced image is subjected to a two-dimensional wavelet transform to obtain the low-frequency component and high-frequency component of the input image, namely: ； in, represents the low-frequency component, represents the high-frequency components in the horizontal, vertical and diagonal directions of the input image, represents the two-dimensional wavelet transform, represents the image retained after the input image is reduced in dimension; The low-frequency component and the high-frequency component are subjected to double bicubic interpolation, the extracted frequency features are extended, and the low-frequency spatial information hypergraph and the high-frequency spatial information hypergraph of the input image are constructed by the following formula: ； in, represents the low-frequency spatial information hypergraph, represents the high-frequency spatial information hypergraph, Represents a complete function for constructing the low-frequency spatial information hypergraph and the high-frequency spatial information hypergraph; The corresponding elements of the low-frequency spatial information hypergraph and the high-frequency spatial information hypergraph are added to obtain the spatial information hypergraph of the input image, and the spatial information hypergraph is converted by the following formula to obtain the spatial information hypergraph operator: ； in, represents the spatial information hypergraph operator, represents the spatial information hypergraph, and The diagonal matrices representing the hyperedge degree and vertex degree respectively, E is initialized to the identity matrix; The input image is used as an input feature and is input into the spatial hypergraph self-attention module together with the spatial information hypergraph operator. The output feature is then processed together with the spatial information hypergraph operator by at least one spatial hypergraph self-attention module to obtain the spatial feature.

6. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 5, characterized in that: In the spatial hypergraph self-attention module: The input feature is multiplied by the spatial information hypergraph operator to obtain a spatial information feature graph, namely: ； in, , m represents the number of the spatial hypergraph self-attention modules, M represents the total number of the spatial hypergraph self-attention modules, represents the spatial features output by the previous spatial hypergraph self-attention module, Indicates that the feature input to the spatial hypergraph self-attention module is the input image, represents the spatial information feature map, represents the weight coefficient for calculating the spatial information feature map; Extract the query matrix, key matrix and value matrix from the spatial information feature graph, and then perform a multi-head self-attention mechanism operation on the query matrix, the key matrix and the value matrix to obtain the spatial features output by the current spatial hypergraph self-attention module, that is: ； in, represents the spatial features output by the current spatial hypergraph self-attention module, represents the query matrix, represents the bond matrix, represents the value matrix, Represents the operation of the multi-head self-attention mechanism.

7. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 1, characterized in that: In the semantic hypergraph branch in step S2: The abundance matrix is ​​parsed from the input image using a non-negative matrix factorization algorithm; According to the abundance matrix, the semantic hypergraph of the input image is constructed by the following formula: ； in, represents the semantic hypergraph, represents the abundance matrix, represents a complete function for constructing the semantic hypergraph; The semantic hypergraph is converted into a semantic hypergraph operator by the following formula: ； in, represents the semantic hypergraph operator, and The diagonal matrices representing the hyperedge degree and vertex degree respectively, E is initialized to the identity matrix; The input image is used as an input feature and is input into the semantic hypergraph 3D module together with the semantic hypergraph operator. The output feature is then processed together with the semantic hypergraph operator by no less than one semantic hypergraph 3D module to obtain the semantic feature.

8. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 7, characterized in that: In the semantic hypergraph 3D module: The input feature is multiplied by the semantic hypergraph operator to obtain a semantic information feature graph, namely: ； in, , o represents the number of the semantic hypergraph 3D modules, represents the total number of 3D modules of the semantic hypergraph, represents the semantic features output by the previous semantic hypergraph 3D module, Indicates that the feature input to the semantic hypergraph 3D module is the input image, represents the semantic information feature graph, Represents the weight coefficient for calculating the semantic information feature map; The semantic information feature map is then subjected to three-dimensional convolution processing to obtain the semantic features output by the current semantic hypergraph 3D module.

9. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 1, characterized in that: The sensitive band extraction branch in step S2 includes a weight acquisition module and a reference residual module; wherein: In the weight acquisition module, pixel value statistics are performed on each band in the spectral feature, the spatial feature and the semantic feature to estimate the probability distribution of the pixel value of each band; the entropy value of each band in each feature is calculated by the maximum entropy formula, and the band with the largest entropy is selected as the sensitive band of each feature; The reference residual module includes no less than 3 cascaded residual blocks, wherein between two adjacent residual blocks, the sensitive bands of the three features are used as weights, multiplied with the corresponding elements of the input features of the previous residual block, and the multiplied features are added with the input features of the next residual block to obtain output features; each residual block includes no less than 2 cascaded convolutional layers, and the input features of the current residual block are convolved by the convolutional layer and then added with the corresponding elements of the input features of the current residual block to obtain output features; the output features obtained by the last residual block are the high-quality semantic features.

10. The method for super-resolution reconstruction of remote sensing hyperspectral images based on hybrid neural network according to claim 1, characterized in that: In the restoration and reconstruction branch in step S2, The spectral features, the spatial features, the semantic features and the corresponding elements of the high-quality semantic features are added, and the added features are sequentially subjected to at least one convolution operation and a transposed convolution operation, and the obtained features are added to the corresponding elements of the preliminary super-resolution image obtained by upsampling the input image, and the added features are sequentially subjected to at least one convolution operation to obtain the reconstructed remote sensing hyperspectral image.

Images