A hyperspectral image super-resolution reconstruction method, device, equipment and medium

Abstract

This invention provides a method, apparatus, device, and medium for super-resolution reconstruction of hyperspectral images, relating to the field of remote sensing image processing technology. The invention acquires and preprocesses an input low-resolution hyperspectral image, generates a full-image semantic object mask using a pre-trained semantic segmentation model, thereby obtaining a semantic index map; then, it constructs a hierarchical global spectral prototype memory and aggregates cross-sample category-level spectral priors through momentum updates; next, it designs a deep reconstruction network composed of multiple cascaded semantic prototype interaction units, combining local prototype perception, object pixel cross-attention, window self-attention, and a feedforward network to enhance the features of the input initial feature map; finally, it reconstructs and outputs a high-resolution image through sub-pixel convolution. This invention requires no auxiliary images, effectively alleviates spectral distortion and boundary blurring, and improves reconstruction fidelity and detail representation.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and more specifically, to a method, apparatus, device, and medium for super-resolution reconstruction of hyperspectral images. Background Technology

[0002] Hyperspectral images (HSI), due to their "map-spectrum integration" characteristic, have wide applications in remote sensing, environmental monitoring, precision agriculture, and geological exploration. However, limited by the physical constraints of imaging sensors and signal-to-noise ratio requirements, hyperspectral images are usually acquired with relatively low spatial resolution, severely restricting their potential for detailed ground feature identification and analysis. To improve their spatial detail representation capabilities, hyperspectral image super-resolution (HSI-SR) technology has become a research hotspot in recent years. Existing methods are mainly divided into two categories: one relies on blind unmixing strategies driven by physical models, such as Vertex Component Analysis (VCA), and the other is based on deep learning frameworks, using convolutional neural networks (CNNs) or Transformer structures for end-to-end reconstruction. However, these methods still face several key bottlenecks in practical applications.

[0003] First, traditional blind demixing algorithms are generally based on the assumption of "pure pixels," meaning that there are pixels in an image composed of only a single substance. However, in low-resolution hyperspectral images, pixel mixing is extremely common, leading to severe distortion of the extracted endmember spectra, making it difficult to use as a reliable reconstruction benchmark. Second, current mainstream deep learning methods mostly employ pixel-level mapping mechanisms, lacking the ability to explicitly model high-level semantic objects (such as buildings, vegetation, and water bodies). This neglect of semantic structure makes it impossible for the network to distinguish between maintaining spectral smoothness within the same semantic region and retaining sharp transitions at different semantic boundaries, easily leading to problems such as over-smoothing of the spectrum or spectral crosstalk at boundaries. Furthermore, due to computational resource constraints, existing HSI-SR methods are usually trained on small image patches, and a single image patch often contains only a limited number of semantic categories. This causes the attention mechanism to calculate weights based only on local statistical information, making it difficult to accurately characterize cross-category spectral differences, thus resulting in insensitivity of feature representation to the global semantic distribution.

[0004] In recent years, some studies have attempted to introduce semantic priors to improve HSI reconstruction performance. For example, Zhu (2021) and Wu (2023) explored semantically guided spectral super-resolution methods, but their work largely relies on multi-source auxiliary data (such as high-resolution multispectral images) and is not applicable to single-image HSI super-resolution scenarios. Other studies have applied prototype or memory mechanisms to hyperspectral denoising and classification tasks (such as SERT+prototype networks), but have not built a global spectral prototype library oriented towards semantic categories, nor have they deeply integrated it with the super-resolution reconstruction process. The recently proposed GEWDiff model uses a mask generated by the Segment Anything Model (SAM) as a geometric condition to guide the diffusion process, but it only treats the mask as a spatial shape constraint, failing to mine the semantic category information behind it, and has not established a cross-sample semantic spectral prototype interaction mechanism.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] The present invention aims to provide a method, apparatus, device and medium for super-resolution reconstruction of hyperspectral images, in order to solve the problems of spectral distortion, boundary blurring and loss of detail caused by the common existence of mixed pixels, incomplete semantic distribution of training samples and lack of object-level spatial constraints in existing hyperspectral image super-resolution methods.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A method for super-resolution reconstruction of hyperspectral images includes:

[0009] S1, acquire and preprocess the input low-resolution hyperspectral image;

[0010] S2, after adjusting the preprocessed hyperspectral image to a preset size, input it into the pre-trained semantic segmentation model to generate a full-image semantic object mask, and after non-overlap processing, obtain a semantic index map;

[0011] S3, Construct a global spectral prototype memory library containing a multi-layer global semantic spectral dictionary based on the semantic index graph;

[0012] S4, after dividing the low-resolution hyperspectral image into several small image blocks, the initial feature map corresponding to each image block is extracted through shallow feature extraction;

[0013] S5, the initial feature map input is a deep reconstruction network composed of multiple cascaded semantic prototype interaction units; the deep reconstruction network has the same number of layers as the global spectral prototype memory. Each layer of the semantic prototype interaction unit calls the global semantic spectral dictionary of the corresponding layer, and sequentially performs local prototype calculation, calls the momentum-updated dictionary, attention interaction and feedforward network, and outputs enhanced features.

[0014] S6, the last layer of the deep reconstruction network is upsampled and reconstructed to output a high-resolution hyperspectral image.

[0015] Preferably, the preprocessing includes dimensionality reduction and normalization of the low-resolution hyperspectral image, specifically:

[0016] For the low-resolution hyperspectral image Dimensionality reduction is performed to generate a three-channel image; where, , Image size, This represents the number of spectral bands. It is a real number;

[0017] like If it contains RGB bands, then extract the corresponding RGB bands; if If the spectrum includes bands near RGB, then extract the corresponding bands near RGB; otherwise, extract the bands near RGB. After performing PCA transformation, the first three principal components were extracted;

[0018] The extracted components form a three-channel image. And normalize it.

[0019] Preferably, when adjusting the preprocessed hyperspectral image to a preset size, the method includes upsampling using bicubic interpolation or bilinear interpolation, or dividing the image into blocks using a sliding window, so as to adjust the image size to the input size of the pre-trained semantic segmentation model.

[0020] The non-overlapping processing involves taking the maximum confidence level for overlapping regions.

[0021] Preferably, the global semantic spectral dictionary is used to store a global prototype library of the correspondence between semantic categories and spectral features;

[0022] The semantic index graph is defined as follows: ,in, The maximum number of categories is preset in the semantic index graph. , Image size;

[0023] Each of the global semantic spectrum dictionaries The shape is The global spectral prototype memory consists of L global semantic spectral dictionaries, represented as follows: :in, Number of feature channels;

[0024] Each line in the global semantic spectrum dictionary The global spectral feature prototype corresponding to the k-th semantic category.

[0025] Preferably, it also includes initializing the global spectral prototype memory:

[0026] For the first level dictionary Traverse all low-resolution hyperspectral images in the training set and their corresponding semantic index graphs. For each type of semantics Through semantic index graph Filter out the set of pixels belonging to category k from all images; calculate the average spectral feature of all pixels in the pixel set of category k, and assign the calculated average spectral feature to... The k-th line is completed. Initialization;

[0027] For subsequent dictionary layers, to avoid feature redundancy across multiple dictionary layers, based on Perform differentiated initialization, that is: with Using the base as a basis, subsequent dictionaries are generated through orthogonal transformations to ensure the orthogonality of the spectral prototypes of each dictionary level. The expression is as follows:

[0028] ;

[0029] in, For the first Layer-wide global semantic spectrum dictionary; for The corresponding orthogonal matrix satisfies , Let T be the identity matrix, and T be the transpose symbol.

[0030] Preferably, the deep reconstruction network is composed of L layers of cascaded semantic prototype interaction units (SPIUs), the first... The input to the SPUI layer is the features from the previous layer. Perform the following operations in sequence:

[0031] First, based on the semantic index map and the initial feature map, masked average pooling is used to filter the set of pixels of the kth class in the current image patch. Then to The local spectral prototype is obtained by averaging the features within the region, and its expression is:

[0032] ;

[0033] in, For the local spectral prototype of the k-th semantic object; This is the set of pixels corresponding to the k-th semantic object in the current image patch; For the first Coordinates in the image features input to the layer semantic prototype interaction unit The pixel feature vector of the location; These are pixel coordinates;

[0034] When updating the momentum of the dictionary, the global semantic spectrum dictionary at the corresponding level is updated using the local prototype, as shown in the expression:

[0035] ;

[0036] in, For the first In the layer-wide global semantic spectrum dictionary, the global spectrum prototype corresponding to the k-th semantic object is the k-th row of the global semantic spectrum dictionary; The momentum coefficient;

[0037] like If so, skip the current category. Update steps;

[0038] Then, image features are used as the query vector. The global semantic spectrum dictionary after momentum update consists of key vector K and value vector V. Each pixel feature is matched to its corresponding global semantic spectrum prototype, thus injecting the global spectral prior into the image features to obtain global attention features, expressed as:

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] in, For layer normalization; , , They are query vectors The learnable weight matrix of key vector K and value vector V; For the first Hierarchical global semantic spectrum dictionary; , which is the feature dimension, used for scale normalization of attention; For the first Image features input to the layer semantic prototype interaction unit; For the first Layer-wide global semantic spectrum dictionary; This is the attention weight matrix; For activation functions; This is a global attention feature;

[0044] Finally, the global attention features are used to recover spatial texture through window self-attention, resulting in self-attention features, which are then passed through a feedforward network to output enhanced features, expressed as:

[0045] ;

[0046] ;

[0047] in, This is a characteristic of self-attention; For the first Enhanced features of layer output; For window self-attention; It is a multilayer perceptron, i.e., a feedforward network.

[0048] Preferably, it further includes: performing optimization training using a joint loss constraint function, the expression of which is:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] in, For joint loss constraints; This represents the pixel-level L1 reconstruction loss. The loss is the spectral angle mapping loss, used to constrain spectral shape consistency; The semantic consistency loss is used to ensure that the reconstructed prototype is consistent with the dictionary; This is a true high-resolution hyperspectral image; To reconstruct the output high-resolution hyperspectral image; for Extracted semantic prototype; For the first Layer-wide global semantic spectrum dictionary; , These are the weighting coefficients; , These are the L1 and L2 norms, respectively. It is an inverse cosine function used to calculate the angle between two spectral vectors.

[0054] The present invention also provides a hyperspectral image super-resolution reconstruction apparatus, comprising:

[0055] An image preprocessing unit is used to acquire and preprocess the input low-resolution hyperspectral image;

[0056] The semantic perception unit is used to adjust the preprocessed hyperspectral image to a preset size and then input it into the pre-trained semantic segmentation model to generate a full-image semantic object mask, and after non-overlap processing, obtain a semantic index map;

[0057] The memory building unit is used to build a global spectral prototype memory containing a multi-layer global semantic spectral dictionary based on the semantic index graph;

[0058] The shallow feature extraction unit is used to divide the low-resolution hyperspectral image into several small image blocks and then extract the initial feature map corresponding to each image block through shallow feature extraction.

[0059] A deep network unit is used to input the initial feature map into a deep reconstruction network consisting of multiple cascaded semantic prototype interaction units. The number of layers in the deep reconstruction network is consistent with the number of layers in the global spectral prototype memory. Each layer of the semantic prototype interaction unit calls the global semantic spectral dictionary of the corresponding layer, and sequentially performs local prototype calculation, calls the momentum-updated dictionary, attention interaction and feedforward network, and outputs enhanced features.

[0060] The image reconstruction unit is used to upsample and reconstruct the last layer of the deep reconstruction network to output a high-resolution hyperspectral image.

[0061] The present invention also provides a hyperspectral image super-resolution reconstruction device, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement a hyperspectral image super-resolution reconstruction method as described above.

[0062] The present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium is located, implement a hyperspectral image super-resolution reconstruction method as described above.

[0063] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) This invention introduces the semantic object mask of a general segmentation model (such as Semantic-SAM) into a single HSI super-resolution task for the first time, providing a strong spatial structure prior, enabling the network to distinguish the boundaries and internal regions of different semantic objects, thereby applying object-level spatial constraints during the reconstruction process;

[0065] (2) The present invention constructs a hierarchical global spectral prototype memory library with momentum update, which effectively solves the problem of missing semantic categories in patch image training, realizes cross-sample category-level spectral knowledge transfer, and enables each training sample to access the semantic spectral distribution of the entire dataset.

[0066] (3) The present invention proposes an “object-pixel cross attention” mechanism, which enables each pixel to adaptively query the spectral prototype of all semantic categories. Even when there is an error in the segmentation, robust matching can still be achieved through spectral similarity, avoiding spectral pollution caused by masking errors.

[0067] (4) The present invention adopts a layer-by-layer interactive design, which makes the spectral prototype gradually refined with the network depth, forming a positive feedback loop of "improved image clarity → more accurate prototype extraction → better guided reconstruction". The deep SPUI extracts local prototypes based on the clearer features restored by the shallow network, and then updates the memory bank of the corresponding layer to achieve iterative optimization of the spectral prototype.

[0068] (5) The present invention does not require high-resolution auxiliary images (such as panchromatic or multispectral images) of the same scene throughout the process, and is suitable for typical application conditions in actual remote sensing scenarios where only a single LR HSI image is acquired, thus having stronger practicality and deployment flexibility.

[0069] In summary, this invention constructs an end-to-end single-image hyperspectral image super-resolution framework through semantic object-guided local prototype perception, momentum-driven global memory update, object-pixel cross-attention fusion, and hierarchical feature refinement. This framework significantly improves spatial detail and boundary sharpness while maintaining the consistency of hyperspectral curve shape, effectively suppressing spectral crosstalk and over-smoothing, and providing a new paradigm for hyperspectral image super-resolution that combines semantic perception capabilities with spectral fidelity.

[0070] This invention can effectively integrate high-level semantic object priors and global spectral structure knowledge without relying on high-resolution auxiliary images of the same scene, to achieve single-image HSI super-resolution reconstruction with both high spatial detail and high spectral fidelity. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0072] Figure 1 This is a schematic diagram of a hyperspectral image super-resolution reconstruction method provided in Example 1.

[0073] Figure 2 This is a flowchart illustrating a hyperspectral image super-resolution reconstruction method provided in Example 1.

[0074] Figure 3 This is a schematic diagram of the internal structure of the semantic prototype interaction unit provided in Implementation Example 1.

[0075] Figure 4 This is a schematic diagram of a hyperspectral image super-resolution reconstruction device provided in Embodiment 2.

[0076] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0078] Example 1

[0079] Embodiment 1 of the present invention provides a method for super-resolution reconstruction of hyperspectral images, which can be implemented by a hyperspectral image super-resolution reconstruction device (hereinafter referred to as reconstruction device), and in particular, executed by one or more processors within the reconstruction device.

[0080] In this embodiment, the reconstruction device may be an electronic device equipped with a processor, which carries a computer program for the hyperspectral image super-resolution reconstruction method and the computer program can be executed, such as a computer, smartphone, smart tablet, workstation, etc., without limitation.

[0081] like Figures 1-2 As shown, a hyperspectral image super-resolution reconstruction method includes steps S1 to S6.

[0082] S1: Acquire and preprocess the input low-resolution hyperspectral image.

[0083] This embodiment uses a single low-resolution hyperspectral image (LR HSI) as input, and aims to reconstruct a high-resolution hyperspectral image (HR HSI) with high spatial resolution and high spectral fidelity. The entire process does not rely on high-resolution auxiliary images (such as panchromatic or multispectral images) of the same scene.

[0084] In this step, the preprocessing includes dimensionality reduction and normalization of the low-resolution hyperspectral image, which can be achieved through the following operations:

[0085] For the low-resolution hyperspectral image Dimensionality reduction is performed to generate a three-channel image; where, , Image size, This represents the number of spectral bands. It is a real number;

[0086] like If it contains RGB bands, then extract the corresponding RGB bands; if If the spectrum includes bands near RGB, then extract the corresponding bands near RGB; otherwise, extract the bands near RGB. After performing PCA transformation, the first three principal components were extracted;

[0087] The extracted components form a three-channel image. and normalize to Interval.

[0088] S2, after adjusting the preprocessed hyperspectral image to a preset size, input it into the pre-trained semantic segmentation model to generate a full-image semantic object mask, and after non-overlap processing, obtain a semantic index map.

[0089] Because pre-trained semantic segmentation is resolution-sensitive, when the height (H) and width (W) of the hyperspectral image are larger than the input size of the pre-trained semantic segmentation model, a sliding window can be used for segmentation; when they are smaller than the input size, upsampling is performed. Bicubic or bilinear interpolation can be used for upsampling to normalize the image. Image upscaling to the recommended input size of a pre-trained semantic segmentation model (such as Semantic-SAM) (e.g.) ).

[0090] A mapping relationship can be established between the output label set of the pre-trained segmentation model and the category set of the pre-trained semantic segmentation model in the global spectral prototype memory. This can be used to connect the spatial category information of semantic segmentation with the feature representation information of the spectral prototype, achieving accurate alignment and complementarity between the two, thereby improving the performance of hyperspectral image demixing, super-resolution or classification tasks.

[0091] Image input pre-trained semantic segmentation model obtains A binary mask. Downsample all masks back to their original size. Then, non-overlapping processing is performed (for example, maximizing the confidence score of overlapping regions based on the output of the pre-trained semantic segmentation model) to obtain the semantic index map. ,in, The maximum number of categories preset in the semantic index graph is used to initialize the memory. , This refers to the image size.

[0092] The semantic index map is generated before training based on a low-resolution hyperspectral image with a complete field of view. During training, image patches and corresponding index map patches are synchronously input into the network.

[0093] S3, Based on the semantic index graph, construct a global spectral prototype memory containing a multi-layer global semantic spectral dictionary.

[0094] In this embodiment, the global semantic spectral dictionary is used to store a global prototype library of the correspondence between semantic categories and spectral features.

[0095] Each of the global semantic spectrum dictionaries The shape is The global spectral prototype memory is then set to consist of L global semantic spectral dictionaries, represented as follows: :in, The number of feature channels;

[0096] Each line in the global semantic spectrum dictionary The global spectral feature prototype corresponding to the k-th semantic category.

[0097] When initializing the global spectral prototype memory, the average spectral features of all data in the training set are used. Initialization is performed, and subsequent levels of memory are initialized using orthogonal initialization or mean initialization, specifically as follows:

[0098] For the first level dictionary Traverse all low-resolution hyperspectral images in the training set and their corresponding semantic index graphs. For each type of semantics Through semantic index graph Filter out the set of pixels belonging to category k from all images; calculate the average spectral feature of all pixels in the pixel set of category k, and assign the calculated average spectral feature to... The k-th line is completed. Initialization.

[0099] For subsequent dictionary layers, to avoid feature redundancy across multiple dictionary layers, based on Differential initialization can be performed using orthogonal or mean initialization methods.

[0100] Orthogonal initialization: Using the base as a basis, subsequent dictionaries are generated through orthogonal transformations to ensure the orthogonality of the spectral prototypes of each dictionary level. The expression is as follows:

[0101] ;

[0102] in, For the first Layer-wide global semantic spectrum dictionary; for The corresponding orthogonal matrix satisfies , Let T be the identity matrix, and T be the transpose symbol.

[0103] Mean initialization: direct reuse The mean statistics will Initialize to The same mean and small random perturbations (balancing stability and diversity).

[0104] The initialized multi-level dictionary is stored in the network as learnable parameters or a buffer.

[0105] The construction process of the global spectral prototype memory is essentially "semantic constraints (index graph) → structural definition (multi-layer dictionary) → prior initialization (training set statistics) → storage binding (network parameters)". The semantic index graph determines the "category dimension" of the dictionary, ensuring that the dictionary is aligned with the semantic space of the image. The multi-layer structure adapts to the hierarchical feature interaction of the semantic prototype interaction unit SPUI, with each layer of the dictionary corresponding to the semantic prototype interaction of one layer of SPUI. Initialization introduces the global spectral prior of the training set, transforming the dictionary from an "empty template" into a physically meaningful "semantic-spectral mapping library".

[0106] S4, after dividing the low-resolution hyperspectral image into several small image blocks, the initial feature map corresponding to each image block is extracted through shallow feature extraction.

[0107] In this embodiment, the low-resolution hyperspectral image is divided into several small image blocks in a non-overlapping manner. Then, each small image block is input into a shallow feature extraction network (usually composed of 1-2 convolutional layers + activation functions) to output the corresponding initial feature map.

[0108] S5, the initial feature map input is a deep reconstruction network composed of multiple cascaded semantic prototype interaction units; the deep reconstruction network has the same number of layers as the global spectral prototype memory. Each layer of the semantic prototype interaction unit calls the global semantic spectral dictionary of the corresponding layer, and sequentially performs local prototype calculation, calls the momentum-updated dictionary, attention interaction and feedforward network, and outputs enhanced features.

[0109] In this embodiment, the deep reconstruction network is composed of L layers of cascaded semantic prototype interaction units (SPIUs). The input to the SPUI layer is the features from the previous layer. ,like Figure 3 As shown, the local prototype calculation, the dictionary updated with momentum, attention interaction and feedforward network are executed sequentially.

[0110] (1) Local prototype computation (local sensing)

[0111] First, local prototype computation (i.e., local perception) is performed, based on the semantic index graph M, for the input features. Perform in-class regional averaging to extract the local spectral prototype of the k-th class. The specific steps are as follows:

[0112] Based on the semantic index map M and the initial feature map, Masked Pooling is used to filter the set of pixels of the kth class in the current image patch. Then on Internal features Taking the average yields the local prototype. The expression is:

[0113] ;

[0114] in, For the local spectral prototype of the k-th semantic object; This is the set of pixels corresponding to the k-th semantic object in the current image patch; For the first Coordinates in the image features input to the layer semantic prototype interaction unit The pixel feature vector of the location; These are pixel coordinates.

[0115] This step leverages the spatial constraints of pre-trained semantic segmentation models (such as Semantic-SAM) to eliminate random noise in low-resolution (LR) images through region averaging, achieving a more robust "object-level unmixing" than VCA.

[0116] (2) Dictionary momentum update (memory and update)

[0117] To address the issue of missing classes during image patch training, local prototypes are used to update the corresponding level's Global Semantic Spectrum Dictionary (GSSD), aggregating statistical characteristics of the entire dataset. Local Prototypes After being transformed into features at the adaptive dictionary dimension by MLP, local prototypes are integrated into the corresponding level of the global semantic spectrum dictionary through momentum update rules. The expression is:

[0118] ;

[0119] in, For the first In the hierarchical global semantic spectrum dictionary, the global spectrum prototype corresponding to the k-th semantic object is the k-th row of the global semantic spectrum dictionary; Let be the momentum coefficient; if If so, skip the current category. The update steps.

[0120] Through momentum updates, each category prototype in the memory gradually aggregates the statistical properties of the entire dataset, overcoming the bias of a single training sample.

[0121] Dictionary momentum updates are performed during the training phase; during the inference phase, the global semantic spectrum dictionary, which is fixed after training, is invoked.

[0122] (3) Attention interaction (object pixel cross attention)

[0123] Image features query vector Global semantic spectrum dictionary updated with momentum Given a key vector K and a value vector V, each pixel feature is matched to the corresponding global semantic spectrum prototype, thereby injecting the global spectral prior into the image features to obtain global attention features.

[0124] Specifically, input features After LN (layer normalization), multiply by the weight Obtain the query vector Q;

[0125] Global Semantic Spectrum Dictionary Multiply by weight , The key vector K and the value vector V are obtained respectively;

[0126] Calculate the similarity between Q and K, and obtain the attention weight matrix using Softmax. ;

[0127] Attention weight moments After weighted summation with V, and compared with the original features Residual connections yield global attention features. The expressions are as follows:

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] in, For layer normalization; , , They are query vectors The learnable weight matrix of key vector K and value vector V; For the first Hierarchical global semantic spectrum dictionary; , which is the feature dimension, used for scale normalization of attention; For the first Image features input to the layer semantic prototype interaction unit; For the first Layer-wide global semantic spectrum dictionary; This is the attention weight matrix; For activation functions; This is a global attention feature.

[0133] The cross-attention mechanism employs a global full-scale query strategy. That is, the similarity of image pixel features at any location is calculated with all semantic category prototypes in the global semantic spectral dictionary. Even in cases of local bias or classification errors in the semantic mask, image pixels can still match the correct semantic prototype in the dictionary based on their spectral feature similarity, improving the network's robustness to segmentation noise.

[0134] (4) Window space self-attention

[0135] Finally, to recover any blocky effects that might occur after injecting the prototype and to capture spatial textures, a window size of [size missing] was used. Multi-head self-attention (W-MSA).

[0136] The global attention features are obtained by restoring spatial texture through window self-attention, and then the enhanced features are output through a feedforward network.

[0137] Specifically, global attention features After normalization by an LN layer, Q / K / V values ​​for multi-head self-attention within a window are generated. Self-attention is then calculated within the window (e.g., 8×8), and the output is concatenated with the original feature residuals to obtain... The expression is:

[0138] ;

[0139] in, This is a characteristic of self-attention; For window self-attention.

[0140] (5) Feedforward network

[0141] Nonlinear transformations are applied to the features to enhance their expressive power. Specifically, After passing through an LN, the data is input into an MLP (Multilayer Perceptron); the MLP output is concatenated with the original feature residuals to obtain the first... Output features of layer SPUI The expression is:

[0142] ;

[0143] in, For the first Enhanced features of layer output; It is a multilayer perceptron, i.e., a feedforward network.

[0144] Compared to global attention, this step reduces GPU memory usage, ensuring that GPU memory does not run out during patch training.

[0145] This embodiment employs a hierarchical memory design, with each SPUI layer having its own independent memory. The deep memory is updated based on clearer features recovered from the shallower network. For example, the second SPUI layer receives features enhanced by the first SPUI layer, resulting in a local prototype perception result closer to the true spectral distribution. Consequently, the updated second-layer memory prototype is more refined. This layer-by-layer interaction forms a positive feedback loop of "improved image clarity → more accurate prototype extraction → better guided reconstruction," achieving iterative refinement of the spectral prototype. After processing the L-layer SPUI, the deep features are fed into the upsampling and reconstruction output module.

[0146] S6, the last layer of the deep reconstruction network is upsampled and reconstructed to output a high-resolution hyperspectral image.

[0147] This step employs a sub-pixel convolution structure: first, the number of feature channels is expanded to s²·B (s is the upsampling factor, such as 4; B is the original number of hyperspectral bands) through 1×1 convolution; then, the channel dimensions are rearranged into spatial dimensions through the pixel shuffle operation, and finally, a high-resolution hyperspectral image cube with a size of sH×sW×B is output.

[0148] In a preferred embodiment, the method further includes: performing optimization training using a joint loss constraint function, expressed as:

[0149] ;

[0150] ;

[0151] ;

[0152] ;

[0153] in, For joint loss constraints; This represents the pixel-level L1 reconstruction loss. The loss is the spectral angle mapping loss, used to constrain spectral shape consistency; The semantic consistency loss is used to ensure that the reconstructed prototype is consistent with the dictionary; This is a true high-resolution hyperspectral image; To reconstruct the output high-resolution hyperspectral image; for Extracted semantic prototype; For the first Hierarchical global semantic spectrum dictionary; , These are the weighting coefficients; , These are the L1 and L2 norms, respectively. It is an inverse cosine function used to calculate the angle between two spectral vectors.

[0154] In this embodiment, the Semantic-SAM segmentation model processing and semantic index graph in step S2 can be generated offline and do not participate in gradient calculation. Therefore, the inference speed depends only on the computational cost of the deep reconstruction network.

[0155] This embodiment was validated on typical remote sensing datasets (such as Pavia University and Indian Pines). The input LR HSI image patch size was 32×32×B (offline image segmentation and global dictionary generation can be performed on unsegmented images), the upsampling factor s=4, and the output HR HSI was 128×128×B. The reconstruction results are significantly better than existing methods in terms of spatial details (such as building edges and vegetation textures) and spectral fidelity (spectral angle mapper index is reduced by more than 5%), which fully demonstrates the effectiveness and practicality of the present invention in single-image HSI super-resolution tasks.

[0156] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0157] (1) More faithful spectral features, restoring the true spectral curve. Existing techniques (such as traditional interpolation and ordinary deep learning methods) often suffer from "spectral distortion" problems (such as distorted spectral curve shape and loss of correlation between bands). This method directly constrains the angle between the reconstructed spectrum and the true spectrum through spectral angle mapping loss; and injects the spectral prototype prior at the dataset level through a global semantic spectral dictionary. Finally, after high-resolution reconstruction, the spectral curve is highly consistent with the spectral characteristics of the real scene, avoiding the "pseudo-spectrum" problem.

[0158] (2) More refined spatial texture while also considering semantic boundaries. Existing hyperspectral super-resolution methods mostly focus on "pixel-level restoration," which easily leads to problems such as blurred spatial texture and unclear semantic boundaries (such as the edges of vegetation / buildings). This method guides the clear definition of the semantic spatial boundaries of the image through the Semantic-SAM semantic mask; and combines "local pixel features + global semantic spectral prototype" through the Semantic Prototype Interaction Unit (SPIU) to ensure the spatial continuity of semantic objects while restoring texture. The final reconstructed high-resolution image not only retains fine spatial details, but also accurately distinguishes the boundaries of different semantic objects.

[0159] (3) More robust to noise in low-resolution images. Low-resolution hyperspectral images (LR HSI) often suffer from sensor noise, degradation, and blurring, and existing methods are weak in resisting noise interference. This method eliminates local random noise by averaging pixel features within semantic categories through local prototype calculation; and reduces the impact of single-sample noise by aggregating the spectral statistical properties of the entire dataset through momentum updates of the global semantic spectral dictionary. Even if the input LR HSI is noisy, it can still output stable and reliable high-resolution results.

[0160] (4) Higher computational efficiency and lower memory usage. Existing attention-based hyperspectral super-resolution methods (such as global self-attention) often suffer from "large computational load and memory explosion" due to the high dimensionality of hyperspectral data. This method replaces global attention with window-space self-attention, reducing the computational complexity from "the square of the number of pixels" to "the pixel level within the window"; through object-pixel cross-attention with "the number of semantic categories (e.g., 16)" as the key dimension, the computational complexity is only (N is the number of pixels). While ensuring performance, the computation and memory costs have been significantly reduced, and training / inference of larger image patches is supported.

[0161] Example 2

[0162] like Figure 4 As shown, the second embodiment of the present invention also provides a hyperspectral image super-resolution reconstruction apparatus, comprising:

[0163] An image preprocessing unit is used to acquire and preprocess the input low-resolution hyperspectral image;

[0164] The semantic perception unit is used to adjust the preprocessed hyperspectral image to a preset size and then input it into the pre-trained semantic segmentation model to generate a full-image semantic object mask, and after non-overlap processing, obtain a semantic index map;

[0165] The memory building unit is used to build a global spectral prototype memory containing a multi-layer global semantic spectral dictionary based on the semantic index graph;

[0166] The shallow feature extraction unit is used to divide the low-resolution hyperspectral image into several small image blocks and then extract the initial feature map corresponding to each image block through shallow feature extraction.

[0167] A deep network unit is used to input the initial feature map into a deep reconstruction network consisting of multiple cascaded semantic prototype interaction units. The number of layers in the deep reconstruction network is consistent with the number of layers in the global spectral prototype memory. Each layer of the semantic prototype interaction unit calls the global semantic spectral dictionary of the corresponding layer, and sequentially performs local prototype calculation, calls the momentum-updated dictionary, attention interaction and feedforward network, and outputs enhanced features.

[0168] The image reconstruction unit is used to upsample and reconstruct the last layer of the deep reconstruction network to output a high-resolution hyperspectral image.

[0169] Example 3

[0170] The third embodiment of the present invention also provides a hyperspectral image super-resolution reconstruction device, which includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement the hyperspectral image super-resolution reconstruction method as described above.

[0171] Example 4

[0172] The fourth embodiment of the present invention also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor of the device in which the computer-readable storage medium is located, implement the hyperspectral image super-resolution reconstruction method described above.

[0173] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for super-resolution reconstruction of hyperspectral images, characterized in that, include: S1, acquire and preprocess the input low-resolution hyperspectral image; S2, after adjusting the preprocessed hyperspectral image to a preset size, input it into the pre-trained semantic segmentation model to generate a full-image semantic object mask, and after non-overlap processing, obtain a semantic index map; S3, construct a global spectral prototype memory library containing a multi-layer global semantic spectral dictionary based on the semantic index graph; wherein, the global semantic spectral dictionary is used to store the global prototype library of the correspondence between semantic categories and spectral features; S4, after dividing the low-resolution hyperspectral image into several small image blocks, the initial feature map corresponding to each image block is extracted through shallow feature extraction; S5, the initial feature map input is a deep reconstruction network composed of multiple cascaded semantic prototype interaction units; the deep reconstruction network has the same number of layers as the global spectral prototype memory. Each layer of the semantic prototype interaction unit calls the global semantic spectral dictionary of the corresponding layer, and sequentially performs local prototype calculation, calls the momentum-updated dictionary, attention interaction and feedforward network, and outputs enhanced features. S6, the last layer of the deep reconstruction network's enhanced features are upsampled and reconstructed to output a high-resolution hyperspectral image; The deep reconstruction network is composed of L layers of cascaded semantic prototype interaction units (SPIUs). The input to the SPUI layer is the features from the previous layer. Perform the following operations in sequence: First, based on the semantic index map and the initial feature map, masked average pooling is used to filter the set of pixels belonging to semantic category k in the current image patch. Then to The local spectral prototype is obtained by averaging the features within the region, and its expression is: ； in, For the local spectral prototype of semantic category k; This is the set of pixels corresponding to semantic category k in the current image patch; For the first Pixel coordinates in the image features input to the layer semantic prototype interaction unit The pixel feature vector of the location; These are pixel coordinates; When updating the momentum of the dictionary, the global semantic spectrum dictionary at the corresponding level is updated using the local prototype, as shown in the expression: ； in, For the first In the layer-wide global semantic spectrum dictionary, the global spectrum prototype corresponding to semantic category k is the kth row of the global semantic spectrum dictionary; The momentum coefficient; like Then skip the current semantic category. Update steps; Then, image features are used as the query vector. The global semantic spectrum dictionary after momentum update consists of key vector K and value vector V. Each pixel feature is matched to its corresponding global semantic spectrum prototype, thus injecting the global spectral prior into the image features to obtain global attention features, expressed as: ； ； ； ； in, For layer normalization; , , They are query vectors The learnable weight matrix of key vector K and value vector V; For the first Layer-wide global semantic spectrum dictionary; , which is the feature dimension, used for scale normalization of attention; For the first Image features input to the layer semantic prototype interaction unit; This is the attention weight matrix; For activation functions; This is a global attention feature; Finally, the global attention features are used to recover spatial texture through window self-attention, resulting in self-attention features, which are then passed through a feedforward network to output enhanced features, expressed as: ； ； in, This is a characteristic of self-attention; For the first Enhanced features of layer output; For window self-attention; It is a multilayer perceptron.

2. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that... The preprocessing includes dimensionality reduction and normalization of the low-resolution hyperspectral image, specifically: For the low-resolution hyperspectral image Dimensionality reduction is performed to generate a three-channel image; where, , Image size, This represents the number of spectral bands. It is a real number; like If it contains RGB bands, then extract the corresponding RGB bands; if If the spectrum includes bands near RGB, then extract the corresponding bands near RGB; otherwise, extract the bands near RGB. After performing PCA transformation, the first three principal components were extracted; The extracted components form a three-channel image. And normalize it.

3. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that... When adjusting the preprocessed hyperspectral image to a preset size, methods include using bicubic interpolation or bilinear interpolation for upsampling, or using a sliding window to divide the image into blocks, in order to adjust the image size to the input size of the pre-trained semantic segmentation model. The non-overlapping processing involves taking the maximum confidence level for overlapping regions.

4. The hyperspectral image super-resolution reconstruction method according to claim 1, characterized in that... , The semantic index graph is defined as follows: ,in, The maximum number of categories is preset in the semantic index graph. , Image size; The shape of each of the global semantic spectrum dictionaries is as follows: The global spectral prototype memory consists of L global semantic spectral dictionaries, represented as follows: :in, The number of feature channels; For the first Layer-wide global semantic spectrum dictionary; Each line in the global semantic spectrum dictionary The global spectral feature prototype corresponding to semantic category k.

5. The hyperspectral image super-resolution reconstruction method according to claim 4, characterized in that... This also includes the initialization of the global spectral prototype memory: For the first level dictionary Traverse all low-resolution hyperspectral images in the training set and their corresponding semantic index graphs. ; For each semantic category, use a semantic index graph Filter out the set of pixels in all images that belong to semantic category k; Calculate the average spectral feature of all pixels within the pixel set of semantic category k, and assign the calculated average spectral feature to... The k-th line is completed. Initialization; For subsequent dictionary layers, to avoid feature redundancy across multiple dictionary layers, based on Perform differentiated initialization, that is: with Using the base as a basis, subsequent dictionaries are generated through orthogonal transformations to ensure the orthogonality of the spectral prototypes of each dictionary level. The expression is as follows: ； in, For the first Layer-wide global semantic spectrum dictionary; for The corresponding orthogonal matrix satisfies , Let T be the identity matrix, and T be the transpose symbol.

6. The method for super-resolution reconstruction of hyperspectral images according to claim 1, characterized in that... It also includes: using a joint loss constraint function for optimization training, the expression of which is: ； ； ； ； in, For joint loss constraints; This represents the pixel-level L1 reconstruction loss. The loss is the spectral angle mapping loss, used to constrain spectral shape consistency; The semantic consistency loss is used to ensure that the reconstructed prototype is consistent with the dictionary; For true high-resolution hyperspectral images; To reconstruct the output high-resolution hyperspectral image; for Extracted semantic prototype; For the first Layer-wide global semantic spectrum dictionary; , These are the weighting coefficients; , These are the L1 and L2 norms, respectively. It is an inverse cosine function used to calculate the angle between two spectral vectors.

7. A hyperspectral image super-resolution reconstruction apparatus, used to implement the hyperspectral image super-resolution reconstruction method as described in any one of claims 1-6, characterized in that, include: An image preprocessing unit is used to acquire and preprocess the input low-resolution hyperspectral image; The semantic perception unit is used to adjust the preprocessed hyperspectral image to a preset size and then input it into the pre-trained semantic segmentation model to generate a full-image semantic object mask, and after non-overlap processing, obtain a semantic index map; The memory building unit is used to build a global spectral prototype memory containing a multi-layer global semantic spectral dictionary based on the semantic index graph; The shallow feature extraction unit is used to divide the low-resolution hyperspectral image into several small image blocks and then extract the initial feature map corresponding to each image block through shallow feature extraction. A deep network unit is used to input the initial feature map into a deep reconstruction network consisting of multiple cascaded semantic prototype interaction units. The number of layers in the deep reconstruction network is consistent with the number of layers in the global spectral prototype memory. Each layer of the semantic prototype interaction unit calls the global semantic spectral dictionary of the corresponding layer, and sequentially performs local prototype calculation, calls the momentum-updated dictionary, attention interaction and feedforward network, and outputs enhanced features. The image reconstruction unit is used to upsample and reconstruct the last layer of the deep reconstruction network to output a high-resolution hyperspectral image.

8. A hyperspectral image super-resolution reconstruction device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that can be executed by the processor to implement a hyperspectral image super-resolution reconstruction method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor of the device on which the computer-readable storage medium resides, implement a hyperspectral image super-resolution reconstruction method as described in any one of claims 1-6.

Images