Method and system for reconstructing hyperspectral image based on RGB image
By parsing endmember and abundance information from RGB images and combining multi-scale and attention mechanisms, hyperspectral images are generated, solving the problem of insufficient reconstruction accuracy in existing methods and achieving higher-precision hyperspectral image reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU UNIV
- Filing Date
- 2026-01-05
- Publication Date
- 2026-05-01
AI Technical Summary
Existing deep learning methods for reconstructing hyperspectral images based on RGB images are too complex, making it difficult to construct high-precision hyperspectral images. Furthermore, they fail to effectively consider the material distribution and pixel mixing features in real HSI images, resulting in reconstruction results that lack physical interpretability and high precision.
By parsing endmember and abundance information of RGB images, and combining multi-scale and attention mechanism modules, endmember spectral representations and abundance information are generated. A physical model is used for spectral mixing mechanism, and finally feature enhancement and fusion are performed to construct a hyperspectral image.
It achieves a more realistic reflection of the distribution patterns of matter, improves the ability to capture spectral features under different lighting and ground object scenarios, solves the problems of blurred edges and loss of details in abundance estimation under complex mixed scenarios, and improves reconstruction accuracy.
Smart Images
Figure CN121961948A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of hyperspectral image reconstruction technology, and in particular to a method and system for reconstructing hyperspectral images based on RGB images. Background Technology
[0002] Hyperspectral images (HSI) contain rich spectral information and are widely used in image classification, object detection, and agricultural planning. However, acquiring hyperspectral images is very difficult due to the greater complexity and cost of hyperspectral sensors, as well as their susceptibility to interference from sensor noise and atmospheric effects.
[0003] In contrast, RGB images are cheaper, faster, and more readily available, so researchers often use RGB images to obtain hyperspectral data. Early methods of reconstructing HSI images from RGB images typically employed extensive prior knowledge, using simple connections to map RGB to HSI images. However, these methods did not account for the complex nonlinear spectral relationships between RGB and HSI images. With the continuous improvement of computing power, more and more researchers are using deep learning to simulate these feature extraction and mapping processes, achieving promising reconstruction results.
[0004] Methods for reconstructing HSI images from RGB images can generally be divided into reconstruction methods based on classical machine learning and reconstruction methods based on deep learning. Reconstruction methods based on classical machine learning primarily rely on classical machine learning theory, using mathematical models to construct the mapping relationship between RGB images and HSI images. Commonly used methods include dictionary learning, sparse representation, and low-rank decomposition. These methods typically utilize external prior information, treating the RGB image as a dimensionality-reduced representation of the HSI image in the spectral dimension, and achieving spectral information completion by constructing a linear or weakly nonlinear relationship between the RGB image and the HSI.
[0005] With the continuous advancement of deep learning theory, some deep network models have been applied to the study of HSI image reconstruction from RGB images, such as Convolutional Neural Networks (CNN), Generative Adversarial Networks (GAN), and Graph Convolutional Networks (GCN). These methods extract feature information by utilizing complex network structures, leading to the widespread proposal of increasingly sophisticated deep learning-based fusion classification methods.
[0006] Current deep learning methods for reconstructing HSI images from RGB images have certain drawbacks: First, existing methods often focus on using complex structures to reconstruct HSI images, neglecting the physical mixing information inherent in real HSI images. This weakens the representation of material distribution in the real world, resulting in reconstructed HSI images lacking physical interpretability. Second, existing methods do not consider the aliasing of hyperspectral images, treating pixels as single-material spectra for direct regression. This ignores the essential characteristic that pixels are often mixtures of multiple material spectra, lacking explicit decomposition of material abundance and endmember composition. Consequently, the models struggle when processing images with similar colors. However, when dealing with "different objects with the same spectrum" or "different spectra with the same object" phenomena, the resolution is weak, making it difficult to construct high-precision hyperspectral data. Third, existing methods usually mix hyperspectral spatial texture features with spectral features, failing to effectively interact between the two features. Furthermore, many methods are limited by single-scale convolution operations, making it difficult to capture long-distance dependencies and multi-scale contextual information in images. They cannot fully utilize global abundance distribution patterns to assist in the reconstruction of local pixels. When facing regions with complex textures or large changes in object scale, they are prone to loss of detail or poor spectral consistency, making it difficult to leverage the advantages of extracted feature information.
[0007] In summary, existing deep learning methods for reconstructing HSI images from RGB images are too complex and make it difficult to construct high-precision hyperspectral images. Summary of the Invention
[0008] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the deep learning method for reconstructing HSI images based on RGB images in the prior art is too complicated and it is difficult to construct high-precision hyperspectral images.
[0009] To address the aforementioned technical problems, this invention provides a method for reconstructing hyperspectral images based on RGB images, comprising:
[0010] Step S1: Select the RGB image in space;
[0011] Step S2: Process the RGB image to obtain the endmember information of the RGB image;
[0012] Step S3: Generate an endmember spectral representation based on the endmember information of the RGB image;
[0013] Step S4: Encode, decode, and perform attention modeling on the RGB image to obtain the abundance information of the RGB image;
[0014] Step S5: Construct a coarse hyperspectral image based on the endmember information and abundance information of the RGB image, and perform feature enhancement and fusion on the coarse hyperspectral image and the RGB image respectively to obtain the final hyperspectral image.
[0015] In one embodiment of the present invention, the method for processing the RGB image in step S2 to obtain the endmember information of the RGB image includes:
[0016] The feature map is obtained by performing a two-dimensional convolution on the RGB image. For feature maps Global average pooling is used to form a two-dimensional feature sequence. This is used to characterize the overall spectral distribution characteristics of the current RGB image patch, and then the two-dimensional feature sequence is... Two-dimensional feature sequences are obtained by inputting into a multilayer perceptron (MLP). Then, the two-dimensional feature sequence Endmember information of the RGB image is obtained through a fully connected layer. .
[0017] In one embodiment of the present invention, the method for generating an endmember spectral representation based on the endmember information of the RGB image in step S3 includes:
[0018] endmember information of RGB images Features are obtained by inputting a fully connected layer and a ReLU activation function. ;Then the features Features are obtained through a multi-head self-attention mechanism. Output features after multi-head self-attention mechanism With end-member information The fused features are obtained by fusing through residual connections. Subsequently, layer normalization was applied to the fused features. After normalization, the fused features are finally processed through a fully connected layer with a sigmoid activation function. Mapping to the hyperspectral channel dimension space yields the final endmember spectral representation. .
[0019] In one embodiment of the present invention, the method for encoding, decoding, and attention modeling the RGB image in step S4 to obtain the abundance information of the RGB image includes:
[0020] During the encoding stage, the input RGB image is first processed using a 3x3 two-dimensional convolution, followed by secondary downsampling to progressively reduce the spatial dimension of the feature map. Furthermore, each downsampled feature undergoes attention modeling via a multi-scale and attention module to obtain the abundance information of the RGB image. The secondary downsampling includes a first downsampling and a second downsampling. The first downsampling uses a 2×2 convolution with a stride of 2, and the second downsampling uses a 4×4 convolution with a stride of 4.
[0021] During the decoding stage, the abundance information of the RGB image is... Two upsampling operations are performed, and the features after each upsampling are then subjected to attention modeling using a multi-scale and attention module to obtain the features. Both upsampling operations are bilinear interpolation upsampling.
[0022] Features An initial abundance feature map is generated by inputting a 1×1 convolution. Simultaneously, the initial abundance feature map is residually concatenated with the RGB image to obtain the abundance map. Finally, the softmax activation function was used to optimize the abundance map. After applying normalization constraints, the abundance information of the RGB image is finally obtained. ;
[0023] The multi-scale and attention module includes three parallel branches. The three parallel branches use three convolutional kernels of different scales to extract input features. The output of each scale convolution is then enhanced with a ReLU activation function to enhance the nonlinear expression. The outputs of the ReLU activation functions of the three parallel branches are respectively input into the attention mechanism module. The outputs of the three attention mechanism modules are then concatenated as the output of the multi-scale and attention module.
[0024] In one embodiment of the present invention, the attention mechanism module includes parallel channel attention and spatial attention to achieve adaptive feature enhancement, wherein,
[0025] The channel attention mechanism extracts channel-level statistical information from the input features using global average pooling and models inter-channel dependencies using 1×1 convolution to obtain the output of the channel attention. ;
[0026] The spatial attention mechanism performs average pooling and max pooling on the channel dimensions of the input features, concatenates the outputs of average pooling and max pooling along the channel dimension, and then applies a 7x7 convolutional kernel and a sigmoid activation function to obtain a spatial weight map. The spatial weight map is then multiplied with the input of the attention mechanism module to obtain the output of the spatial attention mechanism. ;
[0027] The output of the channel attention With the output of the spatial attention The input is concatenated along the channel dimension and the output of the attention mechanism module is obtained by using the ReLU activation function.
[0028] In one embodiment of the present invention, the method for constructing a coarse hyperspectral image based on the endmember information and abundance information of the RGB image in step S5 includes:
[0029] endmember information of RGB images Abundance information of RGB images A rough hyperspectral image is obtained by performing matrix multiplication.
[0030] In one embodiment of the present invention, the method of performing feature enhancement and fusion on the coarse hyperspectral image and the RGB image in step S5 to obtain the final hyperspectral image includes:
[0031] Feature extraction from the RGB image is performed using a 3×3 convolutional layer, followed by batch normalization and GeLU activation to enhance the non-linear expressive power of the features. Finally, the output of the GeLU activation function is passed through a 3×3 convolution and combined with batch normalization to obtain the spatial features. ;
[0032] The coarse hyperspectral image is compressed using a 1×1 convolution, then the compressed features are modeled using a depthwise separable convolution. Finally, the output of the depthwise separable convolution is passed through a 1×1 convolution to restore the feature channel count to its original dimension, resulting in spectral enhancement features. ;
[0033] spatial features and spectral enhancement features By stitching along the channel dimension, the fused features are obtained. ; to integrate features The final hyperspectral image is obtained by adding it element-wise to the input coarse hyperspectral image.
[0034] To address the aforementioned technical problems, this invention provides a system for reconstructing hyperspectral images based on RGB images, comprising:
[0035] Selection module: Used to select RGB images in space;
[0036] First processing module: used to process the RGB image to obtain the end-member information of the RGB image;
[0037] Representation module: used to generate an endmember spectral representation based on the endmember information of the RGB image;
[0038] The second processing module is used to encode, decode, and perform attention modeling on the RGB image to obtain the abundance information of the RGB image.
[0039] Fusion module: used to construct a coarse hyperspectral image based on the endmember information and abundance information of the RGB image, and to perform feature enhancement and fusion on the coarse hyperspectral image and the RGB image respectively to obtain the final hyperspectral image.
[0040] To address the aforementioned technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for reconstructing a hyperspectral image based on an RGB image as described above.
[0041] To address the aforementioned technical problems, the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method for reconstructing a hyperspectral image based on an RGB image as described above.
[0042] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:
[0043] The method for reconstructing hyperspectral images based on RGB images described in this invention utilizes a reconstruction network to effectively decompose RGB images into endmember information and abundance information. By introducing a spectral mixing mechanism in the physical model, it solves the problem of lack of physical interpretability in the reconstruction process of existing methods, thereby enabling a more realistic reflection of the material distribution patterns of hyperspectral images in real-world scenarios.
[0044] This invention addresses the problem of poor scene adaptability caused by the reliance on fixed spectral libraries or manual priors for endmembers in traditional methods. It adaptively mines scene features from RGB images to generate endmember spectral representations, ensuring that potential high-dimensional spectral features can still be accurately captured under different lighting and ground object scenes, thus overcoming the deficiency of insufficient expressive power of fixed spectral libraries in complex scenes.
[0045] This invention encodes, decodes, and models attention in RGB images to obtain the abundance information of RGB images. It can capture long-range contextual dependencies and local detail information of images at different resolutions. At the same time, it combines an attention mechanism module to effectively suppress background noise and highlight key texture features, thereby achieving high-precision estimation of abundance in a scene and solving the problems of blurred edges and loss of details in abundance maps in complex mixed scenes. Attached Figure Description
[0046] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0047] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.
[0049] Example 1
[0050] Reference Figure 1 As shown, the present invention relates to a method for reconstructing a hyperspectral image based on an RGB image, comprising:
[0051] Step S1: Select the RGB image in space;
[0052] Step S2: Process the RGB image to obtain the endmember information of the RGB image;
[0053] Step S3: Generate an endmember spectral representation based on the endmember information of the RGB image;
[0054] Step S4: Encode, decode, and perform attention modeling on the RGB image to obtain the abundance information of the RGB image;
[0055] Step S5: Construct a coarse hyperspectral image based on the endmember information and abundance information of the RGB image, and perform feature enhancement and fusion on the coarse hyperspectral image and the RGB image respectively to obtain the final hyperspectral image.
[0056] The following is a detailed description of this embodiment:
[0057] S2. Processing the RGB image to obtain the endmember information of the RGB image includes the following:
[0058] The RGB image is mapped to a 64-channel high-dimensional space through a 2D convolution to obtain the feature map. The obtained feature map Global average pooling is used to form a two-dimensional feature sequence. This is used to characterize the overall spectral distribution characteristics of the current RGB image patch, and then the two-dimensional feature sequence is... Two-dimensional feature sequences are obtained by inputting into a multilayer perceptron (MLP). Then, the two-dimensional feature sequence By mapping back to the three channels through a fully connected layer, the endmember information of the RGB image is obtained. .
[0059] In this embodiment, the multilayer perceptron (MLP) includes two fully connected layers. The ReLU activation function is set after the first fully connected layer, and the sigmoid activation function is set after the second fully connected layer.
[0060] S3. Generating an endmember spectral representation based on the endmember information of the RGB image includes the following:
[0061] endmember information of RGB images As input, endmember features are mapped to a 64-dimensional latent space through a fully connected layer. Then, the ReLU activation function is used to process the output of the fully connected layer to enhance the non-linear expressive power of the features, resulting in the feature set. Then the features Features are obtained through the output of the Multi-Head Attention (MHA) layer. Output features after multi-head self-attention mechanism With the original end-member information The fused features are obtained by fusing through residual joins (element-wise addition). Subsequently, layer normalization was applied to the fused features. The data is then normalized. Finally, the fused features are processed through a fully connected layer with a Sigmoid activation function. Mapping to the hyperspectral channel dimension space yields the final endmember spectral representation. .
[0062] S4. Encode, decode, and perform attention modeling on the RGB image to obtain the abundance information of the RGB image, including the following:
[0063] During the encoding stage, the input RGB image is first processed using a 3x3 two-dimensional convolution, followed by secondary downsampling to progressively reduce the spatial dimension of the feature map. Furthermore, each downsampled feature undergoes attention modeling via a multi-scale and attention module to obtain the abundance information of the RGB image. .
[0064] The secondary downsampling in this embodiment includes a first downsampling and a second downsampling. The first downsampling uses a 2×2 convolution with a stride of 2, and the second downsampling uses a 4×4 convolution with a stride of 4.
[0065] During the decoding stage, the abundance information of the RGB image is... This implementation employs two upsampling operations, both using bilinear interpolation upsampling (UpSampling 2D). This method progressively restores the spatial resolution of the image and recovers spatial information lost during convolution, then concatenates and fuses these with lower-level features. This allows the network to maintain global semantic information while also capturing local details. Furthermore, the features after each upsampling undergo multi-scale and attention module-based attention modeling to ultimately obtain the desired features. .
[0066] In this embodiment, the multi-scale and attention module includes three parallel branches. The three parallel branches use three different scale convolution kernels (3×3 convolution kernel, 5×5 convolution kernel and 7×7 convolution kernel) to extract the input features. After the output of each scale convolution is enhanced with the ReLU activation function to enhance the nonlinear expressive power, the output of the ReLU activation function of the three parallel branches is respectively input into the attention mechanism module. Then the outputs of the three attention mechanism modules are concatenated as the output of the multi-scale and attention module.
[0067] In this embodiment, the attention mechanism module employs a parallel approach of channel attention and spatial attention to adaptively enhance features. Specifically, channel attention extracts channel-level statistical information from the input features using global average pooling and utilizes 1×1 convolutions to model inter-channel dependencies, thereby highlighting key feature channels relevant to abundance estimation and yielding the output of the channel attention mechanism. Spatial attention performs average pooling (AveragePooling2D) and max pooling (MaxPooling2D) on the channel dimensions of the input features. The outputs of average pooling and max pooling are concatenated along the channel dimension. The concatenated result is then passed through a 7x7 convolutional kernel and a sigmoid activation function to obtain a spatial weight map. The spatial weight map is multiplied by the input of the attention mechanism module (i.e., the output of a 3×3 convolutional kernel, a 5×5 convolutional kernel, or a 7×7 convolutional kernel) to obtain the output of the spatial attention. The output of the channel attention With the output of the spatial attention The input is concatenated along the channel dimension and the output of the attention mechanism module is obtained by using the ReLU activation function.
[0068] In the output stage, the features output in the decoding stage are... An initial abundance feature map is generated through a 1×1 convolution. Simultaneously, the initial abundance feature map is concatenated with the residual of the RGB image (element-wise addition) to obtain the abundance map. This is done to preserve the original spatial structure information and enhance the stability of abundance estimation. Finally, the softmax activation function is used to optimize the abundance map. After applying normalization constraints, the abundance information guided by the RGB image is finally obtained. .
[0069] S5. Transfer endmember information of the RGB image Abundance information of RGB images A rough hyperspectral image is obtained by performing matrix multiplication.
[0070] S5. The final hyperspectral image is obtained by performing feature enhancement and fusion on the coarse hyperspectral image and the RGB image, including the following:
[0071] In the spatial feature enhancement stage, continuous convolutional operations are used to process the features. A 3×3 convolutional layer extracts spatial features from the input RGB image, followed by batch normalization (BN) and GeLU activation to enhance the non-linear expressive power of the features. Subsequently, another 3×3 convolutional layer combined with batch normalization further refines the spatial features, yielding the final spatial features. .
[0072] In the spectral feature enhancement stage, a 1×1 convolution is used to compress the number of channels in the coarse hyperspectral image to reduce channel redundancy and computational complexity. Then, depthwise convolution is used to perform spectral modeling on the compressed features to characterize the relationships between different bands. Finally, a 1×1 convolution is used to restore the number of feature channels to the original dimension, resulting in the spectral enhanced features. .
[0073] After completing spatial feature enhancement and spectral feature enhancement, the two feature streams will be... and Concatenate along the channel dimension to form a fused feature. The spliced fusion features The final hyperspectral image is obtained by adding it element-wise to the input coarse hyperspectral image.
[0074] In this embodiment, a corresponding loss function is constructed for the deep learning model (named the reconstruction network) used in the above steps S2-S5. When training the reconstruction network, step S1 requires selecting a hyperspectral image and an RGB image of the same scene in space. Step S2: The hyperspectral image and the RGB image are cropped to obtain n×n image blocks. For example, if the RGB image is divided into 3×3 RGB image blocks, the hyperspectral image is also correspondingly divided into 3×3 hyperspectral image blocks. RGB image blocks and hyperspectral image blocks at the same position correspond one-to-one in space. Then, any RGB image block is selected, and steps S3-S5 are executed to obtain the generated hyperspectral image block. Based on the generated hyperspectral image block and its corresponding actual hyperspectral image block, the generated... The loss function of the reconstruction network is constructed using actual RGB image patches, and the network is trained using this loss function. The following is the formula for the loss function:
[0075] = + +0.1*
[0076]
[0077]
[0078]
[0079] In the formula, For the total loss, This represents the absolute value of the difference between each corresponding pixel in the generated hyperspectral image patch and the actual hyperspectral image patch. For generated The absolute value of the difference between the actual RGB image block and each corresponding pixel. For fusion features The absolute value of each pixel, It is the ground truth of the hyperspectral image patch. It is the ground truth value of hyperspectral image patches predicted by the reconstruction network. For the true value of the RGB image block, yes and The result obtained through matrix multiplication The total number of pixels in the hyperspectral image patch. This represents the total number of pixels in the RGB image block. This refers to norm calculation.
[0080] Using the aforementioned loss function, the Adam optimizer was selected to update the network parameters, with an initial learning rate set to 4 × 10^(-4). A cosine annealing learning rate adjustment strategy was introduced during training to gradually adjust the learning rate throughout the iteration process. After the model reached stationarity, the RGB image was finally input into the network for end-to-end inference, outputting a high-precision hyperspectral image.
[0081] The experimental comparisons are as follows:
[0082] Table 1 shows an experimental comparison of hyperspectral images reconstructed using the method proposed in this invention and the more commonly used hscnn+ network model:
[0083] Table 1
[0084]
[0085] As can be seen from Table 1, the method of the present invention can better reconstruct hyperspectral images from RGB images and has higher accuracy.
[0086] Example 2
[0087] This embodiment provides a system for reconstructing hyperspectral images based on RGB images, including:
[0088] Selection module: Used to select RGB images in space;
[0089] First processing module: used to process the RGB image to obtain the end-member information of the RGB image;
[0090] Representation module: used to generate an endmember spectral representation based on the endmember information of the RGB image;
[0091] The second processing module is used to encode, decode, and perform attention modeling on the RGB image to obtain the abundance information of the RGB image.
[0092] Fusion module: used to construct a coarse hyperspectral image based on the endmember information and abundance information of the RGB image, and to perform feature enhancement and fusion on the coarse hyperspectral image and the RGB image respectively to obtain the final hyperspectral image.
[0093] Example 3
[0094] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for reconstructing a hyperspectral image based on an RGB image as described in Embodiment 1.
[0095] Example 4
[0096] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method for reconstructing a hyperspectral image based on an RGB image as described in Embodiment 1.
[0097] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.
[0098] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0101] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A method for reconstructing a hyperspectral image based on an RGB image, characterized in that, include: Step S1: Select the RGB image in space; Step S2: Process the RGB image to obtain the endmember information of the RGB image; Step S3: Generate an endmember spectral representation based on the endmember information of the RGB image; Step S4: Encode, decode, and perform attention modeling on the RGB image to obtain the abundance information of the RGB image; Step S5: Construct a coarse hyperspectral image based on the endmember information and abundance information of the RGB image, and perform feature enhancement and fusion on the coarse hyperspectral image and the RGB image respectively to obtain the final hyperspectral image.
2. The method for reconstructing a hyperspectral image based on an RGB image according to claim 1, characterized in that: The method for processing the RGB image to obtain the endmember information of the RGB image in step S2 includes: The feature map is obtained by performing a two-dimensional convolution on the RGB image. For feature maps Global average pooling is used to form a two-dimensional feature sequence. This is used to characterize the overall spectral distribution characteristics of the current RGB image patch, and then the two-dimensional feature sequence is... Two-dimensional feature sequences are obtained by inputting into a multilayer perceptron (MLP). Then, the two-dimensional feature sequence Endmember information of the RGB image is obtained through a fully connected layer. .
3. The method for reconstructing a hyperspectral image based on an RGB image according to claim 1, characterized in that: The method for generating endmember spectral representations based on the endmember information of the RGB image in step S3 includes: endmember information of RGB images Features are obtained by inputting a fully connected layer and a ReLU activation function. ;Then the features Features are obtained through a multi-head self-attention mechanism. Output features after multi-head self-attention mechanism With end-member information The fused features are obtained by fusing through residual connections. Subsequently, layer normalization was applied to the fused features. The data is normalized, and finally, the fused features are processed through a fully connected layer with a sigmoid activation function. Mapping to the hyperspectral channel dimension space yields the final endmember spectral representation. .
4. The method for reconstructing a hyperspectral image based on an RGB image according to claim 1, characterized in that: The method for encoding, decoding, and attention modeling the RGB image in step S4 to obtain the abundance information of the RGB image includes: During the encoding stage, the input RGB image is first processed using a 3x3 two-dimensional convolution, followed by secondary downsampling to progressively reduce the spatial dimension of the feature map. Furthermore, each downsampled feature undergoes attention modeling via a multi-scale and attention module to obtain the abundance information of the RGB image. The secondary downsampling includes a first downsampling and a second downsampling. The first downsampling uses a 2×2 convolution with a stride of 2, and the second downsampling uses a 4×4 convolution with a stride of 4. During the decoding stage, the abundance information of the RGB image is... Two upsampling operations are performed, and the features after each upsampling are then subjected to attention modeling using a multi-scale and attention module to obtain the features. Both upsampling operations are bilinear interpolation upsampling. Features An initial abundance feature map is generated by inputting a 1×1 convolution. Simultaneously, the initial abundance feature map is residually concatenated with the RGB image to obtain the abundance map. Finally, the softmax activation function was used to optimize the abundance map. After applying normalization constraints, the abundance information of the RGB image is finally obtained. ; The multi-scale and attention module includes three parallel branches. The three parallel branches use three convolutional kernels of different scales to extract the input features. The output of each scale convolution is then enhanced with a ReLU activation function to enhance the nonlinear expression. The outputs of the ReLU activation functions of the three parallel branches are respectively input into the attention mechanism module. The outputs of the three attention mechanism modules are then concatenated as the output of the multi-scale and attention module.
5. The method for reconstructing a hyperspectral image based on an RGB image according to claim 4, characterized in that: The attention mechanism module includes parallel channel attention and spatial attention to achieve adaptive feature enhancement, wherein, The channel attention mechanism extracts channel-level statistical information from the input features using global average pooling and models inter-channel dependencies using 1×1 convolution to obtain the output of the channel attention. ; The spatial attention mechanism performs average pooling and max pooling on the channel dimensions of the input features, concatenates the outputs of average pooling and max pooling along the channel dimension, and then applies a 7x7 convolutional kernel and a sigmoid activation function to obtain a spatial weight map. The spatial weight map is then multiplied with the input of the attention mechanism module to obtain the output of the spatial attention mechanism. ; The output of the channel attention With the output of the spatial attention The input is concatenated along the channel dimension and the output of the attention mechanism module is obtained by using the ReLU activation function.
6. The method for reconstructing a hyperspectral image based on an RGB image according to claim 1, characterized in that: The method for constructing a coarse hyperspectral image based on the endmember information and abundance information of the RGB image in step S5 includes: endmember information of RGB images Abundance information of RGB images A rough hyperspectral image is obtained by performing matrix multiplication.
7. The method for reconstructing a hyperspectral image based on an RGB image according to claim 1, characterized in that: The method for performing feature enhancement and fusion on the coarse hyperspectral image and the RGB image in step S5 to obtain the final hyperspectral image includes: Feature extraction from the RGB image is performed using a 3×3 convolutional layer, followed by batch normalization and GeLU activation to enhance the non-linear expressive power of the features. Finally, the output of the GeLU activation function is passed through a 3×3 convolution and combined with batch normalization to obtain the spatial features. ; The coarse hyperspectral image is compressed using a 1×1 convolution, then the compressed features are modeled using a depthwise separable convolution. Finally, the output of the depthwise separable convolution is passed through a 1×1 convolution to restore the feature channel count to its original dimension, resulting in spectral enhancement features. ; spatial features and spectral enhancement features By stitching along the channel dimension, the fused features are obtained. ; to integrate features The final hyperspectral image is obtained by adding it element-wise to the input coarse hyperspectral image.
8. A system for reconstructing hyperspectral images based on RGB images, characterized in that, include: Selection module: Used to select RGB images in space; First processing module: used to process the RGB image to obtain the end-member information of the RGB image; Representation module: used to generate an endmember spectral representation based on the endmember information of the RGB image; The second processing module is used to encode, decode, and perform attention modeling on the RGB image to obtain the abundance information of the RGB image. Fusion module: used to construct a coarse hyperspectral image based on the endmember information and abundance information of the RGB image, and to perform feature enhancement and fusion on the coarse hyperspectral image and the RGB image respectively to obtain the final hyperspectral image.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the method for reconstructing a hyperspectral image based on an RGB image as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method for reconstructing a hyperspectral image based on an RGB image as described in any one of claims 1 to 7.