A hyperspectral image reconstruction method and system

Abstract

The application discloses a hyperspectral image reconstruction method and system, and relates to the technical field of hyperspectral image reconstruction.The method comprises the following steps: building and training a hyperspectral image reconstruction network based on local correlation and global continuity modeling, wherein the network comprises multiple spectral reconstruction units, each spectral reconstruction unit comprises multiple spectral relationship modeling modules, each spectral relationship modeling module comprises an inter-spectral relationship modeling module and an intra-spectral spatial modeling module, and the inter-spectral relationship modeling module and the intra-spectral spatial modeling module are respectively used for extracting inter-spectral relationship features and intra-spectral spatial features of spectral characteristics, wherein the inter-spectral relationship features are fusion features of inter-spectral local correlation features and inter-spectral global continuity features; inputting an RGB image into the hyperspectral image reconstruction network that has been trained, and obtaining a reconstructed hyperspectral image.The application efficiently extracts inter-spectral local correlation features and inter-spectral global continuity features, and realizes higher-quality hyperspectral reconstruction through block-level adaptive feature fusion.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a hyperspectral image reconstruction method and system. Background Technology

[0002] Hyperspectral images (HSI) possess rich spectral information and have wide-ranging applications, particularly in computer vision. Leveraging the detailed spectral information collected across multiple bands, HSI images exhibit a richer semantic expression capability than traditional RGB images. Compared to RGB images, HSI images provide more spectral features, giving them unique advantages and broad application prospects in areas such as image classification and object detection. Especially in industries like remote sensing, agricultural monitoring, environmental protection, and medical imaging, HSI images can provide researchers with more precise information, thereby promoting scientific decision-making. However, the complexity of hyperspectral imaging equipment and the high cost of acquisition hinder the widespread adoption of HSI images, impeding practical application. Therefore, acquiring high-quality hyperspectral images at a lower cost is crucial for promoting the widespread application of hyperspectral technology. Considering the relatively low cost of acquiring RGB images, the Hyperspectral Image Reconstruction task has been proposed. This task involves designing effective algorithms, specific mapping relationships, or generative models to infer or reconstruct hyperspectral images from RGB images. This aims to preserve the spectral information of hyperspectral images as much as possible while maintaining low costs. This is of great significance for promoting the application and popularization of hyperspectral image processing technology.

[0003] Traditional hyperspectral reconstruction algorithms often rely on prior knowledge to establish a mapping relationship between RGB images and hyperspectral images. However, these methods often have limited feature representation capabilities, resulting in less than ideal reconstructed hyperspectral images. With the development of deep learning, using Convolutional Neural Networks (CNNs) for hyperspectral reconstruction has gradually become mainstream. Currently, methods using channel attention to model the correlation between spectral bands have been proposed, which can effectively improve the network's reconstruction performance. In addition, due to the superiority of Transformers in modeling long-range dependencies, methods using Transformers to model the spectral band correlation of spectral features have also been proposed. Compared to algorithms that simply use CNNs, this algorithm can uncover more complex inter-spectral feature correlations, thereby significantly improving the reconstruction quality of hyperspectral images.

[0004] However, existing methods do not fully analyze the inter-spectral relationship characteristics of hyperspectral images, resulting in poor quality of the reconstructed hyperspectral images.

[0005] (1) The interspectral relationship of hyperspectral images shows strong local correlation features. The similarity of spectral images in adjacent ranges is high, while the larger the band interval, the lower the similarity of spectral images. The above-mentioned traditional convolutional neural network-based methods cannot efficiently construct the correlation of the spectral band. At the same time, the Transformer-based reconstruction algorithm cannot effectively emphasize the construction of the correlation of the spectral band in the local range, and there are also problems such as large computational load and low modeling efficiency.

[0006] (2) From the global perspective of the spectrum, the spectral response value of the hyperspectral image at each spatial location changes gradually along the band dimension. This characteristic can be called the global continuous characteristic of the spectral image. Existing reconstruction methods ignore the analysis and modeling of this global continuous characteristic between spectra, which hinders the further improvement of the reconstruction quality of hyperspectral images. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a hyperspectral image reconstruction method and system. Starting from the perspectives of modeling local and global inter-spectral characteristics, a hyperspectral image reconstruction network based on local correlation and global continuity modeling is designed. This network introduces a group spectral correlation modeling module to efficiently model inter-spectral correlation within a local range and extract local inter-spectral correlation features. An adjacent spectral continuity modeling module is also introduced, using memory units to model progressively changing features at the global level, thus modeling global inter-spectral continuity features. Finally, block-level adaptive fusion of the two features is used to simultaneously model from both local and global perspectives, resulting in a more complete inter-spectral correlation model and effectively improving the reconstruction quality of hyperspectral images.

[0008] In a first aspect, the present invention provides a hyperspectral image reconstruction method.

[0009] A hyperspectral image reconstruction method, comprising:

[0010] Acquire RGB images captured by an RGB camera;

[0011] An RGB image is input into a trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein, the training process of the hyperspectral image reconstruction network includes:

[0012] Collect several image pairs consisting of RGB images and hyperspectral images, and construct a dataset;

[0013] A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed. The hyperspectral image reconstruction network includes multiple spectral reconstruction units, each of which includes multiple spectral relationship modeling modules. Each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral spatial modeling module, which are used to extract inter-spectral relationship features and intra-spectral spatial features of spectral features, respectively. The inter-spectral relationship features are fused features that integrate inter-spectral local correlation features and inter-spectral global continuity features.

[0014] The dataset is used for iterative training until the loss function is minimized, resulting in a fully trained reconstructed network.

[0015] Secondly, the present invention provides a hyperspectral image reconstruction system.

[0016] A hyperspectral image reconstruction system, comprising:

[0017] RGB image acquisition module, used to acquire RGB images captured by an RGB camera;

[0018] A hyperspectral image reconstruction module is used to input an RGB image into a trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein, the training process of the hyperspectral image reconstruction network includes:

[0019] Collect several image pairs consisting of RGB images and hyperspectral images, and construct a dataset;

[0020] A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed. The hyperspectral image reconstruction network includes multiple spectral reconstruction units, each of which includes multiple spectral relationship modeling modules. Each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral spatial modeling module, which are used to extract inter-spectral relationship features and intra-spectral spatial features of spectral features, respectively. The inter-spectral relationship features are fused features that integrate inter-spectral local correlation features and inter-spectral global continuity features.

[0021] The dataset is used for iterative training until the loss function is minimized, resulting in a fully trained reconstructed network.

[0022] Thirdly, the present invention also provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the method described in the first aspect.

[0023] Fourthly, the present invention also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps of the method described in the first aspect.

[0024] The above one or more technical solutions have the following beneficial effects:

[0025] 1. This invention provides a hyperspectral image reconstruction method and system. Starting from the perspectives of modeling local and global spectral characteristics, a hyperspectral image reconstruction network based on local correlation and global continuity modeling is designed. This network introduces a group spectral correlation modeling module, which efficiently models inter-spectral correlation within a local range by utilizing a group attention module. This solves the problems of inefficiency, high computational cost, and difficulty in efficiently constructing inter-spectral correlation within a local range in existing hyperspectral reconstruction algorithms. Furthermore, an adjacent spectral continuity modeling module is introduced, employing memory units to model the progressively changing features at the global level and extract global continuity features between spectra. This addresses the problem of existing algorithms neglecting the modeling and representation of global continuity between spectra, effectively improving the quality of reconstructed hyperspectral images.

[0026] 2. In this invention, a block-level adaptive fusion module is introduced to address the local correlation features and global continuity features between spectra. This module effectively integrates global continuity features into spectral features in a block-level adaptive manner. The network can model more complete inter-spectral relationships from both local and global perspectives, thereby further improving the reconstruction quality of hyperspectral images. Attached Figure Description

[0027] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0028] Figure 1 This is a schematic diagram of the structure of a hyperspectral image reconstruction network based on local correlation and global continuity modeling in an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the adjacent spectral continuity modeling module in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the block-level adaptive fusion module in an embodiment of the present invention. Detailed Implementation

[0031] It should be noted that the following detailed descriptions are exemplary and are intended only to describe specific embodiments and to provide further explanation of the invention, and are not intended to limit the scope of exemplary embodiments of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0032] To address the problems of inefficiency, high computational cost, and difficulty in efficiently constructing local inter-spectral correlations in existing hyperspectral reconstruction algorithms, as well as the neglect of modeling inter-spectral global continuity, which hinders further improvement in the quality of reconstructed hyperspectral images, this invention proposes a hyperspectral image reconstruction method. Starting from the perspectives of modeling both local and global inter-spectral characteristics, a hyperspectral image reconstruction network based on Local Correlation and Global Continuity Modeling is designed. Specifically, for local inter-spectral correlations, a Group-wise Spectral Correlation Modeling (GrSCM) module is introduced. This module groups spectral features along the spectral band dimension and utilizes a group-wise attention mechanism to efficiently model local inter-spectral correlations, effectively modeling local spectral characteristics. For global inter-spectral continuity, a Neighborhood-wise Spectral Continuity modeling module is introduced. The modeling (NeSCM) module treats spectral features as a time series along the spectral band dimension and uses memory units composed of 3D convolutional neural networks to recursively model the global progressive variation pattern of the spectrum. This extracts the global continuity of spectral features and effectively models global spectral features. Finally, to explore the inherent complementarity between the two modules, a patch-wise adaptive fusion (PAF) module is proposed. By dividing the features generated by the two modules into patches along the spatial dimension, the PAF effectively integrates global continuous features into spectral features in a patch-wise adaptive manner. This effectively integrates the GrSCM and NeSCM modules, promoting the network to model more complete inter-spectral correlations from both local and global perspectives, thereby effectively improving the reconstruction quality of hyperspectral images.

[0033] Example 1

[0034] First, we define the hyperspectral image reconstruction problem. When an RGB camera captures an image, it is affected by the spectral response function (SRF), and the captured image can be represented as:

[0035]

[0036] Where λ is the wavelength range of visible light from 400nm to 700nm, and c∈{R,G,B} represents the three channels corresponding to the RGB image. I represents the spectral response function (SRF) of camera channel c. c (x,y) and H λ (x, y) represent the visible light image of channel c and the hyperspectral image of band λ, respectively. In practice, they can be discretized as follows:

[0037]

[0038] Where B is the number of bands, the task of reconstructing an RGB image into a hyperspectral image can be seen as the inverse process of formula (2), that is, given I c Solving for the hyperspectral image H in the case of (x,y) λ The process of (x,y).

[0039] Considering that the imaging process of hyperspectral images involves sampling at equal intervals from a continuous spectral range, the acquired spectral response data possesses two key characteristics: local correlation and global continuity. To comprehensively analyze these inter-spectral relationship characteristics of hyperspectral images and improve the quality of hyperspectral image reconstruction, this embodiment proposes a hyperspectral image reconstruction method, which specifically includes:

[0040] Acquire RGB images captured by an RGB camera;

[0041] The RGB image is input into the trained hyperspectral image reconstruction network to obtain the reconstructed hyperspectral image.

[0042] The construction and training of the aforementioned hyperspectral image reconstruction network are crucial to ensuring the quality of hyperspectral reconstruction. The core of the Local Correlation and Global Continuity Modeling Network (CCNet) for hyperspectral image reconstruction proposed in this embodiment is: composed of m spectral reconstruction units. For each spectral reconstruction unit, a customized spectral relationship modeling (CSRM) module is used to capture the intra-spectral spatial and inter-spectral relationship characteristics of spectral features. Specifically, in the customized spectral relationship modeling module, inter-spectral relationships are first modeled. On one hand, a group spectral correlation modeling module is designed to model local inter-spectral correlation through a group-based attention strategy; on the other hand, an adjacent spectral continuity modeling module is constructed to model global inter-spectral continuity, using 3D convolution to recursively model the global spectral asymptotic variation. Finally, a block-level adaptive fusion module is proposed to fuse the features of the above two modules, jointly enhancing the network's ability to model local correlation and global continuity. In addition, after modeling the inter-spectral relationships, the feed-forward network (FFN) in the Transformer architecture is used to model the intra-spectral spatial features. Finally, the customized spectral relationship modeling can completely model the inter-spectral relationship features and intra-spectral spatial features, thereby improving the quality of the final hyperspectral reconstruction.

[0043] The following content provides a more detailed introduction to the construction and training of the hyperspectral image reconstruction network.

[0044] Step S1: Obtain several image pairs consisting of RGB images and hyperspectral images to construct a dataset.

[0045] Step S2: Construct a hyperspectral image reconstruction network based on local correlation and global continuity modeling.

[0046] like Figure 1As shown, the RGB image input to the reconstruction network is subjected to 2D convolution to extract spectral features. These spectral features are then processed by multiple spectral reconstruction units to extract inter-spectral features. The extracted inter-spectral features are then reconstructed using 2D convolution to obtain a hyperspectral image. Specifically, the spectral features are input into the spectral reconstruction unit and encoded through a series of multi-layer spectral relationship modeling modules and downsampling layers (two layers in this embodiment) to extract multi-scale depth spectral features. The depth spectral features extracted by the last layer of the spectral relationship modeling module are decoded through a series of multi-layer upsampling layers and spectral relationship modeling modules (two layers in this embodiment). During the decoding process, the decoded features are merged with the encoded features of the same scale. The features generated by the top-level spectral relationship modeling module are output as the reconstructed hyperspectral features.

[0047] The hyperspectral image reconstruction network includes multiple spectral reconstruction units (three in this embodiment). Each spectral reconstruction unit includes multiple spectral relationship modeling modules. Each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral spatial modeling module, which are used to extract inter-spectral relationship features and intra-spectral spatial features of spectral features, respectively. The inter-spectral relationship features are fused features that integrate inter-spectral local correlation features and inter-spectral global continuity features.

[0048] The above-mentioned interspectral relationship modeling module includes:

[0049] The group spectral correlation modeling module is used to model local-range inter-spectral correlations using a group-based attention strategy and extract local inter-spectral correlation features.

[0050] The adjacent spectral continuity modeling module is used to recursively model the global level of inter-spectral continuity using 3D convolution and extract global inter-spectral continuity features.

[0051] The block-level adaptive fusion module is used to adaptively fuse local correlation features and global continuity features between spectra to generate inter-spectral relationship features.

[0052] Among them, the group spectral correlation modeling module and the adjacent spectral continuity modeling module are two parallel modules, and each module will be described in detail below.

[0053] (1) Group-wise Spectral Correlation Modeling (GrSCM) module

[0054] The multi-head attention mechanism in Transformers exhibits excellent performance in hyperspectral reconstruction tasks due to its advantage in long-range modeling. By establishing similarity associations between each spectral band and all other spectral bands, it can model higher-order correlations between spectra to a certain extent. However, hyperspectral images exhibit strong local correlations along the spectral band dimension, with strong correlations between adjacent bands and low correlations between bands at greater distances. Such methods do not consider this characteristic, introducing unnecessary computation when modeling global spectral band correlations and affecting reconstruction results. To address this issue, this embodiment introduces a group spectral correlation modeling module. The extracted spectral features are input into this module, which divides the original spectral features into multiple sub-regions along the channel dimension. Within these sub-regions, a group attention mechanism is used to model the information associations between local spectral bands, thereby extracting local inter-spectral correlation features. This approach enhances the correlation between local inter-spectral features while reducing unnecessary computational costs and optimization burdens.

[0055] Furthermore, assuming spectral characteristics F in The size is R H×W×C First, the input spectral features are grouped along the channel dimension to obtain grouped features. Where i∈{1,2,…,k} represents the group index, and k is the number of groups; then, the spectral features of each group (i.e., each group feature) are sequentially mapped within the group to the latent feature space, which can be expressed as:

[0056]

[0057] in, This represents the linear mapping parameters.

[0058] Subsequently, for each group of features after feature mapping, the spectral features are divided into a head dimension along the channel dimension to obtain the attention feature vector. Where C in This represents the number of feature channels input to the spectral reconstruction unit, and n represents the number of heads. This head dimension can be interpreted as the feature dimension of the spectral band dimension.

[0059] Next, based on the aforementioned attention feature vectors, group similarity between band features is calculated to generate attention weights. Considering that the similarity between band dimensions can be reflected to some extent by the relative positional relationship of their corresponding spectra, and that the closer two bands are, the higher their similarity should be, a learnable relative spectrum encoding (RSE) is introduced as a prompt for relative spectral position to better model the intrinsic relationship between band feature similarity and their relative spectral relationship when generating attention weights. That is, attention weights are generated by combining group similarity and relative spectrum encoding. Then, the softmax function is used to obtain normalized attention weights, and a weighted summation is used to obtain the final attention features. The specific calculation process can be expressed as follows:

[0060]

[0061] in, This is the relative spectrum encoding for the i-th group.

[0062] Finally, the attention features of each group By merging along the head dimension and group dimension respectively, the spectral features F of the constructed local correlation are obtained. att .

[0063] (2) Neighborhood-wise Spectral Continuity Modeling (NeSCM)

[0064] Hyperspectral image processing, in addition to exhibiting local correlation, also possesses global continuity characteristics. From the first band to the last, the response value of a hyperspectral image changes gradually along the band dimension, rather than abruptly. However, existing methods do not fully utilize and model this spectral property. To address this, this embodiment constructs an adjacent spectral continuity modeling module. The extracted spectral features are input into this module, which mainly consists of two branches: forward modeling and backward modeling. This module can effectively model the global continuity of spectral features recursively, thereby extracting global continuity features between spectra.

[0065] like Figure 2 As shown, firstly, following the method of dividing the head dimension in the aforementioned group spectral correlation modeling module, the input spectral features are similarly divided along the channel dimension, specifying a dimension to represent the band dimension of the spectrum, thus obtaining the features. Next, a window of size s is selected, and for each divided spectral feature, a sliding window operation is performed along the band dimension to extract the continuously distributed spectral features. Where xi ∈R n ×H×W×s Then, these features are sequentially input into the forward and backward branches to extract global continuous features in the two directions respectively.

[0066] Both the forward and backward branches include multiple continuous memory units. For the forward branch, a memory feature m0 is first initialized. Then, each spectral band feature, along with the memory feature, is sequentially input into the continuous memory unit. This process is executed recursively to update the memory feature and generate the output feature for the current stage. Similarly, for the backward branch, the calculation process is similar to the forward branch, except that the input order of the spectral band features is reversed.

[0067] For each continuous memory unit, the input data of the unit is the segment memory feature m of the previous stage. t-1 and the spectral characteristics of the current stage x t Both have the same dimension R. n×H×W×s To effectively model interspectral continuity at the global level, two gating strategies were constructed for each continuous memory unit: a forget gate and an output gate. The forget gate incorporates the memory features m from the previous stage. t-1 and the spectral features of the current stage x t Generate forgetting weights to discard some historical information from memory features; specifically, link m along the feature dimension. t-1 and x t And a 3D convolution with a 1×1×1 kernel is used to generate the forgetting weights for the current unit, as expressed by the following formula:

[0068] s t =Sigmoid(Conv3D(Concat(m t-1 ,x t )))(5)

[0069] Among them, s t The forgetting weight of the current unit.

[0070] Then, the memory feature sequence is updated using this forgetting weight to obtain the memory feature m for the current stage. t ,for:

[0071] m t =s t ×m t-1 +(1-s t )×x t (6)

[0072] For the output gate, the memory features m of the current stage are incorporated. tand spectral characteristics x t Then, it is input into a 3D convolution with a 3×3×s convolution kernel to extract the continuity features of the spectrum, as follows:

[0073] y t =Conv3D(Tanh(m t )+x t (7)

[0074] Among them, y t This represents the output of the t-th continuous memory unit, with dimension R. n×H×W×1 .

[0075] Finally, all continuous features generated by the forward and backward branches are merged and then fused to generate a global asymptotic feature F that can characterize global continuous changes. p This refers to the global continuity feature between spectra, which is used as the output feature of this module. In this process, the adjacent spectral continuity modeling module uses 3D convolution to simultaneously select multiple adjacent spectral bands for iterative calculations. While considering global temporal characteristics, it enhances the ability to model the continuity between vector spectral bands, further enhancing the algorithm's ability to model global spectral characteristics between spectra.

[0076] (3) Patch-wise Adaptive Fusion (PAF) module

[0077] Spectral features F obtained by the group spectral correlation modeling module att The spectral correlation between the characteristics of each band within a local region was effectively constructed, and the global asymptotic feature F output by the adjacent spectral continuity modeling module was also demonstrated. p The global continuity between spectra in the original spectral features was captured. These two sets of features emphasize different aspects and are complementary to each other. Modeling the inherent complementarity between these two sets of features is key to obtaining high-quality spectral features. To this end, this embodiment designs a block-level adaptive fusion module, employing a block-level fusion strategy to combine F... p Adaptive fusion into F att In this process, spectral relationship features are extracted to achieve adaptive fusion of local spatial regions, thereby capturing spectral relationships more comprehensively.

[0078] like Figure 3 As shown, firstly, in conjunction with F att To implement a grouping modeling strategy, a Spectral Information Aggregation (SIA) module was designed. Global continuity features between spectra were input into this module, and a grouping-based method was used to aggregate the input spectral features. The grouping operation followed rules similar to those in the group spectral correlation modeling module. The process is represented as follows: in, Let this be the asymptotic feature of group i. Then, within each group, mean pooling and max pooling are performed on the asymptotic features along the channel dimension. Finally, 2D convolution operations are used to fuse these two pooled features to generate the compressed asymptotic feature of group i. The entire process can be represented as:

[0079]

[0080] Among them, C p Let represent the number of channels for the grouped asymptotic features of the i-th group, and merging all channels along the channel dimension. The compressed asymptotic feature F is obtained. cp The feature size is R H×W×k .

[0081] Secondly, for F respectively att and F cp Perform a 1×1 2D convolution operation to map features, then divide the mapped features into different patches along the spatial dimension to obtain... and Where n represents the block number, and assuming the block size is r×r, then each channel feature will obtain a total of Each block-level feature. Further, the cosine similarity between the block-level features is calculated, which yields the attention weights used for fusion, expressed as:

[0082]

[0083] w n (i,*)=softmax(sim n (i,*))(12)

[0084] Where, sim n (i,j) and w n (i,j) represent the attention features of the i-th channel, respectively. Compression asymptotic features of the j-th channel Feature similarity and attention weights between them.

[0085] Finally, using the calculated attention weights, F is fused. att With F cp The process of generating fused interspectral relationship features can be represented as follows:

[0086]

[0087] Among them, F out Indicates the output characteristics of the module. This represents the sequential stitching of all block-level features along channel dimension i and spatial dimension n. By dividing the spectral features of each band into blocks along the spatial dimension and achieving block-level feature fusion, this embodiment can achieve adaptive feature fusion with spatial regional differences. This fusion process explores F... cp and F att The inherent connections between them enable the network to simultaneously model the local correlation and global continuity of the spectrum, which is beneficial for generating more comprehensive interspectral relationship features.

[0088] Step S3: Iteratively train using the dataset until the loss function is minimized to obtain the trained reconstruction network; the constructed loss function includes the average relative error loss and the spectral difference loss.

[0089] Specifically, let's assume Y and Let the real hyperspectral image and the reconstructed hyperspectral image represent the real image and the reconstructed image, respectively. Then, two types of losses are used to optimize the parameters of the entire network, namely:

[0090] (1) Mean Relative Absolute Error (MRAE): Calculates the pixel-wise relative error between the real hyperspectral image and the reconstructed hyperspectral image, and optimizes the network parameters at the pixel-wise level. The loss formula can be expressed as:

[0091]

[0092] Where N represents the number of pixel values ​​in the hyperspectral image.

[0093] (2) Spectral Difference Loss: The L1 distance between any band in the real hyperspectral image and all other bands in the reconstructed hyperspectral image. Specifically, to further model the interspectral relationships of hyperspectral images, a spectral difference loss is proposed. The L1 distance is used to calculate the difference between a band in the reconstructed hyperspectral image and all other bands, thus obtaining a spectral difference loss of size [missing information]. The difference data is then used; similarly, the spectral differences of the real hyperspectral image are calculated using the same rules. The two sets of difference data are then processed using the L1 distance calculation formula to obtain the final spectral difference loss. The specific loss calculation formula can be expressed as:

[0094]

[0095] Finally, the total loss of the model can be expressed as:

[0096] Loss = L MRAE +γL Dif (16)

[0097] Where γ represents the weight of the spectral difference loss.

[0098] Example 2

[0099] This embodiment provides a hyperspectral image reconstruction system, including:

[0100] RGB image acquisition module, used to acquire RGB images captured by an RGB camera;

[0101] A hyperspectral image reconstruction module is used to input an RGB image into a trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein, the training process of the hyperspectral image reconstruction network includes:

[0102] Collect several image pairs consisting of RGB images and hyperspectral images, and construct a dataset;

[0103] A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed. The hyperspectral image reconstruction network includes multiple spectral reconstruction units, each of which includes multiple spectral relationship modeling modules. Each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral spatial modeling module, which are used to extract inter-spectral relationship features and intra-spectral spatial features of spectral features, respectively. The inter-spectral relationship features are fused features that integrate inter-spectral local correlation features and inter-spectral global continuity features.

[0104] The dataset is used for iterative training until the loss function is minimized, resulting in a trained reconstruction network.

[0105] Example 3

[0106] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps in a hyperspectral image reconstruction method as described above.

[0107] Example 4

[0108] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the hyperspectral image reconstruction method described above.

[0109] The steps and methods involved in Embodiments 2 to 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0110] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0111] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention has been described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A hyperspectral image reconstruction method, characterized in that, include: Acquire RGB images captured by an RGB camera; An RGB image is input into a trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein, the training process of the hyperspectral image reconstruction network includes: Collect several image pairs consisting of RGB images and hyperspectral images, and construct a dataset; A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed. The hyperspectral image reconstruction network includes multiple spectral reconstruction units, each of which includes multiple spectral relationship modeling modules. Each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral spatial modeling module, which are used to extract inter-spectral relationship features and intra-spectral spatial features of spectral features, respectively. The inter-spectral relationship features are fused features that integrate inter-spectral local correlation features and inter-spectral global continuity features. The dataset is used for iterative training until the loss function is minimized, resulting in a fully trained reconstructed network. The interspectral relationship modeling module includes: The group spectral correlation modeling module is used to model local-range inter-spectral correlations using a group-based attention strategy and extract local inter-spectral correlation features. The adjacent spectral continuity modeling module is used to recursively model the global level of inter-spectral continuity using 3D convolution and extract global inter-spectral continuity features. The block-level adaptive fusion module is used to adaptively fuse local correlation features and global continuity features between spectra to generate inter-spectral relationship features. Among them, the group spectral correlation modeling module and the adjacent spectral continuity modeling module are two parallel modules; Spectral features are extracted and input into the group spectral correlation modeling module. A group attention mechanism is used to model the information association between local spectral bands, and local correlation features between spectra are extracted, including: The input spectral features are grouped along the channel dimension to obtain grouped features; Perform intra-group linear mapping on each group feature in turn to map the group features to the latent feature space; For each group of features after feature mapping, a head dimension is divided along the channel dimension to obtain the attention feature vector; the head dimension is the feature dimension of the spectral band dimension. Based on the attention feature vector, the group similarity between spectral band features is calculated, and a learnable relative spectral code is introduced as a cue for relative spectral position; the group similarity and the relative spectral code are combined to generate attention weights; the attention weights are normalized to calculate the attention features. Each group of attention features is merged along the head dimension and the group dimension to obtain the interspectral local correlation features; The adjacent spectral continuity modeling module consists of two branches: forward modeling and backward modeling. The extracted spectral features are input into the adjacent spectral continuity modeling module to extract global continuity features between spectra, including: The input spectral features are divided along the channel dimension, and a dimension is specified to represent the band dimension of the spectrum; For each segmented spectral feature, a sliding window operation is performed along the band dimension to extract continuously distributed spectral features. The continuously distributed spectral features are input into the forward and backward branches to extract the continuous features in the two directions respectively; All continuity features generated by the forward and backward branches are merged and then fused to generate interspectral global continuity features that characterize global continuity changes. The forward and backward branches each include multiple continuous memory units. Memory features are initialized by sequentially inputting each spectral feature along with the memory features into the continuous memory unit, updating the memory features, and generating the output features for the current stage. The forward and backward branches are in reverse order. Each continuous memory unit incorporates a forget gate and an output gate, and the input data for the continuous memory unit is the memory features from the previous stage. and the spectral characteristics at the current stage The forgetting gate combines the memory characteristics of the previous stage. and the spectral characteristics at the current stage Generate forgetting weights and use these forgetting weights to update the memory features, thus obtaining the memory features for the current stage. The output gate integrates the memory features of the current stage. and spectral characteristics The continuity features of the spectrum are extracted and output through 3D convolution. .

2. The hyperspectral image reconstruction method as described in claim 1, characterized in that, The block-level adaptive fusion module employs a block-level fusion strategy to adaptively fuse global continuity features between spectra into local correlation features between spectra, extracting inter-spectral relationship features, including: The global continuous features between spectra are input into the spectral information aggregation module. The input features are aggregated using a grouping-based method. In each group, mean pooling and max pooling are performed on the grouped features along the channel dimension. The pooled features are then fused to generate compressed progressive features. The compressed progressive features and the interspectral local correlation features are feature-mapped, and the mapped features are divided into different blocks along the spatial dimension. The block-level feature cosine similarity between the two features is calculated, and then the attention weights used for fusion are obtained. By using the calculated attention weights, compressed progressive features and inter-spectral local correlation features are fused to generate fused inter-spectral relationship features.

3. The hyperspectral image reconstruction method as described in claim 1, characterized in that, The loss function includes average relative error loss and spectral difference loss; The average relative error loss is the pixel-level relative error between the real hyperspectral image and the reconstructed hyperspectral image; The spectral difference loss is the difference between any band in the real hyperspectral image and the reconstructed hyperspectral image and all other bands. distance.

4. The hyperspectral image reconstruction method as described in claim 1, characterized in that, The spectral features of the RGB image are extracted by 2D convolution. The spectral features are then extracted by multiple spectral reconstruction units to extract inter-spectral features. The extracted inter-spectral features are then reconstructed by 2D convolution to obtain the hyperspectral image. In this process, spectral features are input into the spectral reconstruction unit and encoded by multiple consecutive spectral relationship modeling modules and downsampling layers to extract multi-scale depth spectral features. The depth spectral features extracted by the last layer of the spectral relationship modeling module are decoded by multiple consecutive upsampling layers and spectral relationship modeling modules, and the decoded features are merged with the encoded features of the same scale. The features generated by the top-level spectral relationship modeling module are output as the reconstructed hyperspectral features.

5. A hyperspectral image reconstruction system, characterized in that, include: RGB image acquisition module, used to acquire RGB images captured by an RGB camera; A hyperspectral image reconstruction module is used to input an RGB image into a trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein, the training process of the hyperspectral image reconstruction network includes: Collect several image pairs consisting of RGB images and hyperspectral images, and construct a dataset; A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed. The hyperspectral image reconstruction network includes multiple spectral reconstruction units, each of which includes multiple spectral relationship modeling modules. Each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral spatial modeling module, which are used to extract inter-spectral relationship features and intra-spectral spatial features of spectral features, respectively. The inter-spectral relationship features are fused features that integrate inter-spectral local correlation features and inter-spectral global continuity features. The dataset is used for iterative training until the loss function is minimized, resulting in a fully trained reconstructed network. The interspectral relationship modeling module includes: The group spectral correlation modeling module is used to model local-range inter-spectral correlations using a group-based attention strategy and extract local inter-spectral correlation features. The adjacent spectral continuity modeling module is used to recursively model the global level of inter-spectral continuity using 3D convolution and extract global inter-spectral continuity features. The block-level adaptive fusion module is used to adaptively fuse local correlation features and global continuity features between spectra to generate inter-spectral relationship features. Among them, the group spectral correlation modeling module and the adjacent spectral continuity modeling module are two parallel modules; Spectral features are extracted and input into the group spectral correlation modeling module. A group attention mechanism is used to model the information association between local spectral bands, and local correlation features between spectra are extracted, including: The input spectral features are grouped along the channel dimension to obtain grouped features; Perform intra-group linear mapping on each group feature in turn to map the group features to the latent feature space; For each group of features after feature mapping, a head dimension is divided along the channel dimension to obtain the attention feature vector; the head dimension is the feature dimension of the spectral band dimension. Based on the attention feature vector, the group similarity between spectral band features is calculated, and a learnable relative spectral code is introduced as a cue for relative spectral position; the group similarity and the relative spectral code are combined to generate attention weights; the attention weights are normalized to calculate the attention features. Each group of attention features is merged along the head dimension and the group dimension to obtain the interspectral local correlation features; The adjacent spectral continuity modeling module consists of two branches: forward modeling and backward modeling. The extracted spectral features are input into the adjacent spectral continuity modeling module to extract global continuity features between spectra, including: The input spectral features are divided along the channel dimension, and a dimension is specified to represent the band dimension of the spectrum; For each segmented spectral feature, a sliding window operation is performed along the band dimension to extract continuously distributed spectral features. The continuously distributed spectral features are input into the forward and backward branches to extract the continuous features in the two directions respectively; All continuity features generated by the forward and backward branches are merged and then fused to generate interspectral global continuity features that characterize global continuity changes. The forward and backward branches each include multiple continuous memory units. Memory features are initialized by sequentially inputting each spectral feature along with the memory features into the continuous memory unit, updating the memory features, and generating the output features for the current stage. The forward and backward branches are in reverse order. Each continuous memory unit incorporates a forget gate and an output gate, and the input data for the continuous memory unit is the memory features from the previous stage. and the spectral characteristics at the current stage The forgetting gate combines the memory characteristics of the previous stage. and the spectral characteristics at the current stage Generate forgetting weights and use these forgetting weights to update the memory features, thus obtaining the memory features for the current stage. The output gate integrates the memory features of the current stage. and spectral characteristics The continuity features of the spectrum are extracted and output through 3D convolution. .

6. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, complete the steps of a hyperspectral image reconstruction method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of a hyperspectral image reconstruction method as described in any one of claims 1-4.

Images