Hyperspectral image reconstruction method and system

Abstract

The invention discloses a hyperspectral image reconstruction method and system, and relates to the technical field of hyperspectral image reconstruction, and the method comprises the steps: building and training a hyperspectral image reconstruction network based on local correlation and global continuity modeling, the network comprises a plurality of spectrum reconstruction units, each spectrum reconstruction unit comprises a plurality of spectrum relation modeling modules, and each spectrum relation modeling module comprises a plurality of spectrum relation modeling modules; each spectral relationship modeling module comprises an inter-spectral relationship modeling module and an intra-spectral spatial modeling module which are respectively used for extracting inter-spectral relationship characteristics and intra-spectral spatial characteristics of the spectral characteristics, and the inter-spectral relationship characteristics are fusion characteristics fusing inter-spectral local correlation characteristics and inter-spectral global continuity characteristics; and inputting the RGB image into the trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image. According to the method, the inter-spectrum local correlation features and the inter-spectrum global continuity features are efficiently extracted, and high-quality hyperspectral reconstruction is realized through block-level adaptive feature fusion.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a hyperspectral image reconstruction method and system. Background Art

[0002] Hyperspectral images (HSI) have extensive application value in many fields due to their rich spectral information, especially in the field of computer vision. Hyperspectral images show richer semantic expression capabilities than traditional RGB images by virtue of their detailed spectral information collected in multiple bands. Compared with RGB images, hyperspectral images can provide more spectral features, which make hyperspectral images have unique advantages and broad application prospects in many fields such as image classification and target detection. Especially in remote sensing, agricultural monitoring, environmental protection and medical imaging industries, hyperspectral images can provide researchers with more accurate information, thereby promoting scientific decision-making. However, due to the complexity of hyperspectral imaging equipment and the expensive acquisition cost, the popularization of hyperspectral images faces certain obstacles, which is not conducive to practical promotion and application. Therefore, how to obtain high-quality hyperspectral images at a lower cost has become the key to promoting the widespread application of hyperspectral technology. Considering that the acquisition cost of RGB images is relatively low, a hyperspectral image reconstruction task has been proposed. That is, by designing effective algorithms or specific mapping relationships or generation models, hyperspectral images are inferred or reconstructed from RGB images, so as to retain the spectral information of hyperspectral images as much as possible while keeping the cost low. This is of great significance to promoting the application and promotion of hyperspectral image processing technology.

[0003] Traditional hyperspectral reconstruction algorithms often rely on prior knowledge to establish the mapping relationship between RGB images and hyperspectral images, but the feature representation capabilities of these methods are often limited, and the quality of reconstructed hyperspectral images is not ideal. With the development of deep learning, the use of convolutional neural networks (CNNs) to achieve hyperspectral reconstruction tasks has gradually become mainstream. At present, a method of using channel attention to model the correlation between spectral segments has been proposed, which can effectively improve the reconstruction performance of the network. In addition, due to the superiority of Transformer in modeling long-range dependencies, a method of using Transformer to model the spectral segment correlation of spectral features has also been proposed. Compared with the algorithm that simply uses convolutional neural networks, this algorithm can mine more complex inter-spectral feature correlations, thereby greatly improving the reconstruction quality of hyperspectral images.

[0004] However, existing methods do not fully analyze the inter-spectral relationship characteristics of hyperspectral images, which leads to poor quality of the final reconstructed hyperspectral images:

[0005] (1) The inter-spectral relationship of hyperspectral images shows strong local correlation characteristics. The similarity of spectral images in adjacent ranges is high, while the larger the band interval, the lower the similarity of spectral images. The above traditional convolutional neural network-based method cannot efficiently construct the correlation of the spectral band. At the same time, the reconstruction algorithm based on Transformer cannot well emphasize the construction of spectral band correlation in the local range. There are also problems such as large amount of calculation and low modeling efficiency.

[0006] (2) From the global perspective of the spectrum, the spectral response value of the hyperspectral image at each spatial position changes gradually along the band dimension. This characteristic can be called the global continuity characteristic of the spectral image. The existing reconstruction method ignores the analysis and modeling of the global continuity characteristic between spectra, which hinders the further improvement of the reconstruction quality of hyperspectral images. Summary of the invention

[0007] To address the deficiencies of the above-mentioned prior art, the present invention provides a hyperspectral image reconstruction method and system. From the perspectives of modeling local and global characteristics between spectra, a hyperspectral image reconstruction network based on local correlation and global continuity modeling is designed, and a group spectral correlation modeling module is introduced into the network to efficiently model the inter-spectral correlation within a local range and extract the inter-spectral local correlation features; an adjacent spectral continuity modeling module is introduced, and a memory unit is used to model the gradual change characteristics at the global level, and the inter-spectral global continuity features are modeled; and the two features are effectively fused through block-level adaptive modeling, so as to simultaneously model from the local and global perspectives to obtain a more complete inter-spectral correlation, thereby effectively improving the reconstruction quality of the hyperspectral image.

[0008] In a first aspect, the present invention provides a hyperspectral image reconstruction method.

[0009] A hyperspectral image reconstruction method, comprising:

[0010] Get the RGB image taken by the RGB camera;

[0011] Input the RGB image into the trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein the training process of the hyperspectral image reconstruction network includes:

[0012] Obtain several image pairs consisting of RGB images and hyperspectral images to construct a data set;

[0013] A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed; the hyperspectral image reconstruction network includes a plurality of spectral reconstruction units, the spectral reconstruction unit includes a plurality of spectral relationship modeling modules, each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral space modeling module, which are respectively used to extract inter-spectral relationship features and intra-spectral space features of spectral features, wherein the inter-spectral relationship features are fused features of inter-spectral local correlation features and inter-spectral global continuity features;

[0014] The dataset is used for iterative training until the loss function is minimized to obtain a trained reconstructed network.

[0015] In a second aspect, the present invention provides a hyperspectral image reconstruction system.

[0016] A hyperspectral image reconstruction system, comprising:

[0017] RGB image acquisition module, used to acquire RGB images taken by an RGB camera;

[0018] The hyperspectral image reconstruction module is used to input the RGB image into the trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein the training process of the hyperspectral image reconstruction network includes:

[0019] Obtain several image pairs consisting of RGB images and hyperspectral images to construct a data set;

[0020] A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed; the hyperspectral image reconstruction network includes a plurality of spectral reconstruction units, the spectral reconstruction unit includes a plurality of spectral relationship modeling modules, each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral space modeling module, which are respectively used to extract inter-spectral relationship features and intra-spectral space features of spectral features, wherein the inter-spectral relationship features are fused features of inter-spectral local correlation features and inter-spectral global continuity features;

[0021] The dataset is used for iterative training until the loss function is minimized to obtain a trained reconstructed network.

[0022] In a third aspect, the present invention further provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the method described in the first aspect are completed.

[0023] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the method described in the first aspect.

[0024] One or more of the above technical solutions have the following beneficial effects:

[0025] 1. The present invention provides a hyperspectral image reconstruction method and system. From the two perspectives of modeling local and global characteristics between spectra, a hyperspectral image reconstruction network based on local correlation and global continuity modeling is designed. A group spectral correlation modeling module is introduced into the network. The group attention module is used to efficiently model the correlation between spectra in a local range, thereby solving the problems of inefficient modeling methods for correlation between local spectra in existing hyperspectral reconstruction algorithms, large amount of calculation, and difficulty in efficiently constructing correlation between spectra in a local range. An adjacent spectral continuity modeling module is introduced, and a memory unit is used to model the gradual change characteristics at the global level, and the global continuity characteristics between spectra are extracted, thereby solving the problem that the existing algorithms ignore the modeling representation of global continuity between spectra, and can effectively improve the quality of reconstructed hyperspectral images.

[0026] 2. In the present invention, a block-level adaptive fusion module is introduced for the local correlation features between spectra and the global continuity features between spectra, which effectively integrates the global continuity features into the spectral features in a block-level adaptive manner. The network can model a more complete inter-spectral relationship from both local and global perspectives, thereby further improving the reconstruction quality of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0028] Figure 1 A schematic diagram of the structure of a hyperspectral image reconstruction network based on local correlation and global continuity modeling in an embodiment of the present invention;

[0029] Figure 2 Schematic diagram of the structure of the adjacent spectrum continuity modeling module in an embodiment of the present invention;

[0030] Figure 3 Schematic diagram of the structure of the block-level adaptive fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] Aiming at the problems that the existing hyperspectral reconstruction algorithms are not efficient in modeling the inter-spectral local correlation, have a large amount of calculation, and are difficult to efficiently construct the inter-spectral correlation within a local range, and the existing algorithms ignore the modeling representation of the inter-spectral global continuity, which hinders the further improvement of the quality of reconstructed hyperspectral images, the present invention proposes a hyperspectral image reconstruction method. From the two perspectives of modeling the local characteristics and global characteristics of the inter-spectral, a hyperspectral image reconstruction network based on local correlation and global continuity modeling is designed. Specifically, for the local correlation between spectra, a group spectral correlation modeling module (Group-wise Spectral Correlation Modeling, GrSCM) is introduced. By grouping and dividing the spectral features along the spectral segment dimension, the group attention mechanism (Group-wise Attention) is used to efficiently model the inter-spectral correlation within the local range, which can effectively model the spectral characteristics at the local level; for the global continuity between spectra, a neighboring spectral continuity modeling module (Neighborhood-wise Spectral Continuity Modeling) is introduced. The spectral features along the spectral band dimension are regarded as time series, and the memory unit composed of 3D convolutional neural network is used to recursively model the progressive variation pattern of the spectrum at the global level, so as to extract the global continuity characteristics of the spectral features and effectively model the spectral features at the global level; finally, in order to explore the intrinsic complementarity of the two modules, a patch-wise adaptive fusion module (PAF) is proposed. By dividing the features generated by the two modules into patches along the spatial dimension, the global continuity features are effectively integrated into the spectral features in a patch-wise adaptive manner, the GrSCM module and the NeSCM module are effectively integrated, and the network is promoted to model more complete inter-spectral correlations from both local and global perspectives, thereby effectively improving the reconstruction quality of hyperspectral images.

[0033] Embodiment 1

[0034] First, define the hyperspectral image reconstruction problem. When an RGB camera captures an image, it is affected by the spectral response function (SRF). The captured image can be expressed as:

[0035]

[0036] Among them, λ is the wavelength band in the visible light range of 400nm-700nm, c∈{R,G,B} represents the three channels corresponding to the RGB image, represents the spectral response function (SRF) of camera channel c, I c (x,y) and H λ (x, y) represents the visible light image of channel c and the hyperspectral image of band λ respectively. In practice, it can be discretely represented as:

[0037]

[0038] Where B is the number of bands. The task of reconstructing RGB images into hyperspectral images can be regarded as the inverse process of formula (2), that is, given I c (x,y), solve the hyperspectral image H λ (x,y) process.

[0039] Considering that the imaging process of the hyperspectral image is to perform equally spaced sampling from a continuous spectral range, the collected spectral response data has two key characteristics: local correlation and global continuity. In order to fully analyze the above-mentioned inter-spectral relationship characteristics of the hyperspectral image and improve the quality of hyperspectral image reconstruction, this embodiment proposes a hyperspectral image reconstruction method, which specifically includes:

[0040] Get the RGB image taken by the RGB camera;

[0041] The RGB image is input into the trained hyperspectral image reconstruction network to obtain the reconstructed hyperspectral image.

[0042] Among them, the construction and training of the above-mentioned hyperspectral image reconstruction network is the key to ensure the quality of hyperspectral reconstruction. The core of the Local Correlation and Global Continuity Modeling Network (CCNet) for hyperspectral image reconstruction proposed in this embodiment is: composed of m spectral reconstruction units, for each spectral reconstruction unit, a customized spectral relationship modeling (CSRM) module is used to capture the intra-spectral spatial characteristics and inter-spectral relationship characteristics of spectral features. Among them, in the customized spectral relationship modeling module, the inter-spectral relationship is first modeled. On the one hand, a group spectral correlation modeling module is designed to model the inter-spectral correlation of the local range through a group-based attention strategy; on the other hand, an adjacent spectral continuity modeling module is constructed to model the global continuity between spectra, and 3D convolution is used to recursively model the gradual change law of the spectrum at the global level; finally, a block-level adaptive fusion module is proposed to fuse the features of the above two modules to jointly enhance the network's ability to model local correlation and global continuity. In addition, after modeling the inter-spectral relationship, the feed forward neural network (FFN) in the Transformer architecture is used to model the intra-spectral spatial features. The final customized spectral relationship modeling can completely model the inter-spectral relationship features and the intra-spectral spatial features, thereby improving the quality of the final hyperspectral reconstruction.

[0043] The following content provides a more detailed introduction to the construction and training of the above-mentioned hyperspectral image reconstruction network.

[0044] Step S1: Acquire several image pairs consisting of RGB images and hyperspectral images to construct a data set.

[0045] Step S2: building a hyperspectral image reconstruction network based on local correlation and global continuity modeling.

[0046] like Figure 1As shown, the RGB image input to the reconstruction network is subjected to 2D convolution to extract spectral features, and the spectral features are then extracted through multiple spectral reconstruction units for inter-spectral feature extraction. The extracted inter-spectral features are reconstructed by 2D convolution to obtain a hyperspectral image. Among them, the spectral features are input to the spectral reconstruction unit, encoded through continuous multi-layer spectral relationship modeling modules and downsampling layers (two layers in this embodiment), and multi-scale deep spectral features are extracted; the deep spectral features extracted by the last layer of spectral relationship modeling module are decoded through continuous multi-layer upsampling layers and spectral relationship modeling modules (two layers in this embodiment), and the features decoded in the decoding process are merged with the features of the same scale obtained by encoding; the features generated by the top-level spectral relationship modeling module are output as reconstructed hyperspectral features.

[0047] Among them, the hyperspectral image reconstruction network includes multiple spectral reconstruction units (3 are provided in this embodiment), the spectral reconstruction unit includes multiple spectral relationship modeling modules, each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral space modeling module, which are respectively used to extract the inter-spectral relationship features and the intra-spectral space features of the spectral features, and the inter-spectral relationship features are fused features of the local correlation features between spectra and the global continuity features between spectra.

[0048] The above-mentioned spectrum relationship modeling module includes:

[0049] The group spectral correlation modeling module is used to model the local inter-spectral correlation using a group-based attention strategy and extract the inter-spectral local correlation features;

[0050] The adjacent spectrum continuity modeling module is used to recursively model the inter-spectral continuity at the global level using 3D convolution and extract the inter-spectral global continuity features;

[0051] The block-level adaptive fusion module is used to adaptively fuse the local correlation features between spectra and the global continuity features between spectra to generate the inter-spectral relationship features;

[0052] Among them, the group spectrum correlation modeling module and the adjacent spectrum continuity modeling module are two parallel modules, and each module is introduced in detail below.

[0053] (1) Group-wise Spectral Correlation Modeling (GrSCM)

[0054] The multi-head attention mechanism in Transformer shows excellent performance in the hyperspectral reconstruction task due to its advantage of long-distance modeling. By building a similarity association between each spectral band and all other spectral bands, the high-order correlation between spectra can be modeled to a certain extent. However, hyperspectral images have the characteristics of strong local correlation along the spectral band dimension. The correlation between adjacent bands is strong, while the correlation between bands is low when the spectral bands are far apart. This type of method does not take this characteristic into consideration, and will introduce unnecessary calculations when modeling global spectral band associations, which will also affect the reconstruction effect. In order to solve this problem, a group spectral correlation modeling module is introduced in this embodiment. The above-mentioned extracted spectral features are input into the group spectral correlation modeling module. The module divides the original spectral features into multiple sub-regions along the channel dimension, and uses the group attention mechanism in these sub-regions to model the information association between local spectral bands, so as to extract the local correlation features between spectra. In this way, the correlation between local features between spectra is enhanced, while reducing unnecessary computational costs and optimization burdens.

[0055] Furthermore, assuming that the spectral feature F in The size is R H×W×C First, the input spectral features are grouped along the channel dimension to obtain the grouped features Where i∈{1,2,…,k} represents the group index, and k is the number of groups. Then, the spectral features of each group (i.e., each group feature) are linearly mapped within the group, and the group features are mapped to the latent feature space, which can be expressed as:

[0056]

[0057] in, Represents the linear mapping parameters.

[0058] Then, for each feature after feature mapping, the spectral features are divided into the head dimension along the channel dimension to obtain the attention feature vector Among them C in It represents the number of feature channels of the spectral reconstruction unit input, n represents the number of heads, and the head dimension can be interpreted as the feature dimension of the spectral band dimension.

[0059] Afterwards, based on the above attention feature vector, the group similarity between the band features is calculated to generate the attention weight. Considering that the similarity between the band dimensions can be reflected to a certain extent by the relative position relationship of the corresponding spectra, the closer the two bands are, the higher the similarity should be. Therefore, in order to better model the intrinsic relationship between the similarity of band features and their relative spectrum relationship, a learnable relative spectrum encoding (Relative Spectrum Encoding, RSE) is introduced as a prompt for the relative spectrum position when generating the attention weight. That is, the group similarity and relative spectrum encoding are combined to generate the attention weight. Then, the softmax function is used to obtain the normalized attention weight, and the weighted summation method is used to obtain the final attention feature. The specific calculation process can be expressed as:

[0060]

[0061] in, is the relative spectrum code of the i-th group.

[0062] Finally, the attention features of each group Merge along the head dimension and group dimension respectively to obtain the constructed local correlation spectral feature F att .

[0063] (2) Neighborhood-wise Spectral Continuity Modeling (NeSCM)

[0064] In addition to local correlation, hyperspectral image processing also has global continuity characteristics. From the first band to the last band, the response value of the hyperspectral image changes gradually along the band dimension, not all at once. However, existing methods do not fully utilize and model this spectral property. In this regard, this embodiment constructs an adjacent spectral continuity modeling module, and the extracted spectral features are input into the adjacent spectral continuity modeling module. The module is mainly composed of two branches: forward modeling and backward modeling. It can effectively model the global continuity of spectral features in a recursive manner, thereby extracting the global continuity features between spectra.

[0065] like Figure 2 As shown, first, according to the method of dividing the head dimension in the above group spectral correlation modeling module, the input spectral features are divided along the channel dimension, and a dimension is specified to represent the band dimension of the spectrum to obtain the feature After that, a window of size s is selected, and for each divided spectral feature, a sliding window operation is performed along the band dimension to extract the continuously distributed spectral features. where xi ∈R n ×H×W×s ; Then, these features are input into the forward and backward branches respectively to extract the global continuity features in two directions.

[0066] Among them, both the forward and backward branches include multiple continuous memory units. For the forward branch, a memory feature m0 is first initialized, and then each spectral band feature and the memory feature are sequentially input into the continuous memory unit. This process is recursively executed to update the memory feature and generate the output feature of the current stage. Similarly, for the backward branch, the calculation process is similar to the forward branch, except that the input order / sequence of the spectral band features is opposite to that of the forward branch.

[0067] For each continuous memory unit, the input data of the unit is the segment memory feature m of the previous stage t-1 and the spectral characteristics x of the current stage t , both have the same dimension R n×H×W×s In order to effectively model the inter-spectral continuity at the global level, two gating strategies are constructed for each continuous memory unit: forget gate and output gate. Among them, the forget gate combines the memory feature m of the previous stage t-1 and the spectrum characteristics x of the current stage t Generate forget weights to discard some historical information in the memory features. Specifically, link m along the feature dimension t-1 and x t , and use 3D convolution with a 1×1×1 kernel to generate the forget weight of the current unit. The specific formula can be expressed as:

[0068] s t =Sigmoid(Conv3D(Concat(m t-1 ,x t )))(5)

[0069] Among them, s t The forget weight of the current unit.

[0070] Afterwards, the forgetting weight is used to update the memory feature sequence to obtain the memory feature m of the current stage t ,for:

[0071] m t =s t ×m t-1 +(1-s t )×x t (6)

[0072] For the output gate, the memory feature m of the current stage is fused tand the spectral feature x t , and then input it into the 3D convolution with a 3×3×s convolution kernel to extract the continuity characteristics of the spectrum, which is:

[0073] y t =Conv3D(Tanh(m t )+x t )(7)

[0074] Among them, y t Represents the output of the tth continuous memory unit, with dimension R n×H×W×1 .

[0075] Finally, all the continuous features generated by the forward branch and the backward branch are merged and fused to generate a global progressive feature F that can characterize the global continuous change. p , that is, the global continuity feature between spectra, which is used as the output feature of this module. In this process, the adjacent spectrum continuity modeling module uses 3D convolution to simultaneously select multiple adjacent spectrum segments for iterative calculations, while considering the global time series characteristics, and enhances the ability of continuity modeling between vector spectrum segments, further enhancing the algorithm's ability to model global spectral characteristics between spectra.

[0076] (3) Patch-wise Adaptive Fusion (PAF)

[0077] The spectral feature F obtained by the group spectral correlation modeling module att The spectral relationship between the characteristics of each band in the local area is effectively constructed, and the global progressive characteristics F output by the adjacent spectral continuity modeling module p The global continuity characteristics between spectra in the original spectral features are captured. The two sets of features emphasize different aspects and complement each other. Modeling the inherent complementarity between the two sets of features is the key to obtaining high-quality spectral features. To this end, this embodiment designs a block-level adaptive fusion module, which uses a block-level fusion strategy to combine F p Adaptively merge into F att In the process, the inter-spectral relationship features are extracted to achieve adaptive fusion of local spatial regions, thereby capturing the inter-spectral relationship of spectra more comprehensively.

[0078] like Figure 3 As shown, first, combined with F att Based on the group modeling strategy, a spectral information aggregation module (Spectral Information Aggregation, SIA) is designed. The global continuity features between spectra are input into the spectral information aggregation module, and the input spectral features are aggregated using a group-based method. The grouping operation follows the same rules as the group spectral correlation modeling module. The process is expressed as in, It is represented as the progressive feature of the i-th group. Then, in each group, the progressive features of the group are average pooled and max pooled along the channel dimension, and then the two pooled features are fused using a 2D convolution operation to generate the compressed progressive feature of the i-th group. The whole process can be expressed as:

[0079]

[0080] Among them, C p represents the number of channels of the grouped progressive features of the i-th group, and merges all along the channel dimension Get the compressed progressive feature F cp , the characteristic size is R H×W×k .

[0081] Secondly, F att and F cp Perform a 1×1 2D convolution operation to perform feature mapping, and divide the mapping features into different patches along the spatial dimension to obtain and Where n represents the sequence number of the block. Assuming the size of the block is r×r, each channel feature will get block-level features. Further, the cosine similarity between the block-level features is calculated, and then the attention weight for fusion can be obtained, which can be expressed as:

[0082]

[0083] w n (i,*)=softmax(sim n (i,*))(12)

[0084] Among them, sim n (i,j) and w n (i,j) represent the attention features of the i-th channel respectively and the compressed progressive characteristics of the jth channel The feature similarity and attention weight between them.

[0085] Finally, the calculated attention weights are used to fuse F att With F cp , generate the fused inter-spectral relationship features, the specific process can be expressed as:

[0086]

[0087] Among them, F out represents the output characteristics of the module, represents the sequential splicing of all block-level features along the channel dimension i and the spatial dimension n. By dividing the spectral features of each band into blocks along the spatial dimension and implementing block-level feature fusion, this embodiment can implement adaptive feature fusion with spatial regional differences. The fusion process explores F cp and F att The intrinsic correlation between them enables the network to simultaneously model the local correlation and global continuity of the spectrum, which is conducive to generating more comprehensive inter-spectral relationship features.

[0088] Step S3, using the data set to perform iterative training until the loss function is minimized to obtain a trained reconstruction network; wherein the constructed loss function includes average relative error loss and spectral difference loss.

[0089] Specifically, suppose Y and Represent the real hyperspectral image and the reconstructed hyperspectral image respectively, and two losses are used to optimize the parameters of the entire network, which are:

[0090] (1) Mean Relative Absolute Error (MRAE): Calculate the relative error between the real hyperspectral image and the reconstructed hyperspectral image at a pixel-by-pixel granularity to optimize the network parameters. The loss formula can be expressed as:

[0091]

[0092] Where N represents the number of pixel values ​​in the hyperspectral image.

[0093] (2) Spectral Difference Loss: The L1 distance between any band and all other bands in the real hyperspectral image and the reconstructed hyperspectral image. Specifically, in order to further model the spectral relationship between hyperspectral images, a spectral difference loss is proposed. The L1 distance is used to calculate the difference between a band and all other bands in the reconstructed hyperspectral image, thereby obtaining a size of Similarly, the same rules are used to calculate the spectrum difference of the real hyperspectral image. The two sets of difference data are then calculated by the L1 distance calculation formula to obtain the final spectrum difference loss. The specific loss calculation formula can be expressed as:

[0094]

[0095] Finally, the total loss of the model can be expressed as:

[0096] Loss = L MRAE +γL Dif (16)

[0097] Among them, γ represents the weight of the spectral difference loss.

[0098] Embodiment 2

[0099] This embodiment provides a hyperspectral image reconstruction system, including:

[0100] RGB image acquisition module, used to acquire RGB images taken by an RGB camera;

[0101] The hyperspectral image reconstruction module is used to input the RGB image into the trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein the training process of the hyperspectral image reconstruction network includes:

[0102] Obtain several image pairs consisting of RGB images and hyperspectral images to construct a data set;

[0103] A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed; the hyperspectral image reconstruction network includes a plurality of spectral reconstruction units, the spectral reconstruction unit includes a plurality of spectral relationship modeling modules, each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral space modeling module, which are respectively used to extract inter-spectral relationship features and intra-spectral space features of spectral features, wherein the inter-spectral relationship features are fused features of inter-spectral local correlation features and inter-spectral global continuity features;

[0104] The data set is used for iterative training until the loss function is minimized, and a trained reconstruction network is obtained.

[0105] Embodiment 3

[0106] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the hyperspectral image reconstruction method described above are completed.

[0107] Embodiment 4

[0108] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the hyperspectral image reconstruction method described above are completed.

[0109] The steps involved in the above embodiments 2 to 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0110] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0111] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A hyperspectral image reconstruction method, characterized in that: include: Get the RGB image taken by the RGB camera; Input the RGB image into the trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein the training process of the hyperspectral image reconstruction network includes: Obtain several image pairs consisting of RGB images and hyperspectral images to construct a data set; A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed; the hyperspectral image reconstruction network includes a plurality of spectral reconstruction units, the spectral reconstruction unit includes a plurality of spectral relationship modeling modules, each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral space modeling module, which are respectively used to extract inter-spectral relationship features and intra-spectral space features of spectral features, wherein the inter-spectral relationship features are fused features of inter-spectral local correlation features and inter-spectral global continuity features; The dataset is used for iterative training until the loss function is minimized to obtain a trained reconstructed network.

2. A hyperspectral image reconstruction method as claimed in claim 1, characterized in that: The inter-spectrum relationship modeling module includes: The group spectral correlation modeling module is used to model the inter-spectral correlation in the local range using the group-based attention strategy and extract the inter-spectral local correlation features; The adjacent spectrum continuity modeling module is used to recursively model the inter-spectral continuity at the global level using 3D convolution and extract the inter-spectral global continuity features; The block-level adaptive fusion module is used to adaptively fuse the local correlation features between spectra and the global continuity features between spectra to generate the inter-spectral relationship features; Among them, the group spectrum correlation modeling module and the adjacent spectrum continuity modeling module are two parallel modules.

3. A hyperspectral image reconstruction method as claimed in claim 2, characterized in that: The spectral features are extracted and input into the group spectral correlation modeling module. The group attention mechanism is used to model the information association between local spectral segments and extract the local correlation features between spectra, including: Group the input spectral features along the channel dimension to obtain grouped features; Perform linear mapping within each group feature in turn, and map the group features to the potential feature space; For each feature after feature mapping of each group, a head dimension is divided along the channel dimension to obtain an attention feature vector; the head dimension is a feature dimension of the spectral band dimension; Based on the attention feature vector, the group similarity between the spectral band features is calculated, and the learnable relative spectrum encoding is introduced as a prompt for the relative spectrum position; the attention weight is generated by combining the group similarity and the relative spectrum encoding; the attention weight is normalized and the attention feature is calculated; Each group of attention features is merged along the head dimension and the group dimension to obtain the inter-spectral local correlation features.

4. A hyperspectral image reconstruction method as claimed in claim 2, characterized in that: The adjacent spectrum continuity modeling module consists of two branches: forward modeling and backward modeling. The extracted spectral features are input into the adjacent spectrum continuity modeling module to extract the global continuity features between spectra, including: Divide the input spectral features along the channel dimension and specify a dimension to represent the band dimension of the spectrum; For each divided spectral feature, a sliding window operation is performed along the band dimension to extract the continuously distributed spectral features; The continuously distributed spectral features are input into the forward and backward branches to extract the continuity features in two directions respectively; All the continuity features generated by the forward branch and the backward branch are merged respectively and then fused to generate the inter-spectral global continuity features used to characterize the global continuity changes; The forward and backward branches each include a plurality of continuous memory units, the memory feature is initialized, each spectral feature and the memory feature are sequentially input into the continuous memory unit, the memory feature is updated and the output feature of the current stage is generated; the order of the forward branch and the backward branch is opposite; a forget gate and an output gate are constructed in the continuous memory unit, and the input data of the continuous memory unit is the memory feature m of the previous stage. t-1 and the spectral characteristics x of the current stage t , the forget gate combines the memory feature m of the previous stage t-1 and the spectral characteristics x of the current stage t Generate forgetting weights and use the forgetting weights to update the memory features to obtain the memory features m of the current stage t ; The output gate fuses the memory feature m of the current stage t and the spectral feature x t , extracts and outputs the continuity feature y of the spectrum through 3D convolution t .

5. A hyperspectral image reconstruction method as claimed in claim 2, characterized in that: The block-level adaptive fusion module adopts a block-level fusion strategy to adaptively fuse the inter-spectral global continuity features into the inter-spectral local correlation features to extract the inter-spectral relationship features, including: The inter-spectral global continuity features are input into the spectral information aggregation module, and the input features are aggregated using a grouping-based method. In each group, the grouped features are mean pooled and max pooled along the channel dimension, and then the pooled features are fused to generate compressed progressive features. The compressed progressive features and the inter-spectral local correlation features are feature mapped, and the mapped features are divided into different blocks along the spatial dimension to calculate the block-level feature cosine similarity of the two features, and then the attention weights for fusion are obtained; The calculated attention weights are used to fuse the compressed progressive features and the inter-spectral local correlation features to generate the fused inter-spectral relationship features.

6. A hyperspectral image reconstruction method according to claim 1, characterized in that: The loss function includes average relative error loss and spectral difference loss; The average relative error loss is the relative error between the real hyperspectral image and the reconstructed hyperspectral image at a pixel-by-pixel granularity; The spectral difference loss is the L1 distance between any band and all other bands in the real hyperspectral image and the reconstructed hyperspectral image.

7. A hyperspectral image reconstruction method according to claim 1, characterized in that: The spectral features of the RGB image are extracted by 2D convolution, and the spectral features are then extracted through multiple spectral reconstruction units. The extracted spectral features are reconstructed by 2D convolution to obtain a hyperspectral image. Among them, the spectral features are input into the spectral reconstruction unit, encoded through multiple consecutive spectral relationship modeling modules and downsampling layers, and multi-scale deep spectral features are extracted; the deep spectral features extracted by the last layer of spectral relationship modeling module are decoded through multiple consecutive upsampling layers and spectral relationship modeling modules, and the decoded features are merged with the features of the same scale obtained by encoding during the decoding process; the features generated by the top-level spectral relationship modeling module are output as the reconstructed hyperspectral features.

8. A hyperspectral image reconstruction system, characterized in that: include: RGB image acquisition module, used to acquire RGB images taken by an RGB camera; The hyperspectral image reconstruction module is used to input the RGB image into the trained hyperspectral image reconstruction network to obtain a reconstructed hyperspectral image; wherein the training process of the hyperspectral image reconstruction network includes: Obtain several image pairs consisting of RGB images and hyperspectral images to construct a data set; A hyperspectral image reconstruction network based on local correlation and global continuity modeling is constructed; the hyperspectral image reconstruction network includes a plurality of spectral reconstruction units, the spectral reconstruction unit includes a plurality of spectral relationship modeling modules, each spectral relationship modeling module includes an inter-spectral relationship modeling module and an intra-spectral space modeling module, which are respectively used to extract inter-spectral relationship features and intra-spectral space features of spectral features, wherein the inter-spectral relationship features are fused features of inter-spectral local correlation features and inter-spectral global continuity features; The dataset is used for iterative training until the loss function is minimized to obtain a trained reconstructed network.

9. An electronic device, characterized in that: The method comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of a hyperspectral image reconstruction method according to any one of claims 1 to 7 are completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of a hyperspectral image reconstruction method as described in any one of claims 1 to 7.

Images