Hyperspectral image reconstruction method and device, equipment and storage medium

Abstract

The invention provides a hyperspectral image reconstruction method, device and equipment and a storage medium, and relates to the technical field of image reconstruction, and the method comprises the steps: obtaining target two-dimensional compression measurement data of a CASSI system and corresponding target RGB image data; the target hyperspectral image is input into a hyperspectral image reconstruction network, a reconstructed target hyperspectral image is obtained, the reconstruction network comprises a target number of image reconstruction modules, and each image reconstruction module comprises an interpretable fusion module and a fusion guide reconstruction module; the interpretable fusion module learns to obtain a corresponding target spectral response matrix by solving a multi-target optimization problem, and the optimization problem is determined based on the fidelity of target two-dimensional compression measurement data, the constraint condition of target RGB image data and a regularization item; and the fusion guiding reconstruction module is used for guiding reconstruction of the target hyperspectral image through the target spectral response matrix. According to the invention, high-quality spectral image reconstruction is realized.

Description

Technical Field

[0001] This invention relates to the field of image reconstruction technology, and in particular to a hyperspectral image reconstruction method, apparatus, device, and storage medium. Background Technology

[0002] The Single-Dispersion Coded Aperture Snapshot Spectral Imaging (SD-CASSI) system, based on compressed sensing theory, achieves two-dimensional compressed measurement of three-dimensional hyperspectral data through the synergistic effect of the coded aperture and dispersive prism. The system works by first spatially modulating the incident light through the coded aperture, then dispersing it along the spectral dimension through the dispersive prism, ultimately forming a two-dimensional aliasing measurement containing both spatial and spectral information on a monochromatic detector. This innovative single-exposure imaging mechanism improves data acquisition efficiency by 1-2 orders of magnitude compared to traditional scanning spectral imaging techniques. However, the SD-CASSI system faces a severe ill-conditioned inverse problem during reconstruction, resulting in a fundamental information loss when recovering a three-dimensional hyperspectral cube from two-dimensional measurement data. Existing reconstruction algorithms, such as the total variational regularization method, while improving reconstruction quality to some extent, still suffer from high computational complexity and sensitivity to noise. Especially under low signal-to-noise ratio conditions, the reconstruction results are prone to severe spectral distortion and spatial blurring, which greatly limits the system's practical application in complex environments.

[0003] To overcome the limitations of the SD-CASSI system, the Dual-Camera Coded Aperture Snapshot Spectral Imaging (DC-CASSI) technology was developed. This DC-CASSI system innovatively introduces red, green, and blue (RGB) cameras as auxiliary imaging units. A beam splitter divides the incident light into two paths: one path enters the traditional Coded Aperture Snapshot Spectral Imaging (CASSI) imaging channel to acquire compressed spectral information, while the other path is captured by the RGB camera to obtain a high spatial resolution color image. This dual-channel design fully utilizes the rich spatial detail information of the RGB images. Through advanced information fusion algorithms, it not only improves the spatial resolution by more than 30% but also significantly enhances the system's robustness to noise. The core advantage of the DC-CASSI system lies in its multimodal data fusion capability. The spatial information provided by the RGB images effectively compensates for the information loss in the CASSI channel during reconstruction, while the CASSI channel ensures the system's high spectral resolution characteristics.

[0004] Despite the significant advantages exhibited by DC-CASSI technology, its implementation still faces several key technical challenges. The most prominent issue is the system's high dependence on the Spectral Response Matrix (SRM) calibration. Existing methods typically treat the SRM as a fixed parameter, but in reality, the SRM of different RGB camera models varies significantly and drifts over time. In other words, existing systems lack the ability to adapt to SRM parameters, and system performance deteriorates noticeably when ambient lighting conditions or camera characteristics change. These problems are particularly pronounced in long-term monitoring tasks, leading not only to a gradual degradation in reconstruction quality but also potentially causing misjudgments of key spectral attention output features, severely hindering the widespread adoption of this technology in high-precision applications such as environmental monitoring and precision agriculture.

[0005] How to effectively reconstruct high-quality spectral images has become a pressing technical problem that needs to be solved in the industry. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a hyperspectral image reconstruction method, apparatus, device, and storage medium, which effectively achieves high-quality spectral image reconstruction.

[0007] In a first aspect, the present invention provides a hyperspectral image reconstruction method, the method comprising the following steps: Acquire the target two-dimensional compressed measurement data of the coded aperture snapshot spectral imaging CASSI system and the target RGB image data corresponding to the target two-dimensional compressed measurement data; The target two-dimensional compressed measurement data and the target RGB image data are input into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0008] According to a hyperspectral image reconstruction method provided by the present invention, the step of inputting the target two-dimensional compressed measurement data and the target RGB image data into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image includes: The target RGB image data and the original hyperspectral image corresponding to the target RGB image data are input into the interpretable fusion module in the first image reconstruction module among the target number of image reconstruction modules to obtain the first spectral response matrix corresponding to the first image reconstruction module; The first spectral response matrix corresponding to the first image reconstruction module and the target two-dimensional compressed measurement data are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the first hyperspectral image corresponding to the first image reconstruction module. Step 1: Input the target RGB image data and the first hyperspectral image into the interpretable fusion module in the next image reconstruction module to obtain the second spectral response matrix corresponding to the next image reconstruction module; input the second spectral response matrix and the target two-dimensional compressed measurement data into the fusion-guided reconstruction module in the next image reconstruction module, and guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the second hyperspectral image corresponding to the next image reconstruction module; determine the second hyperspectral image as the first hyperspectral image; Repeat step one until the second hyperspectral image corresponding to the last image reconstruction module in the target number of image reconstruction modules is obtained, and determine the second hyperspectral image corresponding to the last image reconstruction module as the target hyperspectral image.

[0009] According to a hyperspectral image reconstruction method provided by the present invention, the step of inputting the target RGB image data and the first hyperspectral image into the interpretable fusion module in the next image reconstruction module to obtain the second spectral response matrix corresponding to the next image reconstruction module includes: Based on the first hyperspectral image, the target RGB image data, the first spectral response matrix, and the gradient descent step size, calculate the intermediate variable of the spectral response matrix corresponding to the next image reconstruction module; The second spectral response matrix is ​​determined based on the intermediate variables of the spectral response matrix and the target RGB image data using an interpretable self-attention mechanism module.

[0010] According to a hyperspectral image reconstruction method provided by the present invention, the multi-view visual transformer includes a pixel-level spectral attention branch and a channel-level spatial convolution branch; The step of inputting the second spectral response matrix and the target two-dimensional compressed measurement data into the fusion-guided reconstruction module in the next image reconstruction module, and guiding image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module of the next image reconstruction module to obtain the second hyperspectral image corresponding to the next image reconstruction module includes: Using the pixel-level spectral attention branch, the local spectral correlation of the first hyperspectral image is processed by a self-attention mechanism based on the second spectral response matrix and the target two-dimensional compressed measurement data to obtain the spectral attention output features of the first hyperspectral image; Using the channel-level spatial convolution branch, the spatial features of the first hyperspectral image are extracted by the convolution module based on the second spectral response matrix and the target two-dimensional compressed measurement data. The spectral attention output features and the spatial features are integrated by a cross-fusion module to output a merged feature; Based on the merged features, the second hyperspectral image is generated.

[0011] According to a hyperspectral image reconstruction method provided by the present invention, the training process of the hyperspectral image reconstruction network includes the following steps: Initialize network parameters; the initial network parameters include the weight information of the interpretable fusion module and the fusion-guided reconstruction module in the image reconstruction module; Multiple training samples are acquired; the training samples include sample two-dimensional compressed measurement data and sample RGB image data corresponding to the sample two-dimensional compressed measurement data. The sample RGB image data and the original hyperspectral image corresponding to each sample RGB image data are input into the interpretable fusion module in the first image reconstruction module to obtain the third spectral response matrix corresponding to the first image reconstruction module; The third spectral response matrices corresponding to the first image reconstruction module and the two-dimensional compressed measurement data of each sample are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the third hyperspectral images corresponding to the first image reconstruction module. Step 2: Input the RGB image data of each sample and the third hyperspectral image of each sample into the interpretable fusion module of the next image reconstruction module to obtain the fourth spectral response matrix corresponding to the next image reconstruction module; input the fourth spectral response matrix and the two-dimensional compressed measurement data of each sample into the fusion-guided reconstruction module of the next image reconstruction module, and guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the fourth hyperspectral image corresponding to the next image reconstruction module; determine each fourth hyperspectral image as the third hyperspectral image. Repeat step two until a preset convergence condition is met; the preset convergence condition is that the value of the multi-objective loss function is minimized. The model parameters that satisfy the preset convergence condition are determined as the optimal network parameters, and the hyperspectral image reconstruction network is determined based on the optimal network parameters.

[0012] According to a hyperspectral image reconstruction method provided by the present invention, before determining the model parameters that satisfy the preset convergence condition as the optimal network parameters, and determining the hyperspectral image reconstruction network based on the optimal network parameters, the method further includes: Based on each of the fourth hyperspectral images, determine the data fidelity loss, spectral consistency loss, and spatial detail loss; The value of the multi-objective loss function is calculated based on the data fidelity loss, the spectral consistency loss, and the spatial detail loss.

[0013] In a second aspect, the present invention also provides a hyperspectral image reconstruction apparatus, which includes the following modules: The acquisition module is used to acquire the target two-dimensional compressed measurement data of the coded aperture snapshot spectral imaging CASSI system and the target RGB image data corresponding to the target two-dimensional compressed measurement data; A reconstruction module is used to input the target two-dimensional compressed measurement data and the target RGB image data into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0014] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the hyperspectral image reconstruction method as described above.

[0015] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image reconstruction method as described above.

[0016] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the hyperspectral image reconstruction method as described above.

[0017] The hyperspectral image reconstruction method, apparatus, device, and storage medium provided by this invention first acquire the target two-dimensional compressed measurement data and the corresponding target RGB image data from the coded aperture snapshot spectral imaging (CASSI) system. Then, the target two-dimensional compressed measurement data and the target RGB image data are input into a hyperspectral image reconstruction network to obtain the reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0018] This invention addresses the difficulty of spectral response matrix (SRM) calibration in DC-CASSI systems by proposing a learnable SRM estimation strategy based on an attention mechanism. Specifically, the corresponding target spectral response matrix is ​​learned by solving a multi-objective optimization problem through an interpretable fusion module in the image reconstruction module, effectively achieving high-quality spectral image reconstruction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts of the hyperspectral image reconstruction method provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the framework of the interpretable fusion module provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the structure of the multi-view visual transformer provided by the present invention.

[0023] Figure 4 This is the second flowchart of the hyperspectral image reconstruction method provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the hyperspectral image reconstruction device provided by the present invention.

[0025] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The following is combined Figures 1-6 The present invention describes a hyperspectral image reconstruction method, apparatus, device, and storage medium.

[0028] Figure 1 This is one of the flowcharts illustrating the hyperspectral image reconstruction method provided by the present invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain the target two-dimensional compressed measurement data and the target RGB image data corresponding to the target two-dimensional compressed measurement data from the coded aperture snapshot spectral imaging CASSI system.

[0029] This invention designs a hybrid reconstruction architecture based on dual-camera coded aperture snapshot spectral imaging (DC-CASSI) technology, innovatively combining the physical interpretability of traditional optimization methods with the representational advantages of deep learning methods, achieving high-precision, adaptive reconstruction of hyperspectral images. Addressing the difficulty of spectral response matrix (SRM) calibration in DC-CASSI systems, this invention proposes a learnable SRM estimation strategy based on an attention mechanism and designs corresponding spectral consistency constraints. Furthermore, this invention constructs a dual-branch processing structure integrating a multi-view vision transformer (MVT) and convolution operations, capable of simultaneously capturing spatial details and spectral attention output features of hyperspectral images, maintaining excellent spectral fidelity while improving reconstruction quality.

[0030] The image reconstruction process includes the following steps: First, acquire the target two-dimensional compressed measurement data and the target RGB image data corresponding to the target two-dimensional compressed measurement data from the coded aperture snapshot spectral imaging CASSI system.

[0031] Specifically, it can be understood that traditional CASSI uses only one detector, receiving coded and dispersed two-dimensional compressed measurements. This value itself is highly aliased, and the reconstruction process is akin to solving a very difficult inverse problem, heavily reliant on prior knowledge such as the sparsity assumption of the image. If the scene does not conform to these assumptions, the reconstruction effect will be significantly reduced. This invention employs a dual-camera coded aperture snapshot spectral imaging CASSI system, DC-CASSI.

[0032] The innovation of DC-CASSI lies in the addition of a separate camera for capturing ordinary RGB (or panchromatic) images. This camera works in sync with the CASSI's optical path, capturing two images simultaneously in a single snapshot: one is a conventional, aliased CASSI compressed measurement (captured by the CASSI camera), and the other is a clear, unmodulated and undispersed RGB image (captured by the RGB camera).

[0033] The workflow for DC-CASSI to acquire target 2D compressed measurement data and corresponding target RGB image data is as follows: 1. Beam splitting: The incident light is split into two paths by a beam splitter.

[0034] 2. CASSI path: One beam of light enters the traditional CASSI optical path, and after modulation by the coded aperture plate and dispersion by the prism, the target two-dimensional compressed measurement data is finally formed on the CASSI camera.

[0035] 3. RGB path: Another path of light directly enters the RGB camera, forming a clear, high spatial resolution RGB image, which is the target RGB image data.

[0036] 4. Synchronous acquisition: The two cameras are triggered synchronously in hardware to ensure that the scene is captured at the same instant.

[0037] Step 102: Input the target two-dimensional compressed measurement data and the target RGB image data into the hyperspectral image reconstruction network to obtain the reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and the regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0038] Specifically, after obtaining the target two-dimensional compressed measurement data and the corresponding target RGB image data, the present invention performs image reconstruction through a pre-trained hyperspectral image reconstruction network.

[0039] The hyperspectral image reconstruction network includes a target number of image reconstruction modules, which include an Interpretable Fusion Module (IFM) and a Fusion-Guided Reconstruction Module (FRM).

[0040] The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and the regularization term.

[0041] The multi-objective optimization problem is expressed by the following formula (1): (1) in, This represents the estimation of the reconstructed hyperspectral image. For the estimated spectral response matrix SRM, For the target two-dimensional compressed measurement data of the CASSI system, The original hyperspectral image, The sensor matrix of the CASSI system represents the hardware coding process. For target RGB image data, It is a learnable spectral response matrix (SRM). For regularization terms (such as sparsity), scalar functions, For SRM regularization terms, , , It is a hyperparameter that balances the weights of various components.

[0042] The design of this objective function takes into account three key factors: the fidelity of the CASSI measurement data, the constraints of the RGB image, and the regularization term, to ensure that the reconstruction process conforms to the physical imaging model and makes full use of multi-source information.

[0043] The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix. Specifically, the hyperspectral image is reconstructed using the learned target spectral response matrix (SRM) and the target two-dimensional compressed measurement data as inputs, through the parallel dual-branch structure of the multi-view visual transformer (MVT) of the fusion-guided reconstruction module.

[0044] The method provided in this embodiment first acquires the target two-dimensional compressed measurement data and the corresponding target RGB image data from the coded aperture snapshot spectral imaging (CASSI) system. Then, the target two-dimensional compressed measurement data and the target RGB image data are input into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0045] This invention addresses the difficulty of spectral response matrix (SRM) calibration in DC-CASSI systems by proposing a learnable SRM estimation strategy based on an attention mechanism. Specifically, the corresponding target spectral response matrix is ​​learned by solving a multi-objective optimization problem through an interpretable fusion module in the image reconstruction module, effectively achieving high-quality spectral image reconstruction.

[0046] According to a hyperspectral image reconstruction method provided by the present invention, target two-dimensional compressed measurement data and target RGB image data are input into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image, including: Input the target RGB image data and the original hyperspectral image corresponding to the target RGB image data into the interpretable fusion module of the first image reconstruction module in the target number of image reconstruction modules to obtain the first spectral response matrix corresponding to the first image reconstruction module; The first spectral response matrix and target two-dimensional compressed measurement data corresponding to the first image reconstruction module are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the first hyperspectral image corresponding to the first image reconstruction module. Step 1: Input the target RGB image data and the first hyperspectral image into the interpretable fusion module of the next image reconstruction module to obtain the second spectral response matrix corresponding to the next image reconstruction module; input the second spectral response matrix and the target two-dimensional compressed measurement data into the fusion-guided reconstruction module of the next image reconstruction module, and guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module of the next image reconstruction module to obtain the second hyperspectral image corresponding to the next image reconstruction module; determine the second hyperspectral image as the first hyperspectral image; Repeat step one until the second hyperspectral image corresponding to the last image reconstruction module in the target number of image reconstruction modules is obtained, and the second hyperspectral image corresponding to the last image reconstruction module is determined as the target hyperspectral image.

[0047] Specifically, in some embodiments, the image reconstruction process in step 102 includes the following steps: First, the target RGB image data and the corresponding original hyperspectral image are input into the interpretable fusion module of the first image reconstruction module among the target number of image reconstruction modules to obtain the first spectral response matrix corresponding to the first image reconstruction module.

[0048] Specifically, regarding SRM learning issues: (2) The problem is solved iteratively using the proximal gradient descent (PGD) method, which consists of two steps: 1. Gradient descent step: Calculate intermediate variables : (3) in, : No. SRM intermediate variables in the next iteration : The spectral response matrix (SRM) learned in the previous stage. The current stage's reconstructed hyperspectral image estimate (initially 0 or a low-resolution estimate) is derived from the output of the previous stage's FRM. Gradient descent step size, scalar The target is RGB image data.

[0049] 2. Proximal mapping step: (4) in, The current updated spectral response matrix (SRM). Value matrix: An additional operator introduced for optimizing the solution.

[0050] Formula (4) above can be used to address complex SRM learning problems by introducing an Explainable Self-Attention Mechanism (ESAM), as follows: Calculated via ESAM module : Within the ESAM submodule, there is the Query module. :right The key is obtained through a linear transformation. :right The value is obtained through a linear transformation. : Generated via the TLCB module.

[0051] Output of the ESAM module: , Representing the scaling factor in the self-attention mechanism, the ESAM module transforms SRM learning into an attention mechanism, enhancing robustness to noise and camera differences.

[0052] Furthermore, the first spectral response matrix corresponding to the first image reconstruction module and the target two-dimensional compressed measurement data are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the first hyperspectral image corresponding to the first image reconstruction module.

[0053] Specifically, the process of solving the image reconstruction problem is illustrated below: The image reconstruction problem is as follows: (5) in, This represents the estimation of the reconstructed hyperspectral image. For the target two-dimensional compressed measurement data of the CASSI system, The sensing matrix of the CASSI system, The original hyperspectral image, , It is a hyperparameter that balances the weights of various components. For target RGB image data, It is a learnable spectral response matrix.

[0054] The image reconstruction problem is solved using the Half Quadratic Splitting (HQS) method, which introduces an auxiliary variable z to decompose the problem into: (6) in, This represents the current stage of hyperspectral image reconstruction estimate (initially 0 or a low-resolution estimate). , For the target two-dimensional compressed measurement data, For target RGB image data, , The spectral response matrix is ​​learnable. The sensing matrix of the CASSI system, This represents the hyperspectral image estimate reconstructed in the previous stage (initially 0 or a low-resolution estimate). This represents the penalty parameter of HQS. This represents the auxiliary variable from the previous stage. This represents the hyperspectral image estimate reconstructed in the previous stage (initially 0 or a low-resolution estimate).

[0055] Among them, auxiliary variables It is expressed as follows (7) Auxiliary variables for the current stage (stage k). Auxiliary variables (introducing the HQS algorithm). Regularization term : The penalty parameter for HQS.

[0056] Auxiliary variables By using a Multi-view Vision Transformer (MVT) to model image priors, the MVT design fully considers the characteristics of hyperspectral images. The PSA branch focuses on processing the continuity of the spectral dimension, while the CSC branch extracts local features in the spatial dimension. This dual-branch structure achieves complementary advantages through CFB, where spectral attention output features and spatial features undergo independent convolution and activation processing. This design leverages the Transformer architecture's ability to capture long-range dependencies while combining it with the CNN's ability to extract local features, demonstrating excellent performance in tests. The network maintains image texture details by preserving feature map resolution, and the number of channels in each layer is optimized to ensure that the model achieves optimal representation in both the spectral and spatial dimensions.

[0057] Step 1: Input the target RGB image data and the first hyperspectral image into the interpretable fusion module of the next image reconstruction module to obtain the second spectral response matrix corresponding to the next image reconstruction module; input the second spectral response matrix and the target two-dimensional compressed measurement data into the fusion-guided reconstruction module of the next image reconstruction module, and guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the second hyperspectral image corresponding to the next image reconstruction module; determine the second hyperspectral image as the first hyperspectral image; Repeat step one until the second hyperspectral image corresponding to the last image reconstruction module in the target number of image reconstruction modules is obtained, and the second hyperspectral image corresponding to the last image reconstruction module is determined as the target hyperspectral image.

[0058] According to a hyperspectral image reconstruction method provided by the present invention, target RGB image data and a first hyperspectral image are input into an interpretable fusion module in a next image reconstruction module to obtain a second spectral response matrix corresponding to the next image reconstruction module, including: Based on the first hyperspectral image, the target RGB image data, the first spectral response matrix, and the gradient descent step size, calculate the intermediate variables of the spectral response matrix corresponding to the next image reconstruction module; The second spectral response matrix is ​​determined based on the intermediate variables of the spectral response matrix and the target RGB image data through an interpretable self-attention mechanism module.

[0059] Specifically, in some embodiments, the implementation steps of the adaptive learning spectral response matrix (SRM) include: First, based on the first hyperspectral image, the target RGB image data, the first spectral response matrix, and the gradient descent step size, the intermediate variable of the spectral response matrix corresponding to the next image reconstruction module is calculated.

[0060] Specifically, regarding SRM learning issues: (2) The problem is solved iteratively using the proximal gradient descent (PGD) method, which consists of two steps: 1. Gradient descent step: Calculate intermediate variables : (3) in, : No. SRM intermediate variables in the next iteration : The spectral response matrix (SRM) learned in the previous stage. The current stage's reconstructed hyperspectral image estimate (initially 0 or a low-resolution estimate) is derived from the output of the previous stage's FRM. Gradient descent step size, scalar The target is RGB image data.

[0061] Furthermore, the second spectral response matrix is ​​determined based on the intermediate variables of the spectral response matrix and the target RGB image data through an interpretable self-attention mechanism module.

[0062] Specifically, the near-end mapping step is used to compute via the ESAM module. : (4) in, The current updated spectral response matrix (SRM). Value matrix: An additional operator introduced for optimizing the solution.

[0063] The above formula (4) can be handled by introducing an interpretable self-attention mechanism (ESAM), as follows: Calculated via ESAM module The steps are shown below: Within the ESAM submodule, the query matrix :right The key matrix is ​​obtained by linear transformation. :right The value matrix is ​​obtained by linear transformation. : Generated via the TLCB module.

[0064] Output of the ESAM module: , Representing the scaling factor in the self-attention mechanism, the ESAM module transforms SRM learning into an attention mechanism, enhancing robustness to noise and camera differences.

[0065] in, Figure 2 This is a schematic diagram of the framework of the interpretable fusion module provided by the present invention, as shown below. Figure 2 As shown: Input target RGB image data The updated spectral response matrix compared to the previous stage The spectral response matrix updated in the previous stage go through Processing of target RGB image data go through The handling, Let be the gradient descent step size, a scalar. The output of the k-1 stage interpretable fusion module is used; element-wise subtraction of the two outputs yields the updated spectral response matrix for the k-1 stage. ; Spectral response matrix updated for stage k-1 After processing, the query matrix is ​​obtained. For target RGB image data The process is performed to obtain the Keymatrix. Perform matrix multiplication on the two to calculate the score; normalize the score to obtain the k-stage updated spectral response matrix.

[0066] Then, perform the following on the target two-dimensional compressed measurement data y Processing yields one output; the spectral response matrix is ​​updated for k stages. The output of the k-1 stage can be explained by the fusion module. and intermediate variables Perform element-by-element merging to obtain ,Will Output of the k-1 stage interpretable fusion module Let's do " "Process to obtain two outputs; update the spectral response matrix in k stages." With intermediate variables Input them together into the TLCB module to obtain the value matrix. , value matrix and Perform element-wise multiplication, then combine with the target RGB image data. Perform element-wise multiplication to obtain three outputs; perform element-wise addition on the above three outputs to obtain the output of the k-stage interpretable fusion module.

[0067] This invention proposes a learnable dynamic SRM mechanism, which transforms the SRM learning process into the computation of an attention mechanism through the ESAM module, automatically adapting to the spectral response characteristics of different RGB cameras, avoiding the problem of precise calibration required by traditional methods, and significantly reducing the SRM estimation error in experiments.

[0068] According to the hyperspectral image reconstruction method provided by the present invention, the multi-view visual transformer includes a pixel-level spectral attention branch and a channel-level spatial convolution branch; The second spectral response matrix and the target two-dimensional compressed measurement data are input into the fusion-guided reconstruction module in the next image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module of the next image reconstruction module, resulting in the second hyperspectral image corresponding to the next image reconstruction module, including: By utilizing pixel-level spectral attention branches, the local spectral correlation of the first hyperspectral image is processed based on the second spectral response matrix and the target two-dimensional compressed measurement data through a self-attention mechanism to obtain the spectral attention output features of the first hyperspectral image; Using channel-level spatial convolution branches, spatial features of the first hyperspectral image are extracted by the convolution module based on the second spectral response matrix and the target two-dimensional compressed measurement data. The cross-fusion module integrates spectral attention output features and spatial features to output merged features; A second hyperspectral image is generated based on the merged features.

[0069] Specifically, the image reconstruction problem is represented as follows: in, This represents the estimation of the reconstructed hyperspectral image. For the target two-dimensional compressed measurement data of the CASSI system, The sensing matrix of the CASSI system, The original hyperspectral image, , It is a hyperparameter that balances the weights of various components. For target RGB image data, It is a learnable spectral response matrix.

[0070] The image reconstruction problem is solved using the Half Quadratic Splitting (HQS) method, which introduces an auxiliary variable z to decompose the problem into: (6) in, The current stage of hyperspectral image reconstruction estimation (initially 0 or low-resolution estimation). , For the target two-dimensional compressed measurement data, For target RGB image data, , The spectral response matrix is ​​learnable. The sensing matrix of the CASSI system, This represents the hyperspectral image estimate reconstructed in the previous stage (initially 0 or a low-resolution estimate). This represents the penalty parameter of HQS. This represents the auxiliary variable from the previous stage. This represents the hyperspectral image estimate reconstructed in the previous stage (initially 0 or a low-resolution estimate).

[0071] Among them, auxiliary variables It is expressed as follows (7) Auxiliary variables for the current stage (stage k). Auxiliary variables (introducing the HQS algorithm). Regularization term : The penalty parameter for HQS.

[0072] Among them, auxiliary variables Image priors are modeled using a Multi-view Vision Transformer (MVT). Figure 3 This is a schematic diagram of the structure of the multi-view visual transformer provided by the present invention. The specific network architecture is as follows: Figure 3 As shown, the Multi-view Vision Transformer (MVT) consists of two core processing branches: the pixel-level spectral attention (PSA) branch (i.e., a pixel-wise spectral attention mechanism) and the channel-level spatial convolution (CSC) branch (i.e., a channel-wise spatial convolutional network). The PSA branch handles local spectral correlations through a self-attention mechanism, while the CSC branch uses convolutional modules to extract spatial features. The outputs of the two branches are integrated through a cross-fusion module (CFB). In the pixel-wise spectral attention mechanism: input features are processed, and the value v is calculated. m Query q m and key k m For query q m and key k m Perform element-wise multiplication, and then multiply the result by the value v. mElement-wise multiplication yields the fused spectral features, which are then processed by a channel-wise spatial convolutional network. The fused spectral and spatial features are processed by Conv2d convolution and sigmoid activation functions, respectively, and finally merged into output features after Conv2d convolution by a cross-fusion module. The entire network structure maintains the input resolution to avoid information loss and ensures the preservation of accurate spectral characteristics and spatial details. The MVT design fully considers the characteristics of hyperspectral images; the PSA branch focuses on processing the continuity of the spectral dimension, while the CSC branch extracts local features in the spatial dimension. This dual-branch structure achieves complementary advantages through CFB, where the spectral attention output features and spatial features undergo independent convolution and activation processing. This design leverages the advantages of the Transformer architecture in capturing long-range dependencies while combining the ability of CNNs to extract local features, demonstrating excellent performance in tests. The network maintains image texture details by preserving feature map resolution, and the number of channels in each layer is optimized to ensure optimal representation in both the spectral and spatial dimensions.

[0073] Specifically, by utilizing pixel-level spectral attention branches, the local spectral correlation of the first hyperspectral image is processed based on the second spectral response matrix and the target two-dimensional compressed measurement data through a self-attention mechanism, thereby obtaining the spectral attention output features of the first hyperspectral image.

[0074] The PSA branch is used to model the global correlation between different spectral channels (such as the relationship between near-infrared and visible light bands), and the processing procedure is as follows: (1) Generate query ,key ,value : in, , , , For learnable weight matrix, This is the initial hyperspectral estimate from IFM or the output from the previous stage (the image to be reconstructed).

[0075] (2) Calculate the spectral attention output: in, The spectral attention output features of the first hyperspectral image, For value, As key, For query, For dimensions.

[0076] Specifically, using the channel-level spatial convolution branch, the spatial features of the first hyperspectral image are extracted by the convolution module based on the second spectral response matrix and the target two-dimensional compressed measurement data, thus obtaining the spatial features of the first hyperspectral image. The CSC branch processing procedure is as follows: Convolutional feature extraction, which processes spatial dimensions through multiple convolutional layers: in, The spatial features of the first hyperspectral image, For the initial hyperspectral estimate from IFM or the output of the previous stage (the image to be reconstructed), Conv represents a 3×3 convolutional layer used to extract local spatial features (such as edges and textures), and GELU is the Gaussian error linear unit activation function used to enhance nonlinear expressive power.

[0077] Furthermore, the spectral attention output features and spatial features are integrated through a cross-fusion module to output a merged feature. Specifically, the outputs of the two branches are first modulated separately: in, The modulated spectral attention output characteristics, The spatial features after modulation The spectral attention output features of the first hyperspectral image are represented by the sigmoid function, which generates a weight mask of 0-1 to identify important feature regions. The spatial features of the first hyperspectral image.

[0078] Furthermore, a second hyperspectral image is generated based on the merged features. Specifically, modulation features are merged and output for adaptive weighted fusion of spectral and spatial features, suppressing redundant information. in, This is the second hyperspectral image output. The modulated spectral attention output characteristics, The modulated spatial features.

[0079] The MVT module proposed in this invention captures both local spectral correlations and global spatial features through a dual-branch structure of PSA and CSC, and then achieves cross-fusion through CFB, effectively maintaining spectral accuracy and spatial details.

[0080] According to the hyperspectral image reconstruction method provided by the present invention, the training process of the hyperspectral image reconstruction network includes the following steps: Initialize network parameters; the initial network parameters include the weight information of the interpretable fusion module and the fusion-guided reconstruction module in the image reconstruction module; Acquire multiple training samples; the training samples include two-dimensional compressed measurement data of the samples and the corresponding RGB image data of the samples; Input the RGB image data of each sample and the original hyperspectral image corresponding to the RGB image data of each sample into the interpretable fusion module in the first image reconstruction module to obtain the third spectral response matrix corresponding to the first image reconstruction module; The third spectral response matrices and two-dimensional compressed measurement data of each sample corresponding to the first image reconstruction module are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the third hyperspectral images corresponding to the first image reconstruction module. Step 2: Input the RGB image data of each sample and each third hyperspectral image into the interpretable fusion module of the next image reconstruction module to obtain the fourth spectral response matrix corresponding to the next image reconstruction module; input the fourth spectral response matrix and the two-dimensional compressed measurement data of each sample into the fusion-guided reconstruction module of the next image reconstruction module, and guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the fourth hyperspectral image corresponding to the next image reconstruction module; determine each fourth hyperspectral image as the third hyperspectral image. Repeat step two until the preset convergence condition is met; the preset convergence condition is that the value of the multi-objective loss function is minimized. The model parameters that meet the preset convergence conditions are determined as the optimal network parameters, and the hyperspectral image reconstruction network is determined based on the optimal network parameters.

[0081] Specifically, in some embodiments, the training steps of the hyperspectral image reconstruction network include the following: Step (1): Initialize network parameters. These network parameters include the weight information of the interpretable fusion module and the fusion-guided reconstruction module within the image reconstruction module, for example... It is a hyperparameter that balances the weights of various components.

[0082] Step (2): Obtain multiple training samples. These training samples include two-dimensional compressed measurement data of the samples and the corresponding RGB image data of the samples. For example, training samples can be randomly selected from CAVE, KAIST, or Harvard datasets, containing two-dimensional compressed measurement data of the samples from the Coded Aperture Snapshot Spectral Imaging (CASSI) system and the corresponding RGB image data of the samples.

[0083] Step (3): Input the RGB image data of each sample and the original hyperspectral image corresponding to the RGB image data of each sample into the interpretable fusion module in the first image reconstruction module to obtain the third spectral response matrix corresponding to the first image reconstruction module.

[0084] Specifically, through the interpretable fusion module IFM, the third spectral response matrix SRM is adaptively learned using ESAM, that is, the mapping relationship between the RGB image data of each sample and the original hyperspectral data is established.

[0085] Step (4): Input the third spectral response matrices and two-dimensional compressed measurement data of each sample corresponding to the first image reconstruction module into the fusion-guided reconstruction module in the first image reconstruction module. Guide the image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the third hyperspectral images corresponding to the first image reconstruction module.

[0086] Specifically, based on the fusion-guided reconstruction module FRM, the learned third spectral response matrices SRM and sample two-dimensional compressed measurement data are used as inputs. The hyperspectral image is reconstructed through the parallel dual-branch structure of the image visual transformer MVT, resulting in the third hyperspectral images corresponding to the first image reconstruction module.

[0087] Step (5): Execute step two: Input the RGB image data of each sample and the third hyperspectral image into the interpretable fusion module of the next image reconstruction module to obtain the fourth spectral response matrix corresponding to the next image reconstruction module; input the fourth spectral response matrix and the two-dimensional compressed measurement data of each sample into the fusion-guided reconstruction module of the next image reconstruction module, guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the fourth hyperspectral image corresponding to the next image reconstruction module; determine the fourth hyperspectral image as the third hyperspectral image. That is, input the reconstruction result of the fusion-guided reconstruction module into the interpretable fusion module IFM of the next image reconstruction module, and repeat the operations of steps (3) and (4) until the last image reconstruction module. Step (6): Repeat step two until the preset convergence condition is met; the preset convergence condition is that the value of the multi-objective loss function is minimized.

[0088] Specifically, calculate the multi-objective loss function, which includes data fidelity loss, spectral consistency loss, and spatial detail loss; evaluate the convergence condition; if convergence fails, update the network parameters through backpropagation and return to step (2); if convergence occurs, save the optimal model parameters.

[0089] Step (7): Determine the model parameters that meet the preset convergence conditions as the optimal network parameters, and determine the hyperspectral image reconstruction network based on the optimal network parameters.

[0090] This invention proposes a hyperspectral image reconstruction method based on an interpretable fusion-sensory depth unfolding framework. It innovatively combines the mathematical interpretability of the physical model with the adaptive learning capability of deep learning, and realizes high-quality image reconstruction of the dual-camera coded aperture snapshot spectral imaging DC-CASSI system.

[0091] According to the hyperspectral image reconstruction method provided by the present invention, before determining the model parameters that satisfy the preset convergence condition as the optimal network parameters and determining the hyperspectral image reconstruction network based on the optimal network parameters, the method further includes: Based on each fourth hyperspectral image, determine the data fidelity loss, spectral consistency loss, and spatial detail loss; The value of the multi-objective loss function is calculated based on data fidelity loss, spectral consistency loss, and spatial detail loss.

[0092] Specifically, in some embodiments, the model parameters that meet preset convergence conditions are determined as the optimal network parameters. Before determining the hyperspectral image reconstruction network based on these optimal network parameters, a multi-objective loss function is also determined. An example of this process is as follows: First, the data fidelity loss, spectral consistency loss, and spatial detail loss are determined based on each fourth hyperspectral image.

[0093] Among them, data fidelity loss and spatial detail loss are , For two-dimensional compression measurement of the CASSI system, It is the sensing matrix of the CASSI system. The original hyperspectral image; spectral consistency loss is .

[0094] Furthermore, based on data fidelity loss, spectral consistency loss, and spatial detail loss, the value of the multi-objective loss function is calculated. For example, the multi-objective loss function is the structural similarity index loss.

[0095] The method provided in this embodiment designs a multi-objective loss function, including data fidelity loss, spectral consistency loss, and spatial detail loss, to guide the neural network in progressively optimizing the hyperspectral image reconstruction results. These optimization objectives include measurement data consistency loss, spectral response matrix (SRM) learning loss, and spatial-spectral attention output feature preservation loss. This method combines the physical interpretability of traditional optimization methods with the powerful representational capabilities of deep learning methods, effectively overcoming the dependence of traditional methods on precise system calibration and significantly improving reconstruction accuracy and robustness. Through this physical model-based deep learning approach, the algorithm's dependence on large-scale labeled data is greatly reduced.

[0096] Figure 4 This is the second flowchart of the hyperspectral image reconstruction method provided by the present invention, illustrating the model training process, as shown below. Figure 4 As shown, the method includes a training phase and a testing phase. The training phase includes: network initialization, inputting random training samples, processing the images through image reconstruction models 1 to K to obtain image reconstruction results, and then calculating the loss function based on the image reconstruction results. Furthermore, it determines whether the network has converged and decides whether to terminate or adjust the network parameters.

[0097] The testing phase includes: loading the network and weights, inputting test samples, and reconstructing the hyperspectral image.

[0098] The hyperspectral image reconstruction apparatus provided by the present invention is described below. The hyperspectral image reconstruction apparatus described below can be referred to in correspondence with the hyperspectral image reconstruction method described above.

[0099] Figure 5 This is a schematic diagram of the hyperspectral image reconstruction device provided by the present invention, as shown below. Figure 5 As shown, the hyperspectral image reconstruction device 500 includes the following modules: The acquisition module 510 is used to acquire the target two-dimensional compressed measurement data of the coded aperture snapshot spectral imaging CASSI system and the target RGB image data corresponding to the target two-dimensional compressed measurement data; The reconstruction module 520 is used to input the target two-dimensional compressed measurement data and the target RGB image data into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0100] The apparatus provided in this embodiment includes an acquisition module 510 for acquiring target two-dimensional compressed measurement data and target RGB image data corresponding to the target two-dimensional compressed measurement data from the coded aperture snapshot spectral imaging (CASSI) system; and a reconstruction module 520 for inputting the target two-dimensional compressed measurement data and target RGB image data into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules, each including an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module guides the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0101] This invention addresses the difficulty of spectral response matrix (SRM) calibration in DC-CASSI systems by proposing a learnable SRM estimation strategy based on an attention mechanism. Specifically, the corresponding target spectral response matrix is ​​learned by solving a multi-objective optimization problem through an interpretable fusion module in the image reconstruction module, effectively achieving high-quality spectral image reconstruction.

[0102] According to the present invention, a hyperspectral image reconstruction apparatus 500 is provided, wherein the reconstruction module 520 is specifically used for: The target RGB image data and the original hyperspectral image corresponding to the target RGB image data are input into the interpretable fusion module in the first image reconstruction module among the target number of image reconstruction modules to obtain the first spectral response matrix corresponding to the first image reconstruction module; The first spectral response matrix corresponding to the first image reconstruction module and the target two-dimensional compressed measurement data are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the first hyperspectral image corresponding to the first image reconstruction module. Step 1: Input the target RGB image data and the first hyperspectral image into the interpretable fusion module in the next image reconstruction module to obtain the second spectral response matrix corresponding to the next image reconstruction module; input the second spectral response matrix and the target two-dimensional compressed measurement data into the fusion-guided reconstruction module in the next image reconstruction module, and guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the second hyperspectral image corresponding to the next image reconstruction module; determine the second hyperspectral image as the first hyperspectral image; Repeat step one until the second hyperspectral image corresponding to the last image reconstruction module in the target number of image reconstruction modules is obtained, and determine the second hyperspectral image corresponding to the last image reconstruction module as the target hyperspectral image.

[0103] According to a hyperspectral image reconstruction method provided by the present invention, the reconstruction module 520 is further configured to: Based on the first hyperspectral image, the target RGB image data, the first spectral response matrix, and the gradient descent step size, calculate the intermediate variable of the spectral response matrix corresponding to the next image reconstruction module; The second spectral response matrix is ​​determined based on the intermediate variables of the spectral response matrix and the target RGB image data using an interpretable self-attention mechanism module.

[0104] According to a hyperspectral image reconstruction method provided by the present invention, the multi-view visual transformer includes a pixel-level spectral attention branch and a channel-level spatial convolution branch; The reconstruction module 520 is further configured to: Using the pixel-level spectral attention branch, the local spectral correlation of the first hyperspectral image is processed by a self-attention mechanism based on the second spectral response matrix and the target two-dimensional compressed measurement data to obtain the spectral attention output features of the first hyperspectral image; Using the channel-level spatial convolution branch, the spatial features of the first hyperspectral image are extracted by the convolution module based on the second spectral response matrix and the target two-dimensional compressed measurement data. The spectral attention output features and the spatial features are integrated by a cross-fusion module to output a merged feature; Based on the merged features, the second hyperspectral image is generated.

[0105] According to a hyperspectral image reconstruction method provided by the present invention, the apparatus further includes a training module; The training module is used for: Initialize network parameters; the initial network parameters include the weight information of the interpretable fusion module and the fusion-guided reconstruction module in the image reconstruction module; Multiple training samples are acquired; the training samples include sample two-dimensional compressed measurement data and sample RGB image data corresponding to the sample two-dimensional compressed measurement data. The sample RGB image data and the original hyperspectral image corresponding to each sample RGB image data are input into the interpretable fusion module in the first image reconstruction module to obtain the third spectral response matrix corresponding to the first image reconstruction module; The third spectral response matrices corresponding to the first image reconstruction module and the two-dimensional compressed measurement data of each sample are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the third hyperspectral images corresponding to the first image reconstruction module. Step 2: Input the RGB image data of each sample and the third hyperspectral image of each sample into the interpretable fusion module of the next image reconstruction module to obtain the fourth spectral response matrix corresponding to the next image reconstruction module; input the fourth spectral response matrix and the two-dimensional compressed measurement data of each sample into the fusion-guided reconstruction module of the next image reconstruction module, and guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the fourth hyperspectral image corresponding to the next image reconstruction module; determine each fourth hyperspectral image as the third hyperspectral image. Repeat step two until a preset convergence condition is met; the preset convergence condition is that the value of the multi-objective loss function is minimized. The model parameters that satisfy the preset convergence condition are determined as the optimal network parameters, and the hyperspectral image reconstruction network is determined based on the optimal network parameters.

[0106] According to the hyperspectral image reconstruction apparatus 500 provided by the present invention, the training module is further configured to: Based on each fourth hyperspectral image, determine the data fidelity loss, spectral consistency loss, and spatial detail loss; The value of the multi-objective loss function is calculated based on the data fidelity loss, the spectral consistency loss, and the spatial detail loss.

[0107] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 The electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute a hyperspectral image reconstruction method, which includes: Acquire the target two-dimensional compressed measurement data of the coded aperture snapshot spectral imaging CASSI system and the target RGB image data corresponding to the target two-dimensional compressed measurement data; The target two-dimensional compressed measurement data and the target RGB image data are input into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0108] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the hyperspectral image reconstruction method provided by the above methods, the method comprising: Acquire the target two-dimensional compressed measurement data of the coded aperture snapshot spectral imaging CASSI system and the target RGB image data corresponding to the target two-dimensional compressed measurement data; The target two-dimensional compressed measurement data and the target RGB image data are input into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0110] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image reconstruction method provided by the methods described above, the method comprising: Acquire the target two-dimensional compressed measurement data of the coded aperture snapshot spectral imaging CASSI system and the target RGB image data corresponding to the target two-dimensional compressed measurement data; The target two-dimensional compressed measurement data and the target RGB image data are input into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral image reconstruction method, characterized in that, include: Acquire the target two-dimensional compressed measurement data of the coded aperture snapshot spectral imaging CASSI system and the target RGB image data corresponding to the target two-dimensional compressed measurement data; The target two-dimensional compressed measurement data and the target RGB image data are input into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

2. The hyperspectral image reconstruction method according to claim 1, characterized in that, The step of inputting the target two-dimensional compressed measurement data and the target RGB image data into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image includes: The target RGB image data and the original hyperspectral image corresponding to the target RGB image data are input into the interpretable fusion module in the first image reconstruction module among the target number of image reconstruction modules to obtain the first spectral response matrix corresponding to the first image reconstruction module; The first spectral response matrix corresponding to the first image reconstruction module and the target two-dimensional compressed measurement data are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the first hyperspectral image corresponding to the first image reconstruction module. Step 1: Input the target RGB image data and the first hyperspectral image into the interpretable fusion module in the next image reconstruction module to obtain the second spectral response matrix corresponding to the next image reconstruction module; input the second spectral response matrix and the target two-dimensional compressed measurement data into the fusion-guided reconstruction module in the next image reconstruction module, and guide image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module to obtain the second hyperspectral image corresponding to the next image reconstruction module; determine the second hyperspectral image as the first hyperspectral image; Repeat step one until the second hyperspectral image corresponding to the last image reconstruction module in the target number of image reconstruction modules is obtained, and the second hyperspectral image corresponding to the last image reconstruction module is determined as the target hyperspectral image.

3. The hyperspectral image reconstruction method according to claim 2, characterized in that, The step of inputting the target RGB image data and the first hyperspectral image into the interpretable fusion module in the next image reconstruction module to obtain the second spectral response matrix corresponding to the next image reconstruction module includes: Based on the first hyperspectral image, the target RGB image data, the first spectral response matrix, and the gradient descent step size, calculate the intermediate variable of the spectral response matrix corresponding to the next image reconstruction module; The second spectral response matrix is ​​determined based on the intermediate variables of the spectral response matrix and the target RGB image data using an interpretable self-attention mechanism module.

4. The hyperspectral image reconstruction method according to claim 2, characterized in that, The multi-view visual transformer includes a pixel-level spectral attention branch and a channel-level spatial convolution branch; The step of inputting the second spectral response matrix and the target two-dimensional compressed measurement data into the fusion-guided reconstruction module in the next image reconstruction module, and guiding image reconstruction through the multi-view visual transformer in the fusion-guided reconstruction module of the next image reconstruction module to obtain the second hyperspectral image corresponding to the next image reconstruction module includes: Using the pixel-level spectral attention branch, the local spectral correlation of the first hyperspectral image is processed by a self-attention mechanism based on the second spectral response matrix and the target two-dimensional compressed measurement data to obtain the spectral attention output features of the first hyperspectral image; Using the channel-level spatial convolution branch, the spatial features of the first hyperspectral image are extracted by the convolution module based on the second spectral response matrix and the target two-dimensional compressed measurement data. The spectral attention output features and the spatial features are integrated by a cross-fusion module to output a merged feature; Based on the merged features, the second hyperspectral image is generated.

5. The hyperspectral image reconstruction method according to claim 1, characterized in that, The training process of the hyperspectral image reconstruction network includes the following steps: Initialize network parameters; the initial network parameters include the weight information of the interpretable fusion module and the fusion-guided reconstruction module in the image reconstruction module; Multiple training samples are acquired; the training samples include sample two-dimensional compressed measurement data and sample RGB image data corresponding to the sample two-dimensional compressed measurement data. Each sample RGB image data and the corresponding original hyperspectral image are input into the interpretable fusion module in the first image reconstruction module to obtain each third spectral response matrix corresponding to the first image reconstruction module; The third spectral response matrices corresponding to the first image reconstruction module and the two-dimensional compressed measurement data of each sample are input into the fusion-guided reconstruction module in the first image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain the third hyperspectral images corresponding to the first image reconstruction module. Step 2: Input the RGB image data of each sample and the third hyperspectral image of each sample into the interpretable fusion module in the next image reconstruction module to obtain the fourth spectral response matrix corresponding to the next image reconstruction module; Each of the fourth spectral response matrices and each of the sample two-dimensional compressed measurement data are input into the fusion-guided reconstruction module in the next image reconstruction module. The image reconstruction is guided by the multi-view visual transformer in the fusion-guided reconstruction module to obtain each of the fourth hyperspectral images corresponding to the next image reconstruction module. Each of the fourth hyperspectral images is determined as each of the third hyperspectral images. Repeat step two until a preset convergence condition is met; the preset convergence condition is that the value of the multi-objective loss function is minimized. The model parameters that satisfy the preset convergence condition are determined as the optimal network parameters, and the hyperspectral image reconstruction network is determined based on the optimal network parameters.

6. The hyperspectral image reconstruction method according to claim 5, characterized in that, Before determining the model parameters that satisfy the preset convergence condition as the optimal network parameters, and before determining the hyperspectral image reconstruction network based on the optimal network parameters, the method further includes: Based on each of the fourth hyperspectral images, determine the data fidelity loss, spectral consistency loss, and spatial detail loss; The value of the multi-objective loss function is calculated based on the data fidelity loss, the spectral consistency loss, and the spatial detail loss.

7. A hyperspectral image reconstruction device, characterized in that, include: The acquisition module is used to acquire the target two-dimensional compressed measurement data of the coded aperture snapshot spectral imaging CASSI system and the target RGB image data corresponding to the target two-dimensional compressed measurement data; A reconstruction module is used to input the target two-dimensional compressed measurement data and the target RGB image data into a hyperspectral image reconstruction network to obtain a reconstructed target hyperspectral image. The hyperspectral image reconstruction network includes a target number of image reconstruction modules. Each image reconstruction module includes an interpretable fusion module and a fusion-guided reconstruction module. The interpretable fusion module learns the corresponding target spectral response matrix by solving a multi-objective optimization problem. The multi-objective optimization problem is determined based on the fidelity of the target two-dimensional compressed measurement data, the constraints of the target RGB image data, and a regularization term. The fusion-guided reconstruction module is used to guide the reconstruction of the target hyperspectral image through the target spectral response matrix.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the hyperspectral image reconstruction method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hyperspectral image reconstruction method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the hyperspectral image reconstruction method as described in any one of claims 1 to 6.

Images