CASSI system and hyperspectral image reconstruction method based on orthogonal two-dispersion

Abstract

This invention provides a coded aperture snapshot spectral imaging system and hyperspectral image reconstruction method based on orthogonal dual-dispersion. The system acquires horizontal and vertical compressed measurement images by synchronously compressing samples through orthogonally set horizontal and vertical dispersion channels and a shared spatial modulation mask. The reconstruction method includes: establishing an optimization equation containing prior terms based on an imaging mathematical model, constructing a dual-dispersion depth unfolding framework (DDDU), and decomposing the optimization equation into data subproblems and prior subproblems. The data subproblems are independently optimized through closed-form solutions of horizontal and vertical dispersion, preserving the complementary information structure of the two channels; the prior subproblems are jointly denoised and feature-fused using a dual-dispersion information fusion Transformer (DDIFT). High-fidelity hyperspectral image reconstruction is achieved through iterative solution. This invention innovatively combines orthogonal dual-dispersion channel design to improve the sampling rate and effectively utilizes complementary spectral information through an algorithm combining physical models and deep learning, thereby improving imaging and reconstruction performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of spectral imaging and hyperspectral reconstruction, and particularly relates to an orthogonal double-dispersion coded aperture snapshot spectral imaging (ODD-CASSI) system and a hyperspectral image reconstruction method thereof. BACKGROUND

[0002] Hyperspectral imaging (HSI) technology can continuously record a high-spectral-resolution image sequence, usually containing tens to hundreds of spectral images, providing key data for analyzing the spatial distribution and characteristic spectral properties of the target. As a new way of acquiring image and spectral information, coded aperture snapshot spectral imaging (CASSI) compresses three-dimensional data into two-dimensional projections through a single exposure, greatly reducing the acquisition time and the amount of data. However, the traditional CASSI relies on a single dispersion channel, resulting in low sampling rate and insufficient spatial-spectral information separation capability in complex scenes, and the reconstructed results have structural blur and spectral distortion.

[0003] Existing reconstruction methods have significant limitations: traditional iterative optimization methods (such as GAP and TwIST) rely on manual priors, are prone to over-smoothing, and have weak generalizability; end-to-end deep learning methods improve efficiency but ignore the physical degradation model and have poor interpretability; the plug-and-play (PnP) framework embeds a deep denoiser into an optimization algorithm, but the iterative efficiency is low; and the deep unfolding network combines the advantages of model-driven and data-driven methods, but is designed mainly for single-dispersion CASSI systems and is difficult to adapt to new multi-channel imaging architectures.

[0004] Therefore, there are still some important problems to be studied: first, how to break through the limitations of single-dispersion CASSI systems and design a multi-dispersion snapshot spectral imaging system to improve the sampling capability of hyperspectral data; second, how to design a reconstruction algorithm that adapts to the new system to improve the reconstruction efficiency and accuracy and realize the collaborative optimization of the physical model and deep learning. To solve the above problems, the present application proposes an orthogonal double-dispersion coded aperture snapshot spectral imaging (ODD-CASSI) system and a hyperspectral image reconstruction method thereof. SUMMARY

[0005] The present application aims to provide an orthogonal double-dispersion coded aperture snapshot spectral imaging (ODD-CASSI) system to break through the limited sampling capability of existing snapshot spectral imaging systems and to construct a hyperspectral image reconstruction method that combines physical models and deep learning, achieving efficient reconstruction.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] An orthogonal dual-dispersion-based coded aperture snapshot spectral imaging (ODD-CASSI) system, comprising:

[0008] An orthogonal dual-dispersion sampling module for generating horizontal and vertical dispersion compressed measurement images, splitting the coded mask modulated light signal into two paths through a beam splitter, inputting the two paths into a horizontal dispersion channel and a vertical dispersion channel respectively, and using a horizontal dispersion grating and a vertical dispersion grating to respectively disperse and expand the spectral information in the horizontal and vertical directions; synchronously capturing the compressed measurement images of the two channels by a detector and transmitting them to a reconstruction module;

[0009] A depth unfolding reconstruction module for reconstructing a spectral image cube, inputting the horizontal and vertical compressed measurement images into a dual-dispersion depth unfolding framework (DDDU) based on the dual-dispersion depth unfolding framework (DDDU), updating two intermediate variables through two closed-form solutions of horizontal and vertical dispersion data sub-problems, inputting the two intermediate variables into a dual-dispersion information fusion Transformer (DDIFT), and performing cross-channel feature alignment and noise suppression through a bidirectional fusion branch module (BFCB);

[0010] An information fusion module for extracting multi-level features of the horizontal and vertical dispersion channels through a multi-scale U-Net architecture of the dual-dispersion information fusion Transformer (DDIFT); using a bidirectional fusion branch module (BFCB) as a unit of the dual-dispersion information fusion Transformer (DDIFT) to perform splicing, multi-head self-attention fusion and separation operations on the multi-level dual-channel features, generate shared features and channel-independent features; and integrating multi-scale features through full-scale skip connection to preserve spatial details and spectral consistency;

[0011] A reconstruction output module for fusing dual-channel features through a 1x1 convolution layer and outputting a restored spectral image cube at the last stage; and outputting the reconstruction result through a visualization interface.

[0012] A hyperspectral image reconstruction method of an orthogonal dual-dispersion-based coded aperture snapshot spectral imaging (ODD-CASSI) system, comprising the following steps:

[0013] Using an orthogonal dual-dispersion-based coded aperture snapshot spectral imaging (ODD-CASSI) system, synchronously compressively sampling target spectral information through a horizontal dispersion channel and a vertical dispersion channel to obtain a horizontal dispersion compressed measurement image and a vertical dispersion compressed measurement image, wherein the horizontal dispersion channel and the vertical dispersion channel share the same spatial modulation coded mask and the dispersion directions are orthogonal;

[0014] For the input orthogonal double-dispersion coded aperture snapshot spectral imaging measurement image, an optimization solving equation for solving the spectral image containing a priori term is established in combination with an imaging mathematical model of the orthogonal double-dispersion coded aperture snapshot spectral image;

[0015] A double-dispersion depth unfolding framework (DDDU) is constructed, and the optimization solving equation for solving the spectral image containing the a priori term is decomposed into a data subproblem and a priori subproblem, wherein the data subproblem is further decomposed into a horizontal dispersion data subproblem and a vertical dispersion data subproblem; the horizontal and vertical dispersion data subproblems have closed-form solutions, and are independently optimized through horizontal and vertical closed-form solutions respectively, and the complementary information structure of the double-dispersion channel is reserved; the priori subproblem performs joint denoising and feature fusion on two solutions of the optimized data subproblem through a double-dispersion information fusion Transformer (DDIFT), and outputs a denoising result;

[0016] Based on the double-dispersion depth unfolding framework (DDDU), the data subproblem and the priori subproblem are iteratively solved, a loss of a predicted spectral image reconstructed and a real spectral image is calculated until convergence, and finally a high-fidelity spectral image is output.

[0017] Preferably, a hardware structure of the orthogonal double-dispersion coded aperture snapshot spectral imaging (ODD-CASSI) system comprises:

[0018] one objective lens, one random coding mask, one beam splitter, four relay lenses, a horizontal dispersion grating, a vertical dispersion grating, and two detectors;

[0019] The coded light signal is split into two paths by the beam splitter, and the two paths respectively pass through the horizontal dispersion grating and the vertical dispersion grating, and horizontal direction dispersion unfolded compressed measurement images and vertical direction dispersion unfolded compressed measurement images are respectively generated on the two detectors.

[0020] Preferably, a function expression of the imaging mathematical model of the orthogonal double-dispersion coded aperture snapshot spectral image is:

[0021] g h =Φ h f h

[0022] g v =Φ v f v

[0023] wherein g h and g v are horizontal and vertical dispersion compressed measurement images of the orthogonal double-dispersion coded aperture snapshot spectral imaging respectively; Φ h and Φ vmeasurement matrices of horizontal and vertical dispersion channels, respectively, which are both block-diagonal sparse matrices, generated by the same random encoding mask along horizontal and vertical dispersion shifts, respectively; f h and f v are vectorized representations of the horizontal and vertical dispersed spectral images, respectively;

[0024] The function expression of the optimization solution equation of the spectral image including the priori term is:

[0025]

[0026] wherein, is the estimated value of the vectorized form of the spectral image; t and r are balance factors; R(f h ) and D(f v ) are the priori terms of the spectral image.

[0027] Preferably, the function expression of the optimization solution equation of the horizontal and vertical dispersed data sub-problems is:

[0028]

[0029]

[0030] wherein, and are the estimated values of the vectorized form of the horizontal and vertical dispersed spectral images, respectively; μ h and μ v are penalty coefficients; and are auxiliary variables introduced;

[0031] The optimization method of the closed-form solution of the horizontal and vertical dispersed data sub-problems includes:

[0032] By using the block-diagonal sparsity of the horizontal and vertical measurement matrices and, the matrix inversion operation is simplified to a pixel-by-pixel linear operation;

[0033] The diagonal elements of and are pre-computed for accelerating the update of and in the iterative process; and and

[0034] The expression of the closed-form solution of the optimization solution equation of the horizontal and vertical dispersed data sub-problems is:

[0035]

[0036] wherein, diag(·) represents extracting elements from the diagonal line thereof to form a vector;

[0037] ​​The function expression of the optimization solution equation of the prior sub-problem is:

[0038]

[0039] The above formula is regarded as a joint denoising task of with noise level and with noise level , then the optimization solution equation of the prior sub-problem is further expressed as:

[0040]

[0041] where Denoiser is a DDIFT denoiser, and is set to

[0042] t k , r k , is the element of the vector inferred by the average pooling layer and the multi-layer perception (MLP) layer once at the k-th stage, when k = 1, from the initial values of the horizontal and vertical dispersed spectral images and , for estimating the noise level of each stage in the depth unfolding of the double-dispersed channel information;

[0043] The initial values of the horizontal and vertical dispersed spectral images and input to the first stage (k = 1) are generated by the horizontal and vertical dispersed compressed measurement images g h and g v and the corresponding measurement matrices Φ h and Φ v , respectively, and and of the subsequent iteration stages i = 1, 2,..., k-1 are generated by the horizontal and vertical shifts of z i , respectively, and the horizontal and vertical shifts are used to simulate the horizontal and vertical dispersion of the ODD-CASSI system.

[0044] Preferably, the structure of the double-dispersed information fusion Transformer (DDIFT) comprises:

[0045] Based on the improved U-Net encoder-decoder structure, multi-level features are extracted through four double convolution encoding layers, four deconvolution decoding layers restore the feature map size, and full-scale skip connection is used to fuse low, medium and high level features; wherein, a bidirectional fusion branch module (BFCB) is used as the core unit of the double dispersion information fusion Transformer (DDIFT) for mining and retaining the multi-level shared characteristics and independent characteristics of the double dispersion information; finally, the double dispersion information fusion Transformer (DDIFT) fuses the double channel features through a 1x1 convolution layer to generate a denoised spectral image.

[0046] Preferably, the bidirectional fusion branch module (BFCB) is used to perform the following operations:

[0047] The horizontal dispersion feature X h ∈R H×W×C and the vertical dispersion feature X v ∈R H×W×C are spliced along the channel dimension to obtain X cat ∈R H ×W×2C ;

[0048] The layer normalization (LN), multi-head self-attention (MSA) and feed-forward full connection layer (FFN) are sequentially performed on X enhanced ∈R H×W×2C ;

[0049] X enhanced is separated into updated X h ∈R H×W×C and X v ∈R H×W×C by the Chunk operation.

[0050] The application further protects a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the above-mentioned hyperspectral image reconstruction method of the CASSI system based on orthogonal double dispersion when executing the computer program.

[0051] The application further protects a computer readable storage medium storing a computer program, characterized in that the computer program is executed by a processor to implement the above-mentioned hyperspectral image reconstruction method of the CASSI system based on orthogonal double dispersion.

[0052] Compared with the prior art, the application has the following beneficial technical effects:

[0053] The application provides an orthogonal double dispersion-based coded aperture snapshot spectral imaging (ODD-CASSI) system, which is based on a double dispersion orthogonal sampling design, synchronously captures complementary information through horizontal and vertical dispersion channels, and improves a data sampling rate; in addition, the application explores a hyperspectral image reconstruction algorithm suitable for the orthogonal double dispersion-based coded aperture snapshot spectral imaging system, combines the advantages of a physical model and deep learning, mines multi-level features of double dispersion information in a double dispersion depth unfolding framework (DDDU) through double dispersion information fusion Transformer (DDIFT) and a bidirectional fusion branch module (BFCB), and extracts and fuses shared features.

[0054] The application performs comparative experiments on simulation data sets CAVE and KAIST and real data, the average PSNR of the method reaches 42.15 dB, the SSIM reaches 0.986, and the SAM reaches 2.768 on the simulation data set, in addition, the method is superior to the most advanced depth unfolding algorithm (9-stg) of CASSI in a lower stage (3-stg), has lower calculation cost (53.5%) and better image quality (SSIM 0.981, PSNR 40.39), and has less artifact noise and the highest spectral curve consistency on real data. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A flowchart of a hyperspectral image reconstruction method of an orthogonal double dispersion-based coded aperture snapshot spectral imaging (ODD-CASSI) system in embodiment 1 of the application is shown;

[0056] Figure 2 A hardware structure schematic diagram of an orthogonal double dispersion coded aperture snapshot spectral imaging (ODD-CASSI) system in embodiment 1 of the application is shown, which shows the layout of a mask, a beam splitter, a dispersion grating and a detector;

[0057] Figure 3 A double dispersion depth unfolding framework (DDDU) and a double dispersion information fusion Transformer (DDIFT) structure diagram in embodiment 1 of the application are shown, which labels data sub-problems, prior sub-problems and a bidirectional fusion branch module (BFCB) flow;

[0058] Figure 4 A comparison diagram of calculation cost and reconstruction performance of the method and different algorithms in the prior art on KAIST and CAVE simulation data sets in embodiment 1 of the application is shown;

[0059] Figure 5 A comparison diagram of real experiment reconstruction results and spectral curves in embodiment 1 of the application is shown;

[0060] Figure 6Ablation experiment comparison chart of the present method and baseline in embodiment 1 of the present application is shown. DETAILED DESCRIPTION

[0061] The present application will be further described below in connection with specific embodiments, which are intended to explain the present application but not to limit it.

[0062] In order to make the personnel in the art better understand the present application scheme, the technical scheme in the embodiments of the present application will be clearly and completely described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person of ordinary skill in the art without making creative labor should belong to the scope of protection of the present application.

[0063] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0064] The application provides a hyperspectral image reconstruction method based on an orthogonal double dispersion coding aperture snapshot spectral imaging (ODD-CASSI) system. The method uses a horizontal dispersion channel and a vertical dispersion channel to synchronously compress and sample spectral information of a target through an orthogonal double dispersion coding aperture snapshot spectral imaging (ODD-CASSI) system, so as to obtain a horizontal dispersion compressed measurement image and a vertical dispersion compressed measurement image. The two channels share the same spatial modulation coding mask, and the dispersion directions are orthogonal. For the input orthogonal double dispersion coding aperture snapshot spectral imaging measurement image, an optimization solving equation for solving a spectral image is established in combination with an imaging mathematical model of the orthogonal double dispersion coding aperture snapshot spectral image. A double dispersion deep unfolding framework (DDDU) is constructed, and the optimization solving equation for solving the spectral image containing the priori term is decomposed into a data sub-problem and a priori sub-problem. The data sub-problem is further decomposed into a horizontal dispersion data sub-problem and a vertical dispersion data sub-problem. The horizontal and vertical dispersion data sub-problems have closed-form solutions, and are independently optimized through horizontal and vertical closed-form solutions, respectively, so as to retain the complementary information structure of the double dispersion channels. The priori sub-problem is used for jointly denoising and feature fusion of two solutions of the optimized data sub-problem through a double dispersion information fusion Transformer (DDIFT), and a denoising result is output. Based on the double dispersion deep unfolding framework, the data sub-problem and the priori sub-problem are iteratively solved until convergence, and finally a high-fidelity spectral image is output. The application is described below in combination with related drawings and specific examples, and the specific content is as follows.

[0065] Embodiment 1

[0066] The application provides an orthogonal double dispersion coding aperture snapshot spectral imaging (ODD-CASSI) system, comprising:

[0067] The orthogonal double dispersion sampling module is configured to:

[0068] The horizontal and vertical dispersion compressed measurement images are generated by splitting the coding mask modulated light signal into two paths through a beam splitter, inputting the light signal into a horizontal dispersion channel and a vertical dispersion channel respectively, and using a horizontal dispersion grating and a vertical dispersion grating to respectively perform horizontal and vertical dispersion unfolding on the spectral information.

[0069] The compressed measurement images of the two channels are synchronously captured by a detector and transmitted to a reconstruction module.

[0070] The deep unfolding reconstruction module is configured to:

[0071] For reconstructing a spectral image cube, based on a double-dispersion deep unfolding framework (DDDU), inputting horizontal and vertical compressed measurement images into the DDDU, updating intermediate variables through two closed-form solutions of horizontal and vertical dispersion data sub-problems, inputting the two intermediate variables into a double-dispersion information fusion Transformer (DDIFT), and performing cross-channel feature alignment and noise suppression through a bidirectional fusion branch module (BFCB);

[0072] The information fusion module is configured to:

[0073] The multi-scale U-Net architecture of the DDIFT extracts multi-level features of the horizontal and vertical dispersion channels; the BFCB is used as a unit of the DDIFT to perform splicing, multi-head self-attention fusion and separation operations on the multi-level double-channel features, to generate shared features and channel-independent features; and the multi-scale features are integrated through full-scale skip connection to retain spatial details and spectral consistency.

[0074] The reconstruction output module is configured to:

[0075] The double-channel features are fused through a 1x1 convolutional layer to output a restored spectral image cube in the last stage.

[0076] The reconstruction result is output through a visualization interface.

[0077] Based on the above, the present application further proposes a hyperspectral image reconstruction method based on an orthogonal double-dispersion coded aperture snapshot spectral imaging (ODD-CASSI) system, please refer to Figure 1 , Figure 1 A flowchart of the above method is shown, including the following steps:

[0078] Step 1:

[0079] Through the orthogonal double-dispersion coded aperture snapshot spectral imaging system, the horizontal dispersion channel and the vertical dispersion channel are used to synchronously compress and sample the target spectral information, to obtain a horizontal dispersion compressed measurement image and a vertical dispersion compressed measurement image, wherein the double channels share the same spatial modulation coding mask, and the dispersion directions are orthogonal.

[0080] Specifically, in the embodiments of the present disclosure, the hardware structure of the orthogonal double-dispersion coded aperture snapshot spectral imaging system includes:

[0081] one objective lens, one random coding mask, one beam splitter, four relay lenses, a horizontal dispersion grating, a vertical dispersion grating, and two detectors.

[0082] As Figure 2As shown, the coded light signal is split into two paths by a beam splitter, and then passes through a horizontal dispersion grating and a vertical dispersion grating respectively, and generates a horizontally dispersed compressed measurement image and a vertically dispersed compressed measurement image on two detectors respectively.

[0083] Step 2:

[0084] For the input orthogonal double-dispersion coded aperture snapshot spectral imaging measurement image, an optimization solving equation for solving the spectral image containing a priori term is established in combination with an imaging mathematical model of the orthogonal double-dispersion coded aperture snapshot spectral image.

[0085] Specifically, in the embodiment of the present disclosure, the function expression of the imaging mathematical model of the orthogonal double-dispersion coded aperture snapshot spectral image is g h = Φ h f h and g v = Φ v f v , wherein g h and g v are horizontal and vertical dispersion compressed measurement images of the orthogonal double-dispersion coded aperture snapshot spectral imaging, Φ h and Φ v are measurement matrices of horizontal and vertical dispersion channels, the measurement matrices of horizontal and vertical dispersion channels are block-diagonal sparse matrices, and are generated by the same random coding mask along the horizontal and vertical dispersion offsets, f h and f v are vectorized representations of horizontal and vertical dispersion spectral images; and the function expression of the optimization solving equation for solving the spectral image containing a priori term is:

[0086]

[0087] In the above formula, f h is an estimated value in a vectorized form of the spectral image, t and r are balance factors, and R(f v ) and D(f h ) are priori terms of the spectral image.

[0088] Further specifically, by introducing auxiliary variables z v and z h , the optimization problem can be rewritten as:

[0089]

[0090] The above formula has a Lagrange form:

[0091]

[0092] In the above formula, μ v and μv is a penalty coefficient.

[0093] Step 3:

[0094] A double-dispersion depth unfolding framework (DDDU) is constructed to decompose the optimization solving equation of solving spectral image containing prior terms into a data sub-problem and a prior sub-problem, wherein the data sub-problem is further decomposed into a horizontal dispersion data sub-problem and a vertical dispersion data sub-problem, the horizontal and vertical dispersion data sub-problems have closed-form solutions, and the horizontal and vertical closed-form solutions are independently optimized to retain the complementary information structure of the double-dispersion channel;

[0095] Specifically, the implementation of the double-dispersion depth unfolding framework (DDDU) includes:

[0096] As shown in the following formula: Figure 3 for the optimization solving equation of solving spectral image containing prior terms, the optimization solving equation is decomposed into a data sub-problem and a prior sub-problem, wherein the data sub-problem is further decomposed into a horizontal dispersion data sub-problem and a vertical dispersion data sub-problem, and the function expression of the optimization solving equation of the horizontal dispersion data sub-problem and the vertical dispersion data sub-problem is:

[0097]

[0098] In the above formula, and are the estimated values of the horizontal and vertical dispersion spectral image in vector form, respectively, and k represents the kth stage.

[0099] The closed-form solution optimization method of the horizontal and vertical dispersion data sub-problems includes:

[0100] The block diagonal sparsity of the sum of the horizontal and vertical measurement matrices is used to simplify the matrix inversion operation into a pixel-by-pixel linear operation;

[0101] The diagonal elements of and are pre-calculated to speed up the update of and in the iteration process;

[0102] The expression of the closed-form solution of the horizontal and vertical dispersion data sub-problems is:

[0103]

[0104] In the above formula, diag(·) represents extracting elements from the diagonal line to form a vector.

[0105] The function expression of the optimization solving equation of the prior sub-problem is:

[0106]

[0107] The above equation can be seen as a joint denoising task with noise levels of and noise levels of The optimization solution equation of the prior sub-problem is further expressed as:

[0108]

[0109] In the above equation, Denoiser is a DDIFT denoiser, and is set as

[0110] t k , r k , is the element of the vector inferred by the average pooling layer and the multi-layer perception (MLP) layer once for the initial values of the horizontal and vertical dispersed spectral images and at the k-th stage when k = 1, which is used to estimate the noise level of each stage in the depth unfolding of the double-dispersed channel information;

[0111] The initial values of the horizontal and vertical dispersed spectral images and input to the first stage (k = 1) are initialized to generate the horizontal and vertical dispersed compressed measurement images g h and g v and the corresponding measurement matrices Φ h and Φ v , and and of the subsequent iteration stages i = 1, 2,..., k-1 are generated by shifting z i along the horizontal and vertical directions, respectively, and the horizontal and vertical shifts are used to simulate the horizontal and vertical dispersion of the ODD-CASSI.

[0112] Step 4:

[0113] The prior sub-problem performs joint denoising and feature fusion on the two solutions of the optimized data sub-problem through a double-dispersed information fusion Transformer (DDIFT) to output a denoising result.

[0114] Specifically, the structure of the double-dispersed information fusion Transformer (DDIFT) includes:

[0115] As Figure 3As shown, based on the improved U-Net encoder-decoder structure, multi-level features are extracted through four double convolutional encoding layers, four deconvolutional decoding layers restore the feature map size, and full-scale skip connection is used to fuse low, medium and high level features; wherein a bidirectional fusion branch module (BFCB) is used as the core unit of DDIFT to mine and retain the multi-level shared and independent characteristics of double dispersion information; finally, DDIFT fuses double-channel features through a 1x1 convolutional layer to generate a denoised spectral image.

[0116] Further need to be explained is that DDIFT first reconstructs data from the horizontal dispersion channel using a convolutional layer with a kernel size of 3x3 and its corresponding noise level Initialize the horizontal dispersion feature X h , and at the same time reconstruct data from the vertical dispersion channel and its corresponding noise level Initialize the vertical dispersion spectral feature X v . Then, input X h and X v together into BFCB, and use the characteristics of BFCB to fully extract and fuse double dispersion information. Through the hierarchical fusion mechanism, DDIFT can align the double dispersion information features at different levels, further improving the feature expression ability and network performance.

[0117] In the final stage of feature processing, DDIFT inputs X h and X v extracted by the last BFCB module into a convolutional layer with a kernel size of 3x3, and further enhances and refines them. And add the enhanced features to the initial inputs and respectively to generate new features X' h and X' v . Finally, DDIFT concatenates X' h and X' v and performs feature fusion through a convolutional layer with a kernel size of 1x1 to output the denoised image z k .

[0118] As shown in Figure 3 , the bidirectional fusion branch module (BFCB) is the core unit of DDIFT, and the specific process is as follows:

[0119] Concatenate the horizontal dispersion feature X h ∈R H×W×C and the vertical dispersion feature X v ∈R H×W×C along the channel dimension to X cat ∈R H ×W×2C ;

[0120] Layer Normalization (LN), Multi-Head Self-Attention (MSA) and Feed-Forward Neural Network (FFN) are sequentially performed on the output enhanced feature X enhanced ∈R H×W×2C ;

[0121] X is separated into updated X enhanced and X h ∈R H×W×C and X v ∈R H×W×C by Chunk operation, and the channel independence is preserved.

[0122] The BFCB can not only fully exploit the sharing characteristics of the double dispersion information, but also preserve its independent characteristics, realize efficient and robust feature fusion, and effectively improve the denoising performance of the DDIFT.

[0123] Step 5:

[0124] Based on the double dispersion deep unfolding framework, the data subproblem and the prior subproblem are iteratively solved, the loss of the reconstructed predicted spectral image and the real spectral image is calculated, and the high-fidelity spectral image is finally output until convergence.

[0125] In this embodiment, simulation experiments are carried out based on public data sets CAVE and KAIST, wherein:

[0126] (1) The CAVE consists of 32 spectral scenes, each scene has 31 spectral bands, the resolution is 512x512 pixels, and the database of 32 scenes is divided into five parts of east, skin and hair, oil painting, food and beverage, and real and fake.

[0127] (2) The KAIST consists of 30 spectral scenes, each scene has 31 spectral bands, and the resolution is 2704x3376 pixels.

[0128] (3) In the simulation experiment, the CAVE data set is used as the training set, and 10 scenes are selected from the KAIST data set as the test set, and the selected scenes in the KAIST are cropped to a data size of 256x256x28. For fairness, all algorithms use the same coding mask.

[0129] In addition, in order to verify the spectral imaging performance, in the simulation experiment, three image quality indicators, namely peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and spectral angle similarity (SAM), and one calculation cost indicator, namely gigaflops (GFlops), are used. PSNR measures the spatial fidelity of the reconstructed image, SSIM quantifies the similarity between two images in space, SAM calculates the angle between the reconstructed spectrum and the true spectrum, the larger the PSNR and SSIM values, the better the reconstruction performance, and the smaller the SAM, the more similar the spectral shape. GFlops measures the model complexity and calculation amount, and the smaller the GFlops, the lower the calculation cost.

[0130] The application builds an orthogonal double dispersion coded aperture snapshot spectral imaging system to collect real measurement data and perform real experiments, specifically:

[0131] (1) Two reflective blazed gratings are used as horizontal and vertical dispersion elements, and a real mask is used as a coding mask to build an orthogonal double dispersion coded aperture snapshot spectral imaging system. A resolution target is illuminated by an active light source, and the resolution target pattern is photographed to obtain real measurement data.

[0132] (2) Based on the CAVE and KAIST data sets, the model suitable for the real mask is retrained, and the real measurement data is reconstructed.

[0133] As shown in Figure 4 , Figure 5 The performance of the method of the application on the simulation data set and the real data is better than all the listed methods, which shows that the method of the application has higher performance in spectral imaging. The performance of our DDDU-9stg in the selected 10 KAIST scenes is better than that of all other methods, the average PSNR reaches 42.15dB, the average SSIM reaches 0.986, and the average SAM reaches 2.768. In addition, the method of the application is better than the most advanced deep unfolding algorithm (DERNN-LNLT-9stg*) of the existing CASSI in the lower stage (DDDU-3stg), has lower calculation cost (53.5%) and better image quality (SSIM 0.981, PSNR 40.39), and in addition, the method of the application has less artifact noise and the highest spectral curve consistency on real data.

[0134] As shown in Figure 6As shown, in the experiment, compared with the baseline (DAUHST-3stg) which only uses horizontal dispersive spectral information, the initial fusion of horizontal and vertical dispersive spectral information improved the average PSNR / SSIM by 1.94 dB and 0.016, respectively, demonstrating the theoretical effectiveness of the orthogonal dual-dispersion system design. After fully using the method of this invention, the average PSNR / SSIM improved by 3.18 dB and 0.022, respectively, indicating that effective dual-dispersion fusion can produce higher reconstruction quality.

[0135] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A hyperspectral image reconstruction method based on a quadratically dispersive CASSI system, characterized in that, The method is realized based on a CASSI system with orthogonal dispersion, and the system comprises: An orthogonal dispersion sampling module is configured to generate horizontal and vertical dispersion compressed measurement images, split the light signal modulated by an encoding mask into two paths through a beam splitter, input the two paths into a horizontal dispersion channel and a vertical dispersion channel respectively, and use a horizontal dispersion grating and a vertical dispersion grating to respectively disperse and expand the spectral information in the horizontal and vertical directions; the compressed measurement images of the two channels are synchronously captured by a detector and transmitted to a reconstruction module; A depth expansion reconstruction module is configured to reconstruct a spectral image cube, input the horizontal and vertical compressed measurement images into a double dispersion depth expansion framework, update two intermediate variables through two closed-form solutions of the horizontal and vertical dispersion data sub-problems, input the two intermediate variables into a double dispersion information fusion Transformer, and perform cross-channel feature alignment and noise suppression through a bidirectional fusion branch module; An information fusion module is configured to extract multi-level features of the horizontal and vertical dispersion channels through a multi-scale U-Net architecture of the double dispersion information fusion Transformer; use a bidirectional fusion branch module as a unit of the double dispersion information fusion Transformer to splice, multi-head self-attention fuse and separate the multi-level double-channel features, generate shared features and channel-independent features, and integrate multi-scale features through full-scale skip connection to retain spatial details and spectral consistency; A reconstruction output module is configured to fuse the double-channel features through a 1x1 convolution layer, output a restored spectral image cube in the last stage, and output the reconstruction result through a visualization interface; The method comprises the following steps: A CASSI system with orthogonal dispersion is used to synchronously compress and sample the spectral information of a target through a horizontal dispersion channel and a vertical dispersion channel, and horizontal dispersion compressed measurement images and vertical dispersion compressed measurement images are obtained, wherein the horizontal dispersion channel and the vertical dispersion channel share the same spatial modulation encoding mask, and the dispersion directions are orthogonal; An optimization solving equation for solving the spectral image is established by combining the imaging mathematical model of the input CASSI system with orthogonal dispersion and the imaging mathematical model of the CASSI system with orthogonal dispersion; A double dispersion depth expansion framework is constructed, and the optimization solving equation for solving the spectral image is decomposed into a data sub-problem and a prior sub-problem, wherein the data sub-problem is further decomposed into a horizontal dispersion data sub-problem and a vertical dispersion data sub-problem; the horizontal and vertical dispersion data sub-problems have closed-form solutions, and are independently optimized through the horizontal and vertical closed-form solutions respectively to retain the complementary information structure of the double dispersion channels; the prior sub-problem performs joint denoising and feature fusion on the two solutions of the optimized data sub-problem through a double dispersion information fusion Transformer, and outputs a denoising result; the structure of the double dispersion information fusion Transformer comprises: Based on the improved U-Net encoder-decoder structure, multi-level features are extracted through four double convolution encoding layers, four deconvolution decoding layers restore the feature map size, and full-scale skip connection is used to fuse low, medium and high level features; wherein, a bidirectional fusion branch module is used as the core unit of the dual-dispersion information fusion Transformer, which is used to mine and retain the multi-level shared characteristics and independent characteristics of the dual-dispersion information; finally, the dual-dispersion information fusion Transformer fuses the dual-channel features through a 1×1 convolution layer to generate a denoised spectral image; the bidirectional fusion branch module is used to perform the following operations: horizontal dispersion features vertical dispersion features along the channel dimension ; outputting enhanced features by sequentially performing layer normalization, multi-head self-attention, and feed-forward fully connected layers ; By Chunk operation separated into updated and ; Based on the dual-dispersion deep unfolding framework, the data sub-problem and the prior sub-problem are iteratively solved, the loss of the reconstructed predicted spectral image and the real spectral image is calculated, and the high-fidelity spectral image is finally output until convergence.

2. The method of claim 1, wherein, The hardware structure of the orthogonal dual-dispersion coded aperture snapshot spectral imaging system comprises: an objective lens, a random coding mask, a beam splitter, four relay lenses, a horizontal dispersion grating, a vertical dispersion grating, and two detectors; The coded light signal is divided into two paths by the beam splitter, and the horizontal direction dispersion unfolded compressed measurement image and the vertical direction dispersion unfolded compressed measurement image are generated on the two detectors respectively.

3. The method of claim 1, wherein, The function expression of the imaging mathematical model of the orthogonal dual-dispersion coded aperture snapshot spectral image is: wherein, and are the horizontal and vertical dispersed compressed measurement images of the orthogonal dual-dispersive coded-aperture snapshot spectral imaging, respectively; and are the measurement matrices of the horizontal and vertical dispersed channels, respectively, which are both block-diagonal sparse matrices generated by the same random coding mask along the horizontal and vertical dispersed shifts, respectively; and are the vectorized representations of the horizontal and vertical dispersed spectral images, respectively; The function expression of the optimization solving equation for solving the spectral image containing the prior term is: wherein is an estimate of the spectral image in vectorized form; and is a balancing factor; and is a prior term for the spectral image.

4. The method of claim 3, wherein, The function expression of the optimization solving equation of the horizontal and vertical dispersion data sub-problem is: wherein, and are the estimated values of the horizontal and vertical dispersed spectral image vectorized forms, respectively; and are the penalty coefficients; and are the introduced auxiliary variables; The optimization method of the closed-form solution of the horizontal and vertical dispersion data sub-problem comprises: The block diagonal sparsity of the sum of the horizontal and vertical measurement matrices is used to simplify the matrix inversion operation into a pixel-by-pixel linear operation; pre-computations and diagonal elements of the matrix and updates in the iteration process; The expression of the closed-form solution of the optimization solving equation of the horizontal and vertical dispersion data sub-problem is: Wherein, diag(·) represents extracting elements from the diagonal line to form a vector; " / " represents a pixel-by-pixel division operation; The function expression of the optimization solving equation of the prior sub-problem is: The above equation is considered as a joint denoising task with and having a noise level of and having a noise level of and having a noise level of the optimization solution equation of the priori sub-problem is further expressed as: in, Denoiser Set the DDIFT noise denoiser. , ; When k=1, the initial values ​​are derived from the horizontal and vertical dispersive spectral images. and The elements of the vector inferred in one step through the average pooling layer and the multilayer perceptron layer at the kth stage are used to estimate the noise level at each stage in the deep unfolding of the dual-dispersion channel information. Input to the first stage are the initial values of the horizontally and vertically dispersed spectral images and are generated from the horizontally and vertically dispersed compressed measurement images and and the corresponding measurement matrices and are initialized, and the subsequent iteration stages i = 1, 2,..., k - 1 of and are generated from are generated by horizontal and vertical shifts that are used to simulate the horizontal and vertical dispersion of the ODD-CASSI system.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the hyperspectral image reconstruction method of the CASSI system based on the orthogonal dual-dispersion as claimed in any one of claims 1-4.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program is executed by the processor to realize the hyperspectral image reconstruction method of the CASSI system based on the orthogonal dual-dispersion as claimed in any one of claims 1-4.

Images