Compressed Spectral Imaging Reconstruction Method and Apparatus Based on Degradation Estimation Recurrent Neural Network

Abstract

This invention provides a compressed spectral imaging reconstruction method and apparatus based on a degradation estimation recurrent neural network. By sharing parameters across different stages, the deep unfolded network is converted into a recurrent neural network. This parameter sharing significantly reduces the number of network parameters and memory usage, and allows the deep unfolded network to learn representations for reconstruction from inputs at different stages. This reduces the parameter sparsity of the deep unfolded network, fully exploits its reconstruction potential, and enhances its reconstruction performance, thereby improving the quality of the reconstructed image. Furthermore, this invention introduces a degradation estimation network to estimate the degradation matrix and noise level, improving the accuracy of solutions to the data and prior subproblems. Simultaneously, the use of local and non-local transformation networks to solve the prior subproblems more fully exploits the local and non-local priors of the hyperspectral image, further improving the quality of the reconstructed image.

Description

Technical Field

[0001] This invention belongs to the field of image reconstruction technology, specifically relating to a compressed spectral imaging reconstruction method and apparatus based on a degradation estimation recurrent neural network. Background Technology

[0002] Hyperspectral imaging (HSI) contains detailed representations of scenes and has broad application prospects in fields such as face recognition and object tracking. However, imaging systems composed of a single 1D or 2D sensor suffer from long scanning times, which is not conducive to real-time imaging of dynamic scenes. In recent years, coded aperture snapshot spectral imaging systems (CASSI) based on compressed sensing theory have attracted widespread attention. However, a key challenge currently facing CASSI systems is how to accurately reconstruct three-dimensional spectral data from two-dimensional compressed spectral images.

[0003] Existing hyperspectral image reconstruction methods include model-based methods, end-to-end neural network-based methods, and deep unfolded network-based methods. However, model-based methods use manually customized priors, requiring manual parameter tuning, resulting in slow reconstruction speed and poor reconstruction quality. End-to-end neural network-based methods leverage the powerful representational capabilities of neural networks to achieve good performance; however, because neural networks are black boxes, they lack interpretability. Furthermore, designing a high-performance neural network architecture is not easy. Deep unfolded networks combine the advantages of the former two by deeply unfolding model-based methods into a set of stacked neural networks, achieving good performance. However, deep unfolded networks require stacking multiple neural networks, leading to a large number of parameters and significant memory consumption. Moreover, previous deep unfolded networks did not simultaneously estimate the degradation matrix used to solve the data subproblem and the noise level used to solve the prior subproblem within a single framework, resulting in suboptimal reconstruction results.

[0004] In summary, existing hyperspectral image reconstruction methods result in poor quality of the reconstructed spectral images. Summary of the Invention

[0005] To address the aforementioned problems in the prior art, this invention provides a compressed spectral imaging reconstruction method and apparatus based on a degradation estimation recurrent neural network. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a compressed spectral imaging reconstruction method based on a degradation estimation recurrent neural network, comprising:

[0007] S100 acquires compressed spectral images and a perception matrix through the CASSI system, and extracts feature maps from the compressed spectral images;

[0008] S200, construct an optimization problem for reconstructing the original hyperspectral image based on the compressed spectral image and the perception matrix, and decompose the solution process of the optimization problem into alternately solving a data subproblem and a priori subproblem;

[0009] S300, Obtain a pre-constructed degradation estimation recurrent neural network, which includes a degradation estimation network and local and non-local transformation networks; Use the feature map and the perceptual matrix as inputs to the degradation estimation network, and use the degradation estimation network to estimate the residual and noise level between the degradation matrix and the perceptual matrix; Input the residual, the perceptual matrix, the compressed spectral image, and the feature map into the data subproblem to obtain the solved hyperspectral image; Wherein, the residual controls the accuracy of solving the data subproblem, and the noise level controls the accuracy of solving the prior subproblem;

[0010] S400, the solved hyperspectral image and the noise level are used as inputs to a local and nonlocal transformation network, and the prior subproblem is solved using the local and nonlocal transformation network to output the estimated hyperspectral image;

[0011] S500: Use the estimated hyperspectral image as the input for the next iteration, repeat S300-S400 until the number of iterations is reached, and obtain the final original hyperspectral image.

[0012] In a second aspect, the present invention provides a compressed spectral imaging reconstruction device based on a degradation estimation recurrent neural network, comprising:

[0013] The feature extraction module is configured to acquire compressed spectral images and a perception matrix through the CASSI system, and extract feature maps from the compressed spectral images.

[0014] The problem decomposition module is configured to construct an optimization problem for reconstructing the original hyperspectral image based on the compressed spectral image and the perception matrix, and to decompose the solution process of the optimization problem into alternately solving a data subproblem and a priori subproblem.

[0015] The parameter estimation module is configured to acquire a pre-constructed degradation estimation recurrent neural network, which includes a degradation estimation network and local and non-local transformation networks; take the feature map and the perceptual matrix as inputs to the degradation estimation network, and use the degradation estimation network to estimate the residual and noise level between the degradation matrix and the perceptual matrix; input the residual, the perceptual matrix, the compressed spectral image, and the feature map into the data subproblem to obtain the solved hyperspectral image; wherein, the residual controls the accuracy of solving the data subproblem, and the noise level controls the accuracy of solving the prior subproblem;

[0016] The image estimation module is configured to take the solved hyperspectral image and the noise level as input to a local and nonlocal transformation network, and use the local and nonlocal transformation network to solve the prior subproblem, and output the estimated hyperspectral image.

[0017] The image reconstruction module is configured to use the estimated hyperspectral image as input for the next iteration, repeating the processing steps of the parameter estimation module and the image estimation module until the required number of iterations is reached, thus obtaining the final original hyperspectral image. Beneficial effects:

[0018] 1. This invention provides a compressed spectral imaging reconstruction method based on a degradation estimation recurrent neural network. By sharing the parameters of the deep unfolded network at different stages, the deep unfolded network is transformed into a recurrent neural network. This not only significantly reduces the number of network parameters and memory usage, but also enables the neural network to learn representations for reconstruction from inputs at different stages, thereby improving the quality of the reconstructed image.

[0019] 2. This invention proposes using a degradation estimation network to estimate the difference between the perceptual matrix and the degradation matrix, addressing the problem that the perceptual matrix cannot fully reflect the degradation process in real-world shooting scenarios, thereby improving the quality of reconstructed images. Previous methods for obtaining the degradation matrix using deep unfolded networks can be mainly divided into two types: one directly uses the perceptual matrix as the degradation matrix, and the other learns the degradation matrix through a neural network. The first method does not consider the difference between the perceptual matrix and the degradation process caused by device errors. In the second method, directly modeling the degradation process presents certain challenges. The degradation estimation network proposed in this invention uses the perceptual matrix as a reference to estimate the residual between the perceptual matrix and the degradation matrix from the CASSI system, combining the advantages of both methods while overcoming their respective shortcomings.

[0020] 3. This invention proposes to use a degradation estimation network to estimate the noise level in the prior subproblem. Explicitly estimating the noise level improves the denoising capability of the denoiser, thereby reconstructing the hyperspectral image more clearly.

[0021] 4. This invention proposes a local and non-local transform network that mines local and non-local prior information of hyperspectral images in a data-driven manner. This solves the problem that previous deep unfolding networks could not simultaneously mine local and non-local prior information of hyperspectral images, thereby improving the quality of reconstructed images. Previous deep unfolding networks used methods based on convolutional neural networks and Transformer networks, respectively, to mine prior information in a data-driven manner. However, convolutional neural networks cannot model the non-local priors of hyperspectral images, and Transformer networks have limitations in mining local priors, resulting in insufficient texture and detail modeling.

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] Figure 1a This is a schematic diagram of the CASSI system provided by the present invention;

[0024] Figure 1b This is a schematic diagram of the recurrent neural network provided by the present invention;

[0025] Figure 2 This is a schematic flowchart of a compressed spectral imaging reconstruction method based on a degradation estimation recurrent neural network provided by the present invention;

[0026] Figure 3 This is a schematic diagram of the overall process of the degradation estimation recurrent neural network provided by the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of the degenerate learning convolutional block provided by the present invention;

[0028] Figure 5 This is a schematic diagram of local and non-local blocks provided by the present invention;

[0029] Figure 6 This is a schematic diagram of the local / non-local multi-head self-attention module provided by the present invention;

[0030] Figure 7 This is a schematic diagram of the gated convolutional feedforward neural network provided by the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0032] Referring to Figure 1 to Figure 7 This invention provides a compressed spectral imaging reconstruction method based on a degradation estimation recurrent neural network, comprising:

[0033] S100 acquires compressed spectral images and a perception matrix through the CASSI system, and extracts feature maps from the compressed spectral images;

[0034] It is worth noting that hyperspectral image reconstruction methods primarily rely on spectrometers to acquire data. Spectrometers collect hyperspectral scene data through line scanning or area scanning. However, acquiring hyperspectral data by spectrometers has two main drawbacks: pushbroom imaging results in slow imaging speed, with a long time required to scan a single spectral scene; hyperspectral images contain dozens or hundreds of channels, requiring significant storage space—one hyperspectral image is equivalent to storing dozens of ordinary color images, typically occupying tens or hundreds of megabytes of storage. This large storage space consumption makes disk-to-memory throughput a bottleneck during model training and inference. The slow imaging speed, large memory consumption, and throughput bottleneck of hyperspectral images limit their application in real-time computer vision tasks. Compressed spectral imaging technology, on the other hand, compresses a snapshot of a three-dimensional hyperspectral scene along the spectral dimension into a two-dimensional observation image, thereby achieving real-time imaging. Among existing compressed spectral imaging systems, the Coordinated Aperture Snapshot Spectroscopic Imaging System (CASSI) stands out and has become a promising research direction. However, a core issue of CASSI systems is how to faithfully reconstruct the original three-dimensional hyperspectral image from the two-dimensional observation image. Therefore, this invention relates to the problem of three-dimensional hyperspectral image reconstruction in the CASSI system.

[0035] The CASSI system encodes a spectral scene using an encoding template and one or more dispersive elements, and captures a two-dimensional projection of the scene's three-dimensional spectral image using a grayscale camera, as illustrated in the diagram. Figure 1a As shown.

[0036] The CASSI system first uses encoding templates. Modulated hyperspectral image signals Where H and W represent the height and width of the hyperspectral image, respectively, and N... λ Indicates the number of wavelengths. (The first...) The image modulated at each wavelength can be represented as:

[0037]

[0038] Here, ⊙ represents element-wise product. The modulated image is then shifted through a dispersive prism; this process can be represented as:

[0039]

[0040] in, Representative wavelength is The displacement distance of the image is calculated. Finally, the displacement images are summed to obtain a two-dimensional observation image, which is represented as follows:

[0041]

[0042] This represents a two-dimensional compressed spectral image. Formulas (1)-(3) can be rewritten in the following vector form:

[0043] y=Φx+n (4)

[0044] in, It is the perception matrix, which is usually regarded as the coding template after dispersion shift, and n represents additive Gaussian noise.

[0045] A key problem in CASSI systems is recovering a high-quality image x from a degraded y. Since image restoration is an ill-conditioned inverse problem, the prior term, often referred to as the regularization term, is used to constrain the solution space. From a Bayesian perspective, this can be solved by solving a maximum a posteriori estimation problem.

[0046]

[0047] Where log p(y|x) represents the log-likelihood term of the observed image y, and log p(x) represents the prior term of the clean image x, independent of the compressed spectral image y. Equation (5) can define the following energy function:

[0048]

[0049] in, R(x) is the data fidelity term, R(x) is the prior term, and λ is the hyperparameter balancing the importance of the two terms. The HQS algorithm first introduces the auxiliary variable z, and equation (6) can be reformulated as:

[0050]

[0051] This is a constrained optimization problem that can be solved by minimizing the following problem:

[0052]

[0053] Here, μ is the penalty parameter. This problem can be solved by alternately solving a data subproblem and a priori subproblem:

[0054]

[0055]

[0056] The data subproblem (9a) typically has a closed-form solution that depends on the perception matrix Φ:

[0057] x k =z k-1 +Φ T [(y-Φz k-1 ) / (μ+ΦΦT (10)

[0058] From a Bayesian perspective, the prior subproblem (9b) corresponds to a noise level of Noisy image x k Noise reduction:

[0059]

[0060] HQS algorithm iterative update x k and z k The process continues until convergence, thus obtaining the reconstructed hyperspectral image.

[0061] S200: Based on the compressed spectral image and the perception matrix, construct an optimization problem to reconstruct the original hyperspectral image, and decompose the solution process of the optimization problem into alternately solving a data subproblem and a priori subproblem;

[0062] This invention first converts the deep unfolded network into a recurrent neural network, such as... Figure 1b As shown, the network uses the output of the current loop as the input of the next loop, and solves a data subproblem and a prior subproblem alternately in each loop.

[0063] This recurrent neural network performs a deep expansion of the semi-quadratic splitting (HQS) algorithm within the framework of maximum a posteriori estimation. The specific steps are as follows:

[0064] The problem in equation (4) is transformed into a Bayesian problem, namely equation (5). Equation (5) has the energy function in equation (6), and solving equation (6) is an ill-conditioned inverse problem. HQS introduces an auxiliary variable z, and equation (6) can be reformulated as equation (7). Equation (7) is a constrained optimization problem, which can be solved by minimizing equation (8). Equation (8) can be solved by alternately solving a data subproblem (9a) and a prior subproblem (9b). The data subproblem (9a) has the following closed-form solution:

[0065]

[0066] Where I is an identity matrix. Given a block diagonal sensing matrix... This invention can calculate diagonal matrices.

[0067]

[0068] Where, φ i Obtained through the encoding template M. Using the matrix inversion formula, the inverse matrix of equation (12) can be rewritten as:

[0069]

[0070] According to equation (13):

[0071]

[0072] Inserting equations (13), (14), and (15) into equation (12) and simplifying the formula, we have:

[0073]

[0074] Here, Diag() represents taking the diagonal elements of the matrix to form a vector. In this way, the data subproblem can be solved in a closed form.

[0075] The solution of the data subproblem (16) is highly dependent on the degenerate matrix. Degeneracy matrix It affects the accuracy of solving the data subproblems each time.

[0076] The perception matrix Φ is typically obtained by capturing monochromatic light of different wavelengths using a camera. However, the perception matrix obtained in this way is affected by factors such as exposure time, light intensity, and equipment errors, leading to a degraded matrix when shooting actual scenes. There is a difference between the perception matrix Φ and the actual degradation matrix. Therefore, this invention uses a neural network to estimate the degradation matrix from the CASSI system. The difference between the data subproblem and the perception matrix Φ makes the solution more accurate each time.

[0077] From a Bayesian perspective, the prior subproblem (9b) corresponds to a noise level of Noisy image x k The noise level denoising is then performed, as shown in equation (11). Solving the noise level denoising problem is crucial, yet it is unknown in the reconstruction task. To facilitate solving equation (9b), this invention allows... is a learnable parameter related to the iteration, which is the reciprocal of the square of the noise level.

[0078] S300: Obtain a pre-constructed degradation estimation recurrent neural network, which includes a degradation estimation network and local and non-local transformation networks; use the feature map and the perceptual matrix as inputs to the degradation estimation network, and use the degradation estimation network to estimate the residual and noise level between the degradation matrix and the perceptual matrix; input the residual, the perceptual matrix, the compressed spectral image, and the feature map into the data subproblem to obtain the solved hyperspectral image; wherein, the residual controls the accuracy of solving the data subproblem, and the noise level controls the accuracy of solving the prior subproblem;

[0079] To solve the data subproblem and the prior subproblem, refer to... Figure 3This invention proposes a degradation estimation network. The degradation estimation network comprises multiple sequentially connected degradation learning convolutional blocks, a global average pooling layer, and a multilayer perceptron; see [link to related documentation]. Figure 4 Each degenerate learning convolutional block comprises a first convolutional layer, a first depthwise convolutional layer, and a second convolutional layer connected in sequence. Within the degenerate learning convolutional block, the feature map and the receptive matrix are convolved through its three convolutional layers to obtain the residual between the receptive matrix and the degenerate matrix; a global average pooling layer performs average pooling on the residual; and multiple receptive layers perform a nonlinear transformation on the average pooled residual to obtain the noise level. Figures 3-7 The matrix splicing operator is used In other words, the matrix addition operator uses Matrix multiplication operators The pixel-by-pixel multiplication operator for matrices is represented by ⊙.

[0080] The degradation estimation network uses multiple degradation learning convolutional blocks (DLCBs) to estimate the perceptual matrix Φ and the degradation matrix from the input of the current loop and the CASSI system. The residuals between Subsequently, the residuals The penalty parameter μ used to control convergence is estimated using a global average pooling (GAP) and a multilayer perceptron (MLP). k and the noise level η used to solve the prior subproblem. k Therefore, solving the data subproblem becomes solving equation (17), and solving the a priori subproblem becomes solving equation (18).

[0081]

[0082] z k =Denoiser(x k η k (18)

[0083] To solve the prior subproblem, refer to Figure 3This invention proposes a Local and Nonlocal Transform Network (LNLT). The LNLT is a U-shaped network comprising multiple Local and Nonlocal Blocks (LNLBs), two third convolutional layers, two downsampling layers, two upsampling layers, two matrix concatenation operators, two fourth convolutional layers, and a first matrix addition operator. The first third convolutional layer, the first local and nonlocal block, the first downsampling layer, the second local and nonlocal block, and the second downsampling layer are sequentially connected. The output of the second downsampling layer is connected to the input of the first upsampling layer. The outputs of the first upsampling layer and the second local and nonlocal block are both connected to the input of the first matrix concatenation operator. The output of the first matrix concatenation operator is connected to the input of the first fourth convolutional layer. The output of the first fourth convolutional layer is connected to the input of the fifth local and non-local block. The output of the fifth local and non-local block is connected to the input of the second upsampling layer. The outputs of the second upsampling layer and the sixth local and non-local block are both connected to the input of the second matrix concatenation operator. The output of the second matrix concatenation operator is connected to the input of the second fourth convolutional layer. The output of the second fourth convolutional layer is connected to the input of the sixth local and non-local block. The output of the sixth local and non-local block is connected to the input of the second fourth convolutional layer. The hyperspectral image of the second fourth convolutional layer and the input of the first fourth convolutional layer are both input to the first matrix addition operator. The first matrix addition operator outputs the hyperspectral image for the next cycle. Each third and fourth convolutional layer in the local and non-local transform network is used to convolve the input features. Each local and non-local block is used to perform a non-linear transformation on the convolutional features. Each downsampling layer is used to downsample the input features. Each matrix concatenation operator is used to concatenate the input features. Each upsampling layer is used to upsample the input features.

[0084] First, LNLT uses a Conv3x3 operation to concatenate the x values. k and η after broadcast k Mapped to a shallow feature Then, X0 obtains deep features through multiple local and non-local blocks (LNLBs) at different levels and up and down sampling modules between different levels. Finally, regarding X d Generate a residual image using a Conv3x3 operation. By x k The denoised image z is obtained by summing the sum with R. k .

[0085] refer to Figure 5The Local and Non-Local Block (LNLB) consists of three normalization layers (LN), three second matrix addition operators, and a local multi-head self-attention module (Local). The system consists of a Non-Local Multi-Head Self-Attention (MSA) module and a Gated Convolutional Feedforward Neural Network (GDFN). The output of the first normalized layer is connected to the input of the local multi-head self-attention module. The output of the local multi-head self-attention module and the input of the first normalized layer are both connected to the input of the first second matrix addition operator. The output of the first second matrix addition operator is connected to the input of the second normalized layer. The output of the second normalized layer is connected to the input of the non-local multi-head self-attention module. The output of the non-local multi-head self-attention module and the output of the first second matrix addition operator are both connected to the input of the second second matrix addition operator. The output of the second second matrix addition operator is connected to the input of the third normalized layer. The output of the third normalized layer is connected to the input of the Gated Convolutional Feedforward Neural Network. The output of the Gated Convolutional Feedforward Neural Network and the output of the second second matrix addition operator are both connected to the input of the third second matrix addition operator. The output of the third second matrix addition operator serves as the output of both the local and non-local blocks.

[0086] See Figure 6 The local multi-head self-attention module and the non-local multi-head self-attention module have the same structure, both including: 4 fifth convolutional layers, 3 second depth convolutional layers, 3 matrix shape reshaping operators, and 2 matrix multiplication operators; the outputs of the three fifth convolutional layers are connected to a second depth convolutional layer, the output of each second depth convolutional layer is connected to a matrix shape reshaping operator, the first and second matrix shape reshaping operators are connected to a matrix multiplication operator, the output of the third matrix shape reshaping operator is connected to the input of the second matrix multiplication operator, the output of the second matrix multiplication operator is connected to the input of the first matrix multiplication operator, the output of the first matrix multiplication operator is connected to the input of the last fifth convolutional layer, and the output of the last fifth convolutional layer serves as the output of either the local multi-head self-attention module or the non-local multi-head self-attention module.

[0087] Local multi-head self-attention (Local MSA) and non-local multi-head self-attention (Non-Local MSA) use the same method to embed the input as queries, keys, and values; the difference lies in how they compute the self-attention map. Figure 6 The calculation process for Local MSA and Non-Local MSA is demonstrated. Let the inputs for Local MSA and Non-Local MSA be represented as... Both Local MSA and Non-Local MSA use a Conv1x1 and a DConv3x3 to represent X in Embedded as a query key Sum Where DConv is a depthwise convolution:

[0088]

[0089] in, Indicates Conv1x1, This represents DConv3x3.

[0090] Local Multi-Head Self-Attention (Local MSA) divides the input into non-overlapping windows of size M x M, treating each pixel within the window as an element and computing self-attention within the window. In Local MSA, the query Q, key K, and value V are reshaped as follows: Then, Q L X L V L Split into h heads along the last dimension: The dimensions of each head are ask s and keys The inner product produces a shape of attention mapping It represents the degree of correlation between different pixels within a window. In summary, the calculation process for Local MSA is as follows:

[0091] X 0ut =W p Concat(head1, ..., head) h )+X in (20)

[0092]

[0093] in, It is a learnable position embedding that represents the positional relationship between different pixels within a window.

[0094] Non-Local Multi-Head Self-Attention (Non-Local MSA) divides the input into NxN non-overlapping windows, treating all pixels within each window as a single element and computing self-attention across windows. In Non-Local MSA, the query Q, key K, and value V are reshaped as follows: Then, Q NL K NL V NL Split into h heads along the last dimension: The dimensions of each head are ask s and keys The inner product produces a shape of attention mapping It represents the magnitude of the correlation between different windows. In summary, the calculation process for Non-Local MSA is as follows:

[0095] X out =W p Concat(head1, ..., head) h )+X in (twenty two)

[0096]

[0097] in, It is a learnable position embedding that represents the positional relationship between different windows.

[0098] S400 takes the solved hyperspectral image and noise level as input to a local and nonlocal transformation network, and uses the local and nonlocal transformation network to solve the prior subproblem, outputting the estimated hyperspectral image.

[0099] refer to Figure 7 The gated convolutional feedforward neural network includes: three sixth convolutional layers, two third-depth convolutional layers, and one matrix multiplication operator per pixel. The two sixth convolutional layers have the same output, and the output of each sixth convolutional layer is connected to a third-depth convolutional layer. The output of each third-depth convolutional layer is connected to the input of the matrix multiplication operator per pixel. The output of the matrix multiplication operator per pixel is connected to the input of the last sixth convolutional layer. The output of the sixth convolutional layer serves as the output of the gated convolutional feedforward neural network.

[0100] LNLB finally requires a gated convolutional feedforward network (GDFN), the structure of which is as follows: Figure 7 As shown, the gating mechanism is formulated as an element-wise product of parallel branches of a linear transformation layer, where one branch is non-linearly activated by the GELU activation function. GDFN also introduces depthwise convolution to encode information from adjacent spatial pixel locations to learn the local image structure for efficient image reconstruction.

[0101] Finally, the output denoised image z k This serves as the input for the next iteration of the data subproblem, thus completing the iterative alternation of solving the data subproblem and the prior subproblem.

[0102] S500: The estimated hyperspectral image is used as the input for the next iteration. S300-S400 are repeated until the number of iterations is reached, and the final raw hyperspectral image is obtained.

[0103] This step involves inputting the estimated hyperspectral image into the degenerate estimation recurrent neural network in the current iteration and repeating S300-S400 to estimate the hyperspectral image for the next iteration. It then determines whether the number of iterations has reached the preset number of iterations. If it has, the last estimated hyperspectral image is determined as the final original hyperspectral image. The original hyperspectral image is a reconstructed image of the compressed spectral image.

[0104] This invention provides a compressed spectral imaging reconstruction device based on a degradation estimation recurrent neural network, comprising:

[0105] The feature extraction module is configured to acquire compressed spectral images and a perception matrix through the CASSI system, and extract feature maps from the compressed spectral images.

[0106] The problem decomposition module is configured to construct an optimization problem for reconstructing the original hyperspectral image based on the compressed spectral image and the perception matrix, and decompose the solution process of the optimization problem into alternately solving a data subproblem and a priori subproblem.

[0107] The parameter estimation module is configured to acquire a pre-constructed degradation estimation recurrent neural network, which includes a degradation estimation network and local and non-local transformation networks. It takes the feature map and the perceptual matrix as input to the degradation estimation network and uses the degradation estimation network to estimate the residual between the degradation matrix and the perceptual matrix, as well as the noise level. The residual, the perceptual matrix, the compressed spectral image, and the feature map are then input into the data subproblem to obtain the solved hyperspectral image. The residual controls the accuracy of solving the data subproblem, and the noise level controls the accuracy of solving the prior subproblem.

[0108] The image estimation module is configured to take the solved hyperspectral image and noise level as input to a local and nonlocal transformation network, and use the local and nonlocal transformation network to solve the prior subproblem, and output the estimated hyperspectral image.

[0109] The image reconstruction module is configured to use the estimated hyperspectral image as input for the next iteration, repeating the processing of the parameter estimation module and the image estimation module until the number of iterations is reached, and finally obtaining the original hyperspectral image.

[0110] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0111] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0112] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network, characterized in that, include: S100 acquires compressed spectral images and a perception matrix through the CASSI system, and extracts feature maps from the compressed spectral images; S200, construct an optimization problem for reconstructing the original hyperspectral image based on the compressed spectral image and the perception matrix, and decompose the solution process of the optimization problem into alternately solving a data subproblem and a priori subproblem; S300, Obtain a pre-constructed degradation estimation recurrent neural network, which includes a degradation estimation network and local and non-local transformation networks; Use the feature map and the perceptual matrix as inputs to the degradation estimation network, and use the degradation estimation network to estimate the residual and noise level between the degradation matrix and the perceptual matrix; Input the residual, the perceptual matrix, the compressed spectral image, and the feature map into the data subproblem to obtain the solved hyperspectral image; Wherein, the residual controls the accuracy of solving the data subproblem, and the noise level controls the accuracy of solving the prior subproblem; S400, the solved hyperspectral image and the noise level are used as inputs to a local and nonlocal transformation network, and the prior subproblem is solved using the local and nonlocal transformation network to output the estimated hyperspectral image; S500: Use the estimated hyperspectral image as the input for the next iteration, repeat S300-S400 until the number of iterations is reached, and obtain the final original hyperspectral image.

2. The compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network according to claim 1, characterized in that, The degradation estimation network includes multiple sequentially connected degradation learning convolutional blocks, global average pooling layers, and multilayer perceptrons; each degradation learning convolutional block includes a first convolutional layer, a first depthwise convolutional layer, and a second convolutional layer connected in sequence.

3. The compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network according to claim 2, characterized in that, The degradation learning convolutional block uses its own three convolutional layers to convolve the feature map and the perceptual matrix to obtain the residual between the perceptual matrix and the degradation matrix; the global average pooling layer performs average pooling on the residual, and the multi-layer perceptual layer performs nonlinear transformation on the average pooled residual to obtain the noise level.

4. The compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network according to claim 1, characterized in that, The local and non-local transformation network is a U-shaped network; the local and non-local transformation network includes multiple local and non-local blocks, two third convolutional layers, two downsampling layers, two upsampling layers, two matrix concatenation operators, two fourth convolutional layers, and one first matrix addition operator; The system consists of three convolutional layers: the first third convolutional layer, the first local and non-local block, the first downsampling layer, the second local and non-local block, and the second downsampling layer. The output of the second downsampling layer is connected to the input of the first upsampling layer. The outputs of the first upsampling layer and the second local and non-local block are both connected to the input of the first matrix concatenation operator. The output of the first matrix concatenation operator is connected to the input of the first fourth convolutional layer. The output of the first fourth convolutional layer is connected to the input of the fifth local and non-local block. The output of the fifth local and non-local block is connected to the input of the second upsampling layer. The outputs of the second upsampling layer and the sixth local and non-local block are both connected to the input of the second matrix concatenation operator. The output of the second matrix concatenation operator is connected to the input of the second fourth convolutional layer. The output of the second fourth convolutional layer is connected to the input of the sixth local and non-local block. The output of the sixth local and non-local block is connected to the input of the second fourth convolutional layer. The hyperspectral image from the output of the second fourth convolutional layer and the input of the first fourth convolutional layer are both input to the first matrix addition operator. The first matrix addition operator outputs the hyperspectral image for the next cycle.

5. The compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network according to claim 4, characterized in that, Each third and fourth convolutional layer in the local and non-local transform network is used to convolve the input features, and each local and non-local block is used to perform non-linear transformations on the convolved features; each downsampling layer is used to downsample the input features; each matrix concatenation operator is used to concatenate the input features, and each upsampling layer is used to upsample the input features.

6. The compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network according to claim 4, characterized in that, The local and non-local blocks include three normalization layers, three second matrix addition operators, a local multi-head self-attention module, a non-local multi-head self-attention module, and a gated convolutional feedforward neural network. In this configuration, the output of the first normalized layer is connected to the input of the local multi-head self-attention module. The output of the local multi-head self-attention module and the input of the first normalized layer are both connected to the input of the first second matrix addition operator. The output of the first second matrix addition operator is connected to the input of the second normalized layer. The output of the second normalized layer is connected to the input of the non-local multi-head self-attention module. The output of the non-local multi-head self-attention module and the output of the first second matrix addition operator are both connected to the input of the second second matrix addition operator. The output of the second second matrix addition operator is connected to the input of the third normalized layer. The output of the third normalized layer is connected to the input of the gated convolutional feedforward neural network. The output of the gated convolutional feedforward neural network and the output of the second second matrix addition operator are both connected to the input of the third second matrix addition operator. The output of the third second matrix addition operator serves as the output of both the local and non-local blocks.

7. The compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network according to claim 6, characterized in that, The local multi-head self-attention module and the non-local multi-head self-attention module have the same structure, both including: 4 fifth convolutional layers, 3 second depth convolutional layers, 3 matrix shape reshaping operators and 2 matrix multiplication operators; The outputs of the three fifth convolutional layers are connected to a second depth convolutional layer, the output of each second depth convolutional layer is connected to a matrix shape reshaping operator, the first and second matrix shape reshaping operators are connected to a matrix multiplication operator, the output of the third matrix shape reshaping operator is connected to the input of the second matrix multiplication operator, the output of the second matrix multiplication operator is connected to the input of the first matrix multiplication operator, the output of the first matrix multiplication operator is connected to the input of the last fifth convolutional layer, and the output of the last fifth convolutional layer serves as the output of either a local multi-head self-attention module or a non-local multi-head self-attention module.

8. The compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network according to claim 6, characterized in that, The gated convolutional feedforward neural network includes: three sixth convolutional layers, two third depth convolutional layers, and one matrix pixel-wise multiplication operator; In this system, both sixth convolutional layers have the same output. The output of each sixth convolutional layer is connected to a third deep convolutional layer. The output of each third deep convolutional layer is connected to the input of a matrix pixel-by-pixel multiplication operator. The output of the matrix pixel-by-pixel multiplication operator is connected to the input of the last sixth convolutional layer. The output of the sixth convolutional layer serves as the output of the gated convolutional feedforward neural network.

9. The compressed spectral imaging reconstruction method based on a degenerate estimation recurrent neural network according to claim 1, characterized in that, The S500 includes: The estimated hyperspectral image is input into the degradation estimation recurrent neural network in the current iteration, and S300-S400 is repeated to predict the hyperspectral image in the next iteration. Determine whether the number of iterations has reached the preset number of iterations. If it has, then determine the last estimated hyperspectral image as the final original hyperspectral image; the original hyperspectral image is the reconstructed image of the compressed spectral image.

10. A compressed spectral imaging reconstruction device based on a degenerate estimation recurrent neural network, characterized in that, include: The feature extraction module is configured to acquire compressed spectral images and a perception matrix through the CASSI system, and extract feature maps from the compressed spectral images. The problem decomposition module is configured to construct an optimization problem for reconstructing the original hyperspectral image based on the compressed spectral image and the perception matrix, and to decompose the solution process of the optimization problem into alternately solving a data subproblem and a priori subproblem. The parameter estimation module is configured to acquire a pre-constructed degradation estimation recurrent neural network, which includes a degradation estimation network and local and non-local transformation networks; take the feature map and the perceptual matrix as inputs to the degradation estimation network, and use the degradation estimation network to estimate the residual and noise level between the degradation matrix and the perceptual matrix; input the residual, the perceptual matrix, the compressed spectral image, and the feature map into the data subproblem to obtain the solved hyperspectral image; wherein, the residual controls the accuracy of solving the data subproblem, and the noise level controls the accuracy of solving the prior subproblem; The image estimation module is configured to take the solved hyperspectral image and the noise level as input to a local and nonlocal transformation network, and use the local and nonlocal transformation network to solve the prior subproblem, and output the estimated hyperspectral image. The image reconstruction module is configured to use the estimated hyperspectral image as input for the next iteration, repeating the processing of the parameter estimation module and the image estimation module until the number of iterations is reached, to obtain the final original hyperspectral image.

Images