An end-to-end spectral reconstruction method and system

Abstract

This invention discloses an end-to-end spectral reconstruction method, comprising the following steps: S1, reconstructing a hyperspectral image; S2, using the obtained hyperspectral image as input to an optimized neural network, training the neural network parameters according to the end-to-end spectral reconstruction model, and calculating the loss function; S3, backpropagating errors to the coding template to be designed, changing the design parameters of the coding template, and continuing to train the model until a high-quality hyperspectral image is reconstructed. By employing the above technical solution, the design of the optical system is combined with the reconstruction method. Based on the coded aperture snapshot system, an adaptive coding template is added according to the end-to-end spectral reconstruction method, replacing the traditional random coding template, thereby improving the quality of spectral reconstruction.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral image processing technology, specifically to an end-to-end spectral reconstruction method and system. Background Technology

[0002] Spectral imaging captures the spectral power distribution of a scene or object as a three-dimensional data cube, comprising multiple two-dimensional images of the same scene measured at different wavelengths. Hyperspectral images typically contain dozens or even hundreds of spectral channels, with images in different spectral bands containing different spatial and spectral information. Therefore, its applications are very broad, including medical imaging, remote sensing, defense and surveillance, and food quality assessment.

[0003] With the rapid development of computational reconstruction, snapshot spectral imagers have also been further developed. These technologies, including Computed Tomography Spectroscopy (CTIS) and Prism Mask Spectral Video Imaging System (PMVIS), are capable of capturing complete hyperspectral images in a single shot. Based on compressed sensing theory, Wagadarikar et al. proposed Coded Aperture Snapshot Spectral Imaging (CASSI) (e.g., Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt., vol. 47, no. 10, pp. B44–B51, Apr 2008).

[0004] Furthermore, coded aperture snapshot spectral imaging has developed rapidly in recent years. In the halftone domain, Ulichney proposed a uniformly distributed random unbiased noise model—the blue noise model. This model can be applied to coded aperture templates. The only variable component in a coded aperture snapshot spectral imaging system is the coded aperture template. The coded aperture template determines the number of measurements acquired by the sensor and the quality of spectral reconstruction. Therefore, designing a reasonable coded aperture template will further improve the quality of spectral reconstruction. In the paper "Spatiotemporal blue noise coded aperture design for multi-shot compressive spectral imaging" (Claudia V Correa, Henry Arguello, and Gonzalo RArce, "Spatiotemporal blue noise coded aperture design for multi-shot compressive spectral imaging," JOSAA, vol. 33, no. 12, pp. 2312–2322, 2016), Correa et al. designed a spatial blue noise coded aperture based on the finite isometry property of the coded aperture snapshot spectral imaging sensing matrix, thus improving the quality of spectral imaging. Meanwhile, convolutional neural networks (CNNs) have been widely used in spectral reconstruction. Compared with traditional sparse recovery and dictionary learning, CNNs offer superior computational power and feature learning mapping. Currently, a new research direction is to replace iterative optimization in compressed sensing with deep neural networks. This method uses data-driven approaches instead of empirical design, further improving the quality of spectral reconstruction. However, data-driven approaches only optimize the spectral reconstruction process at the algorithmic level and do not consider joint optimization of the optical system design.

[0005] Based on the above analysis, the problems and shortcomings of existing technologies are as follows: Most current coded aperture snapshot spectral imaging technologies use fixed, random coded aperture templates, without designing more rationally structured templates. Furthermore, while improving the quality of spectral image reconstruction is achieved at the algorithm level, the joint optimization of the entire optical system is not considered, thus affecting the overall quality of spectral image reconstruction.

[0006] The challenges in addressing the above problems and deficiencies lie in: how to design a more rationally structured coding aperture template; and how to design a joint optimization of the entire optical system. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the prior art by proposing an end-to-end spectral reconstruction method and system. This method optimizes the design of the coded aperture template and neural network, and the learnable parameters are the coded aperture template and neural network parameters. This can minimize the deviation between the real image and the reconstructed image, and further improve the quality of spectral image reconstruction.

[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0009] An end-to-end spectral reconstruction method includes the following steps:

[0010] S1, Reconstructing the hyperspectral image

[0011] S1-1 constructs a sensing model for the adaptive coded aperture snapshot system using an adaptive coded template:

[0012] y = Hf + n

[0013] Where y is the original image acquired by the sensor, H represents the system observation matrix, f represents the original hyperspectral image to be recovered, and n is the sensor noise;

[0014] S1-2. Based on the above model, the hyperspectral image is initially reconstructed;

[0015] S2. Use the obtained hyperspectral image as input to optimize the neural network, train the neural network parameters based on the end-to-end spectral reconstruction model, and calculate the loss function.

[0016] S3. Backpropagate the error to the coding template to be designed, change the design parameters of the coding template, and continue to train the model until a high-quality hyperspectral image is reconstructed.

[0017] Preferably, in step S1, the system observation matrix H is obtained by the following formula: H = TPD

[0018] Where T∈{0,1} represents the encoding template state, 0 represents off, and 1 represents on; P represents the discretization model of the dispersive element; D represents spatial extraction related to the sensor pixel size, and the discretization model of the dispersive element is specifically represented as P∈R M(N+L-1)×MNL .

[0019] Preferably, the original image acquired by the sensor The system observation matrix The original hyperspectral image to be recovered, f∈R MNL The sensing noise Where R represents the dimension, M represents the length of the original spatial image, N represents the width of the original spatial image, and L represents the number of spectral bands.

[0020] Preferably, in steps S1-2, the method for initially reconstructing the hyperspectral image is to use image priors as regularization to constrain the solution space and solve a minimization problem to obtain the hyperspectral image:

[0021]

[0022] Where τ is the balance parameter, μ is the penalty parameter, and e∈R MNL As an auxiliary variable, The subscript 2 in the middle represents the L2 norm, and the 2 in the upper right corner represents the square. Solving this equation can be divided into two subproblems, thus finding the minimized f and e.

[0023] Preferably, the neural network optimized in step S2 is a CA-Unet network.

[0024] Preferably, the end-to-end spectral reconstruction model in step S2 is:

[0025]

[0026] Where θ represents the optimization parameters of the spectral reconstruction network, including the coding template design parameters, H is the system observation matrix used for spectral reconstruction network optimization, M(·) represents the entire spectral reconstruction network, and αR(t) represents regularization to achieve coding template binarization.

[0027] Preferably, R(t) is expressed as:

[0028]

[0029] Among them, t i,j ∈{0,1} represents the binarization of the encoding template.

[0030] Preferably, the loss function is expressed as:

[0031]

[0032] in, For the hyperspectral value of each reconstructed pixel, spectral loss minimizes the angle between the reconstructed image and the ground truth spectral features.

[0033] Preferably, in step S3, the coding template design parameter is the finite isometric property constant δ of the system sensing matrix, where δ is represented as a function of the random coded aperture pattern structure: T = t(δ), where t(·) represents the mapping function from the finite isometric property constant δ to the system observation matrix.

[0034] The present invention also discloses an end-to-end spectral reconstruction system, including an objective lens, an adaptive coding template, a relay lens, a prism, and a grayscale camera. The adaptive coding template includes a memory, a processor, and a computer program stored in the memory and capable of executing the end-to-end spectral reconstruction method on the processor.

[0035] This invention has the following characteristics and beneficial effects:

[0036] By adopting the above technical solution, the design and reconstruction methods of the optical system are combined. Based on the coded aperture snapshot system, an adaptive coding template is added according to the end-to-end spectral reconstruction method, which replaces the traditional random coding template, thereby improving the quality of spectral reconstruction. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of an end-to-end spectral reconstruction method according to the present invention.

[0039] Figure 2 This is a diagram of an end-to-end spectral reconstruction system according to the present invention. Detailed Implementation

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0041] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0042] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0043] This invention provides an end-to-end spectral reconstruction method, such as... Figure 1 As shown, it includes the following steps:

[0044] S1, Reconstructing the hyperspectral image

[0045] S1-1 constructs a sensing model for the adaptive coded aperture snapshot system using an adaptive coded template:

[0046] y = Hf + n

[0047] Where y is the original image acquired by the sensor, H represents the system observation matrix, f represents the original hyperspectral image to be recovered, and n is the sensor noise. Specifically, the original image acquired by the sensor... The system observation matrix The original hyperspectral image to be recovered, f∈R MNL The sensing noise Where R represents the dimension, M represents the length of the original spatial image, N represents the width of the original spatial image, and L represents the number of spectral bands.

[0048] The system observation matrix H is obtained by the following formula:

[0049] H = TPD

[0050] Where T∈{0,1} represents the encoding template state, 0 represents off, and 1 represents on; P represents the discretization model of the dispersive element; D represents spatial extraction related to the sensor pixel size, and the discretization model of the dispersive element is specifically represented as P∈R M(N+L-1)×MNL .

[0051] S1-2. Based on the above model, the hyperspectral image is initially reconstructed.

[0052] Specifically, the initial method for reconstructing the hyperspectral image involves using image priors as regularization to constrain the solution space and solving a minimization problem to obtain the hyperspectral image:

[0053]

[0054] Where τ is the balance parameter, μ is the penalty parameter, and e∈R MNLAs an auxiliary variable, The subscript 2 in the middle represents the L2 norm, and the 2 in the upper right corner represents the square. Solving this equation can be divided into two subproblems, thus finding the minimized f and e.

[0055]

[0056]

[0057] The gradient descent method is used to solve the above problem.

[0058]

[0059] Where λ is the step size of the gradient descent, set to 0.01. This is a hyperspectral image.

[0060] S2. Obtain the hyperspectral image As input to optimize the neural network, the neural network parameters are trained based on the end-to-end spectral reconstruction model, and the loss function is calculated.

[0061] Specifically, optimizing the neural network into a CA-Unet network can be represented as p(·), and minimizing e can be represented as:

[0062]

[0063] Understandably, the optimized neural network CA-Unet mainly consists of two parts: the Unet module and the channel attention module. In the Unet network, the first convolutional layer uses a 3×3×31 filter to generate a tensor with a feature size of 64 to enhance the sparsity of the spectral gradient. Then, the network generates multi-scale features, with a shrinking path using max pooling and an expanding path using upper convolutional layers. For each level, two convolutional layers encode spatial spectral features. The scaled features are connected to the upper scaled features via skip connections. In the channel attention module, the 64-channel, M×N feature map is extracted into 1×1×64 global information, then fully connected to obtain a 64 / r dimensional vector. ReLU activation is then applied, followed by another full connection to transform the 64 / r dimensional vector back to a 64 dimensional vector, and finally sigmoid activation to ensure the values ​​are between 0 and 1, resulting in the weight matrix. The weight matrix is ​​then multiplied by the feature map. Finally, we use 31 convolutional layers of 3×3×64 to generate a tensor of the original hyperspectral cube size, which is the recovered hyperspectral image.

[0064] Furthermore, the end-to-end spectral reconstruction model is as follows:

[0065]

[0066] Where θ represents the optimization parameters of the spectral reconstruction network, including the coding template design parameters, H is the system observation matrix used for spectral reconstruction network optimization, M(·) represents the entire spectral reconstruction network, and αR(t) represents regularization to achieve coding template binarization. Based on the end-to-end spectral reconstruction model, the coding aperture template can be jointly optimized and designed.

[0067] The R(t) is expressed as:

[0068]

[0069] Among them, t i,j ∈{0,1} represents the binarization of the encoding template.

[0070] The loss function is expressed as:

[0071]

[0072] in, For the hyperspectral value of each reconstructed pixel, the spectral loss minimizes the angle between the reconstructed image and the ground truth spectral features. The loss function value can be calculated using the above formula.

[0073] S3. Backpropagate the error to the coding template to be designed, change the design parameters of the coding template, and continue to train the model until a high-quality hyperspectral image is reconstructed.

[0074] Specifically, the design parameter for the coding template is the finite isometric property constant δ of the system sensing matrix, where δ is a function of the random coded aperture pattern structure: T = t(δ), where t(·) represents the mapping function from the finite isometric property constant δ to the system observation matrix. This function ensures maximum separation of the designed coding pattern in the horizontal direction and maximizes incoherence in the vertical direction. The initial δ is set to 0.5.

[0075] Error backpropagation uses gradient descent:

[0076]

[0077]

[0078] Where λ and ε are the gradient descent step sizes, with λ set to 0.001 and ε set to 0.002.

[0079] In this embodiment, the number of iterations is set to 20. The final δ and the designed adaptive coding template pattern are obtained, and a high-quality hyperspectral image is reconstructed.

[0080] This invention also discloses an end-to-end spectral reconstruction system, such as Figure 2As shown, it includes an objective lens, an adaptive coding template, a relay lens, a prism, and a grayscale camera. The adaptive coding template includes a memory, a processor, and a computer program stored in the memory and capable of executing the end-to-end spectral reconstruction method on the processor.

[0081] The working principle of the above technical solution:

[0082] The target scene is transmitted to the adaptive coding template through the objective lens. The hyperspectral image is reconstructed using the end-to-end spectral reconstruction method described above. The reconstructed hyperspectral image is then transmitted through a relay lens and a prism to a grayscale camera. The reconstructed hyperspectral image is then optimized by an optimization neural network and sent to the adaptive coding template for iteration, ultimately outputting a high-quality hyperspectral image.

[0083] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. An end-to-end spectral reconstruction method, characterized by, The method comprises the following steps: S1, reconstructing a hyperspectral image S1-1, constructing a sensing model of an adaptive coded aperture snapshot system by an adaptive coding template: y = Hf + n Wherein, y is an original image collected by a sensor, H represents a system observation matrix, f represents an original hyperspectral image to be recovered, and n is a sensor noise; S1-2, reconstructing a hyperspectral image according to the above model; The method for the preliminary reconstruction of the hyperspectral image is to use an image prior as a regularization to constrain a solution space, and to obtain the hyperspectral image by solving a minimization problem: where τ is a balancing parameter, μ is a penalty parameter, e ∈ R MNL is an auxiliary variable, The subscript 2 in the middle denotes the L2 norm, the 2 in the upper right corner denotes squaring, and the solution of the equation can be divided into two sub-problems to find the minimized f and e; S2, taking the obtained hyperspectral image as an input of an optimization neural network, training neural network parameters according to an end-to-end spectral reconstruction model, and calculating a loss function; The loss function is expressed as: wherein, For each pixel point of the reconstruction, the spectral loss minimizes the angle between the reconstructed image and the true spectral feature. S3, error is back-propagated to a coded template to be designed, coded template design parameters are changed, the model is continuously trained, and a high-quality hyperspectral image is reconstructed, The error back-propagation adopts a gradient descent method: The coded template design parameter is a constant δ of a finite isometry property of a system sensing matrix, and δ is expressed as a function of a random coded aperture pattern structure: T = t(δ), wherein t(·) represents a mapping function of the constant δ of the finite isometry property to the system observation matrix.

2. The end-to-end spectral reconstruction method of claim 1, wherein, In the step S1, the system observation matrix H is obtained by the following formula: H = TPD wherein T∈{0,1} represents the encoding template state, 0 represents off, and 1 represents on; P represents a dispersion element discretization model; D represents a spatial decimation related to a sensor pixel size, and the dispersion element discretization model is specifically represented as P∈R M(N +L-1)×MNL .

3. The end-to-end spectral reconstruction method of claim 2, wherein, The raw image collected by the sensor The system observation matrix The original hyperspectral image to be recovered f ∈ R MNL The sensor noise Wherein, R represents the dimension, M represents the length of the spatial raw image, N represents the width of the spatial raw image, and L represents the number of spectral bands.

4. The end-to-end spectral reconstruction method of claim 1, wherein, The optimization neural network in the step S2 is a CA-Unet network.

5. The end-to-end spectral reconstruction method of claim 4, wherein, The end-to-end spectral reconstruction model in the step S2 is: Wherein, θ represents optimization parameters of the spectral reconstruction network, including coded template design parameters, H is a system observation matrix, is used for optimization of the spectral reconstruction network, M(·) represents the entire spectral reconstruction network, αR(t) represents regularization, and binary coding of the coded template is realized.

6. The end-to-end spectral reconstruction method of claim 5, wherein, The R(t) is expressed as: where t i,j ∈ {0,1} denotes binarization of the encoding template.

7. An end-to-end spectral reconstruction system, comprising: The system comprises an objective lens, an adaptive coded template, a relay lens, a prism, and a gray-scale camera, the adaptive coded template comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is the end-to-end spectral reconstruction method according to any one of claims 1-6.

Images