Phase mask constrained image reconstruction method and device based on end-to-end deep learning, equipment and storage medium

Abstract

The invention relates to an end-to-end deep learning-based phase mask constrained image reconstruction method, apparatus and device, and a storage medium. The method comprises the following steps: acquiring hyperspectral image data, inputting a single two-dimensional hyperspectral image into a CASSI imaging system, and constraining the single two-dimensional hyperspectral image by using a binary phase mask; performing spectral information compression on the constrained single two-dimensional hyperspectral image, performing image reconstruction through a compressed sensing algorithm, and outputting a measurement value; a measurement value output by the CASSI imaging system is constrained by using regularization constraint; randomly initialized or pre-trained weights are used to initialize the U-Net network, the training set and the verification set are used to carry out joint training and verification on the phase mask and the U-Net network, and joint optimization of the phase mask and the U-Net network is realized; and carrying out hyperspectral image reconstruction by using a U-Net network, and evaluating the quality of a reconstructed image by continuously iterating and jointly optimizing parameters of a phase mask and a network model.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging technology, and in particular to an image reconstruction method, device, equipment and storage medium based on end-to-end deep learning and phase mask constraints. Background Art

[0002] The development of early computational optical imaging technology was inspired by human vision. Imaging deals with capturing image representations of the world, while computer vision extracts meaningful information from these images for high-level tasks such as classification, detection, and segmentation. Together, these two steps, imaging and vision, constitute the visual perception pipeline. The imaging step can be interpreted as a physical encoder that converts a 3D scene into a low-dimensional representation (i.e., an image), while the vision step can be interpreted as a digital decoder that uses this representation to perform high-level tasks. Although perception is more closely linked to vision, the downstream effects of imaging on perception have been overlooked.

[0003] The current mainstream design methods of optical coding elements (OCEs) are designed separately and independently from the downstream target tasks, and are optimized and evaluated using traditional design methods. Some optical optimization methods rely on a rich ecosystem of optimization tools, but do not allow joint optimization across the entire imaging pipeline because the number of parameters to be optimized becomes very large, and thus existing optical systems remain isolated. Currently, research on the joint design of optical elements and downstream tasks has just begun. The basic idea is to model the optical element as a layer of a neural network, and the trainable parameters that characterize the optical element are coupled with the network that performs the decoding task (i.e., reconstruction, classification, segmentation, etc.) and then jointly trained, so that the parameters of the optical element can be adjusted according to the properties of the training database and the needs of the computational decoder.

[0004] Computational Optical Imaging (COI) systems exploit OCE to encode high-dimensional scenes in single or multiple snapshots and decode them using computational algorithms. The performance of COI systems is highly dependent on the design of their main components: the OCE pattern and the computational method used to perform a given task. COI measurements are modeled by a fully differentiable image formation model that considers the physics-based propagation of light and its interaction with OCE, and the parameters defining the OCE and the computational decoder can be optimized in an end-to-end (E2E) manner.

[0005] In 2018, Vincent Sitzmann et al. established a fully differentiable simulation model to map the real source image to the reconstructed image, and jointly optimized the optical parameters and image processing algorithm parameters to minimize the deviation between the real image and the reconstructed image on a large image set. In 2022, Roman Jacome et al. proposed an end-to-end joint optimization of the sensing architecture and the reconstruction network. The trainable optical parameters are the CA in the CASSI system and the color-coded aperture in the MCFA system. The sigmoid activation function and regularization function are used to binarize the trainable variables to achieve the purpose.

[0006] Although the current joint design of optical components and algorithms has achieved certain research performance improvements, there are still many limitations in existing methods, including: the optimization of the parameters of the optical coding element is only related to the loss function of the computational decoding network output calculation, which makes the performance of the E2E network more dependent on / inclined to the best / optimal computational decoding network training rather than the best optical coding design; lack of constraints on the measurement values ​​corresponding to the optical coding element and careful optimization considerations; the optional space of the parameters of the optical coding element is severely limited, which reduces the degree of freedom of training and limits the performance of the computational decoding network after training. The main reason for this problem is that the intermediate output (coding measurement) of E2E is not considered independently in training, and the optimization is only performed on the output.

[0007] Therefore, it is necessary to implement an image reconstruction method, apparatus, device and storage medium based on end-to-end deep learning and phase mask constraints. Summary of the invention

[0008] In order to address the deficiencies in the prior art, the object of the present invention is to provide an image reconstruction method, apparatus, device and storage medium based on end-to-end deep learning and phase mask constraints.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect of the present invention, a phase mask constrained image reconstruction method based on end-to-end deep learning is disclosed.

[0011] Specifically, the method comprises the following steps:

[0012] S1. Mask constraint: Obtain hyperspectral image data, input a single two-dimensional hyperspectral image into the CASSI imaging system, and then use a binary phase mask to constrain the single two-dimensional hyperspectral image after entering the CASSI imaging system.

[0013] Phase masks do not absorb or reflect part of the light intensity like amplitude masks, so they can reduce light loss in the optical system. This means that more light energy can pass through the system, improving the light intensity and signal-to-noise ratio of the image.

[0014] S2. Initial reconstruction: Use the CASSI imaging system to compress the spectral information of the constrained single two-dimensional hyperspectral image, reconstruct the image through the compressed sensing algorithm, and output the measurement value.

[0015] Step S2 achieves efficient and high-resolution spectral imaging, reducing sampling time and storage requirements while maintaining high image quality.

[0016] S3. Measurement value constraint: Regularization constraints are used to constrain the measurement values ​​output by the CASSI imaging system.

[0017] By imposing additional constraints on the set of measurements output from the CASSI imaging system, the performance of the optical imaging system can be improved through different regularization functions, focusing on the distribution of the measurements (reducing data variance), and thus representing the data more compactly, resulting in better reconstruction quality.

[0018] S4. Joint optimization: Determine the U-Net network, initialize the U-Net network using randomly initialized or pre-trained weights, divide the constrained hyperspectral image data into a training set and a validation set, and use the training set and the validation set to jointly train and validate the phase mask and the U-Net network to achieve joint optimization of the phase mask and the U-Net network.

[0019] The network parameters are optimized by minimizing the loss function. Multiple iterations are performed during the training process, and the network weights and optical layer parameters are updated through the back-propagation algorithm.

[0020] S5. Reconstruct image: Use the U-Net network to reconstruct the hyperspectral image, continuously iterate and optimize the parameters of the phase mask and network model, and use PSNR or SSIM to evaluate the quality of the reconstructed image.

[0021] When using U-Net networks or other methods for hyperspectral image reconstruction, it is a common process to improve the quality of the reconstructed image by continuously iteratively optimizing the model parameters. PSNR (Peak Signal-to-Noise Ratio) is a commonly used image quality evaluation indicator that measures the degree of difference between the original image and the reconstructed image. A higher PSNR value generally means better image quality.

[0022] Preferably, according to the present invention, the graphic reconstruction method is implemented by using an image reconstruction system, and the image reconstruction system includes an optical encoding system, an intermediate output layer and a decoder.

[0023] According to a preferred embodiment of the present invention, the optical encoding system is an optical system with phase mask constraints, which is used to obtain projection measurement values ​​entering a computational decoder to generate outputs for any task.

[0024] The optical encoding system and the decoder are modeled as layers in a U-Net network, and the task error estimated by the optical system is propagated from the output of the decoder to the optical layer to update the weights of the decoder and the optical system.

[0025] The intermediate output layer is used to impose regularization constraints on the measured values ​​and execute a set of regularization functions in the output of the optical encoding element.

[0026] Preferably, according to the present invention, in step S4, the network optimization is implemented using the following formula:

[0027]

[0028] Among them, {Φ * , *} represent the optimal optical coding parameter set and optimal weight of the network, respectively. Represents the task function and loss function of this system regarding the optical encoding parameter set and weights Links to image reconstruction tasks, optical encoding systems and decoder represents the training database of K components, f k is the training data set, d k As the output of the neural decoder, R ρ (Φ) and R σ (θ) represents the regularization function in the optical parameters and weights acting on the decoder, ρ and σ are regularization parameters, μ is the regularization parameter, is a matrix containing the compressed measurements of all training batches, i.e.

[0029] Preferably, according to the present invention, in step S1, a single two-dimensional hyperspectral image is constrained using a binary phase mask, which is implemented using the following formula:

[0030]

[0031] in, is the regularization function, S is the number of snapshots, Φ is the parameterized optical coding element used to characterize the sensing system, i, j, l are coefficients, and p1 and p2 are two hyperparameters that provide the variability of the function curve.

[0032] Preferably, according to the present invention, in the image reconstruction process in step S5, in the reconstruction task, the regularizer shown in the following first formula is used to minimize the variance; in the classification task, the regularizer shown in the following second formula is used to maximize the variance:

[0033] R rec (Y)=||σ Y ||2.

[0034] in, is a matrix containing the compressed measurements of all training batches, i.e. σ Y is the variance of Y;

[0035] R cla (Y)=||σ max -σ Y ||2,

[0036] in, is a matrix containing the compressed measurements of all training batches, i.e. σ Y is the variance of Y, σ max is the maximum variance reference value and is an adjustable hyperparameter.

[0037] Preferably, according to the present invention, the gradient of the loss function of the U-Net network relative to the optical encoding element is:

[0038]

[0039] in, is the loss function, Φ is the parameterized optical encoding element used to characterize the sensing system, and the loss function Link to the image reconstruction task, To encode measured values, optical encoding systems and decoder R ρ (Φ) and R σ (θ) represents the regularization function acting on the optical parameters and weights of the decoder, R(Y) represents the regularization function acting on the intermediate measurements, and ρ, σ and μ are used as regularization parameters.

[0040] In a second aspect of the present invention, a phase mask-constrained image reconstruction device based on end-to-end deep learning is disclosed. The device comprises a mask constraint module, an initial reconstruction module, a measurement value constraint module, a joint optimization module and a reconstruction image module.

[0041] The mask constraint module is used to obtain hyperspectral image data, input a single two-dimensional hyperspectral image into the CASSI imaging system, and after entering the CASSI imaging system, use a binary phase mask to constrain the single two-dimensional hyperspectral image.

[0042] The initial reconstruction module is used to compress the spectral information of the constrained single two-dimensional hyperspectral image using the CASSI imaging system, reconstruct the image through a compressed sensing algorithm, and output a measurement value.

[0043] The measurement value constraint module uses regularization constraints to constrain the measurement values ​​output by the CASSI imaging system.

[0044] The joint optimization module is used to determine a U-Net network, initialize the U-Net network using randomly initialized or pre-trained weights, divide the constrained hyperspectral image data into a training set and a validation set, and use the training set and the validation set to jointly train and validate the phase mask and the U-Net network to achieve joint optimization of the phase mask and the U-Net network;

[0045] The image reconstruction module is used to reconstruct the hyperspectral image using the U-Net network, continuously and iteratively optimize the parameters of the phase mask and the network model, and evaluate the quality of the reconstructed image using PSNR or SSIM.

[0046] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor; and a memory, wherein the memory stores instructions, and when the instructions are executed by the at least one processor, the at least one processor executes the above-mentioned end-to-end deep learning-based phase mask constrained image reconstruction method.

[0047] In a fourth aspect of the present invention, a machine-readable storage medium is disclosed, which stores executable instructions, which, when executed, enable the machine to perform the above-mentioned end-to-end deep learning-based phase mask constrained image reconstruction method.

[0048] Compared with the prior art, the advantages of the present invention are:

[0049] Although the current joint design of optical elements and algorithms has achieved certain research performance improvements, the existing methods still have many limitations. The present invention proposes an image reconstruction method, device, equipment and storage medium based on end-to-end deep learning and phase mask constraints. Based on the joint design of optical elements and algorithms, the optical coding element is optimized to improve the light transmittance; regularized prior constraints are imposed on the optimized optical elements to improve the degree of freedom of the optimized optical elements. In order to further improve the performance of the optical imaging system, the present invention imposes constraints on the set of measured values ​​output from the imaging model, and improves the performance of the optical imaging system through different regularization functions, so that the data can be represented more compactly and better reconstruction quality can be obtained, which provides a new idea and method for the research and application of end-to-end computational optical imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of an image reconstruction method based on phase mask constraints of end-to-end deep learning in the present invention;

[0051] Figure 2 It is a traditional computational optical imaging model;

[0052] Figure 3 It is a flowchart of joint optimization of phase mask and reconstruction algorithm in lensless imaging. DETAILED DESCRIPTION

[0053] The present disclosure is further described below with reference to the accompanying drawings and embodiments:

[0054] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0055] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0056] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0057] Example 1

[0058] This embodiment provides an image reconstruction method based on phase mask constraints based on end-to-end deep learning. Figure 1 As shown, the method comprises the following steps:

[0059] S1. Mask constraint: Obtain hyperspectral image data, input a single two-dimensional hyperspectral image into the CASSI imaging system, and then use a binary phase mask to constrain the single two-dimensional hyperspectral image after entering the CASSI imaging system. The phase mask does not absorb or reflect part of the light intensity like the amplitude mask, so it can reduce the light loss in the optical system. This means that more light energy can pass through the system, improving the light intensity and signal-to-noise ratio of the imaging.

[0060] S2, initial reconstruction: The CASSI imaging system is used to compress the spectral information of the constrained single two-dimensional hyperspectral image, and the image is reconstructed through the compressed sensing algorithm to output the measurement value. Step S2 achieves efficient and high-resolution spectral imaging, reducing sampling time and storage requirements while maintaining high image quality.

[0061] S3, measurement value constraints: Regularization constraints are used to constrain the measurement values ​​output by the CASSI imaging system. By imposing additional constraints on the set of measurement values ​​output from the CASSI imaging system, the performance of the optical imaging system is improved through different regularization functions, focusing on the distribution of the measurement values ​​(reducing data variance), and representing the data more compactly, thereby obtaining better reconstruction quality.

[0062] S4, Joint Optimization: Determine the U-Net network, use random initialization or pre-trained weights to initialize the U-Net network, and divide the constrained hyperspectral image data into a training set and a validation set. Use the training set and validation set to jointly train and validate the phase mask and U-Net network, and continuously iterate and optimize the parameters of the phase mask and network model. Use the minimization loss function to optimize the network parameters, perform multiple iterations during the training process, and update the network weights and optical layer parameters through the back propagation algorithm.

[0063] S5. Reconstructed image: Use the U-Net network to reconstruct the hyperspectral image, optimize the model parameters through continuous iteration, and use PSNR (peak signal-to-noise ratio) or SSIM (structural similarity index) to evaluate the quality of the reconstructed image.

[0064] The principles of existing computational optical imaging models are as follows: Figure 2 As shown:

[0065] (1) The COI system is modeled as an optical layer. During training, a set of images passes through the optical system to obtain projection measurements that enter the computational decoder to produce output for any task. The COI system is represented in vector form as shown in formula (1):

[0066]

[0067] in, To encode the measurement value, it is the vector representation of Y; is the sensing matrix of the system, i.e., the optical encoder, which simulates the three basic operators of the COI system: propagation, interaction with CEs Φ = {Φ l} l＝1,...,L and the sensing process; Φ is the parameterized optical coding element used to characterize the sensing system; Represents the underlying scene given by the input domain, m<<n; ω is additive noise.

[0068] (2) Optical encoding elements and computational decoding networks Co-optimization. In the COI system, the optical coding system and decoder Modeled as layers, the task error estimated by the COI system is propagated from the output of the decoder to the optical layer, updating the weights of the decoder and the optical system. The system jointly optimizes the set of CAs Parameter Q and the selected The cost function is composed of the loss function L of the implementation task task (·,·) and customizable regularization function The first module involves the sensing protocol, which consists of an optical layer that learns the sensing matrix through a set of CAs as trainable parameters. The optical layer is directly connected to the second module, called the task module, which consists of a selected trainable DNN with various hidden layers to complete the task.

[0069] (3) The end-to-end joint optimization can be expressed as an optimization problem as shown in formula (2) to train the encoder-decoder parameters:

[0070]

[0071] Among them, {Φ * ,θ *} represent the optimal optical coding parameter set and optimal weight of the network, respectively. Represents the task function and loss function of this system regarding the optical encoding parameter set and weights Linked to a specific inference task. For example, mean squared error (MSE) and cross entropy metrics are often used for reconstruction and classification tasks, respectively. Optical Coding Systems and decoder represents the training database of K components, f k is the training data set, d k The output of the neural decoder can be a target image, a classification vector, a segmentation map, etc. ρ (Φ) and R σ(θ) represents the regularization function acting on the optical parameters and weights of the decoder, respectively, and ρ and σ are regularization parameters. Regularization functions have been widely used in training neural networks because R σ (θ) has been shown to reduce the overfitting problem, which is a common problem when training DNNs.

[0072] (4) Optical parameters R ρ (Φ) This regularization can also induce desired properties for the optical coding element, such as transmittance in CA, number of shots, etc. The optical coding element is updated according to the task loss function and the physical constraints given by the regularization. Regularizations are different from network weights because they directly change the value of physical optics. Therefore, it is useful to improve the desired properties of CE.

[0073] (5) The binary constraint is solved by proposing a set of functions whose minimum is uniquely obtained when the elements of CA are (0) or (1). It is incorporated in (3) as a regularizer

[0074]

[0075] in, is the regularization function, S is the number of snapshots, Φ is the parameterized optical encoding element used to characterize the sensing system, i, j, l are coefficients, and p1, p2 are two hyperparameters that provide the variability of the function curve.

[0076] (6) The main idea of ​​adding regularization to training is to use the chain rule to calculate the combined loss function with respect to Φ The gradient of , the task loss function and the regularization function directly affect the design of the optical coding element.

[0077]

[0078] in, is the loss function, Φ is the parameterized optical encoding element used to characterize the sensing system, and the loss function Link to the image reconstruction task, To encode measured values, optical encoding systems and decoder R ρ (Φ) and R σ (θ) denotes the regularization function acting on the optical parameters and weights of the decoder, respectively, and ρ and σ serve as regularization parameters.

[0079] In the implementation of the phase mask constrained image reconstruction method based on end-to-end deep learning described in this embodiment: First, a set of images passes through an optical system with phase mask constraints to obtain projection measurements that enter a computational decoder to produce outputs for any task. The optical encoding system and decoder are modeled as layers, and the task error estimated by the system is propagated from the output of the decoder to the optical layer to update the weights of the decoder and the optical system. Secondly, the system jointly optimizes the set of CAs The parameter Θ and the selected The cost function is composed of the loss function L of the implementation task task (·,·) and customizable regularization function The encoder and decoder steps are modeled by following a two-module process. The first module involves the sensing protocol, which contains an optical layer that learns the sensing matrix, which is composed of an optical layer that learns the perception matrix through a set of CAs as trainable parameters. The optical layer is directly connected to the second module, which is the task module, which is composed of a selected trainable DNN with various hidden layers to complete the task. In this embodiment, the DNN adopts U-Net.

[0080] In the end-to-end optimization framework, both the optical encoding system and the decoder are modeled as layers in a neural network. This design allows the task error to be propagated from the output of the decoder back to the optical layer through the back-propagation algorithm, thereby updating the weights of the decoder and the optical system simultaneously. This approach can achieve joint optimization of the entire system and improve overall performance.

[0081] Currently, there are two main problems with training optical encoding elements:

[0082] (1) The training has high requirements on the physical constraints of the regularization function, which will reduce the degrees of freedom of the optical encoding element;

[0083] (2) Due to the optical encoder It is the first layer of the E2E network, and the gradient will disappear. Most of the optimization is in calculating the decoder The main reason for this problem is that the intermediate outputs of E2E (encoding measurements) are not considered independently in the training, and the optimization is only performed on the optical encoding. Output is carried out.

[0084] In order to overcome the optical coding element training problem, the present invention adds an extra intermediate output layer in the E2E joint training framework and imposes additional regularization constraints on the measurement values ​​of the intermediate layer output, that is, a set of regularization functions are executed in the optical coding element output.

[0085] The improved E2E optimization framework to be adopted in the present invention is as follows Figure 3As shown. At this time, the optimization problem becomes:

[0086]

[0087] Among them, {Φ * ,θ *} represent the optimal optical coding parameter set and optimal weight of the network, respectively. Represents the task function and loss function of this system regarding the optical encoding parameter set and weights Link to image reconstruction tasks. Optical Coding Systems and decoder represents the training database of K components, f k is the training data set, d k As the output of the neural decoder. ρ (Φ) and R σ (θ) represents the regularization function in the optical parameters and weights of the decoder, respectively, and ρ and σ are regularization parameters. μ is the regularization parameter, is a matrix containing the compressed measurements of all training batches, i.e.

[0088] The binary phase mask constraint is also solved by proposing a set of functions, analogous to the case when the elements of CA are (-1) or (1), the minimum of these functions is the only one obtained.

[0089]

[0090] in, is the regularization function, S is the number of snapshots, Φ is the parameterized optical encoding element used to characterize the sensing system, i, j, l are coefficients, and p1, p2 are two hyperparameters that provide the variability of the function curve.

[0091] In order to further improve the performance of the optical imaging system, the present invention imposes constraints on the set of measured values, improves the performance of the optical imaging system through different regularization functions, and focuses on the distribution of the measured values ​​(reducing the data variance), so that the data can be represented more compactly and better reconstruction quality can be obtained. According to these criteria, the present invention proposes an E2E optimization method based on intermediate output regularization, that is, regularization based on the variance of the encoded observations to improve the characteristics of OE: controlling the variance can make the data representation more compact. The present invention minimizes the variance in the reconstruction task and maximizes the variance in the classification task. By reducing the variance of the measurement set, better reconstruction quality can be obtained.

[0092] In the regularization process, in order to control the concentration of the measured value distribution, it is necessary to adjust the hyperparameter μ in formula (5). In the reconstruction task, the variance is minimized, and the function expression of the regularizer is as shown in formula (7):

[0093] R rec (Y)=||σ Y ||2. (7)

[0094] in, is a matrix containing the compressed measurements of all training batches, i.e. σ Y is the variance of Y.

[0095] In the classification task, the present invention identifies the differences between different categories from the image. If the distribution of the measured values ​​is wider, that is, the variance is larger, CD can better identify the category. Therefore, the variance maximization can be promoted by the regularization function shown in formula (8):

[0096] R cla (Y)=||σ max -σ Y ||2, (8)

[0097] in, is a matrix containing the compressed measurements of all training batches, i.e. σ Y is the variance of Y, σ max is the maximum variance reference value and is an adjustable hyperparameter.

[0098] According to the chain rule, the gradient of the loss function relative to the optical encoding element is:

[0099]

[0100] in, is the loss function, Φ is the parameterized optical encoding element used to characterize the sensing system, and the loss function Link to the image reconstruction task, To encode measured values, optical encoding systems and decoder R ρ (Φ) and R σ (θ) represents the regularization function acting on the optical parameters and weights of the decoder, R(Y) represents the regularization function acting on the intermediate measurements, and ρ, σ and μ are used as regularization parameters.

[0101] In this embodiment, a joint optimization framework of an image reconstruction algorithm with phase mask constraints based on end-to-end deep learning is proposed. First, a binary phase mask that can improve light transmittance is designed to replace the coded aperture CA for optimization; secondly, the loss function and regularization constraints corresponding to the phase mask are constructed; finally, an E2E optimization method based on intermediate output regularization is adopted, that is, regularization based on the variance of the coded observation value, a set of regularization functions is used to overcome the defects of the existing encoder optimization, and some prior knowledge about the computational task, the statistical characteristics of the encoder output (measurement) and the sensor model is utilized. The method proposed in the present invention provides a new idea and method for the research and application in the field of end-to-end computational optical imaging.

[0102] Example 2

[0103] This embodiment provides an image reconstruction device based on end-to-end deep learning and phase mask constraints. The device includes a mask constraint module, an initial reconstruction module, a measurement value constraint module, a joint optimization module and an image reconstruction module.

[0104] The mask constraint module is used to obtain hyperspectral image data, input a single two-dimensional hyperspectral image into the CASSI imaging system, and after entering the CASSI imaging system, use a binary phase mask to constrain the single two-dimensional hyperspectral image.

[0105] The initial reconstruction module is used to compress the spectral information of the constrained single two-dimensional hyperspectral image using the CASSI imaging system, reconstruct the image through a compressed sensing algorithm, and output a measurement value.

[0106] The measurement value constraint module uses regularization constraints to constrain the measurement values ​​output by the CASSI imaging system.

[0107] The joint optimization module is used to determine the U-Net network, initialize the U-Net network using randomly initialized or pre-trained weights, divide the constrained hyperspectral image data into a training set and a validation set, and use the training set and the validation set to jointly train and validate the phase mask and the U-Net network to achieve joint optimization of the phase mask and the U-Net network.

[0108] The image reconstruction module is used to reconstruct the hyperspectral image using the U-Net network, continuously and iteratively optimize the parameters of the phase mask and the network model, and evaluate the quality of the reconstructed image using PSNR or SSIM.

[0109] Example 3

[0110] This embodiment also provides an electronic device, comprising: at least one processor; and a memory, wherein the memory stores instructions, and when the instructions are executed by the at least one processor, the at least one processor executes the image reconstruction method based on end-to-end deep learning and phase mask constraints as described above.

[0111] In this embodiment, the electronic device may include, but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, and the like.

[0112] Example 4

[0113] This embodiment also provides a machine-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the phase mask constrained image reconstruction method based on end-to-end deep learning as described above.

[0114] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0115] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0116] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0117] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] The above-described embodiments are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the scope of protection determined by the claims of the present invention.

Claims

1. A phase mask constrained image reconstruction method based on end-to-end deep learning, characterized in that: The method comprises the following steps: S1, mask constraint: obtain hyperspectral image data, input a single two-dimensional hyperspectral image into the CASSI imaging system, and after entering the CASSI imaging system, use a binary phase mask to constrain the single two-dimensional hyperspectral image; S2, initial reconstruction: use the CASSI imaging system to compress the spectral information of the constrained single two-dimensional hyperspectral image, reconstruct the image through the compressed sensing algorithm, and output the measurement value; S3, measurement value constraint: use regularization constraints to constrain the measurement values ​​output by the CASSI imaging system; S4, joint optimization: determine the U-Net network, use random initialization or pre-trained weights to initialize the U-Net network, and divide the constrained hyperspectral image data into a training set and a validation set, use the training set and the validation set to jointly train and validate the phase mask and the U-Net network, and achieve joint optimization of the phase mask and the U-Net network; S5. Reconstructed image: Use the U-Net network to reconstruct the hyperspectral image, iteratively optimize the parameters of the phase mask and network model, and use PSNR or SSIM to evaluate the quality of the reconstructed image.

2. The phase mask constrained image reconstruction method based on end-to-end deep learning according to claim 1, characterized in that: The graphic reconstruction method is implemented by using an image reconstruction system, which includes an optical encoding system, an intermediate output layer and a decoder.

3. The phase mask constrained image reconstruction method based on end-to-end deep learning according to claim 2, characterized in that: The optical encoding system is an optical system with phase mask constraints for obtaining projection measurements that enter a computational decoder to produce output for an arbitrary task; The optical encoding system and the decoder are modeled as layers in a U-Net network, and the task error estimated by the optical system is propagated from the output of the decoder to the optical layer to update the weights of the decoder and the optical system; The intermediate output layer is used to impose regularization constraints on the measured values ​​and execute a set of regularization functions in the output of the optical encoding element.

4. The phase mask constrained image reconstruction method based on end-to-end deep learning according to claim 1, characterized in that: In step S4, the network optimization is implemented using the following formula: Among them, {Φ * ,θ * } represent the optimal optical coding parameter set and optimal weight of the network, respectively. Represents the task function and loss function of this system regarding the optical encoding parameter set and weights Links to image reconstruction tasks, optical encoding systems and decoder represents the training database of K components, f k is the training data set, d k As the output of the neural decoder, R ρ (Φ) and R σ (θ) represents the regularization function in the optical parameters and weights acting on the decoder, ρ and σ are regularization parameters, μ is the regularization parameter, is a matrix containing the compressed measurements of all training batches, i.e.

5. The phase mask constrained image reconstruction method based on end-to-end deep learning according to claim 1, characterized in that: In step S1, a single two-dimensional hyperspectral image is constrained using a binary phase mask, which is implemented using the following formula: in, is the regularization function, S is the number of snapshots, Φ is the parameterized optical coding element used to characterize the sensing system, i, j, l are coefficients, and p1 and p2 are two hyperparameters that provide the variability of the function curve.

6. The phase mask constrained image reconstruction method based on end-to-end deep learning according to claim 1, characterized in that: In the image reconstruction process in step S5, in the reconstruction task, the regularizer shown in the following first formula is used to minimize the variance; in the classification task, the regularizer shown in the following second formula is used to maximize the variance: R rec (Y)=||σ Y ||2. in, is a matrix containing the compressed measurements of all training batches, i.e. σ Y is the variance of Y; R cla (Y)=||σ max -s Y ||2, in, is a matrix containing the compressed measurements of all training batches, i.e. σ Y is the variance of Y, σ max is the maximum variance reference value and is an adjustable hyperparameter.

7. The phase mask constrained image reconstruction method based on end-to-end deep learning according to claim 1, characterized in that: The gradient of the loss function of the U-Net network relative to the optical encoding element is: in, is the loss function, Φ is the parameterized optical encoding element used to characterize the sensing system, and the loss function Link to the image reconstruction task, To encode measured values, optical encoding systems and decoder R ρ (Φ) and R σ (θ) represents the regularization function acting on the optical parameters and weights of the decoder, R(Y) represents the regularization function acting on the intermediate measurements, and ρ, σ and μ are used as regularization parameters.

8. A phase mask constrained image reconstruction device based on end-to-end deep learning, characterized in that: The device comprises a mask constraint module, an initial reconstruction module, a measurement value constraint module, a joint optimization module and a reconstruction image module; The mask constraint module is used to obtain hyperspectral image data, input a single two-dimensional hyperspectral image into the CASSI imaging system, and after entering the CASSI imaging system, constrain the single two-dimensional hyperspectral image using a binary phase mask; The initial reconstruction module is used to compress the spectral information of the constrained single two-dimensional hyperspectral image using the CASSI imaging system, reconstruct the image through a compressed sensing algorithm, and output a measurement value; The measurement value constraint module constrains the measurement value output by the CASSI imaging system using regularization constraints; The joint optimization module is used to determine a U-Net network, initialize the U-Net network using randomly initialized or pre-trained weights, divide the constrained hyperspectral image data into a training set and a validation set, and use the training set and the validation set to jointly train and validate the phase mask and the U-Net network to achieve joint optimization of the phase mask and the U-Net network; The image reconstruction module is used to reconstruct the hyperspectral image using the U-Net network, continuously and iteratively optimize the parameters of the phase mask and the network model, and evaluate the quality of the reconstructed image using PSNR or SSIM.

9. An electronic device, characterized in that: include: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to perform the image reconstruction method based on end-to-end deep learning with phase mask constraints as described in any one of claims 1 to 7.

10. A machine-readable storage medium, characterized in that: The machine stores executable instructions which, when executed, enable the machine to perform the phase mask constrained image reconstruction method based on end-to-end deep learning as described in any one of claims 1 to 5.

Images