Lens-free imaging reconstruction method driven by end-to-end preprocessing network

Abstract

The invention provides a lens-free imaging reconstruction method driven by an end-to-end preprocessing network. The method comprises the following steps that S1, an original image passes through a diffuser lens-free imaging system to obtain a measurement image; s2, constructing a lightweight convolutional neural network CNN-ISP-net, and performing training by using the training set; s3, inputting the measurement image and the point spread function into the trained lightweight convolutional neural network for preprocessing, and obtaining the preprocessed measurement image and the point spread function; s4, constructing and training an Le-ADMM network for image reconstruction; and S5, jointly inputting the preprocessed point spread function and the measured image into the trained Le-ADMM network for image reconstruction. According to the method, the measurement image and the measurement point spread function are preprocessed through the lightweight convolutional neural network CNN-ISP-net, the quality of the measurement image is optimized, the measurement point spread function is closer to an ideal point spread function, and the purpose of improving the effect of the reconstructed image is achieved.

Description

Technical Field

[0001] The present invention relates to the field of lens-free imaging technology, and in particular to a lens-free imaging reconstruction method driven by an end-to-end preprocessing network. Background Art

[0002] Lensless imaging is an emerging imaging technology that replaces lenses with computing. Compared with traditional imaging systems, it is compact and convenient.

[0003] Patent application number CN112950750A proposes a lensless camera image reconstruction method based on a coded mask and a Learned-TSVD algorithm. The training method of the image preprocessing network includes: using different JPEG quantization factors to perform JPEG compression operations on the original images in the training data image set; using the preprocessed image of the original image of the image preprocessing network, and using the rate estimation network of a differentiable JPEG simulator to calculate the rate of the preprocessed image; using the differentiable JPEG simulator to reconstruct the preprocessing result of the original image, and calculate the distortion between the reconstructed image and the corresponding original image; using a predefined rate-distortion loss function to obtain a rate-distortion loss value, and adjusting the parameters of the image preprocessing network according to the rate-distortion loss value; adjusting the parameters of the rate estimation network; adjusting the parameters of the image preprocessing network and the rate estimation network in an alternating iterative manner to obtain the trained image preprocessing network and rate estimation network.

[0004] Patent application number CN112950750A proposes a lensless camera image reconstruction method based on a coded mask and a learned-TSVD algorithm. The coded mask is used to encode the light propagation process, and the separable properties of the coded mask and the TSVD algorithm are used to convert the original large-scale system measurement matrix into two smaller-scale left and right system measurement matrices. Next, a neural network is constructed to cyclically train the left and right system measurement matrices to reduce the error caused by approximate operations on the final result. Finally, the image is reconstructed using the TSVD algorithm and a regularization algorithm. This invention uses the learned system measurement matrix for subsequent calculations, making the overall reconstruction process more resistant to noise. The learned system measurement matrix can better reconstruct scene images at other distances, solving the problem of low depth of field in other reconstruction algorithms.

[0005] However, the current lensless imaging is affected by the non-ideal characteristics of CMOS sensors, and the point spread function (PSF) of the lensless imaging system has problems such as blurring and distortion.

[0006] To address these issues, the present invention proposes a lensless imaging reconstruction method driven by an end-to-end preprocessing network. Based on image signal processing (ISP), a lightweight fully convolutional neural network is designed. This network learns the error between the measured and idealized PSF caustics, thereby compensating for the measured PSF. The processed PSF is then used for image reconstruction. The reconstructed image network framework employed in the present invention is a parameter-learnable ADMM network, also known as the Le-ADMM network. Summary of the Invention

[0007] The purpose of this paper is to propose a lensless imaging reconstruction method driven by an end-to-end preprocessing network. To address the problems of noise, underexposure, and unclear details in the reconstructed image caused by using a CMOS sensor to collect the PSF in a lensless imaging system, a lightweight convolutional neural network (CNN-ISP-net) based on the ISP process is designed. Through the image preprocessing network, the measured point spread function is brought closer to the ideal point spread function, thereby improving the reconstructed image quality.

[0008] To achieve the above object, the technical solution of the present invention is: an end-to-end preprocessing network driven lens-free imaging reconstruction method, comprising the following steps:

[0009] S1, the original image is passed through the diffuser lensless imaging system to obtain the measurement image;

[0010] S2. Build a lightweight convolutional neural network CNN-ISP-net and train it using the training set;

[0011] S3, inputting the measured image and the point spread function into the trained lightweight convolutional neural network for preprocessing to obtain a preprocessed measured image and point spread function;

[0012] S4. Build and train the Le-ADMM network for image reconstruction.

[0013] S5. The preprocessed point spread function and the measured image are input into the trained Le-ADMM network for image reconstruction.

[0014] Preferably, in the diffuser lensless imaging system, the relationship between the measurement value b captured by the sensor and the reconstructed image x is expressed as:

[0015] b=FΦx (1)

[0016] The Φ matrix is ​​the transfer function composed of the point spread function, and F is the diffuser clipping function.

[0017] Preferably, the lightweight convolutional neural network CNN-ISP-net includes 7 Conv+ReLU layers and 1 Conv layer, and the output of each layer of the network serves as the input of the next layer.

[0018] Preferably, the first layer of the lightweight convolutional neural network CNN-ISP-net uses 8 filters of size 3×3×1 for convolution to extract local features; each layer of the 2nd to 7th layers uses 8 filters of size 3×3×8 for convolution to perform deeper feature extraction on the features extracted in the first layer; the 8th layer uses a convolution kernel of size 3×3×8 to integrate the features extracted previously.

[0019] Preferably, the lightweight convolutional neural network CNN-ISP-net is trained using three training sets, including an HDR+ dataset, a SID dataset, and a manually collected and processed point spread function dataset.

[0020] Preferably, the HDR+ dataset is constructed by capturing a series of underexposed natural scenes as original images, and then using DCraw to white balance and enhance the original images as control images of the original images.

[0021] Preferably, the SDI dataset is constructed by collecting short-exposure and noisy original images of different scenes, then using DCraw to restore the underexposed images, and using Photoshop's automatic enhancement tool to process the images to achieve an enhanced effect, and using the processed images as control images.

[0022] Preferably, the manually collected and processed point spread function data set is composed of low-light point spread functions at different distances. The diffuser lensless imaging system is used to collect point spread function images at different distances of 1-10 cm with an interval of 1 cm. Point sources of different pixel sizes of 0-70 are used at an interval of 10 pixels at each distance, and then the collected point spread function images are preprocessed by the ISP process; the unprocessed point spread function image is used as the original image, and the preprocessed point spread function image is used as the control image.

[0023] Preferably, the lightweight convolutional neural network CNN-ISP-net uses the difference between the restored image and the true image in linear and logarithmic space as the loss function:

[0024]

[0025] Among them I r is the restored image, G r is a real image, ε is a very small value, and 0<ε≤0.05.

[0026] Preferably, the S4 is specifically:

[0027] Convert the solution of reconstructed image into a regularized optimization problem:

[0028]

[0029] where Ψ is the sparse transform, λ is a tuning parameter for adjusting the sparsity level, r is the radius of the PSF, and x is the reconstructed image. is the final reconstructed image;

[0030] Substitute Equation 3 into the ADMM algorithm optimization model:

[0031]

[0032] in g(u)=λ||u||1=λ||Ψx||1;

[0033] The Lagrangian function of formula (10) is:

[0034]

[0035] Among them, α1, α2, and α3 are the Lagrange multipliers corresponding to u, v, and w respectively, and ρ1, ρ2, and ρ3 are Lagrange penalty functions;

[0036] The iterative update equation of formula (11) is as formula (12), where T τ (n) = sgn(n)(n|-τ) + , τ is the vector soft threshold, H represents the conjugate transpose of the matrix, F represents the Fourier transform matrix, and I is the identity matrix;

[0037]

[0038] Each computational iteration is considered as a network layer of the Le-ADMM network. Each layer of the network trains the parameters of the ADMM to integrate the physical model into the network structure.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention uses a lightweight convolutional neural network CNN-ISP-net to preprocess the point spread function and the measured image, making the measured point spread function closer to the ideal point spread function and achieving better image reconstruction. In addition, through the preprocessing stage, the CNN-ISP-net can better adjust and optimize the quality of the original image, realizing four functions: bad pixel correction, automatic exposure, denoising, and image enhancement. This provides more accurate and rich information for the subsequent reconstruction process, thereby improving the quality of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the overall network framework diagram of the present invention;

[0042] Figure 2 This is a diagram of the lens-free imaging model of the diffuser of the present invention;

[0043] Figure 3 This is a diagram of the lightweight convolutional neural network CNN-ISP-net framework of the present invention;

[0044] Figure 4 This is a comparison diagram of the effects of reconstructing point sources in one embodiment of the present invention;

[0045] Figure 5 Reconstructing images for point sources of different sizes in one embodiment of the present invention;

[0046] Figure 6 This is a graph of the PSF autocorrelation function without CNN-ISP-net preprocessing in one embodiment of the present invention;

[0047] Figure 7 This is a graph of the PSF autocorrelation function after CNN-ISP-net preprocessing in one embodiment of the present invention;

[0048] Figure 8 This is a comparison chart of the joint reconstruction effects of U-net and CNN-ISP-net on Le-ADMM in one embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0050] refer to Figure 1 The present invention proposes an end-to-end preprocessing network driven lens-free imaging reconstruction method, comprising the following steps:

[0051] S1, the original image is passed through the diffuser lensless imaging system to obtain the measurement image;

[0052] S2. Build a lightweight convolutional neural network CNN-ISP-net and train it using the training set;

[0053] S3, inputting the measured image and the point spread function into the trained lightweight convolutional neural network for preprocessing to obtain a preprocessed measured image and point spread function;

[0054] S4. Build and train the Le-ADMM network for image reconstruction.

[0055] S5. The preprocessed point spread function and the measured image are input into the trained Le-ADMM network for image reconstruction.

[0056] In this embodiment, the diffuser lensless imaging is adopted, and its model is as follows: Figure 2 ; The imaging process can be divided into two processes. In the first process, P i,j The scene light is irradiated on the diffuser through the spatial sampling plane. In the second process, the light is modulated by the diffuser and irradiated on the sensor, where z is the distance from the image point to the sampling plane, C z is the distance from the sampling plane to the diffuser, and V' is the distance from the diffuser to the image sensor.

[0057] In this lens-less imaging system, the relationship between the measurements b captured by the sensor and the reconstructed image x can be expressed using linear matrix multiplication:

[0058] b=FΦx (1)

[0059] Where the Φ matrix is ​​the transfer function of the point spread function, and F is the diffuser clipping function. This linear model can be approximated as the convolution of the point spread function:

[0060] b=F*h*x (2)

[0061] Where h is the system PSF; here * represents 2D discrete linear convolution. To find the reconstructed image in equation (1), it can be transformed into the following regularized optimization problem:

[0062]

[0063] where Ψ is the sparse transform, λ is a tuning parameter for adjusting the sparsity level, r is the radius of the PSF, and x is the reconstructed image. is the final reconstructed image.

[0064] In this embodiment, the lensless imaging system processes single-channel grayscale images. Therefore, to improve system efficiency and resource utilization, the ISP system was simplified. Color-related functions were removed from the original ISP process, ultimately retaining only four functions: bad pixel correction, automatic exposure, denoising, and image enhancement.

[0065] The proposed lightweight CNN-ISP-net framework is as follows Figure 3As shown in the figure, the framework consists of seven Conv+ReLU layers and one Conv layer. In image processing tasks, pooling layers can reduce image resolution, leading to information loss and hindering bad pixel correction. Therefore, the bad pixel correction network does not use pooling layers. To reduce network complexity, two convolutional layers are used to perform bad pixel correction and auto-exposure. Bilateral filtering is used for noise reduction, and to obtain sufficient information, pooling layers are not used. Therefore, a six-layer Conv+ReLU network is designed in the filtering network to perform convolutional learning. The following is a detailed introduction to the lightweight CNN-ISP-net.

[0066] The point spread function and the measured image are fed into the first layer of the network, which uses 8 filters of size 3×3×1 for convolution in order to extract local features of the image. These features are represented as 8 different feature maps to facilitate image processing in subsequent network layers.

[0067] The eight different feature maps are fed into the Conv+ReLU layers 2-7. Each layer uses eight 3×3×8 filters to perform deeper feature extraction based on the features extracted in the first layer. The output of each layer serves as the input for the next layer. The Conv+ReLU layers 2-7 perform four functions: bad pixel correction, automatic exposure, denoising, and image enhancement.

[0068] The last Conv layer uses a 3×3×8 convolution kernel to integrate the previously extracted features and complete the image reconstruction output.

[0069] The network of the present invention is trained using three training sets: the HDR+ dataset, the SID dataset, and a manually collected and processed point spread function dataset. The three datasets have different characteristics, allowing the network to be trained with different focuses. The HDR+ dataset focuses on denoising and detail enhancement. This dataset captures a series of underexposed natural scenes as raw images, which are then white-balanced and enhanced using DCraw to serve as control images for real scenes. The SDI dataset focuses on denoising in low-light environments. This dataset captures short-exposure, noisy raw images of different scenes, then uses DCraw to restore the underexposed images. The images are then further processed using Photoshop's automatic enhancement tool to achieve an enhanced effect. The processed images serve as control images. The manually collected dataset consists of low-light point spread functions at different distances. Using the aforementioned diffuser lensless imaging system, point spread function images are collected at different distances of 1-10 cm (with 1 cm intervals). At each distance, point sources of different pixel sizes (10 pixels intervals) from 0-70 are used. The collected point spread function images are then preprocessed using the ISP process. The unprocessed point spread function image is used as the original image, and the preprocessed point spread function image is used as the control image.

[0070] The present invention uses the difference between the restored image and the true image in linear and logarithmic space as the loss function, as shown in formula (4):

[0071]

[0072] Among them I r is the restored image, G r is a true image, 0 < ε ≤ 0.05. Since the restored image I is in linear space, its image intensity is proportional to the scene radiance, but not proportional to human visual perception. Therefore, to conform to the subjective perception of the human visual system, a logarithmic loss is introduced into the loss function.

[0073] In this embodiment, Le-ADMM is formed by iteratively expanding the ADMM algorithm. In the network, each computational iteration of the ADMM algorithm is used as a network layer of Le-ADMM. Each network layer can learn and train the parameters of ADMM. In this way, the physical model is integrated into the network structure, thereby improving the efficiency and performance of the network.

[0074] The model optimized by the ADMM algorithm is:

[0075]

[0076] where x∈R n and z∈R nis the variable to be optimized, f(x) and g(z) are convex functions, Ax+Bz=c is the linear constraint, A and B are matrices used to represent the coefficients in the linear constraint, and c is a constant vector or matrix representing the value on the right side of the equality constraint.

[0077] In order to solve this optimization problem, ADMM constructs an augmented Lagrangian function:

[0078]

[0079] where y T is the Lagrange multiplier, and ρ is the Lagrange penalty parameter.

[0080] The iterative steps for solving formula (6) are:

[0081]

[0082] In the formula, the superscript k represents the kth iteration. From formulas (7), (8), and (9), we can see that the ADMM algorithm gradually approaches the optimal solution of the original problem by alternately updating x, z, and the Lagrange multiplier y.

[0083] Substituting formula (3) into formula (5) we get the optimized model:

[0084]

[0085] in g(u)=λ||u||1=λ||Ψx||1. The Lagrangian function of formula (10) is:

[0086]

[0087] Among them, α1, α2, and α3 are the Lagrange multipliers corresponding to u, v, and w respectively, u, v, and w are the variables to be optimized respectively, and ρ1, ρ2, and ρ3 are Lagrange penalty functions.

[0088] The iterative update equation of formula (11) is shown in formula (12), where T τ (n) = sgn(n)(n|-τ) + , τ is the vector soft threshold, H represents the conjugate transpose of the matrix, F represents the Fourier transform matrix, and I is the identity matrix.

[0089]

[0090] In this embodiment, the results analysis and conclusions are as follows:

[0091] This example first verifies the reconstruction effect of CNN-ISP-net on the ADMM algorithm. Figure 4, the comparison results show that the reconstructed image using the CNN-ISP-net network is better. Then use the PSF of different point sources to conduct experiments, and the results are as follows Figure 5 : It can be seen that the PSF processed by CNN-ISP-net shows a higher spatial resolution in the point source reconstruction process. The reconstructed image can more clearly show the details and edge information of the point source, and the contrast between the point source and the background is more obvious, effectively reducing blur and distortion. Figure 6 and Figure 7 From the comparison of the PSF autocorrelation functions in , it can be seen that the peak of the PSF autocorrelation function after CNN-ISP-net preprocessing is more concentrated than that without CNN-ISP-net preprocessing, indicating that the better the PSF concentration, the higher the imaging quality. At the same time, the full width at half maximum of the PSF autocorrelation function is narrower, indicating that the reconstruction of the point source has a more significant improvement in spatial resolution, the detailed information in the image can be presented more accurately, and the noise level is lower.

[0092] The experimental results show that the joint reconstruction effect of CNN-ISP-net and Le-ADMM network is better than that of U-net post-processing. Figure 8 As shown in the figure, both U-net and CNN-ISP-net significantly improve the reconstruction performance of Le-ADMM. However, the present invention uses CNN-ISP-net to correct the PSF image, and the image reconstruction performance is slightly better than the direct denoising of the reconstructed image by U-net. It also shows significant advantages in network architecture and computational parameters, which is of great significance for the convenience and miniaturization of lensless imaging systems.

[0093] In summary, through the preprocessing stage, the CNN-ISP-net of the present invention can better adjust and optimize the quality of the original image, provide more accurate and rich information for the subsequent reconstruction process, and improve the quality of the reconstructed image.

[0094] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. An end-to-end preprocessing network driven lens-free imaging reconstruction method, characterized in that: The following steps are involved: S1, the original image is passed through the diffuser lensless imaging system to obtain the measured image; S2. Build a lightweight convolutional neural network CNN-ISP-net and train it using the training set; S3, inputting the measured image and the point spread function into the trained lightweight convolutional neural network for preprocessing, to obtain the preprocessed measured image and point spread function; S4. Build and train the Le-ADMM network for image reconstruction. S5. The preprocessed point spread function and the measured image are input into the trained Le-ADMM network for image reconstruction.

2. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 1, characterized in that: In the diffuser lensless imaging system, the relationship between the measurement value b captured by the sensor and the reconstructed image x is expressed as: b=FΦx (1) The Φ matrix is ​​the transmission function composed of the point spread function, and F is the diffuser clipping function.

3. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 1, characterized in that: The lightweight convolutional neural network CNN-ISP-net includes 7 Conv+ReLU layers and 1 Conv layer, and the output of each layer of the network serves as the input of the next layer.

4. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 3, characterized in that: The first layer of the lightweight convolutional neural network CNN-ISP-net uses 8 filters of size 3×3×1 for convolution to extract local features; each layer of the 2nd to 7th layers uses 8 filters of size 3×3×8 for convolution to perform deeper feature extraction on the features extracted in the first layer; the 8th layer uses a convolution kernel of size 3×3×8 to integrate the previously extracted features.

5. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 1, characterized in that: The lightweight convolutional neural network CNN-ISP-net is trained using three training sets, including an HDR+ dataset, a SID dataset, and a manually collected point spread function dataset.

6. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 5, characterized in that: The HDR+ dataset is constructed by capturing a series of underexposed natural scenes as original images, and then using DCraw to white balance and enhance the original images as control images of the original images.

7. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 5, characterized in that: The construction of the SDI dataset is specifically to collect short-exposure and noisy original images of different scenes, then use DCraw to restore the underexposed images, and use Photoshop's automatic enhancement tool to process the images to achieve an enhanced effect, and use the processed images as control images.

8. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 5, characterized in that: The manually collected and processed point spread function data set is composed of low-light point spread functions at different distances. The diffuser lensless imaging system is used to collect point spread function images at different distances of 1-10 cm with an interval of 1 cm. Point sources of different pixel sizes of 0-70 are used at an interval of 10 pixels at each distance. The collected point spread function images are then preprocessed by the ISP process. The unprocessed point spread function image is used as the original image, and the preprocessed point spread function image is used as the control image.

9. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 1, characterized in that: The lightweight convolutional neural network CNN-ISP-net uses the difference between the restored image and the real image in linear and logarithmic space as the loss function: Among them I r is the restored image, G r is a real image, ε is a very small value, and 0<ε≤0.

05.

10. The end-to-end preprocessing network driven lens-free imaging reconstruction method according to claim 1, characterized in that: The S4 is specifically: Convert the solution of reconstructed image into a regularized optimization problem: where Ψ is the sparse transform, λ is a tuning parameter for adjusting the sparsity level, r is the radius of the PSF, and x is the reconstructed image. is the final reconstructed image; Substitute formula 3 into the ADMM algorithm optimization model: among them g(u)=λ||u||1=λ||Ψx||1; The Lagrangian function of formula (10) is: Among them, α1, α2, α3 are the Lagrange multipliers corresponding to u, v, and w respectively, and ρ1, ρ2, ρ3 are Lagrange penalty functions; The iterative update equation of formula (11) is as follows: τ (n) = sgn(n)(n|-τ) + , τ is the vector soft threshold, H represents the conjugate transpose of the matrix, F represents the Fourier transform matrix, and I is the identity matrix; Each calculation iteration is regarded as a network layer of the Le-ADMM network, and each layer of the network learns and trains the parameters of ADMM to integrate the physical model into the network structure.

Images