An end-to-end pre-processing network driven lensless imaging reconstruction method

Abstract

The application provides an end-to-end preprocessing network driven lensless imaging reconstruction method, comprising the following steps: S1, obtaining a measurement image through a diffuser lensless imaging system from an original image; S2, constructing a light-weight convolutional neural network CNN-ISP-net and training the light-weight convolutional neural network CNN-ISP-net using a training set; S3, inputting the measurement image and a point spread function into the trained light-weight convolutional neural network for preprocessing to obtain a preprocessed measurement image and a preprocessed point spread function; S4, constructing and training a Le-ADMM network for image reconstruction; and S5, jointly inputting the preprocessed point spread function and the measurement image into the trained Le-ADMM network for image reconstruction. The light-weight convolutional neural network CNN-ISP-net is used for preprocessing the measurement image and the measurement point spread function respectively, the quality of the measurement image is optimized, the measurement point spread function is closer to an ideal point spread function, and the purpose of improving the effect of a reconstructed image is achieved.

Description

Technical Field

[0001] This invention relates to the field of lensless imaging technology, and more specifically to a lensless imaging reconstruction method driven by an end-to-end preprocessing network. Background Technology

[0002] Lensless imaging is an emerging imaging technology that replaces lenses with computation, and it is more compact and convenient than traditional imaging systems.

[0003] Patent application CN112950750A proposes a lensless camera image reconstruction method based on coded masks and the Learned-TSVD algorithm. The training method for this image preprocessing network includes: performing JPEG compression on the original images in the training data image set using different JPEG quantization factors; calculating the bitrate of the preprocessed image using the preprocessed image from the original image preprocessing network and a bitrate estimation network using a differentiable JPEG simulator; reconstructing the original image using the preprocessed result from the differentiable JPEG simulator and calculating the distortion between the reconstructed image and the corresponding original image; obtaining the rate-distortion loss value using a predefined rate-distortion loss function and adjusting the parameters of the image preprocessing network based on the rate-distortion loss value; adjusting the parameters of the bitrate estimation network; and adjusting the parameters of the image preprocessing network and the bitrate estimation network through alternating iterations to obtain the trained image preprocessing network and bitrate estimation network.

[0004] Patent application CN112950750A proposes a lensless camera image reconstruction method based on coded masks and the Learned-TSVD algorithm. It uses coded masks to encode the light propagation process and leverages the separability of the coded masks and the TSVD algorithm to transform the original large-scale system measurement matrix into two smaller left and right system measurement matrices. Next, a neural network is constructed and trained iteratively on the left and right system measurement matrices to reduce the error caused by approximation calculations in the final result. Finally, the image is reconstructed using the TSVD algorithm and a regularization algorithm. This invention uses the learned system measurement matrix for subsequent calculations, making the overall reconstruction process more robust to noise. The learned system measurement matrix can also effectively reconstruct scene images at other distances, solving the problem of low depth of field in other reconstruction algorithms.

[0005] However, lensless imaging is currently affected by the non-ideal characteristics of CMOS sensors, and the point spread function (PSF) of lensless imaging systems suffers from problems such as blurring and distortion.

[0006] Therefore, to address the above problems, this invention proposes an end-to-end preprocessing network-driven lensless imaging reconstruction method. Based on Image Signal Processing (ISP), a lightweight fully convolutional neural network is designed. This network learns the error between the measured point spread function caustic image and the ideal point spread function caustic image, thereby compensating for the measured point spread function. The processed point spread function is then used for image reconstruction. The reconstruction network framework adopted in this invention is a parameter-learnable ADMM network, specifically the Le-ADMM network. Summary of the Invention

[0007] The purpose of this invention is to propose an end-to-end preprocessing network-driven lensless imaging reconstruction method. Addressing the issues of noise, underexposure, and lack of detail in reconstructed images caused by using CMOS sensors to acquire PSF data in lensless imaging systems, a lightweight convolutional neural network CNN-ISP-net based on the ISP process is designed. Through an image preprocessing network, the measured point spread function is made closer to the ideal point spread function, thereby improving the quality of the reconstructed image.

[0008] To achieve the above objectives, the technical solution of the present invention is: an end-to-end preprocessing network-driven lensless imaging reconstruction method, comprising the following steps:

[0009] S1. The original image is processed by a diffuser lensless imaging system to obtain a measurement image;

[0010] S2. Construct a lightweight convolutional neural network CNN-ISP-net and train it using the training set;

[0011] S3. Input the measurement image and point spread function into the trained lightweight convolutional neural network for preprocessing to obtain the preprocessed measurement image and point spread function;

[0012] S4. Construct and train the Le-ADMM network for image reconstruction.

[0013] S5. Input the preprocessed point spread function and the measured image into the trained Le-ADMM network for image reconstruction.

[0014] Preferably, in the diffuser lensless imaging system, the relationship between the measured value b captured by the sensor and the reconstructed image x is expressed as:

[0015] b=FΦx (1)

[0016] Where the Φ matrix is ​​the transfer function composed of point diffusion functions, and F is the diffuse trimming function.

[0017] Preferably, the lightweight convolutional neural network CNN-ISP-net includes 7 Conv+ReLU layers and 1 Conv layer, with the output of each layer serving as the input to the next layer.

[0018] Preferably, the first layer of the lightweight convolutional neural network CNN-ISP-net uses eight 3×3×1 filters for convolution to extract local features; the second to seventh layers each use eight 3×3×8 filters for convolution to perform deeper feature extraction on the features extracted in the first layer; the eighth layer uses a 3×3×8 convolution kernel to integrate the previously extracted features.

[0019] Preferably, the lightweight convolutional neural network CNN-ISP-net is trained using three training sets, including the HDR+ dataset, the SID dataset, and a manually acquired point spread function dataset.

[0020] Preferably, the HDR+ dataset is constructed by capturing a series of underexposed natural scenes as original images, and then using DCraw to perform white balance and enhancement on the original images as a reference image.

[0021] Preferably, the construction of the SDI dataset specifically involves acquiring short-exposure and noisy original images of different scenes, then using DCraw to restore the underexposed images, and using Photoshop's automatic enhancement tools to process the images to achieve an enhancement effect, with the processed images serving as control images.

[0022] Preferably, the manually acquired point spread function dataset consists of low-light point spread functions at different distances. Using the lensless imaging system of the diffuser, point spread function images are acquired at different distances of 1-10cm with 1cm intervals. Point sources of different pixel sizes (0-70) are used at 10-pixel intervals at each distance. The acquired point spread function images are then preprocessed using the ISP (In-Screen Image Processing) procedure. The unprocessed point spread function image is used as the original image, and the preprocessed point spread function image is used as the control image.

[0023] Preferably, the lightweight convolutional neural network CNN-ISP-net uses the difference between the recovered image and the real image in linear and logarithmic space as the loss function:

[0024]

[0025] Where I r It is the restored image, G r It is a real image, ε is a very small value, and 0 < ε ≤ 0.05.

[0026] Preferably, S4 specifically includes:

[0027] The problem of reconstructing the image is transformed into a regularization optimization problem:

[0028]

[0029] Where Ψ is the sparse transform, λ is the adjustment parameter for adjusting the sparsity level, r is the radius of the PSF, and x is the reconstructed image. It is the final reconstructed image;

[0030] Substituting Equation 3 into the ADMM algorithm optimization model:

[0031]

[0032] in g(u)=λ||u||1=λ||Ψx||1;

[0033] The Lagrange function of equation (10) is:

[0034]

[0035] Where α1, α2, and α3 are the Lagrange multipliers corresponding to u, v, and w, respectively, and ρ1, ρ2, and ρ3 are the Lagrange penalty functions;

[0036] The iterative update equation of equation (11) is as shown in equation (12), where T τ (n)=sgn(n)(n|-τ) + τ is the vector soft threshold, H represents the conjugate transpose of the matrix, F represents the Fourier transform matrix, and I is the identity matrix;

[0037]

[0038] Each computational iteration is treated as a layer in the Le-ADMM network, and each layer is trained to learn the parameters of ADMM in order to incorporate the physical model into the network structure.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] This invention uses a lightweight convolutional neural network, CNN-ISP-net, to preprocess the point spread function and the measured image, making the measured point spread function closer to the ideal point spread function and achieving better image reconstruction. Furthermore, through the preprocessing stage, CNN-ISP-net can better adjust and optimize the quality of the original image, realizing four functions: bad pixel correction, automatic exposure, noise reduction, and image enhancement. This provides more accurate and richer information for the subsequent reconstruction process, improving the quality of the reconstructed image. Attached Figure Description

[0041] Figure 1 This is a diagram of the overall network framework of the present invention;

[0042] Figure 2 This is a model diagram of the lensless imaging system of the diffuser of the present invention;

[0043] Figure 3 This is a diagram of the lightweight convolutional neural network CNN-ISP-net framework of the present invention;

[0044] Figure 4 This is a comparison diagram of the reconstructed point source effect in one embodiment of the present invention;

[0045] Figure 5 These are reconstructed images of point sources of different sizes in one embodiment of the present invention;

[0046] Figure 6 This is a PSF autocorrelation function graph in one embodiment of the present invention without CNN-ISP-net preprocessing;

[0047] Figure 7 This is a graph of the PSF autocorrelation function after preprocessing CNN-ISP-net in one embodiment of the present invention;

[0048] Figure 8 This is a comparison chart of the joint reconstruction effects of U-net and CNN-ISP-net on Le-ADMM in one embodiment of the present invention. Detailed Implementation

[0049] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0050] refer to Figure 1 This invention proposes an end-to-end preprocessing network-driven lensless imaging reconstruction method, comprising the following steps:

[0051] S1. The original image is processed by a diffuser lensless imaging system to obtain a measurement image;

[0052] S2. Construct a lightweight convolutional neural network CNN-ISP-net and train it using the training set;

[0053] S3. Input the measurement image and point spread function into the trained lightweight convolutional neural network for preprocessing to obtain the preprocessed measurement image and point spread function;

[0054] S4. Construct and train the Le-ADMM network for image reconstruction.

[0055] S5. Input the preprocessed point spread function and the measured image into the trained Le-ADMM network for image reconstruction.

[0056] In this embodiment, diffuser-based lensless imaging is used, and its model is as follows: Figure 2 The imaging process can be divided into two stages. In the first stage, P... i,j Scene light passes through the spatial sampling plane and illuminates the diffuser. In the second process, the light is modulated by the diffuser and illuminates the sensor, where z is the distance from the image point to the sampling plane, and C... z V' is the distance from the sampling plane to the diffuser, and V' is the distance from the diffuser to the image sensor.

[0057] In this lensless imaging system, linear matrix multiplication can be used to represent the relationship between the sensor-captured measurement value b and the reconstructed image x:

[0058] b=FΦx (1)

[0059] Where the Φ matrix represents the transfer function composed of point spread functions, and F is the diffuse trimming function. This linear model can be approximated as the convolution of point spread functions:

[0060] b=F*h*x (2)

[0061] Where h is the system PSF; here * denotes 2D discrete linear convolution. To find the reconstructed image in equation (1), it can be transformed into the following regularization optimization problem:

[0062]

[0063] Where Ψ is the sparse transform, λ is the adjustment parameter for adjusting the sparsity level, r is the radius of the PSF, and x is the reconstructed image. This is the final reconstructed image.

[0064] In this embodiment, the lensless imaging system processes single-channel grayscale images. Therefore, to improve system efficiency and resource utilization, the ISP system is simplified. Color-related functions are removed from the original ISP workflow, ultimately retaining only four functions: bad pixel correction, automatic exposure, noise reduction, and image enhancement.

[0065] The proposed lightweight CNN-ISP-net framework, such as Figure 3As shown, the framework consists of 7 Conv+ReLU layers and 1 Conv layer. In image processing tasks, pooling layers may reduce image resolution, leading to information loss and hindering bad pixel correction; therefore, the bad pixel correction network does not use pooling layers. To reduce network complexity, 2 convolutional layers are used for both bad pixel correction and automatic exposure. Bilateral filtering is used for noise reduction, and to obtain sufficient information, pooling layers are also avoided. Therefore, a 6-layer Conv+ReLU network is designed in the filtering network to complete convolutional learning. The following is a detailed introduction to the lightweight CNN-ISP-net.

[0066] The point spread function and the measured image are fed into the first layer of the network. This layer uses eight 3×3×1 filters for convolution to extract local features of the image. These features are represented as eight different feature maps, which facilitate image processing in subsequent network layers.

[0067] Eight different feature maps are fed into Conv+ReLU layers 2-7. Each of layers 2-7 uses eight 3×3×8 filters to perform deeper feature extraction on the features extracted in the first layer. The output of each layer serves as the input for the next layer. The Conv+ReLU layers 2-7 perform four functions: bad pixel correction, automatic exposure, noise reduction, and image enhancement.

[0068] The final Conv layer uses a 3×3×8 convolutional kernel to integrate the previously extracted features, completing the image reconstruction output.

[0069] The network of this invention is trained using three training sets: the HDR+ dataset, the SID dataset, and a manually acquired and processed point spread function (PSF) dataset. These three datasets have different characteristics, allowing for different training focuses. The HDR+ dataset focuses on denoising and detail enhancement. This dataset captures a series of underexposed natural scenes as raw images, then uses DCraw to perform white balance and enhancement on the raw images, serving as a control image for real-world scenes. The SDI dataset focuses on denoising in low-light environments. This dataset captures short-exposure and noisy raw images of different scenes, then uses DCraw to restore the underexposed images, followed by further processing using Photoshop's automatic enhancement tools to achieve an enhanced effect. The processed images serve as a control image. The manually acquired dataset consists of low-light PSF images at different distances. Using the aforementioned diffuser lensless imaging system, PSF images are acquired at different distances (1-10 cm intervals), using point sources of 0-70 pixels (10 pixels intervals) at each distance. The acquired PSF images are then preprocessed using an ISP (In-Screen Preprocessing) workflow. The unprocessed point spread function image is used as the original image, and the preprocessed point spread function image is used as the control image.

[0070] This invention uses the difference between the recovered image and the real image in the linear and logarithmic spaces as the loss function, as shown in equation (4).

[0071]

[0072] Where I r It is the restored image, G r The image is a real image, where 0 < ε ≤ 0.05. Since the recovered image I lies in a linear space, its image intensity is proportional to the scene's radiance, but not proportional to human visual perception. Therefore, to conform to the subjective perception of the human visual system, a logarithmic loss is introduced into the loss function.

[0073] In this embodiment, Le-ADMM is derived from the iterative expansion of the ADMM algorithm. In the network, each computational iteration of the ADMM algorithm is used as a network layer of Le-ADMM. Each layer of the network can learn and train the parameters of ADMM. In this way, the physical model is integrated into the network structure, thereby improving the efficiency and performance of the network.

[0074] The model optimized by the ADMM algorithm is:

[0075]

[0076] where x∈R n and z∈R nLet f(x) and g(z) be the variables to be optimized, f(x) and g(z) be convex functions, Ax + Bz = c be the linear constraint, A and B are matrices used to represent the coefficients in the linear constraint, and c is a constant vector or matrix representing the right-hand side value of the equality constraint.

[0077] To solve this optimization problem, ADMM constructs the augmented Lagrangian function:

[0078]

[0079] Where y T ρ is the Lagrange multiplier, and ρ is the Lagrange penalty parameter.

[0080] The iterative steps for solving equation (6) are as follows:

[0081]

[0082] In the formula, the superscript k indicates the k-th iteration. As can be seen from equations (7), (8), and (9), the ADMM algorithm gradually approaches the optimal solution of the original problem by alternately updating x, z, and the Lagrange multiplier y.

[0083] Substituting equation (3) into equation (5) yields the optimized model:

[0084]

[0085] in g(u)=λ||u||1=λ||Ψx||1. The Lagrangian function of equation (10) is:

[0086]

[0087] Where α1, α2, and α3 are the Lagrange multipliers corresponding to u, v, and w, respectively, u, v, and w are the variables to be optimized, and ρ1, ρ2, and ρ3 are the Lagrange penalty functions.

[0088] The iterative update equation of equation (11) is shown in equation (12), where T τ (n)=sgn(n)(n|-τ) + τ is the vector soft threshold, H represents the conjugate transpose of the matrix, F represents the Fourier transform matrix, and I is the identity matrix.

[0089]

[0090] In this embodiment, the results analysis and conclusions are as follows:

[0091] This embodiment first verifies the reconstruction effect of CNN-ISP-net on the ADMM algorithm, such as Figure 4The comparison results clearly show that the image reconstruction effect using the CNN-ISP-net network is better. Further experiments using PSF with different point sources yielded the following results: Figure 5 It can be seen that the PSF processed by CNN-ISP-net exhibits higher spatial resolution in the point source reconstruction process. The reconstructed image can more clearly present the details and edge information of the point source, and the contrast between the point source and the background is more obvious, effectively reducing blurring and distortion. Meanwhile, from... Figure 6 and Figure 7 The comparison of PSF autocorrelation functions shows that the peak value of the PSF autocorrelation function after using CNN-ISP-net preprocessing is more concentrated than that without it, indicating that the better the PSF concentration, the higher the image quality. At the same time, the full width at half maximum (FWHM) of the PSF autocorrelation function is narrower, indicating that the reconstruction of point sources has a more significant improvement in spatial resolution, and the details in the image can be presented more accurately, while having a lower noise level.

[0092] The joint reconstruction effect of CNN-ISP-net and Le-ADMM network was obtained through experiments, and compared with the post-processing effect of U-net. The experimental results are as follows: Figure 8 As shown, both U-net and CNN-ISP-net significantly improve the reconstruction performance of Le-ADMM. However, this invention uses CNN-ISP-net to correct PSF images, and its image reconstruction performance is slightly better than U-net's direct denoising of reconstructed images. Furthermore, it exhibits significant advantages in network architecture and computational parameters, which is of great significance for the convenience and miniaturization of lensless imaging systems.

[0093] In summary, through the preprocessing stage, CNN-ISP-net can better adjust and optimize the quality of the original image, providing more accurate and richer information for the subsequent reconstruction process and improving the quality of the reconstructed image.

[0094] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A lensless imaging reconstruction method driven by an end-to-end preprocessing network, characterized in that, Includes the following steps: S1. The original image is processed by a diffuser lensless imaging system to obtain a measurement image; S2. Construct a lightweight convolutional neural network CNN-ISP-net and train it using the training set; S3. Input the measurement image and point spread function into the trained lightweight convolutional neural network for preprocessing to obtain the preprocessed measurement image and point spread function; S4. Construct and train the Le-ADMM network for image reconstruction. S5. Input the preprocessed point spread function and the measured image into the trained Le-ADMM network for image reconstruction. In the diffuser lensless imaging system, the relationship between the measured value b captured by the sensor and the reconstructed image x is expressed as: b=FΦx (1) Where the Φ matrix is ​​the transfer function composed of point diffusion functions, and F is the diffuse volume trimming function; The lightweight convolutional neural network CNN-ISP-net consists of 7 Conv+ReLU layers and 1 Conv layer, with the output of each layer serving as the input to the next layer. The lightweight convolutional neural network CNN-ISP-net uses eight 3×3×1 filters for convolution in its first layer to extract local features; layers 2-7 each use eight 3×3×8 filters for convolution to extract deeper features based on the features extracted in the first layer; and the eighth layer uses a 3×3×8 convolution kernel to integrate the features extracted in the previous layers. The lightweight convolutional neural network CNN-ISP-net is trained using three training sets, including the HDR+ dataset, the SID dataset, and a manually acquired point spread function dataset. The HDR+ dataset is constructed by capturing a series of underexposed natural scenes as raw images, and then using DCraw to perform white balance and enhancement on the raw images as a reference image.

2. The end-to-end preprocessing network-driven lensless imaging reconstruction method according to claim 1, characterized in that, The construction of the SID dataset specifically involves collecting short-exposure and noisy original images of different scenes, then using DCraw to restore the underexposed images, and using Photoshop's automatic enhancement tools to process the images to achieve an enhancement effect. The processed images are then used as control images.

3. The end-to-end preprocessing network-driven lensless imaging reconstruction method according to claim 1, characterized in that, The manually acquired point spread function dataset consists of low-light point spread functions at different distances. Using the lensless imaging system of the diffuser, point spread function images are acquired at different distances of 1-10cm with 1cm intervals. Point sources of different pixel sizes (0-70 pixels) are used at 10-pixel intervals at each distance. The acquired point spread function images are then preprocessed using the ISP (In-Screen Imaging) process. The unprocessed point spread function image is used as the original image, and the preprocessed point spread function image is used as the control image.

4. The end-to-end preprocessing network-driven lensless imaging reconstruction method according to claim 1, characterized in that, The lightweight convolutional neural network CNN-ISP-net uses the difference between the recovered image and the real image in both linear and logarithmic space as the loss function: Where I r It is the restored image, G r It is a true image, 0 < ε ≤ 0.

05.

5. The end-to-end preprocessing network-driven lensless imaging reconstruction method according to claim 1, characterized in that, Specifically, S4 is: The problem of reconstructing the image is transformed into a regularization optimization problem: Where Ψ is the sparse transform, λ is the adjustment parameter for adjusting the sparsity level, r is the radius of the PSF, and x is the reconstructed image. It is the final reconstructed image; Substituting Equation 3 into the ADMM algorithm optimization model: among them g(u)=λ||u||1=λ||Ψx||1; The Lagrange function of equation (10) is: Where α1, α2, and α3 are the Lagrange multipliers corresponding to u, v, and w, respectively, and ρ1, ρ2, and ρ3 are the Lagrange penalty functions; The iterative update equation of equation (11) is as shown in equation (12), where T τ (n)=sgn(n)(|n|-τ) + τ is the vector soft threshold, H represents the conjugate transpose of the matrix, F represents the Fourier transform matrix, and I is the identity matrix; Each computational iteration is treated as a layer in the Le-ADMM network, and each layer learns and trains the parameters of ADMM to incorporate the physical model into the network structure.

Images