Two-stage lens-free camera imaging method and lens-free camera

Abstract

The invention relates to a two-stage lensless camera imaging method and a lensless camera, and the method comprises the following steps: obtaining an image collected by the lensless camera, inputting the image into a pre-trained space-variant deconvolution network, generating a plurality of different deconvolution intermediate results by the space-variant deconvolution network according to the input image, and outputting the deconvolution intermediate results to the lensless camera; performing weighted synthesis to obtain a final target deconvolution result as a first-stage recovery result; and inputting the first-stage recovery result into a pre-trained conditional diffusion model, performing visual quality enhancement, and obtaining a second-stage recovery result as a final lens-free camera imaging result. Compared with the prior art, the imaging quality of the lens-free camera is effectively improved, so that the imaging effect of the lens-free camera can be close to that of a lens camera; in an application scene where the size and cost of the camera are limited, the scheme of the invention is expected to replace a traditional lens camera to carry out high-quality imaging.

Description

Technical Field

[0001] The present invention relates to the technical field of lensless cameras, and in particular to a two-stage lensless camera imaging method and a lensless camera. Background Art

[0002] Lensless camera is an emerging imaging system that replaces traditional lenses by placing amplitude or phase masks near the sensor to achieve a highly integrated, lightweight and low-cost imaging solution. However, due to the lack of a lens that can focus light, the images directly captured by the lensless camera are very blurry and unrecognizable. In order to restore the original captured scene from the image captured by the lensless camera, researchers have proposed a series of scene reconstruction algorithms for lensless cameras. Generally speaking, previous work approximates the lensless imaging process as a process of convolution between the point spread function of a lensless imaging system and the original scene. Therefore, traditional methods such as Wiener deconvolution attempt to reconstruct images through deconvolution. Based on Wiener deconvolution, the recent deep learning-based method FlatNet attempts to enhance the visual quality of imaging results by training an end-to-end lensless imaging model using paired scene graphs and lensless imaging results.

[0003] Existing lensless imaging methods mainly include two disadvantages: poor consistency between the reconstructed image and the original scene, and poor visual quality of the reconstructed image. First, the lack of consistency between the reconstructed image and the original scene is due to the inaccurate modeling of the lensless imaging process by the current methods. Current methods mostly use a translation-invariant convolution model to model the lensless imaging process, that is, current work believes that the point spread functions generated by light incident from different angles are the same, and then a deconvolution module can be used to restore the original scene. However, considering the diffraction effect of light, the translation-invariant convolution model is an approximation of the real lensless imaging process, that is, the point spread functions generated by incident light at different angles will be slightly different, which will cause the scene restored by the deconvolution module to be inconsistent with the original scene.

[0004] Secondly, the visual quality of current lensless reconstructed images is poor. This is because existing methods only rely on the original signal to obtain the reconstructed image, resulting in the irreparable loss of high-frequency details caused by filtering and noise during the lensless imaging process. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a two-stage lensless camera imaging method and a lensless camera with better restoration effect.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A two-stage lensless camera imaging method comprises the following steps:

[0008] The image captured by the lensless camera is obtained and input into a pre-trained space-variant deconvolution network. The space-variant deconvolution network generates multiple different deconvolution intermediate results according to the input image, and weights and synthesizes them into the final target deconvolution result as the first-stage restoration result.

[0009] The first-stage restoration result is input into a pre-trained conditional diffusion model for visual quality enhancement to obtain the second-stage restoration result as the final lens-free camera imaging result.

[0010] Furthermore, the space-variant deconvolution network includes K×K learnable deconvolution kernels, and generates K×K deconvolution intermediate results accordingly, and the K×K deconvolution intermediate results respectively correspond to the K×K grids into which the target deconvolution result is divided.

[0011] Furthermore, the expression of the generation process of the deconvolution intermediate result is:

[0012]

[0013] Where y is the input image, F and F -1 represent discrete Fourier transform and its inverse transform, respectively. is the ith deconvolution intermediate result, p (i) is the i-th learnable deconvolution kernel.

[0014] Furthermore, the synthetic expression of the target deconvolution result is:

[0015]

[0016]

[0017] d i (u,v)=(uu i ) 2 +(vv i ) 2 .

[0018] In the formula, x int (u, v) is the deconvolution synthesis result corresponding to the image coordinate (u, v), w i (u, v) is the spatial weight of the i-th deconvolution result, is the ith deconvolution intermediate result, (u i , v i ) is the center of the i-th grid, d i (u, v) is the distance from coordinates u, v to the i-th grid.

[0019] Furthermore, the conditional diffusion model includes a pre-trained image diffusion model.

[0020] Furthermore, the conditional diffusion model also includes a conditional control module, which uses the first stage restoration result as a condition to add high-frequency details to the output result of the image diffusion model on the basis of ensuring structural consistency.

[0021] Furthermore, the method further comprises: during the training process of the space-variant deconvolution network, obtaining an original scene image corresponding to the input image of the lensless camera, constructing a reconstruction loss of the first-stage restoration result, and thus training the space-variant deconvolution network;

[0022] During the training process of the conditional diffusion model, the original scene image corresponding to the restoration result of the first stage is also obtained, and the reconstruction loss of the restoration result of the second stage is constructed, so as to train the conditional diffusion model.

[0023] Furthermore, the method also includes: directly inputting the final lens-free camera imaging result obtained into a lens-free face verification model for face verification.

[0024] The present invention also provides a lens-less camera, which performs imaging by using a two-stage lens-less camera imaging method as described in any one of claims 1 to 8.

[0025] Furthermore, the lensless camera is a lensless camera or an imaging device using a diffractive optical element.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] (1) The present invention achieves high visual quality and high restoration of the original scene through a two-stage lensless imaging method. Compared with the best existing technology, the present invention more accurately models the lensless imaging process, introduces a space-variant deconvolution network, and learns the different point spread function patterns generated by different incident angles of light in a data-driven manner, thereby achieving a better restoration effect on the original captured scene. In addition, the present invention introduces a conditional diffusion model based on generative prior as a generative prior to restore the high-frequency details lost in the lensless imaging process, thereby obtaining a scene reconstruction result with better visual quality than previous methods.

[0028] (2) The present invention also trains a conditional diffusion model based on the pre-trained diffusion model. The model takes the lens-free imaging results of the first stage as conditions, adds high-frequency details on the basis of ensuring structural consistency, and thus obtains high-quality lens-free reconstruction results.

[0029] (3) The method of the present invention can be combined with the lens-free face verification model to perform end-to-end optimization, thereby improving the accuracy of the lens-free face verification system. At the same time, the present invention directly verifies the face of the lens-free face imaging results, which enhances the privacy protection effect of the lens-free face verification system compared with the method of first reconstruction and then recognition. In addition, the lens-free face recognition system obtained by the end-to-end optimization method proposed by the present invention has a certain robustness for faces at different positions and angles in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic flow chart of a two-stage lensless camera imaging method provided in an embodiment of the present invention;

[0031] Figure 2 A flow chart showing image processing result changes of a two-stage lensless camera imaging method provided in an embodiment of the present invention;

[0032] Figure 3 This is a comparison chart of results of a two-stage lensless camera imaging method provided in an embodiment of the present invention and other lensless imaging methods. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0034] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0036] Example 1

[0037] like Figure 1 As shown, this embodiment provides a two-stage lensless camera imaging method, comprising the following steps:

[0038] S1: Acquire images collected by a lensless camera;

[0039] S2: Input the acquired image into a pre-trained space-variant deconvolution network, which generates multiple different deconvolution intermediate results according to the input image, and weights and synthesizes them into the final target deconvolution result as the first stage restoration result;

[0040] S3: The first stage restoration result is input into the pre-trained conditional diffusion model for visual quality enhancement to obtain the second stage restoration result as the final lens-free camera imaging result.

[0041] In other words, in the first stage, this method aims to restore the structural content of the original scene from the shooting results of the lensless camera, that is, to restore the original scene that can ensure data consistency. In order to improve data consistency, a self-learning space-variant deconvolution method is proposed, which can learn the different point spread function patterns generated by different incident angles of light in a data-driven way, and thus can restore the original scene more accurately.

[0042] In the second stage, in order to improve the visual quality of the generated image, the method uses a pre-trained diffusion model as a generative prior to restore the high-frequency details lost in the lensless imaging process. Preferably, the method further trains a conditional diffusion model based on the pre-trained diffusion model, which uses the lensless imaging results of the first stage as a condition to add high-frequency details while ensuring structural consistency, thereby obtaining high-quality lensless reconstruction results.

[0043] Specifically, Figure 2 As shown, the first stage of this method uses a space-variant deconvolution network to convert the input into an output containing the low-frequency structural information of the original scene. The output image restores the content of the original scene as accurately as possible, ensuring the consistency between the lens-free imaging result and the original scene, but its image visual quality is low.

[0044] That is, the space-variant deconvolution network converts the lens-free output into the original scene with low-frequency structural information. It has K×K learnable deconvolution kernels. After applying these deconvolution kernels to the input image taken by the lens-free camera, K×K different deconvolution intermediate results can be obtained; then these K×K different deconvolution intermediate results are used to synthesize the final target deconvolution result.

[0045] If the target deconvolution result is divided into K×K grids, then the K×K deconvolution intermediate results correspond to different K×K grids in the target deconvolution result. The deconvolution here is implemented in the frequency domain. Specifically, the deconvolution process can be expressed as follows:

[0046]

[0047] Where y is the input image, F and F -1 represent discrete Fourier transform and its inverse transform, respectively. is the ith deconvolution intermediate result, p (i) is the i-th learnable deconvolution kernel.

[0048] Then the K×K deconvolution intermediate results are synthesized into the final target deconvolution result x int (u, v):

[0049]

[0050] Where u, v represent image coordinates, and w i (u, v) represents the spatial weight of the i-th deconvolution result, specifically:

[0051]

[0052] d i (u,v)=(uu i ) 2 +(vv i ) 2 .

[0053] In the formula, (u i , v i ) is the center of the i-th grid, d i (u, v) is the distance from coordinates u, v to the i-th grid.

[0054] Preferably, after obtaining the target deconvolution result x int After (u, v), in order to achieve efficient training of the space-variant deconvolution network, this method also obtains the original scene image corresponding to the lens-free camera input image, constructs the reconstruction loss of the first-stage restoration result, and thus trains the space-variant deconvolution network.

[0055] Specifically, this method introduces an image mapping network U-Net to perform subsequent denoising and enhancement on the deconvolution results. A set of paired images taken by a lens-free camera and their corresponding original scenes are used to train the space-variant deconvolution network end-to-end to achieve the purpose of restoring the original scene structure information from the lens-free image.

[0056] In the second stage, this method trains a conditional diffusion model with generative priors, which takes the output of the first stage as conditional input and outputs the final lens-free imaging result. The lens-free imaging result improves the visual quality of the output of the first stage and maintains the same consistency with the original scene as the output of the first stage.

[0057] It should be noted that in order for a general image diffusion model to be able to restore natural images according to conditions, some adjustments need to be made to the model. First, a pre-trained image diffusion model, such as StableDiffusion, needs to be prepared. Then, an additional conditional control module needs to be trained. During the training process, only this conditional control module is trained, and the weights of the original pre-trained image diffusion model are not modified. In this way, a model that can generate images according to conditions can be obtained while retaining the generative priors of the original pre-trained model. The conditional diffusion model used in the present invention can be a variety of different diffusion model structures, such as ControlNet and StableSR.

[0058] In summary, in the first stage, this method uses a space-variant convolutional network to decode the input image taken by the lensless camera into an image containing the low-frequency structure of the original scene, and then in the second stage, this method uses a conditional diffusion model to utilize the low-frequency structure of the original scene to restore the original scene with higher visual quality.

[0059] Preferably, the method also includes: directly inputting the final lensless camera imaging result into a lensless face verification model for face verification, which enhances the privacy protection effect of the lensless face verification system compared to the method of first reconstruction and then recognition.

[0060] like Figure 3 The results are shown in Figure 1, which are the comparison results of this method with other different lens-free imaging methods, where: Figure 3 Among them, (a) is the image taken by a lensless camera, (b) is the image after Wiener deconvolution, (c) is the imaging result of the most advanced method FlatNet, (d) is the imaging result of combining the output of FlatNet with the generative prior DiffBIR, (e) is the imaging result of the method proposed in the present invention, and (f) is the real result of the original scene; it can be seen that the imaging result of the method proposed in the present invention has a better restoration effect.

[0061] Example 2

[0062] This embodiment provides a lensless camera, which uses a two-stage lensless camera imaging method as described in Embodiment 1 to perform imaging.

[0063] It should be noted that the lensless camera used in the present invention can be any type of lensless camera or an imaging device using a diffractive optical element.

[0064] The result of the conditional diffusion model used in the present invention can be various conditional diffusion model structures, such as StableSR and ControlNet.

[0065] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A two-stage lensless camera imaging method, characterized in that: The following steps are involved: The image captured by the lensless camera is obtained and input into a pre-trained space-variant deconvolution network. The space-variant deconvolution network generates multiple different deconvolution intermediate results according to the input image, and weights and synthesizes them into the final target deconvolution result as the first-stage restoration result. The first-stage restoration result is input into a pre-trained conditional diffusion model for visual quality enhancement to obtain the second-stage restoration result as the final lens-free camera imaging result.

2. A two-stage lensless camera imaging method according to claim 1, characterized in that: The space-variant deconvolution network includes K×K learnable deconvolution kernels and generates K×K deconvolution intermediate results accordingly, and the K×K deconvolution intermediate results respectively correspond to the K×K grids into which the target deconvolution result is divided.

3. A two-stage lensless camera imaging method according to claim 2, characterized in that: The expression of the generation process of the deconvolution intermediate result is: Where y is the input image, F and F -1 represent discrete Fourier transform and its inverse transform, respectively. is the ith deconvolution intermediate result, p (i) is the i-th learnable deconvolution kernel.

4. A two-stage lensless camera imaging method according to claim 2, characterized in that: The synthetic expression of the target deconvolution result is: d i (u,v)=(uu i ) 2 +(vv i ) 2 . In the formula, x int (u,v) is the deconvolution synthesis result corresponding to the image coordinate (u,v), w i (u,v) is the spatial weight of the i-th deconvolution result, is the ith deconvolution intermediate result, (u i ,v i ) is the center of the i-th grid, d i (u,v) is the distance from coordinates u,v to the i-th grid.

5. A two-stage lensless camera imaging method according to claim 1, characterized in that: The conditional diffusion model includes a pre-trained image diffusion model.

6. A two-stage lensless camera imaging method according to claim 5, characterized in that: The conditional diffusion model further includes a conditional control module, which uses the first stage restoration result as a condition to add high-frequency details to the output result of the image diffusion model on the basis of ensuring structural consistency.

7. A two-stage lensless camera imaging method according to claim 1, characterized in that: The method further includes: during the training process of the space-variant deconvolution network, obtaining an original scene image corresponding to the lensless camera input image, constructing a reconstruction loss of the first-stage restoration result, and thus training the space-variant deconvolution network; During the training process of the conditional diffusion model, the original scene image corresponding to the restoration result of the first stage is also obtained, and the reconstruction loss of the restoration result of the second stage is constructed, so as to train the conditional diffusion model.

8. A two-stage lensless camera imaging method according to claim 1, characterized in that: The method further includes: directly inputting the final lensless camera imaging result obtained into a lensless face verification model for face verification.

9. A lensless camera, characterized in that: The lensless camera uses a two-stage lensless camera imaging method as described in any one of claims 1-8 to perform imaging.

10. The lensless camera according to claim 9, characterized in that: The lensless camera is a lensless camera or an imaging device using a diffractive optical element.

Images