Lens-free camera image reconstruction method and device based on pre-training generation model

Abstract

The invention discloses a lens-free camera image reconstruction method and device based on a pre-training generative model. The method comprises the following steps: constructing a lens-free camera system; collecting a scene image by adopting a lens-free camera system; preprocessing the scene image; performing feature extraction on the preprocessed image; a pre-trained diffusion model is used as image prior; and inputting the image features after feature extraction into a diffusion model, and iteratively denoising to reconstruct a high-quality image. An optical system can be obviously simplified, and the size and cost of the camera are reduced; the image reconstruction quality is improved by using a pre-training generation model; more flexible image reconstruction control is realized through text guidance; and the calculation efficiency is improved, so that the method can be applied to resource-limited equipment.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lensless camera image reconstruction method and device based on a pre-trained generative model. Background Art

[0002] Traditional cameras typically use complex lens systems for focusing and imaging, which makes them bulky and expensive. To address this issue, researchers have proposed the concept of lensless cameras, which replace traditional lenses with specialized masks. However, existing lensless camera systems still have shortcomings in image reconstruction quality, making them difficult to meet the requirements of practical applications.

[0003] The main problems in the existing technology include: 1. Poor image reconstruction quality, lack of details and clarity; 2. The reconstruction algorithm is highly complex and difficult to process in real time; 3. The lack of ability to effectively utilize scene prior knowledge limits the further improvement of reconstruction effects.

[0004] Therefore, a new lens-free camera system and image reconstruction method are urgently needed to improve the quality of reconstructed images and effectively utilize scene prior knowledge. Summary of the Invention

[0005] According to an embodiment of the present invention, a lensless camera image reconstruction method based on a pre-trained generative model is provided, comprising the following steps: Building a lens-less camera system; A lens-less camera system is used to capture scene images; Preprocessing the scene image; Perform feature extraction on the preprocessed image; Utilize pre-trained diffusion model as image prior; The extracted image features are input into the diffusion model and denoised iteratively to reconstruct a high-quality image.

[0006] Furthermore, the lens-less camera system includes a mask and an image sensor disposed adjacent to each other.

[0007] Furthermore, preprocessing the scene image includes the following steps: Separate the scene image according to the color channel to obtain four independent channel images of R, Gr, Gb and B; Perform a separable linear transformation on each channel image to obtain a transformed channel image; Stack the transformed channel images into a single 4-channel tensor.

[0008] Furthermore, the pre-trained ControlNet network is used to extract features from the pre-processed images.

[0009] Furthermore, the diffusion model is the StableDiffusion2.1 model.

[0010] Furthermore, the extracted image features are input into the diffusion model, and iterative denoising to reconstruct a high-quality image includes the following steps: Input the extracted image features into the diffusion model; Perform T iterations of denoising, where T is the predefined number of diffusion steps. In each iteration, the inverse diffusion process of the diffusion model is used to predict and remove the noise component in the current image, gradually restoring a clearer image. After the iteration is completed, the reconstructed image is output.

[0011] Furthermore, the image features after feature extraction are input into the diffusion model, and iterative denoising to reconstruct a high-quality image also includes the following steps: inputting a scene text description into the diffusion model.

[0012] According to another embodiment of the present invention, a lensless camera image reconstruction device based on a pre-trained generative model is provided, comprising: An image acquisition module, which constructs a lens-less camera system and uses the lens-less camera system to acquire scene images; A preprocessing module, which preprocesses the scene image; Feature extraction module, the feature extraction module extracts features from the preprocessed image; Iterative reconstruction module,The iterative reconstruction module uses the pre-trained diffusion model as the image prior,,inputs the image features after feature extraction into the diffusion model, and iteratively denoises,to reconstruct a high-quality image.

[0013] According to another embodiment of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned lensless camera image reconstruction method based on the pre-trained generative model is implemented.

[0014] The present invention also provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to run the above-mentioned lens-free camera image reconstruction method based on the pre-trained generative model.

[0015] According to the lensless camera image reconstruction method and device based on the pre-trained generative model according to the embodiments of the present invention, it is possible to significantly simplify the optical system and reduce the size and cost of the camera; improve the image reconstruction quality by using the pre-trained generative model; achieve more flexible image reconstruction control through text guidance; and improve computational efficiency, making the method applicable to resource-constrained devices.

[0016] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart of a lens-less camera image reconstruction method based on a pre-trained generative model according to an embodiment of the present invention; Figure 2 2 is a schematic structural diagram of a lens-less camera system according to a lens-less camera image reconstruction method based on a pre-trained generative model according to an embodiment of the present invention; Figure 3 Schematic diagram of a mask pattern of a lens-less camera image reconstruction method based on a pre-trained generative model according to an embodiment of the present invention; Figure 4 Schematic diagram of a point spread function of a lens-less camera image reconstruction method based on a pre-trained generative model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings to further illustrate the present invention.

[0019] First, combine Figures 1 to 4 A lensless camera image reconstruction method based on a pre-trained generative model according to an embodiment of the present invention is described. The method is used for image reconstruction of a lensless camera and has a wide range of application scenarios.

[0020] like Figures 1 to 4 As shown, the lensless camera image reconstruction method based on the pre-trained generative model according to an embodiment of the present invention comprises the following steps: A lensless camera system was constructed, comprising a mask and an image sensor. The mask pattern was obtained by performing an outer product operation on a separable M-sequence binary signal. Specifically, an M-sequence binary signal of length 255 was used to perform an outer product operation to obtain a two-dimensional pattern. This pattern was then photolithographically fabricated onto a 0.2 mm thick chrome plate on a glass substrate. Finally, the pattern was converted into a physical form: small holes or transparent areas on the chrome plate, each with a size of 25 microns. In this embodiment, the mask pattern was designed based on the M-sequence and possessed specific mathematical properties that facilitated subsequent image reconstruction. The image sensor used was an AlliedVision 1800U-500 board-level camera with the following parameters: pixel size: 2.2 μm, total pixel count: 5 million, bit depth: 12 bits. The mask was placed adjacent to the image sensor to achieve the desired optical properties. The specific distance between the image sensor and the mask could be adjusted based on actual conditions.

[0021] like Figure 3 As shown in FIG, a simplified schematic diagram of the outer product pattern of a separable M sequence binary signal is shown, where the black squares represent the openings on the mask and the white areas represent the opaque parts; Figure 4 As shown, the corresponding point spread function is shown, which uses concentric circles to simulate the diffraction pattern of light. The color gradually fades from the center to the outside, representing the imaging pattern of a single point light source on the sensor after passing through the mask.

[0022] A lens-less camera system is used to capture scene images, specifically RAW images.

[0023] The scene image is preprocessed, including the following steps: separating the scene image according to the color channel to obtain four independent channel images of R, Gr, Gb and B; performing a separable linear transformation on each channel image to obtain the transformed channel image; the linear transformation formula is as follows:

[0024] in, , the size of Ck is hi×wi, φl∈R^(ho×hi) and φr∈R^(wi×wo) are learnable matrices.

[0025] For each channel Ck (where k represents one of R, Gr, Gb, or B), a linear transformation is applied. This transformation is defined by two learnable matrices, φl and φr, which apply to the left (or input) and right (or output) sides of the channel, respectively. φl has dimensions ho × hi, meaning it transforms the height of the input channel from hi to ho; φr has dimensions wi × wo, meaning it transforms the width of the input channel from wi to wo. Through this transformation, each channel Ck is transformed into a new channel Co_k of dimensions ho × wo.

[0026] The transformed channel images are stacked into a single 4-channel tensor Co, Co∈R^(4×ho×wo); the four transformed channels (Co_R, Co_Gr, Co_Gb, Co_B) are stacked into a single 4-channel tensor Co. The size of this tensor is 4×ho×wo, where 4 represents the number of channels, and ho and wo are the height and width after transformation, respectively.

[0027] A pre-trained ControlNet network is used to extract features from the pre-processed image, and the extracted features are used to guide the generation of the subsequent diffusion model. The 4-channel tensor obtained by applying a separable linear transform to the RAW image is used as the input of the ControlNet network. The ControlNet network uses a copy of a pre-trained diffusion model (specifically its UNet encoder portion). During initialization, several additional convolutional layers with weights initialized to zero are added to the UNet encoder to adjust feature dimensions or add additional control signals. By using the output of the separable transform as a control signal, ControlNet can significantly enhance the generation capability of the diffusion model. By adjusting the input and parameters of ControlNet, users can fine-tune the generated image.

[0028] Use the pre-trained StableDiffusion2.1 diffusion model as image prior; The extracted image features are input into the diffusion model, and iterative denoising is performed to reconstruct a high-quality image. The steps include: inputting the output of ControlNet into the diffusion model as a condition; performing T iterative denoising, where T is a predefined number of diffusion steps. In each iteration, the diffusion model predicts and removes the noise in the image based on the current image state (the image containing noise) and the conditional input (the output from the control network), gradually restoring a clearer image; after the iteration is completed, the reconstructed image is output.

[0029] Further, if Figures 1 to 4As shown, in this embodiment, the image features after feature extraction are input into the diffusion model, and iterative denoising to reconstruct a high-quality image also includes the following steps: inputting the scene text description into the diffusion model to further guide the diffusion process, specifically, converting the text description into an embedding vector, and using the embedding vector as an additional condition in the diffusion process.

[0030] According to another embodiment of the present invention, a lensless camera image reconstruction device based on a pretrained generative model is provided, comprising: an image acquisition module, which constructs a lensless camera system and uses the lensless camera system to capture scene images; a preprocessing module, which preprocesses the scene images; a feature extraction module, which extracts features from the preprocessed images; and an iterative reconstruction module, which uses a pretrained diffusion model as an image prior, inputs the extracted image features into the diffusion model, and iteratively denoises to reconstruct a high-quality image.

[0031] According to another embodiment of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned lensless camera image reconstruction method based on the pre-trained generative model is implemented.

[0032] An image generation model is trained by combining a diffusion model and a separable reconstruction loss framework. The image generation training method is as follows: Data acquisition: A prototype camera was used to capture images of the LAION-aesthetics dataset projected onto a 34cm×34cm PC screen at a distance of 60cm from the camera. The exposure time was set to 12ms, and raw Bayer images with a 12-bit depth were saved. A total of approximately 55,000 images were collected, of which 500 were used for testing and the rest for training. Black screen images were collected for background correction.

[0033] During the training process, a loss function combining diffusion model loss and separable reconstruction loss is used. The formula is as follows:

[0034] Among them, LC is the control network loss, defined as: LC=E[||ε-εθ(zt, t, y, Cψ(Co))||^2], ε is the added noise, εθ is the denoising network, zt is the noise sample at time t, y is the text description (if any), Cψ(Co) is a control network, which generates conditional information based on certain conditions (such as text description y) and potential representation Co. This information is used to guide the generation process of the denoising network εθ; Lsep is the separable reconstruction loss, defined as: Lsep=||I-fconv(Co)||^2; I is the target image, fconv is the learned 3×3 convolutional layer; λ is the weight coefficient for balancing the two losses.

[0035] The diffusion loss (LC) measures the model's ability to recover the original image from noise. It is calculated based on the difference between the noisy sample and the output of the denoising network. The separable reconstruction loss (Lsep) ensures that the image generated by the model is similar to the target image under a certain transformation, which is achieved through a learned 3×3 convolutional layer. To balance these two losses, a weight coefficient λ is introduced.

[0036] The training process uses the AdamW optimizer with a learning rate of 5×10^(-5). The model iterates on the training set for 500k steps to optimize the control network parameters ψ and the separable transformation parameters φl and φr. These parameters together determine the quality and style of the images generated by the model. Through continuous iteration and optimization, the model gradually learns how to generate high-quality images from a given input (such as a text description or latent representation).

[0037] The experimental results are as follows. On a self-collected test set, this method achieved a PSNR (Peak Signal-to-Noise Ratio) of 21.89, an SSIM (Structural Similarity) of 0.541, a LPIPS (Perceptual Similarity) of 0.276, and a CLIP (Text-Image Similarity) score of 24.38. While reconstruction quality significantly degrades when no separable loss is present, adding text description guidance further improves reconstruction, particularly in terms of detail recovery.

[0038] The method of the present invention can also be applied to the following scenarios: ultra-thin camera modules in mobile devices, medical endoscopic imaging systems, visual sensors in space-constrained environments such as automobiles, hidden cameras in security monitoring, and special imaging needs in scientific research (such as high-speed photography, spectral imaging, etc.).

[0039] The present invention also provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to run the above-mentioned lens-free camera image reconstruction method based on the pre-trained generative model.

[0040] Above, refer to Figures 1 to 4 A lensless camera image reconstruction method and apparatus based on a pre-trained generative model according to an embodiment of the present invention are described. These methods can significantly simplify the optical system and reduce the size and cost of the camera. The pre-trained generative model is used to improve image reconstruction quality. Text-guided image reconstruction allows for more flexible control. Computational efficiency is improved, making the method applicable to resource-constrained devices.

[0041] It should be noted that, in this specification, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the elements.

[0042] Although the present invention has been described in detail through the above preferred embodiments, it should be understood that the above description is not intended to limit the present invention. After reading the above description, various modifications and substitutions of the present invention will become apparent to those skilled in the art. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A lensless camera image reconstruction method based on a pre-trained generative model, characterized in that: The following steps are included: Building a lens-less camera system; Using the lens-free camera system to collect scene images; Preprocessing the scene image; Perform feature extraction on the preprocessed image; Utilize pre-trained diffusion models as image priors; The image features after feature extraction are input into the diffusion model, and denoising is performed iteratively to reconstruct a high-quality image.

2. The lensless camera image reconstruction method based on a pre-trained generative model as claimed in claim 1, characterized in that: The lens-less camera system includes a mask and an image sensor disposed adjacent to each other.

3. The lensless camera image reconstruction method based on a pre-trained generative model as claimed in claim 1, characterized in that: The preprocessing of the scene image comprises the following steps: Separate the scene image according to color channels to obtain four independent channel images of R, Gr, Gb and B; Performing a separable linear transformation on each of the channel images to obtain a transformed channel image; Stack the transformed channel images into a single 4-channel tensor.

4. The lensless camera image reconstruction method based on a pre-trained generative model as claimed in claim 1, characterized in that: It is characterized in that The pre-trained ControlNet network is used to extract features from the pre-processed images.

5. The lensless camera image reconstruction method based on a pre-trained generative model as claimed in claim 1, characterized in that: The diffusion model is the StableDiffusion2.1 model.

6. The lensless camera image reconstruction method based on a pre-trained generative model as claimed in claim 1, characterized in that: The step of inputting the extracted image features into the diffusion model and iteratively denoising to reconstruct a high-quality image comprises the following steps: Inputting the extracted image features into the diffusion model; Perform T iterations of denoising, where T is a predefined number of diffusion steps, and in each iteration, use the inverse diffusion process of the diffusion model to predict and remove the noise component in the current image, and gradually restore a clearer image; After the iteration is completed, the reconstructed image is output.

7. The lensless camera image reconstruction method based on a pre-trained generative model as claimed in claim 6, characterized in that: The step of inputting the extracted image features into the diffusion model and iteratively denoising to reconstruct a high-quality image further comprises the following steps: inputting a scene text description into the diffusion model.

8. A lensless camera image reconstruction device based on a pre-trained generative model, characterized in that: Include: An image acquisition module, wherein the image acquisition module constructs a lens-less camera system and uses the lens-less camera system to acquire scene images; A preprocessing module, wherein the preprocessing module preprocesses the scene image; A feature extraction module, wherein the feature extraction module extracts features from the preprocessed image; The iterative reconstruction module uses a pre-trained diffusion model as an image prior, inputs the image features after feature extraction into the diffusion model, and iteratively denoises to reconstruct a high-quality image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the lens-free camera image reconstruction method based on the pre-trained generative model as described in any one of claims 1 to 7 is implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the lens-free camera image reconstruction method based on the pre-trained generative model as described in any one of claims 1-7.

Images