Lensless imaging device and lensless image reconstruction method based on deep learning
Through lensless imaging devices and deep learning methods, the problems of large size, high cost and poor image quality caused by optical component-assisted imaging in traditional computing imaging technology are solved, and high-resolution lensless image reconstruction is achieved, reducing costs and Simplified operation.
Patent Information
- Application Number
- CN202410981932.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-15
AI Technical Summary
Existing computational imaging technologies require optical components to assist imaging, resulting in large system size, high cost, complex operation and low image resolution.
A lensless imaging device is designed, including a housing, a diffuser, a sensor and a handle, and a lensless image reconstruction network model is constructed using deep learning methods. Images are acquired and trained through the lensless imaging device to generate high-resolution reconstruction images.
Lensless imaging is achieved, reducing costs, simplifying operations, reducing device volume, and improving image quality.
Smart Images

Figure CN118972684A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of lens imaging, and more specifically, to a lens-free imaging device and a lens-free image reconstruction method based on deep learning. Background Art
[0002] Computational imaging is a cutting-edge technology that combines optical imaging with digital signal processing. It achieves resolution and functionality beyond that of traditional imaging systems by optimizing and analyzing the data captured by the optical system.
[0003] Traditional computational imaging technology requires optical elements for auxiliary imaging. This method not only increases the size and weight of the system, but also requires high precision in manufacturing and adjusting the optical elements, resulting in high manufacturing costs. Secondly, the operating steps of the device are relatively complicated, and the device is large and difficult to move conveniently. At the same time, there is also the problem of difficulty in fixing and adjusting the position of optical elements. When reconstructing lensless images, the resolution is low and the image quality is poor.
[0004] The prior art discloses a dense sample lensless microscopic imaging device and method based on deep learning. The method uses a single-side light source to illuminate the sample under test, and uses an image sensor to capture the image of the sample under test. Finally, a deep neural network is used to reconstruct the sample image. The method is costly, complex to operate, and bulky. Summary of the invention
[0005] In view of the defects of the imaging devices and methods in the prior art, such as high cost, complex operation, large device size and poor image quality, the present invention provides a lens-free imaging device and a lens-free image reconstruction method based on deep learning.
[0006] The primary purpose of the present invention is to solve the above technical problems. The technical solution of the present invention is as follows:
[0007] A lensless imaging device comprises a housing, a diffuser, a sensor, and a sensor bearing surface;
[0008] The sensor bearing surface is arranged inside the housing and moves along the axis direction of the housing; the sensor is arranged at the center position of the sensor bearing surface;
[0009] A first through hole is arranged at the center of the wall surface of the housing opposite to the sensor bearing surface, and a diffuser is arranged on a side of the first through hole close to the sensor bearing surface.
[0010] Furthermore, the lens-free imaging device also includes a handle and an aperture; one end of the handle is connected to the sensor bearing surface, and the other end extends out of the housing; the aperture is arranged on a side of the first through hole away from the sensor bearing surface.
[0011] Deep learning-based lensless image reconstruction method, including:
[0012] S1: obtaining lens-free images of several target objects using a lens-free imaging device; obtaining real images of the target objects using a camera;
[0013] S2: constructing a training data set based on the lens-free images of the plurality of target objects and the real images of the target objects;
[0014] S3: using the training data set to train the constructed lensless image reconstruction network model to obtain a trained lensless image reconstruction network model;
[0015] S4: Acquire a lens-free image of the target object using a lens-free imaging device, input the image into a trained lens-free image reconstruction network model, and obtain a reconstructed image of the target object.
[0016] Furthermore, before the lens-free imaging device acquires the lens-free images of the plurality of target objects, the method further includes:
[0017] The distance between the sensor bearing surface and the first through hole is adjusted so that a linear point diffusion pattern with a contrast higher than a first preset value is formed on the sensor, and the current distance is fixed.
[0018] Furthermore, the constructed lensless image reconstruction network model is trained using the training data set to obtain a trained lensless image reconstruction network model, including:
[0019] S301: inputting the lens-free images of the plurality of target objects into a generator to obtain generated images;
[0020] S302: Inputting the generated image and the real image of the target object into a discriminator for training to obtain an error and a trained discriminator;
[0021] S303: Using the error to train a generator to obtain a trained generator;
[0022] S304: Use the trained generator as a new generator and the trained discriminator as a new discriminator, and repeat steps S301 to S303 until the output of the discriminator reaches a second preset value.
[0023] Furthermore, the generator includes a first convolutional layer, a first batch normalized layer, a first activation layer, a second convolutional layer, a second batch normalized layer, a second activation layer, a third convolutional layer, a third batch normalized layer, a third activation layer, a fourth convolutional layer, a fourth batch normalized layer, a fourth activation layer, a first transposed convolutional layer, a fifth batch normalized layer, a fifth activation layer, a second transposed convolutional layer, a sixth batch normalized layer, a sixth activation layer, a third transposed convolutional layer, a seventh batch normalized layer, a seventh activation layer, a fourth transposed convolutional layer, and an eighth activation layer, which are connected in sequence.
[0024] Furthermore, the loss function of the generator is as follows:
[0025]
[0026] is the lens-free image, is a real image; is the nth image generated by the generator; n is the sample image number; N is the number of training samples; L GI is the perceptual loss function;
[0027] Furthermore, the perceptual loss function is as follows:
[0028]
[0029] For content loss, To combat losses;
[0030]
[0031] is the lens-free image, is the real image; G θG (I TM ) is the image generated by the generator; W i,j is the width of each feature map in the network; H i,j is the height of each feature map in the network; φ i,j (·) is the feature map obtained by the jth convolution and activation before the i-th maximum pooling layer in the network; is the reconstructed image; x and y represent the elements of row y and column x in the feature map matrix.
[0032] Furthermore, the discriminator includes a fifth convolutional layer, a ninth activation layer, a sixth convolutional layer, an eighth batch normalized layer, a tenth activation layer, a seventh convolutional layer, a ninth batch normalized layer, an eleventh activation layer, an eighth convolutional layer, a tenth batch normalized layer, a twelfth activation layer, a ninth convolutional layer, and a thirteenth activation layer, which are connected in sequence.
[0033] Furthermore, the loss function of the discriminator is as follows:
[0034]
[0035] D θD (I GM ) is the probability that the discriminator network identifies the input as a real image; D θD [G θG (I TM )] is the probability that the discriminator network determines that the image is generated; To identify the probability distribution of real images; The probability distribution of the generated image is discriminative.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] The present invention uses a lens-free imaging device to obtain a lens-free image, and reconstructs the image through a lens-free image reconstruction network model, and finally obtains a reconstructed image of the target object. The device and method can form an image without a lens, has low cost, simple operation, small device size, and good image quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A structural diagram of a lens-free imaging device provided in an embodiment.
[0039] Figure 2 A flow chart of a lensless image reconstruction method based on deep learning provided in an embodiment.
[0040] Figure 3 A flow chart of a lens-free image reconstruction network model training provided in an embodiment.
[0041] Figure 4 A diagram of the generator structure provided for an embodiment.
[0042] Figure 5 A structural diagram of a discriminator provided in an embodiment. DETAILED DESCRIPTION
[0043] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;
[0044] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;
[0045] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0046] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0047] Example
[0048] like Figure 1 As shown, a lens-free imaging device comprises a housing 1, a diffuser 2, a sensor 3, and a sensor bearing surface 4;
[0049] The sensor bearing surface 4 is arranged inside the housing 1 and moves along the axis direction of the housing 1; the sensor 3 is arranged at the center of the sensor bearing surface 4;
[0050] A first through hole 6 is disposed at the center of a wall surface of the housing 1 opposite to the sensor carrying surface 4 , and a diffuser 2 is disposed on a side of the first through hole 6 close to the sensor carrying surface 4 .
[0051] It should be noted that the diffuser randomly scatters the incident light, making the light more uniform on the imaging surface, avoiding the problem of too large dark areas or too large bright areas, and helping to improve the imaging quality.
[0052] Furthermore, a lens-free imaging device also includes a handle 5 and an aperture 7; one end of the handle 5 is connected to the sensor bearing surface 4, and the other end extends out of the housing 1; the aperture 7 is arranged on a side of the first through hole 6 away from the sensor bearing surface 4.
[0053] It should be noted that a lens-free imaging device does not require the use of an optical lens and has low cost.
[0054] In a specific embodiment, the aperture may be made of black tape or a light shielding plate, and is used to control the light flux entering the imaging system, thereby adjusting the brightness and exposure time of the imaging, and affecting the resolution of the imaging.
[0055] It should be noted that the provision of the handle can more conveniently adjust the distance between the sensor bearing surface and the first through hole.
[0056] like Figure 2 As shown, the lens-free image reconstruction method based on deep learning includes:
[0057] S1: obtaining lens-free images of several target objects using a lens-free imaging device; obtaining real images of the target objects using a camera;
[0058] S2: constructing a training data set based on the lens-free images of the plurality of target objects and the real images of the target objects;
[0059] S3: using the training data set to train the constructed lensless image reconstruction network model to obtain a trained lensless image reconstruction network model;
[0060] S4: Acquire a lens-free image of the target object using a lens-free imaging device, input the image into a trained lens-free image reconstruction network model, and obtain a reconstructed image of the target object.
[0061] It should be noted that this method can achieve high-resolution and high-precision reconstruction of lens-free images, and the use of hardware such as GPU can accelerate parallel computing and further improve the reconstruction speed.
[0062] In a specific embodiment, the lens-free image reconstruction network model can use an adaptive moment estimation optimization algorithm (ADAM) or a deep convolutional generative adversarial network (DCGAN).
[0063] Furthermore, before the lens-free imaging device acquires the lens-free images of the plurality of target objects, the method further includes:
[0064] The distance between the sensor bearing surface and the first through hole is adjusted so that a linear point diffusion pattern with a contrast higher than a first preset value is formed on the sensor, and the current distance is fixed.
[0065] In one specific embodiment, light from a point source is scattered by a diffuser to form a high contrast point spread pattern on the sensor.
[0066] Furthermore, if Figure 3 As shown, the constructed lensless image reconstruction network model is trained using the training data set to obtain a trained lensless image reconstruction network model, including:
[0067] S301: inputting the lens-free images of the plurality of target objects into a generator to obtain generated images;
[0068] S302: Inputting the generated image and the real image of the target object into a discriminator for training to obtain an error and a trained discriminator;
[0069] S303: Using the error to train a generator to obtain a trained generator;
[0070] S304: Use the trained generator as a new generator and the trained discriminator as a new discriminator, and repeat steps S301 to S303 until the output of the discriminator reaches a second preset value.
[0071] In a specific embodiment, the second preset value is 0.5.
[0072] It should be noted that the alternating training process of the generator network and the discriminator network will be repeated continuously, so that the generator network can generate results that are closer and closer to the real image.
[0073] Furthermore, if Figure 4 As shown, the generator includes a first convolutional layer, a first batch normalized layer, a first activation layer, a second convolutional layer, a second batch normalized layer, a second activation layer, a third convolutional layer, a third batch normalized layer, a third activation layer, a fourth convolutional layer, a fourth batch normalized layer, a fourth activation layer, a first transposed convolutional layer, a fifth batch normalized layer, a fifth activation layer, a second transposed convolutional layer, a sixth batch normalized layer, a sixth activation layer, a third transposed convolutional layer, a seventh batch normalized layer, a seventh activation layer, a fourth transposed convolutional layer, and an eighth activation layer, which are connected in sequence.
[0074] It should be noted that the first convolutional layer, the first batch normalized layer, and the first activation layer combination have the effect of downsampling, the second convolutional layer, the second batch normalized layer, and the second activation layer combination have the effect of downsampling, the third convolutional layer, the third batch normalized layer, and the third activation layer combination have the effect of downsampling, and the fourth convolutional layer, the fourth batch normalized layer, and the fourth activation layer combination have the effect of downsampling.
[0075] It should be noted that the first transposed convolutional layer, the fifth batch normalized layer, and the fifth activation layer combination have the effect of upsampling, the second transposed convolutional layer, the sixth batch normalized layer, and the sixth activation layer combination have the effect of upsampling, the third transposed convolutional layer, the seventh batch normalized layer, and the seventh activation layer combination have the effect of upsampling, and the fourth transposed convolutional layer and the eighth activation layer combination have the effect of upsampling.
[0076] In a specific embodiment, the lens-free image input to the generator has three channels, namely R channel, G channel and B channel; the height is 64 pixels and the width is 64 pixels. The first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer all contain 4x4 convolution kernels, the step size is 2, and the padding is 1. The input dimension of the first convolution layer is 3x 64x 64, the output dimension of the first activation layer is 64x 32x 32, the input dimension of the second convolution layer is 64x 32x 32, the output dimension of the second activation layer is 128x 16x 16, the input dimension of the third convolution layer is 128x 16x 16, the output dimension of the third activation layer is 256x8x 8, the input dimension of the fourth convolution layer is 256x 8x 8, and the output dimension of the fourth activation layer is 512x 4x 4.
[0077] In a specific embodiment, the first transposed convolution layer, the second transposed convolution layer, the third transposed convolution layer, and the fourth transposed convolution layer all contain a 4×4 convolution kernel, a stride of 2, and a padding of 1. The input dimension of the first transposed convolution layer is 512x 4x4, and the output dimension of the fifth activation layer is 256x 8x 8. The input dimension of the second transposed convolution layer is 256x 8x 8, and the output dimension of the sixth activation layer is 128x 16x 16. The input dimension of the third transposed convolution layer is 128x 16x 16, and the output dimension of the seventh activation layer is 64x 32x 32. The input dimension of the fourth transposed convolution layer is 64x 32x 32. The final output of the eighth activation layer generates an image with 3 channels, namely, R channel, G channel, and B channel; the height is 64 pixels and the width is 64 pixels.
[0078] It should be noted that the transposed convolution layer upsamples the low-resolution feature map to generate a higher-resolution feature map. Compared with the simple upsampling operation, the transposed convolution layer can better capture the feature expression in the image generation process by learning parameters, improve the learning ability of the model, and thus generate higher-resolution image output.
[0079] In a specific embodiment, the first activation layer, the second activation layer, the third activation layer, the fourth activation layer, the fifth activation layer, the sixth activation layer, and the seventh activation layer use the ReLU activation function, and the eighth activation layer uses the Tanh activation function.
[0080] It should be noted that the eighth activation layer limits the output value to the range of [-1, 1]. This ensures that the generated image pixel values are within a reasonable range, which is convenient for subsequent processing and use.
[0081] Furthermore, the loss function of the generator is as follows:
[0082]
[0083] is the lens-free image, is a real image; is the nth image generated by the generator; n is the sample image number; N is the number of training samples; L GI is the perceptual loss function;
[0084] Furthermore, the perceptual loss function is as follows:
[0085]
[0086] For content loss, To combat losses;
[0087]
[0088] is the lens-free image, is the real image; G θG (I TM ) is the image generated by the generator; W i,j is the width of each feature map in the network; H i,j is the height of each feature map in the network; φ i,j (·) is the feature map obtained by the jth convolution and activation before the i-th maximum pooling layer in the network; is the reconstructed image; x and y represent the elements of row y and column x in the feature map matrix.
[0089] Furthermore, if Figure 5As shown, the discriminator includes a fifth convolutional layer, a ninth activation layer, a sixth convolutional layer, an eighth batch normalized layer, a tenth activation layer, a seventh convolutional layer, a ninth batch normalized layer, an eleventh activation layer, an eighth convolutional layer, a tenth batch normalized layer, a twelfth activation layer, a ninth convolutional layer, and a thirteenth activation layer, which are connected in sequence.
[0090] It should be noted that the fifth convolutional layer and the ninth activation layer combination has the function of downsampling, the sixth convolutional layer, the eighth batch normalization layer, and the tenth activation layer combination has the function of downsampling, the seventh convolutional layer, the ninth batch normalization layer, and the eleventh activation layer combination has the function of downsampling, the eighth convolutional layer, the tenth batch normalization layer, and the twelfth activation layer combination has the function of downsampling, and the ninth convolutional layer and the thirteenth activation layer combination has the function of downsampling.
[0091] In a specific embodiment, the fifth convolution layer, the sixth convolution layer, the seventh convolution layer, and the eighth convolution layer all include a 4×4 convolution kernel, a step size of 2, and a padding of 1. The ninth convolution layer includes a 4×4 convolution kernel, a step size of 1, and a padding of 0. Since the number of input channels of the first convolution layer is small (usually 3), the overhead of calculating BN is relatively small, and thus omitting the BN layer in the fifth convolution layer can improve computational efficiency.
[0092] In a specific embodiment, the generated image of the input of the fifth convolutional layer has 3 channels, namely R channel, G channel and B channel; the height is 64 pixels and the width is 64 pixels. The output dimension of the ninth activation layer is 64x 32x 32. The input dimension of the sixth convolutional layer is 64x 32x 32, and the output dimension of the tenth activation layer is 128x 16x 16. The input dimension of the seventh convolutional layer is 128x 16x 16, and the output dimension of the eleventh activation layer is 256x 8x 8. The input dimension of the eighth convolutional layer is 256x 8x 8, and the output dimension of the twelfth activation layer is 512x 4x 4. The input dimension of the ninth convolutional layer is 512x 4x 4, and the output dimension of the thirteenth activation layer is 1x 1x 1.
[0093] In a specific embodiment, the ninth activation layer, the tenth activation layer, the eleventh activation layer, and the twelfth activation layer use the LeakyReLU activation function, and the thirteenth activation layer uses the sigmoid activation function.
[0094] Furthermore, the loss function of the discriminator is as follows:
[0095]
[0096] D θD (I GM ) is the probability that the discriminator network identifies the input as a real image; D θD [G θG(I TM )] is the probability that the discriminator network determines that the image is generated; To identify the probability distribution of real images; The probability distribution of the generated image is discriminative.
[0097] It should be noted that if D θD (I GM ) is closer to 1, the θD [G θG (I TM )] is closer to 0, the better the discriminator network training effect is, and the better it can distinguish the source of the image.
[0098] The same or similar reference numerals correspond to the same or similar components;
[0099] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;
[0100] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A lensless imaging device, characterized in that: It comprises a housing (1), a diffuser (2), a sensor (3), and a sensor bearing surface (4); The sensor bearing surface (4) is arranged inside the housing (1) and moves along the axial direction of the housing (1); the sensor (3) is arranged at the center of the sensor bearing surface (4); A first through hole (6) is arranged at the center of the wall surface of the housing (1) opposite to the sensor bearing surface (4), and a diffuser (2) is arranged on the side of the first through hole (6) close to the sensor bearing surface (4).
2. A lens-free imaging device according to claim 1, characterized in that: It also comprises a handle (5) and an aperture (7); one end of the handle (5) is connected to the sensor bearing surface (4), and the other end extends out of the housing (1); the aperture (7) is arranged on a side of the first through hole (6) away from the sensor bearing surface (4).
3. A lens-free image reconstruction method based on deep learning, characterized in that: include: S1: Acquire lens-free images of a plurality of target objects using the lens-free imaging device according to claim 1 or 2; Using a camera to obtain a real image of the target object; S2: constructing a training data set based on the lens-free images of the plurality of target objects and the real images of the target objects; S3: using the training data set to train the constructed lensless image reconstruction network model to obtain a trained lensless image reconstruction network model; S4: Use the lensless imaging device described in claim 1 or 2 to obtain a lensless image of the target object, input it into a trained lensless image reconstruction network model, and obtain a reconstructed image of the target object.
4. The lensless image reconstruction method based on deep learning according to claim 3, characterized in that: Before the lens-free imaging device acquires lens-free images of a plurality of target objects, the method further includes: The distance between the sensor bearing surface and the first through hole is adjusted so that a linear point diffusion pattern with a contrast higher than a first preset value is formed on the sensor, and the current distance is fixed.
5. The lensless image reconstruction method based on deep learning according to claim 3, characterized in that: The constructed lensless image reconstruction network model is trained using the training data set to obtain a trained lensless image reconstruction network model, including: S301: inputting the lens-free images of the plurality of target objects into a generator to obtain generated images; S302: Inputting the generated image and the real image of the target object into a discriminator for training to obtain an error and a trained discriminator; S303: Using the error to train a generator to obtain a trained generator; S304: Use the trained generator as a new generator and the trained discriminator as a new discriminator, and repeat steps S301 to S303 until the output of the discriminator reaches a second preset value.
6. The lensless image reconstruction method based on deep learning according to claim 5, characterized in that: The generator includes a first convolutional layer, a first batch normalized layer, a first activation layer, a second convolutional layer, a second batch normalized layer, a second activation layer, a third convolutional layer, a third batch normalized layer, a third activation layer, a fourth convolutional layer, a fourth batch normalized layer, a fourth activation layer, a first transposed convolutional layer, a fifth batch normalized layer, a fifth activation layer, a second transposed convolutional layer, a sixth batch normalized layer, a sixth activation layer, a third transposed convolutional layer, a seventh batch normalized layer, a seventh activation layer, a fourth transposed convolutional layer, and an eighth activation layer, which are connected in sequence.
7. The lensless image reconstruction method based on deep learning according to claim 5 or 6, characterized in that: The loss function of the generator is as follows: is the lens-free image, is a real image; is the nth image generated by the generator; n is the sample image number; N is the number of training samples; L GI is the perceptual loss function.
8. The lensless image reconstruction method based on deep learning according to claim 7, characterized in that: The perceptual loss function is as follows: For content loss, To combat losses; is the lens-free image, is a real image; is the image generated by the generator; W i,j is the width of each feature map in the network; H i,j is the height of each feature map in the network; φ i,j (·) is the feature map obtained by the jth convolution and activation before the i-th maximum pooling layer in the network; is the reconstructed image; x and y represent the elements of row y and column x in the feature map matrix.
9. The lensless image reconstruction method based on deep learning according to claim 5, characterized in that: The discriminator includes a fifth convolutional layer, a ninth activation layer, a sixth convolutional layer, an eighth batch normalization layer, a tenth activation layer, a seventh convolutional layer, a ninth batch normalization layer, an eleventh activation layer, an eighth convolutional layer, a tenth batch normalization layer, a twelfth activation layer, a ninth convolutional layer, and a thirteenth activation layer, which are connected in sequence.
10. The lensless image reconstruction method based on deep learning according to claim 5 or 9, characterized in that: The loss function of the discriminator is as follows: D θD (I GM ) is the probability that the discriminator network identifies the input as a real image; D θD [G θG (I TM )] is the probability that the discriminator network determines that the image is generated; To identify the probability distribution of real images; The probability distribution of the generated image is discriminative.
