Reflection-type Fourier lamination method based on camera matrix

Abstract

The invention relates to the field of image super-resolution, in particular to an image reconstruction method, and particularly relates to a Fourier laminated image reconstruction method based on a deep learning network. A reflective far-field Fourier lamination imaging system is utilized, a series of low-resolution images are shot through a camera array, deep learning is combined with traditional Fourier lamination, the low-resolution images are reconstructed by utilizing a neural network, the acquisition number of the low-resolution images is reduced, the requirement of the traditional Fourier lamination for redundancy is met, and the method has the advantages of high resolution and high resolution. And the sampling speed and the reconstruction speed of Fourier lamination imaging are improved.

Description

Technical Field

[0001] The present invention relates to the field of image super-resolution, and more specifically, to an image reconstruction method, specifically a Fourier stack image reconstruction method based on a deep learning network. Background Art

[0002] Stacked imaging is a computational optical imaging method proposed in the last century that uses deconvolution to solve the phase problem. In the early 21st century, Rodenbur proposed a coherent diffraction stack imaging technology (PIE), which uses the stacking relationship of objects in the spatial domain to constrain and achieve the reconstruction of large-scale objects. In 2013, Zheng et al. proposed Fourier ptychographic microscopy (FPM) technology. Unlike PIE, this technology constrains the light field in the frequency domain of the object, so it can increase the object's spectrum, thereby equivalently improving the object's resolution. Fourier stacked microscopy combines phase recovery algorithms with aperture synthesis technology. This technology constrains the light field in the frequency domain of the object, so it can increase the image's spectrum, thereby equivalently improving the image resolution.

[0003] In recent years, the idea of ​​Fourier stacking has gradually been applied to far-field imaging. In traditional stacked diffraction imaging, a spatially confined light pattern (probe) is used to illuminate the object, and the diffraction pattern generated in the far field is observed, while in the far field, the Fourier transform of the probe is convolved with the Fourier transform of the object. The object is translated to several positions, and the diffraction pattern corresponding to each position is captured, and then merged using an iterative algorithm. However, the current methods require multiple repeated samplings, long dark field exposure times, and the need for redundant information result in long data acquisition times, and faster acquisition speeds and frequencies are required in many scenarios. If the image acquisition time can be reduced and the number of acquired images can be reduced, the overall imaging speed will be further improved and the quality will be higher.

[0004] In recent years, with the rapid development of deep learning, convolutional neural networks (CNN) have been increasingly used in computer vision and image processing, such as image super-resolution, image denoising, phase reconstruction, target detection, etc. Convolutional neural networks learn the difference between input images and output images through a large amount of training data, and then find the correct nonlinear mathematical mapping relationship between the two.

[0005] Therefore, we can combine deep learning with traditional Fourier stacking and use neural networks to reconstruct low-resolution images, reduce the number of low-resolution images collected, and improve the sampling and reconstruction speed of Fourier stack imaging. Summary of the invention

[0006] The purpose of the present invention is to solve the deficiencies in the current technology and propose a Fourier stack image reconstruction method based on a deep learning network, which can effectively improve the sampling and system imaging efficiency.

[0007] The technical solution provided by the present invention is as follows:

[0008] A first aspect: The present invention provides a super-resolution imaging sampling system, including a far-field reflective Fourier stack imaging optical path and a camera matrix, for collecting a number of low-resolution image intensity maps of known sizes.

[0009] In the second aspect, the present invention also provides a training method for an image reconstruction system, an image reconstruction method and a system.

[0010] The technical solution of the super-resolution imaging sampling system provided by the present invention is: a Fourier stack image reconstruction method based on a deep learning network. The steps are as follows:

[0011] Step 1: Create an image of the object using a liquid crystal spatial light modulator

[0012] Step 2: Use the array camera matrix system to collect several low-resolution images of known size

[0013] Step 3: Calculate the system magnification using the image size from step 2 and the camera target size

[0014] Step 4: Build a deep learning network for super-resolution image reconstruction

[0015] Step 5: Use the images collected in step 2 and the corresponding true values ​​to create a training data set and train the network

[0016] Step 6: Evaluate and optimize the network using widely used image quality assessment metrics

[0017] Step 7: Input the low-resolution image into the network trained and optimized in step 5 to obtain a high-resolution reconstructed image.

[0018] The present invention is further described in detail below in conjunction with the accompanying drawings BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of collecting images, training networks, and reconstructing high-resolution images using networks in an embodiment of the present invention

[0020] Figure 2 Schematic diagram of a reflective Fourier stack sampling system provided by an embodiment of the present invention

[0021] Figure 3 A diagram of a super-resolution image reconstruction network structure provided by an embodiment of the present invention

[0022] Figure 4 Data samples on a spatial light modulator and corresponding measurements from a camera array provided for an embodiment of the present invention

[0023] Figure 5 The high-resolution reconstructed image output by the network after thresholding and the corresponding true value provided by the embodiment of the present invention DETAILED DESCRIPTION

[0024] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0025] Step 1: Use liquid crystal spatial light modulator to create object images. More than 2,600 vector clipart images were produced, and more than 20,000 600×800 binary images were generated through image enhancement methods such as rotation, flipping, and scaling.

[0026] Step 2: Use the array camera matrix to collect low-resolution images: Build an imaging system. The specific imaging system structure is as follows: Figure 2 As shown, a 650nm laser light source is used for illumination, and a spatial filter is used to collimate the light source. The object image is created using a liquid crystal spatial light modulator (SLM), which contains 600×800 pixels and a pixel pitch of 20μm. The reflected phase modulated wave is imaged onto a camera array. The camera array consists of a 4×4 array of full-shutter cameras, and the offset between the optical axes of adjacent lenses is about 33 mm. The direction of each camera is adjusted so that the target appears in the center of the image it captures, and a group of 16 low-resolution images are taken at a time. During the sampling process, each camera obtains an average of 5 frames of images to suppress noise. It is illuminated by a laser equivalent coherent light source and captured by the camera array. In the case of the transfer function H(μ, v) = [P(λFμ, λFv), each camera is roughly simulated as a low-pass filter on the target field, where P(x, y) is the pupil function and F is the focal length. In the same camera array, the position (x i ,y i The transfer function of the first camera centered at ) is P(λFμ-x i ,λFv-y i ). The corresponding coherent impulse response of the camera is:

[0027]

[0028] where h o (x, y) is the point spread function of the camera located at the center of the (x, y) plane. In practical applications, camera tilt, focus state changes and (x i ,y i ) uncertainty has a significant impact on how well the phase function is known, where the phase function is:

[0029]

[0030] But for now, the array camera measurement model is defined as:

[0031]

[0032] Step 3: Calculate the system magnification, put the image formed by the spatial light modulator SLM on the camera, calculate the pixel size it occupies on the camera target surface, and calculate the system magnification β according to the magnification formula.

[0033] Step 4: Construct an image reconstruction network. The specific structure is as follows Figure 3 As shown in the figure, the network consists of an initial convolutional layer, 14 dense blocks, 7 transition layers, 7 upsampling layers and a final convolutional layer. The dense blocks and transition layers refer to DensenNet. Each dense block consists of a normalization layer, a convolutional layer, an activation function and other modules. The upsampling layer replaces the maximum pooling of the transition layer with deconvolution. The final result is upsampled so that the input image of the network matches the output image in terms of dimension.

[0034] Step 5: Construct a data set and loss function, filter and preprocess the collected images, obtain a training data set and train the network. The low-resolution image of each camera is cropped into a 90×120 area SLM imaging. The example training image and the corresponding measurement results are shown in Figure 4 As shown in the figure. The 23200 image data was split into 20000 training images and 3200 test images. The network was first trained for 100 epochs in PyTorch using binary cross entropy loss, Adam optimizer and learning rate 0.0003. Then a sub-dataset was created by selecting SSIM data with poor reconstruction effect. We used the sub-dataset to fine-tune the network for 20 batches and used the following loss:

[0035] l=l BCE +λl SSIM

[0036] We choose λ = 0.01 to avoid the influence of smoothing data.

[0037] Step 6: Use widely used image quality evaluation indicators to evaluate and optimize the network. The comparison results are shown in the following table:

[0038] enter MSE SSIM BCE network 0.0428 0.8117 0.1776

[0039] Step 7: Input the low-resolution image into the trained and optimized network and threshold it to obtain the high-resolution reconstruction result output by the network, such as Figure 5 shown.

Claims

1. A reflective Fourier stacking method based on a camera matrix, characterized in that: The steps are as follows Step 1: Create an image of the object using a liquid crystal spatial light modulator Step 2: Use the array camera matrix system to collect several low-resolution images of known size Step 3: Calculate the system magnification using the image size from step 2 and the camera target size Step 4: Build a deep learning network for super-resolution image reconstruction Step 5: Use the images collected in step 2 and the corresponding true values ​​to create a training data set and train the network Step 6: Evaluate and optimize the network using widely used image quality assessment metrics Step 7: Input the low-resolution image into the network trained and optimized in step 5 to obtain a high-resolution reconstructed image.

2. The camera matrix-based reflective Fourier stacking method according to claim 1, characterized in that: The specific method of obtaining an image using a spatial light modulator in step 1 is to make multiple vector clipping images and generate multiple binary images through image enhancement methods such as rotation, flipping, and scaling.

3. The camera matrix-based reflective Fourier stacking method according to claim 1, characterized in that: The camera array used to collect object images consists of 16 full-shutter cameras, and the offset between adjacent camera lenses is about 33mm.

4. The camera matrix-based reflective Fourier stacking method according to claim 1, characterized in that: The specific method of step 2 using the camera array to collect the low-resolution intensity map of the image obtained in step 1 is: use laser illumination, obtain spherical light illumination after collimation and beam expansion, reflect the light from the spatial light modulator, adjust the camera array position so that the array center converges with the equivalent camera aperture center, and use the camera array to take a low-resolution intensity map of a set of samples.

5. The camera matrix-based reflective Fourier stacking method according to claim 1, characterized in that: The image is illuminated by a laser equivalent coherent light source and captured by a camera array. In the case of a transfer function H(μ, v) = P(λFμ, λFv), each camera is roughly simulated as a low-pass filter on the target field, where P(x, y) is the pupil function and F is the focal length. In the same camera array, the position (x i ,y i ) is centered on the first camera and the transfer function is P(λFμ-x i ,λFν-y i ). The corresponding coherent impulse response of the camera is: where h o (x, y) is the point spread function of the camera located at the center of the (x, y) plane. In practical applications, camera tilt, focus state changes and (x i ,y i ) uncertainty has a significant impact on how well the phase function is known, where the phase function is: But for now, the array camera measurement model is defined as:

6. According to the camera matrix-based reflective Fourier stacking method of claim 1, the network structure in step 4 is characterized by: It consists of an initial convolutional layer, 14 dense blocks, 7 transition layers, 7 upsampling layers, and a final convolutional layer. The dense blocks and transition layers refer to DensenNet. Each dense block consists of a normalization layer, a convolutional layer, an activation function, and other modules. The upsampling layer replaces the maximum pooling of the transition layer with deconvolution. The final result is upsampled so that the input image of the network matches the output image in terms of dimension.

7. The camera matrix-based reflective Fourier stacking method according to claim 1, characterized in that: In step 5, the collected low-resolution images are pre-processed and screened by downsampling to produce training data sets and sub-data sets.

8. The image reconstruction network structure according to claim 6, characterized in that: The following losses were used: l=l BCE +l SSIM 9. The camera matrix-based reflective Fourier stacking method according to claim 1, characterized in that: The camera array collects 16 low-resolution images at a time.

Images