A camera matrix based reflective fourier lamination method

Abstract

The present application relates to the field of image super-resolution, more particularly, to an image reconstruction method, in particular to a Fourier stack image reconstruction method based on a deep learning network. By using a reflective far-field Fourier stack imaging system, a series of low-resolution images are captured by a camera array, deep learning is combined with traditional Fourier stack, and a neural network is used to reconstruct low-resolution images, reduce the number of low-resolution image acquisition, reduce the redundancy requirement of traditional Fourier stack, and improve the sampling speed and reconstruction speed of Fourier stack imaging.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution, and more specifically, to an image reconstruction method, specifically a Fourier layer image reconstruction method based on deep learning networks. Background Technology

[0002] Layered imaging is a computational optical imaging method proposed in the last century that uses deconvolution to solve phase problems. In the early 21st century, Rodenbur proposed a coherent diffraction layered imaging technique (PIE), which uses the layered relationship of objects in the spatial domain as a constraint to achieve the reconstruction of large-scale objects. In 2013, Zheng et al. proposed Fourier ptychographic microscopy (FPM). Unlike PIE, this technique constrains the light field in the frequency domain of the object, thus increasing the object's spectrum and effectively improving its resolution. Fourier ptychographic microscopy combines phase retrieval algorithms and aperture synthesis techniques. This technique constrains the light field in the frequency domain of the object, thus increasing the image's spectrum and effectively improving its resolution.

[0003] In recent years, the concept of Fourier stacking has been increasingly applied to far-field imaging. In traditional stacked diffraction imaging, a spatially constrained light pattern (probe) illuminates the object, and the resulting diffraction pattern in the far field is observed. In the far field, the Fourier transform of the probe is convolved with the Fourier transform of the object. The object is translated to several positions, and the diffraction pattern corresponding to each position is captured. Then, an iterative algorithm is used to merge them. However, current methods require multiple repeated samplings. Long exposure times in dark fields and the need for redundant information lead to long data acquisition times, while many scenarios require faster acquisition speeds and frequencies. If image acquisition time and the number of images acquired can be reduced, the overall imaging speed will be further improved, and the quality will also be higher.

[0004] In recent years, with the rapid development of deep learning, Convolutional Neural Networks (CNNs) have found increasing applications in computer vision and image processing, such as image super-resolution, image denoising, phase reconstruction, and object detection. CNNs learn the differences between input and output images through a large amount of training data, thereby finding the correct nonlinear mathematical mapping relationship between them.

[0005] Therefore, we can combine deep learning with traditional Fourier stacks and use neural networks to reconstruct low-resolution images, reducing the number of low-resolution images to be acquired and improving the sampling and reconstruction speed of Fourier stack imaging. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of current technologies by proposing a Fourier layered image reconstruction method based on deep learning networks, which can effectively improve sampling and system imaging efficiency.

[0007] The technical solution provided by this invention is as follows:

[0008] Firstly, the present invention provides a super-resolution imaging sampling system, including a far-field reflective Fourier stacked imaging optical path and a camera matrix, for acquiring several low-resolution image intensity maps of known size.

[0009] Secondly, the present invention also provides a training method, an image reconstruction method, and a system for an image reconstruction system.

[0010] The super-resolution imaging sampling system technical solution provided by this invention is: a Fourier layered image reconstruction method based on deep learning networks, the steps of which are as follows:

[0011] Step 1: Create an image of the object using a liquid crystal spatial light modulator;

[0012] Step 2: Use an array camera matrix system consisting of 4×4 full shutter cameras to acquire several low-resolution images of known size, with an offset of 33mm between adjacent camera lenses;

[0013] Step 3: Using the image size and camera target size from Step 2, calculate the system magnification M;

[0014] Step 4: Construct a deep learning network for super-resolution image reconstruction. The network consists of an initial convolutional layer, 14 dense blocks, 7 transition layers, 7 upsampling layers, and a final convolutional layer. The dense blocks and transition layers are based on DenseNet. Each dense block consists of a normalization layer, a convolutional layer, and an activation function module. The upsampling layer replaces the max pooling of the transition layer with deconvolution. An upsampling operation is performed on the final result to match the dimension of the input image to the output image.

[0015] Step 5: Use the images and corresponding ground truth values ​​collected in Step 2 to create a training dataset and train the network;

[0016] Step 6: Evaluate and optimize the network using mean squared error (MSE), structural similarity (SSIM), and binary cross-entropy loss function (BCE) as image quality assessment metrics.

[0017] Step 7: Input the low-resolution image into the network trained and optimized in Step 5 to obtain a high-resolution reconstructed image.

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the process of acquiring images, training the network, and reconstructing high-resolution images using the network in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of a reflective Fourier stacked sampling system provided in an embodiment of the present invention;

[0021] Figure 3 This is a diagram of the super-resolution image reconstruction network structure provided in an embodiment of the present invention;

[0022] Figure 4 Data samples on a spatial light modulator and corresponding measurements from a camera array are provided for embodiments of the present invention.

[0023] Figure 5 The high-resolution reconstructed image and corresponding ground truth value of the network output after thresholding are provided in the embodiments of the present invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings.

[0025] Step 1: Object images were created using a liquid crystal spatial light modulator, resulting in over 2600 vector clipping images. More than 20,000 images were then generated through image enhancement methods such as rotation, flipping, and scaling. A binary image.

[0026] Step 2: Acquire low-resolution images using an array camera matrix: Build an imaging system, the specific structure of which is as follows... Figure 2 As shown, a 650nm laser light source is used for illumination, and the light source is collimated using a spatial filter. The image of the object is created using a liquid crystal spatial light modulator (SLM), which includes... Pixels, pixel pitch The reflected phase-modulated wave is imaged onto a camera array. This camera array consists of... The array consists of full-shutter cameras, with an offset of approximately 33 mm between the optical axes of adjacent lenses. Each camera is oriented so that the target appears in the center of its captured image, and a set of 16 low-resolution images is captured at a time. During sampling, each camera acquires an average of 5 frames to suppress noise. Illumination is provided by a laser equivalent coherent light source, and the images are captured by the camera array. (The last sentence appears to be incomplete and possibly refers to a transfer function.) In this case, each camera is roughly simulated as a low-pass filter on the target field, where It is the pupil function, and F is the focal length. In the same camera array, by position... The transfer function of the first camera centered at the center is The corresponding coherent impulse response of this camera is:

[0027] ,

[0028] in It is located in The spread function of the camera point at the center of the plane. The phase function is defined as:

[0029]

[0030] However, for now, the array camera measurement model is defined as:

[0031] .

[0032] Step 3: Calculate the system magnification. Place the image formed by the spatial light modulator (SLM) on the camera and calculate the pixel size it occupies on the camera target surface. Calculate the system magnification M according to the magnification formula.

[0033] Step 4: Construct the image reconstruction network, the specific structure of which is as follows: Figure 3 As shown, the network consists of an initial convolutional layer, 14 dense blocks, 7 transition layers, 7 upsampling layers, and a final convolutional layer. The dense blocks and transition layers are modeled after DensenNet. Each dense block comprises a normalization layer, a convolutional layer, and an activation function. The upsampling layers replace the max pooling of the transition layers with deconvolution. Upsampling is performed on the final result to match the dimensions of the input image to the output image.

[0034] Step 5: Construct the dataset and loss function. Filter and preprocess the acquired images to obtain the training dataset and train the network. Low-resolution images from each camera are cropped to... SLM imaging of the region. Example training images and corresponding measurement results are shown below. Figure 4 As shown, the 23,200 image dataset was decomposed into 20,000 training images and 3,200 test images. The network was first trained for 100 epochs in PyTorch using binary cross-entropy loss, the Adam optimizer, and a learning rate of 0.0003. Then, a subset of the dataset was created by selecting SSIM data with poor reconstruction performance. We used this subset to fine-tune the network in 20 batches, using the following loss:

[0035] L = L BCE + βL SSIM .

[0036] We chose β=0.01 to avoid the effects of smoothing the data.

[0037] Step 6: Evaluate and optimize the network using widely used image quality evaluation metrics. The comparison results are shown in the table below.

[0038] enter MSE SSIM BCE network 0.0428 0.8117 0.1776

[0039] Step 7: After thresholding the low-resolution image input into the trained and optimized network, the high-resolution reconstruction result output by the network is obtained, such as... Figure 5 As shown.

Claims

1. A reflection-based Fourier stacking method based on a camera matrix, characterized in that, The steps are as follows: Step 1: Create an image of the object using a liquid crystal spatial light modulator; Step 2: Use an array camera matrix system consisting of 4×4 full shutter cameras to acquire several low-resolution images of known size, with an offset of 33mm between adjacent camera lenses; Step 3: Calculate the system magnification using the image size and camera target size from Step 2; Step 4: Construct a deep learning network for super-resolution image reconstruction. The network consists of an initial convolutional layer, 14 dense blocks, 7 transition layers, 7 upsampling layers, and a final convolutional layer. The dense blocks and transition layers are based on DenseNet. Each dense block consists of a normalization layer, a convolutional layer, and an activation function module. The upsampling layer replaces the max pooling of the transition layer with deconvolution; the final result is upsampled so that the input image matches the output image in dimension. Step 5: Use the images and corresponding ground truth values ​​collected in Step 2 to create a training dataset and train the network; Step 6: Evaluate and optimize the network using mean squared error (MSE), structural similarity (SSIM), and binary cross-entropy loss function (BCE) as image quality assessment metrics. Step 7: Input the low-resolution image into the network trained and optimized in Step 5 to obtain a high-resolution reconstructed image; The specific method for acquiring low-resolution images using a camera array in step two is as follows: laser illumination is used, and after collimation and beam expansion, spherical illumination is obtained. The light is reflected by a spatial light modulator, and the position of the camera array is adjusted so that the center of the array coincides with the center of the equivalent camera aperture. A set of low-resolution images of samples are then captured using the camera array. This camera matrix system utilizes laser equivalent coherent light source illumination, which is captured by a camera array, in the transfer function In this case, each camera is roughly simulated as a low-pass filter on the target field, where It is the pupil function, where F is the focal length; in the same camera array, by position The transfer function of the first camera centered at the center is: The coherent impulse response of this camera is: ， in It is located in The spread function of the camera point at the center of the plane is defined by the phase function as follows: ， The array camera measurement model is defined as follows: 。 2. The reflection-based Fourier stacking method based on a camera matrix according to claim 1, characterized in that, The specific method for obtaining images using a spatial light modulator in step one is as follows: create multiple vector clipping images, and generate multiple binary images through image enhancement methods such as rotation, flipping, and scaling.

3. The reflection-based Fourier stacking method based on a camera matrix according to claim 1, characterized in that, In step four, the collected low-resolution images are preprocessed and filtered through downsampling to create the training dataset and sub-datasets.

4. The image reconstruction network structure in step three of the camera matrix-based reflective Fourier stacking method according to claim 1 is characterized in that... The following loss was used: L = L BCE + βL SSIM。 5. The reflection-based Fourier stacking method based on a camera matrix according to claim 1, characterized in that, The camera array acquires 16 low-resolution images at a time.

Images