A light field super-resolution three-dimensional reconstruction method and system

Abstract

The present invention discloses a light field super-resolution 3D reconstruction method and system. The method comprises the following steps: (1) taking a static sample as a target, obtaining its high-resolution actual 3D data; (2) performing simulated sampling, obtaining a low-resolution light field projection matching the hardware of the light field imaging system and a high-resolution light field projection of the target resolution; (3) training a super-resolution network model as training data until the model converges to obtain a light field super-resolution reconstruction model; (4) inputting the light field image data obtained by the light field imaging system with a static or dynamic sample as a target into the light field super-resolution reconstruction model, obtaining a light field image of the target resolution; (5) performing 3D reconstruction, obtaining a high-resolution 3D image of the static or dynamic sample. The present invention can obtain a light field multi-view image beyond the diffraction limit at an ultra-high speed, and finally realize microscopic imaging of dynamic samples, such as sub-micron biological dynamic processes.

Description

Technical Field

[0001] The present invention belongs to the field of biophoton microscopic imaging, and more specifically, relates to a light field super-resolution three-dimensional reconstruction method and system. Background Art

[0002] Light field microscopy has an extremely high volume imaging rate. Compared with conventional high-resolution three-dimensional imaging technology that needs to scan and collect multiple two-dimensional images to obtain three-dimensional information, it only needs to collect a single image to obtain the three-dimensional information of the image, which greatly improves the three-dimensional imaging speed and provides the possibility for high-speed dynamic imaging of living bodies. Light field microscopy technology adds a microlens array to the detection light path so that the acquired two-dimensional images can contain both lateral and angular information. However, since this method needs to obtain lateral and angular information simultaneously through a single image, the lateral information needs to be undersampled, and the spatial resolution is lower than that of conventional three-dimensional imaging technology.

[0003] In order to compensate for the low spatial resolution of light field microscopy, a super-resolution algorithm based on deep learning is currently used to perform light field super-resolution three-dimensional imaging. However, the current two-dimensional light field image super-resolution algorithm based on deep learning uses a high-specification light field camera to collect a high-resolution light field image as the target image, and then simulates it as a low-resolution light field image through degradation, and then pairs the two for training to improve the spatial resolution of the light field image. Although the above method is committed to improving the spatial resolution, it is still limited by the resolution limit of the training samples and cannot break through the diffraction limit of the light field imaging principle. Compared with other three-dimensional imaging methods, the resolution capability is still relatively low. This makes it difficult for current light field imaging technology to observe many submicron biological dynamic processes. Summary of the invention

[0004] In response to the above defects or improvement needs of the prior art, the present invention provides a light field super-resolution three-dimensional reconstruction method and system, the purpose of which is to simulate and generate light field images beyond the diffraction limit that cannot be obtained by actual shooting, and pair them with light field images that can be obtained by actual shooting, while maintaining the advantage of light field in quickly collecting three-dimensional information, the spatial resolution exceeds the diffraction limit of the light field, thereby solving the technical problem that the existing high-resolution three-dimensional imaging technology cannot perform microscopic imaging of dynamic samples.

[0005] To achieve the above object, according to one aspect of the present invention, a light field super-resolution 3D reconstruction method is provided, which is characterized by comprising the following steps:

[0006] (1) Taking static samples as targets, obtain their high-resolution three-dimensional data;

[0007] (2) performing simulation sampling on the high-resolution actual three-dimensional data obtained in step (1) to obtain a low-resolution light field projection that matches the hardware of the light field imaging system and a high-resolution light field projection of the target resolution;

[0008] (3) using the low-resolution light field projection matched with the light field imaging system hardware obtained in step (2) and the high-resolution light field projection of the target resolution as training data, training the super-resolution network model until the model converges to obtain a light field super-resolution reconstruction model;

[0009] (4) inputting the light field image data obtained by the light field imaging system with the static or dynamic sample as the target into the light field super-resolution reconstruction model obtained in step (3) to obtain a light field image with the target resolution;

[0010] (5) Performing three-dimensional reconstruction on the target resolution light field image obtained in step (4) to obtain a high-resolution three-dimensional image of the static or dynamic sample.

[0011] Preferably, in the light field super-resolution 3D reconstruction method, the low-resolution light field projection matching the light field imaging system hardware is obtained according to the following method:

[0012] The high-resolution actual three-dimensional data is subjected to light field projection according to the theoretical wave optics model to obtain the low-resolution light field projection that matches the light field imaging system hardware; N is the acquisition period at the same angle, that is, each interval of N pixels is an image of the same viewing angle.

[0013] Preferably, in the light field super-resolution 3D reconstruction method, the high-resolution light field projection of the target resolution is obtained according to the following method:

[0014] After performing N-fold interpolation on the high-resolution actual three-dimensional data, the interpolated high-resolution three-dimensional data is obtained, and the interpolated high-resolution three-dimensional data is undersampled N times by light field projection according to the theoretical wave optics model to obtain a high-resolution light field projection of the target resolution; N is the acquisition period at the same angle, that is, each interval of N pixels is an image of the same viewing angle; or:

[0015] The high-resolution actual three-dimensional data is rotated and projected to obtain simulated light field parallax images of different viewing angles, and the images are respectively aligned with the viewing angle images of the low-resolution light field projection that matches the light field imaging system hardware to obtain the angle information of the three-dimensional data; and a high-resolution light field projection of the target resolution is constructed.

[0016] Preferably, in the light field super-resolution 3D reconstruction method, the super-resolution network model is used to perform super-resolution reconstruction on the images of each viewing angle of the low-resolution light field projection and the high-resolution light field projection; preferably includes a light field image information extraction module and an image super-resolution module;

[0017] The light field image information extraction module uses convolution modules with residual connections of different sizes to act on each perspective image of the low-resolution light field projection, perform multiple convolutions and multi-perspective fusion, obtain feature fusion information of different perspectives, and submit it to the image super-resolution module;

[0018] The image super-resolution module is used to up-sample the feature fusion information of different viewing angles to obtain a high-resolution light field projection.

[0019] Preferably, in the light field super-resolution 3D reconstruction method, the loss function Loss used in training the super-resolution network model is: the average of the loss function values ​​of each viewing angle image of the low-resolution light field projection and each viewing angle image of the high-resolution light field projection.

[0020] According to another aspect of the present invention, a light field super-resolution 3D reconstruction system is provided, which includes a super-resolution network model; training data acquisition of the super-resolution network model: taking a static sample as a target, acquiring its high-resolution actual 3D data; performing simulated sampling on the high-resolution actual 3D data to obtain a low-resolution light field projection that matches the hardware of a light field imaging system and a high-resolution light field projection at a target resolution; using the low-resolution light field projection that matches the hardware of the light field imaging system and the high-resolution light field projection at the target resolution as training data.

[0021] Preferably, in the light field super-resolution 3D reconstruction system, the loss function of the super-resolution network model is selected as follows: the loss function Loss used for training the super-resolution network model is: the average of the loss function values ​​of each viewing angle image of the low-resolution light field projection and each viewing angle image of the high-resolution light field projection.

[0022] Preferably, the super-resolution network model of the light field super-resolution 3D reconstruction system comprises a light field image information extraction module and an image super-resolution module;

[0023] The light field image information extraction module uses convolution modules with residual connections of different sizes to act on each perspective image of the low-resolution light field projection, perform multiple convolutions and multi-perspective fusion, obtain feature fusion information of different perspectives, and submit it to the image super-resolution module;

[0024] The image super-resolution module is used to up-sample the feature fusion information of different viewing angles to obtain a high-resolution light field projection.

[0025] According to another aspect of the present invention, a lossless simulated light field projection acquisition method is provided, which comprises the following steps:

[0026] After interpolating the actual three-dimensional data N times, the interpolated three-dimensional data is obtained, and the interpolated three-dimensional data is undersampled N times for light field projection according to the theoretical wave optics model to obtain the lossless simulated light field projection; N is the acquisition period at the same angle, that is, every N pixels are the same viewing angle image.

[0027] According to another aspect of the present invention, a lossless simulated light field projection acquisition method is provided, which comprises the following steps:

[0028] The actual three-dimensional data is rotated and projected to obtain parallax images of different viewing angles of the simulated light field, which are respectively aligned with the viewing angle images obtained by light field projection of the actual three-dimensional data to obtain angle information of the three-dimensional data and construct the lossless simulated light field projection.

[0029] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0030] The present invention achieves super-resolution of light field images through simulation, pairs them with real-shot under-sampled light field images, and trains them with a deep learning network. This enables the acquisition of light field multi-view images beyond the diffraction limit at ultra-high speed, and ultimately realizes microscopic imaging of dynamic samples, such as sub-micron biological dynamic processes.

[0031] The preferred solution provides two methods for generating lossless simulated super-resolution light field images: first, interpolating the high-resolution actual three-dimensional data taken statically to simulate the sample after expansion, and then performing light field projection based on the theoretical wave optics model to simulate the light field downsampling process, and pairing it with the simulated low-resolution image of the light field before expansion, extracting each perspective of the paired data, and the high-resolution perspective image obtained at this time offsets the downsampling of the light field image, and has no resolution degradation compared to the actual three-dimensional data, while the actual low-resolution perspective image is downsampled N times, and finally, after training through a convolutional network, the light field image of the actual high-speed moving living sample is super-resolved. Second, according to the different perspectives of the light field, the high-resolution actual three-dimensional data taken statically is rotated and projected at different angles to simulate the non-downsampled high-resolution light field image of each perspective, and then paired with the corresponding perspective of the low-resolution light field, and finally learned through a convolutional network, and finally the light field image super-resolution of the actual high-speed moving living sample is achieved. The light field super-resolution reconstruction model is trained with lossless super-resolution light field images, and the training is easy to converge, and the training effect is good. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a light field super-resolution three-dimensional reconstruction method provided by an embodiment of the present invention;

[0033] Figure 2 A flow chart of a method for generating network training data provided by an embodiment of the present invention;

[0034] Figure 3 Flow chart of the light field 3D super-resolution reconstruction algorithm according to an embodiment of the present invention

[0035] Figure 4 This is a three-dimensional reconstruction effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0037] The light field super-resolution 3D reconstruction method provided by the present invention comprises the following steps:

[0038] (1) Taking static samples as the target, obtain their high-resolution actual three-dimensional data; if dynamic samples are taken as the target, the usual high-resolution three-dimensional image acquisition method relies on scanning multiple two-dimensional images to obtain three-dimensional information, such as light sheet tomography fluorescence imaging and confocal scanning fluorescence imaging. Since the three-dimensional image is acquired by scanning, the resolution of the final three-dimensional image is only limited by the imaging system, and no additional downsampling is introduced. However, it is impossible to acquire multiple two-dimensional images at the same time, so this high-resolution actual three-dimensional data can only be obtained from static samples, or ignoring the distortion caused by a certain delay, it can be obtained from samples with slower movement. If the dynamic sample is scanned and imaged, the motion blur will seriously affect the subsequent training process and training effect of the super-resolution imaging model, resulting in the failure of the model to converge.

[0039] (2) performing simulation sampling on the high-resolution actual three-dimensional data obtained in step (1) to obtain a low-resolution light field projection that matches the hardware of the light field imaging system and a high-resolution light field projection of the target resolution;

[0040] The low-resolution light field projection that matches the light field imaging system hardware is preferably obtained according to the following method:

[0041] The high-resolution actual three-dimensional data is subjected to light field projection according to the theoretical wave optics model to obtain the low-resolution light field projection that matches the light field imaging system hardware; N is the acquisition period at the same angle, that is, each interval of N pixels is an image of the same viewing angle, which is determined by the microlens parameters in the light field optical path and the pixel size of the acquisition camera.

[0042] The high-resolution light field projection of the target resolution is preferably obtained by the following method:

[0043] After N-fold interpolation of the high-resolution actual three-dimensional data, the interpolated high-resolution three-dimensional data is obtained, and the interpolated high-resolution three-dimensional data is undersampled N times by light field projection according to the theoretical wave optics model. At this time, the undersampling process is offset by the interpolation N times, and the high-resolution light field projection of the target resolution is obtained. The principle of light field undersampling is to take points in turn at every other pixel, and the intensity change of the image in the local area is very small, so downsampling after interpolation is almost lossless compared to directly sampling the original value.

[0044] or:

[0045] The high-resolution actual three-dimensional data is rotated and projected to obtain simulated light field parallax images of different viewing angles, which are respectively aligned with the viewing angle images of the low-resolution light field projection that matches the light field imaging system hardware to obtain the angle information of the three-dimensional data, and construct the high-resolution light field projection of the target resolution.

[0046] This step simulates sampling of the high-resolution actual 3D data, so that its lateral resolution exceeds the diffraction limit of ordinary light field images, and achieves the same lateral resolution as the original high-resolution 3D data while maintaining the angular resolution of the light field. Experiments show that if the resolution of the light field data is too low compared to the target resolution, even if a super-resolution 3D reconstruction network is used, the reconstruction effect is not ideal. The low-resolution light field projection obtained by the present invention that matches the hardware of the light field imaging system is used to perform subsequent super-resolution 3D reconstruction training on the light field image, and theoretically the best super-resolution 3D reconstruction effect can be obtained.

[0047] (3) using the low-resolution light field projection matched with the light field imaging system hardware obtained in step (2) and the high-resolution light field projection of the target resolution as training data, training the super-resolution network model until the model converges to obtain a light field super-resolution reconstruction model;

[0048] The super-resolution network model is used to perform super-resolution reconstruction on the images of each viewing angle of the low-resolution light field projection and the high-resolution light field projection; preferably includes a light field image information extraction module and an image super-resolution module;

[0049] The light field image information extraction module uses convolution modules with residual connections of different sizes to act on each perspective image of the low-resolution light field projection, perform multiple convolutions and multi-perspective fusion, obtain feature fusion information of different perspectives, and submit it to the image super-resolution module;

[0050] The image super-resolution module is used to up-sample the feature fusion information of different viewing angles to obtain a high-resolution light field projection.

[0051] The loss function Loss used in the training super-resolution network model is: the average of the loss function values ​​of each viewing angle of the low-resolution light field projection and each viewing angle of the high-resolution light field projection, written as:

[0052]

[0053] Among them, N is the number of training samples input to the network, x, y, u, and v refer to the spatial coordinates of the perspective image and the three-dimensional angle coordinates of the perspective image, respectively. is the network prediction result, I is the high-resolution target image, and Loss EPI is the perspective constraint loss function, || ||1 is the L1 norm, that is, the absolute value is calculated, Differentiate in x, y, u, and v dimensions respectively. Polar diagram for network prediction, E polar diagram for target image.

[0054] During training, the learning rate is gradually reduced according to the number of training cycles to prevent the network loss function from oscillating and difficult to converge.

[0055] (4) inputting the light field image data obtained by the light field imaging system with the static or dynamic sample as the target into the light field super-resolution reconstruction model obtained in step (3) to obtain a light field image with the target resolution;

[0056] (5) Performing three-dimensional reconstruction on the target resolution light field image obtained in step (4) to obtain a high-resolution three-dimensional image of the static or dynamic sample.

[0057] The light field super-resolution 3D reconstruction system provided by the present invention comprises a super-resolution network model; the super-resolution network model preferably comprises a light field image information extraction module and an image super-resolution module;

[0058] The light field image information extraction module uses convolution modules with residual connections of different sizes to act on each perspective image of the low-resolution light field projection, perform multiple convolutions and multi-perspective fusion, obtain feature fusion information of different perspectives, and submit it to the image super-resolution module;

[0059] The image super-resolution module is used to up-sample the feature fusion information of different viewing angles to obtain a high-resolution light field projection.

[0060] Train as follows:

[0061] Acquiring training data: Taking a static sample as a target, acquiring its high-resolution actual three-dimensional data; performing simulated sampling on the high-resolution actual three-dimensional data to obtain a low-resolution light field projection that matches the hardware of the light field imaging system and a high-resolution light field projection of the target resolution; using the low-resolution light field projection that matches the hardware of the light field imaging system and the high-resolution light field projection of the target resolution as training data;

[0062] Loss function selection: The loss function Loss used in the training super-resolution network model is: the average of the loss function values ​​of each viewing angle of the low-resolution light field projection and each viewing angle of the high-resolution light field projection, written as:

[0063]

[0064] Among them, N is the number of training samples input to the network, x, y, u, and v refer to the spatial coordinates of the perspective image and the three-dimensional angle coordinates of the perspective image, respectively. is the network prediction result, I is the high-resolution target image, and Loss EPI is the perspective constraint loss function, || ||1 is the L1 norm, that is, the absolute value is calculated, Differentiate in x, y, u, and v dimensions respectively. Polar diagram for network prediction, E polar diagram for target image.

[0065] Training convergence: During training, the learning rate is gradually reduced according to the number of training cycles to prevent the network loss function from oscillating and difficult to converge.

[0066] The lossless simulated light field projection acquisition method provided by the present invention comprises the following steps:

[0067] After interpolating the actual three-dimensional data N times, the interpolated three-dimensional data is obtained, and the interpolated three-dimensional data is undersampled N times for light field projection according to the theoretical wave optics model to obtain the lossless simulated light field projection; N is the acquisition period at the same angle, that is, every N pixels are the same viewing angle image, which is determined by the microlens parameters in the light field optical path and the pixel size of the acquisition camera.

[0068] The lossless simulated light field projection acquisition method provided by the present invention comprises the following steps:

[0069] The actual three-dimensional data is rotated and projected to obtain simulated light field parallax images of different viewing angles, which are respectively aligned with the viewing angle images obtained by light field projection of the actual three-dimensional data to obtain angle information of the three-dimensional data and construct the lossless simulated light field projection.

[0070] The following are examples:

[0071] Example 1

[0072] This embodiment provides a light field super-resolution 3D reconstruction method. Figure 1 As shown, the simulated high-resolution light field image is paired with the corresponding low-resolution light field image, and the paired high-resolution and low-resolution multi-view images are extracted and input into the convolutional neural network for training. The super-resolution of the light field image is achieved through network forward reasoning, and the super-resolution 3D image is obtained by combining the light field 3D reconstruction network, which can perform high-resolution 3D imaging of dynamic samples. The following steps are included:

[0073] (1) Taking static samples as targets, obtain their high-resolution three-dimensional data;

[0074] (2) performing simulation sampling on the high-resolution actual three-dimensional data obtained in step (1) to obtain a low-resolution light field projection that matches the hardware of the light field imaging system and a high-resolution light field projection of the target resolution;

[0075] Specifically, the following method is used to generate training data:

[0076] like Figure 2 As shown, a low-resolution light field projection acquisition that matches the light field imaging system hardware:

[0077] Based on the theoretical wave optics model, the static high-resolution actual 3D data is projected with light field to generate a simulated low-resolution light field image. The low-resolution light field image size (number of pixels) is the same as the high-resolution actual 3D data, but it also contains the 2D spatial projection information and 3D angle information of the 3D data. Then, based on the optical principle of the light field imaging system, the light field image is extracted from the perspective. The specific method is as follows: Figure 1 As shown in , at this time, N*N low-resolution perspective images are obtained. Each perspective is downsampled N times relative to the original high-resolution 3D data, and this data is used as the low-resolution input data of the network. The specific calculation method is as follows:

[0078] First, the point spread function required for light field projection is calculated according to the actual light field optical path. Based on the theoretical wave optics model (Michael Broxton, Logan Grosenick, Samuel Yang, Noy Cohen, Aaron Andalman, Karl Deisseroth, and Marc Levoy, "Wave optics theory and 3-D deconvolution for the light field microscope," Opt. Express 21, 25418-25439 (2013)), the statically captured high-resolution actual three-dimensional data is projected to generate a simulated low-resolution light field image. At this time, the low-resolution light field image has the same size (number of pixels) as the high-resolution actual three-dimensional data, but also contains the two-dimensional projection information and three-dimensional angle information of the three-dimensional data.

[0079] Then, based on the principle of light field imaging, the light field image is extracted from the perspective. The specific method is as follows: Figure 1 As shown in , every N pixels are the same perspective image, N is determined by the microlens parameters in the light field optical path and the pixel size of the acquisition camera. At this time, low-resolution perspective images of N*N perspectives are obtained. Each perspective is downsampled N times relative to the original high-resolution three-dimensional data, and this data is used as the low-resolution input data of the network.

[0080] High-resolution light field projection acquisition matching the target resolution:

[0081] In order to offset the downsampling process of the light field, the high-resolution actual three-dimensional data is interpolated N times to simulate the expanded samples, and then the light field image is simulated by the same point spread function for light field projection, and then the perspective extraction is performed. The perspective image obtained at this time just offsets the N-fold undersampling of the light field image, and obtains a simulated light field image that is not undersampled and exceeds the diffraction limit that cannot be collected in reality. In this way, a set of low-resolution light field perspective images and corresponding lossless high-resolution light field perspective images can be obtained as the target training set. The specific calculation is as follows:

[0082] First, the high-resolution actual 3D data is interpolated N times using the bicubic interpolation algorithm to simulate the sample after dilation N times, and then the light field projection is simulated using the same point spread function as the low-resolution light field data, and the view extraction is performed in the same way. At this time, the number of view angles of the view image obtained remains unchanged at N*N, and the parallax of each view angle remains unchanged, but the sampling rate of each view angle is increased by N times compared to the low-resolution light field image, which is consistent with the original high-resolution 3D image.

[0083] (3) using the low-resolution light field projection matched with the light field imaging system hardware obtained in step (2) and the high-resolution light field projection of the target resolution as training data, training the super-resolution network model until the model converges to obtain a light field super-resolution reconstruction model;

[0084] (4) inputting the light field image data obtained by the light field imaging system with the static or dynamic sample as the target into the light field super-resolution reconstruction model obtained in step (3) to obtain a light field image with the target resolution;

[0085] (5) Performing three-dimensional reconstruction on the target resolution light field image obtained in step (4) to obtain a high-resolution three-dimensional image of the static or dynamic sample. The specific method is as follows:

[0086] After obtaining the light field super-resolution image, it is arranged to obtain a multi-view image and input into the 3D image reconstruction module, such as Figure 3 As shown. First, the image is subjected to light field image extraction. Different convolution operations are set according to the light field image parameters: dilated convolution and conventional convolution. The former is used to expand the convolution receptive field to extract the correlation information between different perspectives (Multi-Scale Context Aggregation by Dilated Convolutions), while the latter uses non-dilated convolution to extract spatial information within a certain perspective. After the continuous action of the two, the multi-view image enters the image rearrangement and upsampling module. Each single-view image is extracted from the multi-view image and stacked to form a three-dimensional image stack. The sub-pixel convolution is progressively upsampled to make the lateral size of the single-view image consistent with the lateral size of the three-dimensional image. The stack is then converted into a three-dimensional spatially distributed image stack through a U-Net convolutional neural network based on residual connections. (Unet structure reference: MultiResUNet: Rethinking the U-Net architecture for multimodal biomedicalimage segmentation)

[0087] Example 2

[0088] This embodiment provides a light field super-resolution 3D reconstruction method. Figure 1 As shown, the simulated high-resolution light field image is paired with the corresponding low-resolution light field image, and the paired high-resolution and low-resolution multi-view images are extracted and input into the convolutional neural network for training. The super-resolution of the light field image is achieved through network forward reasoning, and the super-resolution 3D image is obtained by combining the light field 3D reconstruction network. The following steps are included:

[0089] (1) Taking static samples as targets, obtain their high-resolution three-dimensional data;

[0090] (2) performing simulation sampling on the high-resolution actual three-dimensional data obtained in step (1) to obtain a low-resolution light field projection that matches the hardware of the light field imaging system and a high-resolution light field projection of the target resolution;

[0091] Specifically, the following method is used to generate training data:

[0092] like Figure 2 As shown, the low-resolution light field projection matching the light field imaging hardware is acquired:

[0093] Based on the theoretical wave optics model, the static high-resolution actual 3D data is projected with light field to generate a simulated low-resolution light field image. The size (number of pixels) of the low-resolution light field image is the same as the high-resolution actual 3D data, but it also contains the 2D projection information and 3D angle information of the 3D data. Then, based on the optical principle of the light field image, the light field image is extracted from the perspective. The specific method is as follows: Figure 1 As shown in , at this time, N*N low-resolution perspective images are obtained. Each perspective is downsampled N times relative to the original high-resolution 3D data, and this data is used as the low-resolution input data of the network. The specific calculation method is as follows:

[0094] First, the point spread function required for light field projection is calculated according to the actual light field optical path (Michael Broxton, Logan Grosenick, Samuel Yang, Noy Cohen, Aaron Andalman, Karl Deisseroth, and Marc Levoy, "Wave optics theory and 3-D deconvolution for the light field microscope," Opt. Express 21, 25418-25439 (2013)). Based on the theoretical wave optics model, the statically captured high-resolution actual three-dimensional data is projected into a light field to generate a simulated low-resolution light field image. At this time, the low-resolution light field image has the same size (number of pixels) as the high-resolution actual three-dimensional data, but also contains the two-dimensional projection information and three-dimensional angle information of the three-dimensional data.

[0095] Then, based on the optical principle of light field images, the light field image is extracted from the perspective. The specific method is as follows: Figure 1 As shown in , every N pixels are the same perspective image, N is determined by the microlens parameters in the light field optical path and the pixel size of the acquisition camera. At this time, low-resolution perspective images of N*N perspectives are obtained. Each perspective is downsampled N times relative to the original high-resolution three-dimensional data, and this data is used as the low-resolution input data of the network.

[0096] The light field projection of high-resolution green matching the target resolution is obtained:

[0097] The high-resolution actual 3D data is projected in 3D rotation to simulate different light field perspectives, and a high-resolution perspective image is generated that corresponds to the low-resolution perspective image one by one and has the same parallax. The high-resolution perspective image generated in this way obtains the angle information of the 3D data while maintaining the high resolution of the original 3D data. Thus, a set of low-resolution light field perspective images and corresponding lossless high-resolution light field perspective images can be obtained as the target training set. The details are as follows:

[0098] The high-resolution actual three-dimensional data is subjected to three-dimensional rotation projection, and the rotation angles of different viewing angles are calculated according to the light field theory to simulate the parallax of the light field at different viewing angles. Then, the three-dimensional data at this angle is intensity superimposed and projected to generate N*N high-resolution viewing angle images that correspond one to one with the low-resolution viewing angle images and have consistent parallax. The lateral resolution of the high-resolution viewing angle images generated in this way maintains the same resolution as the original high-resolution three-dimensional data, and is consistent with the angle information obtained from the low-resolution data.

[0099] (3) using the low-resolution light field projection matched with the light field imaging system hardware obtained in step (2) and the high-resolution light field projection of the target resolution as training data, training the super-resolution network model until the model converges to obtain a light field super-resolution reconstruction model;

[0100] (4) inputting the light field image data obtained by the light field imaging system with the static or dynamic sample as the target into the light field super-resolution reconstruction model obtained in step (3) to obtain a light field image with the target resolution;

[0101] (5) Performing three-dimensional reconstruction on the target resolution light field image obtained in step (4) to obtain a high-resolution three-dimensional image of the static or dynamic sample. The specific steps are the same as those of the three-dimensional reconstruction in Example 1.

[0102] The light field super-resolution method based on a neural network provided by the present invention can super-resolution a low-resolution light field image, so that its lateral resolution exceeds the diffraction limit of the light field imaging system, while maintaining the angular resolution of the light field, achieving the same lateral resolution as the original high-resolution three-dimensional data, and further improving the quality of subsequent three-dimensional reconstruction results. It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A light field super-resolution 3D reconstruction method, characterized in that: The following steps are involved: (1) Taking static samples as targets, obtain their high-resolution three-dimensional data; (2) performing simulation sampling on the high-resolution actual three-dimensional data obtained in step (1) to obtain a low-resolution light field projection that matches the hardware of the light field imaging system and a high-resolution light field projection of the target resolution; (3) using the low-resolution light field projection matched with the light field imaging system hardware obtained in step (2) and the high-resolution light field projection of the target resolution as training data, training the super-resolution network model until the model converges to obtain a light field super-resolution reconstruction model; (4) inputting the light field image data obtained by the light field imaging system with the static or dynamic sample as the target into the light field super-resolution reconstruction model obtained in step (3) to obtain a light field image with the target resolution; (5) Performing three-dimensional reconstruction on the target resolution light field image obtained in step (4) to obtain a high-resolution three-dimensional image of the static or dynamic sample.

2. The light field super-resolution 3D reconstruction method according to claim 1, characterized in that: The low-resolution light field projection matching the light field imaging system hardware is obtained according to the following method: The high-resolution actual three-dimensional data is subjected to light field projection according to the theoretical wave optics model to obtain the low-resolution light field projection that matches the light field imaging system hardware; N is the acquisition period at the same angle, that is, each interval of N pixels is an image of the same viewing angle.

3. The light field super-resolution 3D reconstruction method according to claim 1, characterized in that: The high-resolution light field projection of the target resolution is obtained according to the following method: After performing N-fold interpolation on the high-resolution actual three-dimensional data, the interpolated high-resolution three-dimensional data is obtained, and the interpolated high-resolution three-dimensional data is undersampled N times by light field projection according to a theoretical wave optics model to obtain a high-resolution light field projection of the target resolution; N is an acquisition period at the same angle, that is, each interval of N pixels is an image at the same viewing angle; or: The high-resolution actual three-dimensional data is rotated and projected to obtain simulated light field parallax images of different viewing angles, and the images are respectively aligned with the viewing angle images of the low-resolution light field projection that matches the light field imaging system hardware to obtain the angle information of the three-dimensional data; and a high-resolution light field projection of the target resolution is constructed.

4. The light field super-resolution 3D reconstruction method according to claim 1, characterized in that: The super-resolution network model is used to perform super-resolution reconstruction on the images of each viewing angle of the low-resolution light field projection and the high-resolution light field projection; preferably includes a light field image information extraction module and an image super-resolution module; The light field image information extraction module uses convolution modules with residual connections of different sizes to act on each perspective image of the low-resolution light field projection, perform multiple convolutions and multi-perspective fusion, obtain feature fusion information of different perspectives, and submit it to the image super-resolution module; The image super-resolution module is used to up-sample the feature fusion information of different viewing angles to obtain a high-resolution light field projection.

5. The light field super-resolution 3D reconstruction method according to claim 4, characterized in that: The loss function Loss used in the training super-resolution network model is: the average of the loss function values ​​of each viewing angle image of the low-resolution light field projection and each viewing angle image of the high-resolution light field projection.

6. A light field super-resolution 3D reconstruction system, characterized in that: The steps of implementing the light field super-resolution 3D reconstruction method as described in any one of claims 1 to 5 include: a super-resolution network model; obtaining training data for the super-resolution network model: taking a static sample as a target, obtaining its high-resolution actual 3D data; performing simulated sampling on the high-resolution actual 3D data to obtain a low-resolution light field projection that matches the hardware of a light field imaging system and a high-resolution light field projection at a target resolution; using the low-resolution light field projection that matches the hardware of the light field imaging system and the high-resolution light field projection at the target resolution as training data.

7. The light field super-resolution 3D reconstruction system according to claim 6, characterized in that: Selection of loss function of the super-resolution network model: The loss function Loss used in training the super-resolution network model is: the average of the loss function values ​​of each viewing angle image of the low-resolution light field projection and each viewing angle image of the high-resolution light field projection.

8. The light field super-resolution 3D reconstruction system according to claim 6, characterized in that: The super-resolution network model includes a light field image information extraction module and an image super-resolution module; The light field image information extraction module uses convolution modules with residual connections of different sizes to act on each perspective image of the low-resolution light field projection, perform multiple convolutions and multi-perspective fusion, obtain feature fusion information of different perspectives, and submit it to the image super-resolution module; The image super-resolution module is used to up-sample the feature fusion information of different viewing angles to obtain a high-resolution light field projection.

Images