A Fourier light field microscope super-resolution imaging method and system

Abstract

The present invention discloses a Fourier light field microscope super-resolution imaging method and system. The method comprises the following steps: (1) inputting a single multi-view light field image with a low angular sampling rate obtained by a Fourier light field microscope into an angle difference network to obtain multiple single-view light field images with a high angular sampling rate; (2) reconstructing the multiple single-view light field images with a high angular sampling rate obtained in step (1) into three-dimensional images on a perspective basis, and superimposing them in the same order as the multiple single-view images in the training sample sampling data of the angle difference network to obtain the Fourier light field microscope super-resolution imaging. The present invention provides a Fourier light field microscope super-resolution imaging method, which interpolates more perspective images in the actual captured light field image through a deep learning algorithm without generating crosstalk, and simultaneously obtains a light field PSF that matches the light field image, and then uses a three-dimensional reconstruction algorithm to obtain a high-quality reconstruction result.

Description

Technical Field

[0001] The present invention belongs to the field of biophotonics, and more specifically, relates to a Fourier light field microscope super-resolution imaging method and system. Background Art

[0002] Recently, there has been growing interest in the development of Fourier light-field microscopy (FLFM), which can record the spatial and angular information of a sample in a single exposure, and then reconstruct the sample's 3D volume using a subsequent reconstruction algorithm. This strategy effectively overcomes the fundamental limitations of traditional light-field microscopy (LFM) in terms of reconstruction artifacts and computational cost, significantly improving the reconstruction quality.

[0003] However, in order to obtain higher lateral spatial resolution and field of view, Fourier light field microscopy loses its perspective information, and obtaining more perspective information is the key to obtaining high-resolution, high-fidelity three-dimensional reconstructions. For example, in optical projection tomography (OPT) technology, researchers attempt to obtain projection views of the sample from more angles to obtain higher-resolution three-dimensional reconstructions. However, this is not easy to achieve in Fourier light field microscopy. Considering the spatial barrier limitations of the field of view and aperture (large lens) of the Fourier light field microscope, it is impossible to simultaneously obtain images from more angles without perspective crosstalk, resulting in its imaging resolution being unable to be further improved. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a Fourier light field microscope super-resolution imaging method and system, the purpose of which is to use a super-resolution method to simulate and improve the angular resolution, simulate the microlens array of Fourier square microscopy that cannot be arranged in practice, and thus reconstruct a super-resolution Fourier square microscopic image, thereby solving the market problem of Fourier square microscopes and the technical problem that the spatial barrier limitation of the aperture and the crosstalk of viewing angle make it difficult to further improve the resolution.

[0005] To achieve the above object, according to one aspect of the present invention, a Fourier light field microscope super-resolution imaging method is provided, comprising the following steps:

[0006] (1) A single multi-view light field image with a low angular sampling rate obtained by a Fourier light field microscope is input into an angle difference network to obtain multiple single-view light field images with a high angular sampling rate;

[0007] (2) reconstructing the multiple single-view light field images with high angular sampling rates obtained in step (1) into three-dimensional images perspective by perspective, and superimposing them in the same order as the multiple single-view images in the training sample sampling data of the angle difference network to obtain the Fourier light field microscope super-resolution imaging.

[0008] Preferably, the Fourier light field microscope super-resolution imaging method, step (1) is specifically:

[0009] Iterate the multi-view light field image with a low angular sampling rate and the angular interpolation images of adjacent views as the initial multi-view light field image with a high angular sampling rate; the angular interpolation images of adjacent views are interpolated in the angular dimension using an interpolation algorithm such as bi-tri interpolation of the light field images of adjacent views;

[0010] The angle difference network is based on a deep learning model and includes a fusion channel and a convolution channel. The fusion channel is used to superimpose the original low-angle sampling rate image stack and the interpolated image stack in the channel dimension, and extract high-dimensional information through multiple convolution processes, including multiple fusion modules; the convolution channel includes multiple convolution modules, which are used to further extract information from the front-end fusion module and are composed of multiple convolution layers.

[0011] Preferably, the Fourier light field microscope super-resolution imaging method, wherein the angle difference network based on the deep learning model is trained according to the following method:

[0012] (1-1) Obtain training samples:

[0013] (1-1-1) Taking a static or dynamic sample as a target, photographing it with a high-resolution three-dimensional imaging microscope to obtain a high-resolution three-dimensional image; the high-resolution three-dimensional imaging microscope includes but is not limited to a wide-field microscope and a confocal microscope;

[0014] (1-1-2) obtaining a point spread function of the hardware Fourier light field microscopy system according to the optical parameters of the hardware Fourier light field microscopy system, performing low angular sampling rate fluctuation optical simulation on the high-resolution actual three-dimensional image obtained in step (1-1-1) using the point spread function, and obtaining a single point spread function image of the hardware Fourier light field microscopy system, wherein the single point spread function image includes a multi-view point spread image array with low angular sampling, and the multi-view point spread image array corresponds to the microlens array of the hardware Fourier light field microscopy system;

[0015] In a preferred embodiment, pixel resampling is performed on the actually captured image to obtain the high-resolution actual three-dimensional image;

[0016] (1-1-3) obtaining a point spread function of the imaginary Fourier light field microscopy system based on the optical parameters of the imaginary Fourier light field microscopy system, using the point spread function to perform high angular sampling rate wave optics simulation on the high-resolution actual three-dimensional image obtained in step (1-1-1), and obtaining a plurality of single-viewpoint point spread function images of the imaginary Fourier light field microscopy system, wherein each point spread function image includes a point spread image of one viewpoint with high angular sampling; the plurality of point spread function images correspond to the microlens array of the imaginary Fourier light field microscopy system;

[0017] (1-1-4) performing perspective-by-perspective convolution on the high-resolution three-dimensional image obtained in step (1-1-1) with the single multi-perspective point spread function image obtained in step (1-1-2) and the multiple single-perspective point spread function images obtained in step (1-1-3), respectively, to obtain a single multi-perspective low angular sampling rate light field image and multiple single-perspective high angular sampling rate light field images for forward projection of the actual three-dimensional image, and using the single multi-perspective low angular sampling rate light field image as the input of a training sample, and using its corresponding multiple single-perspective high angular sampling rate light field images as the output of the training sample;

[0018] A single multi-view low angular sampling rate light field image, each view is represented as:

[0019] Projection low ={Resize(O(x,y,z))*|E(x,y,v,z)| 2} v＝1,2,...,v0

[0020] Multiple single-view high-angular sampling rate light field images, the images of each view are represented as follows:

[0021] Projection high ={Resize(O(x,y,z))*|E(x,y,v,z)| 2} v＝1,2,...,ratio·v0

[0022] Projection high Projection low are light field projection images with high and low angular sampling rates, respectively. Resize is resampling, O(x, y, z) is a three-dimensional image, E(x, y, v, z) is the propagation function after the microlens described above, and v0 is the number of viewing angles at the original angular sampling rate.

[0023] (1-2) Model training.

[0024] Preferably, in the Fourier light field microscope super-resolution imaging method, the microlens positions simulated by the imaginary Fourier light field microscopy imaging system are arranged at integer magnification differences of the microlens positions of the hardware Fourier light field microscopy imaging system.

[0025] Preferably, in the Fourier light field microscope super-resolution imaging method, the microlens spacing for simulating the multi-view light field image with a high angular sampling rate can be calculated based on the angular interpolation magnification as follows:

[0026] Δd=d / ratio

[0027] Wherein d is the actual micro-lens pitch, that is, the micro-lens pitch of the lens array simulated by the multi-view light field image with low angular sampling, and ratio is the angular interpolation ratio.

[0028] Preferably, the Fourier light field microscope super-resolution imaging method, for any Fourier light field microscope imaging system, obtains the point spread function given by the imaging system according to its optical parameters, specifically:

[0029] S1. Obtain the spatial positions of all microlenses in the lens array for multi-view light field image simulation through the microlens spacing and microlens arrangement.

[0030] S2. Determine the transmittance function T based on the optical parameters of the microlens MLA , and according to the wave optics theory, calculate the point spread function PSF (x, y, v) after different microlenses, that is:

[0031] PSF(x,y,v)=|E(x,y,v;z))| 2

[0032] =Fresnel l1 (E1(x,y;z)×T MLA (x,y,v;x v ,y v ))

[0033] Where E1 is the electric field distribution of the point source O(x, y, z) propagating to the microlens plane, T MLA is the transmittance function of a single simulated microlens, x v ,y v is the center coordinate of a microlens in the densely sampled microlens array, Fresnel l1 is the Fresnel diffraction propagation distance l1, the distance from the l1 microlens plane to the camera plane.

[0034] Preferably, in the Fourier light field microscope super-resolution imaging method, the microlens arrangement adopted by the Fourier light field microscope imaging system is: hexagonal arrangement or square orthogonal arrangement.

[0035] Preferably, in the Fourier light field microscope super-resolution imaging method, the steps (1-2) are specifically as follows:

[0036] Through forward propagation through a dense residual network, we finally obtain a network output that is consistent with the image size of the high-angle sampling rate, and complete the iterative convergence of the network model by optimizing the following loss function;

[0037]

[0038] Where N is the number of pixels in the view image stack, and y is They are respectively the high-resolution actual three-dimensional image obtained in step (1-1-1) and the three-dimensional image reconstructed from multiple single-view light field images with a high angular sampling rate output by the deep learning model angle difference network.

[0039] Preferably, in the Fourier light field microscope super-resolution imaging method, the reconstruction into a three-dimensional image is specifically performed by: performing an iterative deconvolution operation on a single single-view light field image and its corresponding point spread function image of the viewing angle to obtain a three-dimensional image of the viewing angle;

[0040] The stacking of the three-dimensional images in the training order is specifically: stacking and splicing the three-dimensional images of each perspective in the training order according to the spatial position, which not only avoids crosstalk between perspectives, but also effectively corrects the three-dimensional image, thereby improving the imaging quality.

[0041] According to another aspect of the present invention, there is provided a Fourier light field microscope super-resolution imaging system, comprising an angle difference network and a perspective-by-perspective reconstruction module;

[0042] The angle difference network is used to intelligently interpolate a single multi-view light field image with a low angular sampling rate acquired by the Fourier light field microscope to obtain multiple single-view light field images with a high angular sampling rate;

[0043] The perspective-by-perspective reconstruction module is used to reconstruct multiple single-perspective light field images with high angular sampling rates output by the angle difference network into three-dimensional images, and superimpose them in the same order as the multiple single-perspective images in the training sample sampling data of the angle difference network to obtain the Fourier light field microscope super-resolution imaging.

[0044] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0045] The present invention provides a Fourier light field microscope super-resolution imaging method, which uses a deep learning algorithm to interpolate more perspective images into the actual captured light field image without generating crosstalk, while obtaining a matching light field PSF, and then uses a three-dimensional reconstruction algorithm to obtain high-quality reconstruction results.

[0046] The present invention combines the principle of light field imaging, based on the multi-view generation algorithm and the angular interpolation algorithm, converts the multi-view image with low angular sampling rate into the multi-view image with dense sampling, thereby reducing the

[0047] It alleviates the problem of reconstruction artifacts of traditional deconvolution algorithms in low angular sampling rate images, improves its axial tomography capability, and overcomes the limitations of traditional light field imaging technology in observing high-speed dynamic biological processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the angle difference network structure provided by an embodiment of the present invention;

[0049] Figure 2 Schematic diagram of the microlens distribution of the hardware Fourier light field microscopy imaging system provided by an embodiment of the present invention;

[0050] Figure 3 Schematic diagram of the microlens distribution of an imaginary Fourier light field microscopy imaging system provided by an embodiment of the present invention;

[0051] Figure 4 2 is a schematic diagram of a single-view, high-angular sampling rate light field image provided by an embodiment of the present invention;

[0052] Figure 5 This is a diagram showing the reconstruction effect of the Fourier light field microscope super-resolution imaging method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0054] The present invention provides a Fourier light field microscope super-resolution imaging method, comprising the following steps:

[0055] (1) A single multi-view light field image with a low angular sampling rate obtained by a Fourier light field microscope is input into an angular difference network based on a deep learning model to obtain multiple single-view light field images with a high angular sampling rate; in a preferred embodiment, the multi-view light field image with a low angular sampling rate and the angular interpolation images of adjacent views thereof are used as the initial multi-view light field image with a high angular sampling rate for iteration; the angular interpolation images of adjacent views can be obtained by interpolation in the angular dimension using an interpolation algorithm such as double-triple interpolation of the light field images of adjacent views.

[0056] The angle difference network based on the deep learning model is as follows Figure 1 As shown, it includes a fusion channel and a convolution channel, wherein: the fusion channel is used to superimpose the original low-angular sampling rate image stack and the interpolation image stack in the channel dimension, and extract high-dimensional information through multiple convolution processes, including multiple fusion modules; the convolution channel includes multiple convolution modules, which are used to further extract information from the front-end fusion module and are composed of multiple convolution layers.

[0057] The angle difference network based on the deep learning model is trained as follows:

[0058] (1-1) Obtain training samples:

[0059] (1-1-1) Taking a static or dynamic sample as a target, photographing it with a high-resolution three-dimensional imaging microscope to obtain a high-resolution three-dimensional image; the high-resolution three-dimensional imaging microscope includes but is not limited to a wide-field microscope and a confocal microscope;

[0060] In a preferred embodiment, the actually captured image is pixel-resampled to obtain the high-resolution actual three-dimensional image to conform to the Fourier light field imaging parameters.

[0061] (1-1-2) obtaining a point spread function of the hardware Fourier light field microscopy system according to the optical parameters of the hardware Fourier light field microscopy system, performing low angular sampling rate fluctuation optical simulation on the high-resolution actual three-dimensional image obtained in step (1-1-1) using the point spread function, and obtaining a single point spread function image of the hardware Fourier light field microscopy system, wherein the single point spread function image includes a multi-view point spread image array with low angular sampling, and the multi-view point spread image array corresponds to the microlens array of the hardware Fourier light field microscopy system;

[0062] (1-1-3) obtaining a point spread function of the imaginary Fourier light field microscopy system based on the optical parameters of the imaginary Fourier light field microscopy system, using the point spread function to perform high angular sampling rate wave optics simulation on the high-resolution actual three-dimensional image obtained in step (1-1-1), and obtaining a plurality of single-viewpoint point spread function images of the imaginary Fourier light field microscopy system, wherein each point spread function image includes a point spread image of one viewpoint with high angular sampling; the plurality of point spread function images correspond to the microlens array of the imaginary Fourier light field microscopy system;

[0063] The microlens positions simulated by the hypothetical Fourier light field microscopy system are arranged as integer-magnification differences of the microlens positions of the hardware Fourier light field microscopy system; that is, the microlens array simulated by the multi-view light field image with a high angular sampling rate adds an integer number of lenses between adjacent lenses of the lens array simulated by the multi-view light field image with a low angular sampling rate, so that the multi-view light field image with a high angular sampling rate is a magnification interpolation image of the multi-view light field image with a low angular sampling rate.

[0064] According to the angular interpolation magnification, the microlens spacing of the multi-view light field image simulation with high angular sampling rate can be calculated as:

[0065] Δd=d / ratio

[0066] Wherein d is the actual micro-lens pitch, that is, the micro-lens pitch of the lens array simulated by the multi-view light field image with low angular sampling, and ratio is the angular interpolation ratio.

[0067] It should be noted that due to the limited field of view and aperture limitations of actual Fourier light field microscopes, the microlenses of hardware Fourier light field microscopy systems are closely arranged, but their angular resolution is low; therefore, for a hypothetical Fourier light field microscopy system with a higher angular sampling rate, its phantom microlens arrays must overlap with each other, which cannot be achieved by a hardware microlens array. For a phantom Fourier light field microscopy system, if a single multi-perspective point spread function image is formed according to the usual microlens array simulation, then the point spread elements at different spatial positions will crosstalk, resulting in poor results when the high angular resolution light field image is reconstructed into a three-dimensional image. The present invention obtains multiple single-perspective point spread images separately, thereby effectively avoiding the crosstalk problem, and is applied to the final three-dimensional image after later reconstruction to obtain a better three-dimensional reconstruction effect through image correction.

[0068] (1-1-4) Performing perspective-by-perspective convolution on the high-resolution three-dimensional image obtained in step (1-1-1) with the single multi-perspective point spread function image obtained in step (1-1-2) and the multiple single-perspective point spread function images obtained in step (1-1-3) to obtain a single multi-perspective low angular sampling rate light field image and multiple single-perspective high angular sampling rate light field images for forward projection of the actual three-dimensional image. The single multi-perspective low angular sampling rate light field image is used as the input of a training sample, and its corresponding multiple single-perspective high angular sampling rate light field images are used as the output of the training sample.

[0069] A single multi-view low angular sampling rate light field image, each view is represented as:

[0070] Projection low ={Resize(O(x,y,z))*|E(x,y,v,z)| 2} v＝1,2,...,v0

[0071] Multiple single-view high-angular sampling rate light field images, the images of each view are represented as follows:

[0072] Projectio high ={Resize(O(x,y,z))*|E(x,y,v,z)| 2} v＝1,2,...,ratio·v0

[0073] Projection jigh Projection low are light field projection images with high and low angular sampling rates, respectively. Resize is resampling, O(x, y, z) is a three-dimensional image, E(x, y, v, z) is the propagation function after the microlens described above, and v0 is the number of viewing angles at the original angular sampling rate.

[0074] For any Fourier light field microscopy imaging system, the point spread function of the imaging system is obtained according to its optical parameters, specifically:

[0075] S1. Obtain the spatial positions of all microlenses in the lens array for multi-view light field image simulation by using the microlens spacing and microlens arrangement. Microlens arrangements commonly used in Fourier light field microscopy systems include hexagonal and square orthogonal arrangements.

[0076] S2. Determine the transmittance function T based on the optical parameters of the microlens MLA , and according to the wave optics theory, calculate the point spread function PSF (x, y, v) after different microlenses, that is:

[0077] PSF(x,y,v)=|E(x,y,v;z))| 2

[0078] =Fresnel l1 (E1(x,y;z)×T MLA (x,y,v;x v ,y v ))

[0079] Where E1 is the electric field distribution of the point source O(x, y, z) propagating to the microlens plane, T MLA is the transmittance function of a single simulated microlens, x v ,y v is the center coordinate of a microlens in the densely sampled microlens array, Fresnel l1 is the Fresnel diffraction propagation distance l1, the distance from the l1 microlens plane to the camera plane.

[0080] (1-2) Model training

[0081] By performing forward propagation through a dense residual network, we can finally obtain a network output that is consistent with the image size of a high-angle sampling rate, and complete the iterative convergence of the network model by optimizing the following loss function.

[0082]

[0083] Where N is the number of pixels in the view image stack, and y is They are respectively the high-resolution actual three-dimensional image obtained in step (1-1-1) and the three-dimensional image reconstructed from multiple single-view light field images with a high angular sampling rate output by the deep learning model angle difference network.

[0084] (2) reconstructing the multiple single-view light field images with high angular sampling rates obtained in step (1) into three-dimensional images, and superimposing them in the same order as the multiple single-view images in the training sample sampling data of the angle difference network, to obtain the Fourier light field microscope super-resolution imaging. The reconstruction into a three-dimensional image is specifically: performing an iterative deconvolution operation on a single single-view light field image and its corresponding point spread function image of the view to obtain the three-dimensional image of the view;

[0085] The stacking of the three-dimensional images in the training order is specifically: stacking and splicing the three-dimensional images of each perspective in the training order according to the spatial position, which not only avoids crosstalk between perspectives, but also effectively corrects the three-dimensional image, thereby improving the imaging quality.

[0086] The Fourier light field microscope super-resolution imaging system provided by the present invention includes an angle difference network and a perspective-by-perspective reconstruction module;

[0087] The angle difference network is used to intelligently interpolate a single multi-view light field image with a low angular sampling rate acquired by the Fourier light field microscope to obtain multiple single-view light field images with a high angular sampling rate;

[0088] The perspective-by-perspective reconstruction module is used to reconstruct multiple single-perspective light field images with high angular sampling rates output by the angle difference network into three-dimensional images, and superimpose them in the same order as the multiple single-perspective images in the training sample sampling data of the angle difference network to obtain the Fourier light field microscope super-resolution imaging.

[0089] The following are examples:

[0090] The Fourier light field microscope super-resolution imaging method provided in this embodiment includes the following steps:

[0091] (1) A single multi-view light field image with a low angular sampling rate obtained by a Fourier light field microscope is input into an angular difference network based on a deep learning model to obtain multiple single-view light field images with a high angular sampling rate; the multi-view light field image with a low angular sampling rate and the angular interpolation images of adjacent views therein are used as the initial multi-view light field image with a high angular sampling rate for iteration; the angular interpolation images of adjacent views are obtained by interpolation in the angular dimension using the light field images of adjacent views for double-triple interpolation or other interpolation.

[0092] The angle difference network based on the deep learning model has a structure as follows Figure 1 As shown, it consists of a fusion module and a convolution module. The fusion module superimposes the original low-angular sampling rate image stack and the interpolated image stack in the channel dimension and extracts high-dimensional information through multiple convolution processes. The convolution module further extracts information from the front-end fusion module and is composed of multiple layers of conventional convolution layers. The fusion module consists of a single channel stack layer, and the information extraction is specifically composed of multiple residual convolution modules. Each residual convolution module consists of a convolution kernel size of 1 and 3, and integrates the network features formed by convolutions of different scales through local skip connections.

[0093] The angle difference network based on the deep learning model is trained as follows:

[0094] (1-1) Obtain training samples:

[0095] (1-1-1) Taking a static or dynamic sample as a target, photographing it with a high-resolution three-dimensional imaging microscope to obtain a high-resolution three-dimensional image; the high-resolution three-dimensional imaging microscope includes but is not limited to a wide-field microscope and a confocal microscope;

[0096] Pixel resampling is performed on the actually captured image to obtain the high-resolution actual three-dimensional image to comply with the Fourier light field imaging parameters.

[0097] (1-1-2) obtaining a point spread function of the hardware Fourier light field microscopy system according to the optical parameters of the hardware Fourier light field microscopy system, performing low angular sampling rate fluctuation optical simulation on the high-resolution actual three-dimensional image obtained in step (1-1-1) using the point spread function, and obtaining a single point spread function image of the hardware Fourier light field microscopy system, wherein the single point spread function image includes a multi-view point spread image array with low angular sampling, and the multi-view point spread image array corresponds to the microlens array of the hardware Fourier light field microscopy system;

[0098] The microlens distribution of the hardware Fourier light field microscopy system is arranged in a hexagonal pattern, such as Figure 2 shown.

[0099] (1-1-3) obtaining a point spread function of the imaginary Fourier light field microscopy system based on the optical parameters of the imaginary Fourier light field microscopy system, using the point spread function to perform high angular sampling rate wave optics simulation on the high-resolution actual three-dimensional image obtained in step (1-1-1), and obtaining a plurality of single-viewpoint point spread function images of the imaginary Fourier light field microscopy system, wherein each point spread function image includes a point spread image of one viewpoint with high angular sampling; the plurality of point spread function images correspond to the microlens array of the imaginary Fourier light field microscopy system;

[0100] The microlens positions simulated by the hypothetical Fourier light field microscopy system are arranged as integer-magnification differences of the microlens positions of the hardware Fourier light field microscopy system; that is, the microlens array simulated by the multi-view light field image with a high angular sampling rate adds an integer number of lenses between adjacent lenses of the lens array simulated by the multi-view light field image with a low angular sampling rate, so that the multi-view light field image with a high angular sampling rate is a magnification interpolation image of the multi-view light field image with a low angular sampling rate.

[0101] According to the angular interpolation magnification, the microlens spacing of the multi-view light field image simulation with high angular sampling rate can be calculated as:

[0102] Δd=d / ratio

[0103] Wherein d is the actual micro-lens pitch, that is, the micro-lens pitch of the lens array simulated by the multi-view light field image with low angular sampling, and ratio is the angular interpolation ratio.

[0104] In this embodiment, the angular interpolation magnification is set to 2, and the microlens distribution of the hypothetical Fourier light field microscopy imaging system is obtained as follows: Figure 3 shown.

[0105] Perform initial interpolation for low angular sampling rates, that is, interpolate and extrapolate the initial value of the dense angular sampling image from the adjacent view images, which can be expressed as:

[0106] View i =Interpolation(View i-1 ,View i+1 )

[0107] View i is the 3D image of the i-th view in the high sampling rate light field projection after interpolation. Interpolation is the bicubic interpolation algorithm. View i-1 ,View i+1 They are the perspective images projected at the original angular sampling rate. The perspective sorting of the interpolation algorithm should be fixed.

[0108] (1-1-4) Performing perspective-by-perspective convolution on the high-resolution three-dimensional image obtained in step (1-1-1) with the single multi-perspective point spread function image obtained in step (1-1-2) and the multiple single-perspective point spread function images obtained in step (1-1-3) to obtain a single multi-perspective low angular sampling rate light field image and multiple single-perspective high angular sampling rate light field images for forward projection of the actual three-dimensional image. The single multi-perspective low angular sampling rate light field image is used as the input of a training sample, and its corresponding multiple single-perspective high angular sampling rate light field images are used as the output of the training sample.

[0109] A single multi-view low angular sampling rate light field image, each view is represented as

[0110] Projectio low ={Resize(O(x,y,z))*|E(x,y,v,z)| 2} v＝1,2,...,v0

[0111] Multiple single-view high-angular sampling rate light field images, such as Figure 4 As shown, the images of each perspective are represented as:

[0112] Projection high ={Resize(O(x,y,z))*|E(x,y,v,z)| 2} v＝1,2,...,ratio·v0

[0113] Among them Projectio high Projection loware light field projection images with high and low angular sampling rates, respectively. Resize is resampling, O(x, y, z) is a three-dimensional image, E(x, y, v, z) is the propagation function after the microlens described above, and v0 is the number of viewing angles at the original angular sampling rate.

[0114] For any Fourier light field microscopy imaging system, the point spread function of the imaging system is obtained according to its optical parameters, specifically:

[0115] S1. Obtain the spatial positions of all microlenses in the lens array for multi-view light field image simulation by using the microlens spacing and microlens arrangement. Microlens arrangements commonly used in Fourier light field microscopy systems include hexagonal and square orthogonal arrangements.

[0116] S2. Determine the transmittance function T based on the optical parameters of the microlens MLA , and according to the wave optics theory, calculate the point spread function PSF (x, y, v) after different microlenses, that is:

[0117] PSF(x,y,v)=|E(x,y,v;z))| 2

[0118] =Fresnel l1 (E1(x,y;z)×T MLA (x,y,v;x v ,y v ))

[0119] Where E1 is the electric field distribution of the point source O(x, y, z) propagating to the microlens plane, T MLA is the transmittance function of a single simulated microlens, x v ,y v is the center coordinate of a microlens in the densely sampled microlens array, Fresnel l1 is the Fresnel diffraction propagation distance l1, the distance from the l1 microlens plane to the camera plane.

[0120] (1-2) Model training

[0121] By performing forward propagation through a dense residual network, we can finally obtain a network output that is consistent with the image size of a high-angle sampling rate, and complete the iterative convergence of the network model by optimizing the following loss function.

[0122]

[0123] Where N is the number of pixels in the view image stack, and y is They are respectively the high-resolution actual three-dimensional image obtained in step (1-1-1) and the three-dimensional image reconstructed from multiple single-view light field images with a high angular sampling rate output by the deep learning model angle difference network.

[0124] (2) reconstructing the multiple single-view light field images with high angular sampling rates obtained in step (1) into three-dimensional images perspective by perspective, and superimposing them in the same order as the multiple single-view images in the training sample sampling data of the angle difference network to obtain the Fourier light field microscope super-resolution imaging.

[0125] The reconstruction into a three-dimensional image is specifically: performing an iterative deconvolution operation on a single single-view light field image and its corresponding point spread function image to obtain a three-dimensional image of the view;

[0126] The stacking of the three-dimensional images in the training order is specifically: stacking and splicing the three-dimensional images of each perspective in the training order according to the spatial position, which not only avoids crosstalk between perspectives, but also effectively corrects the three-dimensional image, thereby improving the imaging quality.

[0127] The reconstruction results are as follows Figure 5 shown.

[0128] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Fourier light field microscope super-resolution imaging method, characterized in that: The following steps are involved: (1) A single multi-view light field image with a low angular sampling rate obtained by a Fourier light field microscope is input into an angle difference network to obtain multiple single-view light field images with a high angular sampling rate; specifically: Iterate the multi-view light field image with a low angular sampling rate and the angular interpolation images of adjacent views as the initial multi-view light field image with a high angular sampling rate; the angular interpolation images of adjacent views are interpolated in the angular dimension using an interpolation algorithm such as bi-tri interpolation of the light field images of adjacent views; The angle difference network is based on a deep learning model and includes a fusion channel and a convolution channel. The fusion channel is used to superimpose the original low-angle sampling rate image stack and the interpolated image stack in the channel dimension and extract high-dimensional information through multiple convolution processes, including multiple fusion modules. The convolution channel includes multiple convolution modules, which are used to further extract information from the front-end fusion module and are composed of multiple convolution layers. (2) The multiple single-view light field images with high angular sampling rate obtained in step (1) are reconstructed into three-dimensional images perspective by perspective, and are superimposed in the same order as the multiple single-view images in the training sample sampling data of the angle difference network to obtain the Fourier light field microscope super-resolution imaging.

2. The Fourier light field microscope super-resolution imaging method according to claim 1, characterized in that: The angle difference network based on the deep learning model is trained as follows: (1-1) Obtain training samples: (1-1-1) Using a high-resolution three-dimensional imaging microscope to actually photograph a static or dynamic sample to obtain a high-resolution three-dimensional image; the high-resolution three-dimensional imaging microscope includes but is not limited to a wide-field microscope and a confocal microscope; (1-1-2) obtaining a point spread function of the hardware Fourier light field microscopy system based on the optical parameters of the hardware Fourier light field microscopy system, performing low angular sampling rate fluctuation optical simulation on the high-resolution actual three-dimensional image obtained in step (1-1-1) using the point spread function, and obtaining a single point spread function image of the hardware Fourier light field microscopy system, wherein the single point spread function image includes a multi-view point spread image array with low angular sampling, and the multi-view point spread image array corresponds to the microlens array of the hardware Fourier light field microscopy system; (1-1-3) obtaining a point spread function of the imaginary Fourier light field microscopy system based on the optical parameters of the imaginary Fourier light field microscopy system, using the point spread function to perform high angular sampling rate fluctuation optical simulation on the high-resolution actual three-dimensional image obtained in step (1-1-1), and obtaining a plurality of single-viewing angle point spread function images of the imaginary Fourier light field microscopy system, wherein each point spread function image includes a point spread image of one viewing angle with high angular sampling; the plurality of point spread function images correspond to the microlens array of the imaginary Fourier light field microscopy system; (1-1-4) performing per-view convolution on the high-resolution three-dimensional image obtained in step (1-1-1) with the single multi-view point spread function image obtained in step (1-1-2) and the multiple single-view point spread function images obtained in step (1-1-3), respectively, to obtain a single multi-view low angular sampling rate light field image and multiple single-view high angular sampling rate light field images for forward projection of the actual three-dimensional image, using the single multi-view low angular sampling rate light field image as the input of a training sample, and using its corresponding multiple single-view high angular sampling rate light field images as the output of the training sample; A single multi-view low angular sampling rate light field image, each view is represented as: ， Multiple single-view high-angular sampling rate light field images, the images of each view are represented as follows: ， in and They are light field projection images with high / low angular sampling rates respectively, and Resize is resampling. For a three-dimensional image, is the propagation function after the microlens mentioned above, is the number of viewing angles at the original angular sampling rate; (1-2) Model training.

3. The Fourier light field microscope super-resolution imaging method according to claim 2, characterized in that: Step (1-1-2) performs pixel resampling on the actually captured image to obtain the high-resolution actual three-dimensional image.

4. The Fourier light field microscope super-resolution imaging method according to claim 2, wherein: The microlens positions simulated by the hypothetical Fourier light field microscopy imaging system are arranged with integer magnification differences of the microlens positions of the hardware Fourier light field microscopy imaging system.

5. The Fourier light field microscope super-resolution imaging method according to claim 4, characterized in that: According to the angular interpolation magnification, the microlens spacing of the multi-view light field image simulation with high angular sampling rate can be calculated as: ， in is the actual microlens pitch, that is, the microlens pitch of the lens array simulated by the multi-view light field image with low angular sampling, and ratio is the angular interpolation magnification.

6. The Fourier light field microscopy super-resolution imaging method according to any one of claims 1 to 5, characterized in that: For any Fourier light field microscopy imaging system, the point spread function given by the imaging system is obtained according to its optical parameters, specifically: S1. Obtaining the spatial positions of all microlenses in the lens array for multi-view light field image simulation by microlens spacing and microlens arrangement; S2. Determine the transmittance function based on the optical parameters of the microlens , and according to the wave optics theory, calculate the point spread function after different microlenses ,Right now: ； in is the electric field distribution of the point source O(x,y,z) propagating to the microlens plane, is the transmittance function of a single simulated microlens, is the center coordinate of a microlens in the densely sampled microlens array, Fresnel diffraction propagation distance, The distance from the microlens plane to the camera plane.

7. The Fourier light field microscope super-resolution imaging method according to claim 6, characterized in that: The microlens arrangement adopted by the Fourier light field microscopy imaging system is: hexagonal arrangement and square orthogonal arrangement.

8. The Fourier light field microscope super-resolution imaging method according to claim 3, wherein: The steps (1-2) are specifically as follows: Through the dense residual network, forward propagation is performed to finally obtain the network output consistent with the high angular sampling rate image size, and the following loss function is optimized , complete the iterative convergence of the network model; ， Where N is the number of pixels in the view image stack, and They are respectively the high-resolution actual three-dimensional image obtained in step (1-1-1) and the three-dimensional image reconstructed from multiple single-view light field images with a high angular sampling rate output by the deep learning model angle difference network.

9. The Fourier light field microscope super-resolution imaging method according to claim 1, wherein: The reconstruction into a three-dimensional image is specifically: performing an iterative deconvolution operation on a single single-view light field image and its corresponding point spread function image to obtain a three-dimensional image of the view; The three-dimensional images are superimposed in the order of training. Specifically, the three-dimensional images of each perspective are superimposed and spliced ​​in the order of training according to the spatial position. This can not only avoid crosstalk between perspectives, but also effectively correct the three-dimensional images, thereby improving the imaging quality.

10. A Fourier light field microscope super-resolution imaging system, characterized in that: Includes angle difference network and perspective reconstruction module; The angle difference network is used to intelligently interpolate a single multi-view light field image with a low angular sampling rate acquired by a Fourier light field microscope to obtain multiple single-view light field images with a high angular sampling rate; specifically: Iterate the multi-view light field image with a low angular sampling rate and the angular interpolation images of adjacent views as the initial multi-view light field image with a high angular sampling rate; the angular interpolation images of adjacent views are interpolated in the angular dimension using an interpolation algorithm such as bi-tri interpolation of the light field images of adjacent views; The angle difference network is based on a deep learning model and includes a fusion channel and a convolution channel. The fusion channel is used to superimpose the original low-angle sampling rate image stack and the interpolated image stack in the channel dimension and extract high-dimensional information through multiple convolution processes, including multiple fusion modules. The convolution channel includes multiple convolution modules, which are used to further extract information from the front-end fusion module and are composed of multiple convolution layers. The perspective-by-perspective reconstruction module is used to reconstruct multiple single-perspective light field images with high angular sampling rates output by the angle difference network into three-dimensional images, and superimpose them in the same order as the multiple single-perspective images in the training sample sampling data of the angle difference network to obtain the Fourier light field microscope super-resolution imaging.

Images