Self-supervised light field three-dimensional high-resolution reconstruction method and system and medium

Abstract

The invention discloses a self-supervised light field three-dimensional high-resolution reconstruction method and system and a medium, and belongs to the field of microscopic imaging, and the method comprises the steps: employing a light field imaging system and a wide field imaging system to carry out the imaging of a sample, and obtaining a low-resolution light field image of the sample and a high-resolution wide field image of the sample; constructing a self-supervised training sample through simulation mapping to obtain a high-low resolution simulation light field training image pair; performing self-supervised training on the two-dimensional super-resolution network by taking the low-resolution simulation light field image frame as an input and the high-resolution simulation light field image frame as a label; and step 3, carrying out super-resolution on image frames of each view angle of the sample low-resolution light field image obtained in the step 1 by using the trained two-dimensional super-resolution network, outputting a high-resolution reasoning light field image frame, and carrying out three-dimensional reconstruction to obtain a super-resolution target sample three-dimensional image. According to the method, rapid and real-time light field super-resolution three-dimensional imaging can be realized, and the axial resolution hardware limit of light field imaging is broken through.

Description

Technical Field

[0001] The present invention belongs to the field of microscopic imaging, and more specifically, relates to a self-supervised light field three-dimensional high-resolution reconstruction method, system and medium. Background Art

[0002] Light field microscopy has an extremely high volume imaging rate. Compared with conventional high-resolution three-dimensional imaging technology that needs to scan and collect multiple two-dimensional images to obtain three-dimensional information, it only needs to collect a single image to obtain the three-dimensional information of the image, which greatly improves the three-dimensional imaging speed and provides the possibility for high-speed dynamic imaging of living bodies. Light field microscopy technology adds a microlens array to the detection light path so that the acquired two-dimensional images can contain both lateral and angular information. However, since this method needs to obtain lateral and angular information simultaneously through a single image, the lateral information needs to be undersampled. Compared with conventional three-dimensional imaging technology, the spatial resolution is lower and the temporal resolution is higher.

[0003] At present, it is generally necessary to combine the light field reconstruction algorithm with deep learning to achieve light field imaging. Fluorescence microscopy images enhanced by deep learning show higher resolution and signal-to-noise ratio. However, the existing light field reconstruction algorithms combined with deep learning still face the following problems: the need to obtain a large amount of labeled data, the acquisition of labeled data is time-consuming, the training speed is slow, and it is difficult to generalize under different conditions, which limits its application scenarios. Summary of the invention

[0004] In view of the defects of the prior art and the need for improvement, the present invention provides a self-supervised light field three-dimensional high-resolution reconstruction method, system and medium, the purpose of which is to provide high-resolution information through wide-field imaging, and to improve the light field resolution by self-supervised learning in the imaging process of similar samples, thereby solving the technical problems of existing light field super-resolution imaging, such as slow speed, poor fidelity of axial super-resolution imaging, and difficulty in generalization.

[0005] To achieve the above-mentioned purpose, according to one aspect of the present invention, a self-supervised light field three-dimensional high-resolution reconstruction method is provided, comprising: self-supervised training stages S1-S3 and reconstruction stage S4; S1, simulating and mapping the high-resolution wide-field image of the sample to generate high-resolution simulated light field image frames at different viewing angles; S2, downsampling each of the high-resolution simulated light field image frames to obtain a corresponding low-resolution simulated light field image frame, the resolution of the low-resolution simulated light field image frame matches the resolution of the low-resolution light field image of the sample, wherein the low-resolution light field image includes light field image frames at each viewing angle; S3, using the low-resolution simulated light field image frame as input and the high-resolution simulated light field image frame as a label, self-supervising training is performed on a two-dimensional super-resolution network; S4, using the trained two-dimensional super-resolution network to perform super-resolution processing on the low-resolution light field image of a target body of the same category as the sample, and performing three-dimensional reconstruction on the image obtained by the super-resolution processing to obtain a super-resolution three-dimensional image of the target body.

[0006] Furthermore, when the target object is the sample, the method also includes: imaging the sample using a wide-field imaging system and a light-field imaging system respectively, and correspondingly obtaining a high-resolution wide-field image and a low-resolution light-field image of the sample; wherein a switching device is provided at the intersection of the optical paths of the wide-field imaging system and the light-field imaging system, and the switching device is used to connect to one of the wide-field imaging system and the light-field imaging system for imaging at the same time.

[0007] Furthermore, the respectively using a wide-field imaging system and a light-field imaging system to image the sample specifically includes: respectively using a wide-field imaging system and a light-field imaging system to image the same field of view of the sample.

[0008] Furthermore, the switching device is a flippable reflector; when the flippable reflector is flipped into the optical path, one of the wide-field imaging system and the light-field imaging system performs imaging; when the flippable reflector is flipped out of the optical path, the other of the wide-field imaging system and the light-field imaging system performs imaging.

[0009] Furthermore, S1 specifically includes: deconvolving the high-resolution wide-field image of the sample; performing a convolution operation on the deconvolved high-resolution wide-field image using the point spread functions of the light field image at different depths and directions to obtain a high-resolution simulated light field image frame at a corresponding viewing angle.

[0010] Furthermore, when the low-resolution light field image of the sample is an image obtained by spatial light field imaging, the point spread function of the light field image at different depths and directions is obtained by rearranging the point spread function of the spatial light field image in the phase space domain.

[0011] Furthermore, S4 specifically includes: inputting the light field image frames at each viewing angle in the low-resolution light field image of the target body into the trained two-dimensional super-resolution network to obtain high-resolution inference light field image frames at each viewing angle; performing deconvolution operations on the high-resolution inference light field image frames at each viewing angle with the point spread function at the corresponding depth and direction, respectively, to obtain a super-resolution three-dimensional image of the target body.

[0012] Furthermore, the two-dimensional super-resolution network is a lightweight convolutional neural network.

[0013] According to another aspect of the present invention, a self-supervised light field three-dimensional high-resolution reconstruction system is provided, comprising: a processor; a memory storing a computer executable program, wherein when the program is executed by the processor, the processor executes the self-supervised light field three-dimensional high-resolution reconstruction method as described above.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for self-supervised light field three-dimensional high-resolution reconstruction as described above is implemented.

[0015] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0016] (1) A self-supervised light field 3D high-resolution reconstruction method is provided. The method uses the resolution information provided by the high-resolution wide-field image and adopts a self-supervised learning two-dimensional super-resolution network to improve the resolution of the light field images at each viewing angle, thereby achieving fast and real-time light field super-resolution 3D imaging, breaking through the axial resolution hardware limit of light field imaging, and having a high speed. Compared with the existing light field super-resolution 3D imaging technology based on high-resolution factual 3D data, the method has good practicality and verifiability because the wide-field image data acquisition difficulty is low and the system is convenient, and it is easy to realize the optical path fusion with the light field system.

[0017] (2) When the sample is reconstructed with high resolution in three dimensions, the switching device is used to sequentially realize two types of imaging of the sample, namely wide-field imaging and light-field imaging, to achieve in-situ acquisition and ensure that wide-field imaging and light-field imaging are performed on the same field of view of the sample. This can obtain light-field images and wide-field images with higher consistency, improve the super-resolution performance of the self-supervised network for low-resolution light-field images, and make the method have good generalization performance;

[0018] (3) A lightweight convolutional neural network is selected as the two-dimensional super-resolution network to achieve a lightweight design, thereby further improving the speed of light field super-resolution three-dimensional imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a method for self-supervised light field three-dimensional high-resolution reconstruction provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a self-supervised light field three-dimensional high-resolution reconstruction method provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the optical path structure of a microscopic imaging device provided in an embodiment of the present invention;

[0022] Figure 4 Examples of light field images and wide field images obtained by the method provided in the embodiments of the present invention;

[0023] Figure 5 An example of deconvolution of a wide field image in the method provided by an embodiment of the present invention;

[0024] Figure 6 An example of high-resolution light field image three-dimensional reconstruction in the method provided in an embodiment of the present invention;

[0025] Figure 7 For the spatial light field, the method provided in the embodiment of the present invention is used to perform high-resolution light field image 3D reconstruction and the effect of directly performing light field image 3D reconstruction is compared;

[0026] Figure 8 For Fourier light field, the method provided by the embodiment of the present invention is used to perform high-resolution light field image 3D reconstruction and the effect of directly performing light field image 3D reconstruction is compared. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0029] Embodiment 1

[0030] A self-supervised light field 3D high-resolution reconstruction method, combined with Figure 1-Figure 8 , the self-supervised light field three-dimensional high-resolution reconstruction method in this embodiment is described in detail. Figure 1The method includes a self-supervised training phase (operation S1-operation S3) and a reconstruction phase (operation S4), and its principle is as follows Figure 2 shown.

[0031] In operation S1, a high-resolution wide-field image of a sample is simulated mapped to generate high-resolution simulated light-field image frames at different viewing angles.

[0032] Before performing operation S1, it is necessary to obtain a high-resolution wide-field image of the sample; before performing operation S4, it is necessary to obtain a low-resolution light-field image of the target object.

[0033] When the target object in the reconstruction phase is not the sample used in the self-supervised training phase, the process of acquiring the high-resolution wide-field image of the sample and the process of acquiring the low-resolution light-field image of the target object are independent of each other.

[0034] When the target object in the reconstruction phase is the sample used in the self-supervised training phase, the method further includes: imaging the sample using a wide-field imaging system and a light-field imaging system, respectively, to obtain a high-resolution wide-field image and a low-resolution light-field image of the sample. Preferably, the wide-field imaging system and the light-field imaging system are used to image the same field of view of the sample, respectively, to obtain a light-field image and a wide-field image with higher consistency.

[0035] In this embodiment, a switching device is provided at the intersection of the optical paths of the wide-field imaging system and the light-field imaging system, and the switching device is used to connect to one of the wide-field imaging system and the light-field imaging system for imaging at the same time.

[0036] Preferably, the switching device is a flippable reflector. When the flippable reflector is flipped into the optical path, one of the wide-field imaging system and the light-field imaging system performs imaging; when the flippable reflector is flipped out of the optical path, the other of the wide-field imaging system and the light-field imaging system performs imaging.

[0037] In this embodiment, simulation mapping is performed according to the parameters of the light field imaging system to obtain high-resolution simulated light field image frames at various viewing angles as label data of self-supervised training samples. Specifically, operation S1 specifically includes the following sub-operations S11-S12.

[0038] In sub-operation S11, a high-resolution wide-field image of the sample is deconvolved.

[0039] In sub-operation S12, a convolution operation is performed on the deconvolved high-resolution wide-field image using point spread functions at different depths and in different directions of the light field image to obtain a high-resolution simulated light field image frame at a corresponding viewing angle.

[0040] When the low-resolution light field image of the sample is an image obtained by spatial light field imaging, the point spread function of the light field image at different depths and directions is obtained by rearranging the point spread function of the spatial light field image in phase space domain.

[0041] The arrangement is as follows: the spatial light field point spread function is converted from the airspace domain (x, y, z) to the phase space domain (x, y, z, view). The conversion method is to extract the pixels at the same position in each microlens to obtain the viewing angle information, and then extract the corresponding two-dimensional point spread function for each viewing angle and each depth.

[0042] Operation S2, downsampling each high-resolution simulated light field image frame to obtain a corresponding low-resolution simulated light field image frame, the resolution of the low-resolution simulated light field image frame matches the resolution of the low-resolution light field image of the sample, wherein the low-resolution light field image includes light field image frames at each viewing angle.

[0043] In this embodiment, low-resolution simulated light field image frames at various viewing angles are used as training inputs, and corresponding high-resolution simulated light field image frames are used as label data to form self-supervised training sample images.

[0044] Operation S3 performs self-supervised training on a two-dimensional super-resolution network using low-resolution simulated light field image frames as input and high-resolution simulated light field image frames as labels.

[0045] Preferably, the two-dimensional super-resolution network is a lightweight convolutional neural network, such as UNet, etc. Since a lightweight convolutional neural network is used, the training speed is fast and the self-supervised training requirements are met. It should be noted that the two-dimensional super-resolution network can also be other two-dimensional networks.

[0046] The two-dimensional super-resolution network is self-supervised trained as follows: the training loss function and convergence conditions apply to conventional image super-resolution loss functions and convergence conditions.

[0047] In this embodiment, the objects of the self-supervised training stage and the reconstruction stage are samples of the same type, and the image characteristics are highly consistent. Self-supervised training is performed, so there is no need to train with a large amount of data to make the two-dimensional super-resolution network have good generalization performance, and there is no distortion during image super-resolution due to the small amount of training data and overfitting of the two-dimensional super-resolution network.

[0048] Operation S4, using the trained two-dimensional super-resolution network, super-resolution processing is performed on the low-resolution light field image of the target of the same category as the sample, and three-dimensional reconstruction is performed on the image obtained by the super-resolution processing to obtain a super-resolution three-dimensional image of the target.

[0049] In this embodiment, operation S4 specifically includes the following sub-operation S41-sub-operation S42.

[0050] In sub-operation S41, light field image frames at each viewing angle in the low-resolution light field image of the target object are input into the trained two-dimensional super-resolution network to obtain high-resolution inference light field image frames at each viewing angle.

[0051] In sub-operation S42, the high-resolution inference light field image frames at each viewing angle are deconvolved with the point spread functions at the corresponding depth and direction to obtain a super-resolution three-dimensional image of the target volume.

[0052] The method is described by taking the light field imaging system as a Fourier light field imaging system, the light field image as a Fourier light field image, and the target body as a sample as an example. The method includes the following steps 1 to 4.

[0053] See also Figure 3 , the same commercial inverted fluorescence microscope is used to build the Fourier light field optical path and the wide field optical path, and a switching device based on a flip mirror is designed to switch between the two modes. The same field of view of the fluorescence microscope can be used for wide field detection and Fourier light field detection; among them, wide field detection provides high-resolution label data, and Fourier light field provides information from different perspectives. The same scene is imaged using two modalities, and the network training can be completed using only one wide field image.

[0054] Step 1: Use the Fourier light field imaging system and the wide field imaging system to image the same field of view of the sample, and obtain a low-resolution light field image and a high-resolution wide field image of the sample. The light field image is composed of image frames from various viewing angles, such as Figure 4 shown.

[0055] Step 2, construct self-supervised training samples. Perform simulation mapping on the high-resolution wide-field image obtained in step 1 to obtain a high-resolution simulated light field image frame as the label data of the self-supervised training sample; downsample the high-resolution simulated light field image frame to obtain a corresponding low-resolution simulated light field image frame as the input image of the self-supervised training sample, and the low-resolution simulated light field image frame matches the image frame resolution of the low-resolution light field image. The specific process of simulation mapping is as follows:

[0056] Step 2-1: Deconvolve the high-resolution wide-field image obtained in step 1 to obtain a deconvoluted high-resolution wide-field image, such as Figure 5 shown.

[0057] Step 2-2: According to the imaging parameters of the light field imaging system, the point spread functions at different depths and directions of the light field image are used to perform convolution operations on the high-resolution wide-field image after deconvolution to obtain high-resolution simulated light field image frames of each viewing angle. In this embodiment, the point spread functions of 41 layers of 7 viewing angles of the Fourier light field are used to perform layer-by-layer and perspective-by-perspective convolution on the high-resolution wide-field image after deconvolution, and 287 high-resolution light field images are obtained by simulation mapping, which are then cropped into 4600 32*32 pixel images as label data.

[0058] Step 2-3: Downsample the high-resolution simulated light field image frame obtained in step 2-2 to obtain a corresponding low-resolution simulated light field image frame. The downsampling method of this embodiment is to merge adjacent pixels to obtain a simulated light field view with the same resolution as the actual light field view as the input data of the network.

[0059] Step 2-4: Use low-resolution simulated light field image frames at each viewing angle as training input, and use high-resolution simulated light field image frames at the corresponding viewing angle as label data to form self-supervised training sample images.

[0060] Step 3: Self-supervised training: The self-supervised training samples obtained in step 2 are used to train a two-dimensional super-resolution network to obtain a light field image frame super-resolution network with converged training.

[0061] The two-dimensional super-resolution network is trained according to the following method: the self-supervised training data obtained in step 2 is used as the training data of the two-dimensional super-resolution network; the two-dimensional super-resolution network used in this embodiment is based on the Unet network, the learning rate is 0.0004, the loss function is the mean square error loss function, and the convergence condition is determined by the quality of the actual light field inferred by the network, which is generally around 400 iterations.

[0062] Step 4: Input the light field image frames of each viewing angle of the low-resolution light field image of the sample obtained in step 1 into the light field image frame super-resolution network obtained in step 3, output high-resolution reasoning light field image frames of each viewing angle with a resolution equivalent to that of the high-resolution wide-field image, and perform three-dimensional reconstruction to obtain a super-resolution three-dimensional image of the sample. This embodiment uses Richard-Lucy deconvolution to perform three-dimensional reconstruction, and the result is as follows: Figure 6 shown.

[0063] Since the network parameters are small and it is optimized for a single scene, the network can be trained in a few minutes. The Fourier light field image is input into the trained two-dimensional super-resolution network to obtain a super-resolution light field image. A well-trained two-dimensional super-resolution network can increase the viewing angle of the Fourier light field in each direction to the resolution of a wide-field image, while maintaining the large depth of field of the Fourier light field.

[0064] The obtained super-resolution light field map is reconstructed in three dimensions to obtain a high-resolution three-dimensional image. The sample three-dimensional image reconstructed according to the theoretical deconvolution of the output image has the advantages of high resolution, near isotropy, and large depth of field, such as Figure 7 As shown, the resolution of the high-resolution three-dimensional reconstruction image obtained in this embodiment is greatly improved compared with the resolution of the Fourier light field reconstruction image.

[0065] If the Fourier light field imaging system in the above example is replaced by a spatial light field imaging system, the implementation process is basically the same as the above example, except that step 2-2 is as follows: re-arrange the pixels of the spatial light field image to obtain information of the sample at different viewing angles, and re-arrange the simulated spatial light field point spread function to obtain point spread functions in different directions and layers, and then convolve with the wide field image to obtain high-resolution simulated light field image frames of each viewing angle; input the spatial light field map into the trained two-dimensional super-resolution network to obtain a super-resolution light field map. A well-trained network can effectively improve the reconstruction resolution of the spatial light field while maintaining its advantage of a large field of view. Figure 8 The resolution of the high-resolution three-dimensional reconstruction image obtained in this embodiment is greatly improved compared with the resolution of the spatial light field reconstruction image.

[0066] Embodiment 2

[0067] A self-supervised light field three-dimensional high-resolution reconstruction system includes: a processor; a memory storing a computer executable program, which, when executed by the processor, enables the processor to execute the self-supervised light field three-dimensional high-resolution reconstruction method. The related technical solution is the same as that of the first embodiment, and will not be repeated here.

[0068] Embodiment 3

[0069] A computer-readable storage medium stores a computer program, and when the program is executed by a processor, the above-mentioned self-supervised light field three-dimensional high-resolution reconstruction method is implemented. The related technical solution is the same as that of the first embodiment, and will not be repeated here.

[0070] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A self-supervised light field 3D high-resolution reconstruction method, characterized in that: include: Self-supervised training phases S1-S3 and reconstruction phase S4; S1, simulate mapping the high-resolution wide-field image of the sample to generate high-resolution simulated light-field image frames at different viewing angles; S2, downsampling each of the high-resolution simulated light field image frames to obtain a corresponding low-resolution simulated light field image frame, wherein the resolution of the low-resolution simulated light field image frame matches the resolution of the low-resolution light field image of the sample, wherein the low-resolution light field image includes light field image frames at each viewing angle; S3, using the low-resolution simulated light field image frame as input and the high-resolution simulated light field image frame as a label, performing self-supervised training on a two-dimensional super-resolution network; S4, using the trained two-dimensional super-resolution network to perform super-resolution processing on the low-resolution light field image of the target of the same category as the sample, and to perform three-dimensional reconstruction on the image obtained by the super-resolution processing to obtain a super-resolution three-dimensional image of the target.

2. The self-supervised light field three-dimensional high-resolution reconstruction method according to claim 1, characterized in that: When the target is the sample, the method further comprises: The sample is imaged using a wide-field imaging system and a light-field imaging system respectively, and a high-resolution wide-field image and a low-resolution light-field image of the sample are obtained accordingly; Wherein, a switching device is provided at the intersection of the optical paths of the wide-field imaging system and the light-field imaging system, and the switching device is used to connect to one of the wide-field imaging system and the light-field imaging system at the same time for imaging.

3. The self-supervised light field three-dimensional high-resolution reconstruction method according to claim 2, characterized in that: The respectively using a wide-field imaging system and a light-field imaging system to image the sample specifically includes: respectively using a wide-field imaging system and a light-field imaging system to image the same field of view of the sample.

4. The self-supervised light field three-dimensional high-resolution reconstruction method according to claim 2, characterized in that: The switching device is a flippable reflector; When the flippable reflector is flipped into the optical path, one of the wide-field imaging system and the light-field imaging system performs imaging; When the flippable reflector is flipped out of the optical path, the other of the wide-field imaging system and the light-field imaging system performs imaging.

5. The self-supervised light field three-dimensional high-resolution reconstruction method according to any one of claims 1 to 4, characterized in that: The S1 specifically includes: Deconvolute high-resolution widefield images of the sample; The point spread functions of the light field image at different depths and directions are used to perform convolution operations on the deconvolved high-resolution wide-field image to obtain a high-resolution simulated light field image frame at the corresponding viewing angle.

6. The self-supervised light field three-dimensional high-resolution reconstruction method according to claim 5, characterized in that: When the low-resolution light field image of the sample is an image obtained by spatial light field imaging, the point spread function of the light field image at different depths and directions is obtained by rearranging the point spread function of the spatial light field image in phase space domain.

7. The self-supervised light field 3D high-resolution reconstruction method according to claim 5, characterized in that: The S4 specifically includes: Inputting the light field image frames at each viewing angle in the low-resolution light field image of the target object into the trained two-dimensional super-resolution network to obtain high-resolution inference light field image frames at each viewing angle; The high-resolution inference light field image frames at each viewing angle are deconvolved with the point spread functions in the corresponding depth and direction to obtain a super-resolution three-dimensional image of the target volume.

8. The self-supervised light field three-dimensional high-resolution reconstruction method according to claim 1, characterized in that: The two-dimensional super-resolution network is a lightweight convolutional neural network.

9. A self-supervised light field 3D high-resolution reconstruction system, characterized in that: include: processor; A memory storing a computer executable program, wherein when the program is executed by the processor, the processor executes the self-supervised light field three-dimensional high-resolution reconstruction method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the self-supervised light field three-dimensional high-resolution reconstruction method according to any one of claims 1 to 8 is implemented.

Images