Self-supervised light field three-dimensional high-resolution reconstruction method, system and medium

Abstract

The present invention discloses a self-supervised light field 3D high-resolution reconstruction method, system, and medium, belonging to the field of microscopic imaging. The method comprises: imaging a sample using a light field imaging system and a widefield imaging system to obtain a low-resolution light field image and a high-resolution widefield image of the sample; constructing self-supervised training samples through simulated mapping to obtain high-resolution and low-resolution simulated light field training image pairs; self-supervised training of a two-dimensional super-resolution network using low-resolution simulated light field image frames as input and high-resolution simulated light field image frames as labels; and utilizing the trained two-dimensional super-resolution network to super-resolve image frames from each perspective of the sample's low-resolution light field image obtained in the first step, outputting high-resolution inferred light field image frames, and performing 3D reconstruction to obtain a super-resolved 3D image of the target sample. This method can achieve fast and real-time light field super-resolution 3D imaging, breaking through the axial resolution hardware limit of light field imaging.

Description

Technical Field

[0001] The present invention belongs to the field of microscopic imaging, and more specifically, relates to a method, system and medium for self-supervised light field three-dimensional high-resolution reconstruction. Background Art

[0002] Light-field microscopy has an extremely high volumetric imaging rate. Compared to conventional high-resolution three-dimensional imaging techniques that require scanning and acquiring multiple two-dimensional images to obtain three-dimensional information, it only needs to acquire a single image to obtain the image's three-dimensional information. This greatly improves the three-dimensional imaging speed and makes high-speed dynamic imaging of living bodies possible. Light-field microscopy technology adds a microlens array to the detection light path so that the acquired two-dimensional images can contain both lateral and angular information. However, since this method requires acquiring both lateral and angular information simultaneously through a single image, the lateral information needs to be undersampled. Compared to conventional three-dimensional imaging technology, the spatial resolution is lower and the temporal resolution is higher.

[0003] Currently, light field reconstruction algorithms combined with deep learning are generally required to achieve light field imaging. Fluorescence microscopy images enhanced by deep learning exhibit higher resolution and signal-to-noise ratio. However, existing deep learning-based light field reconstruction algorithms still face the following challenges: requiring large amounts of labeled data, time-consuming label data acquisition, slow training speed, and difficulty generalizing under different conditions, thus limiting their application scenarios. Summary of the Invention

[0004] In response to the defects of the existing technology and the need for improvement, the present invention provides a self-supervised light field three-dimensional high-resolution reconstruction method, system and medium. Its purpose is to provide high-resolution information through wide-field imaging, and use self-supervised learning to improve the light field resolution during the imaging process of similar samples, thereby solving the technical problems of existing light field super-resolution imaging, such as slow speed, poor fidelity of axial super-resolution imaging, and difficulty in generalization.

[0005] To achieve the above-mentioned objectives, according to one aspect of the present invention, a self-supervised light field three-dimensional high-resolution reconstruction method is provided, comprising: self-supervised training stages S1-S3 and a reconstruction stage S4; S1, performing simulated mapping on the high-resolution wide-field image of the sample to generate high-resolution simulated light field image frames at different viewing angles; S2, downsampling each of the high-resolution simulated light field image frames to obtain corresponding low-resolution simulated light field image frames, wherein the resolution of the low-resolution simulated light field image frames matches the resolution of the low-resolution light field image of the sample, wherein the low-resolution light field image includes light field image frames at each viewing angle; S3, performing self-supervised training on a two-dimensional super-resolution network using the low-resolution simulated light field image frames as input and the high-resolution simulated light field image frames as labels; S4, using the trained two-dimensional super-resolution network, performing super-resolution processing on the low-resolution light field image of a target of the same category as the sample, and performing three-dimensional reconstruction on the image obtained by the super-resolution processing to obtain a super-resolution three-dimensional image of the target.

[0006] Furthermore, when the target object is the sample, the method also includes: imaging the sample using a wide-field imaging system and a light-field imaging system respectively, and obtaining a high-resolution wide-field image and a low-resolution light-field image of the sample accordingly; wherein a switching device is provided at the intersection of the optical paths of the wide-field imaging system and the light-field imaging system, and the switching device is used to connect to one of the wide-field imaging system and the light-field imaging system for imaging at the same time.

[0007] Furthermore, the respectively using a wide-field imaging system and a light-field imaging system to image the sample specifically includes: respectively using a wide-field imaging system and a light-field imaging system to image the same field of view of the sample.

[0008] Furthermore, the switching device is a flippable mirror; when the flippable mirror is flipped into the optical path, one of the wide-field imaging system and the light-field imaging system performs imaging; when the flippable mirror is flipped out of the optical path, the other of the wide-field imaging system and the light-field imaging system performs imaging.

[0009] Furthermore, the S1 specifically includes: deconvolving the high-resolution wide-field image of the sample; performing a convolution operation on the deconvolved high-resolution wide-field image using the point spread functions at different depths and directions of the light field image to obtain a high-resolution simulated light field image frame at the corresponding viewing angle.

[0010] Furthermore, when the low-resolution light field image of the sample is an image obtained by spatial light field imaging, the point spread function of the light field image at different depths and directions is obtained by rearranging the point spread function of the spatial light field image in the phase space domain.

[0011] Furthermore, the S4 specifically includes: inputting the light field image frames at each viewing angle in the low-resolution light field image of the target object into the trained two-dimensional super-resolution network to obtain high-resolution inference light field image frames at each viewing angle; performing deconvolution operations on the high-resolution inference light field image frames at each viewing angle with the point spread function in the corresponding depth and direction, respectively, to obtain a super-resolution three-dimensional image of the target object.

[0012] Furthermore, the two-dimensional super-resolution network is a lightweight convolutional neural network.

[0013] According to another aspect of the present invention, a self-supervised light field three-dimensional high-resolution reconstruction system is provided, comprising: a processor; and a memory storing a computer-executable program, which, when executed by the processor, causes the processor to perform the self-supervised light field three-dimensional high-resolution reconstruction method described above.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for self-supervised light field three-dimensional high-resolution reconstruction as described above is implemented.

[0015] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0016] (1) A self-supervised light field three-dimensional high-resolution reconstruction method is provided. The method uses the resolution information provided by the high-resolution wide-field image and adopts a self-supervised learning two-dimensional super-resolution network to improve the resolution of the light field images at each perspective, thereby achieving fast and real-time light field super-resolution three-dimensional imaging, breaking through the axial resolution hardware limit of light field imaging, and having a high speed. Compared with the existing light field super-resolution three-dimensional imaging technology based on high-resolution factual three-dimensional data, the method has good practicality and verifiability because the wide-field image data acquisition difficulty is low and the system is convenient, and it is easy to realize the optical path fusion with the light field system.

[0017] (2) When performing three-dimensional high-resolution reconstruction of the sample, a switching device is used to sequentially realize two types of imaging of the sample - wide-field imaging and light-field imaging, to achieve in-situ acquisition, ensuring that wide-field imaging and light-field imaging are performed on the same field of view of the sample, and obtaining more consistent light-field images and wide-field images, thereby improving the super-resolution performance of the self-supervised network for low-resolution light-field images and making the method have good generalization.

[0018] (3) A lightweight convolutional neural network is selected as the two-dimensional super-resolution network to achieve a lightweight design, thereby further improving the speed of light field super-resolution three-dimensional imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a method for self-supervised light field 3D high-resolution reconstruction provided by an embodiment of the present invention;

[0020] Figure 2 Schematic diagram of the self-supervised light field 3D high-resolution reconstruction method provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the optical path structure of a microscopic imaging device provided in an embodiment of the present invention;

[0022] Figure 4 Examples of light field images and wide field images obtained by the method provided in the embodiments of the present invention;

[0023] Figure 5 An example of deconvolution of a wide-field image in the method provided in an embodiment of the present invention;

[0024] Figure 6 An example of high-resolution light field image 3D reconstruction in the method provided in an embodiment of the present invention;

[0025] Figure 7 For spatial light fields, the method provided in the embodiment of the present invention is used to perform high-resolution 3D reconstruction of light field images, and the results are compared with those of direct 3D reconstruction of light field images.

[0026] Figure 8 For Fourier light field, the effects of high-resolution 3D reconstruction of light field images using the method provided by the embodiment of the present invention are compared with those of direct 3D reconstruction of light field images. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0028] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0029] Example 1

[0030] A self-supervised light field 3D high-resolution reconstruction method, combined with Figures 1-8 , the self-supervised light field 3D high-resolution reconstruction method in this embodiment is described in detail. Figure 1, the method includes a self-supervised training phase (operation S1-operation S3) and a reconstruction phase (operation S4), the principle of which is as follows Figure 2 shown.

[0031] In operation S1 , a high-resolution wide-field image of a sample is simulated and mapped to generate high-resolution simulated light-field image frames at different viewing angles.

[0032] Before performing operation S1, a high-resolution wide-field image of the sample needs to be acquired; before performing operation S4, a low-resolution light-field image of the target object needs to be acquired.

[0033] When the target object in the reconstruction phase is not the sample used in the self-supervised training phase, the acquisition process of the sample's high-resolution wide-field image and the acquisition process of the target object's low-resolution light-field image are independent of each other.

[0034] When the target object in the reconstruction phase is the sample used in the self-supervised training phase, the method further includes: imaging the sample using a widefield imaging system and a lightfield imaging system, respectively, to obtain a high-resolution widefield image and a low-resolution lightfield image of the sample. Preferably, the widefield imaging system and the lightfield imaging system are used to image the same field of view of the sample to obtain a more consistent lightfield image and widefield image.

[0035] In this embodiment, a switching device is provided at the intersection of the optical paths of the wide-field imaging system and the light-field imaging system, and the switching device is used to simultaneously access one of the wide-field imaging system and the light-field imaging system for imaging.

[0036] Preferably, the switching device is a flip mirror. When the flip mirror is flipped into the optical path, one of the wide-field imaging system and the light-field imaging system performs imaging; when the flip mirror is flipped out of the optical path, the other of the wide-field imaging system and the light-field imaging system performs imaging.

[0037] In this embodiment, simulation mapping is performed according to the parameters of the light field imaging system to obtain high-resolution simulated light field image frames at various viewing angles as label data for self-supervised training samples. Specifically, operation S1 includes the following sub-operations S11 and S12.

[0038] In sub-operation S11, a high-resolution wide-field image of the sample is deconvolved.

[0039] In sub-operation S12, a convolution operation is performed on the deconvolved high-resolution wide-field image using point spread functions at different depths and directions of the light field image to obtain a high-resolution simulated light field image frame at a corresponding viewing angle.

[0040] When the low-resolution light field image of the sample is an image obtained by spatial light field imaging, the point spread functions of the light field image at different depths and directions are obtained by rearranging the point spread functions of the spatial light field image in the phase space domain.

[0041] The arrangement is as follows: the spatial light field point spread function is converted from the airspace domain (x, y, z) to the phase space domain (x, y, z, view). The conversion method is to extract the pixels at the same position in each microlens to obtain the viewing angle information, and then extract the corresponding two-dimensional point spread function for each viewing angle and each depth.

[0042] In operation S2, each high-resolution simulated light field image frame is downsampled to obtain a corresponding low-resolution simulated light field image frame, where the resolution of the low-resolution simulated light field image frame matches the resolution of the low-resolution light field image of the sample, wherein the low-resolution light field image includes light field image frames at each viewing angle.

[0043] In this embodiment, low-resolution simulated light field image frames at various viewing angles are used as training inputs, and corresponding high-resolution simulated light field image frames are used as label data to form self-supervised training sample images.

[0044] Operation S3 performs self-supervised training on a two-dimensional super-resolution network using the low-resolution simulated light field image frame as input and the high-resolution simulated light field image frame as a label.

[0045] Preferably, the 2D super-resolution network is a lightweight convolutional neural network, such as UNet. Because lightweight convolutional neural networks are used, training speed is fast and meets the requirements of self-supervised training. It should be noted that the 2D super-resolution network can also be other 2D networks.

[0046] The 2D super-resolution network is self-supervised trained as follows: the training loss function and convergence conditions apply to conventional image super-resolution loss functions and convergence conditions.

[0047] In this embodiment, the objects of the self-supervised training stage and the reconstruction stage are samples of the same type, and the image characteristics are highly consistent. Self-supervised training is performed, so there is no need to train with a large amount of data to ensure that the two-dimensional super-resolution network has good generalization performance, and there is no distortion during image super-resolution due to the small amount of training data and overfitting of the two-dimensional super-resolution network.

[0048] Operation S4, using the trained two-dimensional super-resolution network, super-resolution processing is performed on the low-resolution light field image of the target of the same category as the sample, and three-dimensional reconstruction is performed on the image obtained by the super-resolution processing to obtain a super-resolution three-dimensional image of the target.

[0049] In this embodiment, operation S4 specifically includes the following sub-operation S41 and sub-operation S42.

[0050] In sub-operation S41 , the light field image frames at each viewing angle in the low-resolution light field image of the target object are input into the trained two-dimensional super-resolution network to obtain high-resolution inference light field image frames at each viewing angle.

[0051] In sub-operation S42 , deconvolution operations are performed on the high-resolution inferred light field image frames at each viewing angle and the point spread functions at the corresponding depths and directions to obtain a super-resolution three-dimensional image of the target volume.

[0052] The method is described by taking the light field imaging system as a Fourier light field imaging system, the light field image as a Fourier light field image, and the target object as a sample as an example. The method includes the following steps 1 to 4.

[0053] See Figure 3 Using a commercial inverted fluorescence microscope, they constructed both the Fourier light field and widefield optical paths. A switching mechanism based on a flip-up mirror was designed to switch between the two modalities. This allows both widefield and Fourier light field detection within the same field of view of the fluorescence microscope. Widefield detection provides high-resolution label data, while Fourier light field provides information from different perspectives. By imaging the same scene using both modalities, the network can be trained using only a single widefield image.

[0054] Step 1: Use the Fourier light field imaging system and the wide field imaging system to image the same field of view of the sample, and obtain a low-resolution light field image and a high-resolution wide field image of the sample. The light field image consists of image frames from various perspectives, such as Figure 4 shown.

[0055] Step 2: Construct self-supervised training samples. Perform simulation mapping on the high-resolution wide-field image obtained in step 1 to obtain a high-resolution simulated light field image frame, which serves as the label data for the self-supervised training sample. Downsample the high-resolution simulated light field image frame to obtain a corresponding low-resolution simulated light field image frame, which serves as the input image for the self-supervised training sample. The low-resolution simulated light field image frame matches the image frame resolution of the low-resolution light field image. The specific process of simulation mapping is as follows:

[0056] Step 2-1: Deconvolve the high-resolution wide-field image obtained in step 1 to obtain a deconvoluted high-resolution wide-field image, such as Figure 5 shown.

[0057] Step 2-2: Based on the imaging parameters of the light field imaging system, the deconvolved high-resolution widefield image is convolved using point spread functions at different depths and directions of the light field image to obtain high-resolution simulated light field image frames at each viewing angle. In this embodiment, the deconvolved high-resolution widefield image is convolved layer by layer and view by view using point spread functions at 41 layers of the 7 viewing angles of the Fourier light field. Simulation mapping yields 287 high-resolution light field images, which are then cropped into 4,600 32*32 pixel images as label data.

[0058] Step 2-3: Downsample the high-resolution simulated light field image frame obtained in step 2-2 to obtain a corresponding low-resolution simulated light field image frame. The downsampling method in this embodiment is to merge adjacent pixels to obtain a simulated light field view with the same resolution as the actual light field view as the input data of the network.

[0059] Step 2-4: Use low-resolution simulated light field image frames at each viewing angle as training input and high-resolution simulated light field image frames at the corresponding viewing angle as label data to form self-supervised training sample images.

[0060] Step 3: Self-supervised training: Use the self-supervised training samples obtained in step 2 to train a 2D super-resolution network to obtain a converged light field image frame super-resolution network.

[0061] The 2D super-resolution network is trained as follows: the self-supervised training data obtained in step 2 is used as the training data for the 2D super-resolution network. The 2D super-resolution network used in this embodiment is based on the Unet network, with a learning rate of 0.0004 and a loss function of mean square error. The convergence condition is determined by the quality of the actual light field inferred by the network, which is generally around 400 iterations.

[0062] Step 4: Input the light field image frames of each perspective of the low-resolution light field image of the sample obtained in step 1 into the light field image frame super-resolution network obtained in step 3, output high-resolution inference light field image frames of each perspective with a resolution equivalent to that of the high-resolution wide-field image, and perform 3D reconstruction to obtain a super-resolution 3D image of the sample. This embodiment uses Richard-Lucey deconvolution for 3D reconstruction, and the results are as follows: Figure 6 shown.

[0063] Because the network has small parameters and is optimized for a single scene, training can be completed in minutes. The Fourier light field image is fed into a trained 2D super-resolution network to produce a super-resolved light field image. A well-trained 2D super-resolution network can improve the viewing angle of the Fourier light field to the resolution of a wide-field image in every direction, while maintaining the Fourier light field's characteristic large depth of field.

[0064] The obtained super-resolution light field map is reconstructed into three dimensions to obtain a high-resolution three-dimensional image. The sample three-dimensional image obtained by theoretical deconvolution of the output image has the advantages of high resolution, near isotropy, and large depth of field, such as Figure 7 As shown, the resolution of the high-resolution three-dimensional reconstruction image obtained in this embodiment is greatly improved compared to the resolution of the Fourier light field reconstruction image.

[0065] If the Fourier light field imaging system in the above example is replaced with a spatial light field imaging system, the implementation process is basically the same as the above example. The difference is that its step 2-2 is specifically as follows: the spatial light field image is pixel-permuted to obtain information of different viewpoints of the sample, and the simulated spatial light field point spread function is permuted to obtain point spread functions in different directions and layers. It is then convolved with the wide-field image to obtain high-resolution simulated light field image frames of each viewpoint; the spatial light field image is input into a trained two-dimensional super-resolution network to obtain a super-resolution light field image. A well-trained network can effectively improve the reconstruction resolution of the spatial light field while maintaining its advantage of a large field of view. Figure 8 The resolution of the high-resolution three-dimensional reconstruction image obtained in this embodiment is greatly improved compared to the resolution of the spatial light field reconstruction image.

[0066] Example 2

[0067] A self-supervised light field 3D high-resolution reconstruction system comprises: a processor; and a memory storing a computer-executable program. When executed by the processor, the program causes the processor to perform the self-supervised light field 3D high-resolution reconstruction method described above. The related technical solutions are the same as those in Example 1 and are not further described here.

[0068] Example 3

[0069] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned self-supervised light field three-dimensional high-resolution reconstruction method. The related technical solutions are the same as those in Example 1 and will not be repeated here.

[0070] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A self-supervised light field 3D high-resolution reconstruction method, characterized in that: include: Self-supervised training phases S1-S3 and reconstruction phase S4; S1, simulates mapping the high-resolution wide-field image of the sample to generate high-resolution simulated light-field image frames at different viewing angles; S2, downsampling each of the high-resolution simulated light field image frames to obtain a corresponding low-resolution simulated light field image frame, wherein the resolution of the low-resolution simulated light field image frame matches the resolution of the low-resolution light field image of the sample, wherein the low-resolution light field image includes a light field image frame at each viewing angle; S3, performing self-supervised training on a two-dimensional super-resolution network using the low-resolution simulated light field image frame as input and the high-resolution simulated light field image frame as a label; S4, using the trained two-dimensional super-resolution network, performing super-resolution processing on a low-resolution light field image of a target of the same category as the sample, and performing three-dimensional reconstruction on the image obtained by the super-resolution processing to obtain a super-resolution three-dimensional image of the target; When the target object is the sample, the self-supervised light field three-dimensional high-resolution reconstruction method further includes: using a wide-field imaging system and a light field imaging system to image the same field of view of the sample, respectively, to obtain a high-resolution wide-field image and a low-resolution light field image of the sample; The S1 specifically includes: deconvolving the high-resolution wide-field image of the sample; performing a convolution operation on the deconvolved high-resolution wide-field image using the point spread functions at different depths and directions of the low-resolution light field image to obtain a high-resolution simulated light field image frame at the corresponding viewing angle.

2. The self-supervised light field 3D high-resolution reconstruction method according to claim 1, characterized in that: A switching device is provided at the intersection of the optical paths of the wide-field imaging system and the light-field imaging system, and the switching device is used to simultaneously connect to one of the wide-field imaging system and the light-field imaging system for imaging.

3. The self-supervised light field 3D high-resolution reconstruction method according to claim 2, wherein: The switching device is a flippable reflector; When the flippable mirror is flipped into the optical path, one of the wide-field imaging system and the light-field imaging system performs imaging; When the flippable mirror is flipped out of the optical path, the other of the wide-field imaging system and the light-field imaging system performs imaging.

4. The self-supervised light field 3D high-resolution reconstruction method according to claim 1, wherein: When the low-resolution light field image of the sample is an image obtained by spatial light field imaging, the point spread function of the light field image at different depths and directions is obtained by rearranging the point spread function of the spatial light field image in the phase space domain.

5. The self-supervised light field 3D high-resolution reconstruction method according to claim 1, wherein: The S4 specifically includes: Inputting the light field image frames at each viewing angle in the low-resolution light field image of the target object into the trained two-dimensional super-resolution network to obtain high-resolution inference light field image frames at each viewing angle; The high-resolution inference light field image frames at each viewing angle are deconvolved with the point spread functions in the corresponding depth and direction to obtain a super-resolution three-dimensional image of the target body.

6. The self-supervised light field 3D high-resolution reconstruction method according to claim 1, wherein: The two-dimensional super-resolution network is a lightweight convolutional neural network.

7. A self-supervised light field 3D high-resolution reconstruction system, characterized in that: include: processor; A memory storing a computer executable program, wherein when the program is executed by the processor, the processor executes the self-supervised light field three-dimensional high-resolution reconstruction method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for self-supervised light field three-dimensional high-resolution reconstruction according to any one of claims 1 to 6 is implemented.

Images