A scanning-free high-resolution four-dimensional light field microscopic imaging method and system
Through scanning-free light field spatial resolution enhancement network and three-dimensional deconvolution technology, the problem of difficult time and spatial resolution in light field microscopy is solved, and four-dimensional high-resolution imaging is realized, reducing costs and expanding application scope.
Patent Information
- Application Number
- CN202410155015.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-04
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing light field microscopy imaging technology cannot achieve 4D high-resolution imaging with high time and spatial resolution at the same time, and it relies on complex mechanical scanning and expensive hardware, and has limited application scope.
Adopt a scanning-free light field spatial resolution enhancement network, obtain training data through multiple exposures, input low-resolution light field video for feature extraction and fusion, output high-resolution video, and realize four-dimensional high-resolution reconstruction through three-dimensional deconvolution , avoiding mechanical scanning process.
It realizes high-temporal and spatial resolution light field microscopy, reduces costs, is suitable for more observation scenarios, and is not affected by sample motion artifacts, and has a wide range of applications.
Smart Images

Figure CN117934285B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of computational optics, computational photography, computer vision and computer graphics, and more particularly to a non-scanning high-resolution four-dimensional light field microscopic imaging method and system. Background Art
[0002] At present, light field microscopy can collect three-dimensional information of a scene in a single exposure, making three-dimensional high-speed imaging of living organisms possible and becoming an important tool to further promote the development of life sciences and brain sciences.
[0003] In the prior art, light field microscopy is to place a microlens array into the image plane of a microscope, and move the sensor plane back to the focal length of the microlens. By rearranging the imaging results in space, the original two-dimensional light field acquisition image is converted into two-dimensional space and two-dimensional angle, and the two-dimensional spatial resolution is improved by two-dimensional mechanical scanning. If two-dimensional mechanical scanning is not used, the reuse of the original two-dimensional space and two-dimensional angle will reduce the spatial sampling rate by the angle number times, making the spatial resolution of three-dimensional imaging insufficient. If two-dimensional mechanical scanning is used, the temporal resolution of three-dimensional imaging will decrease by times the scanning period. Therefore, the prior art cannot achieve four-dimensional high-resolution (three-dimensional space, one-dimensional time) imaging.
[0004] Therefore, in view of the problem that it is difficult to balance the temporal and spatial resolutions of (scanning) light field microscopes, how to provide a non-scanning four-dimensional high-resolution light field microscopy method and system is an issue that technical personnel in this field urgently need to solve. Summary of the invention
[0005] In view of this, the present invention provides a scanless four-dimensional high-resolution light field microscopy imaging method and system. The scanless light field spatial resolution enhancement network restores the input high-frame rate and low-resolution light field video to a high-frame rate and high-resolution light field video, and then obtains a three-dimensional volume through deconvolution three-dimensional reconstruction, thereby achieving scanless light field four-dimensional (three-dimensional space, one-dimensional time) high-resolution reconstruction with good generalization, wide applicability and low cost, while taking into account high temporal and spatial resolution.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A non-scanning high-resolution four-dimensional light field microscopic imaging method, comprising:
[0008] Take static samples as targets and expose them multiple times to obtain training data;
[0009] Input the training data into the scanless light field spatial resolution enhancement network, rearrange the input low spatial resolution static light field image into a macro-pixel image and a phase space image, extract the macro-pixel features of the macro-pixel image and the phase space features of the phase space image respectively, fuse the macro-pixel features and the phase space features to output the reconstruction result, and train the scanless light field spatial resolution enhancement network with the high spatial resolution light field data;
[0010] Input the scanless high frame rate and low spatial resolution living body dynamic light field image into the trained scanless light field spatial resolution enhancement network to obtain a high frame rate and high spatial resolution living body dynamic light field video;
[0011] The high-frame-rate and high-spatial-resolution in vivo dynamic light field video is 3D deconvolved and 3D reconstructed to obtain a 4D high-resolution 3D volume reconstructed video.
[0012] Preferably, the specific process of obtaining training data is as follows:
[0013] Take multiple exposures of static samples to collect high spatial resolution light field data and use it as the true value;
[0014] A single-exposure low spatial resolution static light field image is extracted from the high-resolution light field data to form paired training data.
[0015] Preferably, the three-dimensional reconstruction method includes an iteratively optimized reconstruction method or is implemented by a deep three-dimensional reconstruction network.
[0016] A non-scanning high-resolution four-dimensional light field microscopic imaging system, comprising:
[0017] Data acquisition module: used to expose static samples multiple times to obtain training data;
[0018] Network training module: used to input training data into the scanless light field spatial resolution enhancement network, rearrange the input low spatial resolution static light field image into macro-pixel image and phase space image, extract macro-pixel features of the macro-pixel image and phase space features of the phase space image respectively, fuse the macro-pixel features and the phase space features and output the reconstruction result, and train the scanless light field spatial resolution enhancement network with high spatial resolution light field data;
[0019] Network inference module: used to input the scanless high frame rate and low spatial resolution living body dynamic light field image into the trained scanless light field spatial resolution enhancement network to obtain a high frame rate and high spatial resolution living body dynamic light field video;
[0020] 3D reconstruction module: used to perform 3D deconvolution and 3D reconstruction on high frame rate and high spatial resolution live dynamic light field videos to obtain 4D high-resolution 3D volume reconstruction videos.
[0021] Preferably, the specific process of obtaining training data is as follows:
[0022] Take multiple exposures of static samples to collect high spatial resolution light field data and use it as the true value;
[0023] A single-exposure low spatial resolution static light field image is extracted from the high-resolution light field data to form paired training data.
[0024] Preferably, the three-dimensional reconstruction method includes an iteratively optimized reconstruction method or is implemented by a deep three-dimensional reconstruction network.
[0025] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a non-scanning four-dimensional high-resolution light field microscopic imaging method and system, which has the following advantages:
[0026] 1) The present invention decouples the light field reconstruction process into the restoration of spatial resolution and the mapping of three-dimensional reconstruction, thereby achieving light field microscopic imaging with high temporal and spatial resolution.
[0027] 2) Different from scanning light field microscopes that require complex mechanical control and are easily affected by violent movement of the sample to produce motion artifacts, the present invention does not require complex and expensive hardware additions, has high resolution and low cost.
[0028] 3) Different from the end-to-end 3D reconstruction of light field images based on deep learning (the current light field 3D reconstruction technology based on deep learning directly maps from the light field end to the 3D reconstruction. On the one hand, the data difference is very large, and on the other hand, the real training data cannot be obtained in most scenarios, resulting in a heavy reliance on simulation data for network training. In practical applications, the scope of application is limited), the present invention only limits the scope of the network to the data domain of the light field. The input and output are both light fields, but the resolution is different. Therefore, the data difference is small, the network modeling is easier, and the interpretability is stronger. In addition, the training data only uses light field images, so it can be applied to more observation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0030] Figure 1 A flow chart of the scanning-free high-resolution four-dimensional light field microscopic imaging method provided by the present invention.
[0031] Figure 2This is the model training process of the present invention.
[0032] Figure 3 A flowchart of the network reasoning process provided by the present invention.
[0033] Figure 4 Schematic diagram of the scanning-free high-resolution four-dimensional light-field microscopic imaging system provided by the present invention. DETAILED DESCRIPTION
[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0035] The embodiment of the present invention discloses a non-scanning high-resolution four-dimensional light field microscopic imaging method, such as Figure 1 As shown, including:
[0036] Taking static samples as the target, multiple exposures are performed to obtain training data. Specifically, multiple exposures are performed on static samples to collect high spatial resolution light field data and use it as the true value. A single exposure low spatial resolution static light field image is extracted from the high-resolution light field data to form paired training data.
[0037] The training data is input into the scanless light field spatial resolution enhancement network, and the input low spatial resolution static light field image is rearranged into a macro-pixel image and a phase space image, and the macro-pixel features of the macro-pixel image and the phase space features of the phase space image are extracted respectively, and the macro-pixel features and the phase space features are fused to output the reconstruction result, and the scanless light field spatial resolution enhancement network is trained with high spatial resolution light field data, such as Figure 2 As shown;
[0038] The scanless high frame rate and low spatial resolution live dynamic light field image is input into the trained scanless light field spatial resolution enhancement network to obtain a high frame rate and high spatial resolution live dynamic light field video. The specific reasoning process is as follows: Figure 3 As shown, when processing the input scanless high frame rate and low spatial resolution living dynamic light field image during the inference process, the macro-pixel feature, phase space feature extraction module and feature fusion module are trained according to the above training process, and the parameters in the module are fixed.
[0039] The high frame rate and high spatial resolution live dynamic light field video is subjected to 3D deconvolution and 3D reconstruction to obtain a 4D high resolution 3D volume reconstruction video. The 3D reconstruction method includes an iterative optimization reconstruction method (Richardson-Lucy iterative deconvolution algorithm, etc.) or is implemented through a deep 3D reconstruction network.
[0040] A scanning-free high-resolution four-dimensional light-field microscopy system, such as Figure 4 As shown, including:
[0041] Data acquisition module: used to expose static samples multiple times to obtain training data; the specific process of obtaining training data is: expose static samples multiple times, collect high spatial resolution light field data and use it as the true value; extract a single exposure low spatial resolution static light field image from the high-resolution light field data to form paired training data.
[0042] Network training module: used to input training data into the scanless light field spatial resolution enhancement network, and rearrange the input low spatial resolution static light field image into a macro-pixel image and a phase space image, respectively extract the macro-pixel features of the macro-pixel image and the phase space features of the phase space image, and output the reconstruction result after fusing the macro-pixel features and the phase space features, and train the scanless light field spatial resolution enhancement network with high spatial resolution light field data;
[0043] Network inference module: used to input the scanless high frame rate and low spatial resolution live dynamic light field image into the trained scanless light field spatial resolution enhancement network to obtain a high frame rate and high spatial resolution live dynamic light field video; the specific inference process is as follows Figure 3 As shown, when processing the input scanless high frame rate and low spatial resolution living dynamic light field image during the inference process, the macro-pixel feature, phase space feature extraction module and feature fusion module are trained according to the above training process, and the parameters in the module are fixed.
[0044] 3D reconstruction module: used to perform 3D deconvolution and 3D reconstruction of high-frame-rate and high-spatial-resolution live dynamic light field videos to obtain 4D high-resolution 3D volume reconstruction videos. 3D reconstruction methods include iterative optimization reconstruction methods (Richardson-Lucy iterative deconvolution algorithm, etc.) or through deep 3D reconstruction networks.
[0045] The present invention innovatively uses a neural network to extract the intrinsic correlation of the light field imaging process (the correlation between single exposure and multiple exposures in data, and the input image of the single exposure itself satisfies the structural constraints of the light field data in terms of macro-pixels and phase space characteristics), and thereby restores the spatial resolution of the light field imaging without the need for a two-dimensional mechanical scanning process. On the one hand, it does not require the complex hardware assembly process required for mechanical scanning, and can be applied to various light field microscopic imaging systems. On the other hand, it decouples the high-resolution reconstruction task of the light field, realizes the application of deep learning technology in a more light field observation scenario, and thus realizes four-dimensional high-resolution light field microscopic imaging.
[0046] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0047] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A non-scanning high-resolution four-dimensional light field microscopy imaging method, characterized in that: include: Take static samples as targets and expose them multiple times to obtain training data; Input the training data into the scanless light field spatial resolution enhancement network, rearrange the input low spatial resolution static light field image into a macro-pixel image and a phase space image, extract the macro-pixel features of the macro-pixel image and the phase space features of the phase space image respectively, fuse the macro-pixel features and the phase space features to output the reconstruction result, and train the scanless light field spatial resolution enhancement network with the high spatial resolution light field data; Input the scanless high frame rate and low spatial resolution living body dynamic light field image into the trained scanless light field spatial resolution enhancement network to obtain a high frame rate and high spatial resolution living body dynamic light field video; Perform 3D deconvolution and 3D reconstruction on the high frame rate and high spatial resolution live dynamic light field video to obtain a 4D high-resolution 3D volume reconstruction video; The specific process of obtaining training data is as follows: Take multiple exposures of static samples to collect high spatial resolution light field data and use it as the true value; A single-exposure low spatial resolution static light field image is extracted from the high-resolution light field data to form paired training data.
2. A scanning-free high-resolution four-dimensional light field microscopy imaging method according to claim 1, characterized in that: The 3D reconstruction method includes an iteratively optimized reconstruction method or is implemented through a deep 3D reconstruction network.
3. A non-scanning high-resolution four-dimensional light field microscopy imaging system, characterized in that: include: Data acquisition module: used to expose static samples multiple times to obtain training data; Network training module: used to input training data into the scanless light field spatial resolution enhancement network, rearrange the input low spatial resolution static light field image into macro-pixel image and phase space image, extract macro-pixel features of the macro-pixel image and phase space features of the phase space image respectively, fuse the macro-pixel features and the phase space features and output the reconstruction result, and train the scanless light field spatial resolution enhancement network with high spatial resolution light field data; Network inference module: used to input the scanless high frame rate and low spatial resolution living body dynamic light field image into the trained scanless light field spatial resolution enhancement network to obtain a high frame rate and high spatial resolution living body dynamic light field video; 3D reconstruction module: used to perform 3D deconvolution and 3D reconstruction on the high frame rate and high spatial resolution live dynamic light field video to obtain a 4D high-resolution 3D volume reconstruction video; The specific process of obtaining training data is as follows: Take multiple exposures of static samples to collect high spatial resolution light field data and use it as the true value; A single-exposure low spatial resolution static light field image is extracted from the high-resolution light field data to form paired training data.
4. The scanning-free high-resolution four-dimensional light field microscopic imaging system according to claim 3, characterized in that: The 3D reconstruction method includes an iteratively optimized reconstruction method or is implemented through a deep 3D reconstruction network.
Citation Information
Patent Citations
Space-time joint optimization reconstruction method and system
CN110706346A
Non-scanning hyperresolution optical three-dimensional microscopic imaging method
CN110824681A
Light field super-resolution three-dimensional reconstruction method and system
CN113870433A
Light field three-dimensional reconstruction method and device, electronic equipment and computer storage medium
CN115100352A
Microscopic imaging system and method based on deep learning and light field imaging
CN115220211A
