Three-photon high-speed imaging system based on deep learning

Abstract

The invention relates to a three-photon high-speed imaging system based on deep learning, and the system comprises a three-photon image collection unit which is used for obtaining original three-photon imaging data; the deep learning super-resolution noise reduction network training unit is used for training the constructed deep learning super-resolution noise reduction network; wherein the deep learning super-resolution noise reduction network is a generative adversarial network; and the deep learning image recovery unit is used for carrying out image recovery on the actually acquired three-photon imaging data by adopting the trained deep learning super-resolution noise reduction network. Compared with the prior art, high-speed imaging of the three-photon microscope is realized, and relatively high signal-to-noise ratio and spatial resolution are maintained at the same time.

Description

Technical Field

[0001] The present invention relates to the field of optical microscopic imaging technology, and in particular to a three-photon high-speed imaging system based on deep learning. Background Art

[0002] As a non-invasive observation method, optical microscopy is widely used in the structural and functional imaging of biological tissues and the observation of various microscopic structures. The fluorescence microscope obtained by combining it with fluorescent labeling technology can achieve specific imaging and has become one of the most powerful tools for life science research.

[0003] As a high-resolution fluorescence microscopy technique based on nonlinear excitation effects, three-photon microscopy usually uses a 1300nm or longer wavelength laser to emit three simultaneously absorbed photons to excite fluorescent dyes, which is suitable for neuroscience and oncology applications. Compared with traditional single-photon or two-photon lasers, three-photon lasers have the following advantages: 1) The use of longer wavelengths can reduce the impact of light scattering and increase the penetration depth of the illumination beam into the sample; 2) The nonlinear characteristics of three-photon microscopy limit the excitation target to a smaller volume, reduce out-of-focus light, and minimize photobleaching on biological samples. The above advantages give it advantages in visualizing in vivo and ex vivo tissue morphology and physiology at the cellular level deep in scattering tissues and rapid volume imaging.

[0004] Compared with two-photon imaging, three-photon microscopy requires higher single pulse energy. Under the current hardware limitations, the pulse repetition frequency of three-photon lasers is very low, usually around 1MHz. This hardware condition causes three-photon imaging to fail to balance imaging speed and imaging resolution when scanning. Up to now, the imaging speed of three-photon imaging applications is mostly at a frame rate of 5-6Hz. Such imaging conditions have the following disadvantages: 1) The imaging speed is too slow, which requires longer imaging time when imaging neuronal structures. This is not friendly to some samples with high time requirements and unstable states. In addition, the single pulse energy of three-photons is high. Although long-term imaging will not produce obvious immune responses, it will more or less cause neuronal abnormalities; 2) In neuronal calcium imaging, the speed of 5-6Hz is not enough to record the calcium signals of the f series in the GCaMP series, and these f series calcium signals are often an important tool for decoding the relationship between behavior and neural activity.

[0005] In summary, there is an urgent need to design a high-speed three-photon imaging system that can achieve high-speed imaging of three-photon microscopy while maintaining a relatively high signal-to-noise ratio and spatial resolution. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a three-photon high-speed imaging system based on deep learning, which can realize high-speed imaging of three-photon microscope while maintaining relatively high signal-to-noise ratio and spatial resolution.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] The present invention provides a three-photon high-speed imaging system based on deep learning, comprising:

[0009] A three-photon image acquisition unit, used for obtaining original three-photon imaging data;

[0010] A deep learning super-resolution denoising network training unit is used to train the constructed deep learning super-resolution denoising network;

[0011] The deep learning image restoration unit is used to perform image restoration on the actually collected three-photon imaging data using a trained deep learning super-resolution denoising network.

[0012] Preferably, the three-photon image acquisition unit comprises a three-photon microscopic imaging device and a storage device.

[0013] Preferably, the three-photon microscopic imaging device controls the sampling rate, signal-to-noise ratio and imaging depth of three-photon imaging by adjusting the excitation power and scanning frequency parameters.

[0014] Preferably, the three-photon imaging data is one of three-photon structural imaging data and three-photon calcium imaging data.

[0015] Preferably, the training set acquisition process of the deep learning super-resolution denoising network includes:

[0016] Perform noise reduction on the original three-photon imaging data to obtain label data of the training set;

[0017] The original three-photon imaging data is downsampled to obtain the input data of the training set.

[0018] Preferably, the original three-photon imaging data is subjected to denoising to obtain label data of the training set, and the three-dimensional structural imaging data or the three-photon calcium imaging data containing the time signal is subjected to two-dimensional processing frame by frame, or a three-dimensional network structure is used to perform overall denoising and super-resolution on the three-photon imaging data to obtain label data of the training set.

[0019] Preferably, the denoising process is performed on the original three-photon imaging data to obtain the label data of the training set, specifically: one or more of a multi-frame averaging method, a spatial adaptive filtering method, a frequency domain filtering method, a diffusion-based filtering method, a statistical method, and a deep learning-based denoising method are used to perform denoising and enhancement processing on the original three-photon imaging data to obtain the label data of the training set.

[0020] Preferably, the deep learning super-resolution denoising network is a generative adversarial network.

[0021] Preferably, the generator of the generative adversarial network adopts a network structure including but not limited to an encoder-decoder network Unet and a deep back projection network DBPN; the discriminator of the generative adversarial network adopts a classification network structure including but not limited to VGG16.

[0022] Preferably, the deep learning super-resolution denoising network is a cascaded super-resolution network and denoising network based on a deep learning model.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The deep learning-based three-photon imaging system of the present invention restores low signal-to-noise ratio and low-resolution three-photon images through a deep learning super-resolution denoising network, thereby achieving high-speed imaging of a three-photon microscope while maintaining a relatively high signal-to-noise ratio and spatial resolution, thereby improving the research efficiency of neuroscience including neuronal calcium signal analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural block diagram of the three-photon high-speed imaging system based on deep learning of the present invention;

[0026] Figure 2 It is a deep learning image restoration unit;

[0027] Figure 3 It is a structural diagram of the feature extraction module;

[0028] Figure 4 It is a schematic diagram of the structure of the DBPN super-resolution module;

[0029] Figure 5 It is the GAN network training unit;

[0030] Figure 6 This is a picture of the restoration effect of three-photon structural imaging of mouse brain slices in vitro;

[0031] Figure 7 This is a picture of the recovery effect of three-photon calcium imaging of brain tissue in mice in vivo;

[0032] Figure 8This is the effect of three-photon three-dimensional structural imaging restoration of mouse brain slices in vitro. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] Example

[0035] This embodiment uses mouse brain neurons as data samples, and uses a three-photon high-speed imaging system based on deep learning to improve the resolution and signal-to-noise ratio of neuronal structure imaging and calcium imaging signals under the same scanning time.

[0036] like Figure 1 and Figure 2 As shown, this embodiment provides a three-photon high-speed imaging system based on deep learning, including:

[0037] A three-photon image acquisition unit, used for obtaining original three-photon imaging data;

[0038] A deep learning super-resolution denoising network training unit is used to train the constructed deep learning super-resolution denoising network;

[0039] The deep learning image restoration unit is used to perform image restoration on the actually collected three-photon imaging data using a trained deep learning super-resolution denoising network.

[0040] Next, each unit of the system of the present invention is introduced in detail.

[0041] 1. Three-photon image acquisition unit

[0042] The three-photon image acquisition unit includes a three-photon microscopic imaging device and a storage device. The three-photon microscopic imaging device controls the sampling rate, signal-to-noise ratio and imaging depth of three-photon imaging by adjusting the excitation power and scanning frequency parameters.

[0043] The three-photon microscopy device uses a laser with an operating wavelength of 1300 nm (Light Conversion, CB3-20W and I-OPA-TW-F) and a 6210H galvanometer (Cambridge Technology, 6210H6SD11154) to scan the sample. After a series of filters (Semrock), all excited fluorescence is detected by a GaAsP photomultiplier tube (Hamamatsu, H7422-40), and the signal emitted by the photomultiplier tube is amplified by an amplifier (Hamamatsu C12419). The control and data acquisition of the entire three-photon imaging system are controlled by MATLAB software (Vidrio Technologies, LLC, ScanImage); all images are collected using a water-immersion 25× objective (Olympus, XLPLN25XWMP2, 25X NA1.05).

[0044] The scanning mode of the three-photon image acquisition unit can be selected as linear scanning or nonlinear scanning.

[0045] 2. Deep learning super-resolution denoising network training unit

[0046] The deep learning super-resolution denoising network of this embodiment is a generative adversarial network;

[0047] The training process of the generative adversarial network includes:

[0048] 1) Perform multi-frame averaging on the low signal-to-noise ratio, high-resolution images acquired by the three-photon image acquisition unit to improve their signal-to-noise ratio and obtain the label data (ground truth) of the training data set.

[0049] 2) Down-sampling the low signal-to-noise ratio and low-resolution images acquired by the three-photon image acquisition unit to obtain low signal-to-noise ratio and low-resolution three-photon images that simulate actual acquisition as input data of the training data set.

[0050] 3) Using the training set data obtained in the above steps to train the neural network, thereby obtaining a network model that can simultaneously perform noise reduction and super-resolution on three-photon images.

[0051] Optionally, the original three-photon imaging data is subjected to denoising to obtain label data of the training set, and the three-dimensional structural imaging data or the three-photon calcium imaging data containing the time signal is subjected to two-dimensional processing frame by frame, or a three-dimensional network structure is used to perform overall denoising and super-resolution on the three-photon imaging data to obtain label data of the training set.

[0052] Optionally, one or more of a multi-frame averaging method, a spatial adaptive filtering method, a frequency domain filtering method, a diffusion-based filtering method, a statistical method, and a deep learning-based denoising method are used to perform noise reduction and enhancement processing on the original three-photon imaging image to obtain a training ground truth.

[0053] In some embodiments of the present invention, the generator of the generative adversarial network adopts an end-to-end super-resolution denoising network structure including but not limited to Unet, a deep back projection network DBPN, a convolutional neural network (CNN) or a neural network model based on an attention mechanism, such as (Vision Transformer, ViT). Further, a feature extraction module and a DBPN super-resolution module are used to form the generator in the GAN network, and the feature extraction module and the DBPN super-resolution module are respectively composed of Figure 3 , Figure 4 The discriminator of the generated adversarial network adopts a classification network structure including but not limited to VGG16. Figure 5 The training flow chart of the GAN network training unit is shown.

[0054] Optionally, the model selected by the deep learning network training unit may be a single network model with simultaneous noise reduction and super-resolution functions, or a network model for super-resolution and a network model for noise reduction may be trained separately in a cascade manner.

[0055] During the training process, low signal-to-noise ratio, low-resolution images of mouse brain neuron structures are input into the deep neural network, and the generator generates predicted high signal-to-noise ratio, high-resolution images; the predicted images are compared with the labeled images in the training set to obtain the loss function, and finally the loss value is used to calculate the gradient and update the network parameters; the loss function includes both the generator loss function and the discriminator loss function.

[0056] The deep learning image restoration unit uses the trained GAN network generator model to reduce noise and super-resolve the low signal-to-noise ratio and low-resolution mouse neuron images actually collected, thereby obtaining the predicted high signal-to-noise ratio and high-resolution neuron images. Through frame-by-frame processing, a three-dimensional neuron structure image or neuron activity image with restored signal-to-noise ratio and resolution is obtained.

[0057] Using the deep learning image recovery unit in this embodiment can effectively restore the low signal-to-noise ratio and low-resolution three-photon data actually collected into high signal-to-noise ratio and high-resolution three-photon image data. Figure 6 The restoration effect of the three-photon structural imaging data of an in vitro mouse brain slice using an embodiment of the present invention is demonstrated, and from left to right are the collected original image data and the restored high signal-to-noise ratio image.

[0058] Using the deep learning image restoration unit in this embodiment can effectively restore the low signal-to-noise ratio and low-resolution three-photon calcium imaging data actually collected into high signal-to-noise ratio and high-resolution three-photon calcium imaging data. Figure 7 The restoration effect of the three-photon calcium imaging data of the in vivo brain tissue of mice using this embodiment is demonstrated, and from left to right are the collected original image data and the restored high signal-to-noise ratio image.

[0059] Using the deep learning image restoration unit in this embodiment can effectively restore the low signal-to-noise ratio, low-resolution three-dimensional three-photon data actually collected into high signal-to-noise ratio, high-resolution three-dimensional three-photon image data. Figure 8 The restoration effect of the three-photon three-dimensional structural imaging data of an in vitro mouse brain slice using an embodiment of the present invention is demonstrated, and from left to right are the collected original image data and the restored high signal-to-noise ratio image.

[0060] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A three-photon high-speed imaging system based on deep learning, characterized in that: include: A three-photon image acquisition unit, used for obtaining original three-photon imaging data; A deep learning super-resolution denoising network training unit is used to train the constructed deep learning super-resolution denoising network; The deep learning image restoration unit is used to perform image restoration on the actually collected three-photon imaging data using a trained deep learning super-resolution denoising network.

2. A three-photon high-speed imaging system based on deep learning according to claim 1, characterized in that: The three-photon image acquisition unit includes a three-photon microscopic imaging device and a storage device.

3. A three-photon high-speed imaging system based on deep learning according to claim 2, characterized in that: The three-photon microscopic imaging device controls the sampling rate, signal-to-noise ratio and imaging depth of three-photon imaging by adjusting the excitation power and scanning frequency parameters.

4. The three-photon high-speed imaging system based on deep learning according to claim 1, characterized in that: The three-photon imaging data is one of three-photon structural imaging data and three-photon calcium imaging data.

5. A three-photon high-speed imaging system based on deep learning according to claim 4, characterized in that: The training set acquisition process of the deep learning super-resolution denoising network includes: Perform noise reduction on the original three-photon imaging data to obtain label data of the training set; The original three-photon imaging data is downsampled to obtain the input data of the training set.

6. A three-photon high-speed imaging system based on deep learning according to claim 5, characterized in that: The original three-photon imaging data is subjected to denoising to obtain label data of the training set, and the three-dimensional structural imaging data or the three-photon calcium imaging data containing the time signal is subjected to two-dimensional processing frame by frame, or the three-photon imaging data is subjected to overall denoising and super-resolution using a three-dimensional network structure to obtain label data of the training set.

7. The three-photon high-speed imaging system based on deep learning according to claim 5, characterized in that: The denoising process of the original three-photon imaging data to obtain the label data of the training set is specifically: one or more of a multi-frame averaging method, a spatial adaptive filtering method, a frequency domain filtering method, a diffusion-based filtering method, a statistical method, and a deep learning-based denoising method are used to perform denoising and enhancement processing on the original three-photon imaging data to obtain the label data of the training set.

8. The three-photon high-speed imaging system based on deep learning according to claim 1, characterized in that: The deep learning super-resolution denoising network is a generative adversarial network.

9. A three-photon high-speed imaging system based on deep learning according to claim 8, characterized in that: The generator of the generative adversarial network adopts, including but not limited to, an encoder-decoder network Unet and a deep back projection network DBPN; the discriminator of the generative adversarial network adopts, including but not limited to, a classification network structure of VGG16.

10. The three-photon high-speed imaging system based on deep learning according to claim 1, characterized in that: The deep learning super-resolution denoising network is a cascaded super-resolution network and denoising network based on a deep learning model.

Images