Target perception high-speed imaging method based on deep learning

Abstract

The invention discloses a target perception high-speed imaging method based on deep learning. The method comprises the following steps: a) acquiring two-photon microscopic imaging data; b) performing neuron segmentation and activity information extraction in a target area: developing a target sensing model based on deep learning, performing three-stage processing of filtering enhancement-U-shaped network-parameter optimization to obtain a binary matrix, and multiplying the binary matrix with high-image-quality image data to obtain a segmentation result; c) building a deep learning network for training; and d) performing high-speed imaging on the target neurons by using the trained network. According to the method, the problems of data processing loss rate and positioning deviation in the traditional image segmentation technology are solved, the problems of long reconstruction time and the like caused by reduction of sampling data volume are solved, hardware parts of optical imaging equipment do not need to be transformed, the economic cost is remarkably reduced, the popularization potential of the method in various fields is enhanced, and the method is suitable for popularization and application. High-speed reconstruction of the activity of the target neuron is realized, and dynamic change detail information issued by the target neuron is restored.

Description

Technical Field

[0001] The present invention focuses on the frontier fields of biological imaging and medical image processing, and specifically relates to a target perception high-speed imaging method based on deep learning. Background Art

[0002] Two-photon microscopy uses the two-photon absorption process to excite fluorescent molecules by absorbing two near-infrared photons at the same time, achieving high-resolution observation and recording of deep biological samples, and is suitable for three-dimensional imaging of living animals. Compared with traditional single-photon imaging technology, two-photon microscopy technology has the advantages of deeper imaging depth and higher spatial resolution; compared with confocal microscopy technology, it has better biological tissue penetration ability. Therefore, it is of great significance for brain science / neuroscience research.

[0003] At present, the main challenges faced by two-photon microscopy are: 1. The balance between imaging depth and resolution. As the imaging depth increases, optical aberrations and scattering effects will lead to a decrease in imaging resolution and image quality, which places higher requirements on the design of the imaging system. 2. The phototoxicity problem of biological samples. Long-term laser irradiation can cause serious phototoxicity to living samples, limiting the imaging time. In addition, the point scanning imaging mode of traditional two-photon microscopy fundamentally limits its imaging flux and three-dimensional perception speed, and is easily disturbed by complex living imaging environments. This is also an important problem that needs to be solved urgently, and it is of great significance for brain science research such as revealing the brain's cognition, emotion, and behavior. Summary of the invention

[0004] Aiming at the widely used two-photon microscopy, the present invention proposes a target-aware high-speed imaging method based on deep learning to solve the problem of balancing imaging depth and resolution in two-photon microscopy. It can reproduce the neuronal activity image in the target area with higher frame rate and image quality, which has important guiding significance for the application of two-photon microscopy technology in neuroscience research.

[0005] The present invention adopts the following technical solution:

[0006] The deep learning-based high-speed imaging method for target-aware neuron segmentation is described in detail in the following steps:

[0007] Step 1) Two-photon microscopic imaging data acquisition: for the living organism transfected with the fluorescent probe, two-photon microscopic imaging is performed and microscopic imaging video data is acquired;

[0008] Step 2) Segmentation of neurons and extraction of activity information in the target area: Develop a target perception model based on deep learning, process the microscopic imaging video data obtained in step 1) through a beam splitter to obtain low-image-quality imaging video and high-image-quality image data; use the low-image-quality image data as model input, and process it in turn through a filter enhancement module, a U-shaped network module, and a parameter optimization module to obtain a binary matrix that retains key neuron information; multiply the binary matrix by the collected high-image-quality image data to obtain a segmentation result, that is, a low-frame-rate high-image-quality image that retains key neuron information;

[0009] Step 3) Building a deep learning network integrating compressed sensing for training: Building a deep learning network integrating compressed sensing, using the neuron segmentation image data obtained in step 2) and processed by the target perception model, and performing self-supervised training from low frame rate to high frame rate by downsampling;

[0010] Step 4) Using the trained network to perform high-speed imaging of the target neurons: The microscopic imaging video data to be processed is processed through step 2) to obtain segmented neuron image data (low frame rate and high image quality), and then processed through step 3) through the trained deep learning network for high-speed imaging restoration, thereby obtaining a high frame rate and high image quality neuron activity image.

[0011] The step 1) is specifically as follows:

[0012] Step 1.1) Into a specific region of a living biological sample to be observed, fluorescent protein is introduced by stereotaxic brain injection technology to achieve cell transfection.

[0013] Step 1.2) After the fluorescent protein is fully expressed, the sample is imaged and observed using a microscope, and the device collects video data of the fluorescent image, i.e., an image with a low frame rate and low image quality.

[0014] In the step 2), the low image quality data obtained after being processed by the beam splitter is used as the imaging light path, and the high image quality image data is used as the reference light path.

[0015] The filtering enhancement module in step 2) is specifically:

[0016] The microscopic imaging video data was filtered through a mean filter to suppress image noise. Subsequently, a filter for Poisson noise was designed to reduce noise interference and improve signal clarity. Temporal filtering and homomorphic filtering techniques were further used to eliminate signal fluctuations and anomalies. Finally, the signal-to-noise ratio (SNR) filter was used in combination with the non-negative matrix factorization (NMF) technology to process the SNR traces, achieving effective separation of different independent cell signals and outputting the filtered enhanced image.

[0017] The U-shaped network module in step 2) is specifically:

[0018] A U-shaped network based on deep learning is constructed. Through the self-supervised training process, the network weights are optimized to obtain the optimal parameter configuration. The optimal parameter configuration is further used to output the probability map corresponding to the image.

[0019] The parameter optimization module in step 2) is specifically:

[0020] Input the probability map obtained by the U-shaped network module, set the probability threshold to 0.98, retain the pixel area with a probability value greater than 0.98, and compare it with the image after filtering and enhancement processing to ensure that the data processed by the U-shaped network has a high degree of overlap with the previous data, thereby eliminating invalid information;

[0021] Then, the mean radius of the neuron (meanR) was set to 5.85, and the mean area parameter was calculated and determined as the estimated value of the neuron area to ensure the accuracy of the segmentation. The mean area parameter was calculated according to the following formula:

[0022]

[0023] Among them, meanR is the average radius of neurons, and pixSize is the pixel size value of 0.78.

[0024] Then, by setting the intersection-over-union ratio threshold (Thresh_IOU) to 0.7, the neurons in the image are evaluated to be fused, that is, Thresh_IOU is greater than 0.7, or separated, that is, Thresh_IOU is less than 0.7. If it is determined to be fused, it is regarded as one neuron, and the minimum area parameter is introduced and set to 46.2+120 to remove neurons that do not meet the minimum area size requirement to ensure the accurate morphological contour of the neuron; if it is determined to be separated, it is regarded as two neurons; finally, the continuous frame parameter (tau) is set to That is, within 30 consecutive frames, if a neuron appears more than 6 times, the neuron is retained to ensure that the firing activity of the neuron conforms to the calcium dynamics model. Finally, a binary matrix that highlights the key information of the neuron is obtained.

[0025] The step 3) is specifically as follows:

[0026] Step 3.1) Establishing a high-speed reconstruction network integrating compressed sensing and deep learning, wherein the neural network consists of a feature extraction module and an image reconstruction module;

[0027] Step 3.2) During the training phase, the target neuron image data after segmentation in step 2) (i.e., the downsampled low frame rate and high image quality neuron activity image) is used as the network input, and the network outputs a high frame rate and high image quality neuron activity image to complete the self-supervised training.

[0028] In the step 3.1):

[0029] The feature extraction module is divided into two parts: time extraction and spatial extraction. The spatial feature extraction module is mainly composed of 3 convolutional layers, and the temporal feature extraction module is composed of 8 ResDNet blocks;

[0030] The image reconstruction module consists of a pixel jittering layer and three convolutional layers.

[0031] Beneficial effects of the present invention:

[0032] The present invention realizes high-speed reconstruction of target neuron activities in two-photon microscopy imaging, overcomes the problems of data processing loss rate and positioning deviation in traditional image segmentation technology, and solves the problems of long reconstruction time caused by reduced sampling data volume. There is no need to modify the hardware part of the optical imaging equipment, which significantly reduces the economic cost and enhances its popularization potential in various fields. At the same time, it can realize high-speed reconstruction of target neuron activities and restore the dynamic change details information emitted by the target neurons, which is of great significance for application expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of the target perception high-speed imaging method based on deep learning provided by the present invention.

[0034] Figure 2 It is a flow chart of the target area neuron automatic segmentation and activity information extraction model provided by the present invention.

[0035] Figure 3 It is a schematic diagram of high-speed reconstruction of neuron activity information by the deep learning network provided by the present invention. DETAILED DESCRIPTION

[0036] The process of target perception high-speed imaging method based on deep learning is as follows Figure 1 As shown, the specific steps are as follows:

[0037] (a) Two-photon microscopy data acquisition: Perform two-photon microscopy imaging and obtain image data for living organisms transfected with fluorescent probes;

[0038] (b) Neuron segmentation and activity information extraction in the target area: Develop a target perception model based on deep learning, and process the microscopic imaging video data obtained in step 1) through a beam splitter to obtain imaging videos with lower image quality and image data with higher image quality, where the low image quality data is used as the imaging optical path and the higher image quality image data is used as the reference optical path.

[0039] The image data with lower image quality is used as the model input, and is processed by the filter enhancement module, the U-shaped network module and the parameter optimization module in sequence to obtain a binary matrix that retains the key information of neurons; the binary matrix is ​​multiplied by the collected image data with higher image quality to obtain the segmentation result, that is, a low frame rate and high image quality image that retains the key information of neurons;

[0040] (c) Building a deep learning network for training: Building a deep learning network that integrates compressed sensing, using the neuron segmentation image data (low frame rate and high image quality) obtained in step 2) and processed by the target perception model, and performing self-supervised training from low frame rate to high frame rate by downsampling;

[0041] (d) Using the trained network to perform high-speed imaging of the target neuron: The microscopic imaging video data to be processed is processed through step 2) to obtain segmented neuron image data (low frame rate and high image quality), and then processed through step 3) through the trained deep learning network for high-speed imaging restoration, thereby obtaining a neuron activity image with a high frame rate and high image quality.

[0042] Further, step (a) two-photon microscopy data acquisition is performed according to the following steps:

[0043] Step 1: Introduce fluorescent protein into a specific area of ​​the biological living sample to be observed through brain stereotaxic injection technology to achieve cell transfection.

[0044] Step 2: After the fluorescent protein is fully expressed, the sample is imaged using a microscope and the video data of the fluorescent image is collected through the device (low frame rate and low image quality).

[0045] Further, if Figure 2 As shown, step (b) is performed as follows:

[0046] Step 1: Develop a target perception model based on deep learning, use the collected low-quality microscopic imaging video as input data, introduce a mean filter to suppress image noise; then, design a filter for Poisson noise to reduce noise interference and improve signal clarity; further use time filtering and homomorphic filtering technology to eliminate signal fluctuations and anomalies. Finally, use signal-to-noise ratio (SNR) filtering, combined with non-negative matrix factorization (NMF) technology to process SNR traces, achieve effective separation of different independent cell signals, and output the filtered enhanced image.

[0047] Step 2: Based on the image obtained in step 1, input it into the self-developed U-shaped network based on deep learning, and optimize the network weights through the self-supervised training process to obtain the optimal parameter configuration. Further, using the optimal parameter configuration, the probability map corresponding to the image is output.

[0048] Step 3, using the probability map of step 2, set the probability threshold to filter out the area belonging to the neuron structure and eliminate interference information; introduce the minimum area parameter to further eliminate the interference of invalid data. The spatial distribution and morphological outline of neurons are restored through the average radius and average area parameters. The intersection-over-union ratio threshold is calculated and used to accurately evaluate whether the neurons are fused or separated. If it is judged to be fused, it is regarded as one neuron, otherwise it is regarded as two neurons. In addition, the continuous frame parameters are set to ensure the coherence and integrity of the neuron activity information, and finally a binary matrix that highlights the key information of the neurons is obtained. The binary matrix is ​​then multiplied with the microscopic imaging video with higher image quality to obtain the segmentation result, that is, image data with low frame rate and high image quality.

[0049] The segmentation algorithm of the invention step (b) adopts originality, and the pre-processing and post-processing stages in the segmentation are the uniqueness of the segmentation algorithm. In the pre-processing stage, a mean filter is first introduced to suppress noise in a targeted manner. Secondly, a filter based on Poisson noise is designed to effectively reduce noise interference and improve signal clarity. At the same time, the filtering technology of the spatiotemporal dimension (in line with calcium dynamics-fast rise and slow fall) is also adopted to eliminate signal fluctuations and anomalies. Subsequently, the signal-to-noise ratio (SNR) filtering is also integrated, and the SNR trace is processed by non-negative matrix decomposition (NMF) technology to effectively separate the independent signals of different cells. In the post-processing stage, after completing parameter setting and optimization, the neuron segmentation image data is generated by means of precise algorithm calculation by the index (probability threshold, minimum area parameter, average radius and average area parameter, intersection-combination ratio threshold, continuous frame parameter) set by oneself.

[0050] Further, if Figure 3 As shown, step (c) is performed as follows:

[0051] Step 1: Establish a deep learning network based on compressed sensing. The neural network consists of a feature extraction module and an image reconstruction module. The feature extraction module is divided into two parts: time extraction and space extraction. The spatial feature extraction is mainly composed of 3 convolutional layers, and the time feature extraction module is composed of 8 ResDNet blocks; the image reconstruction module is composed of a pixel jitter layer and 3 convolutional layers.

[0052] Step 2, during the training phase, use the segmented target neuron image data (low frame rate and high image quality) dataset, take the downsampled low frame rate and high image quality neuron activity image as the network input, and take the high frame rate and high image quality neuron activity image as the network target output, and perform self-supervised training.

[0053] Step 3: After training, when the loss function decreases and stabilizes within a certain numerical range, the network can restore the low frame rate and high image quality neuron activity image to a high frame rate and high image quality neuron activity image with high precision, and retain the weights at this time for subsequent inference.

[0054] In the high-speed imaging part of step (c), the present invention uses the concept of snapshot compression imaging for the first time in the field of two-photon imaging, ie, biological imaging.

[0055] Further, step (d) is carried out as follows:

[0056] Step 1: During the testing phase, load the network weights obtained in step (c).

[0057] Step 2: restore the neuron activity image (low frame rate and high image quality) in step (b) through a trained deep learning network to obtain a neuron activity image with high frame rate and high image quality.

Claims

1. A target perception high-speed imaging method based on deep learning, characterized in that: The following steps are involved: Step 1) Two-photon microscopic imaging data acquisition: for the living organism transfected with the fluorescent probe, two-photon microscopic imaging is performed and microscopic imaging video data is acquired; Step 2) Segmentation of neurons and extraction of activity information in the target area: Develop a target perception model based on deep learning, process the microscopic imaging video data obtained in step 1) through a beam splitter to obtain low-image-quality imaging video and high-image-quality image data; use the low-image-quality image data as model input, and process it in turn through a filter enhancement module, a U-shaped network module, and a parameter optimization module to obtain a binary matrix that retains key neuron information; multiply the binary matrix by the collected high-image-quality image data to obtain a segmentation result, that is, a low-frame-rate high-image-quality image that retains key neuron information; Step 3) Building a deep learning network integrating compressed sensing for training: Building a deep learning network integrating compressed sensing, using the neuron segmentation image data obtained in step 2) and processed by the target perception model, and performing self-supervised training from low frame rate to high frame rate by downsampling; Step 4) Using the trained network to perform high-speed imaging of the target neuron: The microscopic imaging video data to be processed is processed through step 2) to obtain segmented neuron image data, and then the trained deep learning network is used for high-speed imaging restoration in step 3), thereby obtaining a neuron activity image with high frame rate and high image quality.

2. The target perception high-speed imaging method based on deep learning according to claim 1, characterized in that: The step 1) is specifically as follows: Step 1.1) Into a specific region of a living biological sample to be observed, fluorescent protein is introduced by stereotaxic brain injection technology to achieve cell transfection. Step 1.2) After the fluorescent protein is fully expressed, the sample is imaged and observed using a microscope, and the device collects video data of the fluorescent image, i.e., an image with a low frame rate and low image quality.

3. The target perception high-speed imaging method based on deep learning according to claim 1, characterized in that: In the step 2), the low image quality data obtained after being processed by the beam splitter is used as the imaging light path, and the high image quality image data is used as the reference light path.

4. The target perception high-speed imaging method based on deep learning according to claim 1, characterized in that: The filtering enhancement module in step 2) is specifically: The microscopic imaging video data was filtered through a mean filter to suppress image noise. Subsequently, a filter was designed for Poisson noise to reduce noise interference and improve signal clarity. Temporal filtering and homomorphic filtering techniques were further used to eliminate signal fluctuations and anomalies. Finally, signal-to-noise ratio filtering was used in combination with non-negative matrix decomposition technology to process SNR traces, effectively separating different independent cell signals and outputting filtered and enhanced images.

5. The target perception high-speed imaging method based on deep learning according to claim 1, characterized in that: The U-shaped network module in step 2) is specifically: A U-shaped network based on deep learning is constructed. Through the self-supervised training process, the network weights are optimized to obtain the optimal parameter configuration. The optimal parameter configuration is further used to output the probability map corresponding to the image.

6. The target perception high-speed imaging method based on deep learning according to claim 1, characterized in that: The parameter optimization module in step 2) is specifically: Input the probability map obtained by the U-shaped network module, and retain the pixel area with a probability value greater than the probability threshold by setting the probability threshold, and compare it with the image after filtering and enhancement processing to ensure that the data processed by the U-shaped network has a high degree of overlap with the previous data, thereby eliminating invalid information; Then, the mean radius of the neuron meanR is set, and the average area parameter AvgArea is calculated to determine the estimated value of the neuron area to ensure the accuracy of the segmentation; The average area parameter is calculated according to the following formula: Among them, meanR is the average radius of neurons, and pixSize is the pixel size value of 0.

78. Then, the intersection-over-union ratio threshold is used to evaluate whether the neurons in the image are fused or separated. If they are judged to be fused, they are regarded as one neuron, and the minimum area parameter is introduced to remove neurons that do not meet the minimum area size requirement; If it is judged to be separated, it is regarded as two neurons; Finally, the continuous frame parameter is set to n / f, that is, in the continuous f frames, if a neuron appears more than n times, the neuron is retained, and finally a binary matrix highlighting the key information of the neuron is obtained.

7. The target perception high-speed imaging method based on deep learning according to claim 1, characterized in that: The step 3) is specifically as follows: Step 3.1) Establishing a high-speed reconstruction network integrating compressed sensing and deep learning, wherein the neural network consists of a feature extraction module and an image reconstruction module; Step 3.2) During the training phase, the target neuron image data after segmentation in step 2) is used as network input, and the network outputs a neuron activity image with a high frame rate and high image quality, completing self-supervised training.

8. The target perception high-speed imaging method based on deep learning according to claim 7, characterized in that: In the step 3.1): The feature extraction module is divided into two parts: time extraction and spatial extraction. The spatial feature extraction module is mainly composed of 3 convolutional layers, and the temporal feature extraction module is composed of 8 ResDNet blocks; The image reconstruction module consists of a pixel jittering layer and three convolutional layers.

Images