Super-high resolution fluorescence microscopy based on subvoxel convolutional neural network

Abstract

The present invention provides an ultra-high resolution fluorescence microscopy imaging method based on a subvoxel convolutional neural network. An ultra-high resolution three-dimensional optical microscopy imaging model is constructed and trained based on an end-to-end three-dimensional subvoxel convolutional neural network that combines a multi-branch structure and residual learning. The original optical low-resolution image is mapped to a positioning image of a three-dimensional ultra-high resolution fluorescent probe through the imaging model, thereby being applied to three-dimensional ultra-high resolution fluorescence microscopy imaging. Compared with existing microscopic imaging technologies, the ultra-high resolution fluorescence microscopy imaging method based on a subvoxel convolutional neural network significantly improves the axial resolution of ultra-high resolution fluorescence microscopy imaging and reduces the computational complexity of three-dimensional fluorescence microscopy super-resolution reconstruction. Moreover, the ultra-high resolution three-dimensional optical microscopy imaging model successfully obtained through network training does not require additional manual parameter adjustment or additional manual intervention, and is suitable for fast, flexible, three-dimensional ultra-high resolution fluorescence microscopy imaging.

Description

Technical Field

[0001] The present invention belongs to the technical field of microscopic imaging, and in particular relates to an ultra-high resolution fluorescence microscopic imaging method based on a subvoxel convolutional neural network. Background Art

[0002] Ultra-high resolution fluorescence localization microscopy is an indispensable tool in biomedical research, which has greatly promoted scholars' exploration of the microscopic world. Compared with traditional fluorescence microscopy, by combining the single-molecule localization method with a light-switchable fluorescent probe, the ultra-high resolution fluorescence localization microscopy method breaks through the optical diffraction limit and can image the intracellular organelles or molecular structures of living cells at the nanoscale.

[0003] However, current ultra-high resolution fluorescence microscopy technology is mainly focused on the two-dimensional scale. With the development of biomedicine, researchers hope to be able to observe the three-dimensional structure of cells and their spatial relationships at the nanoscale, that is, to perform three-dimensional ultra-high resolution imaging. However, due to phenomena such as light scattering in cells, the currently proposed single-molecule localization method cannot effectively resolve depth (Z-axis) information, resulting in the axial resolution of three-dimensional fluorescence microscopy being lower than the lateral resolution.

[0004] As an emerging technology, deep learning has been successfully applied to ultra-high resolution fluorescence microscopy research and has made important progress. However, existing ultra-high resolution fluorescence microscopy methods based on deep learning mainly achieve ultra-high resolution imaging through two-dimensional convolution operations. Although it can effectively improve the performance of ultra-high resolution fluorescence microscopy in two dimensions, its axial resolution is still much lower than the lateral resolution.

[0005] Compared with traditional two-dimensional convolutional neural networks, three-dimensional convolutional neural networks can simultaneously process the lateral and axial information of three-dimensional images and effectively learn image features at a three-dimensional scale. However, in the process of training deep three-dimensional convolutional neural networks, as the network deepens, it is easy to cause gradient vanishing or gradient exploding problems, making it difficult for network training to converge. Summary of the invention

[0006] In order to solve the above problems, an imaging method is provided to significantly improve the axial resolution of ultra-high resolution fluorescence microscopy imaging and reduce the computational complexity of three-dimensional fluorescence microscopy super-resolution reconstruction. The present invention adopts the following technical solutions:

[0007] The present invention provides an ultra-high resolution fluorescence microscopy imaging method based on a subvoxel convolutional neural network, which is characterized in that it comprises the following steps: step S1, constructing a deep learning model based on a three-dimensional subvoxel convolutional neural network and training it, and using the trained deep learning model as an ultra-high resolution three-dimensional optical microscopy imaging model; step S2, obtaining an original three-dimensional fluorescence microscopy image sequence to be imaged; step S3, inputting the original three-dimensional fluorescence microscopy image sequence into the ultra-high resolution three-dimensional optical microscopy imaging model to obtain an ultra-high resolution fluorescence probe positioning result corresponding to each frame; step S4, superimposing the ultra-high resolution fluorescence probe positioning results of all frames to obtain an ultra-high three-dimensional fluorescence microscopy imaging image, wherein the three-dimensional subvoxel convolutional neural network comprises a plurality of three-dimensional convolutional layers, a three-branch multi-scale convolutional module, and a residual module jump-connected to the three-dimensional convolutional layer.

[0008] The ultra-high resolution fluorescence microscopy method based on subvoxel convolutional neural network provided by the present invention may also have such technical features, wherein the ultra-high resolution three-dimensional optical microscopy model is obtained by the following steps: step S1-1, constructing a deep learning model based on the three-dimensional subvoxel convolutional neural network; step S1-2, obtaining simulation training data; step S1-3, constructing a loss function; step S1-4, using the simulation training data as input, using the loss function to optimize the deep learning model using the Adam optimization algorithm to traverse training, and obtain the ultra-high resolution three-dimensional optical microscopy model.

[0009] The ultra-high resolution fluorescence microscopy method based on subvoxel convolutional neural network provided by the present invention may also have such a technical feature, wherein the simulation training data is obtained based on the following steps: step S1-2-1, generating a number of fluorescent molecules randomly distributed in the simulation structure template in the three-dimensional grid area of ​​predetermined pixels, and performing simulation imaging of the current distribution based on the optical microscopy imaging system; step S1-2-2, randomly changing the positions of the activated fluorescent molecules in the simulation structure template, and performing simulation imaging again using the optical microscopy imaging system; step S1-2-3, repeating step S1-2-2 a predetermined number of times and obtaining a predetermined frame of simulated optical images; step S1-2-4, using the forward model to simulate the optical imaging process by convolving the impulse function at the real position of the fluorescent molecule with the three-dimensional PSF to obtain a noise-free simulated optical image; step S1-2-5, adding background noise to the noise-free simulated optical image one by one to obtain a noisy simulated optical image, and using the noisy simulated optical image as the simulation training data.

[0010] The ultra-high resolution fluorescence microscopy imaging method based on subvoxel convolutional neural network provided by the present invention may also have such a technical feature, wherein the loss function is a combination of MSE and L1 regularization:

[0011]

[0012] In the formula, y is the training label, the image of the actual distribution of fluorescent molecules, is the predicted output image of the network, and N is the number of mini-batches of images that the network traverses each time it is optimized.

[0013] The ultra-high resolution fluorescence microscopy imaging method based on subvoxel convolutional neural network provided by the present invention may also have such technical features, wherein, in the optimization traversal training, the training cycle is 600, the number N of small batch images of each optimization traversal of the network is 8, and the initial learning rate is 0.0001.

[0014] The ultra-high resolution fluorescence microscopy imaging method based on subvoxel convolutional neural network provided by the present invention can also have the following technical features: there are 10 three-dimensional convolutional layers, the first convolutional layer is used to extract shallow features of the image, the second to fourth convolutional layers introduce jump connections to form a residual module for realizing local residual learning, the fifth to seventh convolutional layers are used to learn deep features of the same scale, and a three-branch multi-scale convolutional module is located after the fifth to seventh convolutional layers. The three-branch multi-scale convolutional module uses branches with different receptive fields to learn multi-scale features, and adds jump connections for residual learning. The sixth convolutional layer is connected to the ninth convolutional layer to realize feature fusion of different levels and scales, and the tenth convolutional layer is used to realize upsampling operation.

[0015] Function and Effect of the Invention

[0016] According to the ultra-high resolution fluorescence microscopy imaging method based on subvoxel convolutional neural network of the present invention, a deep learning model is constructed based on an end-to-end three-dimensional subvoxel convolutional neural network combining a multi-branch structure and residual learning, and the deep learning model is trained to obtain an ultra-high resolution three-dimensional optical microscopy imaging model, through which the original optical low-resolution image is mapped to the positioning image of the three-dimensional ultra-high resolution fluorescent probe, so as to be applied to three-dimensional ultra-high resolution fluorescence microscopy imaging. Since the three-dimensional subvoxel convolutional neural network realizes the feature fusion between the shallow network and the deep network through jump connection, the error of network training is greatly reduced and converges to a small value, thereby improving the accuracy of fluorescence positioning.

[0017] Compared with the existing microscopic imaging technology, the ultra-high resolution fluorescence microscopic imaging method based on subvoxel convolutional neural network of the present invention significantly improves the axial resolution of ultra-high resolution fluorescence microscopic imaging and reduces the computational complexity of three-dimensional fluorescence microscopic super-resolution reconstruction. Moreover, the ultra-high resolution three-dimensional optical microscopic imaging model successfully obtained through network training does not require additional manual parameter adjustment or additional manual intervention, and is suitable for fast, flexible, three-dimensional ultra-high resolution fluorescence microscopic imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of an ultra-high resolution fluorescence microscopy imaging method based on a subvoxel convolutional neural network in an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of the structure of a three-dimensional subvoxel convolutional neural network in an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of a phantom model used in a simulation experiment in an embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of an original imaging image of a phantom model in an embodiment of the present invention;

[0022] Figure 5 It is a schematic diagram of a three-dimensional ultra-high resolution fluorescence microscopy imaging result reconstructed by an ultra-high resolution fluorescence microscopy imaging method based on a subvoxel convolutional neural network in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the ultra-high resolution fluorescence microscopy imaging method of the subvoxel convolutional neural network of the present invention is specifically described below in combination with the embodiments and drawings.

[0024] <Example>

[0025] Figure 1 It is a flow chart of an ultra-high resolution fluorescence microscopy imaging method based on a subvoxel convolutional neural network in an embodiment of the present invention.

[0026] like Figure 1 As shown, the ultra-high resolution fluorescence microscopy imaging method based on subvoxel convolutional neural network includes the following steps:

[0027] Step S1, constructing and training a deep learning model based on a three-dimensional subvoxel convolutional neural network, and using the trained deep learning model as an ultra-high resolution three-dimensional optical microscopy imaging model.

[0028] In this embodiment, an end-to-end subvoxel convolutional neural network combining a multi-branch structure and residual learning is used for three-dimensional ultra-high resolution fluorescence microscopy imaging. The initial input of the neural network is the original low-resolution optical image, and the corresponding final output is an ultra-high resolution three-dimensional fluorescence localization image.

[0029] In step S1, the ultra-high resolution three-dimensional optical microscopy imaging model acquisition process is as follows:

[0030] Step S1-1, constructing a deep learning model based on a three-dimensional subvoxel convolutional neural network.

[0031] Figure 2 Schematic diagram of the structure of a three-dimensional subvoxel convolutional neural network in an embodiment of the present invention.

[0032] In this embodiment, Figure 2 As shown in the figure, the 3D subvoxel convolutional neural network contains 10 3D convolutional layers and a three-branch multi-scale convolutional module. Among them, the first convolutional layer is used to extract shallow features of the image. Then, the second to fourth convolutional layers introduce jump connections to form a residual module for local residual learning. Three convolutional layers are stacked before the three-branch multi-scale convolutional module, namely the fifth to seventh convolutional layers, for learning deep features of the same scale. The three-branch multi-scale convolutional module uses branches with different receptive fields to learn multi-scale features, and adds jump connections for residual learning. The sixth convolutional layer is connected to the ninth convolutional layer to realize feature fusion of different levels and scales. The tenth convolutional layer (subvoxel convolutional layer) at the end of the network is used to implement upsampling operations.

[0033] Figure 3 is a schematic diagram of a phantom model used in a simulation experiment in an embodiment of the present invention, Figure 4 Schematic diagram of the original imaging image of the phantom model in the embodiment of the present invention.

[0034] Step S1-2, based on Figure 3 The phantom model shown obtains simulation training data. Specifically:

[0035] Step S1-2-1, generating a number of fluorescent molecules randomly distributed in the simulation structure template in a 32x32x32 three-dimensional grid area (excluding the small border at the corner of the image), and performing simulation imaging of the current distribution based on the optical microscopy imaging system;

[0036] Step S1-2-2, randomly changing the positions of the activated fluorescent molecules in the simulation structure template, and performing simulation imaging again using an optical microscopic imaging system;

[0037] Step S1-2-3, in order to simulate the process of continuous flow of fluorescent molecules in biological tissues within the imaging area, the above step S1-2-2 is repeated 1500 times to obtain 1500 frames of original simulated optical images, such as Figure 4 As shown;

[0038] Step S1-2-4, using the forward model to simulate the optical imaging process by convolving the impulse function at the real position of the fluorescent molecule with the three-dimensional PSF to obtain a noise-free simulated optical image;

[0039] Step S1-2-5, adding background noise to the noise-free simulated optical images one by one to obtain noisy simulated optical images, and using the noisy simulated optical images as simulation training data.

[0040] In this embodiment, the original optical low-resolution image is directly mapped to the positioning image of the three-dimensional ultra-high resolution fluorescent probe through a neural network, so the loss function has a great influence on the training of the network. The use of the mean square error (MSE) loss function commonly used in image processing to train the deep learning model will make the reconstructed image too smooth and unable to retain the image detail information well, which is not suitable for the three-dimensional fluorescent probe positioning task. Therefore, this embodiment uses the MSE combined with the L1 regularized loss function to train the network.

[0041] Step S1-3, construct the loss function of MSE combined with L1 regularization, which is expressed as follows:

[0042]

[0043] Where y is the training label, and the image of the actual distribution of fluorescent molecules. is the predicted output image of the network, and N is the number of mini-batches of images that the network traverses each time it is optimized.

[0044] Step S1-4, using the simulation training data as input, using the loss function to optimize the deep learning model using the Adam optimization algorithm to traverse the training, and obtaining an ultra-high resolution three-dimensional optical microscopy imaging model.

[0045] The specific training process is to use the Adam optimization algorithm to train a total of 600 cycles (i.e., traverse the training set 600 times) on a training set containing 1500 pairs of original 3D simulated optical images and corresponding labels, where the number of small batch images N traversed by the network each time is 8, and the initial learning rate is 0.0001. In order to make the error converge to a smaller value, a learning rate decay strategy is used, specifically, the learning rate decays by half every 100 rounds (reducing the learning rate when the training error is stable).

[0046] Step S2, obtaining the original three-dimensional fluorescence microscopy image sequence to be imaged.

[0047] Step S3, inputting the original three-dimensional fluorescence microscopy image sequence into the ultra-high resolution three-dimensional optical microscopy imaging model to obtain the ultra-high resolution fluorescence probe positioning result corresponding to each frame.

[0048] Step S4, superimposing the ultra-high resolution fluorescent probe positioning results of all frames to obtain an ultra-high resolution three-dimensional fluorescence microscopy image.

[0049] Figure 5It is a schematic diagram of a three-dimensional ultra-high resolution fluorescence microscopy imaging result reconstructed by an ultra-high resolution fluorescence microscopy imaging method based on a subvoxel convolutional neural network in an embodiment of the present invention.

[0050] In this embodiment, the low-resolution original optical image of the phantom model obtained in the experiment is processed based on the ultra-high resolution three-dimensional optical microscopy imaging model obtained by training, and finally a three-dimensional ultra-high resolution optical microscopy imaging image is obtained, such as Figure 5 shown.

[0051] Example Function and Effect

[0052] According to the ultra-high resolution fluorescence microscopy method based on subvoxel convolutional neural network provided in this embodiment, a deep learning model is constructed based on an end-to-end three-dimensional subvoxel convolutional neural network that combines a multi-branch structure and residual learning. The deep learning model is trained to obtain an ultra-high resolution three-dimensional optical microscopy imaging model. Through the imaging model, the original optical low-resolution image is mapped to the positioning image of the three-dimensional ultra-high resolution fluorescent probe, and then applied to three-dimensional ultra-high resolution fluorescence microscopy imaging.

[0053] Compared with the existing microscopic imaging technology, the ultra-high resolution fluorescence microscopic imaging method based on subvoxel convolutional neural network in this embodiment significantly improves the axial resolution of ultra-high resolution fluorescence microscopic imaging and reduces the computational complexity of three-dimensional fluorescence microscopic super-resolution reconstruction. Moreover, the ultra-high resolution three-dimensional optical microscopic imaging model successfully obtained through network training does not require additional manual parameter adjustment or additional manual intervention, and is suitable for fast, flexible, three-dimensional ultra-high resolution fluorescence microscopic imaging.

[0054] In the embodiment, since the three-dimensional subvoxel convolutional neural network realizes feature fusion between the shallow network and the deep network through skip connections, the error of network training is greatly reduced and converges to a smaller value, thereby improving the accuracy of fluorescence localization.

[0055] In the embodiment, the MSE combined with the L1 regularization loss function is used to train the deep learning model, which solves the problem that the reconstructed image is too smooth when the deep learning model is trained by the traditional mean square error (MSE) loss function, resulting in the inability to retain the image detail information well. The model can be better suited for three-dimensional fluorescent probe positioning tasks.

[0056] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A method for ultra-high resolution fluorescence microscopy based on subvoxel convolutional neural network, characterized in that: The following steps are involved: Step S1, constructing a deep learning model based on a three-dimensional subvoxel convolutional neural network and training it, and using the trained deep learning model as an ultra-high resolution three-dimensional optical microscopy imaging model; Step S2, obtaining an original three-dimensional fluorescence microscopic image sequence to be imaged; Step S3, inputting the original three-dimensional fluorescence microscopy image sequence into the ultra-high resolution three-dimensional optical microscopy imaging model to obtain the ultra-high resolution fluorescence probe positioning result corresponding to each frame; Step S4, superimposing the ultra-high resolution fluorescent probe positioning results of all frames to obtain an ultra-high three-dimensional fluorescent microscopic imaging image, The three-dimensional subvoxel convolutional neural network includes a plurality of three-dimensional convolutional layers, a three-branch multi-scale convolutional module, and a residual module jump-connected to the three-dimensional convolutional layer.

2. The method for ultra-high resolution fluorescence microscopy based on subvoxel convolutional neural network according to claim 1, characterized in that: in, The ultra-high resolution three-dimensional optical microscopy imaging model is obtained by the following steps: Step S1-1, constructing a deep learning model based on a three-dimensional subvoxel convolutional neural network; Step S1-2, obtaining simulation training data; Step S1-3, constructing a loss function; Step S1-4, taking the simulation training data as input, using the loss function to optimize the deep learning model using the Adam optimization algorithm to obtain the ultra-high resolution three-dimensional optical microscopy imaging model.

3. The method for ultra-high resolution fluorescence microscopy based on subvoxel convolutional neural network according to claim 2, Features: Wherein, the simulation training data is obtained based on the following steps: Step S1-2-1, generating a plurality of fluorescent molecules randomly distributed in a simulation structure template in a three-dimensional grid area of ​​a predetermined pixel, and performing simulation imaging of the current distribution based on an optical microscopic imaging system; Step S1-2-2, randomly changing the positions of the activated fluorescent molecules in the simulation structure template, and performing simulation imaging again using the optical microscopic imaging system; Step S1-2-3, repeating the step S1-2-2 for a predetermined number of times and acquiring a predetermined frame of simulated optical images; Step S1-2-4, using a forward model to simulate the optical imaging process by convolving the impulse function at the real position of the fluorescent molecule with the three-dimensional PSF to obtain a noise-free simulated optical image; Step S1-2-5, adding background noise to the noise-free simulated optical image one by one to obtain a noisy simulated optical image, and using the noisy simulated optical image as the simulation training data.

4. The method for ultra-high resolution fluorescence microscopy based on subvoxel convolutional neural network according to claim 2, characterized in that: in, The loss function is a combination of MSE and L1 regularization: Where y is the training label, and the image of the actual distribution of fluorescent molecules. is the predicted output image of the network, and N is the number of mini-batches of images that the network traverses each time it is optimized.

5. The method for ultra-high resolution fluorescence microscopy based on subvoxel convolutional neural network according to claim 2, characterized in that: in, In the optimization traversal training, the training cycle is 600, the number of small batch images N of each optimization traversal of the network is 8, and the initial learning rate is 0.0001.

6. The method for ultra-high resolution fluorescence microscopy based on subvoxel convolutional neural network according to claim 1, characterized in that: in, The number of the three-dimensional convolutional layers is 10. The first convolutional layer is used to extract shallow features of the image. The second to fourth convolutional layers introduce skip connections to form a residual module for implementing local residual learning. The 5th to 7th convolutional layers are used to learn deep features of the same scale. The three-branch multi-scale convolution module is after the 5th to 7th convolution layers. The three-branch multi-scale convolution module uses branches with different receptive fields to learn multi-scale features and adds skip connections for residual learning. The sixth convolutional layer is connected to the ninth convolutional layer to achieve feature fusion of different levels and scales. The 10th convolutional layer is used to implement the upsampling operation.

Images