A fluorescence microscopic resolution enhancement method based on physical prior deep learning

Abstract

The application discloses a fluorescent microscopic resolution enhancement method based on physical prior deep learning, which comprises the following steps: firstly, acquiring a low-resolution image and preparing a training data set; secondly, building a Res-U-DBPN network comprising an upper projection module and a lower projection module, wherein the network is a U-shaped network, and the upper projection module and the lower projection module at the same depth are connected through a skip layer for feature fusion; then, constructing a target function and training the network, wherein the target function comprises a physical prior constraint, a sparse constraint and a continuity constraint; finally, matching a sample fluorescent image with a point spread function, removing the background and inputting the trained network to obtain a resolution-enhanced image result. The method has no special requirements for hardware and samples, has high structure generalization and fidelity, does not need to be adjusted for a specific structure or to collect a corresponding high-resolution data set for training, and can enhance the SIM resolution by 1.5 times only once.

Description

Technical Field

[0001] This invention relates to the field of optical super-resolution microscopy, and more particularly to a method for improving the resolution of fluorescence microscopy based on physical prior deep learning. Background Technology

[0002] Fluorescence microscopy is an important tool for observing the microstructure and dynamic processes of cells, and its resolution and imaging speed determine the ability to observe cellular dynamic processes. However, current imaging techniques struggle to achieve both high speed and high resolution. For example, single-molecule localization microscopy (SMLM) and stimulated emission depletion microscopy (STED) require sacrificing imaging time to meet high resolution requirements, while structured illumination microscopy (SIM), although characterized by short imaging time, is limited in imaging resolution.

[0003] Against this backdrop, computational imaging is an effective means to address the trade-off between fast imaging time and high imaging resolution. To date, researchers have proposed various computational imaging algorithms. From the perspective of improving spatial resolution, such as deep learning-based cross-modal techniques and traditional super-resolution algorithms like sparse deconvolution, computational super-resolution techniques can improve the spatial resolution of low-resolution images, enabling them to achieve both high resolution and fast imaging performance. Regarding improving imaging speed, deep learning algorithms can also significantly shorten the imaging time required for high-resolution, low-speed imaging techniques such as STED and SMLM. However, the successful reconstruction of traditional computational super-resolution algorithms relies on prior structural information in the image to solve ill-conditioned inverse problems such as super-resolution reconstruction. The aforementioned deep learning methods are all supervised training methods, requiring the collection of a large amount of high-quality ground-value data with the same structure for training. Furthermore, the trained network can only be applied to data with fixed imaging modes or different structures. Therefore, current computational super-resolution and deep learning algorithms suffer from deficiencies in structure fidelity and generalization. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a physical prior deep learning-based fluorescence microscopy super-resolution imaging method that requires only one training iteration to improve the spatial resolution of imaging results for different structures under different imaging modes.

[0005] The present invention discloses a method for improving the resolution of fluorescence microscopy based on physical prior deep learning, which includes the following steps:

[0006] S1. Low-resolution image acquisition: Reconstruct the multiple sets of original images to obtain the original training data;

[0007] S2. Training Dataset Creation: Preprocess the original training data to obtain a batch of training datasets, and use this dataset as the input to the network;

[0008] S3. Construct the Res-U-DBPN network: This network is a U-shaped network, including an upper projection module and a lower projection module. The upper projection module and the lower projection module at the same depth perform feature fusion through skipping layers.

[0009] S4. Construct the objective function: The objective function consists of three parts, namely physical prior constraints, sparsity constraints, and continuity constraints.

[0010] S5. Train the Res-U-DBPN network: Input the training dataset into the built network, use the optimizer to iteratively optimize, reduce the value of the objective function until the network converges, stop training and obtain the optimal network parameters;

[0011] S6. Fluorescence Image Resolution Enhancement: The sample is imaged to obtain the fluorescence image. The point spread function of the fluorescence image is matched, and the background of the matched fluorescence image is removed. Then, the background-removed fluorescence image is input into the Res-U-DBPN network, and the optimal network parameters are loaded into the Res-U-DBPN network. The output of the network is the image result after resolution enhancement.

[0012] Furthermore, the acquisition method of multiple sets of original images in S1 is as follows: using the SIM hardware system to capture multiple images of the same area, different directions, and different phases of different types of samples under sinusoidal fringe illumination as a set of original images, repeating the above steps to capture different samples and different areas to obtain multiple sets of original images, reconstructing the original data to obtain the SIM training dataset.

[0013] Furthermore, after segmenting the original training data, S2 also needs to perform segmentation, rotation, and data augmentation, and normalize the original image, set an information density threshold, and remove images with information density lower than the information density threshold.

[0014] Furthermore, the Res-U-DBPN network structure is as follows: This network integrates U-NET, residual and DBPN networks, and has a U-shaped network structure. At the same depth, the network contains up-projection modules and down-projection modules of the same feature map size, which are connected by skip layers for feature fusion. Down-projection modules at adjacent depths are connected through downsampling layers, and up-projection modules are connected through upsampling layers.

[0015] Further, the downprojection module specifically works as follows: The downprojection module first obtains the original feature map h0 through a convolutional layer with a stride of 1, then downsamples it using a convolutional layer with a stride of 2 to obtain a low-resolution feature map l0, and then upsamples it through a transposed convolutional layer to obtain a high-resolution feature map h1. The obtained feature map h1 is subtracted from the original feature map h0 to obtain the difference, and then the difference is input into a convolutional layer for downsampling with a stride of 2. The downsampling results l1 and l0 are summed to obtain the final feature map, which is then input into the lower-layer downprojection module.

[0016] Further, the upper projection module specifically works as follows: The upper projection module first obtains the original feature map l0 through a convolutional layer with a stride of 1, then upsamples it through a transposed convolutional layer with a stride of 2 to obtain a high-resolution feature map h0, and then downsamples it through a convolutional layer with a stride of 2 to obtain a low-resolution feature map l1. The obtained feature map l1 is subtracted from the original feature map l0 to obtain the difference, and then the difference is input into the transposed convolutional layer for upsampling with a stride of 2 to obtain a high-resolution feature map h1. The sum of h1 and h0 is combined with the result of the downsampling module in the same layer for feature fusion, and input into the residual block to obtain the final feature map, which is then input into the lower layer upper projection module.

[0017] Furthermore, the objective function is specifically:

[0018]

[0019] Physical prior constraints were used to construct the imaging model of the optical system. The network output image f was convolved with the equivalent srPSF of the dataset to calculate the structural similarity with the network input; continuity constraints R Hessian (x) is a Hessian constraint, denoted as The sparsity constraint is the L1 norm of the network output image f, and α and β are network hyperparameters used to balance sparsity and continuity constraints.

[0020] Furthermore, the midpoint diffusion function matching for improving fluorescence image resolution specifically involves:

[0021] Point spread function matching is performed on the fluorescence image. The pixel size of the image is changed by upsampling / downsampling. The pixel size of the image is then matched using the point spread function. input It should satisfy the Nyquist sampling theorem, i.e., less than λ. input / 4NA input , λ input NA is the excitation wavelength of the point spread function of the original image. input The equivalent numerical aperture of the point spread function of the original image;

[0022] The upsampling and downsampling dimensions should satisfy the following formula:

[0023]

[0024] Where λ srPSF NA srPSF With pixelsize srPSF The excitation wavelength, effective numerical aperture, and pixel size of the point spread function used in the physical prior constraints of S3.2 should be specified, and the sampled image should also satisfy the Nyquist sampling theorem, i.e., less than λ. input / 4NA input .

[0025] Compared with the prior art, the present invention has the following beneficial technical effects:

[0026] (1) Compared with traditional super-resolution algorithms and supervised deep learning algorithms, this method has higher structural generalization and fidelity, and does not require parameter tuning for specific structures or training with corresponding high-resolution datasets.

[0027] (2) The training set needs to be collected using the SIM system. Only one training is needed to improve the resolution by 1.5 times on the basis of SIM, and there are no special requirements for hardware or samples.

[0028] (3) This method does not require iterative optimization. After training, the network can directly process fluorescence images. It is fast and the processing speed is 90% faster than traditional iterative optimization algorithms. Attached Figure Description

[0029] Figure 1 This is a flowchart of network training and testing provided in an embodiment of the present invention;

[0030] Figure 2 This is a diagram of the Res-U-DBPN network structure provided in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the upsampling and downsampling modules provided in an embodiment of the present invention;

[0032] Figure 4 The diagram illustrates the implementation effect of an embodiment of the present invention. Detailed Implementation

[0033] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, the present invention provides a method for improving the resolution of fluorescence microscopy based on physical prior deep learning, comprising the following steps:

[0035] S1. Acquisition of low-resolution images of the training set: Images of different samples under different azimuth angles and phases are captured using the SIM hardware system and used as the original images for traditional SIM reconstruction to obtain the reconstruction results;

[0036] S2. Training dataset creation: Segment and rotate multiple SIM reconstructed images, and perform data augmentation operations;

[0037] S3. Network setup and objective function construction: (e.g.) Figure 1 As shown, an objective function is constructed to quantitatively describe the difference between the network output and the true value. The training dataset obtained in S2 is input into the network, and the Adam optimizer is used for iterative optimization to narrow the value of the objective function. After 40,000 iterations, the network converges, and the network weight parameters are saved as a prerequisite for improving the resolution of fluorescence images.

[0038] S4. Fluorescence Image Resolution Enhancement: The sample is imaged using a fluorescence microscopy system to obtain the fluorescence image. The point spread function of the fluorescence image is matched, and the background of the matched fluorescence image is removed. The background-removed fluorescence image is then input into the training network, and the optimal network parameters obtained in S3 are loaded into the network. The output of the network is the image result after resolution enhancement.

[0039] Preferably, step S3 includes the following sub-steps:

[0040] S3.1, Construct a Res-U-DBPN network, such as Figure 2 As shown, the network constructed using this method has a depth of 4 layers, with an initial feature channel count of 64. The number of feature channels at different depths are 64, 128, 256, and 512, respectively. The network mainly consists of two core modules: an upper projection module and a lower projection module. Its structure is as follows: Figure 3 As shown, the downprojection module first obtains the original feature map h0 through a convolutional layer with a stride of 1, then downsamples it through a convolutional layer with a stride of 2 to obtain the first low-resolution feature map l0, and then upsamples it through a transposed convolutional layer to obtain the high-resolution feature map h1. The obtained feature map h1 is subtracted from the original feature map h0 to obtain the difference, and then the difference is input into a convolutional layer for downsampling with a stride of 2. The downsampling result is the second low-resolution feature map l1. l0 and l1 are summed, downsampled through a max pooling layer, and then input into the next downprojection module.

[0041] The upprojection module first obtains the original feature map l0 through a convolutional layer with a stride of 1, then upsamples it through a transposed convolutional layer with a stride of 2 to obtain a high-resolution feature map h0, and then downsamples it through a convolutional layer with a stride of 2 to obtain a low-resolution feature map l1. The obtained feature map l1 is subtracted from the original feature map l0 to obtain the difference, and then the difference is input into a transposed convolutional layer for upsampling with a stride of 2 to obtain a high-resolution feature map h1. The sum of h1 and h0 is combined with the result of the downsampling module in the same layer for feature fusion, and then input into the residual block. The output result is upsampled through a transposed convolution and input into the next upprojection module.

[0042] S3.2 The loss function consists of three parts: physical prior constraints, sparsity constraints, and continuity constraints.

[0043]

[0044] Physical prior constraints were used to construct an imaging model for the optical system, which was used to convolve the network output image f with the dataset's equivalent srPSF to calculate the structural similarity with the network input y. In this embodiment, the srPSF simulation adopted an approximate numerical aperture of 3, an excitation wavelength of 488 nm, and a pixel size of 32.5 nm, satisfying a first-order Bessel function form; the continuity constraint R... Hessian (f) represents the Hessian constraint, denoted as The sparsity constraint is the L1 norm of the network output image f, and α and β are network hyperparameters used to balance the sparsity and continuity constraints to obtain the best results. The hyperparameter values ​​used in this invention are 1e-6 and 1e-4, respectively. These values ​​should be adjusted according to the actual training situation to obtain the best results.

[0045] Preferably, step S4 includes the following sub-steps:

[0046] like Figure 4 As shown, the fluorescence image is first subjected to point spread function matching. The pixel size of the image is changed by upsampling / downsampling, and the pixel size of the image is then matched using the point spread function. input It should satisfy the Nyquist sampling theorem, i.e., less than λ. input / 4NA input , λ input NA is the excitation wavelength of the point spread function of the original image. input This is the equivalent numerical aperture of the point spread function of the original image. In this embodiment, the image pixel size is 30nm, the equivalent point spread function numerical aperture is 1.49, and the excitation wavelength is 561nm. To achieve matching with srPSF, the downsampling scale should satisfy the following formula:

[0047]

[0048] Where λ srPSF NA srPSF With pixelsize srPSF Given the excitation wavelength, effective numerical aperture, and pixel size of the point spread function used in the physical prior constraints of S3.2, the downsampling scale should be 0.52, and the sampled image should also satisfy the Nyquist sampling theorem, i.e., less than λ. input / 4NA input .

[0049] S4.2. The point spread function matching image is obtained through the method in S4.1. The background fluorescence information in the sample is further removed by the ball rolling algorithm. In this embodiment, the ball rolling radius of the ball rolling algorithm is set to 50 pixels to remove the background. The sample with the background removed is input into the trained network for resolution improvement. The network output is the high-resolution result.

[0050] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A physical-prior deep learning based fluorescence microscopy resolution enhancement method, characterized in that, The method includes the following steps: S1. Low-resolution image acquisition: Reconstruct the multiple sets of original images to obtain the original training data; S2. Training Dataset Creation: Preprocess the original training data to obtain a batch of training datasets, and use this dataset as the input to the network; S3. Construct the Res-U-DBPN network: This network is a U-shaped network, including an upper projection module and a lower projection module. The upper projection module and the lower projection module at the same depth perform feature fusion through skipping layers. S4. Construct the objective function: The objective function consists of three parts, namely physical prior constraints, sparsity constraints, and continuity constraints. S5. Train the Res-U-DBPN network: Input the training dataset into the built network, use the optimizer to iteratively optimize, reduce the value of the objective function until the network converges, stop training and obtain the optimal network parameters; S6. Fluorescence Image Resolution Enhancement: The sample is imaged to obtain the fluorescence image. The point spread function of the fluorescence image is matched, and the background of the matched fluorescence image is removed. Then, the background-removed fluorescence image is input into the Res-U-DBPN network, and the optimal network parameters are loaded into the Res-U-DBPN network. The output of the network is the image result after resolution enhancement.

2. The method of claim 1, wherein the method is based on physical priors and deep learning. The method for acquiring multiple sets of original images in S1 is as follows: using the SIM hardware system to capture multiple images of the same area, different directions, and different phases of different types of samples under sinusoidal fringe illumination as a set of original images. Repeat the above steps to capture different samples and different areas to obtain multiple sets of original images. Reconstruct the original data to obtain the SIM training dataset.

3. The method of claim 1, wherein the method is based on physical priors and deep learning. After segmenting the original training data, S2 also needs to perform segmentation, rotation, and data augmentation, and normalize the original image, set an information density threshold, and remove images with information density lower than the information density threshold.

4. The method for improving the resolution of fluorescence microscopy based on physical prior deep learning according to claim 1, characterized in that, The Res-U-DBPN network is specifically designed as follows: This network integrates U-NET, residual, and DBPN networks, and has a U-shaped structure. At the same depth, the network contains up-projection and down-projection modules with the same feature map size. Feature fusion is achieved through skip-layer connections. Down-projection modules at adjacent depths are connected through downsampling layers, and up-projection modules are connected through upsampling layers.

5. The fluorescence microscopy resolution enhancement method based on physical prior deep learning according to claim 4, characterized in that, The downprojection module is specifically as follows: First, the downprojection module obtains the original feature map h0 through a convolutional layer with a stride of 1. Then, it performs downsampling with a convolutional layer with a stride of 2 to obtain a low-resolution feature map l0. After that, it performs upsampling through a transposed convolutional layer to obtain a high-resolution feature map h1. The obtained feature map h1 is subtracted from the original feature map h0 to obtain the difference. Then, the difference is input into a convolutional layer for downsampling with a stride of 2. The downsampling results l1 and l0 are summed to obtain the final feature map, which is then input into the lower-layer downprojection module.

6. The method for improving the resolution of fluorescence microscopy based on physical prior deep learning according to claim 4, characterized in that, The upprojection module is specifically as follows: First, the upprojection module obtains the original feature map l0 through a convolutional layer with a stride of 1. Then, it performs upsampling with a transposed convolutional layer with a stride of 2 to obtain a high-resolution feature map h0. After that, it performs downsampling with a convolutional layer with a stride of 2 to obtain a low-resolution feature map l1. The obtained feature map l1 is subtracted from the original feature map l0 to obtain the difference. Then, the difference is input into the transposed convolutional layer for upsampling with a stride of 2 to obtain a high-resolution feature map h1. The sum of h1 and h0 is combined with the result of the downsampling module in the same layer for feature fusion. The result is input into the residual block to obtain the final feature map, which is then input into the lower-layer upprojection module.

7. The method for improving the resolution of fluorescence microscopy based on physical prior deep learning according to claim 1, characterized in that, The objective function is specifically: Physical prior constraints were used to construct the imaging model of the optical system. The network output image f was convolved with the equivalent srPSF of the dataset to calculate the structural similarity with the network input; continuity constraints R Hessian (f) represents the Hessian constraint, denoted as The sparsity constraint is the L1 norm of the network output image f, and α and β are network hyperparameters used to balance sparsity and continuity constraints.

8. The method for improving fluorescence microscopy resolution based on physical prior deep learning according to claim 1, characterized in that, The fluorescence image resolution enhancement midpoint spread function matching specifically involves: The fluorescent image is point spread function matched, pixel size of the point spread function matched image is changed through image up / down sampling input The Nyquist sampling theorem should be met, i.e. less than λ input 4NA input , λ input is an excitation wavelength of a point spread function of the original image, NA input is an equivalent numerical aperture of the point spread function of the original image; The upsampling and downsampling dimensions should satisfy the following formula: where λ srPSF , NA srPSF and pixelsize srPSF are the excitation wavelength, effective numerical aperture and pixel size of the point spread function used in the physical prior constraint in S3.2, and the sampled picture should also satisfy the Nyquist sampling theorem, i.e. less than λ input 4NA input .

Images