A dual-supervised training method for structured illumination microstructure reconstruction using deep learning

Abstract

The present invention discloses a method for deep learning reconstruction of structured light microscopy using dual-supervisory training. The method collects original structured light microscopy super-resolution images and original widefield true-value images under conditions of high laser intensity and high exposure time, and collects original modulated widefield images under conditions of low laser intensity and short exposure in the same field of view, performing image enhancement, and uses these as training data sets. A point spread function of the system is generated based on imaging parameters, and a neural network for structured light microscopy reconstruction is constructed. Gradient descent training is performed on the neural network for structured light microscopy reconstruction using dual supervision, based on structured light microscopy super-resolution images and widefield true-value images. The method can further improve the performance of deep learning reconstruction of structured light super-resolution microscopy images. The method is applicable to image reconstruction of mainstream structured light microscopy microscopes, such as stripe-modulated structured light and lattice-modulated structured light microscopes.

Description

Technical Field

[0001] The present invention relates to structured light illumination microscopy technology, and in particular to a dual-supervisory training structured light illumination microscopy deep learning reconstruction method. Background Art

[0002] Structured light microscopy is a super-resolution microscopy technique commonly used in live cell fluorescence imaging and has been widely used in cell biology research. It achieves super-resolution imaging by modulating the sample with structured light illumination and subsequently reconstructing it. The reconstruction algorithm is an important part of structured light illumination imaging. In the past, many traditional structured light illumination microscopy reconstruction algorithms based on mathematical models have been proposed, which are the main methods of structured light illumination microscopy reconstruction. In recent years, deep learning technology has also been used for structured light illumination microscopy reconstruction, which can achieve better reconstruction results than traditional reconstruction methods. The paper Deep learning enables structured illumination microscopy with low light levels and enhanced speed proposes a structured light illumination microscopy reconstruction network based on UNet. The paper Evaluation and development of deep neural networks for image super-resolution in optical microscopy proposes a Fourier channel attention mechanism network for structured light illumination microscopy reconstruction.

[0003] Existing deep learning-based structured light microscopy reconstruction methods use only high-signal-to-noise ratio structured light microscopy images as ground truth to supervise the training of neural networks for super-resolution reconstruction of modulated widefield images. While this training approach can achieve good reconstruction results, it suffers from overfitting during network training, leaving room for further improvement. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned existing technologies, the present invention proposes a dual-supervised training structured illumination microscopic deep learning reconstruction method; using the training method of the present invention to train a neural network to reconstruct structured illumination microscopic images can further improve the image quality of structured illumination microscopic super-resolution reconstruction and improve quantitative indicators such as signal-to-noise ratio and structural similarity.

[0005] The dual-supervisory training structured illumination microscopic deep learning reconstruction method of the present invention comprises the following steps:

[0006] 1) Collecting raw data:

[0007] Setting the field of view of the structured light illumination microscope so that the field of view remains unchanged during the data acquisition process of the structured light illumination microscope;

[0008] Acquire original structured illumination microscopic super-resolution images under high laser intensity and long exposure time conditions. SIM 'And the original wide field true value image I WF '; In order to facilitate network training, the original wide field true value image I is transformed into WF 'Perform 2x upsampling to obtain the original upsampled wide-field true value image I WF2× ';

[0009] The original modulated widefield image I was acquired under low laser intensity and short exposure conditions in the same field of view. Input ';

[0010] 2) Build a training dataset:

[0011] The original structured illumination micro super-resolution image I SIM ', original upsampled wide field ground truth image I WF2× ' and the original modulated wide field image I Input 'Perform image enhancement to obtain structured illumination microscopic super-resolution image I SIM , wide field ground truth image I WF and modulated widefield image I Input , as a training data set;

[0012] 3) Generate the point spread function of the system:

[0013] generating a point spread function of the system according to imaging usage parameters;

[0014] 4) Building a neural network:

[0015] A neural network for structured illumination microscopic reconstruction is built based on a deep learning platform, with a modulated wide-field image as input and a structured illumination microscopic super-resolution image and a wide-field ground-truth image as output.

[0016] 5) Double-supervised alternating training of neural networks:

[0017] The neural network for structured illumination micro reconstruction is trained by gradient descent in a dual-supervisory manner:

[0018] The neural network is supervised and trained using the micro-super-resolution image based on structured illumination.

[0019] Loss function for supervised network training of micro-super-resolution images SIM for:

[0020] Loss SIM =MSE(Ioutput ,I SIM )+λ SSIM (1-SSIM(I output ,I SIM ))

[0021] Among them, MSE and SSIM are the loss functions Loss of supervised network training based on structured illumination micro super-resolution images. SIM The root mean square error and structural similarity, I output is the output image of the neural network, I SIM is the structured illumination microscopic super-resolution image, λ SSIM is the weight coefficient, ranging from 0.05 to 0.15;

[0022] The neural network is supervised and trained based on the wide field true value image. The Pearson correlation coefficient between the neural network output after point spread function convolution and the wide field true value image is calculated to supervise the network training.

[0023] Loss function for supervised network training of images WF for:

[0024]

[0025] Among them, I output is the output image of the neural network, I WF is the wide-field true image, PSF is the point spread function of the system, Represents the convolution operation, i and j are the pixel indexes of the image, (I WF ) i,j is the wide-field ground-truth image of the pixel in the i-th row and j-th column, where M and N represent the number of rows and columns of pixels in the image, respectively;

[0026] In each round of training, the loss function Loss is used for supervised network training based on structured illumination micro-super-resolution images. SIM And the loss function Loss for supervised network training based on wide-field ground-truth images WF Performing dual-supervisory alternating training on the neural network until the neural network converges or reaches the training round, and obtaining a neural network trained with dual-supervisory alternating training;

[0027] 6) Application of Neural Networks:

[0028] After the neural network is trained, the actual application image is reconstructed by inference, taking the modulated wide-field image as input.

[0029] The neural network outputs the structured illumination microscopic super-resolution image and wide-field true value image.

[0030] In step 1), the high laser intensity and long exposure time conditions are as follows: the laser intensity is greater than 40% of the maximum laser intensity, and the exposure time is greater than 100 milliseconds. The low laser intensity and short exposure conditions are as follows: the laser intensity is 1-20% of the maximum laser intensity, and the exposure time is 10-100 milliseconds.

[0031] In step 2), image enhancement includes: performing random cropping, 90° rotation, -90° rotation, horizontal flipping, and vertical flipping operations within the same field of view.

[0032] In step 3), generating a point spread function of the system according to the imaging parameters includes:

[0033] The variance σ of the Gaussian function generated by imaging is:

[0034]

[0035] Among them, λ em is the illumination wavelength used in imaging (unit: nanometers), pixelsize is the pixel size of imaging (unit: nanometers), and NA is the numerical aperture used in imaging;

[0036] Using the variance value σ of the Gaussian function, the Gaussian model is used to generate the point spread function PSF of the system during the imaging process:

[0037]

[0038] Where PSF represents the point spread function of the system, and x and y are integer image coordinates.

[0039] In step 4), a U-shaped network or a residual channel attention mechanism network (RCAN) is used as the mainstream super-resolution network. For lattice-modulated structured light microscopy, the network has 7 input channels and 1 output channel. For stripe-modulated structured light microscopy, the network has 9 input channels and 1 output channel.

[0040] In step 5), the number of training times is 20 to 40, a stochastic gradient descent optimizer is used, and a learning rate is 0.0001 to 0.0005.

[0041] Loss function for supervised network training based on structured illumination microscopic super-resolution images SIM The root mean square error MSE and structural similarity SSIM are:

[0042]

[0043]

[0044] Among them, X represents the image to be compared, Y represents the true value image, and X i,j and Y i,j Represents the pixel of the i-th row and j-th column of the image to be compared and the true value image, μ X and μ Y Represent the mean of the image to be compared and the true image, σ X and σ Y Represent the variance of the image to be compared and the true value image, σ XY Represents the covariance between the image to be compared and the true image, and C1 and C2 are constants.

[0045] Advantages of the present invention:

[0046] The present invention collects original structured light illuminated microscopic super-resolution images and original wide-field true value images under conditions of high laser intensity and high exposure time, and collects original modulated wide-field images under conditions of low laser intensity and short exposure in the same field of view, and uses them as training data sets after image enhancement; gradient descent training is performed on the neural network for structured light illuminated microscopic reconstruction using a dual-supervision form based on structured light illuminated microscopic super-resolution images and wide-field true value images; the present invention can further improve the performance of deep learning in reconstructing structured light super-resolution microscopic images; the present invention is applied to image reconstruction of mainstream structured light illumination microscopes such as stripe modulated structured light and lattice modulated structured light illumination microscopes. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of the dual-supervisory training structured illumination micro deep learning reconstruction method of the present invention;

[0048] Figure 2 Schematic diagram of dual-supervised alternating training of the dual-supervised training structured illumination micro deep learning reconstruction method of the present invention;

[0049] Figure 3 Figure 1 is a diagram showing the reconstruction effect of an embodiment of the dual-supervised training structured light microscopy deep learning reconstruction method according to the present invention applied to lattice illumination structured light microscopy deep learning, wherein (a) is a modulated wide-field image of four common organelles, (b) is a statistical box plot of the peak signal-to-noise ratio index of the actin structure reconstructed image, and (c) is a statistical box plot of the multi-scale structural similarity index of the actin structure reconstructed image;

[0050] Figure 4 The following are reconstruction results of an embodiment of the dual-supervised training structured light microscopy deep learning reconstruction method according to the present invention applied to stripe modulated illumination structured light microscopy deep learning, where (a) is the peak signal-to-noise ratio diagram, (b) is the average error diagram, and (c) is a comparison display diagram of the reconstructed muscle protein image. DETAILED DESCRIPTION

[0051] The present invention will be further described below through specific embodiments in conjunction with the accompanying drawings.

[0052] The dual-supervised training structured illumination micro deep learning reconstruction method of this embodiment is as follows: Figure 1 As shown, the following steps are included:

[0053] 1) Collecting raw data:

[0054] Setting the field of view of the structured light illumination microscope so that the field of view remains unchanged during the data acquisition process of the structured light illumination microscope;

[0055] First, under the conditions of high laser intensity and high exposure time: the laser illumination intensity is greater than 40% of the maximum laser intensity, the exposure time is greater than 200 milliseconds, and the original structured illumination microscopic super-resolution image I is collected. SIM 'And the original wide field true value image I WF '; In order to facilitate network training, the original wide field true value image I is transformed into WF 'Perform 2x upsampling to obtain the original upsampled wide-field true value image I WF2× ';

[0056] Then, in the same field of view, under low laser intensity and short exposure conditions: the laser illumination intensity was 10% of the maximum laser intensity, and the exposure time was 10 milliseconds to acquire the original modulated wide-field image I Input ';

[0057] 2) Build a training dataset:

[0058] The original structured illumination micro super-resolution image I SIM ', original upsampled wide field ground truth image I WF2× ' and the original modulated wide field image I Input 'In the same field of view, random cropping, 90° rotation, -90° rotation, horizontal flipping and vertical flipping operations are performed to enhance the image and obtain the structured illumination microscopic super-resolution image I SIM , wide field ground truth image I WF and modulated widefield image I Input , as the training data set, the structured light illumination micro super-resolution image I SIM and the upsampled wide-field ground-truth image I WF2× The image slice size is 256×256, and the modulated wide-field image I Input The image slice size is 128×128;

[0059] 3) Generate the point spread function of the system:

[0060] The variance σ of the Gaussian function generated by imaging is:

[0061]

[0062] Among them, λ em is the wavelength of illumination used for imaging (unit: nanometers), pixelsize is the pixel size of the imaging (unit:

[0063] nanometers), NA is the numerical aperture used in imaging;

[0064] Using the variance value σ of the Gaussian function, the Gaussian model is used to generate the point spread function PSF of the system during the imaging process:

[0065]

[0066] Where PSF represents the point spread function of the system, x and y are integer image coordinates;

[0067] 4) Building a neural network:

[0068] Based on deep learning platforms such as Pytorch or TensorFlow, a neural network is constructed using mainstream super-resolution networks such as U-net (UNet) or residual channel attention mechanism network (RCAN). For lattice-modulated structured light illumination microscopy, the network has 7 input channels and 1 output channel; for stripe-modulated structured light illumination microscopy, the network has 9 input channels and 1 output channel. A neural network for structured light microscopy reconstruction is constructed on the deep learning platform, with the modulated wide-field image as input and the structured light microscopy super-resolution image and wide-field ground truth image as output.

[0069] 5) Double-supervised alternating training of neural networks:

[0070] The double-supervised form is used to perform gradient descent training on the neural network for microscopic reconstruction of structured illumination, such as Figure 2 As shown, the number of training times is 30, the stochastic gradient descent optimizer is used, and the learning rate is 0.0001:

[0071] Loss function for supervised network training based on structured illumination microscopic super-resolution images SIM The root mean square error MSE and structural similarity SSIM are:

[0072]

[0073] Among them, X represents the image to be compared, Y represents the true value image, and X i，j and Y i，j Represents the pixel of the i-th row and j-th column of the image to be compared and the true value image, μ X and μY Represent the mean of the image to be compared and the true image, σ X and σ Y Represent the variance of the image to be compared and the true value image, σ XY Represents the covariance between the image to be compared and the true image, C1 and C2 are two constants, C1 = 0.01, C2 = 0.03;

[0074] The neural network is supervised and trained using the micro-super-resolution image based on structured illumination.

[0075] Loss function for supervised network training of micro-super-resolution images SIM for:

[0076] Loss SIM =MSE(I output ,I SIM )+λ SSIM (1-SSIM(I output ,I SIM ))

[0077] Among them, MSE and SSIM are the loss functions Loss of supervised network training based on structured illumination micro super-resolution images. SIM The root mean square error and structural similarity, I output is the output image of the neural network, I SIM is the true value of the structured illumination micro-super-resolution image, λ SSIM is the weight coefficient, usually ranging from 0.05 to 0.15;

[0078] The neural network is supervised and trained based on the wide field true value image. The Pearson correlation coefficient between the neural network output after point spread function convolution and the wide field true value image is calculated to supervise the network training.

[0079] Loss function for supervised network training of images WF for:

[0080]

[0081] Among them, I output is the output image of the neural network, I WF is the wide-field true image, PSF is the point spread function of the system, Represents the convolution operation, i and j are the pixel indexes of the image, (I WF ) is the wide-field ground-truth image of the pixel at row i and column j, where M and N represent the width and length of the image, respectively;

[0082] In each round of training, the loss function Loss is used for supervised network training based on structured illumination micro-super-resolution images. SIM And the loss function Loss for supervised network training based on wide-field ground-truth images WF The neural network is trained alternately with dual supervision until the neural network converges or reaches the training round, thereby obtaining a neural network trained with dual supervision. Both the structured illumination microscopic super-resolution image and the wide-field ground truth image are images with high signal-to-noise ratio.

[0083] 6) Application of Neural Networks:

[0084] After the neural network is trained, the actual application image is inferred and reconstructed, with the modulated wide-field image as input, and the neural network outputs the structured illumination microscopic super-resolution image and the wide-field true value image.

[0085] Using the method of the present invention, the original data are images of nuclear pore complex, endoplasmic reticulum, microtubules and actin obtained by lattice modulation structured light illumination microscopy. After the neural network training converges, the network performance is tested on the test set. The results are as follows Figure 3 shown. Figure 3 (a) shows a modulated wide-field image of four common organelles, a ground-truth structured light super-resolution image, an image reconstructed using the RCAN network trained using a traditional scheme, and an image reconstructed using the RCAN network trained using the dual supervision of the present invention. The reconstruction error value is marked in the lower right corner of the figure. Figure 3 (b) Box plots showing the peak signal-to-noise ratio of actin structure reconstruction images obtained by the traditional scheme and the dual-supervisory training of the present invention; Figure 3 (c) is a box plot showing the multi-scale structural similarity of actin structure reconstruction images obtained by the traditional scheme and the dual-supervisory training of the present invention. Figure 3 As shown in the figure, error analysis and image quality quantitative evaluation criteria indicate that the dual-supervised training proposed in this invention can further improve the performance of deep learning reconstruction of structured light super-resolution microscopic images compared with the traditional scheme.

[0086] Using the method of the present invention, the original data are images of nuclear pore complex, endoplasmic reticulum, microtubules and actin obtained by stripe modulated structured light illumination microscope. After the neural network training converges, the network performance is tested on the test set. The results are as follows: Figure 4 shown. Figure 4 (a) and (b) are the peak signal-to-noise ratio and mean error of the reconstructed images of muscle proteins in the BioSR open source dataset using the traditional scheme and the dual-supervisory training of the present invention using the U-shaped network (ScUNet) and the Fourier channel attention network (DFCAN), respectively. The two networks are compared using the traditional method and the dual-supervisory strategy proposed in the present invention; Figure 4(c) is a comparison of the reconstructed muscle protein images of the U-shaped network and the Fourier channel attention network trained by the traditional and the present invention. The reconstruction error value is marked in the lower right corner of the figure. Figure 4 As shown in the figure, error analysis and image quality quantitative evaluation criteria indicate that the dual-supervised training proposed in this invention can further improve the performance of deep learning reconstruction of structured light super-resolution microscopic images compared with the traditional scheme.

[0087] Finally, it should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments; the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A dual-supervisory training structured illumination micro-reconstruction method with deep learning, characterized by: The structured light illumination microscopic deep learning reconstruction method comprises the following steps: 1) Collect raw data: Setting the field of view of the structured light illumination microscope so that the field of view remains unchanged during the data acquisition process of the structured light illumination microscope; Acquire original structured illumination microscopic super-resolution images under high laser intensity and long exposure time conditions. SIM 'And the original wide field true value image I WF '; In order to facilitate network training, the original wide field true value image I is transformed into WF 'Perform 2x upsampling to obtain the original upsampled wide-field true value image I WF2× '; The original modulated widefield image I was acquired under low laser intensity and short exposure conditions in the same field of view. Input '; 2) Build a training dataset: The original structured illumination micro super-resolution image I SIM ', original upsampled wide field ground truth image I WF2× ' and the original modulated wide field image I Input 'Perform image enhancement to obtain structured illumination microscopic super-resolution image I SIM , wide field ground truth image I WF and modulated widefield image I Input , as a training data set; 3) Generate the point spread function of the system: generating a point spread function of the system according to imaging usage parameters; 4) Building a neural network: A neural network for microscopic reconstruction of structured illumination is constructed based on a deep learning platform, with a modulated wide-field image as input. The output is a structured illumination microscopic super-resolution image and a wide-field ground-truth image; 5) Double-supervised alternating training of neural networks: The neural network for structured illumination micro reconstruction is trained by gradient descent in a dual-supervisory manner: The neural network is supervised and trained using structured light illumination micro-super-resolution images. The loss function of the supervised network training based on structured light illumination micro-super-resolution images is Loss. SIM for: Loss SIM =MSE(I output ,I SIM )+λ SSIM (1-SSIM(I output ,I SIM )) Among them, MSE and SSIM are the loss functions Loss of supervised network training based on structured illumination micro super-resolution images. SIM The root mean square error and structural similarity, I output is the output image of the neural network, λ SSIM is the weight coefficient; The neural network is supervised and trained based on wide-field true value images. The Pearson correlation coefficient between the neural network output after point spread function convolution and the wide-field true value image is calculated to supervise the network training. The loss function of the supervised network training based on the wide-field true value image is Loss. WF for: Among them, I output is the output image of the neural network, PSF is the point spread function of the system, represents the convolution operation, (I WF ) i,j is the wide-field ground-truth image of the pixel in the i-th row and j-th column, where M and N represent the number of rows and columns of pixels in the image, respectively; In each round of training, the loss function Loss is used for supervised network training based on structured illumination micro-super-resolution images. SIM And the loss function Loss for supervised network training based on wide-field ground-truth images WF Performing dual-supervisory alternating training on the neural network until the neural network converges or reaches the training round, and obtaining a neural network trained with dual-supervisory alternating training; 6) Application of Neural Networks: After the neural network is trained, the actual application image is inferred and reconstructed, with the modulated wide-field image as input, and the neural network outputs the structured illumination microscopic super-resolution image and the wide-field true value image.

2. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 1), the high laser intensity and high exposure time conditions are that the laser illumination intensity is greater than 40% of the maximum laser intensity, and the exposure time is greater than 100 milliseconds.

3. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 1), the low laser intensity and short exposure conditions are that the laser illumination intensity is 1 to 20% of the maximum laser intensity, and the exposure time is 10 to 100 milliseconds.

4. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 2), image enhancement includes: performing random cropping, 90° rotation, -90° rotation, horizontal flipping, and vertical flipping operations within the same field of view.

5. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 3), generating a point spread function of the system according to the imaging parameters includes: The variance σ of the Gaussian function generated by imaging is: Among them, λ em is the illumination wavelength used for imaging, pixelsize is the pixel size of imaging, and NA is the numerical aperture used in imaging; Using the variance value σ of the Gaussian function, the Gaussian model is used to generate the point spread function PSF of the system during the imaging process: Where x and y are integer image coordinates.

6. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 4), a mainstream super-resolution network is adopted, such as a U-type network or a residual channel attention mechanism network.

7. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 4), for the lattice modulated structured light illumination microscope, the number of input channels of the network is 7, and the number of output channels is 1.

8. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 4), for the fringe modulated structured light illumination microscope, the number of input channels of the network is 9, and the number of output channels is 1.

9. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 5), the number of training times is 20 to 40, a stochastic gradient descent optimizer is used, and a learning rate is 0.0001 to 0.0005.

10. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 5), the loss function Loss of the supervised network training based on the high signal-to-noise ratio structured light illumination super-resolution image is SIM The root mean square error MSE and structural similarity SSIM are: Among them, X represents the image to be compared, Y represents the true value image, and X i,j and Y i,j Represents the pixel of the i-th row and j-th column of the image to be compared and the true value image, μ X and μ Y Represent the mean of the image to be compared and the true image, σ X and σ Y Represent the variance of the image to be compared and the true value image, σ XY Represents the covariance between the image to be compared and the true image, and C1 and C2 are constants.

Images