Structured illumination obvious micro deep learning reconstruction method based on dual-supervised training

Abstract

The invention discloses a structured illumination obvious micro deep learning reconstruction method based on dual-supervised training. The method comprises the following steps: acquiring an original structure illumination obvious micro-super-resolution image and an original wide-field true value image under the conditions of high laser intensity and high exposure time, acquiring an original modulation wide-field image under the conditions of low laser intensity and short exposure in the same visual field, and performing image enhancement to obtain a training data set; generating a point spread function of the system according to imaging use parameters, and constructing a neural network for structured illumination obvious micro-reconstruction; performing gradient descent training on the neural network of the structured illumination obvious micro-reconstruction by using a dual-supervision form based on the structured illumination obvious micro-super-resolution image and the wide-field truth value image; according to the method, the performance of deep learning reconstruction of the structured light super-resolution microscopic image can be further improved; the method is applied to image reconstruction of mainstream structured light illumination microscopes such as stripe modulation structured light illumination microscopes and lattice modulation structured light illumination microscopes.

Description

Technical Field

[0001] The present invention relates to structured light illumination microscopy imaging technology, and in particular to a dual-supervisory training structured light illumination microscopy deep learning reconstruction method. Background Art

[0002] Structured light illumination microscopy is a super-resolution microscopy technique commonly used in living cell fluorescence imaging and has been widely used in cell biology research. It achieves super-resolution imaging by modulating the sample with structured light illumination and performing subsequent reconstruction. The reconstruction algorithm is an important part of structured light illumination imaging. In the past, many traditional structured light illumination microscopy reconstruction algorithms based on mathematical models have been proposed, which are the main methods of structured light illumination microscopy reconstruction. In recent years, deep learning technology has also been used for structured light illumination microscopy reconstruction, which can achieve better reconstruction results than traditional reconstruction methods. The paper Deep learning enables structured illumination microscopy with low light levels and enhanced speed proposes a structured light illumination microscopy reconstruction network based on UNet. The paper Evaluation and development of deep neural networks for image super-resolution in optical microscopy proposes a Fourier channel attention mechanism network for structured light illumination microscopy reconstruction.

[0003] In the existing deep learning-based structured light illumination microscopy reconstruction method, only high signal-to-noise ratio structured light illumination microscopy images are used as the truth value to supervise the training of the neural network to reconstruct the modulated wide-field image for super-resolution. Although this training method can achieve good reconstruction results, it is affected by the overfitting of the network training, and its reconstruction effect still has room for further improvement. Summary of the invention

[0004] In view of the problems existing in the above-mentioned prior art, the present invention proposes a dual-supervised training structured illumination microscopic deep learning reconstruction method; using the training method of the present invention to train a neural network to reconstruct structured illumination microscopic images can further improve the image quality of structured illumination microscopic super-resolution reconstruction, and improve quantitative indicators such as signal-to-noise ratio and structural similarity.

[0005] The dual-supervised training structured light illumination micro-deep learning reconstruction method of the present invention comprises the following steps:

[0006] 1) Collecting raw data:

[0007] The field of view of the structured light illumination microscope is set, and the field of view remains unchanged during the data collection process of the structured light illumination microscope;

[0008] Acquisition of original structured illumination microscopic super-resolution images under high laser intensity and high exposure time conditions SIM ' and the original wide field true value image I WF '; In order to facilitate network training, the original wide-field true value image I is interpolated using the bilinear interpolation method WF 'Perform 2x upsampling to obtain the original upsampled wide-field true value image I WF2× ';

[0009] Collect raw modulated widefield images under low laser intensity and short exposure conditions in the same field of view. Input ';

[0010] 2) Construct training data set:

[0011] The original structured illumination microscopic super-resolution image I SIM ', original upsampled wide field true value image I WF2× ' and the original modulated wide field image I Input 'Perform image enhancement to obtain structured illumination microscopic super-resolution image I SIM , wide field true value image I WF and modulated widefield image I Input , as a training data set;

[0012] 3) Generate the point spread function of the system:

[0013] generating a point spread function of the system according to the imaging usage parameters;

[0014] 4) Building a neural network:

[0015] A neural network for structured illumination microscopic reconstruction was built based on a deep learning platform, with the modulated wide-field image as input and the structured illumination microscopic super-resolution image and wide-field true value image as output;

[0016] 5) Double-supervised alternating training of neural networks:

[0017] The neural network for structured illumination microscopic reconstruction is trained by gradient descent in a dual-supervisory manner:

[0018] The neural network is supervised by using the micro-super-resolution image based on structured illumination.

[0019] Loss function for supervised network training of micro-super-resolution images SIM for:

[0020] Loss SIM =MSE(Ioutput ,I SIM )+λ SSIM (1-SSIM(I output ,I SIM ))

[0021] Among them, MSE and SSIM are the loss functions Loss of supervised network training based on structured illumination micro-super-resolution images. SIM The root mean square error and structural similarity, I output is the output image of the neural network, I SIM is the structured light illumination microscopic super-resolution image, λ SSIM is the weight coefficient, ranging from 0.05 to 0.15;

[0022] The neural network is supervised and trained based on the wide-field true value image. The Pearson correlation coefficient between the neural network output after point spread function convolution and the wide-field true value image is calculated to supervise the network training.

[0023] Loss function for supervised network training of images WF for:

[0024]

[0025] Among them, I output is the output image of the neural network, I WF is the wide-field true image, PSF is the point spread function of the system, Represents the convolution operation, i and j are the pixel indexes of the image, (I WF ) i,j is the wide-field true value image of the pixel in the i-th row and j-th column, M and N represent the number of rows and columns of pixels in the image respectively;

[0026] In each round of training, the loss function Loss is used for supervised network training based on structured illumination micro-super-resolution images. SIM And the loss function Loss for supervised network training based on wide-field true value images WF Performing double-supervised alternating training on the neural network until the neural network converges or reaches the training round, and obtaining a double-supervised alternating trained neural network;

[0027] 6) Application of Neural Networks:

[0028] After the neural network is trained, the actual application image is reconstructed by inference, taking the modulated wide-field image as input.

[0029] The structured illumination microscopic super-resolution image and wide-field true value image are obtained as output of the neural network.

[0030] Wherein, in step 1), the high laser intensity and high exposure time conditions are that the laser illumination intensity is greater than 40% of the maximum laser intensity, and the exposure time is greater than 100 milliseconds. The low laser intensity and short exposure conditions are that the laser illumination intensity is 1-20% of the maximum laser intensity, and the exposure time is 10-100 milliseconds.

[0031] In step 2), the image enhancement includes: performing random cropping, rotating 90°, rotating -90°, horizontal flipping, and vertical flipping operations under the same field of view.

[0032] In step 3), generating a point spread function of the system according to the imaging parameters includes:

[0033] The variance value σ of the Gaussian function generated by imaging is:

[0034]

[0035] Among them, λ em is the illumination wavelength used in imaging (in nanometers), pixelsize is the pixel size of imaging (in nanometers), and NA is the numerical aperture used in imaging;

[0036] Using the variance value σ of the Gaussian function, the Gaussian model is used to generate the point spread function PSF of the system during the imaging process:

[0037]

[0038] Where PSF represents the point spread function of the system, and x and y are integer image coordinates.

[0039] In step 4), a U-type network or a residual channel attention mechanism network RCAN mainstream super-resolution network is used. For lattice modulated structured light illumination microscopy, the number of input channels of the network is 7, and the number of output channels is 1. For stripe modulated structured light illumination microscopy, the number of input channels of the network is 9, and the number of output channels is 1.

[0040] In step 5), the number of trainings is 20 to 40 times, a stochastic gradient descent optimizer is used, and a learning rate is 0.0001 to 0.0005.

[0041] Loss function for supervised network training based on structured illumination micro-super-resolution images SIM The root mean square error MSE and structural similarity SSIM are:

[0042]

[0043]

[0044] Among them, X represents the image to be compared, Y represents the true value image, and X i,j and Y i,j Represent the pixels in the i-th row and j-th column of the image to be compared and the true value image, μ X and μ Y Represent the mean of the image to be compared and the true image, σ X and σ Y Represent the variance of the image to be compared and the true value image, σ XY Represents the covariance between the image to be compared and the true image, C 1 and C 2 are constants respectively.

[0045] Advantages of the present invention:

[0046] The present invention collects original structured light illuminated microscopic super-resolution images and original wide-field true value images under conditions of high laser intensity and high exposure time, and collects original modulated wide-field images under conditions of low laser intensity and short exposure in the same field of view, and uses them as training data sets after image enhancement; gradient descent training is performed on a neural network for structured light illuminated microscopic reconstruction using a dual-supervision form based on structured light illuminated microscopic super-resolution images and based on wide-field true value images; the present invention can further improve the performance of deep learning to reconstruct structured light super-resolution microscopic images; the present invention is applied to image reconstruction of mainstream structured light illumination microscopes such as stripe modulated structured light and lattice modulated structured light illumination microscopes. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the dual-supervised training structured illumination micro deep learning reconstruction method of the present invention;

[0048] Figure 2 A schematic diagram of dual-supervised alternating training of the dual-supervised training structured illumination microscopic deep learning reconstruction method of the present invention;

[0049] Figure 3 The following are reconstruction effect diagrams of an embodiment of the structured light illumination microscopy deep learning reconstruction method according to the dual-supervised training of the present invention applied to the lattice illumination structured light microscope deep learning, wherein (a) is a modulated wide-field image of four common organelles, (b) is a statistical box plot of the peak signal-to-noise ratio index of the actin structure reconstruction image, and (c) is a statistical box plot of the multi-scale structural similarity index of the actin structure reconstruction image;

[0050] Figure 4 The following is a diagram showing the reconstruction effect of stripe modulated illumination structured light microscopy deep learning reconstruction method according to an embodiment of the present invention, applied to the reconstruction of stripe modulated illumination structured light microscopy deep learning, wherein (a) is a peak signal-to-noise ratio diagram, (b) is an average error diagram, and (c) is a comparison display diagram of the reconstructed muscle protein image. DETAILED DESCRIPTION

[0051] The present invention will be further described below through specific embodiments in conjunction with the accompanying drawings.

[0052] The dual-supervised training structured illumination micro-deep learning reconstruction method of this embodiment is as follows: Figure 1 As shown, the following steps are included:

[0053] 1) Collecting raw data:

[0054] The field of view of the structured light illumination microscope is set, and the field of view remains unchanged during the data collection process of the structured light illumination microscope;

[0055] First, under the conditions of high laser intensity and high exposure time: the laser intensity is greater than 40% of the maximum laser intensity, the exposure time is greater than 200 milliseconds, and the original structured illumination microscopic super-resolution image I is collected. SIM ' and the original wide field true value image I WF '; In order to facilitate network training, the original wide-field true value image I is interpolated using the bilinear interpolation method WF 'Perform 2x upsampling to obtain the original upsampled wide-field true value image I WF2× ';

[0056] Then, in the same field of view, under low laser intensity and short exposure conditions: the laser illumination intensity was 10% of the maximum laser intensity, and the exposure time was 10 milliseconds to collect the original modulated wide-field image I Input ';

[0057] 2) Construct training data set:

[0058] The original structured illumination microscopic super-resolution image I SIM ', original upsampled wide field true value image I WF2× ' and the original modulated wide field image I Input 'In the same field of view, random cropping, rotation of 90°, rotation of -90°, horizontal flipping and vertical flipping operations are performed to enhance the image and obtain the structured illumination microscopic super-resolution image I SIM , wide field true value image I WF and modulated widefield image I Input , as the training data set, the structured light illumination microscopic super-resolution image I SIM and the upsampled wide-field ground-truth image I WF2× The image slice size is 256×256, and the modulation wide field image I Input The image slice size is 128×128;

[0059] 3) Generate the point spread function of the system:

[0060] The variance value σ of the Gaussian function generated by imaging is:

[0061]

[0062] Among them, λ em is the wavelength of illumination used for imaging (unit: nanometers), pixelsize is the pixel size of the imaging (unit:

[0063] nanometers), NA is the numerical aperture used in imaging;

[0064] Using the variance value σ of the Gaussian function, the Gaussian model is used to generate the point spread function PSF of the system during the imaging process:

[0065]

[0066] Where PSF represents the point spread function of the system, x and y are integer image coordinates;

[0067] 4) Building a neural network:

[0068] Based on deep learning platforms such as Pytorch or TensorFlow, a neural network is constructed using the mainstream super-resolution network of U-net or residual channel attention mechanism network (RCAN); for lattice modulated structured light illumination microscopy, the number of input channels of the network is 7, and the number of output channels is 1; for stripe modulated structured light illumination microscopy, the number of input channels of the network is 9, and the number of output channels is 1. The deep learning platform constructs a neural network for structured light illumination microscopy reconstruction, with modulated wide-field images as input, and structured light illumination microscopy super-resolution images and wide-field true value images as output;

[0069] 5) Double-supervised alternating training of neural networks:

[0070] The double-supervised form is used to perform gradient descent training on the neural network for structured illumination microscopic reconstruction, such as Figure 2 As shown, the number of training times is 30, the stochastic gradient descent optimizer is used, and the learning rate is 0.0001:

[0071] Loss function for supervised network training based on structured illumination micro-super-resolution images SIM The root mean square error MSE and structural similarity SSIM are:

[0072]

[0073] Among them, X represents the image to be compared, Y represents the true value image, and X i，j and Y i，j Represent the pixels in the i-th row and j-th column of the image to be compared and the true value image, μX and μ Y Represent the mean of the image to be compared and the true image, σ X and σ Y Represent the variance of the image to be compared and the true value image, σ XY Represents the covariance between the image to be compared and the true image, C 1 and C 2 are two constants, C 1 =0.01,C 2 =0.03;

[0074] The neural network is supervised by using the micro-super-resolution image based on structured illumination.

[0075] Loss function for supervised network training of micro-super-resolution images SIM for:

[0076] Loss SIM =MSE(I output ,I SIM )+λ SSIM (1-SSIM(I output ,I SIM ))

[0077] Among them, MSE and SSIM are the loss functions Loss of supervised network training based on structured illumination micro-super-resolution images. SIM The root mean square error and structural similarity, I output is the output image of the neural network, I SIM is the true value of the structured illumination micro-super-resolution image, λ SSIM is the weight coefficient, usually ranging from 0.05 to 0.15;

[0078] The neural network is supervised and trained based on the wide-field true value image. The Pearson correlation coefficient between the neural network output after point spread function convolution and the wide-field true value image is calculated to supervise the network training.

[0079] Loss function for supervised network training of images WF for:

[0080]

[0081] Among them, I output is the output image of the neural network, I WF is the wide-field true image, PSF is the point spread function of the system, Represents the convolution operation, i and j are the pixel indexes of the image, (I WF) is the wide-field true value image of the pixel in the i-th row and j-th column, M and N represent the width and length of the image respectively;

[0082] In each round of training, the loss function Loss is used for supervised network training based on structured illumination micro-super-resolution images. SIM And the loss function Loss for supervised network training based on wide-field true value images WF The neural network is trained alternately with dual supervision until the neural network converges or reaches the training round, and a neural network trained with dual supervision is obtained; both the structured light illumination microscopic super-resolution image and the wide-field true value image are images with high signal-to-noise ratio;

[0083] 6) Application of Neural Networks:

[0084] After the neural network is trained, the actual application image is inferred and reconstructed, with the modulated wide-field image as input, and the structured illumination microscopic super-resolution image and wide-field true value image are obtained as output of the neural network.

[0085] Using the method of the present invention, the original data are images of nuclear pore complex, endoplasmic reticulum, microtubules and actin obtained by lattice modulation structured light illumination microscope. After the neural network training converges, the network performance is tested on the test set. The results are as follows: Figure 3 shown. Figure 3 (a) shows the modulated wide-field image of four common organelles, the true structured light super-resolution image, the image reconstructed by the RCAN network trained with the traditional scheme, and the image reconstructed by the RCAN network trained with the dual supervision of the present invention. The reconstruction error value is marked in the lower right corner of the figure; Figure 3 (b) is a box plot showing the peak signal-to-noise ratio of actin structure reconstruction images obtained by the conventional scheme and the dual-supervisory training of the present invention; Figure 3 (c) is a box plot showing the multi-scale structural similarity index of actin structure reconstruction images obtained by the conventional scheme and the dual-supervisory training of the present invention. Figure 3 As shown, error analysis and image quality quantitative evaluation criteria indicate that, compared with traditional schemes, the dual-supervised training proposed in the present invention can further improve the performance of deep learning reconstruction of structured light super-resolution microscopic images.

[0086] Using the method of the present invention, the original data are images of nuclear pore complex, endoplasmic reticulum, microtubules and actin obtained by stripe modulated structured light illumination microscope. After the neural network training converges, the network performance is tested on the test set. The results are as follows: Figure 4 shown. Figure 4(a) and (b) are the peak signal-to-noise ratio and average error index of the muscle protein in the BioSR open source dataset trained by the traditional scheme and the dual-supervised training of the present invention using the U-shaped network (ScUNet) and the Fourier channel attention network (DFCAN) to train and reconstruct the image. The two networks are compared using the traditional method and the dual-supervised strategy proposed by the present invention; Figure 4 (c) is a comparison of the reconstructed muscle protein images of the U-shaped network and the Fourier channel attention network trained by the traditional and the present invention, and the reconstruction error value is marked in the lower right corner of the figure. Figure 4 As shown, error analysis and image quality quantitative evaluation criteria indicate that, compared with traditional schemes, the dual-supervised training proposed in the present invention can further improve the performance of deep learning reconstruction of structured light super-resolution microscopic images.

[0087] Finally, it should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A dual-supervised training structured illumination micro-deep learning reconstruction method, characterized in that: The structured light illumination micro-deep learning reconstruction method comprises the following steps: 1) Collect raw data: The field of view of the structured light illumination microscope is set, and the field of view remains unchanged during the data collection process of the structured light illumination microscope; Acquisition of original structured illumination microscopic super-resolution images under high laser intensity and high exposure time conditions SIM ' and the original wide field true value image I WF '; In order to facilitate network training, the original wide-field true value image I is interpolated using the bilinear interpolation method WF 'Perform 2x upsampling to obtain the original upsampled wide-field true value image I WF2× '; Collect raw modulated widefield images under low laser intensity and short exposure conditions in the same field of view. Input '; 2) Construct training data set: The original structured illumination microscopic super-resolution image I SIM ', original upsampled wide field true value image I WF2× ' and the original modulated wide field image I Input 'Perform image enhancement to obtain structured illumination microscopic super-resolution image I SIM , wide field true value image I WF and modulated widefield image I Input , as a training data set; 3) Generate the point spread function of the system: generating a point spread function of the system according to the imaging usage parameters; 4) Building a neural network: A neural network for microscopic reconstruction of structured illumination is constructed based on a deep learning platform, with a modulated wide-field image as input. The structured illumination microscopic super-resolution image and wide-field ground-truth image are taken as output; 5) Double-supervised alternating training of neural networks: The neural network for structured illumination microscopic reconstruction is trained by gradient descent in a dual-supervisory manner: The neural network is supervised and trained using structured illumination micro-super-resolution images. The loss function of the supervised network training based on structured illumination micro-super-resolution images is Loss. SIM for: Loss SIM =MSE(I output ,I SIM )+λ SSIM (1-SSIM(I output ,I SIM )) Among them, MSE and SSIM are the loss functions Loss of supervised network training based on structured illumination micro-super-resolution images. SIM The root mean square error and structural similarity, I output is the output image of the neural network, λ SSIM is the weight coefficient; The neural network is supervised and trained based on the wide-field true value image. The Pearson correlation coefficient between the neural network output after point spread function convolution and the wide-field true value image is calculated to supervise the network training. The loss function Loss of the supervised network training based on the wide-field true value image is WF for: Among them, I output is the output image of the neural network, PSF is the point spread function of the system, represents the convolution operation, (I WF ) i,j is the wide-field true value image of the pixel in the i-th row and j-th column, M and N represent the number of rows and columns of pixels in the image respectively; In each round of training, the loss function Loss is used for supervised network training based on structured illumination micro-super-resolution images. SIM And the loss function Loss for supervised network training based on wide-field true value images WF Performing double-supervised alternating training on the neural network until the neural network converges or reaches the training round, and obtaining a double-supervised alternating trained neural network; 6) Application of Neural Networks: After the neural network is trained, the actual application image is inferred and reconstructed, with the modulated wide-field image as input, and the structured illumination microscopic super-resolution image and wide-field true value image are obtained as output of the neural network.

2. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 1), the high laser intensity and high exposure time conditions are that the laser illumination intensity is greater than 40% of the maximum laser intensity, and the exposure time is greater than 100 milliseconds.

3. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 1), the low laser intensity and short exposure conditions are that the laser illumination intensity is 1 to 20% of the maximum laser intensity, and the exposure time is 10 to 100 milliseconds.

4. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 2), the image enhancement includes: performing random cropping, rotating 90°, rotating -90°, horizontal flipping, and vertical flipping operations under the same field of view.

5. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 3), generating a point spread function of the system according to the imaging parameters includes: The variance value σ of the Gaussian function generated by imaging is: Among them, λ em is the illumination wavelength used in imaging, pixelsize is the pixel size of imaging, and NA is the numerical aperture used in imaging; Using the variance value σ of the Gaussian function, the Gaussian model is used to generate the point spread function PSF of the system during the imaging process: Where x and y are integer image coordinates.

6. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 4), a mainstream super-resolution network is adopted, such as a U-type network or a residual channel attention mechanism network.

7. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 4), for the lattice modulated structured light illumination microscope, the number of input channels of the network is 7, and the number of output channels is 1.

8. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 4), for the fringe modulated structured light illumination microscope, the number of input channels of the network is 9, and the number of output channels is 1.

9. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 5), the number of trainings is 20 to 40 times, a stochastic gradient descent optimizer is used, and a learning rate is 0.0001 to 0.0005.

10. The structured light illumination microscopic deep learning reconstruction method according to claim 1, characterized in that: In step 5), the loss function Loss of the supervised network training based on the high signal-to-noise ratio structured light illumination super-resolution image is SIM The root mean square error MSE and structural similarity SSIM are: Among them, X represents the image to be compared, Y represents the true value image, and X i,j and Y i,j Represent the pixels in the i-th row and j-th column of the image to be compared and the true value image, μ X and μ Y Represent the mean of the image to be compared and the true image, σ X and σ Y Represent the variance of the image to be compared and the true value image, σ XY Represents the covariance between the image to be compared and the true image, and C1 and C2 are constants.

Images