Structured light microscopy image reconstruction method and system based on deep learning

Abstract

The present invention discloses a structured light microscopic image reconstruction method and system based on deep learning, which relates to the field of image processing technology. The acquired structured light illumination light slice image is transmitted to a trained deep learning model to output a reconstructed light slice image. The training process of the deep learning model is as follows: grid illumination structured light is used to project stripes on the same observation target to obtain a phase image pair, and the stripe area content of the first type of original image and the second type of original image in the phase image pair is masked to obtain a first type of mask image and a second type of mask image; the first type of mask image and the second type of mask image are input into the deep learning model to obtain a first type of output image and a second type of output image; a total loss is constructed to update and adjust the parameters of the deep learning model; the illumination light slice microscopic image reconstruction method and system improve the image reconstruction quality.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a structured light microscopy image reconstruction method and system based on deep learning. Background Art

[0002] With the development of modern biomedical technology, the requirements for microscopic imaging results are getting higher and higher. In order to break through the resolution limit of optical microscopes and perform three-dimensional imaging, structured light illumination light sectioning (Optical Sectioning Structured Illumination Microscopy, OS-SIM) three-dimensional microscopy technology came into being. Its basic principle is to indicate the area of ​​the observed object in the focal plane by projecting a fringe pattern with a specific spatial frequency. Since there are almost no fringes in the out-of-focus area, the algorithm can then be used to remove the out-of-focus area information to obtain an ideal light sectioning image.

[0003] Existing image reconstruction algorithms include the RMS (Root Mean Square) algorithm proposed by Neil MA et al. in 1997, the HiLo method proposed by Santos S et al. in 2009 (HiLo combines wide-field imaging and speckle imaging, processes two original images, removes out-of-focus information, and achieves light slicing effect), and the Fast and Adaptive Bi-Dimensional Empirical Mode Decomposition-Hilbert Spiral Transform (FABMED-HS) algorithm based on two-dimensional Hilbert transform proposed by Patorski K et al. in 2014. Some of these algorithms are sensitive to noise, resulting in poor imaging quality, while others require specific adaptation of samples and lighting conditions and lack universality. With the rapid development of computer hardware in recent years, methods based on deep learning have begun to receive attention. However, due to the inherent reasons of optical microscopy and structured light illumination light slicing methods, it is difficult or impossible to obtain clean images as the training truth value for image reconstruction, and often only the reconstruction results of traditional methods can be used as training targets, which sets a strict upper limit for the potential of image reconstruction algorithms based on deep learning. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a structured light microscopy image reconstruction method and system based on deep learning, which improves the image reconstruction quality.

[0005] The structured light microscopic image reconstruction method based on deep learning proposed in the present invention transmits the acquired structured light illumination light slice image to a trained deep learning model to output a reconstructed light slice image; the training process of the deep learning model is as follows:

[0006] Step 1: using grid-illuminated structured light to project stripes on the same observed structured light illumination light slice image target to obtain a phase image pair, wherein the phase image pair includes a first type of original image and a second type of original image, and the first type of original image and the second type of original image are images with different stripe phases;

[0007] Step 2: masking the stripe area contents of the first type of original image and the second type of original image in the phase image pair to obtain a first type of mask image and a second type of mask image;

[0008] Step 3: input the first type of original image, the second type of original image, the first type of mask image, and the second type of mask image into the deep learning model to obtain the first type of output image and the second type of output image;

[0009] Step 4: Multiply the second type of output image by the first type of mask image as the input of the loss function, and multiply the first type of original image by the first type of mask image as the input of the loss function to calculate the first type of mask loss; multiply the first type of output image by the second type of mask image as the input of the loss function, and multiply the second type of original image by the second type of mask image as the input of the loss function to calculate the second type of mask loss;

[0010] Step 5: Multiply the first category original image and the first category mask image as the input of the loss function, and multiply the first category mask image and the first category output image as the input of the loss function to calculate the first category image content loss; multiply the second category original image and the second category mask image as the input of the loss function, and multiply the second category mask image and the second category output image as the input of the loss function to calculate the second category image content loss;

[0011] Step 6: Add the first-class mask loss, the second-class mask loss, the first-class image content loss, and the second-class image content loss in sequence to obtain the total loss, and update and adjust the parameters of the deep learning model based on the total loss.

[0012] Furthermore, in step one, the stripes of the same content in the first type of original image and the second type of original image are complementary in brightness and darkness.

[0013] Furthermore, the trained deep learning model is verified as follows:

[0014] Construct a validation data set in the same way as in step 1, calculate the validation total loss value in the same way as in steps 2 to 6, and determine whether the validation total loss value has a downward trend within a certain period of time;

[0015] If yes, continue training the deep learning model;

[0016] If not, the deep learning model training is complete.

[0017] The structured light microscopic image reconstruction system based on deep learning transmits the acquired structured light illumination light slice image to the trained deep learning model to output the reconstructed light slice image; the training process of the deep learning model is as follows:

[0018] Step 1: using grid-illuminated structured light to project stripes on the same observed structured light illumination light slice image target to obtain a phase image pair, wherein the phase image pair includes a first type of original image and a second type of original image, and the first type of original image and the second type of original image are images with different stripe phases;

[0019] Step 2: masking the stripe area contents of the first type of original image and the second type of original image in the phase image pair to obtain a first type of mask image and a second type of mask image;

[0020] Step 3: input the first type of original image, the second type of original image, the first type of mask image, and the second type of mask image into the deep learning model to obtain the first type of output image and the second type of output image;

[0021] Step 4: Multiply the second type of output image by the first type of mask image as the input of the loss function, and multiply the first type of original image by the first type of mask image as the input of the loss function to calculate the first type of mask loss; multiply the first type of output image by the second type of mask image as the input of the loss function, and multiply the second type of original image by the second type of mask image as the input of the loss function to calculate the second type of mask loss;

[0022] Step 5: Multiply the first category original image and the first category mask image as the input of the loss function, and multiply the first category mask image and the first category output image as the input of the loss function to calculate the first category image content loss; multiply the second category original image and the second category mask image as the input of the loss function, and multiply the second category mask image and the second category output image as the input of the loss function to calculate the second category image content loss;

[0023] Step 6: Add the first-class mask loss, the second-class mask loss, the first-class image content loss, and the second-class image content loss in sequence to obtain the total loss, and update and adjust the parameters of the deep learning model based on the total loss.

[0024] Furthermore, in step one, the stripes of the same content in the first type of original image and the second type of original image are complementary in brightness and darkness.

[0025] The advantages of the structured light microscopic image reconstruction method and system based on deep learning provided by the present invention are: the deep learning model training method of self-supervised bidirectional learning is adopted, so that the structured light illumination light section microscopic image reconstruction based on deep learning can be independent of the truth value prior of the traditional image reconstruction algorithm; by combining the mask loss and the image content loss, the deep learning model is able to perceive the clean image content as the truth value from the image with different phase stripes; compared with the existing deep learning network training method using the traditional image reconstruction algorithm as the truth value, the present embodiment has a higher network capacity upper limit, avoids the problem of unsatisfactory training truth value that may be caused by the traditional image reconstruction algorithm, and has universality and robustness in various complex grid structured light illumination light section microscopic image reconstruction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the structure flow of the present invention;

[0027] Figure 2 Schematic diagram of the experimental results of this embodiment, where (a) is the first type of original image, (b) is the first type of output image, (c) is the second type of original image, and (d) is the second type of output image. DETAILED DESCRIPTION

[0028] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0029] like Figure 1 and 2 As shown, the structured light microscopy image reconstruction method based on deep learning proposed in the present invention transmits the acquired structured light illumination light slice image to the trained deep learning model to output the reconstructed light slice image.

[0030] The training process of the deep learning model is as follows:

[0031] Step 1: using grid-illuminated structured light to project stripes on the same observed structured light illumination light slice image target to obtain a phase image pair, wherein the phase image pair includes a first type of original image and a second type of original image, wherein the first type of original image and the second type of original image are images with different stripe phases, that is, the two original images are distinguished by different stripe phases;

[0032] Use grid lighting structured light to project stripes to collect image data; images with the same content are required to contain at least two stripes with different phases, and two phase shifts are the best, because the stripes in the two-phase pattern with the same content are complementary in brightness and darkness; after the stripe phase is determined, all acquisitions should follow this phase; collect as many images with different contents as possible as training data sets;

[0033] Two types of data are distinguished according to different phases; it is required that an image obtained from multiple original images of the first type has a unique corresponding image in multiple original images of the second type, and the two images have the same content but different fringe phases.

[0034] In addition, the acquisition of the observation target (structured light illumination light slice image) is an existing method, for example, the light beam emitted by the light source is modulated by a light field modulation device to form a structured light illumination light field to irradiate the sample; the sample illuminated by the structured light is imaged on the target surface of the area array digital camera through an optical microscope system, and the original image of the structured light illumination is recorded; the original image is used as the structured light illumination light slice image.

[0035] Step 2: masking the stripe area contents of the first type of original image and the second type of original image in the phase image pair to obtain a first type of mask image and a second type of mask image;

[0036] Through manual annotation, corresponding mask images are created for the two types of images, and a total of two mask images are obtained, one for each type; the mask image is such an image: its two-dimensional size is consistent with the original image obtained by acquisition; the pixel value of each point of the mask image is either 0 or 1, and for the part with a pixel value of 0, the corresponding pixel content of the first type of mask image or the second type of mask image is the content of the stripe area, and for the part with a pixel value of 1, the corresponding pixel content of the first type of mask image or the second type of mask image is the content of the non-stripe area.

[0037] Step 3: input the first type of original image, the second type of original image, the first type of mask image, and the second type of mask image into the deep learning model to obtain the first type of output image and the second type of output image;

[0038] The deep learning model used in this embodiment is an existing model framework. The total loss constructed based on this embodiment is used to adjust the network parameters in the model framework, thereby obtaining a deep learning model suitable for microscopic image reconstruction based on grid illumination. In addition, the self-supervised training method set in this embodiment improves the upper limit of the ability of the image reconstruction algorithm based on deep learning and improves the image reconstruction quality.

[0039] Step 4: Multiply the second type of output image by the first type of mask image as the input of the loss function, and multiply the first type of original image by the first type of mask image as the input of the loss function to calculate the first type of mask loss; multiply the first type of output image by the second type of mask image as the input of the loss function, and multiply the second type of original image by the second type of mask image as the input of the loss function to calculate the second type of mask loss;

[0040] ;

[0041] ;

[0042] ;

[0043] in, and They are the first type of mask loss and the second type of mask loss, is the loss function, and are the first type of output image and the second type of output image respectively, and are the first type of mask image and the second type of mask image respectively, and They are the first and second category original images respectively. represents the product; Represents the number of image pixels, Indicates The actual value of the image pixels, Indicates the image The reconstructed pixel value output by the deep learning model is and , the deep learning model is able to learn the information of the complementary light and dark areas of the two types of images, and learn how to restore the stripe image to its corresponding stripe-free true value.

[0044] Step 5: Multiply the first category original image and the first category mask image as the input of the loss function, and multiply the first category mask image and the first category output image as the input of the loss function to calculate the first category image content loss; multiply the second category original image and the second category mask image as the input of the loss function, and multiply the second category mask image and the second category output image as the input of the loss function to calculate the second category image content loss;

[0045] ;

[0046] ;

[0047] in, and They represent the first type of image content loss and the second type of image content loss respectively.

[0048] With the assistance of image content loss, the visual blind spot of the deep learning model in step 4 is completed, so that the deep learning model can learn the information that should be retained during the reconstruction process of the image.

[0049] It should be noted that the multiplication in steps 4 and 5 refers to the multiplication of the corresponding pixel values ​​of the two images. The area with a value of 0 in the mask image (the first type of mask image and the second type of mask image) will mask the corresponding area information of the image multiplied by it, and the area with a value of 1 in the mask image (the first type of mask image and the second type of mask image) will retain the corresponding area information of the image multiplied by it.

[0050] Step 6: Add the first type of mask loss, the second type of mask loss, the first type of image content loss, and the second type of image content loss in sequence to obtain a total loss, and update and adjust the parameters of the deep learning model based on the total loss;

[0051]

[0052] in, Represents the total loss.

[0053] Total loss As the total loss function of the deep learning model for gradient descent to iteratively optimize network parameters, it can achieve network parameter update.

[0054] After the deep learning model is trained, it is verified by the validation data set. The validation data set is sent to the deep learning model in the same way as the training, and the loss of the validation data set is calculated in the same way as steps 4 to 6. If the loss value of the validation data set after passing through the deep learning model no longer shows a downward trend within a certain period of time, the training can be terminated. If there is still a downward trend, the deep learning model will continue to be trained until it is verified by the validation data set.

[0055] like Figure 2 As shown, based on the illumination light slice microscopic image reconstruction method of this embodiment, two types of original images (the first type of original images (a) and the second type of original images (c)) are reconstructed to obtain reconstructed light slice images (the first type of output images (b) and the second type of output images (d)). This embodiment can remove stripes without losing the original content, and the restored stripe area corresponds well to the original content of another phase corresponding area;

[0056] This embodiment adopts a self-supervised two-way learning deep learning model training method, so that the reconstruction of structured light illumination light section microscopy images based on deep learning can be independent of the truth prior of the traditional image reconstruction algorithm; by combining mask loss and image content loss, the deep learning model is able to perceive clean image content as the truth value from images with different phase stripes; compared with the existing deep learning network training method that uses the traditional image reconstruction algorithm as the truth value, this embodiment has a higher network capacity upper limit, avoids the problem of unsatisfactory training truth value that may be caused by the traditional image reconstruction algorithm, and is universal and robust in various complex grid structured light illumination light section microscopy image reconstruction tasks.

[0057] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A structured light microscopy image reconstruction method based on deep learning, characterized in that: The acquired structured light illumination light slice image is transmitted to the trained deep learning model to output the reconstructed light slice image; the training process of the deep learning model is as follows: Step 1: using grid-illuminated structured light to project stripes on the same observed structured light illumination light slice image target to obtain a phase image pair, wherein the phase image pair includes a first type of original image and a second type of original image, and the first type of original image and the second type of original image are images with different stripe phases; Step 2: masking the stripe area contents of the first type of original image and the second type of original image in the phase image pair to obtain a first type of mask image and a second type of mask image; Step 3: input the first type of original image, the second type of original image, the first type of mask image, and the second type of mask image into the deep learning model to obtain the first type of output image and the second type of output image; Step 4: Multiply the second type of output image by the first type of mask image as the input of the loss function, and multiply the first type of original image by the first type of mask image as the input of the loss function to calculate the first type of mask loss; multiply the first type of output image by the second type of mask image as the input of the loss function, and multiply the second type of original image by the second type of mask image as the input of the loss function to calculate the second type of mask loss; Step 5: Multiply the first category original image and the first category mask image as the input of the loss function, and multiply the first category mask image and the first category output image as the input of the loss function to calculate the first category image content loss; multiply the second category original image and the second category mask image as the input of the loss function, and multiply the second category mask image and the second category output image as the input of the loss function to calculate the second category image content loss; Step 6: Add the first-class mask loss, the second-class mask loss, the first-class image content loss, and the second-class image content loss in sequence to obtain the total loss, and update and adjust the parameters of the deep learning model based on the total loss.

2. The structured light microscopy image reconstruction method based on deep learning according to claim 1, characterized in that: In step one, the stripes of the same content in the first type of original image and the second type of original image are complementary in brightness and darkness.

3. The structured light microscopy image reconstruction method based on deep learning according to claim 1, characterized in that: The trained deep learning model is verified as follows: Construct a validation data set in the same way as in step 1, calculate the validation total loss value in the same way as in steps 2 to 6, and determine whether the validation total loss value has a downward trend within a certain period of time; If yes, continue training the deep learning model; If not, the deep learning model training is complete.

4. A structured light microscopy image reconstruction system based on deep learning, characterized in that: The acquired structured light illumination light slice image is transmitted to the trained deep learning model to output the reconstructed light slice image; the training process of the deep learning model is as follows: Step 1: using grid-illuminated structured light to project stripes on the same observed structured light illumination light slice image target to obtain a phase image pair, wherein the phase image pair includes a first type of original image and a second type of original image, and the first type of original image and the second type of original image are images with different stripe phases; Step 2: masking the stripe area contents of the first type of original image and the second type of original image in the phase image pair to obtain a first type of mask image and a second type of mask image; Step 3: input the first type of original image, the second type of original image, the first type of mask image, and the second type of mask image into the deep learning model to obtain the first type of output image and the second type of output image; Step 4: Multiply the second type of output image by the first type of mask image as the input of the loss function, and multiply the first type of original image by the first type of mask image as the input of the loss function to calculate the first type of mask loss; multiply the first type of output image by the second type of mask image as the input of the loss function, and multiply the second type of original image by the second type of mask image as the input of the loss function to calculate the second type of mask loss; Step 5: Multiply the first category original image and the first category mask image as the input of the loss function, and multiply the first category mask image and the first category output image as the input of the loss function to calculate the first category image content loss; multiply the second category original image and the second category mask image as the input of the loss function, and multiply the second category mask image and the second category output image as the input of the loss function to calculate the second category image content loss; Step 6: Add the first-class mask loss, the second-class mask loss, the first-class image content loss, and the second-class image content loss in sequence to obtain the total loss, and update and adjust the parameters of the deep learning model based on the total loss.

5. The structured light microscopy image reconstruction system based on deep learning according to claim 4, characterized in that: In step one, the stripes of the same content in the first type of original image and the second type of original image are complementary in brightness and darkness.

Images