Confocal microscopic image super-resolution reconstruction method based on object plane constraint

Abstract

The invention discloses a confocal microscopic image super-resolution reconstruction method based on object plane constraint, and the method comprises the steps: obtaining a fluorescence and transmission data stack through a confocal microscopic system, carrying out the deconvolution of the fluorescence data stack, and carrying out the cell segmentation through an improved watershed algorithm, and obtaining a three-dimensional cell contour. After a phase transfer function of transmission data is calculated in advance, deconvolution processing is carried out on the transmission data, and object refractive index distribution is inverted from light intensity data. And performing super-resolution operation on the reconstructed refractive index distribution through a GP iterative algorithm by taking the three-dimensional cell contour as an object plane constraint to obtain super-resolution reconstructed object refractive index distribution. According to the invention, the problem of'cone missing 'caused by the limitation of the aperture of the objective lens is solved, the axial resolution of the image is obviously improved, and an innovative solution is provided for the development of the microscopic imaging technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical microscopic imaging, and in particular relates to a confocal microscopic image super-resolution reconstruction method based on object plane constraint. Background Art

[0002] Optical microscopy plays a vital role in biological, material science, and medical research. Despite this, conventional optical microscopes have some limitations in three-dimensional imaging, especially in resolution and imaging speed. Due to the numerical aperture limitation of the objective lens, the "missing cone" problem will occur, which is also the main problem of optical diffraction tomography. This will cause the reconstructed three-dimensional refractive index to be underestimated, and the image will be elongated along the axial direction, further reducing the image resolution. Optical diffraction tomography (ODT) usually requires the combination of phase measurement technology and computed tomography technology to obtain multiple sets of quantitative phase information by rotating the object or changing the illumination direction, and then reconstruct the three-dimensional spatial refractive index distribution of the object in combination with diffraction tomography theory. Its accurate understanding of the sample structure will be seriously affected by various factors.

[0003] Because the light source of traditional wide-field microscopes uniformly illuminates the entire sample, the objective lens collects light signals from the focal plane and out-of-focus areas simultaneously, resulting in poor axial resolution. Especially when imaging thick samples, out-of-focus light will be superimposed on the focal plane image, resulting in background blur and reduced signal-to-noise ratio. In fluorescence imaging, the wide-field illumination of wide-field microscopes will simultaneously excite fluorescent molecules in the entire area of ​​the sample, resulting in accelerated photobleaching, and the emission light of different fluorescence channels may produce crosstalk due to the overlap of out-of-focus signals. The limitations of wide-field microscopes also have a significant impact on cell segmentation. Since out-of-focus light cannot be effectively removed, the image background is blurred and the cell boundaries are not clear (especially in thick samples). The algorithm may mistakenly identify out-of-focus signals as real cell structures. The uniform excitation of out-of-focus light and wide-field illumination will increase background noise, and weak signal areas (such as cell processes) may be masked by noise, resulting in missed segmentation. During three-dimensional segmentation, the axial resolution of wide-field microscopes is low, and the Z-axis stack image is blurred, making it difficult to provide clear three-dimensional structural information. Summary of the Invention

[0004] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a confocal microscopy image super-resolution reconstruction method based on object plane constraints, which solves the "missing cone" problem caused by the numerical aperture limitation of the objective lens, improves the axial resolution of the image, and uses a confocal microscopy system to solve the problems of low axial resolution of traditional wide-field microscopes and the inability to obtain accurate support domain constraint range during cell segmentation.

[0005] The technical solution for achieving the purpose of the present invention is: a confocal microscopy image super-resolution reconstruction method based on object plane constraint, the specific steps are:

[0006] Step 1: Stain the cells to be tested;

[0007] Step 2: Acquire a 3D fluorescence data stack and a 3D transmission data stack using a confocal microscope system;

[0008] Step 3: Deconvolve the fluorescence data and perform cell segmentation to obtain the range of support domain constraints;

[0009] Step 4: Reconstruct the refractive index distribution of the transmission data based on the deconvolution algorithm;

[0010] Step 5: Perform super-resolution calculation on the sample refractive index distribution based on the GP iterative algorithm;

[0011] Step 6: Obtain the super-resolution reconstructed three-dimensional refractive index distribution of the sample.

[0012] Compared with the existing technology, the present invention has the following significant advantages: (1) It utilizes confocal microscopy to effectively block information outside the focal plane, and applies the fluorescence data stack to cell segmentation, significantly improving the accuracy of the cell segmentation results; (2) It uses a modified watershed algorithm for cell segmentation, which does not require the large data sets and computing resources required by deep learning-based methods. The calculation method is simple and more efficient when processing simple scenes such as confocal fluorescence images. (3) It uses a modified GP iterative super-resolution method to solve the "missing cone" problem caused by the limitation of the numerical aperture of the objective lens and the problem of underestimation of the reconstructed three-dimensional refractive index, significantly improving the axial resolution of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Flowchart of the present invention.

[0014] Figure 2 This is a flow chart of the three-dimensional cell segmentation algorithm of the present invention.

[0015] Figure 3 The results of super-resolution 3D refractive index reconstruction of a cell-simulating sphere are shown. (a1) is the original cell xz plane image; (b1) is the inverted refractive index image of the original cell after deconvolution; (c1) is the image of the refractive index distribution reconstructed using the super-resolution method of the present invention; (a2)-(c2) are the corresponding frequency domain images.

[0016] Figure 4Results of a cell confocal transmission imaging experiment. (a) Images of three planes (xy, xz, and yz) of three-dimensional transmission cell data collected by a confocal microscope; (b) images of three planes (xy, xz, and yz) obtained by deconvolution of the confocal transmission image; and (c) images of three planes (xy, xz, and yz) of the three-dimensional refractive index distribution image obtained by super-resolution reconstruction using the present invention. DETAILED DESCRIPTION

[0017] To facilitate understanding and implementation of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0018] A super-resolution reconstruction method for confocal microscopy images based on object plane constraints uses a confocal microscope system to acquire fluorescence and transmission data stacks. After deconvolution of the fluorescence data stack, cells are segmented using a modified watershed algorithm to obtain three-dimensional cell outlines. The transmission data's phase transfer function is precalculated and then deconvolved, allowing the object's refractive index distribution to be inferred from the intensity data. Using the three-dimensional cell outline as an object plane constraint, the reconstructed refractive index distribution is super-resolved using a GP iterative algorithm, resulting in a super-resolution reconstructed object refractive index distribution.

[0019] As an embodiment, the specific steps of a confocal microscopy image super-resolution reconstruction method based on object plane constraint are described in combination with simulation:

[0020] Step 1: stain the cells to be tested. The specific steps are as follows:

[0021] The staining working solution used in the present invention is a cell membrane green fluorescence staining kit (DIO), which presents green fluorescence, is very stable after staining, has very low toxicity, and is one of the most commonly used cell membrane fluorescent probes. The first is the preparation of the working solution. According to the number of samples and the volume of the working solution required for each sample, the total volume of the staining working solution is calculated, and the cell membrane staining working solution is prepared. The adherent cells are then stained. The adherent cells are planted on a cell culture dish, the culture solution is removed by suction, and the cells are washed 2-3 times with a buffer solution. An appropriate staining working solution is added, and the cells are incubated in the dark at 37°C for 5-20 minutes. The working solution is sucked out and rinsed, and a 37°C preheated culture medium is added, and the cells can be observed under a fluorescence microscope.

[0022] Step 2: Collect three-dimensional fluorescence data stacks and three-dimensional transmission data stacks using a confocal microscope system. The specific steps are as follows:

[0023] The imaging system of a confocal microscope system includes a laser light source, a beam splitter, a galvanometer, an objective lens, a pinhole, and a detector. The laser light emitted by the laser light source is focused onto a tiny point on the sample through the beam splitter, galvanometer, and objective lens, stimulating the sample to produce a fluorescence signal. The fluorescence signal returns along the same path as the laser light, is separated from the laser light at the beam splitter, and passes through a pinhole conjugate with the light source to the detector, ensuring that only the signal light from the focal plane can be received by the detector while the out-of-focus scattered light is blocked. The galvanometer scanning ultimately produces a three-dimensional fluorescence data stack. The transmission portion of the confocal microscope system is a partially coherent system, using a second objective lens behind the sample to collect the transmitted light and focus it onto the confocal pinhole and detector to produce a three-dimensional transmission data stack.

[0024] Step 3: Deconvolve the 3D fluorescence data stack and perform cell segmentation to obtain the range of the support domain constraint. The specific steps are as follows:

[0025] The optical transfer function of a confocal 3D fluorescence data stack can be calculated by first simulating a circle with a radius equal to the cutoff frequency of the objective lens:

[0026]

[0027] Among them, (F x ,F y ) is the frequency domain coordinate, f o is the objective lens cutoff frequency.

[0028] Calculate the transmission function of P at each position along the propagation direction to simulate the propagation process of light waves through free space:

[0029]

[0030] Performing a z-axis Fourier transform on the transfer function yields the scattered wave on the Ewald sphere, which is in the shape of a three-dimensional spherical shell. Performing an autocorrelation operation on the three-dimensional spherical shell yields the confocal coherent transfer function:

[0031]

[0032] Then, the confocal optical transfer function can be obtained by performing autocorrelation operation on the coherent transfer function:

[0033]

[0034] Deconvolution of confocal fluorescence images using the optical transfer function:

[0035]

[0036] in, is the Fourier transform of the fluorescence image, and β is the Tihkonv regularization parameter.

[0037] Perform cell segmentation on the deconvolution image. The specific method is as follows:

[0038] To address the impact of artifacts on cell segmentation, image preprocessing is required. First, the image is globally thresholded to remove low-intensity noise. Then, by calculating the volume of each structure, structures with volumes smaller than V1 and larger than V2 are removed based on two thresholds, V1 and V2, to remove large-volume artifacts. The preprocessed image is binarized, and the complement of the resulting three-dimensional image is calculated to create a background marker. Foreground markers are calculated by identifying the regional maximum of the original three-dimensional image masked by the background marker. However, the presence of small local maxima can lead to overdetection of foreground markers. To eliminate false foreground markers, the image is subjected to a three-dimensional Gaussian filter before identifying the regional maximum to reduce false local maxima. Morphological processing is also performed on the image to identify the region containing the foreground marker. Finally, the regional maximum is identified to calculate the foreground marker.

[0039] The images after foreground and background marking are segmented by the watershed algorithm. If there are more foreground markers than cells, too many three-dimensional volumes will be marked as cells, resulting in cell fragmentation. To solve this problem, the segmented cell images are processed by the merging algorithm to finally obtain the range of the support domain constraint.

[0040] Step 4: Reconstruct the refractive index distribution of the transmission data based on the deconvolution algorithm. The specific steps are as follows:

[0041] In order to separate the contributions of the sample and the imaging system, the component related to the imaging system factors, namely the cross transfer function (TCC), is calculated separately:

[0042] TCC(u1,u2)=∫S(u)H(u+u1)H * (u+u2)du

[0043] After the cross transfer function TCC is calculated, the deconvolution operation is performed on the three-dimensional transmission data stack:

[0044]

[0045] in, is the Fourier transform of the intensity distribution I(u) of the three-dimensional object, * represents the complex conjugate, and β is the Tihkonv regularization parameter in deconvolution.

[0046] The frequency domain data obtained after deconvolution Perform an inverse Fourier transform:

[0047]

[0048] Among them, r=(x,y,z) is the spatial coordinate, u=(k x ,k y ,k z ) is the frequency domain coordinate, and f(r) is the object scattering potential distribution.

[0049] Step 5: Perform super-resolution calculation on the sample refractive index distribution based on the GP iterative algorithm. The specific steps are as follows:

[0050] Step 5.1: Perform a three-dimensional Fourier transform on the scattering potential distribution of the object to obtain a frequency domain image, specifically:

[0051]

[0052] Among them, (x, y, z) are spatial coordinates, (k x ,k y ,k z ) is the frequency domain coordinate, G1(k x ,k y ,k z ) represents the frequency domain image of the first iteration.

[0053] Step 5.2: Replace the phase transfer function mask portion of the spectrum graph. Since this is the first iterative operation, both the masked portion and the portion outside the mask use the frequency domain image of this operation. However, in each subsequent iteration, the masked portion of the frequency domain image of each iteration is replaced with the frequency domain image of this operation. The specific operation process will be explained in step 5.6.

[0054] G2(k x ,k y ,k z )=G1(k x ,k y ,k z )*P(k x ,k y ,k z )+G1(k x ,k y ,k z )*(1-P(k x ,k y ,k z ))

[0055] Among them, P(k x ,k y ,k z ) is the phase transfer function mask, specifically:

[0056]

[0057] Among them, Im(H P) is the imaginary part of the phase optical transfer function POTF.

[0058] Step 5.3: Then perform inverse Fourier transform on the frequency domain image to obtain the spatial domain image, specifically:

[0059]

[0060] Step 5.4: Add the range of the support region constraint obtained in step 2 as an object plane constraint to the spatial domain image:

[0061] g3(x,y,z)={g2(x,y,z)-n m}*S(x,y,z)+n m

[0062] Among them, n m is the medium refractive index, and S(x,y,z) is the object surface constraint.

[0063] Step 5.5: Perform a 3D Fourier transform on the spatial domain image to obtain the frequency domain image. The calculation process is the same as step 5.1, specifically:

[0064]

[0065] Step 5.6: Replace the phase transfer function mask portion of the frequency domain image of this iterative operation, specifically:

[0066] G4(k x ,k y ,k z )=G1(k x ,k y ,k z )*P(k x ,k y ,k z )+G3(k x ,k y ,k z )*(1-P(k x ,k y ,k z ))

[0067] The iterative process from steps 5.3 to 5.6 is then repeated, gradually approximating the high-resolution image until a preset number of iterations are reached, resulting in the target super-resolved sample refractive index distribution. This can significantly improve image quality, especially when processing biomedical images with complex geometric structures.

Claims

1. A confocal microscopy image super-resolution reconstruction method based on object plane constraint, characterized in that: The specific steps are: Step 1: Stain the cells to be tested; Step 2: Acquire a 3D fluorescence data stack and a 3D transmission data stack using a confocal microscope system; Step 3: Deconvolve the fluorescence data and perform cell segmentation to obtain the range of support domain constraints; Step 4: Reconstruct the refractive index distribution of the transmission data based on the deconvolution algorithm; Step 5: Perform super-resolution calculation on the sample refractive index distribution based on the GP iterative algorithm; Step 6: Obtain the super-resolution reconstructed three-dimensional refractive index distribution of the sample.

2. The method for super-resolution reconstruction of confocal microscopic images based on object plane constraint according to claim 1, characterized in that: The specific method for staining cells is as follows: According to the number of samples and the volume of working solution required for each sample, calculate the total volume of the staining working solution and prepare the cell membrane staining working solution; Seed the adherent cells on a cell culture dish, remove the culture medium, wash with buffer 2-3 times, add the staining working solution, incubate at 37℃ in the dark for 5-20 minutes, remove the working solution and rinse, and add the culture medium preheated at 37℃.

3. The method for super-resolution reconstruction of confocal microscopic images based on object plane constraint according to claim 1, characterized in that: The specific process of collecting two sets of three-dimensional data stacks, fluorescence and transmission, using a confocal microscope system is as follows: The laser light emitted by the laser light source is focused onto the sample through a beam splitter, a galvanometer and an objective lens to excite the sample and produce a fluorescence signal. The fluorescence signal returns along the same path as the laser, is separated from the laser at the beam splitter, and passes through a pinhole conjugate with the light source to the detector, and is scanned by the galvanometer to finally obtain a three-dimensional fluorescence data stack; the transmission part of the confocal microscope system is a partially coherent system. A second objective lens is used behind the sample to collect the transmitted light and focus it onto the confocal pinhole and detector to obtain a three-dimensional transmission data stack.

4. The method for super-resolution reconstruction of confocal microscopic images based on object plane constraint according to claim 1, characterized in that: The specific method for performing cell segmentation after deconvolution of the three-dimensional fluorescence data stack to obtain the range of the support domain constraint is as follows: Deconvolution of confocal three-dimensional fluorescence data stacks using optical transfer functions; Perform global threshold processing on the image to remove low-intensity noise, then calculate the volume of each structure and remove structures with a volume smaller than V1 and a volume larger than V2 according to the two set thresholds V1 and V2. Binarize the preprocessed image and calculate the complement of the binarized three-dimensional image to create a background mark; Perform 3D Gaussian filtering and morphological processing on the binarized 3D image to obtain the area where the foreground mark is located, and identify the maximum value of the area to calculate the foreground mark; The images after foreground and background marking are segmented by watershed algorithm. If there are more foreground markings than cells, the segmented cell images are processed by merging algorithm to finally obtain the range of support domain constraint.

5. The method for super-resolution reconstruction of confocal microscopic images based on object plane constraint according to claim 4, characterized in that: The specific formula for deconvolution of confocal three-dimensional fluorescence data stacks using the optical transfer function is: Where, is the Fourier transform of the fluorescence image, β is the Tychonoff regularization parameter, OTF is the optical transfer function, and * represents the complex conjugate.

6. The method for super-resolution reconstruction of confocal microscopic images based on object plane constraint according to claim 4, characterized in that: The optical transfer function is specifically: Where, (F x ,F y ) is the frequency domain coordinate, f o is the objective lens cutoff frequency, z is the z-axis spatial coordinate, and λ is the wavelength.

7. The method for super-resolution reconstruction of confocal microscopic images based on object plane constraint according to claim 1, characterized in that: The specific method for reconstructing the refractive index distribution of transmission data based on the deconvolution algorithm is: Calculate the components related to imaging system factors, namely the cross transfer function TCC: TCC(u1,u2)=∫S(u)H(u+u1)H * (u+u2)du Where u is the frequency domain coordinate, S(u) is the light intensity distribution of the light source, u1 and u2 are two spatial frequencies, and H(u+u1)H * (u+u2) is the mutual interference transfer function; Deconvolve a 3D stack of transmission data: in, is the Fourier transform of the intensity distribution I(u) of the three-dimensional object, * represents the complex conjugate, and β is the Tikhonov regularization parameter in deconvolution; The frequency domain data obtained after deconvolution Perform inverse Fourier transform operation to obtain the object scattering potential distribution f(r). The specific operation process is: Among them, r=(x,y,z) is the spatial coordinate, u=(k x ,k y ,k z ) is the frequency domain coordinate, and f(r) is the object scattering potential distribution.

8. The method for super-resolution reconstruction of confocal microscopic images based on object plane constraint according to claim 1, characterized in that: The specific method for super-resolution calculation of sample refractive index distribution based on GP iterative algorithm is as follows: Step 5.1: Perform a three-dimensional Fourier transform on the scattering potential distribution of the object to obtain a frequency domain image, specifically: Among them, (x, y, z) are spatial coordinates, (k x ,k y ,k z ) is the frequency domain coordinate, G1(k x ,k y ,k z ) represents the frequency domain image of the first iteration; Step 5.2: Replace the phase transfer function mask portion of the spectrum graph as follows: G2(k x ,k y ,k z )=G1(k x ,k y ,k z )*P(k x ,k y ,k z )+G1(k x ,k y ,k z )*(1-P(k x ,k y ,k z )) Among them, P(k x ,k y ,k z ) is the phase transfer function mask, specifically: Among them, Im(H P ) is the imaginary part of the phase optical transfer function; Step 5.3: Perform inverse Fourier transform on the frequency domain image to obtain the spatial domain image, specifically: Initially, n = 2; Step 5.4: Add the range of the support region constraint obtained in step 2 as an object plane constraint to the spatial domain image: g n+1 (x,y,z)={g n (x,y,z)-n m }*S(x,y,z)+n m Among them, n m is the medium refractive index, S(x,y,z) is the object plane constraint; Step 5.5: Perform a three-dimensional Fourier transform on the spatial domain image to obtain a frequency domain image, specifically: Step 5.6: Replace the phase transfer function mask portion of the frequency domain image of this iterative operation, specifically: G n+2 (k x ,k y ,k z )=G n-1 (k x ,k y ,k z )*P(k x ,k y ,k z )+G n+1 (k x ,k y ,k z )*(1-P(k x ,k y ,k z )) Let n = n + 1; repeat the iterative calculation process of step 5.3 to step 5.6, gradually approximating the high-resolution image, until the preset number of iterations is reached, and the target super-resolution sample refractive index distribution is obtained.

Images