Super-resolution microscopic imaging method, device, computer device, and storage medium

Abstract

The present invention discloses a super-resolution microscopic imaging method, device, computer device and storage medium. The method includes: collecting a fluorescent signal sequence of a sample to be observed; determining the number of deconvolution iterations; performing a pre-deconvolution iteration on an initial image of each frame in the fluorescent signal sequence, and outputting when the iterations reach half the number of iterations; reconstructing the output image twice; performing a second deconvolution iteration on the twice-reconstructed image, and outputting when the iterations reach the number of iterations or half the number of iterations. The present invention has the advantages of non-parametric, high throughput and high resolution, can overcome the shortcomings and deficiencies of the prior art, and can be widely combined with various imaging modes such as acoustic microscopy, i.e. photoacoustic and ultrasonic microscopic imaging technology.

Description

[Technical field]

[0001] The present invention relates to a super-resolution microscopic imaging method, apparatus, computer device and storage medium, which belong to the fields of computational imaging and super-resolution microscopic imaging. [Background technology]

[0002] The fluorescence super-resolution method SOFI (Super-resolution optical fluctuation imaging) based on the molecular intensity signal fluctuation model is a flexible and cost-effective super-resolution method that utilizes only a physical model of random fluctuations in fluorescent molecular intensity and does not rely on any hardware modulation. Due to its characteristics that are not limited by hardware systems, it can be flexibly coupled to various imaging modes. The disadvantage of such an apparatus is that the time resolution of the current fluorescence super-resolution method is low, and at least 500 to 1000 frames of images must be collected continuously to achieve the expected high-quality super-resolution effect, which inhibits its application to super-resolution imaging of live cells. Therefore, it is necessary to provide a method that makes maximum use of the fluorescence wave behavior detectable per measurement to achieve the desired high time resolution and high throughput. Summary of the Invention [Problem to be solved by the invention]

[0003] Therefore, the present invention provides a super-resolution microscopic imaging method, apparatus, computer device and storage medium that have the advantages of non-parametric, high throughput and high resolution, and can overcome the drawbacks and shortcomings of the prior art.

[0004] A first object of the present invention is to provide a method for super-resolution microscopic imaging.

[0005] A second object of the present invention is to provide a super-resolution microscopic imaging device.

[0006] A third object of the present invention is to provide a computer device.

[0007] A fourth object of the present invention is to provide a storage medium. [Means for solving the problem]

[0008] The first object of the present invention can be achieved by employing the following technical means. 1. A method for super-resolution microscopic imaging, comprising: Acquiring a sequence of fluorescent signals of a sample to be observed; Determining a number of deconvolution iterations; performing a pre-deconvolution iteration on an initial image of each frame in the fluorescence signal sequence, and outputting when the iteration reaches half the number of iterations; reconstructing the output image twice; and performing a second deconvolution iteration on the twice reconstructed image and providing an output when the iterations reach the number of iterations or half the number of iterations.

[0009] In one embodiment, after collecting the series of samples to be observed, the method further includes estimating the background of the fluorescent signal using a wavelet transform to remove background noise.

[0010] In one embodiment, the method of estimating the background of the fluorescent signal and removing the background noise using the wavelet transform specifically includes: estimating a background from the lowest frequencies of the input image using wavelet estimation; blending these two images by inverse wavelet transforming the lowest band into the spatial domain, comparing the result with half the square root of the input image, and keeping the minimum value for each pixel; and looping wavelet estimation using the estimated low-band low-peak background data as a new input image until a preset number of cycles is reached.

[0011] In one embodiment, determining the number of iterations of the deconvolution specifically includes: The method includes using a rolling Fourier ring correlation to determine the number of iterations of the deconvolution when the maximum rolling Fourier ring correlation resolution is reached.

[0012] In one embodiment, the number of iterations of the deconvolution is determined by using the rolling Fourier ring correlation and the number of iterations when the maximum rolling Fourier ring correlation resolution is reached. Specifically, defining a Rolling Fourier Ring correlation as a function of spatial frequency and a spatial frequency discretization of the Rolling Fourier Ring correlation curve to calculate corresponding discrete spatial frequency values; smoothing the noisy rolling Fourier ring correlation curve using an averaging window half-width averaging filter; When the rolling Fourier correlation curve falls below a predetermined threshold value representing the maximum spatial frequency of significant information outside random noise, the frequency is defined as an effective cutoff frequency, and the resolution is defined as the reciprocal of the effective cutoff frequency; The number of iterations when the maximum resolution is reached is set as the number of iterations of the deconvolution.

[0013] In one embodiment, the reconstructing the output image twice specifically includes: Using the fluctuation principle of the fluorescent signal, a first reconstruction of the output image is performed; and applying a joint sparsity-continuity constraint to perform a second reconstruction on the image after the first reconstruction.

[0014] In one embodiment, the first reconstruction of the output image using the fluctuation principle of the fluorescent signal is specifically performed by: obtaining an expression for the fluorescence signal based on a function of the microscope point spread function, the fluorophore brightness constant, and the time-varying variation of the fluorophore brightness; Obtaining a zero time delay time accumulation expression based on the expression of the fluorescent signal using the individual wave characteristics of each fluorescent molecule; The method includes expanding an expression for the time accumulated amount, and if a preset condition is satisfied, regarding the cross-correlation term in the time accumulated amount expansion expression as zero, and expressing the time accumulated amount as a sum of squares of the corresponding brightness constant-weighted point spread function.

[0015] In one embodiment, applying the joint sparsity-continuity constraint to perform a second reconstruction on the image after the first reconstruction can be specifically performed by: constructing a reconstruction model including a first term that is a fidelity term representing a distance between the first reconstructed image and the initial image collected, a second term that represents a continuity constraint on the first reconstructed image, and a third term that represents a sparsity constraint on the first reconstructed image; and performing a second reconstruction on the image after the first reconstruction using the reconstruction model.

[0016] In one embodiment, the deconvolution iterations specifically include: obtaining an iterative formula in a spatial domain iterative maximum likelihood manner; and implementing the iterative calculation of the deconvolution using an acceleration method based on vector extrapolation based on the iterative formula.

[0017] The second object of the present invention can be achieved by employing the following technical means. A super-resolution microscopic imaging device, comprising: an acquisition module for acquiring a sequence of fluorescent signals of a sample to be observed; a determining module for determining the number of iterations of the deconvolution; a pre-deconvolution module for performing a pre-deconvolution iteration on an initial image of each frame in the fluorescence signal sequence and for outputting when the iterations reach half the number of iterations; a reconstruction module for reconstructing the output image twice; a deconvolution module for performing a second deconvolution iteration on the twice reconstructed image and outputting when the iterations reach the iteration number or half the iteration number.

[0018] The third object of the present invention can be achieved by employing the following technical means. A computer device including a processor and a memory for storing a program executable by the processor, The present invention is characterized in that the above-mentioned super-resolution microscopic imaging method is realized when the processor executes a program stored in a memory.

[0019] The fourth object of the present invention can be achieved by employing the following technical means. A storage medium on which a program is stored, When the program is executed by a processor, the above-mentioned super-resolution microscopic imaging method is realized. Effect of the Invention

[0020] The present invention has the following beneficial effects over the prior art: The present invention has the advantage of being highly flexible, and can be widely combined with various imaging modes such as acoustic microscopy, i.e. photoacoustic and ultrasonic microscopy imaging techniques. It has the advantage of being high throughput, and one image obtained by using deconvolution after multiple iterations can be further processed to achieve a 2-fold improvement in three-dimensional spatial resolution. It can achieve high-quality super-resolution effect in 20 frames or less while maintaining super-resolution, and achieve a 50-100-fold improvement in temporal resolution. It has the advantage of being non-parametric, and proposes an automatic parameter estimation means based on rolling Fourier ring correlation to achieve non-parametric, thereby realizing automatic and high-throughput super-resolution imaging. [Brief description of the drawings]

[0021] In order to more clearly describe the technical aspects in the embodiments of the present invention or the prior art, the following briefly describes the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on the structures shown in these drawings without paying creative labor.

[0022] [Figure 1] 1 is a flowchart of a super-resolution microscopic imaging method according to Example 1 of the present invention. [Diagram 2] FIG. 2 is an exemplary result diagram of 3D spatial resolution enhancement according to the first embodiment of the present invention. [Diagram 3] FIG. 4 is an exemplary result diagram of the temporal resolution improvement according to the first embodiment of the present invention. [Figure 4] FIG. 13 is an exemplary result diagram of non-parametric reconstruction capability according to the first embodiment of the present invention. [Diagram 5] FIG. 2 is an exemplary result diagram of automatic high-throughput reconstruction according to the first embodiment of the present invention. [Figure 6] FIG. 11 is a configuration block diagram of a super-resolution microscopic imaging device according to a second embodiment of the present invention. [Figure 7] FIG. 11 is a configuration block diagram of a computer device according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] In order to make the objectives, technical means and advantages of the embodiments of the present invention clearer, the technical means in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative labor belong to the scope of protection of the present invention.

[0024] Example 1 As shown in FIG. 1, this embodiment provides a super-resolution microscopic imaging method realized based on enhanced deconvolution, which includes the following steps:

[0025] In S101, a fluorescent signal sequence of a sample to be observed is collected.

[0026] In one embodiment, the number of frames in the fluorescent signal sequence is 20. It will be understood by those skilled in the art that the number of frames in the fluorescent signal sequence may be 10, 50, etc.

[0027] In one embodiment, for data with weak or no background, the background estimation operation can be omitted, i.e., the background parameter value b is set to zero to avoid removing information. In particular, considering that images under low dose illumination generally only exhibit a low and stable background fluorescence-based noise distribution, values ​​above the image mean are directly set to zero, and the resulting residual image is used for subsequent background estimation.

[0028] In one embodiment, under conditions other than those mentioned above, after step S101, the following may be further included. In S102, the background of the fluorescent signal is estimated using wavelet transform, and background noise is removed.

[0029] In one embodiment, step S102 specifically includes:

[0030] In S1021, the background is estimated from the lowest frequency of the input image using wavelet estimation.

[0031] In one embodiment, a two-dimensional Daubechies-6 wavelet filter is used to multi-decompose the signal into seven stages to extract the lowest band in the frequency domain.

[0032] In S1022, the two images are blended by transforming the lowest band into the spatial domain using an inverse wavelet transform, comparing the result with half the square root of the input image, and keeping the minimum value for each pixel.

[0033] In one embodiment, to prevent the unexpected removal of small useful signals, the two images are combined by inverse wavelet transforming the lowest band into the spatial domain, comparing the result with half the square root of the input image (smoother information), and keeping the minimum value for each pixel. This operation removes high intensity pixels in the background due to inaccurate background estimation.

[0034] In S1023, the estimated low-band low-peak background data is used as a new input image to loop wavelet estimation, i.e., steps S1021 and S1022 are repeated until a preset number of cycles is reached.

[0035] In one embodiment, the preset number of cycles is set to three in order to estimate the true fluorescent background with a minimum distribution.

[0036] In S102, the number of iterations of the deconvolution is determined.

[0037] In one embodiment, determining the number of deconvolution iterations specifically includes using a rolling Fourier ring correlation, and determining the number of iterations when a maximum rolling Fourier ring correlation resolution is reached as the number of deconvolution iterations.

[0038] In one embodiment, the number of iterations of deconvolution is determined by using a rolling Fourier ring correlation (FRC), and the number of iterations when the maximum rolling Fourier ring correlation resolution is reached is the number of iterations of deconvolution, which specifically includes the following:

[0039] In S1021, the rolling Fourier ring correlation is taken as a function of spatial frequency, and a discretization of the spatial frequency of the rolling Fourier ring correlation curve is defined to calculate the corresponding discrete values ​​of the spatial frequency.

[0040] In one embodiment, the rolling Fourier ring correlation measures the statistical correlation between two two-dimensional signals over a series of concentric rings in the Fourier domain. The rolling Fourier ring correlation is measured at spatial frequency q i can be considered as a function of

number

number

number

[0041] Define the spatial frequency discretization of the rolling Fourier correlation curve to calculate the corresponding discrete spatial frequency values, where the maximum frequency fmax is the pixel size p s corresponds to half the reciprocal of fmax=1 / (2p s ) The rolling Fourier correlation curve consists of N / 2 values, and the discretization step length of the spatial frequency, Δf, is as follows:

number

[0042] At S1022, the noisy rolling Fourier ring correlation curve is smoothed using an averaging window half-width averaging filter.

[0043] In one embodiment, an averaging filter with an averaging window half-width (usually equal to 3) is used to smooth the noisy rolling Fourier ring correlation curve.

[0044] In S1023, when the rolling Fourier ring correlation curve falls below a predetermined threshold representing the maximum spatial frequency of significant information outside random noise, the frequency is defined as the effective cutoff frequency, and the resolution is the reciprocal of the effective cutoff frequency.

[0045] In one embodiment, when the rolling Fourier ring correlation curve is below a predetermined threshold that represents the maximum spatial frequency of significant information outside random noise, the frequency is defined as the effective cutoff frequency, and the resolution is the inverse of the effective cutoff frequency. Specifically, a common choice of threshold is a fixed value threshold or σ factor curve. The fixed value is usually a hard threshold of 1 / 7, and the criterion of the σ factor curve can be written as follows:

number

number

[0046] In S1024, the number of iterations when the maximum resolution is reached is set as the number of iterations of the deconvolution.

[0047] In one embodiment, the number of iterations at which full resolution is reached is k, where k is the number of iterations of the deconvolution.

[0048] In S103, a pre-deconvolution iteration is performed on the initial image of each frame in the fluorescence signal sequence, and an output is made when the iteration reaches half the number of iterations.

[0049] In one embodiment, the pre-deconvolution iterations are completed when the iterations reach half the number of iterations determined in step S102, and the effective on / off contrast and signal-to-noise ratio of the signal can be improved by performing pre-deconvolution. The deconvolution iterations specifically include:

[0050] In S1031, an iterative formula is obtained in a maximum likelihood manner using iterations in the spatial domain.

[0051] In one embodiment, the deconvolution employs the Richardson-Lucy (RL) algorithm. The deconvolution model is based on a Poisson noise model and uses iterations in the spatial domain to find maximum likelihood, resulting in the following iterative formula:

number

[0052] In S1032, the iterative calculation of deconvolution is realized using an acceleration method based on vector extrapolation according to the iterative formula.

[0053] In one embodiment, the iterative calculation of the deconvolution is realized using an acceleration scheme based on vector extrapolation to accelerate the iterative convergence speed.

number

[0054] In S104, the output image is reconstructed twice.

[0055] In one embodiment, step S104 specifically includes:

[0056] In S1041, the output image is reconstructed for the first time using the fluctuation principle of the fluorescent signal.

[0057] In S10411, an expression for the fluorescent signal is obtained based on a function of the microscope's point spread function, the fluorescent molecule brightness constant, and the time-dependent variation of the fluorescent molecule brightness.

[0058] In one embodiment, the imaging sample is generally considered to consist of N independent fluorescent molecules located at rk. Assuming that the fluorescent molecules have independent molecular intensities that vary over time, the fluorescent signal located at r and time t can be expressed as: F(r,t) = h(rr k )·c k ω k (t) where h, c, and ω represent the point spread function, the molecular brightness constant, and the time-varying variation function of the molecular brightness, respectively, of the corresponding microscope.

[0059] In S10412, a time accumulation expression of zero time delay is obtained based on the expression of the fluorescent signal using the individual wave characteristics of each fluorescent molecule.

[0060] In one embodiment, the time accumulation amount is a second order time accumulation amount, although those skilled in the art can understand that the time accumulation amount may be a third order time accumulation amount or a fourth order time accumulation amount.

[0061] In one embodiment, based on the fluorescence fluctuation super-resolution imaging technology, such individual wave features of each fluorescent molecule are used to calculate the correlation accumulation amount along the t axis for each pixel to improve the resolution, and the zero time delay second-order time accumulation amount G2 is calculated to obtain the following equation: G2(r)=<δF(r,t) δF(r,t)> t δF(r,t)=F(r,t)-<F(r,t)> t where < >t is the time-averaged function.

[0062] In S10413, an expression for the time accumulated amount is expanded, and if a preset condition is satisfied, the cross-correlation term in the time accumulated amount expansion expression is regarded as zero, and the time accumulated amount is expressed as the sum of squares of the corresponding brightness constant weighted point spread function.

[0063] In one embodiment, the expression for the quadratic time accumulation G2 is expanded to obtain the following equation:

number

[0064] The emission intensity of each fluorescent molecule is assumed to be an uncorrelated independent fluctuation. When i ≠ k (a preset condition), the cross-correlation term in the expansion formula is considered to be zero, and the second-order time cumulative quantity G2 is expressed as the sum of squares of the corresponding brightness constant γ-weighted point spread function as shown in the following formula.

number

[0065] In S1042, the joint sparsity-continuity constraint is applied to perform a second reconstruction on the image after the first reconstruction.

[0066] In S10421, a reconstruction model is constructed. The reconstruction model includes a first term, a second term, and a third term. The first term is a fidelity term and represents a distance between an image after a first reconstruction and an initial image acquired. The second term represents a continuity constraint of the image after the first reconstruction. The third term represents a sparsity constraint of the image after the first reconstruction.

[0067] In one embodiment, we construct the reconstruction model as follows:

number

number

number

number

[0068] In S10422, a second reconstruction is performed on the image after the first reconstruction using the reconstruction model.

[0069] In S106, a second deconvolution iteration is performed on the twice reconstructed image, and an output is made when the iteration reaches the iteration number or half the iteration number.

[0070] In one embodiment, the second deconvolution iteration is completed when the iteration reaches half the number of iterations determined in step S102 (i.e., k / 2 times). The pre-deconvolution and second deconvolution iterations are both k / 2 times, which is just the number of iterations determined in step S102. The specific content of the second deconvolution iteration is referred to step S103 and will not be repeated here. By performing two deconvolution iterations (pre-deconvolution iteration and second deconvolution iteration), the resolution can be further improved while maintaining image quality and minimizing artifacts.

[0071] Those skilled in the art can appreciate that the second deconvolution iteration can also be completed when the iteration number determined in step S102 is reached.

[0072] FIG. 2 is an exemplary result showing the 3D spatial resolution improvement according to the embodiment of the present invention, which can achieve a 2x 3D spatial resolution improvement compared to the conventional confocal mode. The method of the embodiment of the present invention is used to measure the 3D PSF of a quantum dot (QD525) sample, and a standard turntable confocal microscope is used to acquire image series to extract the lateral / axial half-widths of such molecular dot sources, respectively. The lateral / axial half-widths of the conventional confocal mode are 325 / 655 nm, the conventional fluorescence fluctuation mode is 295 / 510 nm, and the embodiment of the present invention is 135 / 334 nm.

[0073] FIG. 3 is an exemplary result showing the improvement in temporal resolution according to an embodiment of the present invention (microtubules using quantum dot markers in COS-7 cells). The embodiment of the present invention can achieve high-quality super-resolution effect with only 20 frames while maintaining super-resolution, achieving 50-100 times improvement in temporal resolution over traditional fluorescence fluctuation techniques (SOFI). When the signal-to-noise ratio gradually decreases, the traditional fluorescence fluctuation technique requires reconstruction of 1000 frames of images to maintain relatively stable performance, but the embodiment of the present invention can reconstruct two-point structures with high fidelity using only 20 frames of images under all conditions.

[0074] 4 and 5 are exemplary results showing the non-parametric reconstruction capability and automatic high-throughput reconstruction of the embodiment of the present invention, respectively. The embodiment of the present invention has the advantage of non-parametric and proposes an automatic parameter estimation means based on rolling Fourier ring correlation to realize non-parametric, thereby realizing automatic high-throughput super-resolution imaging (microtubules with quantum dot markers in COS-7 cells). The embodiment images a large field of view of 2.0 mm x 1.4 mm, containing more than 2000 cells, and consisting of about 2,400,000,000 (2.4 billion) pixels (32.5 nm x 32.5 nm per pixel), spanning almost five orders of magnitude of area spatial scale. In the present embodiment, the reconstruction takes only about 10 minutes, while the conventional fluorescence fluctuation technique (SOFI) takes about 17 hours to achieve similar imaging performance.

[0075] Although the method operations of the above-described embodiments have been described in a particular order, this does not require or imply that the operations must be performed in that particular order, or that all operations must be performed in order to achieve the desired results. Conversely, the steps described may be performed in a different order. Additionally or alternatively, some steps may be omitted, multiple steps may be merged into one step, or one step may be broken down into multiple steps.

[0076] Example 2 As shown in FIG. 6, this embodiment provides a super-resolution microscopic imaging device. The device includes an acquisition module 601, a wavelet transform module 602, a judgment module 603, a pre-deconvolution module 604, a reconstruction module 605, and a deconvolution module 606. The specific description of each module is as follows: The acquisition module 601 is used to acquire a fluorescent signal sequence of a sample to be observed. The wavelet transform module 602 is used to estimate the background of the fluorescent signal using wavelet transform and remove background noise. The judgment module 603 is used to judge the number of iterations of deconvolution. The pre-deconvolution module 604 is used to perform pre-deconvolution iterations on the initial image of each frame in the fluorescent signal sequence, and output when the iterations reach half of the number of iterations. The reconstruction module 605 is used to reconstruct the output image twice. The deconvolution module 606 is used to perform a second deconvolution iteration on the twice reconstructed image and provide an output when the iteration reaches the iteration number or half the iteration number.

[0077] In addition, the system provided in this embodiment is simply described by taking the division of each of the above functional modules as an example. In actual application, in order to complete all or part of the above functions, the above functions can be allocated to be completed by different functional modules as necessary, that is, the internal structure can be divided into different functional modules.

[0078] Example 3 This embodiment provides a computer device. The computer device may be a computer, and includes a processor 702, a memory, an input device 703, a display 704, and a network interface 705 connected via a system bus 701, as shown in FIG. 7. The processor is used to provide calculation and control capabilities. The memory includes a non-volatile storage medium 706 and an internal memory 707. The non-volatile storage medium 706 stores an operating system, a computer program, and a database. The internal memory 707 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 702 executes the computer program stored in the memory, the super-resolution microscopic imaging method of the above-mentioned embodiment 1 is realized as follows. A sequence of fluorescent signals of the sample to be observed is collected. The number of deconvolution iterations is determined. A pre-deconvolution iteration is performed on the initial image of each frame in the fluorescence signal sequence, and an output is generated when the iteration reaches half the number of iterations. The output image is reconstructed twice. A second deconvolution iteration is performed on the twice reconstructed image, with an output when the iterations reach the number of iterations or half the number of iterations.

[0079] In one embodiment, after collecting the series of samples to be observed, the method further includes estimating the background of the fluorescent signal using a wavelet transform and removing background noise.

[0080] In one embodiment, the method of estimating the background of the fluorescent signal using the wavelet transform and removing the background noise includes the steps of: estimating a background from the lowest frequencies of the input image using wavelet estimation; blending these two images by inverse wavelet transforming the lowest band into the spatial domain, comparing the result with half the square root of the input image, and keeping the minimum value for each pixel; and looping wavelet estimation using the estimated low-band low-peak background data as a new input image until a preset number of cycles is reached.

[0081] In one embodiment, determining the number of deconvolution iterations specifically includes: The method includes using a rolling Fourier ring correlation to determine the number of iterations of the deconvolution when the maximum rolling Fourier ring correlation resolution is reached.

[0082] In one embodiment, the number of iterations of the deconvolution is determined by using the rolling Fourier ring correlation and the number of iterations when the maximum rolling Fourier ring correlation resolution is reached. Specifically, defining a Rolling Fourier Ring correlation as a function of spatial frequency and a spatial frequency discretization of the Rolling Fourier Ring correlation curve to calculate corresponding discrete spatial frequency values; smoothing the noisy rolling Fourier ring correlation curve using an averaging window half-width averaging filter; When the rolling Fourier correlation curve falls below a predetermined threshold value representing the maximum spatial frequency of significant information outside random noise, the frequency is defined as an effective cutoff frequency, and the resolution is defined as the reciprocal of the effective cutoff frequency; The number of iterations when the maximum resolution is reached is set as the number of iterations of the deconvolution.

[0083] In one embodiment, the reconstructing the output image twice specifically includes: Using the fluctuation principle of the fluorescent signal, a first reconstruction of the output image is performed; and applying a joint sparsity-continuity constraint to perform a second reconstruction on the image after the first reconstruction.

[0084] In one embodiment, the first reconstruction of the output image using the fluctuation principle of the fluorescent signal is specifically obtaining an expression for the fluorescence signal based on a function of the microscope point spread function, the fluorophore brightness constant, and the time-varying variation of the fluorophore brightness; Obtaining a zero time delay time accumulation expression based on the expression of the fluorescent signal using the individual wave characteristics of each fluorescent molecule; The method includes expanding an expression for the time accumulated amount, and if a preset condition is satisfied, regarding the cross-correlation term in the time accumulated amount expansion expression as zero, and expressing the time accumulated amount as a sum of squares of the corresponding brightness constant-weighted point spread function.

[0085] In one embodiment, applying the joint sparsity-continuity constraint to perform a second reconstruction on an image after a first reconstruction can be specifically performed by: constructing a reconstruction model including a first term that is a fidelity term representing a distance between the first reconstructed image and the initial image collected, a second term that represents a continuity constraint on the first reconstructed image, and a third term that represents a sparsity constraint on the first reconstructed image; and performing a second reconstruction on the image after the first reconstruction using the reconstruction model.

[0086] In one embodiment, the deconvolution iterations specifically include: obtaining an iterative formula in a spatial domain iterative maximum likelihood manner; and implementing the iterative calculation of the deconvolution using an acceleration method based on vector extrapolation based on the iterative formula.

[0087] Example 4 This embodiment provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the super-resolution microscopic imaging method of the above embodiment 1 is realized as follows: A sequence of fluorescent signals of the sample to be observed is collected. The number of deconvolution iterations is determined. A pre-deconvolution iteration is performed on the initial image of each frame in the fluorescence signal sequence, and an output is generated when the iteration reaches half the number of iterations. The output image is reconstructed twice. A second deconvolution iteration is performed on the twice reconstructed image, with an output when the iterations reach the number of iterations or half the number of iterations.

[0088] In one embodiment, after collecting the series of samples to be observed, the method further includes estimating the background of the fluorescent signal using a wavelet transform and removing background noise.

[0089] In one embodiment, the method of estimating the background of the fluorescent signal using the wavelet transform and removing the background noise includes the steps of: estimating a background from the lowest frequencies of the input image using wavelet estimation; blending these two images by inverse wavelet transforming the lowest band into the spatial domain, comparing the result with half the square root of the input image, and keeping the minimum value for each pixel; and looping wavelet estimation using the estimated low-band low-peak background data as a new input image until a preset number of cycles is reached.

[0090] In one embodiment, determining the number of deconvolution iterations specifically includes: The method includes using a rolling Fourier ring correlation to determine the number of iterations of the deconvolution when the maximum rolling Fourier ring correlation resolution is reached.

[0091] In one embodiment, the number of iterations of the deconvolution is determined by using the rolling Fourier ring correlation and the number of iterations when the maximum rolling Fourier ring correlation resolution is reached. Specifically, defining a Rolling Fourier Ring correlation as a function of spatial frequency and a spatial frequency discretization of the Rolling Fourier Ring correlation curve to calculate corresponding discrete spatial frequency values; smoothing the noisy rolling Fourier ring correlation curve using an averaging window half-width averaging filter; When the rolling Fourier correlation curve falls below a predetermined threshold value representing the maximum spatial frequency of significant information outside random noise, the frequency is defined as an effective cutoff frequency, and the resolution is defined as the reciprocal of the effective cutoff frequency; The number of iterations when the maximum resolution is reached is set as the number of iterations of the deconvolution.

[0092] In one embodiment, the reconstructing the output image twice specifically includes: Using the fluctuation principle of the fluorescent signal, a first reconstruction of the output image is performed; and applying a joint sparsity-continuity constraint to perform a second reconstruction on the image after the first reconstruction.

[0093] In one embodiment, the first reconstruction of the output image using the fluctuation principle of the fluorescent signal is specifically obtaining an expression for the fluorescence signal based on a function of the microscope point spread function, the fluorophore brightness constant, and the time-varying variation of the fluorophore brightness; Obtaining a zero time delay time accumulation expression based on the expression of the fluorescent signal using the individual wave characteristics of each fluorescent molecule; The method includes expanding an expression for the time accumulated amount, and if a preset condition is satisfied, regarding the cross-correlation term in the time accumulated amount expansion expression as zero, and expressing the time accumulated amount as a sum of squares of the corresponding brightness constant-weighted point spread function.

[0094] In one embodiment, applying the joint sparsity-continuity constraint to perform a second reconstruction on an image after a first reconstruction can be specifically performed by: constructing a reconstruction model including a first term that is a fidelity term representing a distance between the first reconstructed image and the initial image collected, a second term that represents a continuity constraint on the first reconstructed image, and a third term that represents a sparsity constraint on the first reconstructed image; and performing a second reconstruction on the image after the first reconstruction using the reconstruction model.

[0095] In one embodiment, the deconvolution iterations specifically include: obtaining an iterative formula in a spatial domain iterative maximum likelihood manner; and implementing the iterative calculation of the deconvolution using an acceleration method based on vector extrapolation based on the iterative formula.

[0096] In conclusion, the present invention has the advantage of being highly flexible and can be widely combined with various imaging modes such as acoustic microscopy, i.e. photoacoustic and ultrasonic microscopy imaging techniques. It has the advantage of being high throughput, and one image obtained by using deconvolution after multiple iterations can be further processed to achieve a two-fold improvement in three-dimensional spatial resolution. It can achieve high-quality super-resolution effect in 20 frames or less while maintaining super-resolution, and achieve a 50-100-fold improvement in temporal resolution. It has the advantage of being non-parametric, and proposes an automatic parameter estimation means based on rolling Fourier ring correlation to achieve non-parametric, thereby realizing automatic and high-throughput super-resolution imaging.

[0097] It should be noted that the computer-readable storage medium of this embodiment may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of more than one. More specific examples of the computer-readable storage medium may be, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0098] In this embodiment, the computer-readable storage medium may be any tangible medium that includes or stores a program instructed to be used by or in combination with an instruction execution system, apparatus, or device. Meanwhile, in this embodiment, the computer-readable signal medium may include a data signal propagating in baseband or as part of a carrier wave carrying the computer-readable program. Such a propagating data signal may take various examples including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium may be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The computer program contained in the computer-readable storage medium may be transmitted by any suitable medium including, but not limited to, an electrical wire, an optical cable, RF (radio frequency), etc., or any suitable combination of the above.

[0099] The computer readable storage medium may be written with a computer program for carrying out the present embodiment in one or more programming languages ​​or a combination thereof. The programming languages ​​include object-oriented programming languages ​​such as Java, Python, C++, and conventional procedural programming languages ​​such as C. The program may run entirely on the user computer, partially on the user computer, as an independent package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. When referring to a remote computer, the remote computer may be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0100] The computer program code necessary for the operation of the various parts of the present application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, e.g., the C programming language, conventional programming languages ​​such as Visual Basic, Fortran 2103, Perl, COBOL 2102, PHP, ABAP, e.g., dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer as an independent package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via some form of network such as a local area network (LAN) or wide area network (WAN), connected to an external computer (e.g., via the Internet), or used as a service such as SaaS in a cloud computing environment.

[0101] Additionally, unless expressly stated in the claims, the order of processing elements and sequences described herein, the use of alphanumeric characters, or the use of other names, shall not be used to limit the order of flow and method of the present application. Although the above disclosure has discussed several inventive embodiments presently believed to be useful by way of various examples, such details are for illustrative purposes only, and the appended claims are not intended to be limited to the disclosed embodiments. On the contrary, it should be understood that the claims are intended to cover all modifications and equivalent combinations consistent with the substance and scope of the embodiments of the present application. For example, the implementation of the various components described above may be implemented in hardware devices, but may also be implemented as pure software solutions, such as installation on existing servers or mobile devices.

[0102] Finally, it should be understood that the embodiments described herein are used only to illustrate the principles of the embodiments of the present application. Other variations are possible within the scope of the present application. Thus, by way of example, and not of limitation, alternative configurations of the embodiments of the present application are deemed to be consistent with the teachings of the present application. Thus, the embodiments of the present application are not limited to the embodiments expressly illustrated and described herein.

[0103] (Additional Note) (Appendix 1) Acquiring a sequence of fluorescent signals of a sample to be observed; Determining a number of deconvolution iterations; performing a pre-deconvolution iteration on an initial image of each frame in the fluorescence signal sequence, and outputting when the iteration reaches half the number of iterations; reconstructing the output image twice; and performing a second deconvolution iteration on the twice reconstructed image and outputting when the iterations reach the number of iterations or half the number of iterations. Super-resolution microscopy imaging methods.

[0104] (Appendix 2) After collecting the series of samples to be observed, 2. A super-resolution microscopy imaging method according to claim 1, characterized in that it includes estimating the background of the fluorescent signal using a wavelet transform and removing background noise.

[0105] (Appendix 3) Specifically, the method of estimating the background of the fluorescent signal using the wavelet transform and removing the background noise includes the following steps: estimating a background from the lowest frequencies of the input image using wavelet estimation; blending these two images by inverse wavelet transforming the lowest band into the spatial domain, comparing the result with half the square root of the input image, and keeping the minimum value for each pixel; The super-resolution microscopy imaging method according to claim 2, further comprising: looping wavelet estimation using the estimated low-band low-peak background data as a new input image until a preset number of cycles is reached.

[0106] (Appendix 4) Determining the number of iterations of the deconvolution specifically includes: 2. The super-resolution microscopy imaging method according to claim 1, further comprising determining the number of iterations when the maximum rolling Fourier ring correlation resolution is reached using the rolling Fourier ring correlation as the number of iterations of deconvolution.

[0107] (Appendix 5) The number of iterations when the maximum rolling Fourier ring correlation resolution is reached by using the rolling Fourier ring correlation is set as the number of iterations of the deconvolution, specifically, defining a Rolling Fourier Ring correlation as a function of spatial frequency and a spatial frequency discretization of the Rolling Fourier Ring correlation curve to calculate corresponding discrete spatial frequency values; smoothing the noisy rolling Fourier ring correlation curve using an averaging window half-width averaging filter; When the rolling Fourier correlation curve falls below a predetermined threshold value representing the maximum spatial frequency of significant information outside random noise, the frequency is defined as an effective cutoff frequency, and the resolution is defined as the reciprocal of the effective cutoff frequency; and setting the number of iterations when the maximum resolution is reached as the number of iterations of deconvolution.

[0108] (Appendix 6) Specifically, the output image is reconstructed twice. Using the fluctuation principle of the fluorescent signal, a first reconstruction of the output image is performed; A super-resolution microscopy imaging method as described in Appendix 1, characterized in that it includes applying a joint sparseness-continuity constraint to perform a second reconstruction on the image after the first reconstruction.

[0109] (Appendix 7) Specifically, the first reconstruction of the output image using the fluctuation principle of the fluorescent signal is performed as follows: obtaining an expression for the fluorescence signal based on a function of the microscope point spread function, the fluorophore brightness constant, and the time-varying variation of the fluorophore brightness; Obtaining a zero time delay time accumulation expression based on the expression of the fluorescent signal using the individual wave characteristics of each fluorescent molecule; A super-resolution microscopy imaging method according to claim 6, characterized in that it includes: expanding an expression for the time-accumulated amount; and, if a preset condition is satisfied, regarding the cross-correlation term in the time-accumulated amount expansion expression as zero, and expressing the time-accumulated amount as the sum of squares of the corresponding brightness-constant-weighted point spread functions.

[0110] (Appendix 8) Specifically, applying the joint sparsity-continuity constraint to perform a second reconstruction on an image after a first reconstruction is as follows: constructing a reconstruction model including a first term that is a fidelity term representing a distance between the first reconstructed image and the initial image collected, a second term that represents a continuity constraint on the first reconstructed image, and a third term that represents a sparsity constraint on the first reconstructed image; 7. A super-resolution microscopy imaging method according to claim 6, further comprising: performing a second reconstruction on the image after the first reconstruction using the reconstruction model.

[0111] (Appendix 9) Specifically, the deconvolution iteration is obtaining an iterative formula in a spatial domain iterative maximum likelihood manner; 9. A super-resolution microscopy imaging method according to any one of claims 1 to 8, characterized in that it includes realizing iterative calculation of deconvolution using an acceleration method based on vector extrapolation based on an iterative formula.

[0112] (Appendix 10) an acquisition module for acquiring a sequence of fluorescent signals of a sample to be observed; a determining module for determining the number of iterations of the deconvolution; a pre-deconvolution module for performing a pre-deconvolution iteration on an initial image of each frame in the fluorescence signal sequence and for outputting when the iterations reach half the number of iterations; a reconstruction module for reconstructing the output image twice; and a deconvolution module for performing a second deconvolution iteration on the twice reconstructed image and outputting when the iterations reach the iteration number or half the iteration number. Super-resolution microscopic imaging device.

[0113] (Appendix 11) A computer device including a processor and a memory for storing a program executable by the processor, A computer apparatus that, when the processor executes a program stored in a memory, realizes a super-resolution microscopy imaging method according to any one of claims 1 to 9.

[0114] (Appendix 12) A storage medium on which a program is stored, A storage medium characterized in that, when the program is executed by a processor, the super-resolution microscopy imaging method described in any one of appendices 1 to 9 is realized.

Claims

1. Acquiring a sequence of fluorescent signals of a sample to be observed; Determining a number of deconvolution iterations; performing a pre-deconvolution iteration on an initial image of each frame in the fluorescence signal sequence, and outputting when the iteration reaches half the number of iterations; reconstructing the output image twice; performing a second deconvolution iteration on the twice reconstructed image and outputting when the iterations reach the number of iterations or half the number of iterations. Super-resolution microscopic imaging methods.

2. After collecting the series of samples to be observed, 2. The method of claim 1, further comprising: estimating the background of the fluorescent signal using a wavelet transform; and removing background noise.

3. Determining the number of iterations of the deconvolution specifically includes: The super-resolution microscopy imaging method according to claim 1, further comprising determining the number of iterations when a maximum rolling Fourier ring correlation resolution is reached as the number of iterations of deconvolution.

4. The number of iterations when the maximum rolling Fourier ring correlation resolution is reached by using the rolling Fourier ring correlation is set as the number of iterations of the deconvolution, specifically, defining a Rolling Fourier Ring correlation as a function of spatial frequency and a spatial frequency discretization of the Rolling Fourier Ring correlation curve to calculate corresponding discrete spatial frequency values; smoothing the noisy rolling Fourier ring correlation curve using an averaging window half-width averaging filter; When the rolling Fourier correlation curve falls below a predetermined threshold value representing the maximum spatial frequency of significant information outside random noise, the frequency is defined as an effective cutoff frequency, and the resolution is defined as the reciprocal of the effective cutoff frequency; 4. The method of claim 3, further comprising: determining the number of iterations when a maximum resolution is reached as the number of iterations of the deconvolution.

5. Specifically, the output image is reconstructed twice. A first reconstruction of the output image is performed using the fluctuation principle of the fluorescent signal; The super-resolution microscopy imaging method according to claim 1, further comprising: applying a joint sparseness-continuity constraint to perform a second reconstruction on the image after the first reconstruction.

6. Specifically, the first reconstruction of the output image using the fluctuation principle of the fluorescent signal is performed as follows: obtaining an expression for the fluorescence signal based on a function of the microscope point spread function, the fluorophore brightness constant, and the time-varying variation of the fluorophore brightness; Obtaining a zero time delay time accumulation expression based on the expression of the fluorescent signal using the individual wave characteristics of each fluorescent molecule; The super-resolution microscopy imaging method of claim 5, further comprising: expanding an expression for the time-accumulated amount; and, if a predetermined condition is satisfied, regarding the cross-correlation term in the time-accumulated amount expansion expression as zero; and expressing the time-accumulated amount as the sum of squares of corresponding brightness-constant-weighted point spread functions.

7. Specifically, applying the joint sparsity-continuity constraint to perform a second reconstruction on an image after a first reconstruction is as follows: constructing a reconstruction model including a first term that is a fidelity term representing a distance between the first reconstructed image and the initial image acquired, a second term that represents a continuity constraint of the first reconstructed image, and a third term that represents a sparsity constraint of the first reconstructed image; 6. The method of claim 5, further comprising: performing a second reconstruction on the image after the first reconstruction using the reconstruction model.

8. Specifically, the deconvolution iteration is obtaining an iterative formula in a spatial domain iterative maximum likelihood manner; The super-resolution microscopy imaging method according to any one of claims 1 to 7, further comprising: realizing the iterative calculation of deconvolution using an acceleration method based on vector extrapolation based on an iterative formula.

9. an acquisition module for acquiring a sequence of fluorescent signals of a sample to be observed; a determining module for determining the number of iterations of the deconvolution; a pre-deconvolution module for performing a pre-deconvolution iteration on an initial image of each frame in the fluorescence signal sequence and for outputting when the iterations reach half the number of iterations; a reconstruction module for reconstructing the output image twice; and a deconvolution module for performing a second deconvolution iteration on the twice reconstructed image and outputting when the iterations reach the iteration number or half the iteration number. Super-resolution microscopic imaging device.

10. A computer device including a processor and a memory for storing a program executable by the processor, A computer device which, when the processor executes a program stored in a memory, realizes the super-resolution microscopic imaging method according to any one of claims 1 to 7.

11. A storage medium on which a program is stored, A storage medium, which, when the program is executed by a processor, realizes the super-resolution microscopic imaging method according to any one of claims 1 to 7.

Images