A structured illumination microscopy method based on adaptive hilbert norm

Abstract

The present application relates to a kind of structure light illumination microscopic imaging method based on adaptive hilbert norm, the present application aims at solving the problem of inaccurate illumination parameter estimation value and reconstruction artifact under the condition of uneven illumination stripe in existing SIM reconstruction algorithm.The method of the present application decomposes the single frame or multiple frame SIM original image collected into structure, texture and noise part;By constructing an energy minimization model containing total variation norm and adaptive hilbert norm, the texture part representing illumination pattern is accurately extracted;The extracted illumination pattern image is iteratively reconstructed with SIM original image, and finally the super-resolution image is obtained.Compared with the traditional method, the present application can accurately extract the illumination pattern from single frame original image, effectively process the non-uniform illumination stripe with local bending, and obtain more accurate super-resolution reconstruction result.The method is suitable for a variety of reconstruction methods and application scenarios, and has good application prospect in the improvement of SIM imaging quality and imaging flux.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of optical super-resolution microscopic imaging, and particularly relates to a structured illumination microscopy (SIM) method based on adaptive Hilbert norm. BACKGROUND

[0002] The structural organization of the brain is very complex, from the dendrite and axon fiber with a diameter of less than 1 micrometer, the capillary with a diameter of several micrometers to the neuron cell body, arteriole and venule with a diameter of tens of micrometers, and the whole brain optical imaging must span several orders of magnitude to realize the imaging of fine structures. Therefore, the whole brain optical imaging puts forward the demand for the super-resolution ability and imaging flux of microscopic imaging. The structured illumination microscopy is a high-flux super-resolution imaging technology, which projects a high-frequency sinusoidal stripe structure light onto the sample, uses the principle of frequency spectrum shift to shift the high-frequency information in the sample that cannot be directly detected by the objective lens into a detectable range, and then restores the sample structure beyond the diffraction limit through a subsequent image reconstruction algorithm. When applied to whole brain neural circuit imaging, it can more efficiently observe neurons and other fine structures at a large scale.

[0003] The SIM reconstruction algorithm requires accurate estimation of various parameters of the illumination stripe, including spatial frequency, phase and modulation depth. Existing parameter estimation methods, such as frequency domain cross-correlation, autocorrelation, etc., usually assume that the illumination stripe is uniform in the entire field of view, and most of them require three or more phase-shifted images to work accurately. When facing the local bending or deformation of the stripe caused by factors such as aberration and defocus, these methods will give inaccurate estimation results, resulting in artifacts in the reconstruction results. In addition, they are not suitable for high-throughput imaging scenarios with only a small number of original images. Therefore, following the traditional illumination parameter estimation approach makes it difficult for the current SIM reconstruction algorithm to have a substantial breakthrough.

[0004] From the perspective of image decomposition, the SIM original image can be regarded as a combination of a fluorescent sample, an illumination stripe and noise, so the assumption that the frequency and phase of the illumination stripe remain consistent within the imaging field of view can be abandoned, and the illumination stripe can be accurately extracted. However, how to establish clear and effective constraints on the illumination stripe, and the extracted illumination stripe is not directly compatible with traditional reconstruction methods, has become a problem to be solved. SUMMARY

[0005] The present application aims to overcome the deficiencies of the prior art, and provides a robust, accurate and suitable for high-throughput imaging based on adaptive Hilbert norm structured light illumination microscopic imaging method. The present application is based on the illumination fringe extraction method of adaptive Hilbert norm, which regards the illumination fringe as a kind of local parallel texture with strong directionality. By introducing the structure-texture decomposition idea in image processing, the SIM original image is decomposed into a low-frequency structure part representing the sample fluorescence distribution and a high-frequency texture part representing the illumination fringe, so as to extract the SIM illumination fringe, and then complete the iterative super-resolution reconstruction. When the SIM original image has the defect of non-uniform illumination fringe caused by factors such as optical system aberration, the illumination fringe can be accurately extracted, the reconstruction artifact can be reduced, and more accurate and stable super-resolution results can be obtained.

[0006] The technical scheme of the present application is:

[0007] A structured light illumination microscopic imaging method based on adaptive Hilbert norm, characterized by comprising the following steps:

[0008] Step 1: Based on the structure-texture decomposition model of the image, the single-frame structured light illumination original image y collected is decomposed into a smooth structure part f representing the fluorescence sample, a high-frequency texture part I representing the illumination fringe and a noise part n. By introducing total variation regularization constraint structure part, and using adaptive Hilbert norm constraint illumination fringe part, an optimization objective is constructed.

[0009] Step 2: The energy minimization problem constructed in step 1 is solved by block coordinate descent algorithm, and the illumination fringe is extracted from the input original image. The output I of this process is a two-dimensional image with the same size as the original image, which accurately represents the real illumination fringe pattern acting on the sample, and contains the possible local bending, phase shift and other non-uniform characteristics. This step can be completed for single-frame original image without multiple frame phase shift images.

[0010] Step 3: The illumination fringe image extracted for each original image in step 2 is substituted into the forward model of the iterative super-resolution reconstruction algorithm as a known physical prior information, and an iterative optimization objective is constructed.

[0011] Step 4: The optimization objective constructed in step 3 is solved by alternating direction multiplier method, and the SIM super-resolution image is obtained.

[0012] In step 1, based on the structure-texture decomposition model of the image, the single-frame structured light illumination original image y collected is decomposed into a smooth structure part f representing the fluorescence sample and a high-frequency texture part I representing the illumination fringe and a noise part n. By introducing total variation regularization constraint structure part, and using adaptive Hilbert norm constraint texture part, the corresponding minimum objective function is obtained:

[0013] ;

[0014] is the optimization variable for the optimization problem, i.e. minimizing the objective function ; is the detected single-frame SIM raw image; is the fluorescence distribution of the biological sample to be decomposed, i.e. the structure part; is the illumination fringe pattern to be extracted, i.e. the texture part; ξ is the local frequency field representing the texture direction and frequency; is the total variation norm of the structure part ; is the adaptive Huber norm of the illumination fringe part with respect to the frequency field , which is used to precisely constrain the illumination fringe with strong directionality, and is described as: ; is the local Fourier frame, each is the Fourier transform at the coordinates within the local window ; is the decomposition operator of the local Fourier frame on , which is expressed as is the weighting coefficient with respect to the frequency field , which is defined as:

[0015] ;

[0016] where, is a scale parameter reflecting the deviation of and . denotes the frequency field coordinate with the window center as the origin, denotes the coordinate of the dominant fringe frequency in the frequency field around the point ; when there is a dominant directional fringe around the point , and correspond to two frequency components symmetric about the spectrum center, respectively, so that is as small as possible to make as small as possible, thereby extracting the dominant fringe frequency in the optimization process. When there is no dominant directional fringe around the point , for all to avoid the generation of false fringes; is a positive parameter to prevent weight disappearance and ensure convergence; is the decomposition result of the structure part a diagonal matrix constructed by is a data fidelity term, which ensures that the decomposition result is consistent with the original image; is a non-negative regularization coefficient to balance the above two constraints.

[0017] Step 2 solves the optimization problem constructed in step 1 by a block coordinate descent algorithm, i.e., respectively solving the variables , minimizes the objective function, and the minimization of the objective function is:

[0018] ;

[0019] The optimization problem is divided into three sub-problems, and the three sub-problems are solved as follows:

[0020] (1) Fixing , , The dominant frequency in the small window of can be approximated as: ; is a pre-set threshold parameter to extract the dominant frequency;

[0021] (2) Fixing , , can be calculated by the proximal operator: ; is the proximal operator, which can be iteratively calculated;

[0022] (3) Fixing , , The gradient equation is: ; is the unit matrix; is a symmetric positive definite operator, which can be solved by the conjugate gradient algorithm or Fourier transform.

[0023] The forward model of the iterative super-resolution reconstruction algorithm in step 3 is described as: ;

[0024] is the detected single-frame SIM original image; is the three-dimensional fluorescence distribution of the biological sample to be reconstructed with high resolution; is the extracted illumination fringe image in step (2); is the discrete convolution operator of the microscope optical system, and the core is the point spread function; is the down-sampling operator; is the noise part.

[0025] The iterative optimization objective is described as follows: ;

[0026] The original image number for SIM; The least squares fidelity term has the following mathematical form: ; For regularization terms, For Hessian operators, calculate The second partial derivative at each point. for Mixed norm, its mathematical form is ; for The first-order Schatten norm at point n is defined as the matrix... The sum of all singular values; As a non-negativity constraint, it ensures that the reconstructed fluorophore density distribution has physical meaning, that is, the intensity value at each point is greater than or equal to zero; For linear operators, the mathematical form is: ; To balance the nonnegative regularization coefficients of the two constraints mentioned above.

[0027] Step 4: Solve the Lagrange equation corresponding to the optimization problem in Step 3 using the alternating direction multiplier method.

[0028] ;

[0029] , Auxiliary variables and their dual variables, ; , Lagrange multiplier;

[0030] Using the alternating direction multiplier method, variables The iterative calculation formula is as follows: Initialize the super-resolution image A set of auxiliary variables and its dual variables :

[0031] ;

[0032] Update auxiliary variables:

[0033] ;

[0034] ;

[0035] Update host variables:

[0036]

[0037] ;

[0038] Updating the dual variables:

[0039] ;

[0040] ;

[0041] Finally, the SIM super-resolution image is obtained .

[0042] Advantages of the present application:

[0043] 1. The illumination fringe can be extracted from a single original image.

[0044] 2. The illumination fringe can be accurately extracted when the illumination fringe is not uniform, and an accurate super-resolution reconstruction result is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flow chart of the present application.

[0046] Figure 2 is a schematic diagram of the illumination fringe extraction results of a set of SIM images without a rotation region in three different directions of the present application.

[0047] Figure 3 is a schematic diagram of the fringe extracted on a simulated input image of the present application in a local rotation region.

[0048] Figure 4a is a schematic diagram of the SIM input image of the present application in a local rotation (red square).

[0049] Figure 4b is a schematic diagram of the fringe true value in a rotation region of the present application.

[0050] Figure 4c is a schematic diagram of the part of the fringe extracted in the rotation region of the present application.

[0051] Figure 4d is a spectrum diagram of the fringe true value in a rotation region of the present application.

[0052] Figure 4e is a spectrum diagram of the part of the fringe extracted in the rotation region of the present application.

[0053] Figure 4f is a schematic diagram of the true value in a rotation region of the present application.

[0054] Figure 4g is a reconstruction result diagram in a rotation region of the present application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings.

[0056] The present application proposes a method for extracting illumination stripes based on adaptive Hilbert norm, which regards the illumination stripes as a kind of local parallel texture with strong directionality, decomposes the SIM original image into a low-frequency structure part representing the sample fluorescence distribution and a high-frequency texture part representing the illumination stripes by introducing the structure-texture decomposition idea in image processing, so as to extract the SIM illumination stripes, and then completes the iterative super-resolution reconstruction. The specific process is shown in Figure 1 The main steps are as follows:

[0057] Step 1: Based on the structure-texture decomposition model of the image, the collected single-frame structured light illumination original image y is decomposed into a smooth structure part f representing the fluorescence sample, a high-frequency texture part I representing the illumination stripes and a noise part n. By introducing the total variation regularization constraint structure part, and using the adaptive Hilbert norm constraint illumination stripe part, the minimum objective function is constructed as follows:

[0058] ;

[0059] Step 2: The minimum objective function constructed in step 1 is solved by block coordinate descent algorithm, and the illumination stripes I are extracted from the input original image y,

[0060] (1) Fix the main frequency , , in the small window of ,

[0061] ;

[0062] (2) Fix , , , which can be calculated by the proximal operator: ;

[0063] (3) Fix , , , which is solved by the conjugate gradient descent method, and the gradient equation is: ;

[0064] Step 3: The illumination stripe image extracted for each original image in step 2 is substituted into the forward model of the iterative super-resolution reconstruction algorithm as a known physical prior information, and an iterative optimization objective is constructed.

[0065] The forward model is described as: ;

[0066] The optimization objective is described as: ;

[0067] Step 4: the SIM super-resolution image is obtained by solving the Lagrange equation corresponding to the optimization objective in step 3 through the alternating direction multiplier method :

[0068] ;

[0069] , auxiliary variables and their dual variables, ; , Lagrange multipliers;

[0070] Using the alternating direction multiplier method, the iterative calculation formula of the variable is as follows: initializing the super-resolution image , a set of auxiliary variables and their dual variables :

[0071] ;

[0072] Updating the auxiliary variables:

[0073]

[0074] ;

[0075] Updating the main variables:

[0076]

[0077] ;

[0078] Updating the dual variables:

[0079] ;

[0080] ;

[0081] Finally, the SIM super-resolution image is obtained.

[0082] Next, the application is further described through examples.

[0083] Embodiment:

[0084] Two groups of SIM simulation images are used as simulation cases in this example. In order to simulate the diffraction-limited SIM images, the ground truth images are first multiplied by the illumination stripes, then convolved by the point spread function, and finally added with background fluorescence and noise. A 128x128 pixel size region of one group of images is rotated by 90° to simulate the inhomogeneous illumination stripe condition to prove the good performance of the method. The illumination stripes of both groups of simulation images are obtained by the illumination stripe extraction method based on adaptive Hilbert norm. Figure 2 Figure 2 Three illumination directions are shown, and one phase is selected for each direction. Compared with the frequency and phase of the real stripes, the frequency ratio (wave vector modulus / true value) and phase error are shown in Table 1 and Table 2.

[0085] Table 1 Phase error of the extracted illumination stripes in each direction and phase

[0086]

[0087] Table 2 Frequency ratio (accurate to one pixel) of the extracted illumination stripes in each direction and phase

[0088]

[0089] The stripes extracted by the method on the simulation input images containing local rotated regions are shown in Figure 3 . The method has accurate reconstruction results in both the unrotated region and the rotated region, and there is no mutual influence. The part of the super-resolution reconstruction result in the rotated region is shown in Figure 4a , Figure 4b , Figure 4c , Figure 4d , Figure 4e , Figure 4f , Figure 4g . The reconstruction result of the method is consistent with the true value, the error is very small, the image detail information is good, and the super-resolution quality is high.​

Claims

1. A method of structured illumination microscopy based on adaptive Hubert norm, characterized in that, The method comprises the following steps: Step 1, based on an image-based structure-texture decomposition model, decomposing the collected single-frame structured light illumination original image y into a smooth structure part f representing a fluorescent sample, a high-frequency texture part I representing an illumination fringe, and a noise part n, constraining the structure part by introducing a total variation regularization, and constraining the illumination fringe part by using an adaptive Huber norm, and constructing a minimization objective function; the minimization objective function is: ； To optimize the problem, i.e., minimize the objective function, the optimization variable is: ; The original image of a single-frame SIM detected; The fluorescence distribution of the biological sample to be decomposed, i.e., the structural part; The light stripe pattern to be extracted is the texture portion; A local frequency field characterizing the texture direction and frequency; For structural parts The total variation norm; For the lighting stripe section Regarding frequency fields An adaptive Hilbert norm, used to precisely constrain strongly directional lighting fringes, is described as follows: ; For a local Fourier framework, each For local windows Interior coordinates Fourier transform at the location; For local Fourier framework Below Decomposition operator, The positive regularization coefficient is non-negative. Step 2, solving the minimization objective function constructed in step 1 by using a block coordinate descent algorithm, and separating an illumination fringe image from the input original image y; Step 3, the illumination fringe image extracted from each original image in step 2 is substituted into the forward model of the iterative super-resolution reconstruction algorithm as a known physical prior information to construct an iterative optimization target; the forward model of the iterative super-resolution reconstruction algorithm is as follows: ; for the detected single frame SIM raw image; for the high resolution three-dimensional fluorescence distribution of the biological sample to be reconstructed; for the extracted illumination fringe image in step 2; for the discrete convolution operator of the microscope optical system, the core of which is the point spread function; for the down-sampling operator; noise part; The iterative optimization objective is described as: ; is the original image sequence number of the SIM; is the least square fidelity term, which is mathematically expressed as: ; is the regularization term, is the Hessian operator, which is calculated as the second order partial derivative at each point; is the mixed norm, which is mathematically expressed as ; is the first order Schatten norm at point n, which is defined as the sum of all singular values of the matrix ; is the non-negativity constraint, which ensures that the reconstructed fluorophore density distribution has physical meaning, i.e. the intensity value at each point is greater than or equal to zero; is the linear operator, which is mathematically expressed as ; is the non-negative regularization coefficient; Step 4, solving the iterative optimization objective constructed in step 3 to obtain a SIM super-resolution image.

2. The structured light illumination microscopic imaging method based on an adaptive Huber norm according to claim 1, characterized in that: represents a form is a weighting coefficient with respect to the frequency field is defined as follows: ; where, is a scale parameter reflecting deviation from ; denotes the frequency field coordinate with the window center as the origin, denotes the coordinate of the dominant frequency of the fringe around point in the frequency field ; when there is a dominant fringe around point , the two terms and correspond to two frequency components symmetric about the spectrum center, respectively, so that is as small as possible, and is as small as possible, so as to extract the dominant frequency of the fringe in the optimization process; when there is no dominant fringe around point , all are set to , so as to avoid the generation of false fringes; is a positive parameter to prevent the weight from disappearing and to ensure convergence; is a diagonal matrix composed of , and is a data fidelity term to ensure that the decomposition result is consistent with the original image.

3. The adaptive Hubert norm based structured illumination microscopy method of claim 2, wherein Step 2 is specifically splitting the minimization objective function of step 1 into three sub-problems, and solving the three sub-problems as follows: (1) fixed , , the dominant frequency in the small window at ; is a pre-set threshold parameter to extract the dominant frequency;​ (2) fixed , , By calculating the proximal operator: ; is a proximal operator that can be computed iteratively; (3) fixed , , The gradient equation is: ; is the identity matrix; is a symmetric positive definite operator, is obtained by the conjugate gradient algorithm or Fourier transform.

4. The adaptive Hubert norm based structured illumination microscopy method of claim 3, wherein: Step 4 is specifically solving the iterative optimization objective by using an alternating direction multiplier method, and using corresponding Lagrange equations: ； , Lagrange multipliers, , Lagrange multipliers; Using the alternating direction method of multipliers, the iterative formula for the variables is given by a set of auxiliary variables and their dual variables : ； updating the auxiliary variable: ； ； updating the main variable: ； updating the dual variable: ； Final SIM super-resolution image .

Images