A structured light illumination super-resolution image reconstruction method with joint noise reduction in spatial and frequency domains

Abstract

The present invention discloses a structured light illumination super-resolution image reconstruction method with space-frequency domain joint noise reduction, the method comprising: reading multiple frames of original images collected by a structured light illumination microscopy (SIM); estimating various parameters of the illumination structured light stripes; based on the result of parameter estimation, constructing a SIM super-resolution reconstruction model in the frequency domain using a combined notch method to achieve frequency domain noise reduction of the super-resolution image; and using a regularization model based on Split Bregman optimization in the spatial domain to further achieve spatial domain noise reduction of the SIM super-resolution image. The method of the present invention realizes SIM super-resolution reconstruction by using space-frequency domain joint noise reduction, which can effectively reduce artifacts related to noise and defocus information in the SIM reconstructed image, and significantly improve the quality of SIM reconstruction.

Description

Technical Field

[0001] The present invention belongs to the technical field of super-resolution microscopic imaging, and in particular to a structured light illumination super-resolution image reconstruction method with combined noise reduction in space-frequency domain. Background Art

[0002] SIM microscopy technology projects multiple structured light illumination patterns of different directions and phases onto the sample surface for spatial modulation, which can transfer high-frequency information beyond the optical transfer function (OTF) of the system into the detectable range of the objective lens. After the original image is collected, the high-frequency information can be restored and the resolution can be improved through the SIM reconstruction algorithm. Figure 2 shown.

[0003] The conventional SIM reconstruction algorithm is mainly based on the deconvolution operation of the Wiener filter model. It is essentially an ill-posed inverse problem, and the reconstruction effect is heavily dependent on the accuracy of the structured light fringe parameter estimation and the setting of the filter parameters. There are currently a variety of optimization methods for the parameter estimation problem, which have improved the quality of the reconstructed image to a certain extent, but there are still a variety of artifacts. This is because the complexity of the imaging process causes the original image to contain a variety of noise and interference from defocus information, and the actual noise type and distribution are diverse. However, the noise parameter in the SIM reconstruction model is a single constant, which means that the algorithm assumes that the noise type and level of all areas of the image are consistent. Therefore, the noise and defocus information carried by the original image cannot be accurately suppressed, and a variety of artifacts appear in the reconstructed image, which ultimately leads to a decrease in resolution and unreliable imaging results, and cannot provide accurate information for subsequent quantitative analysis.

[0004] In order to suppress the artifacts caused by noise and defocus information in the image, a variety of SIM reconstruction optimization methods have been proposed, such as fair-SIM, RL-SIM, TV-SIM, MAP-SIM, Hessian-SIM and HiFi-SIM. Although these methods have made certain improvements in reducing artifacts and improving imaging resolution, they also have some potential disadvantages: (1) Both RL-SIM and HiFi-SIM use the Richardson-Lucy (RL) algorithm to perform deconvolution operations on the original image and the reconstructed image respectively. However, the RL algorithm is an iterative method based on statistics. It assumes that the image noise follows a Poisson distribution, which is inconsistent with the actual noise situation of SIM imaging and is prone to additional artifacts and loss of details. (2) fair-SIM uses a Gaussian notch filter to notch and denoise the separated frequency spectra at each level, which is easy to cause gully-shaped areas in the spectrum, which appears as artifacts with independent characteristics in the image domain. To address this problem, HiFi-SIM constructs a composite filter consisting of two Wiener filters for deconvolution, constructing an equivalent OTF close to the actual imaging conditions, and achieving a good balance between improving resolution and reducing artifacts. However, this method contains multiple adjustable parameters that need to be set manually, and requires a high level of professionalism from the user. (3) TV-SIM, MAP-SIM, and Hessian-SIM are all methods based on spatial domain iterative denoising models, which are sensitive to noise and set parameters and are prone to overfitting. Summary of the invention

[0005] The object of the present invention is to provide a SIM image reconstruction method that can directly and effectively remove noise and artifact interference without requiring excessive parameter settings in order to solve the problems existing in the prior art.

[0006] The technical solution to achieve the purpose of the present invention is: a structured light illumination super-resolution image reconstruction method with joint noise reduction in spatial and frequency domains, the method comprising:

[0007] Read multiple frames of original images acquired by the SIM imaging system;

[0008] Preprocess the original image to eliminate interference information in the image frequency domain;

[0009] Estimate structured light illumination stripe parameters, including spatial frequency, modulation and initial phase;

[0010] According to the results of structured light illumination stripe parameter estimation, a combined notch filter method is used to build a SIM reconstruction model to achieve frequency domain denoising of super-resolution images.

[0011] For the super-resolution reconstructed image after frequency domain denoising, a regularization method based on Split Bregman optimization is further used for spatial domain denoising.

[0012] Furthermore, in one of the embodiments, the preprocessing of the original image specifically includes:

[0013] The original image is summed and averaged to obtain an equivalent wide-field image, and a sub-image block that does not contain the biological sample is intercepted and the average value of all pixels is accumulated and taken as the background value of the original image. , and subtract it from the original data The original image after background removal is obtained as shown in the following formula:

[0014]

[0015] in, For the Azimuth The original SIM image collected after the phase structured light stripe modulation, is the spatial coordinate of the pixel point, Indicates the operation of intercepting a sub-image block. and are the width and height of the intercepted sub-image block respectively;

[0016] A sinusoidal grayscale fading function is used to perform boundary fading processing on the original image after background removal, as shown in the following formula:

[0017]

[0018] In the formula, and They are the original images after background removal. The width and height of The width of the faded border area;

[0019] The original image after the boundary fading process is normalized as shown in the following formula:

[0020]

[0021] In the formula, and Represents the operations of taking the maximum pixel value and the minimum pixel value respectively.

[0022] Furthermore, in one embodiment, estimating structured light illumination stripe parameters specifically includes:

[0023] The optical transfer function (OTF) of the SIM imaging system is constructed based on the system optical parameters and theoretical model;

[0024] The preprocessed original image is subjected to Fourier transform, and the spectrum of each level is roughly separated in the frequency domain. The spectrum of each level obtained is expressed as , and , represents the fringe azimuth; , and They represent the zero-order, 1-order and -1-order spectra after rough separation respectively;

[0025] De-noise the spectrum at each level after rough separation;

[0026] The spatial frequency is estimated by the cross-correlation method based on the correlation of the zero-order and higher-order sub-spectra after noise reduction. ;

[0027] Based on the structured light spatial frequency , perform high-order spectrum shift, and use linear regression method to estimate the structured light modulation and initial phase .

[0028] Further, in one of the embodiments, constructing the optical transfer function OTF of the SIM imaging system according to the system optical parameters and the theoretical model specifically includes:

[0029] Combined with the optical parameters of the SIM system, the OTF is generated according to the theoretical model. The formula is:

[0030]

[0031] in, is the system optical transfer function OTF, k is the frequency domain variable, is the system cutoff frequency, is the numerical aperture of the detection objective, is the emission wavelength, is the aberration compensation coefficient.

[0032] Furthermore, in one of the embodiments, a Gaussian notch filter is used to reduce noise on each level of spectrum after coarse separation.

[0033] Furthermore, in one embodiment, the spatial frequency The calculation formula is:

[0034]

[0035] In the formula, Indicates Azimuth structured light spatial frequency, superscript represents conjugation, represents cross correlation, max represents taking the maximum value of the matrix, represents the sub-pixel position index, represents the Gaussian notch function.

[0036] Furthermore, in one embodiment, the structured light spatial frequency , perform high-order spectrum shift, and use linear regression method to estimate the structured light modulation and initial phase , the specific formula is:

[0037]

[0038] In the formula, represents the first-level spectrum after translation, represents the Gaussian notch function after translation, subscript Indicates the operation of taking the overlapping area. Respectively The imaginary and real parts of .

[0039] Furthermore, in one of the embodiments, the method of constructing a SIM reconstruction model based on the structured light illumination stripe parameter estimation result and implementing frequency domain denoising of the super-resolution image by using a combined notch filter method specifically includes:

[0040] According to the estimated value of the structured light illumination stripe parameter, the pre-processed SIM original image is subjected to spectrum separation and shifting, and the stripe azimuth is , level is The spectrum is expressed as ,in, ;

[0041] Gaussian notch function and Butterworth notch function are used for joint notch processing, and the frequency domain denoising of the super-resolution image is performed in combination with the Wiener filter reconstruction model. The frequency domain result is inverse Fourier transformed to obtain the super-resolution reconstructed image.

[0042] Furthermore, in one of the embodiments, the Gaussian notch function and the Butterworth notch function are used for joint notch processing, and the frequency domain noise reduction of the super-resolution image is performed in combination with the Wiener filter reconstruction model, and the frequency domain result is subjected to inverse Fourier transform to obtain the super-resolution reconstructed image. The specific formula is:

[0043]

[0044] In the formula, represents the super-resolution reconstructed image after frequency domain denoising, represents the inverse Fourier transform, the superscript represents the conjugate value, represents the Butterworth filter function, Indicates translation The Butterworth filter function after represents the Gaussian notch function, Indicates translation The Gaussian notch function after Indicates translation The optical transfer function OTF of the system after that; is the Wiener constant, ; is the apodization function, which is used to suppress the ringing effect in the image reconstruction process.

[0045] Furthermore, in one of the embodiments, the spatial domain denoising is performed using a regularization method based on Split Bregman optimization, and the denoising process is expressed as:

[0046]

[0047] In the formula, is the super-resolution image after denoising in the spatial domain, It is a super-resolution image obtained by inverse Fourier transform from frequency domain to spatial domain after denoising. is a noisy image, is the regularization coefficient, which is used to adjust the coefficient between the data term and the regularization term. represents the gradient function, Indicates taking the L1 norm.

[0048] Compared with the prior art, the present invention has the following significant advantages:

[0049] 1) Effectively reduce the noise and defocus information related artifacts in the reconstructed image and improve the reconstruction quality of the super-resolution image.

[0050] 2) The reconstruction method of joint denoising in spatial and frequency domains can preserve the image super-resolution details while reducing noise.

[0051] 3) Based on the known prior knowledge of the system and combined with the physical model, the system OTF can be constructed without causing any loss in reconstruction quality, thus avoiding the problem of experimental measurement required by traditional SIM reconstruction systems.

[0052] 4) The present invention realizes SIM super-resolution reconstruction by using a spatial-frequency domain joint noise reduction method, which can effectively reduce artifacts related to noise and defocus information in the SIM reconstructed image and significantly improve the SIM reconstruction quality.

[0053] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1This is a flow chart of the structured light illumination super-resolution image reconstruction method with joint noise reduction in the spatial and frequency domains of the present invention.

[0055] Figure 2 This is the schematic diagram of the conventional SIM reconstruction simulation based on Wiener filtering, where Figure 2 (a) is a simulated image without structured light modulation. Figure 2 (b) is the structured light illumination stripe (taking 2D-SIM as an example, there are 3 azimuth angles, 60° apart). Figure 2 (c) is the system OTF constructed based on system parameters and physical model. Figure 2 (d) is the spectrum of the original SIM image after Fourier transformation and the spectrum of each level after separation. Figure 2 (e) in the figure is the SIM super-resolution image and corresponding spectrum reconstructed based on the Wiener filter model.

[0056] Figure 3 Schematic diagram of SIM original image preprocessing, where Figure 3 (a) in the figure is a frame of the original SIM image. Figure 3 (b) in Figure 3 The Fourier domain spectrum corresponding to (a) is: Figure 3 (c) in Figure 3 (a) is the image after preprocessing. Figure 3 (d) in Figure 3 (c) The corresponding Fourier domain spectrum.

[0057] Figure 4 Comparison of the reconstruction results of the SIM reconstruction method of the present invention with other methods, where Figure 4 (a), (b), and (c) are the reconstructed images of wide-field and conventional Wiener SIM, Wiener SIM with Gaussian notch denoising, Wiener SIM with combined notch, and SIM with joint denoising in space-frequency domain, respectively; Figure 4 (d), (e), (f), (g), and (h) are Figure 4 Partially enlarged images of (a), (b), and (c); Figure 4 (i), (j), and (k) are the spectra of the super-resolution image reconstructed by Wiener SIM, Gaussian notch Wiener SIM, and the method proposed in the present invention, respectively. Figure 4 The (l), (m), and (n) in the Figure 4 (i), (j), and (k) are the amplitude curves where the straight line in the frequency domain image is located. The horizontal axis represents the distance and the vertical axis represents the intensity. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0060] In one embodiment, a structured light illumination super-resolution image reconstruction method with joint denoising in space-frequency domain is provided, the method comprising:

[0061] Read multiple frames of original images acquired by the SIM imaging system;

[0062] Preprocess the original image to eliminate interference information in the image frequency domain;

[0063] Estimate structured light illumination stripe parameters, including spatial frequency, modulation and initial phase;

[0064] According to the results of structured light illumination stripe parameter estimation, a combined notch filter method is used to build a SIM reconstruction model to achieve frequency domain denoising of super-resolution images.

[0065] For the super-resolution reconstructed image after frequency domain denoising, a regularization method based on Split Bregman optimization is further used for spatial domain denoising.

[0066] Furthermore, in one of the embodiments, the preprocessing of the original image specifically includes:

[0067] The original image is summed and averaged to obtain an equivalent wide-field image, and a sub-image block that does not contain the biological sample is intercepted and the average value of all pixels is accumulated and taken as the background value of the original image. , and subtract it from the original data The original image after background removal is obtained as shown in the following formula:

[0068]

[0069] in, For the Azimuth The original SIM image collected after the phase structured light stripe modulation, is the spatial coordinate of the pixel point, Indicates the operation of intercepting a sub-image block. and are the width and height of the captured sub-image block respectively; (w,h) represents the pixel coordinates, the value range of w is [1,W], and the value range of h is [1,H].

[0070] A sinusoidal grayscale fading function is used to perform boundary fading processing on the original image after background removal, as shown in the following formula:

[0071]

[0072] In the formula, and They are the original images after background removal. The width and height of The width of the faded border area;

[0073] The original image after the boundary fading process is normalized as shown in the following formula:

[0074]

[0075] In the formula, max and min represent the operations of taking the maximum pixel value and the minimum pixel value respectively.

[0076] Furthermore, in one embodiment, estimating structured light illumination stripe parameters specifically includes:

[0077] The optical transfer function (OTF) of the SIM imaging system is constructed based on the system optical parameters and theoretical model;

[0078] The preprocessed original image is subjected to Fourier transform, and the spectrum of each level is roughly separated in the frequency domain. The spectrum of each level obtained is expressed as , and , represents the fringe azimuth; , and They represent the zero-order, 1-order and -1-order spectra after rough separation respectively;

[0079] De-noise the spectrum at each level after rough separation;

[0080] The spatial frequency is estimated by the cross-correlation method based on the correlation of the zero-order and higher-order sub-spectra after noise reduction. ;

[0081] Based on the structured light spatial frequency , perform high-order spectrum shift, and use linear regression method to estimate the structured light modulation and initial phase .

[0082] Further, in one of the embodiments, constructing the optical transfer function OTF of the SIM imaging system according to the system optical parameters and the theoretical model specifically includes:

[0083] Combined with the optical parameters of the SIM system, the OTF is generated according to the theoretical model. The formula is:

[0084]

[0085] in, is the system optical transfer function OTF, k is the frequency domain variable, is the system cutoff frequency, is the numerical aperture of the detection objective, is the emission wavelength, is the aberration compensation coefficient.

[0086] Furthermore, in one of the embodiments, a Gaussian notch filter is used to reduce noise on each level of spectrum after coarse separation.

[0087] Furthermore, in one embodiment, the spatial frequency The calculation formula is:

[0088]

[0089] In the formula, Indicates Azimuth structured light spatial frequency, superscript represents conjugation, represents cross correlation, max represents taking the maximum value of the matrix, represents the sub-pixel position index, represents the Gaussian notch function.

[0090] Furthermore, in one embodiment, the structured light spatial frequency , perform high-order spectrum shift, and use linear regression method to estimate the structured light modulation and initial phase , the specific formula is:

[0091]

[0092] In the formula, represents the first-level spectrum after translation, represents the Gaussian notch function after translation, subscript Indicates the operation of taking the overlapping area. Respectively The imaginary and real parts of .

[0093] Furthermore, in one of the embodiments, the method of constructing a SIM reconstruction model based on the structured light illumination stripe parameter estimation result and implementing frequency domain denoising of the super-resolution image by using a combined notch filter method specifically includes:

[0094] According to the estimated value of the structured light illumination stripe parameter, the pre-processed SIM original image is subjected to spectrum separation and shifting, and the stripe azimuth is , level is The spectrum is expressed as ,in, ;

[0095] Gaussian notch function and Butterworth notch function are used for joint notch processing, and the frequency domain denoising of the super-resolution image is performed in combination with the Wiener filter reconstruction model. The frequency domain result is inverse Fourier transformed to obtain the super-resolution reconstructed image.

[0096] Furthermore, in one of the embodiments, the Gaussian notch function and the Butterworth notch function are used for joint notch processing, and the frequency domain noise reduction of the super-resolution image is performed in combination with the Wiener filter reconstruction model, and the frequency domain result is subjected to inverse Fourier transform to obtain the super-resolution reconstructed image. The specific formula is:

[0097]

[0098] In the formula, represents the super-resolution reconstructed image after frequency domain denoising, represents the inverse Fourier transform, the superscript represents the conjugate value, represents the Butterworth filter function, Indicates translation The Butterworth filter function after represents the Gaussian notch function, Indicates translation The Gaussian notch function after Indicates translation The optical transfer function OTF of the system after that; is the Wiener constant, ; is the apodization function, which is used to suppress the ringing effect in the image reconstruction process.

[0099] Furthermore, in one of the embodiments, the spatial domain denoising is performed using a regularization method based on Split Bregman optimization, and the denoising process is expressed as:

[0100]

[0101] In the formula, is the super-resolution image after denoising in the spatial domain, It is a super-resolution image obtained by inverse Fourier transform from frequency domain to spatial domain after denoising. is a noisy image, is the regularization coefficient, which is used to adjust the coefficient between the data term and the regularization term. represents the gradient function, Indicates taking the L1 norm.

[0102] In one embodiment, a structured light illumination super-resolution image reconstruction system with joint noise reduction in space-frequency domain is provided, the system comprising:

[0103] The first module is used to read multiple frames of original images collected by the SIM imaging system;

[0104] The second module is used to preprocess the original image to eliminate interference information in the image frequency domain;

[0105] The third module is used to estimate the parameters of structured light illumination stripes, including spatial frequency, modulation degree and initial phase;

[0106] The fourth module is used to construct a SIM reconstruction model using a combined notch filter method based on the structured light illumination stripe parameter estimation results to achieve frequency domain denoising of the super-resolution image;

[0107] The fifth module is used to further perform spatial domain denoising on the super-resolution reconstructed image after frequency domain denoising using a regularization method based on SplitBregman optimization.

[0108] For the specific definition of the structured light illumination super-resolution image reconstruction system with joint noise reduction in the space-frequency domain, please refer to the definition of the structured light illumination super-resolution image reconstruction method with joint noise reduction in the space-frequency domain mentioned above, which will not be repeated here. Each module in the structured light illumination super-resolution image reconstruction system with joint noise reduction in the space-frequency domain mentioned above can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0109] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0110] Read multiple frames of original images acquired by the SIM imaging system;

[0111] Preprocess the original image to eliminate interference information in the image frequency domain;

[0112] Estimate structured light illumination stripe parameters, including spatial frequency, modulation and initial phase;

[0113] According to the results of structured light illumination stripe parameter estimation, a combined notch filter method is used to build a SIM reconstruction model to achieve frequency domain denoising of super-resolution images.

[0114] For the super-resolution reconstructed image after frequency domain denoising, a regularization method based on Split Bregman optimization is further used for spatial domain denoising.

[0115] For the specific limitations of each step, please refer to the limitations of the structured light illumination super-resolution image reconstruction method for joint noise reduction in the spatial and frequency domains mentioned above, which will not be repeated here.

[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements:

[0117] Read multiple frames of original images acquired by the SIM imaging system;

[0118] Preprocess the original image to eliminate interference information in the image frequency domain;

[0119] Estimate structured light illumination stripe parameters, including spatial frequency, modulation and initial phase;

[0120] According to the results of structured light illumination stripe parameter estimation, a combined notch filter method is used to build a SIM reconstruction model to achieve frequency domain denoising of super-resolution images.

[0121] For the super-resolution reconstructed image after frequency domain denoising, a regularization method based on Split Bregman optimization is further used for spatial domain denoising.

[0122] For the specific limitations of each step, please refer to the limitations of the structured light illumination super-resolution image reconstruction method for joint noise reduction in the spatial and frequency domains mentioned above, which will not be repeated here.

[0123] The structured light illumination super-resolution image reconstruction method with spatial-frequency domain joint noise reduction provided by the present invention is suitable for the reconstruction of linear structured light illumination super-resolution images (2D-SIM and single-layer 3D-SIM) and the super-resolution image reconstruction of nonlinear structured light illumination microscope (NL-SIM).

[0124] As a specific example, the present invention is further described in detail. Taking 2D-SIM as an example, the specific implementation is as follows:

[0125] In 2D-SIM, the structured light illumination stripes have three azimuth angles, and each azimuth stripe has three equally spaced phases. Azimuth The structured light stripes of the phase are expressed as follows:

[0126]

[0127] in, Respectively represent The modulation degree, spatial frequency and initial phase of the structured light stripes at each azimuth angle. The corresponding SIM original image captured by the camera is:

[0128]

[0129] in, Indicates that the system is The original image captured under the lighting conditions, is the true fluorescence intensity of the sample, is the point spread function, is the noise and defocus information, is the spatial coordinate, Represents a convolution operation.

[0130] The Fourier transform of the collected original image is expressed as follows:

[0131]

[0132] in, The superscript symbol indicates the Fourier transform operation, and They are and The Fourier domain representation of is the frequency domain coordinate, It is the Fourier transform form of the point spread function, called the optical transfer function (OTF) of the system. In conventional SIM imaging systems, fluorescent microsphere sample imaging is usually used for actual measurement and acquisition.

[0133] (1) Original image preprocessing

[0134] The original images collected by the SIM imaging system contain background fluorescence and random noise, such as Figure 3 The background removal method proposed in the present invention is: summing and averaging multiple frames of original images to obtain an equivalent wide-field image, intercepting a sub-image block that does not contain a biological sample, and accumulating and averaging all its pixels as the background value of the original image. , and subtracted from the original data, the specific expression is as follows:

[0135]

[0136] in, For the Azimuth The original SIM image collected after the phase structured light stripe modulation, is the spatial coordinate, To intercept the background area sub-image block operation, W and H are the width and height of the intercepted wide-field image sub-block, is the calculated background value.

[0137] In addition, during the imaging process, the grayscale change characteristics of the boundary part of the camera original image are expressed in the Fourier domain spectrum as follows: Figure 3 The cross-shaped area shown in (b) in FIG. 1 is easy to interfere with the subsequent spectrum separation and parameter estimation. Therefore, a sinusoidal grayscale fading function is used to slow down the grayscale changes of the four boundaries to eliminate their frequency domain feature information, such as Figure 3 As shown in (c) and (d) in the figure, the processing process is expressed as follows:

[0138]

[0139] in, is the original SIM image after background removal. and are the width and height of the original image, The width of the faded border area.

[0140] Finally, the original image after background removal and boundary fading is normalized, which is expressed as follows:

[0141]

[0142] In the formula, max and min represent the operations of taking the maximum and minimum values ​​respectively.

[0143] (2) Structured light parameter estimation

[0144] The system OTF is obtained based on the known system optical parameters and physical model as follows:

[0145]

[0146] in, is the system cutoff frequency, is the numerical aperture of the detection objective, is the emission wavelength, It is the aberration compensation coefficient and can be set manually according to the actual imaging situation.

[0147] The pre-processed raw data is Fourier transformed into the frequency domain. First, the structured light parameters under ideal conditions are used to perform rough separation of the frequency spectra at each level. The process is expressed as follows:

[0148]

[0149] It is worth noting that the separated levels carry noise terms, but the traditional SIM reconstruction algorithm chooses to ignore the noise terms here in the subsequent processing, resulting in artifacts in the reconstruction results. In order to effectively remove noise interference and improve the accuracy of parameter estimation, a Gaussian notch filter is used to notch the spectrum of each level.

[0150]

[0151] in, and They determine the strength and range of the notch respectively.

[0152] make and represent the zero-order, 1-order and -1-order spectra after rough separation, respectively. According to the correlation between the zero-order spectrum component and the higher-order components, the sub-pixel position with the highest correlation is solved by cross-correlation iteration to obtain the first Azimuth structured light spatial frequency , the calculation process is as follows:

[0153]

[0154] Among them, the superscript represents conjugation, represents cross correlation, max represents the maximum value of the matrix, and index represents the sub-pixel position index.

[0155] Based on the estimated results of the spatial frequency of the structured light at various azimuth angles, the higher-order spectrum is shifted. According to the linear relationship between the overlapping areas of the zero-order and higher-order spectra after the shift, the linear regression method can be used to calculate the value of the structured light modulation and initial phase. The specific calculation process is expressed as follows:

[0156]

[0157] Among them, the subscript Represents the operation of taking overlapping areas.

[0158] (3) Super-resolution reconstruction and frequency domain noise reduction

[0159] Based on the estimated structured light fringe parameters, the preprocessed original image is separated. According to the conventional reconstruction model based on the Wiener filter method, directly using the separation level containing noise terms for reconstruction is likely to lead to the appearance of complex artifacts in the reconstructed image, such as Figure 4 This is shown in (a) and (e), especially under low signal-to-noise ratio and thick sample conditions.

[0160] The use of Gaussian notch filter to modulate the corresponding order of OTF has a certain effect in frequency domain noise reduction and can reduce some artifacts in super-resolution images. However, due to the narrowband characteristics of the Gaussian notch function, gully-like areas are prone to appear in the reconstructed spectrum, such as Figure 4 As shown in (j) in the figure, these spectral features are transformed into the image domain and appear as complex artifacts and background interference, which also affects the reconstruction quality. In order to avoid introducing new artifacts and interference while performing the notch operation, the present invention proposes a combined notch method of Gaussian notch and Butterworth notch. The Butterworth filter has a very smooth frequency transition characteristic without ringing, which can suppress the sidelobe protrusion caused by the Gaussian notch filter, so that the spectrum of the notch area and other areas can achieve a natural and smooth transition, such as Figure 4 As shown in (c) and (h), the noise and defocus information are effectively removed, and the spectrum loss introduced by the Gaussian notch filter is compensated. The reconstruction process is expressed as follows:

[0161]

[0162] in, are the levels of spectrum separation based on the estimated values ​​of structured light parameters; represents the conjugate value, the subscript Indicates the azimuth of the structured light fringes, subscript represents the spectrum order, Indicates The spatial frequency of the azimuthal fringes, is the Wiener constant, which can be set manually; and They are Gaussian notch filter and Butterworth notch filter respectively. and is its cutoff frequency and order, which respectively represent the steepness of the filter attenuation region and the frequency position where attenuation begins; is the apodization function, which is used to suppress the ringing effect in the image reconstruction process.

[0163] (4) Super-resolution image spatial domain denoising

[0164] Due to the complexity of the system and the characteristics of the sample, the noise types contained in the original images collected by the SIM imaging system are relatively complex, and the frequency domain noise reduction process based on the combined notch filter cannot completely eliminate the noise interference in the image, such as Figure 4Therefore, this method proposes to perform further denoising in the spatial domain, using the total variation (TV) regularization of the sparse solution operator L1 norm to implement denoising, which can effectively remove the noise in the image while retaining the image detail information. The denoising process is expressed as:

[0165]

[0166] in, is the super-resolution image after denoising in the spatial domain, It is a super-resolution image obtained by inverse Fourier transform from frequency domain to spatial domain after denoising. is a noisy image, is the regularization coefficient, which is used to adjust the coefficient between the data term and the regularization term. represents the gradient function, Indicates taking the L1 norm.

[0167] In order to solve the above optimization process more quickly and accurately, this method proposes to use the Split Bregman iteration method to find the optimal solution. The superscript numbers represent the current iteration number. The iteration process is as follows:

[0168]

[0169] When satisfied When it is less than the set tolerance, the iteration stops. This is the optimized denoised image, otherwise the next iteration is performed. The final reconstructed super-resolution image is as follows Figure 4 As shown in (c) and (h) in Figure 4 As shown in (k) in .

[0170] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A structured light illumination super-resolution image reconstruction method with joint denoising in spatial and frequency domains, characterized in that: The method comprises: Read multiple frames of original images acquired by the SIM imaging system; Preprocess the original image to eliminate interference information in the image frequency domain; Estimate structured light illumination stripe parameters, including spatial frequency, modulation and initial phase; According to the results of structured light illumination stripe parameter estimation, a combined notch filter method is used to build a SIM reconstruction model to achieve frequency domain denoising of super-resolution images. For the super-resolution reconstructed image after frequency domain denoising, a regularization method based on Split Bregman optimization is further used to perform spatial domain denoising; The estimating structured light illumination stripe parameters specifically includes: The optical transfer function (OTF) of the SIM imaging system is constructed based on the system optical parameters and theoretical model; The preprocessed original image is subjected to Fourier transform, and the spectrum of each level is roughly separated in the frequency domain. The spectrum of each level obtained is expressed as , and , represents the fringe azimuth; , and They represent the zero-order, 1-order and -1-order spectra after rough separation respectively; De-noise the spectrum at each level after rough separation; The cross-correlation method is used to estimate the spatial frequency of the structured light based on the correlation between the zero-order and higher-order spectra after noise reduction. ; Based on the structured light spatial frequency , perform high-level sub-spectrum shift, and use linear regression method to estimate the structured light modulation and initial phase ; The method of constructing a SIM reconstruction model based on the result of the structured light illumination stripe parameter estimation and implementing frequency domain noise reduction of the super-resolution image specifically includes: According to the estimated value of the structured light illumination stripe parameter, the pre-processed SIM original image is subjected to spectrum separation and shifting, and the stripe orientation is , level is The spectrum is expressed as ,in, ; Gaussian notch function and Butterworth notch function are used for joint notch processing, and the frequency domain noise reduction of super-resolution images is performed in combination with the Wiener filter reconstruction model, and the frequency domain results are inversely transformed by Fourier to obtain super-resolution reconstructed images; The Gaussian notch function and the Butterworth notch function are used for joint notch processing, and the frequency domain noise reduction of the super-resolution image is performed in combination with the Wiener filter reconstruction model, and the frequency domain result is inverse Fourier transformed to obtain the super-resolution reconstructed image. The specific formula is: ； In the formula, represents the super-resolution reconstructed image after frequency domain denoising, represents the inverse Fourier transform, the superscript represents the conjugate value, represents the Butterworth filter function, Indicates translation The Butterworth filter function after represents the Gaussian notch function, Indicates translation The Gaussian notch function after Indicates translation The optical transfer function OTF of the system after that; is the Wiener constant, ; is the apodization function, which is used to suppress the ringing effect in the image reconstruction process; The regularization method based on Split Bregman optimization is used to perform spatial domain denoising, and the denoising process is expressed as: ； In the formula, is the super-resolution image after denoising in the spatial domain, It is a super-resolution image obtained by inverse Fourier transform from frequency domain to spatial domain after denoising. is a noisy image, is the regularization coefficient, which is used to adjust the coefficient between the data term and the regularization term. represents the gradient function, Indicates taking the L1 norm.

2. The structured light illumination super-resolution image reconstruction method with spatial-frequency domain joint noise reduction according to claim 1 is characterized in that: The preprocessing of the original image specifically includes: The original image is summed and averaged to obtain an equivalent wide-field image, and a sub-image block that does not contain the biological sample is intercepted and the average value of all pixels is accumulated and taken as the background value of the original image. , and subtract it from the original data The original image after background removal is obtained as shown in the following formula: ； in, For the Azimuth The original SIM image collected after the phase structured light stripe modulation, is the spatial coordinate of the pixel point, Indicates the operation of intercepting a sub-image block. and are the width and height of the captured sub-image block respectively; the value range of w is [1, W], and the value range of h is [1, H]; A sinusoidal grayscale fading function is used to perform boundary fading processing on the original image after background removal, as shown in the following formula: ； In the formula, and They are the original images after background removal. The width and height of The width of the faded border area; The original image after the boundary fading process is normalized as shown in the following formula: ； In the formula, and Represents the operations of taking the maximum pixel value and the minimum pixel value respectively.

3. The structured light illumination super-resolution image reconstruction method with joint denoising in spatial and frequency domains according to claim 1, characterized in that: The optical transfer function OTF of the SIM imaging system is constructed according to the system optical parameters and the theoretical model, specifically including: Combined with the optical parameters of the SIM system, the OTF is generated according to the theoretical model. The formula is: ； in, is the system optical transfer function OTF, k is the frequency domain variable, is the system cutoff frequency, is the numerical aperture of the detection objective, is the emission wavelength, is the aberration compensation coefficient.

4. The structured light illumination super-resolution image reconstruction method with joint denoising in spatial and frequency domains according to claim 1, characterized in that: A Gaussian notch filter is used to reduce the noise of each frequency spectrum after rough separation.

5. The structured light illumination super-resolution image reconstruction method with spatial-frequency domain joint noise reduction according to claim 3 is characterized in that: The structured light spatial frequency The calculation formula is: ； In the formula, Indicates The spatial frequency of structured light at azimuth angles, superscript represents conjugation, represents cross correlation, max represents taking the maximum value of the matrix, represents the sub-pixel position index, represents the Gaussian notch function.

6. The structured light illumination super-resolution image reconstruction method with spatial-frequency domain joint noise reduction according to claim 5, characterized in that: The structured light spatial frequency , perform high-level sub-spectrum shift, and use linear regression method to estimate the structured light modulation and initial phase , the specific formula is: ； In the formula, represents the first-level spectrum after translation, represents the Gaussian notch function after translation, subscript Indicates the operation of taking the overlapping area. Respectively The imaginary and real parts of .

Images