A structured light microscopic illumination parameter estimation method and system based on high signal-to-noise ratio spectrum extraction

Abstract

This invention discloses a method and system for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio (SNR) spectrum extraction. The method employs a structured light microscopy imaging system to illuminate a sample with structured light from multiple directions, acquiring multiple raw images with different initial phases under each illumination direction. Each raw image is converted to frequency domain data; frequency domain analysis is performed to separate spectral components; integer pixel alignment is performed on the 0th and +1st order spectra; frequency domain Fourier ring correlation analysis is performed on the +1st order spectral component to obtain a quantitative evaluation result of frequency domain consistency at different spatial frequency domains, determining the frequency range that meets the preset high SNR condition, and constructing a high SNR spectral region in the frequency domain based on this frequency range; partial correlation analysis is performed based on the high SNR spectral region; and the structured light illumination parameters are determined based on the partial correlation analysis. This invention ensures the accuracy and robustness of illumination parameter estimation, reduces computational complexity, improves efficiency, and can meet the requirements of real-time or near-real-time imaging.

Description

Technical Field

[0001] This invention relates to a method for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio spectrum extraction, and also to a system using this method. Background Technology

[0002] Fluorescence resonance energy transfer microscopy (FRET) enables real-time quantitative analysis of biomolecular interactions at the 10 nm scale in living cells. The FRET effect typically occurs within a distance of approximately 1–10 nm between donor and acceptor fluorescent molecules. This distance scale is comparable to the thickness of biomolecules, intracellular proteins, and biomembranes. Therefore, FRET microscopy has significant applications in studying biomolecular interactions, protein conformational changes, and protein regulatory networks in signaling pathways in cell biology. However, traditional wide-field FRET techniques are limited by the optical diffraction limit, with spatial resolution typically struggling to exceed approximately 200 nm. This makes it difficult to effectively distinguish and resolve molecular interaction signals located in the sub-diffraction scale region, thus limiting the application of FRET in resolving fine spatial structures. Structured light super-resolution microscopy (SR-SIM), by introducing structured light illumination with known spatial frequencies and phases onto the sample and reconstructing multiple modulated images, can improve the spatial resolution of microscopy to approximately twice the diffraction limit without significantly increasing phototoxicity and photobleaching. The SIM-FRET imaging technique, which combines SIM technology with FRET microscopy, can significantly improve spatial resolution while maintaining the quantitative capability of FRET, thereby revealing the spatial distribution characteristics and interaction mechanisms of intracellular biomolecules more accurately.

[0003] However, SIM-FRET imaging technology suffers from the following problems in application: the traditional structured light illumination parameter estimation step is time-consuming; cumbersome post-processing steps lead to excessively long image processing time for SIM-FRET, greatly limiting its application in live-cell dynamic quantitative analysis; in the image reconstruction and quantitative analysis process of SIM-FRET microscopy, the overall processing flow typically includes multiple steps such as frequency domain transformation, super-resolution structured light reconstruction, and FRET parameter calculation. Among these, the estimation of structured light illumination parameters is a key prerequisite for achieving high-quality super-resolution reconstruction and reliable FRET quantitative analysis. Existing SIM-FRET parameter estimation methods usually require multiple iterations of correlation analysis in the frequency domain to obtain accurate illumination parameters. This process involves a large number of Fourier transforms and correlation calculations, resulting in high computational complexity and long processing time. Practical applications show that in the complete SIM-FRET image processing flow, the parameter estimation step often occupies the majority of the computation time and is one of the core bottlenecks leading to slow overall reconstruction speed, thus limiting the application of SIM-FRET technology in real-time dynamic quantitative analysis of live cells.

[0004] Various methods have emerged to solve the above problems, but these methods have the following drawbacks:

[0005] (1) One type is the fast estimation method based on spectral features, which realizes wave vector localization through frequency domain peak detection, phase relationship and other methods. The calculation speed is relatively fast, but it depends on the clear separability of ±1 order spectral components. It is easily affected by noise interference under low signal-to-noise ratio or high illumination spatial frequency conditions, which leads to a decrease in estimation accuracy and stability.

[0006] (2) Another type is the iterative search method based on frequency domain cross-correlation, such as the Iterative Cross-Correlation (COR) algorithm, which can iteratively search wave vector coordinates with sub-pixel accuracy. It still has high accuracy and robustness under low signal-to-noise ratio and is widely used in practical applications. However, although the COR algorithm has high accuracy, its computational burden is extremely high. This is because the algorithm needs to perform complex multiplication and addition operations on all frequency points of the entire full-size spectrum at each sub-pixel candidate position in the common area. For SIM imaging, hundreds of full-size cross-correlation calculations are required in each direction; and for three-channel SIM-FRET imaging, it is three times that. This makes it take several seconds to tens of seconds to process a single frame of image, even on modern CPUs, which is completely unable to meet the real-time requirements for observing cell dynamics.

[0007] To accelerate processing, current research is mainly focused on the following two directions:

[0008] 1. Algorithm simplification: Fast parameter estimation algorithms such as dCOR (bisection cross-correlation) have emerged, but these methods either have reduced accuracy under low signal-to-noise ratio and high illumination frequency, or still require a certain amount of iterative search, making it difficult to balance accuracy and robustness.

[0009] 2. Hardware Acceleration: Utilizing the parallel computing capabilities of graphics processing units (GPUs) to accelerate traditional algorithms such as COR. For example, a patent (CN119006281A) discloses a SIM-FRET reconstruction system based on GPU parallel acceleration, which improves the overall processing speed through CPU-GPU heterogeneous computing, asynchronous data transmission, and multi-threaded scheduling. However, this type of solution is essentially a "hard acceleration" of the original computation process and does not reduce the inherent redundancy of the computation itself from the algorithm's fundamental principles. Even with GPU optimization, meaningless calculations for a large number of low signal-to-noise ratio and high-frequency components still exist in traditional COR algorithms, limiting further improvements in speedup and the ceiling of real-time performance. Summary of the Invention

[0010] The first objective of this invention is to provide a structured light microscopy illumination parameter estimation method based on high signal-to-noise ratio spectrum extraction. This method can significantly reduce computational complexity and improve parameter estimation efficiency while ensuring or improving the accuracy and robustness of illumination parameter estimation, thus meeting the requirements of real-time or near-real-time imaging.

[0011] The first objective of this invention is achieved through the following technical measures: a method for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio spectrum extraction, characterized by comprising the following steps:

[0012] S1. The sample is illuminated by structured light microscopy system in multiple directions, and multiple original images with different initial phases are acquired under each illumination direction. There is a preset phase difference relationship between the original images with different initial phases.

[0013] S2. Perform a two-dimensional Fourier transform on each of the acquired raw images to convert them into frequency domain data;

[0014] S3. Based on the phase difference relationship between the original images with different initial phases, the frequency domain data is decoupled to separate the 0th order spectral component and the +1st order spectral component or -1st order spectral component for each illumination direction.

[0015] S4, by locating the +1st order spectral component The peak value determines the integer pixel components of the illumination wave vector. and will Translation This makes it comparable to [previous device] in integer pixel precision. Center-aligned, the aligned spectrum is obtained. Then, frequency domain Fourier ring correlation analysis is performed on the aligned +1 order spectral components to obtain quantitative evaluation results of frequency domain consistency at different spatial frequency domains. Based on the evaluation results, the frequency range that meets the preset high signal-to-noise ratio condition is determined, and the high signal-to-noise ratio spectral region in the frequency domain is constructed using this frequency range.

[0016] S5. Within the obtained high signal-to-noise ratio spectrum region, perform partial correlation analysis on the 0th-order spectral component and the +1st-order spectral component.

[0017] S6. Based on the optimal results of partial correlation analysis, determine the structured light illumination parameters.

[0018] This invention proposes a partial cross-correlation parameter estimation method based on high signal-to-noise ratio (SNR) spectrum extraction. It abandons the traditional approach of "performing full cross-correlation calculations on the complete spectrum and then accelerating the process with hardware." Instead, it addresses the essence of spectral information reliability by performing high SNR reliability analysis on the frequency domain data before parameter estimation. This identifies and selects only high SNR regions in the spectrum that actually contribute to parameter estimation, actively excluding frequency components with excessively low SNR. Correlation analysis and parameter search are performed only within high SNR spectral regions. This significantly reduces the amount of invalid data involved in cross-correlation calculations from the algorithm's source. While ensuring or improving the accuracy and robustness of lighting parameter estimation, it significantly reduces computational complexity and improves parameter estimation efficiency, meeting the needs of real-time or near-real-time imaging.

[0019] In step S2 of this invention, the formula for performing the two-dimensional Fourier transform is as follows:

[0020]

[0021] in, For the original fluorescence image The spectrum signal after Fourier transform; X = DD, DA, AA, representing donor excitation of donor emission channel DD, donor excitation of acceptor emission channel DA, and acceptor excitation of acceptor emission channel AA, respectively; θ = 1, 2, 3: representing different directions; n = -1, 0, 1: representing different phases; r is the spatial coordinate of the original image.

[0022] In step S5 of this invention, the formula for analysis is as follows:

[0023]

[0024] In this calculation, the numerator represents the cross-correlation terms; the denominator is used as a normalization factor. The coordinates are the corresponding frequency spectrum. For the integer pixel components of the illumination wave vector, used to shift the +1 order spectrum to align with the 0 order spectrum; and It is the signal power of the corresponding spectrum at that frequency point; This represents the set of frequency domain coordinates corresponding to the r-th ring band;

[0025] The calculation formula is as follows:

[0026]

[0027] Now, assuming the noise is under independent statistics, the FRC value is a monotonic function of SNR, and its expression is:

[0028] .

[0029] In step S6 of the present invention, the high signal-to-noise ratio spectral region is calculated using the following formula:

[0030]

[0031] Where L is the side length of the high signal-to-noise ratio region; It is the effective cutoff radius obtained based on Fourier ring correlation screening, representing the maximum radial range of high signal-to-noise ratio signal in the spectrum; N is the size of the complete spectrum.

[0032] In step S7 of this invention, the partial correlation analysis includes the following steps:

[0033] (1) Construct multiple candidate parameters within the sub-pixel search range of the illumination wave vector;

[0034] (2) For each candidate parameter, the corresponding correlation evaluation value is calculated only in the high signal-to-noise ratio spectrum region;

[0035] (3) By comparing the correlation evaluation values ​​corresponding to different candidate parameters, the parameter values ​​that make the correlation optimal are determined.

[0036] In step S7⑶ of the present invention, the optimal parameter values ​​are calculated using the following formula:

[0037]

[0038] in, It is a candidate subpixel vector The corresponding partial cross-correlation value; m is the modulation depth of the structured lighting; L is the side length of the high signal-to-noise ratio region; These are the frequency domain coordinates within that region; It is a 0th order spectrum. It is its complex conjugate; It is a +1 order spectrum; These are the integer pixel components of the illumination wave vector; It is a candidate subpixel vector; It is the phase factor corresponding to the initial phase φ0.

[0039] The structured light illumination parameters described in this invention are the illumination wave vector, initial phase, and modulation depth.

[0040] In step S1 of this invention, after the original data acquisition is completed, the original image is preprocessed by one or more of the following methods: background subtraction or dark current correction of the original image, intensity normalization of images acquired from different channels or at different times, noise suppression of the image, registration of multiple frames, and subpixel calibration of dual-channel images, and then the process proceeds to step S2.

[0041] The second objective of this invention is to provide a system for estimating structured light microscopy illumination parameters using the above-described method based on high signal-to-noise ratio spectrum extraction.

[0042] The second objective of this invention is achieved through the following technical measures: a system using the above-described method for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio spectral extraction, characterized by comprising:

[0043] The data acquisition module acquires raw images of multi-directional, multi-phase structured light and performs preprocessing.

[0044] The frequency domain analysis module performs a Fourier transform on the preprocessed original image to convert it to frequency domain data, and then separates the 0th-order and ±1st-order spectral components.

[0045] The high signal-to-noise ratio (SNR) evaluation module evaluates the SNR of frequency domain data based on spectral amplitude or energy distribution to construct a high SNR spectral region in the frequency domain.

[0046] The partial correlation calculation module performs partial correlation analysis on the 0th and +1st order spectral components in the high signal-to-noise ratio spectral region.

[0047] The parameter output module determines and outputs the structured light illumination parameters based on the optimal results of partial correlation analysis.

[0048] The computational tasks of each module in this invention are executed in parallel using parallel computing units.

[0049] Compared with the prior art, the present invention has the following significant effects: (1) This invention adopts a high-reliability spectrum region determination mechanism based on high signal-to-noise ratio (SNR) evaluation of the spectrum. That is, it no longer assumes that the entire common frequency domain is applicable to parameter estimation, but introduces a high SNR evaluation mechanism to determine the high-reliability spectrum support region. This realizes the active screening of high SNR information in the spectrum and the automatic suppression of low SNR regions, which is the basis for subsequent improvement of computational efficiency and robustness.

[0050] (2) This invention employs a partial correlation parameter estimation method based on high signal-to-noise ratio (SNR) spectral regions. Specifically, it performs correlation calculations only within the high SNR spectral regions, rather than performing full cross-correlation calculations on the entire spectrum. By limiting the scope of correlation analysis from the entire spectrum to a defined high SNR region, the computational load is significantly reduced, achieving a transformation in parameter estimation complexity from O(N²) to O(L²). This achieves an order-of-magnitude improvement in parameter estimation speed, breaking through the bottleneck of real-time imaging.

[0051] (3) The high signal-to-noise ratio spectral region of the present invention is not fixed, but is determined according to the spectral characteristics of the current imaging data. That is, under imaging conditions with a high signal-to-noise ratio, the high signal-to-noise ratio spectral region automatically expands to make full use of the effective spectral information; under conditions with a low signal-to-noise ratio or enhanced noise, the high signal-to-noise ratio spectral region automatically shrinks to avoid noise spectrum interference with parameter estimation. This mechanism significantly improves the stability and universality of the method under different sample types, imaging conditions and noise levels.

[0052] (4) This invention is not only applicable to single-channel structured light microscopy imaging, but can also be extended to: multi-directional, multi-phase SIM imaging; multi-channel SIM-FRET imaging, including multiple excitation / emission channels for donor and acceptor; and implementation methods for parameter sharing or independent estimation under multi-channel conditions. This invention can still significantly reduce the overall parameter estimation time in multi-channel SIM-FRET scenarios, providing a feasible algorithmic basis for real-time or near-real-time SIM-FRET quantitative imaging systems.

[0053] (5) The computing module of the system of the present invention can be deployed on CPU, GPU or other parallel computing platforms, and real-time processing is achieved through task scheduling and data transmission. Since pCOR greatly reduces the amount of computation at the algorithm level, it reduces the strong dependence on dedicated parallel computing hardware (such as high-end GPUs), broadens the application scenarios, and makes it possible to achieve near real-time super-resolution image processing on low-to-medium configuration computers or even embedded processors, which is conducive to the development of low-cost, portable microscopic imaging devices.

[0054] (6) Based on the high efficiency of the pCOR algorithm itself, this invention provides a better algorithmic foundation for deeper system-level optimization (such as multi-threaded pipelines and GPU acceleration) and unlocks greater performance potential. Attached Figure Description

[0055] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0056] Figure 1 is an overall flowchart of the method of the present invention;

[0057] Figure 2This is a flowchart comparing the COR algorithm and the pCOR algorithm;

[0058] Figure 3 This is a flowchart of the image data processing of the SIM-FRET real-time imaging system;

[0059] Figure 4 This is a comparison chart of the processing time of each step of the COR algorithm and the pCOR algorithm under the CPU-GPU joint platform;

[0060] Figure 5 This is a comparison chart of the COR algorithm and the pCOR algorithm in multi-task parallel processing on a CPU-GPU joint platform;

[0061] Figure 6 These are comparison images of super-resolution SIM imaging;

[0062] Figure 7 This is a comparison chart of FRET quantitative analysis results. Detailed Implementation

[0063] In existing technologies, the estimation of structured light illumination parameters is typically based on performing cross-correlation analysis or similar correlation calculations within a complete common frequency domain defined by the system cutoff frequency. However, through in-depth research, the inventors have discovered that not all frequency components in the complete common frequency domain effectively contribute to the estimation of illumination parameters. In particular, the high-frequency region near the system cutoff frequency suffers from low signal-to-noise ratios due to significant attenuation of the optical transfer function and increased noise, resulting in unreliable spectral information. These low signal-to-noise ratio spectral components not only fail to provide effective information during parameter estimation but may also introduce noise interference and cause a large number of invalid or redundant calculations, thus becoming one of the main factors limiting the speed and stability of parameter estimation.

[0064] This invention abandons the traditional approach of "performing full cross-correlation calculations on the complete spectrum and then accelerating the process with hardware," and instead proposes a method for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio spectrum extraction, starting from the essence of spectral information reliability. Figures 1-7 As shown, the method includes the following steps:

[0065] S1. The sample is illuminated by structured light microscopy system in multiple directions, and multiple original images with different initial phases are acquired under each illumination direction. There is a preset phase difference relationship between the original images with different initial phases.

[0066] In this embodiment, a structured light-fluorescence resonance energy transfer (SIM-FRET) microscopy imaging system is used to image the sample. The microscopy imaging system includes an FPGA control module, a structured light illumination module, an excitation and emission optical path switching module, and an image acquisition module. During imaging, a periodic structured light pattern with a preset spatial frequency and direction is projected onto the sample using the structured light illumination module. Under each illumination direction, the initial phase of the structured light pattern is controlled to change, thereby acquiring multiple raw images with different initial phases. The above process can be completed under different excitation / emission channels of the donor and acceptor to obtain raw data for subsequent SIM-FRET reconstruction and analysis. The acquired raw images are transmitted in real time to a circular buffer in a PC host computer. When the buffer accumulates 9 frames (SIM mode) or 27 frames (SIM-FRET mode) of data, the subsequent preprocessing process is triggered to avoid timing conflicts between data transmission and computation.

[0067] After the raw data acquisition is completed, the raw images are preprocessed to improve the stability and accuracy of subsequent parameter estimation and image reconstruction. Image preprocessing includes, but is not limited to, one or more of the following operations:

[0068] ① Perform background subtraction or dark current correction on the original image;

[0069] ② Perform intensity normalization processing on images acquired from different channels or at different times;

[0070] ③ Perform noise suppression processing on the image, such as using spatial filtering or frequency domain filtering methods;

[0071] ④ Perform registration processing on multiple frames of images to compensate for the effects of sample drift or system jitter;

[0072] ⑤ Perform sub-pixel calibration on the dual-channel image to compensate for the effects caused by mechanical errors in the optical path.

[0073] The preprocessed image is used as input for the parameter estimation step.

[0074] S2. Perform a two-dimensional Fourier transform on each acquired raw image to convert it to frequency domain data, as shown in the following formula:

[0075]

[0076] in, For the original fluorescence image The spectrum signal after Fourier transform; X = DD, DA, AA, representing donor excitation of donor emission channel DD, donor excitation of acceptor emission channel DA, and acceptor excitation of acceptor emission channel AA, respectively; θ (=1,2,3) and n (= -1,0,1) represent different directions and different phases, respectively; r is the spatial coordinate of the original image.

[0077] S3. Based on the phase difference relationship between the original images with different initial phases, the frequency domain data is decoupled to separate the 0th-order spectral component and the +1st-order or -1st-order spectral component for each illumination direction; that is, the structured light modulation spectral component for each illumination direction is separated. The structured light modulation spectral component consists of a 0th-order spectral component and a first-order spectral component, which is either a +1st-order or -1st-order spectral component. Among them, the 0th-order spectral component mainly contains the low-frequency information of the sample, and the ±1st-order spectral components contain the high-frequency information after structured light modulation.

[0078] For example, the following formula can be used to separate the spectrograms of three original images with different initial phases (known to have a phase difference of 2π / 3) under the same illumination direction:

[0079]

[0080] in,

[0081]

[0082] Where i is an imaginary number; It is the mid-to-low frequency range within the cutoff frequency; , It includes both the high-frequency spectrum outside the cutoff frequency and a portion of the mid-to-low frequency spectrum within the cutoff frequency.

[0083] S4, by locating the +1st order spectral component The peak value determines the integer pixel components of the illumination wave vector. and will Translation This makes it comparable to [previous device] in integer pixel precision. Center-aligned, the aligned spectrum is obtained. ;

[0084] S5. Perform frequency domain Fourier ring correlation analysis on the aligned +1st order spectral components to obtain a quantitative evaluation result of spectral consistency at different spatial frequencies, as shown in the following formula:

[0085]

[0086] In this calculation, the numerator represents the cross-correlation terms; the denominator is used as a normalization factor. The coordinates are the corresponding frequency spectrum. For the integer pixel components of the illumination wave vector, used to shift the +1 order spectrum to align with the 0 order spectrum; and It is the signal power of the corresponding spectrum at that frequency point; This represents the set of frequency domain coordinates corresponding to the r-th ring band (covering a specific spatial frequency range). The calculation formula is as follows (N represents the size of the complete frequency domain):

[0087]

[0088] Now, assuming the noise is under independent statistics, the FRC value is a monotonic function of SNR, and its expression is:

[0089]

[0090] S6. Determine the frequency range that meets the preset high signal-to-noise ratio condition based on the evaluation results, and use this frequency range...

[0091] Constructing a high signal-to-noise ratio spectrum region in the frequency domain:

[0092]

[0093] Where L is the side length of the high signal-to-noise ratio (SNR) region (corresponding to the pixel dimension of the square effective region in the spectrum). It is the effective cutoff radius obtained based on Fourier ring correlation (FRC) filtering, representing the maximum radial range of high signal-to-noise ratio of the signal in the spectrum; N is the size of the complete spectrum (consistent with the pixel resolution of the original image, such as N=512 for 512×512 pixels in this invention).

[0094] S7. Within the obtained high signal-to-noise ratio spectral region, perform partial correlation analysis on the 0th-order spectral component and the +1st-order spectral component:

[0095] (1) Construct multiple candidate parameters within the sub-pixel search range of the illumination wave vector;

[0096] (2) For each candidate parameter, the corresponding correlation evaluation value is calculated only in the high signal-to-noise ratio spectrum region;

[0097] (3) By comparing the correlation evaluation values ​​corresponding to different candidate parameters, determine the parameter values ​​that achieve the optimal correlation:

[0098]

[0099] in, It is a candidate subpixel vector The corresponding partial cross-correlation value (used to determine the accuracy of the sub-pixel vector); m is the modulation depth of the structured lighting (used to describe the brightness contrast of the lighting pattern); L is the side length of the high signal-to-noise ratio region; These are the frequency domain coordinates within that region; It is the 0th order spectrum (containing low-frequency information of the sample). It is its complex conjugate; It is the +1 order spectrum (containing high-frequency information introduced by lighting). It is the integer pixel component of the illumination wave vector (for integer translation alignment of the +1 order spectrum). It is a candidate subpixel vector (used for fine correction of wave vectors); It is the phase factor corresponding to the initial phase φ0.

[0100] S8. Based on the optimal results of partial correlation analysis, determine the structured light illumination parameters, including the illumination wave vector, initial phase, and modulation depth.

[0101] The parameter estimation step is controlled and task-scheduled by the central processing unit (CPU), which can transfer relevant computational tasks and data to parallel computing units for execution. These parallel computing units can be graphics processing units (GPUs) or other parallel computing platforms. By utilizing parallel computing units to accelerate computationally intensive operations such as frequency domain transformation and correlation analysis, the processing speed of parameter estimation can be further improved, thereby meeting the requirements of real-time or near-real-time SIM-FRET imaging.

[0102] The structured light illumination parameters are output for subsequent super-resolution image reconstruction processing. The reconstruction process is as follows:

[0103] After obtaining accurate structured light illumination parameters, the frequency domain data is processed using these parameters, including spectral shifting and spectral overlap. A super-resolution image is then generated using a pre-defined super-resolution reconstruction algorithm. The initial phase is estimated based on the structured light illumination parameters. Illumination frequency wave vector and The separated image is obtained by performing spectral separation on the image, and the formula is as follows:

[0104]

[0105]

[0106] in , , After spectral separation, it contains the 0th-order low-to-mid-range spectrum and ±1-order high-frequency spectrum. θ (=1,2,3) and n (=-1,0,1) represent different orientation angles and phase differences of the image. These are the optical transfer functions of different FRET channels with corresponding frequency shifts, generated through separation. In the image, X = DD, DA, AA are the reconstructed FRET single-channel super-resolution images, and ifft represents the inverse Fourier transform. Let w represent the Gaussian apodization function, w represent the Wiener filter parameters, and r represent the position coordinates in the image spatial domain.

[0107] After performing the same calculations three times, the super-resolution SIM three-channel image was obtained. , , .

[0108] Image reconstruction algorithms can include Wiener filtering, regularized reconstruction, or other super-resolution image reconstruction methods.

[0109] The image reconstruction step is controlled and task-scheduled by the central processing unit (CPU), which can transfer relevant computational tasks and data to parallel computing units for execution. These parallel computing units can be graphics processing units (GPUs) or other parallel computing platforms. By utilizing parallel computing units to accelerate computationally intensive operations such as frequency domain transformation and correlation analysis, the processing speed of image reconstruction can be further improved, thereby meeting the requirements of real-time or near-real-time SIM-FRET imaging.

[0110] Based on the reconstructed super-resolution SIM images, the data from the donor and acceptor channels are further processed to achieve quantitative FRET analysis. The workflow is as follows:

[0111] ① Background correction and intensity masking: This removes non-specific signals from the image, such as camera noise. The formula is as follows:

[0112]

[0113]

[0114] in The value represents the image background value, which can be set manually or corresponds to the pixel value with the highest number of pixels in the histogram of grayscale values ​​from 8 to 1500. This represents the corrected image after background subtraction; mask represents an intensity mask based on a background threshold. This represents the background intensity value of the image. It can be set manually or be the pixel value corresponding to the highest number of pixels in the histogram of grayscale values ​​from 0 to 5000. An empirical coefficient is set to 3.

[0115] ② FRET quantitative analysis: The formula is as follows:

[0116]

[0117]

[0118]

[0119] Among them, at this time To remove the background , The intensity of fluorescence emission due to receptor sensitization. For donor-centric apparent efficiency, G represents the donor-recipient concentration ratio, k represents the sensitization-quenching conversion factor, and a, b, c, and d represent the system crosstalk coefficients. Specifically, the sensitization-quenching conversion factor G, the donor-recipient concentration conversion factor k, and the system crosstalk coefficients a, b, c, and d can be determined by preparing two standard plasmid samples with different fixed FRET efficiencies and a donor-recipient concentration ratio of 1:1, as well as by transfecting donor and recipient plasmid samples separately. The system parameters a, b, c, d, G, and K will vary depending on the FRET donor-recipient and FRET measurement system. In this embodiment, the values ​​were pre-measured to obtain a = 0.04, b = 0.000912663, c = 0.001961017, d = 0.054721272, G = 0.381640369, and k = 2.156691882.

[0120] Based on the imaging results of donors and acceptors under different excitation and emission conditions, FRET efficiency, donor-acceptor ratio, or other parameters related to molecular interactions are calculated, thereby enabling quantitative analysis of the interaction state of biomolecules in the sample.

[0121] Image display formats may include pseudo-color images, intensity distribution maps, or statistical result charts. The apparent efficiency centered on the donor is calculated using the above methods. donor-recipient concentration ratio The pixel values ​​are normalized, and the grayscale image is mapped to a color image using the Parula color map table provided in the OpenCV image library to generate a simulated efficiency map effect, which facilitates the observation of intensity distribution and subtle changes in the image. The final super-resolution SIM-FRET image and its corresponding quantitative analysis results are displayed on the display interface for users to observe, analyze and store.

[0122] A system for estimating structured light microscopy illumination parameters using the above-described high signal-to-noise ratio spectrum extraction method includes:

[0123] The data acquisition module acquires raw images of multi-directional, multi-phase structured light and performs preprocessing.

[0124] The frequency-domain analysis module performs Fourier transform on the preprocessed original image to convert it into frequency-domain data, and then separates the 0th and ±1st order spectral components;

[0125] The spectral high signal-to-noise ratio evaluation module evaluates the signal-to-noise ratio of the frequency-domain data based on the spectral amplitude or energy distribution to construct a high signal-to-noise ratio spectral region in the frequency domain;

[0126] The partial correlation calculation module performs partial correlation analysis on the 0th and +1st order spectral components within the high signal-to-noise ratio spectral region;

[0127] The parameter output module determines the structured light illumination parameters according to the optimal result of the partial correlation analysis and outputs them.

[0128] The calculation tasks of each module are executed in parallel using a parallel computing unit.

[0129] The system of the present invention combines a parallel computing platform and a host computer. The computing module can be deployed on a CPU, GPU or other parallel computing platforms, and real-time processing is achieved through task scheduling and data transmission. This system and method form a close cooperation, providing a complete engineering implementation path for real-time high-frame-rate SIM-FRET imaging.

[0130] Compared with the prior art, the technical effects of the present invention are reflected in:

[0131] 1. Achieve an order-of-magnitude improvement in the parameter estimation speed, breaking through the bottleneck of real-time imaging.

[0132] By abandoning the calculation of the redundant region of the full spectrum and adopting the pCOR partial cross-correlation algorithm, the present invention reduces the computational complexity from O(N^2) of the traditional cross-correlation (COR) to O(L^2), where L is the side length of the calculated high signal-to-noise ratio region and N is the side length of the full frequency domain, and usually L << N. This brings a leap in the intrinsic efficiency of the algorithm.

[0133] When the present invention processes a single-channel SIM image (512×512, 9 original images) on the same hardware (CPU), the parameter estimation time is shortened from about 2215 milliseconds of the traditional COR to about 560 milliseconds of pCOR, with an acceleration of about 4 times. When deployed on a GPU, the improvement is more significant: the single-channel processing time is reduced from about 61.4 milliseconds of COR-GPU to about 18 milliseconds of pCOR-GPU; for a three-channel SIM-FRET image, the total parameter estimation time is reduced from about 184.0 milliseconds to about 54.3 milliseconds. This enables the end-to-end processing (including reconstruction and quantitative analysis) of a single-frame SIM-FRET image to be completed within about 65 milliseconds, and combined with the acquisition time, real-time super-resolution FRET imaging of more than 10 frames per second can be achieved. The acquisition speed and image processing speed of the SIM-FRET system are shown in the following table:

[0134]

[0135] (Table 1)

[0136] 2. While significantly increasing the speed, it maintains or even improves the accuracy and robustness of parameter estimation.

[0137] pCOR does not sacrifice accuracy for speed. The high signal-to-noise ratio region it filters out through Fourier ring correlation (FRC) is precisely the part of the spectrum that contributes most effectively and reliably to parameter estimation. Actively excluding high-frequency noise regions with low signal-to-noise ratio actually reduces noise interference.

[0138] In simulation experiments, even when the signal-to-noise ratio of the input image deteriorates (by adding noise of 200% intensity), the wave vector estimation error of pCOR remains below 0.08 pixels, and the phase error is less than 0.3 radians, performing comparable to traditional COR. In live-cell imaging, the structural similarity (SSIM) index of SIM-FRET images reconstructed based on pCOR and COR is greater than 0.95, and the calculated FRET efficiency distribution histograms are highly consistent. This demonstrates that pCOR, while pursuing extreme speed, fully maintains the accuracy of quantitative analysis.

[0139] 3. It possesses universality and stability, and can be applied under different imaging conditions.

[0140] One of the core advantages of pCOR is its data-driven nature. The size L of its high signal-to-noise ratio region is not fixed, but determined based on the FRC curve of each frame of the image. This allows the algorithm to be applied to different samples, label densities, illumination intensities, and noise levels.

[0141] Under high signal-to-noise ratio (SNR) imaging conditions (such as bright, densely stained samples), the FRC curve decays slowly, and Rc is larger. The algorithm automatically utilizes more spectral information to ensure maximum accuracy. Under low SNR conditions (such as low illumination and rapid live-cell imaging), the FRC curve decays quickly, and Rc is smaller. The algorithm automatically shrinks the calculation area to effectively shield noise and prevent estimation failure. This mechanism ensures the stability and success rate of the method under various experimental scenarios and reduces the tedious manual parameter adjustment required due to image quality fluctuations.

[0142] 4. It reduces reliance on high-performance hardware and broadens application scenarios.

[0143] Because pCOR significantly reduces computational cost at the algorithm level, its performance advantage is already very evident when using only a general-purpose CPU. This reduces the strong reliance on dedicated parallel computing hardware such as high-end GPUs.

[0144] Reduced cost and power consumption: This makes it possible to achieve near real-time super-resolution image processing on low- to medium-configuration computers and even embedded processors, which is beneficial for developing low-cost, portable microscopic imaging devices.

[0145] Increased accessibility: Biologists can use the technology on more common laboratory workstations without having to invest in expensive computing hardware, which promotes the widespread adoption of the technology.

[0146] Simplified system: Reduces reliance on complex heterogeneous (CPU-GPU) programming and optimization, lowering the complexity of software development and maintenance.

[0147] 5. It provides a better algorithmic foundation for system-level optimization, unlocking greater performance potential.

[0148] The high efficiency of the pCOR algorithm itself provides a superior starting point for deeper system-level optimizations (such as multi-threaded pipelines and GPU acceleration).

[0149] Because a single pCOR calculation requires less data and computation, it can make fuller use of on-chip shared memory on GPUs, reducing access latency to global memory and resulting in higher parallel efficiency. Simultaneously, the shorter processing time allows for more time for other tasks such as image acquisition, data transmission, reconstruction, and display, making it more feasible to build a stable, high-frame-rate, end-to-end real-time imaging system. Furthermore, the high signal-to-noise ratio region information extracted by pCOR can itself serve as a real-time monitoring indicator of image quality.

[0150] from Figure 2 It can be seen that pCOR focuses on the high signal-to-noise ratio signal region through the FRC Fourier ring correlation algorithm.

[0151] Because of its smaller computation area, it achieves a more time-saving and even more accurate result than COR.

[0152] from Figure 4 It can be seen that the pCOR algorithm significantly reduces the computation time in the parameter estimation stage, which has a high time ratio, compared with the COR algorithm, making the overall process more time-saving and efficient.

[0153] from Figure 5 It can be seen that the pCOR algorithm significantly shortens the time spent on parameter estimation and is significantly better than the COR algorithm in terms of real-time performance, thus achieving true real-time SIM image reconstruction.

[0154] from Figure 6 It can be seen that wide-field imaging and PCA methods result in blurriness and loss of detail, while pCOR and COR can both produce clear images.

[0155] It presents detailed subcellular structures, with the highest contrast and signal-to-noise ratio, and the best super-resolution effect.

[0156] from Figure 7 It can be seen that the FRET quantitative analysis accuracy of both pCOR and COR is better than that of wide field, and the FRET signal distribution is more concentrated.

Claims

1. A structured light microscopic illumination parameter estimation method based on high signal-to-noise ratio spectrum extraction, characterized by Includes the following steps: S1. The sample is illuminated by structured light microscopy system in multiple directions, and multiple original images with different initial phases are acquired under each illumination direction. There is a preset phase difference relationship between the original images with different initial phases. S2. Perform a two-dimensional Fourier transform on each of the acquired raw images to convert them into frequency domain data; S3. Based on the phase difference relationship between the original images with different initial phases, the frequency domain data is decoupled to separate the 0th order spectral component and the +1st order spectral component or -1st order spectral component for each illumination direction. S4, determine the integer pixel component of the illumination wave vector by locating the peak of the +1st order spectral component ;​​​​​ S5. Perform frequency domain Fourier ring correlation analysis on the aligned +1st order spectral components to obtain quantitative evaluation results of spectral consistency at different spatial frequencies. S6. Determine the frequency range that meets the preset high signal-to-noise ratio condition based on the evaluation results, and construct a high signal-to-noise ratio spectrum region in the frequency domain using this frequency range. S7. Within the obtained high signal-to-noise ratio spectrum region, perform partial correlation analysis on the 0th-order spectral component and the +1st-order spectral component. S8. Based on the optimal results of partial correlation analysis, determine the structured light illumination parameters.

2. The structured light microscopic illumination parameter estimation method based on high signal-to-noise ratio spectrum extraction according to claim 1, characterized in that: In step S7, the partial correlation analysis includes the following steps: (1) Construct multiple candidate parameters within the sub-pixel search range of the illumination wave vector; (2) For each candidate parameter, the corresponding correlation evaluation value is calculated only in the high signal-to-noise ratio spectrum region; (3) By comparing the correlation evaluation values ​​corresponding to different candidate parameters, the parameter values ​​that make the correlation optimal are determined.

3. The structured light microscopic illumination parameter estimation method based on high signal-to-noise ratio spectrum extraction according to claim 2, characterized in that: In step S7⑶, the optimal parameter values ​​are calculated using the following formula: where, is a candidate sub-pixel vector is the corresponding partial cross-correlation value; m is the modulation depth of the structured illumination; L is the side length of the high signal-to-noise region; is the frequency domain coordinate within this region; is the 0th order spectrum, is its complex conjugate; is the +1st order spectrum; is the integer pixel component of the illumination wave vector; is a candidate sub-pixel vector; is the phase factor corresponding to the initial phase φ0.

4. The structured light microscopic illumination parameter estimation method based on high signal-to-noise ratio spectrum extraction according to claim 3, characterized in that: In step S2, the formula for performing the two-dimensional Fourier transform is as follows: wherein, is the original fluorescence image Fourier transformed spectral signals; X = DD, DA, AA, represent the donor excitation donor emission channel DD, the donor excitation acceptor emission channel DA, and the acceptor excitation acceptor emission channel AA, respectively; θ = 1, 2, 3: represent different directions; n = -1, 0, 1: represent different phases; r is the spatial coordinate of the original image.

5. The structured light microscopic illumination parameter estimation method based on high signal-to-noise ratio spectrum extraction according to claim 4, characterized in that: In step S5, the formula for performing frequency domain Fourier ring correlation analysis is as follows: where the numerator represents the cross-correlation term; and the denominator serves as a normalization factor; is the coordinate of the corresponding spectrum; is the integer pixel component of the illumination wave vector used to shift the +1st order spectrum to align with the 0th order spectrum; and is the signal power of the corresponding spectrum at the frequency point; represents the frequency domain coordinate set corresponding to the rth ring band; The calculation formula is as follows: Now, assuming the noise is under independent statistics, the FRC value is a monotonic function of SNR, and its expression is: 。 6. The method for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio spectrum extraction according to claim 5, characterized in that: In step S6, the high signal-to-noise ratio spectral region is calculated using the following formula: Where L is the side length of the high signal-to-noise ratio region; It is the effective cutoff radius obtained based on Fourier ring correlation screening, representing the maximum radial range of high signal-to-noise ratio signal in the spectrum; N is the size of the complete spectrum.

7. The method for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio spectrum extraction according to claim 6, characterized in that: The structured light illumination parameters are the illumination wave vector, initial phase, and modulation depth.

8. The method for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio spectrum extraction according to claim 7, characterized in that: In step S1, after the original data acquisition is completed, the original image is preprocessed by one or more of the following methods: background subtraction or dark current correction of the original image, intensity normalization of images acquired from different channels or at different times, noise suppression of the image, registration of multiple frames of images, and subpixel calibration of dual-channel images, and then the process proceeds to step S2.

9. A system for estimating structured light microscopy illumination parameters based on high signal-to-noise ratio spectral extraction as described in claim 1, characterized in that... include: The data acquisition module acquires raw images of multi-directional, multi-phase structured light and performs preprocessing. The frequency domain analysis module performs a two-dimensional Fourier transform on the preprocessed original image to convert it to frequency domain data, and then separates the 0th order and ±1st order spectral components. The high signal-to-noise ratio (SNR) evaluation module evaluates the SNR of frequency domain data based on spectral amplitude or energy distribution to construct a high SNR spectral region in the frequency domain. The partial correlation calculation module performs partial correlation analysis on the 0th and +1st order spectral components in the high signal-to-noise ratio spectral region. The parameter output module determines and outputs the structured light illumination parameters based on the optimal results of partial correlation analysis.

10. The system according to claim 9, characterized in that: The computational tasks of each module are executed in parallel using parallel computing units.

Images