Structured light super-resolution microscopic imaging method based on contour adaptive coordinate transformation

Abstract

The invention belongs to the technical field of microscopic imaging, and particularly relates to a structured light super-resolution microscopic imaging method based on contour adaptive coordinate transformation. According to the method, traditional interference fringes are replaced by contour matching line scanning illumination, geometric distortion is eliminated in combination with logarithmic polar coordinate transformation, a unitary matrix reconstruction framework is constructed in a transformation domain, 'true double 'resolution is achieved, SIM resolution improvement is upgraded to'true optical capture' from'apparent transfer ', and a physical limit is achieved. And meanwhile, deep learning blocking and dynamic frequency optimization are utilized, so that the technology universally covers a semiconductor to a biological special-shaped structure.

Description

Technical Field

[0001] The present invention belongs to the field of microscopic imaging technology, and in particular relates to a structured light super-resolution microscopic imaging method based on contour adaptive coordinate transformation. Background Art

[0002] Structured illumination microscopy (SIM), a mainstream super-resolution imaging technique, uses the moiré effect to shift high-frequency information into the detectable frequency band of the optical system by spatially modulating the interaction between illumination light and the sample. The theoretical basis for its resolution improvement can be expressed as follows:

[0003]

[0004] in is the spatial frequency of the illumination stripes. Classic SIM techniques (such as the one proposed by Gustafsson et al.) typically use fixed-period, globally uniform stripe illumination to reconstruct super-resolution images using a three-phase shift algorithm. This technique can double the resolution and is widely used in live cell imaging and material surface analysis.

[0005] Multi-band SIM (Kner et al., Nature Methods 2009) improves resolution through frequency-domain fusion but still uses a fixed global illumination pattern. Deep learning-enhanced SIM (Wang et al., Cell 2020) uses a CNN to reduce reconstruction noise but fails to address the illumination mismatch problem. A core drawback of existing techniques is the noise caused by global illumination and the spectral aliasing caused by irregular contours. The uniform illumination of classic SIM results in over 70% of the light energy being applied to uninformative background areas, resulting in high background noise masking weak signals.

[0006] Classical structured illumination microscopy (SIM) produces a moiré effect through fixed-period stripe illumination, with a theoretical resolution limit of However, it has the following essential defects:

[0007] 1) "Pseudo-double" resolution: Relying on interference to indirectly modulate high-frequency information, the actual optical information does not exceed the diffraction limit ;

[0008] 2) Spectral aliasing: Irregular contours lead to illumination distortion;

[0009] 3) Reconstruction noise amplification is significant;

[0010] 4) Background noise: 70% of light energy is wasted in the invalid area, and the signal-to-noise ratio is <28 dB. Summary of the Invention

[0011] To address these issues, the present invention proposes a structured light super-resolution microscopy imaging method based on profile-adaptive coordinate transformation. Through an innovative architecture combining profile-adaptive coordinate transformation with block-optimized illumination, the present invention provides a new solution for super-resolution microscopy. Adaptive SIM increases the amount of real optical information received, overcoming the limitations of the "apparent shift" of moiré fringes and achieving a true doubling of the theoretical maximum resolution at the diffraction limit.

[0012] The technical solution adopted in the present invention is:

[0013] The structured light super-resolution microscopy imaging method based on contour adaptive coordinate transformation includes the following steps:

[0014] S1. After fixing the sample to be tested on the microscope stage, a low-resolution bright field / fluorescence image is captured by the microscopic imaging system, and the captured image is defined as the contour image;

[0015] S2. Input the three obtained contour images into a pre-trained U-Net network for segmentation to obtain multiple independent sub-block areas. The pre-trained U-Net network is a network that splits the input original image into independent sub-blocks according to connected domains, so that each sub-block contains a target object. The output of the network is a mask for each sub-block;

[0016] S3. Generate an adaptive structured light pattern using a contour-adaptive coordinate transformation method, specifically:

[0017] For each sub-block, extract the contour point set according to its mask;

[0018] Calculate the centroid of the contour based on the contour point set;

[0019] Establish a polar coordinate system with the center of mass as the origin and calculate the contour boundary function;

[0020] generating three phase-shifted cosine fringe patterns in a normalized coordinate domain;

[0021] The pattern in the normalized coordinate domain is converted back to the original coordinates through inverse mapping to obtain an adaptive structured light pattern;

[0022] S4. For different phases, synthesize the adaptive structured light patterns obtained by all sub-blocks at the phase into a global pattern, thereby obtaining three global patterns with different phases;

[0023] S5, loading the obtained three global patterns into the DMD controller of the microscopic imaging system, and then projecting the three global patterns in sequence according to the phase. After each projection, the camera captures an image, thereby obtaining three global images;

[0024] S6. Reconstruct the image based on the sub-blocks. Specifically, based on the mask of each sub-block obtained in S2, extract the area corresponding to each sub-block from the three global images obtained, and perform the following processing on each sub-block:

[0025] Convert the three images of the sub-block into the normalized coordinate domain;

[0026] Perform a two-dimensional Fourier transform in the normalized coordinate domain;

[0027] Separate spectral components using a three-phase demodulation algorithm;

[0028] Perform spectrum shifting and splicing to obtain extended spectrum;

[0029] Obtain a super-resolution image in the normalized coordinate domain through inverse Fourier transform;

[0030] Inverse transform back to the original Cartesian coordinates and obtain the super-resolved sub-block image by interpolation;

[0031] S7. The super-resolution images reconstructed from all sub-blocks are spliced ​​together according to their positions, and the overlapping areas are fused using weighted averaging to obtain the entire super-resolution image.

[0032] Furthermore, the obtained contour point set is defined as B, and the centroid equation of the contour is calculated as:

[0033] ,

[0034] in, is the center of mass, ( ) is the Cartesian coordinate of any boundary point in the contour point set B. All boundary points in the contour point set B The sum of the coordinates, All boundary points in the contour point set B The sum of the coordinates, It is the inverse of the total number of boundary points and is used to calculate the coordinate average.

[0035] The polar coordinate system established is:

[0036] ,

[0037] in, ) represents the Cartesian coordinate value of any pixel point in the original image, 、 Represents the centroid coordinates calculated from the contour point set

[0038] Taking the center of mass as the origin, for any angle , contour boundary function Represents the angle from the centroid to the contour boundary The radial distance is obtained by interpolation or Fourier series fitting of the contour point set B. It can be expressed by Fourier series as:

[0039]

[0040] Among them, the coefficient , , It is obtained by fitting the contour point set B using the least squares method, where N is the order of the Fourier series.

[0041] The cosine fringe pattern generated in the normalized coordinate domain is:

[0042] ,

[0043] in, are normalized coordinates, , r is the distance from the pixel to the centroid, is the modulation depth, is the spatial frequency, , Indicates three different phases;

[0044] The method of converting the pattern in the normalized coordinate domain back to the original coordinates (x, y) by inverse mapping to obtain the adaptive structured light pattern is: The inverse transform is ,thereby , .

[0045] Furthermore, in S6, the demodulation matrix equation of the three-phase demodulation algorithm is:

[0046] ,

[0047] in, is the zero-frequency component, which is the background spectrum information after demodulation. is the positive first-order spectral component, is the negative first-order spectral component; M is the demodulation matrix, The inverse matrix of the demodulation matrix; , They are the spectra of the three-phase images, corresponding to the phase offset Frequency domain data of

[0048] The method of shifting and splicing the spectrum to obtain the extended spectrum is:

[0049] ,

[0050] in, is the super-resolution spectrum of the complete frequency domain image after splicing, is the radial frequency, the frequency domain variable in the normalized coordinate ρ direction, is the angular frequency, normalized coordinate The frequency domain variable of the direction, k is the spatial frequency;

[0051] The method of obtaining a super-resolution image in the normalized coordinate domain by inverse Fourier transform is Perform radial coordinate restoration, where r is the radial distance in the actual space after inverse transformation;

[0052] The inverse transformation back to the original Cartesian coordinates is to let , The method of interpolating the super-resolution sub-block image is to use bicubic interpolation to obtain the intensity value of the i-th sub-block in the super-resolution sub-block image at the Cartesian coordinate (x, y):

[0053] ,

[0054] in, are the normalized local coordinates, is the radial coordinate The sampling interval, and is the angular coordinate The sampling interval, and is the weight function of bicubic interpolation; through the bicubic interpolation method, the normalized coordinates The super-resolution image intensity value at Mapping to Cartesian coordinates At this point, we get the super-resolution sub-block image .

[0055] The method of global image fusion is to seamlessly stitch the sub-blocks: , weighted averaging of overlapping areas eliminates seams.

[0056] The beneficial effects of the present invention are:

[0057] 1) Eliminate spectrum aliasing through coordinate transformation Mapping irregular contours to canonical domains;

[0058] 2) Suppress background noise and use a deep learning-driven local illumination strategy to focus light energy on the target area;

[0059] 3) Breaking through the resolution limit, contour-adaptive lighting produces an equivalent frequency multiplication effect: , the measured resolution is increased to twice as much in all aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of the application environment of the present invention;

[0061] Figure 2 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0062] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0063] like Figure 1 Figure 2 shows a schematic diagram of an application scenario for the method of the present invention. The constructed microscopic imaging system includes a laser, collimator and beam expander, reflector, DMD, lens, spatial filter, spectroscope, objective lens, camera, and a lens between the camera and spectroscope. The camera acquires sample images through the lens, spectroscope, and objective lens. The laser provides light source sequentially through the collimator and beam expander, reflector, DMD, lens, spatial filter, and spectroscope, and the DMD serves as the controller. In the solution of the present invention, three original images are required to reconstruct a super-resolution image. In traditional SIM, the same three phase-shifted images are used globally, while the present invention is adaptive, using a different pattern for each sub-block. Therefore, three images are required for each sub-block. Therefore, for the entire sample, the number of images required is 3×N (N is the number of sub-blocks). This can be time-consuming in practice, especially when there are many sub-blocks.

[0064] To solve this problem, the present invention utilizes the fast programming feature of DMD to project the pattern of each sub-block in sequence and collect images. Specifically, the entire collection process can be arranged as follows:

[0065] Step 1: Project the phase 1 pattern of sub-block 1 and collect image 1-1;

[0066] Step 2: Project the phase 2 pattern of sub-block 1 and collect image 1-2;

[0067] Step 3: Project the phase 3 pattern of sub-block 1 and collect images 1-3;

[0068] Then move to sub-block 2 (or if the DMD can display patterns of multiple sub-blocks simultaneously, no movement is required, but usually the patterns of each sub-block are independent and may overlap, so they need to be projected in sequence).

[0069] Since the pattern of each sub-block is designed independently and may cover different areas, in order to avoid crosstalk, it is assumed that only one sub-block pattern is projected at each moment. In this case, a total of 3×N acquisitions are required.

[0070] Another optimization approach is to display the illumination patterns of multiple sub-blocks simultaneously on the same DMD pattern for non-overlapping sub-blocks. This allows a single acquisition to capture images of multiple sub-blocks at the same phase offset. However, this requires that the sub-blocks do not overlap and have a certain spacing. Then, during image processing, the image is segmented and processed separately by sub-block. This way, only three acquisitions (one for each phase) are required to obtain data for all sub-blocks. The specific process is as follows:

[0071] Three global patterns are designed (one for each phase). Each global pattern contains the adaptive structured light patterns of all sub-blocks at that phase (located at the positions of their respective sub-blocks, and the non-overlapping areas are dark fields).

[0072] These three global patterns are projected sequentially and three images are acquired.

[0073] For each sub-block, the corresponding region (each sub-block region) is extracted from the three global images, and then each sub-block is reconstructed.

[0074] This approach can significantly reduce acquisition time, requiring only three acquisitions, but only if the sub-blocks do not overlap and the illumination pattern is zero outside the sub-block area (i.e., it does not illuminate other areas).

[0075] like Figure 2 As shown, it is a flow chart of the present invention, combined with Figure 1 The specific operation process of the system shown is as follows:

[0076] 1. Sample preparation: Fix the sample to be tested (such as biological slices, semiconductor materials, etc.) on the microscope stage.

[0077] 2. System initialization:

[0078] a. Turn on the light source (e.g. 488nm laser), collimate and expand the beam, and then illuminate the DMD.

[0079] b. The light reflected by the DMD passes through the 4f relay system (including two lenses) and the objective lens and is irradiated onto the sample.

[0080] c. Light emitted by the sample (fluorescence or reflected light) is imaged onto the camera through the objective lens and tube lens.

[0081] 3. Contour acquisition:

[0082] a. Take a low-resolution brightfield / fluorescence image (called a profile image) using uniform illumination (DMD fully on or using a uniform grayscale pattern).

[0083] b. Input the contour image into a pre-trained U-Net network for segmentation, obtaining multiple independent sub-block regions (each sub-block contains a target object). The network output is a mask for each sub-block.

[0084] 4. Generate adaptive structured light pattern (for each sub-block and each phase):

[0085] a. For each sub-block, extract the contour point set B according to its mask.

[0086] b. Calculate the centroid of the contour .

[0087] c. Establish a polar coordinate system with the center of mass as the origin and calculate the contour boundary function .

[0088] d. Generate three phase-shifted cosine fringe patterns in the normalized coordinate (ρ, θ) domain:

[0089]

[0090] e. Convert the pattern in the normalized coordinate domain back to the original coordinates (x, y) through the inverse mapping:

[0091] The inverse transform is

[0092] Then

[0093] f. Save the generated pattern (valid only within the sub-block area, with the area outside the area set to 0).

[0094] 5. Synthesize global illumination patterns:

[0095] a. For each phase n (n=1, 2, 3), combine the patterns of all sub-blocks at that phase into a global pattern (note: the sub-block patterns do not overlap, so they can be directly superimposed, but it is necessary to ensure that only one sub-block pattern is illuminated at the same position).

[0096] b. Load the synthesized global pattern into the DMD controller.

[0097] 6. Collect images:

[0098] Three global patterns (phases 1, 2, and 3) are projected in sequence. After each projection, the camera captures an image (a total of three global images, each image contains the illumination information of all sub-blocks at that phase).

[0099] 7. Image reconstruction:

[0100] a. For each sub-block, extract the corresponding region from the three global images (determine the region based on the sub-block mask).

[0101] b. Perform the following processing on each sub-block:

[0102] Convert the three images of the sub-block into the normalized coordinate domain (ρ, θ).

[0103] Perform a two-dimensional Fourier transform in the normalized coordinate domain.

[0104] The spectral components are separated using a three-phase demodulation algorithm (demodulation matrix).

[0105] Perform spectrum shifting and splicing to obtain extended spectrum.

[0106] The inverse Fourier transform is performed to obtain a super-resolution image in the normalized coordinate domain.

[0107] The inverse transformation is performed back to the original Cartesian coordinates, and the super-resolved sub-block images are obtained by interpolation.

[0108] 8. Image fusion: The super-resolution images reconstructed from all sub-blocks are stitched together according to their positions, and the overlapping areas are fused using weighted averaging (for example, using distance weighting) to obtain the entire super-resolution image.

[0109] 9. Display and save: Display and save the final super-resolution image.

[0110] In summary, the present invention replaces traditional interference fringes with contour matching line scanning illumination, combines logarithmic polar coordinate transformation to eliminate geometric distortion, and constructs a unitary matrix reconstruction framework in the transformation domain to achieve "true double" resolution, upgrading SIM resolution from "apparent transfer" to "real optical capture", reaching the physical limit. At the same time, deep learning block and dynamic frequency optimization are used to make the technology universally applicable, covering everything from semiconductors to biological heterogeneous structures.

Claims

1. A structured light super-resolution microscopy imaging method based on contour adaptive coordinate transformation, characterized in that: The following steps are involved: S1. After fixing the sample to be tested on the microscope stage, a low-resolution bright field / fluorescence image is captured by the microscopic imaging system, and the captured image is defined as the contour image; S2. Input the obtained contour image into a pre-trained U-Net network for segmentation to obtain multiple independent sub-block areas. The pre-trained U-Net network is a network that splits the input original image into independent sub-blocks according to connected domains, so that each sub-block contains a target object. The output of the network is a mask for each sub-block; S3. Generate an adaptive structured light pattern using a contour-adaptive coordinate transformation method, specifically: For each sub-block, extract the contour point set according to its mask; Calculate the centroid of the contour based on the contour point set; Establish a polar coordinate system with the center of mass as the origin and calculate the contour boundary function; generating three phase-shifted cosine fringe patterns in a normalized coordinate domain; The pattern in the normalized coordinate domain is converted back to the original coordinates through inverse mapping to obtain an adaptive structured light pattern; S4. For different phases, synthesize the adaptive structured light patterns obtained by all sub-blocks at the phase into a global pattern, thereby obtaining three global patterns with different phases; S5, loading the obtained three global patterns into the DMD controller of the microscopic imaging system, and then projecting the three global patterns in sequence according to the phase. After each projection, the camera captures an image, thereby obtaining three global images; S6. Reconstruct the image based on the sub-blocks. Specifically, based on the mask of each sub-block obtained in S2, extract the area corresponding to each sub-block from the three global images obtained, and perform the following processing on each sub-block: Convert the three images of the sub-block into the normalized coordinate domain; Perform a two-dimensional Fourier transform in the normalized coordinate domain; Separate spectral components using a three-phase demodulation algorithm; Perform spectrum shifting and splicing to obtain extended spectrum; Obtain a super-resolution image in the normalized coordinate domain through inverse Fourier transform; Inverse transform back to the original Cartesian coordinates and obtain the super-resolved sub-block image by interpolation; S7. The super-resolution images reconstructed from all sub-blocks are spliced ​​together according to their positions, and the overlapping areas are fused using weighted averaging to obtain the entire super-resolution image.

2. The structured light super-resolution microscopy imaging method based on contour adaptive coordinate transformation according to claim 1, characterized in that: In S3, the contour point set is defined as B, and the centroid equation for calculating the contour is: ， in, is the center of mass, ( 、 ) is the Cartesian coordinate of any boundary point in the contour point set B, All boundary points in the contour point set B The sum of the coordinates, All boundary points in the contour point set B The sum of the coordinates, It is the inverse of the total number of boundary points and is used to calculate the coordinate average; The polar coordinate system established is: ， in,( 、 ) represents the Cartesian coordinate value of any pixel point in the original image, ( 、 ) represents the centroid coordinates calculated from the contour point set Taking the center of mass as the origin, for any angle , contour boundary function Represents the angle from the centroid to the contour boundary The radial distance is obtained by interpolation or Fourier series fitting of the contour point set B. It is expressed by Fourier series as:

3. Among them, the coefficient , , It is obtained by fitting the contour point set B by the least squares method, where N is the order of the Fourier series; The cosine fringe pattern generated in the normalized coordinate domain is: ， in, are normalized coordinates, , r is the distance from the pixel to the centroid, is the modulation depth, is the spatial frequency, , Indicates three different phases; The method of converting the pattern in the normalized coordinate domain back to the original coordinates (x, y) by inverse mapping to obtain the adaptive structured light pattern is: The inverse transform is ,thereby , .

4. The structured light super-resolution microscopy imaging method based on contour adaptive coordinate transformation according to claim 1, characterized in that: In S6, the demodulation matrix equation of the three-phase demodulation algorithm is: ， in, is the zero-frequency component, which is the background spectrum information after demodulation. is the positive first-order spectral component, is the negative first-order spectral component; M is the demodulation matrix, The inverse matrix of the demodulation matrix; , They are the spectra of the three-phase images, corresponding to the phase offset Frequency domain data of The method of shifting and splicing the spectrum to obtain the extended spectrum is: ， in, is the super-resolution spectrum of the complete frequency domain image after splicing, is the radial frequency, the frequency domain variable in the normalized coordinate ρ direction, is the angular frequency, normalized coordinate The frequency domain variable of the direction, k is the spatial frequency; The method of obtaining a super-resolution image in the normalized coordinate domain by inverse Fourier transform is Perform radial coordinate restoration, where r is the radial distance in the actual space after inverse transformation; The inverse transformation back to the original Cartesian coordinates is to let , The method of interpolating the super-resolution sub-block image is to use bicubic interpolation to obtain the super-resolution sub-block image of the i-th sub-block in the Cartesian coordinates Intensity value at: ， in, are the normalized local coordinates, is the radial coordinate The sampling interval, and is the angular coordinate The sampling interval, and is the weight function of bicubic interpolation; through the bicubic interpolation method, the normalized coordinates The super-resolution image intensity value at Mapping to Cartesian coordinates At this point, we get the super-resolution sub-block image .

Images