Structured light illumination confocal super-resolution measurement system and method

Abstract

The invention relates to a structured light illumination confocal super-resolution measurement system and method, and relates to the technical field of optical measurement. The system comprises a digital micromirror device (DMD), a sample platform, an electric displacement platform, a microscopic imaging assembly, a complementary metal oxide semiconductor (CMOS) imaging assembly and computer equipment, and the DMD is used for adjusting the point light source array and the structured light field. According to the DMD-based confocal super-resolution measurement method, a structured light field and a signal-noise separation technology are combined, and high-precision and high-resolution three-dimensional surface reconstruction can be realized. By using the flexibility of the DMD and the advantages of the super-resolution technology, the spatial resolution and speed of confocal imaging are effectively improved, so that the measurement process is more accurate. In addition, low-rank decomposition and self-supervised deep learning methods are fused, the image reconstruction process is optimized, noise interference is suppressed, and the overall performance of the system is further improved.

Description

Technical Field

[0001] This invention relates to the field of optical measurement technology, and in particular to a structured light illumination confocal super-resolution measurement system and method. Background Technology

[0002] Optics is the science that studies the behavior and properties of light. Optical microscopes have developed gradually based on this, undergoing numerous technological innovations and application expansions. In the past few decades, research in fields such as semiconductor lithography and life sciences has reached the nanoscale, presenting corresponding challenges and higher requirements for measurement techniques. Limited by the diffraction limit, the spatial resolution of ordinary optical microscopes does not exceed 200 nm, significantly narrowing their application range and failing to meet the measurement needs of many research fields, thus preventing them from fully realizing their advantages. Therefore, how to overcome the diffraction limit and achieve ultra-high optical resolution imaging has become a focus and development direction in the field of optical microscopy.

[0003] Thanks to its unique optical design, confocal microscopy is highly practical, providing high-resolution and high-contrast images, excellent real-time imaging and 3D reconstruction capabilities, and good axial tomography. Its lateral resolution is also superior to that of ordinary optical microscopes, theoretically reaching 1.4 times. Traditional confocal microscopes typically employ point-by-point mechanical scanning, resulting in slow imaging speeds, sometimes sacrificing field of view and resolution to improve speed. Therefore, improving scanning efficiency has become a key breakthrough for confocal measurement technology. In recent years, spectroscopic devices such as Nipkow disks, microlens arrays, and digital micromirror devices (DMDs) have been widely used, enabling parallel and rapid scanning and effectively compensating for the shortcomings of traditional point-by-point scanning methods. On the other hand, to meet the demand for higher confocal imaging resolution, various super-resolution methods have been proposed, including Structure Illumination Microscopy (SIM) and Stimulated Emission Depletion Microscopy (STED), which have extremely wide applications. A digital micromirror device (DMD) is an array of micromirrors of a specific size. Through program control, it can easily split light beams and control the parameters of the split beams. Thanks to its advantages of rapid response, high flexibility, and high light energy utilization, DMDs can be used to achieve both high-speed parallel scanning and axial scanning.

[0004] In structured light illumination super-resolution microscopy, current techniques primarily rely on spatial light modulators to control the structured light field, such as liquid crystal SLMs, ferroelectric SLMs, or digital micromirror devices (DMDs). These modulators offer high precision and speed, significantly simplifying operation and reducing instrument complexity. However, the frequency of the fringe illumination field is limited by the diffraction limit of the optical system, resulting in a resolution that can only exceed the diffraction limit by a maximum of one-fold. Furthermore, it is not a "direct" imaging technique, requiring data post-processing and image reconstruction. This process inevitably introduces artificial noise or residues, affecting the resolution of sample details.

[0005] With the continuous development of confocal microscopy technology and the increasing demands for measurement, improving confocal imaging speed and resolution has become a key research focus and technical challenge in this field. Therefore, it is necessary to design a confocal super-resolution measurement method. Summary of the Invention

[0006] This invention relates to a structured light illumination confocal super-resolution measurement system and method, which can improve confocal imaging speed and resolution by combining measurement requirements. The technical solution is as follows: On the one hand, a structured light illumination confocal super-resolution measurement system is provided, which includes a DMD, a sample platform, an electric displacement platform, a microscopic imaging component, a complementary metal-oxide-semiconductor CMOS imaging component, and a computer device; The DMD, the sample platform, the electric displacement platform, the microscopic imaging component, and the CMOS imaging component are all communicatively connected to the computer device. The DMD is used to adjust the point light source array and the structured light field; The sample platform is used to hold the sample to be tested; The electric displacement platform is used to perform displacement scanning; The CMOS imaging component is used to record optical signals; The computer device is used to process data to obtain the results of structured light illumination confocal super-resolution measurements.

[0007] In an optional embodiment, the sample to be tested is implemented as a plane mirror.

[0008] On the other hand, a structured light illumination confocal super-resolution measurement method is provided, which is applied to a computer device within any of the structured light illumination confocal super-resolution measurement systems described above, and the method includes: Constructing the conjugate relationship between DMD and CMOS imaging components; Construct a point light source array corresponding to the DMD; The electric displacement platform is controlled to scan the point light source array to obtain scanning results, which are then converted into image data. Generate confocal super-resolution measurement raw data based on the image data; The original confocal super-resolution measurement data is reconstructed to obtain confocal super-resolution measurement data.

[0009] In an optional embodiment, constructing the conjugate relationship between the DMD and the CMOS imaging component includes: Determine the sample plane mirror and the sample calibration diagram corresponding to the sample plane mirror; Load the sample calibration map into the DMD; The DMD is calibrated using the sample calibration map to establish a conjugate relationship between the DMD and the CMOS imaging component.

[0010] The step of calibrating the DMD using the sample calibration map to establish a conjugate relationship between the DMD and the CMOS imaging component includes: Determine at least three feature points in the sample calibration map, and the coordinates of the feature points corresponding to the features; The sample calibration map is coarsely calibrated using the coordinates of the feature points. In response to the completion of coarse calibration, fine calibration based on axial conjugation is performed, taking into account the relative positions of the DMD and the CMOS imaging component, to establish the conjugation relationship between the DMD and the CMOS imaging component.

[0011] In an optional embodiment, constructing the point light source array corresponding to the DMD includes: Obtain the modulation parameters corresponding to the DMD; Based on the modulation parameters, a point light source array corresponding to the DMD is constructed.

[0012] In an optional embodiment, controlling the electric displacement platform to scan the point light source array and obtain scanning results includes: The electric displacement platform is controlled to scan the point light source array to obtain scanning data; Based on the scanned data, a scan result is generated using a data processing method, which includes at least one of the centroid method, maximum value search method, and data fitting method.

[0013] In an optional embodiment, generating confocal super-resolution measurement raw data based on the image data includes: Combining the principle of fringe projection, a four-step phase-shifting sine and cosine fringe pattern is loaded; The original confocal super-resolution measurement data is generated based on the sine and cosine fringe pattern.

[0014] In an optional embodiment, the reconstruction processing of the original confocal super-resolution measurement data to obtain confocal super-resolution measurement data includes: The confocal super-resolution measurement data is obtained by extracting intrinsic signals from the confocal super-resolution measurement data using the low-rank decomposition method. The low-rank decomposition method is a reconstruction method based on self-supervised deep learning.

[0015] In an optional embodiment, the method further includes: The accuracy was verified using the structured light illumination confocal super-resolution measurement results, and the accuracy verification results were obtained. Adjustment parameters are generated based on the accuracy verification results, and the adjustment parameters are obtained using a modified self-supervised deep learning method.

[0016] The beneficial effects of the technical solution provided by this invention include at least the following: A DMD-based confocal super-resolution measurement method, combined with structured light field and signal-to-noise separation techniques, enables high-precision, high-resolution 3D surface reconstruction. Leveraging the flexibility of DMD and the advantages of super-resolution technology, this invention effectively improves the spatial resolution and speed of confocal imaging, making the measurement process more accurate. Furthermore, the fusion of low-rank decomposition and self-supervised deep learning optimizes the image reconstruction process and suppresses noise interference, thereby further enhancing the overall system performance. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A structural block diagram of a structured light illumination confocal super-resolution measurement system provided in an exemplary embodiment of this application is shown.

[0019] Figure 2 A schematic flowchart of a structured light illumination confocal super-resolution measurement method provided in an exemplary embodiment of this application is shown.

[0020] Figure 3 A schematic flowchart of a structured light illumination confocal super-resolution measurement method provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0022] Figure 1 This application provides a structural block diagram of a structured light illumination confocal super-resolution measurement system according to an exemplary embodiment. Please refer to... Figure 1 The system includes a DMD 110, a sample platform 120, an electric displacement platform 130, a microscopic imaging component 140, a CMOS imaging component 150, and a computer device 160; the DMD, the sample platform, the electric displacement platform, the microscopic imaging component, and the CMOS imaging component are respectively communicatively connected to the computer device.

[0023] Based on function, in this embodiment, the DMD is used to adjust the point light source array and the structured light field; the sample platform is used to carry the sample to be tested; the electric displacement platform is used to realize displacement scanning; the CMOS imaging component is used to record light signals; and the computer device is used to perform data processing to obtain the structured light illumination confocal super-resolution measurement results.

[0024] In an optional embodiment, the sample to be tested is implemented as a plane mirror.

[0025] based on Figure 1 , Figure 2 This illustration shows a flowchart of a structured light illumination confocal super-resolution measurement method provided in an exemplary embodiment of this application, demonstrating the method's application to, for example... Figure 1 Taking the computer device within the system shown as an example, the method includes: Step 201: Establish the conjugate relationship between the DMD and the CMOS imaging component.

[0026] In this embodiment, the conjugation relationship is constructed based on the principle of optical conjugation, which is also the process of adjusting the hardware positions of devices within the system. A computer device is used to guide this process.

[0027] Step 202: Construct a point light source array corresponding to the DMD.

[0028] This process involves configuring the point size and spacing of a point light source array for a computer device.

[0029] Step 203: Control the electric displacement platform to scan the point light source array and obtain the scanning results.

[0030] In this embodiment of the application, the scanning result is implemented as image data.

[0031] Step 204: Generate confocal super-resolution measurement raw data based on image data.

[0032] Optionally, the raw data from the confocal super-resolution measurement are data that has not undergone enhancement and reconstruction processing.

[0033] Step 205: Reconstruct the original confocal super-resolution measurement data to obtain confocal super-resolution measurement data.

[0034] This process involves enhancing and reconstructing the original data to ensure that the extracted new signal is more accurate and cleaner, thereby improving image quality.

[0035] In summary, the method provided in this application, a confocal super-resolution measurement method based on DMD, combined with structured light field and signal-to-noise separation technology, can achieve high-precision, high-resolution three-dimensional surface reconstruction. Utilizing the flexibility of DMD and the advantages of super-resolution technology, this invention effectively improves the spatial resolution and speed of confocal imaging, making the measurement process more accurate. Furthermore, the fusion of low-rank decomposition and self-supervised deep learning optimizes the image reconstruction process and suppresses noise interference, thereby further improving the overall performance of the system.

[0036] Figure 3 This application illustrates a flowchart of another structured light illumination confocal super-resolution measurement method provided in an exemplary embodiment, demonstrating its application in situations such as... Figure 1 Taking the computer equipment within the structured light illumination confocal super-resolution measurement system shown as an example, the method includes: Step 301: Determine the sample plane mirror and the sample calibration diagram corresponding to the sample plane mirror.

[0037] This process is for determining the sample.

[0038] Step 302: Load the sample calibration map into the DMD.

[0039] The sample calibration map is used to perform the calibration process. Optionally, the calibration map includes multiple known feature points for subsequent calibration work.

[0040] Step 303: The DMD is calibrated using the sample calibration map to establish the conjugate relationship between the DMD and the CMOS imaging components.

[0041] In this embodiment, the calibration process includes a step-by-step coarse calibration and a fine calibration process. In this case, the process includes: determining at least three feature points in the sample calibration image and the coordinates of the feature points corresponding to the features; performing coarse calibration on the sample calibration image using the feature point coordinates; and, in response to the completion of the coarse calibration, performing a fine calibration based on axial conjugation, taking into account the relative positions of the DMD and the CMOS imaging component, to establish the conjugation relationship between the DMD and the CMOS imaging component.

[0042] Step 304: Obtain the modulation parameters corresponding to the DMD.

[0043] Step 305: Construct a point light source array corresponding to the DMD based on the modulation parameters.

[0044] In this embodiment, the point light source array is determined based on the modulation parameters of the DMD. In one example, a plane mirror is used as the object to be measured. Multiple sets of point array patterns with different point sizes and spacings are set for measurement, and image processing is performed. A suitable point light source array is determined by the longitudinal response curve of the system. Finally, a 20×20 pixel point light source array is selected, and the spacing between the point light sources is set to be the same as the size.

[0045] Step 306: Control the electric displacement platform to scan the point light source array and obtain scan data.

[0046] This process involves using an electric displacement platform to perform scanning. Optionally, the electric displacement platform is controlled to scan the point light source array to obtain scanning data.

[0047] Step 307: Combine data processing methods to generate scan results based on the scan data.

[0048] In this embodiment, scanning results are generated based on scanned data using a data processing method. This data processing method includes at least one of the centroid method, maximum value search method, and data fitting method. In one example, data processing is performed based on a Gaussian curve and a function model.

[0049] Step 308: Based on the principle of fringe projection, load a four-step phase-shifted sine and cosine fringe pattern.

[0050] Step 309: Generate confocal super-resolution measurement raw data based on sine and cosine fringe patterns.

[0051] This process, known as structured light illumination modulation, uses the fringe projection principle to replace traditional optical interference methods for generating structured light fields. It controls the loading of a four-step phase-shifted sine and cosine fringe pattern onto a DMD chip. By controlling the phase changes of the fringe pattern, different light fields are generated, enabling high-precision capture of the fine structures on the object's surface, providing high-quality raw data for subsequent image reconstruction.

[0052] Step 310: Using the low-rank decomposition method, extract the intrinsic signals from the confocal super-resolution measurement data to obtain the confocal super-resolution measurement data.

[0053] In this embodiment, the low-rank decomposition method is a reconstruction method based on self-supervised deep learning.

[0054] In actual image acquisition, noise interference is common. Low-rank decomposition (LLD) can be used to extract intrinsic signals and remove noise from the mixed signal. Assume the observed image matrix is ​​X∈Rm×n, containing both useful intrinsic signals and noise. The goal of LLD is to decompose it into a low-rank matrix L (representing the intrinsic signals) and a sparse matrix S (representing noise). Its optimization objective can be expressed by the following formula: , Where L is a low-rank matrix (representing the intrinsic signal) and S is a sparse matrix (representing noise). It is the nuclear norm of the matrix. It is a sparsity constraint. and It is the regularization parameter. By optimizing this problem, it is possible to extract clean intrinsic signals from mixed signals and effectively remove noise components.

[0055] Optionally, to further optimize the image reconstruction process, a generative adversarial network (GAN) is employed in conjunction with a self-supervised deep learning method. Specifically, the generator network starts from the intrinsic signals obtained from low-rank decomposition to generate high-resolution reconstructed images; the discriminator network evaluates the similarity between the generated images and the real images and provides feedback to the generator for optimization. The ultimate optimization goal is to train the generator and discriminator through the combined action of adversarial loss and reconstruction loss. Through adversarial training, the generator can gradually improve the quality of the generated images, achieve super-resolution image reconstruction, further enhance the signal quality, and remove noise.

[0056] Step 311: Apply the structured light illumination confocal super-resolution measurement results to perform accuracy verification and obtain the accuracy verification results.

[0057] Step 312: Generate adjustment parameters based on the accuracy verification results, and adjust the parameters using a modified self-supervised deep learning method.

[0058] In this embodiment, a series of experiments were conducted to verify the feasibility and accuracy of the proposed method. First, the imaging effects of a traditional confocal microscope and a DMD confocal super-resolution measurement system were compared at different resolutions. Second, three-dimensional reconstruction and super-resolution imaging experiments were performed on different sample surfaces to evaluate the actual performance of the system. These experiments will more accurately assess the potential of the DMD confocal super-resolution system in practical applications, especially its advantages in improving imaging resolution and imaging speed. Finally, combining quantitative analysis and image processing results, this invention provides empirical support for the application of digital micromirror devices in the field of optical measurement technology.

[0059] In summary, the method provided in this application, a confocal super-resolution measurement method based on DMD, combined with structured light field and signal-to-noise separation technology, can achieve high-precision, high-resolution three-dimensional surface reconstruction. Utilizing the flexibility of DMD and the advantages of super-resolution technology, this invention effectively improves the spatial resolution and speed of confocal imaging, making the measurement process more accurate. Furthermore, the fusion of low-rank decomposition and self-supervised deep learning optimizes the image reconstruction process and suppresses noise interference, thereby further improving the overall performance of the system.

[0060] The method provided in this application, through precise system calibration and optimization, enables high-resolution 3D imaging. First, a calibration step based on the optical conjugation principle ensures a precise conjugation relationship between the DMD and the CMOS camera, providing a reliable foundation for subsequent imaging. Furthermore, optimized design of the DMD point source array, with appropriate selection of point size and spacing, guarantees the system's efficiency and accuracy in acquiring image data. In addition, the use of the fringe projection principle and low-rank decomposition algorithm effectively improves image quality, removes noise, and ensures the clarity of the imaging signal. Finally, combined with a self-supervised deep learning method, generative adversarial networks further optimize the image reconstruction process, significantly improving imaging resolution and quality. This invention not only improves the resolution, imaging speed, and accuracy of confocal microscopy but also provides effective technical support for super-resolution imaging in practical applications.

[0061] The above are merely optional embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A structured light illumination confocal super-resolution measurement system, characterized in that, The system includes a digital micromirror device (DMD), a sample platform, an electric displacement platform, a microscopic imaging component, a complementary metal-oxide-semiconductor (CMOS) imaging component, and a computer device. The DMD, the sample platform, the electric displacement platform, the microscopic imaging component, and the CMOS imaging component are all communicatively connected to the computer device. The DMD is used to adjust the point light source array and the structured light field; The sample platform is used to hold the sample to be tested; The electric displacement platform is used to perform displacement scanning; The CMOS imaging component is used to record optical signals; The computer device is used to process data to obtain the results of structured light illumination confocal super-resolution measurements.

2. The system according to claim 1, characterized in that, The sample to be tested is implemented as a plane mirror.

3. A structured light illumination confocal super-resolution measurement method, characterized in that, The method is applied to a computer device within the structured light illumination confocal super-resolution measurement system as described in claim 1 or 2, and the method includes: Constructing the conjugate relationship between DMD and CMOS imaging components; Construct a point light source array corresponding to the DMD; The electric displacement platform is controlled to scan the point light source array to obtain scanning results, which are then converted into image data. Generate confocal super-resolution measurement raw data based on the image data; The original confocal super-resolution measurement data is reconstructed to obtain confocal super-resolution measurement data.

4. The method according to claim 3, characterized in that, The construction of the conjugate relationship between the DMD and the CMOS imaging components includes: Determine the sample plane mirror and the sample calibration diagram corresponding to the sample plane mirror; Load the sample calibration map into the DMD; The DMD is calibrated using the sample calibration map to establish a conjugate relationship between the DMD and the CMOS imaging component.

5. The method according to claim 4, characterized in that, The step of calibrating the DMD using the sample calibration map to establish a conjugate relationship between the DMD and the CMOS imaging component includes: Determine at least three feature points in the sample calibration map, and the coordinates of the feature points corresponding to the features; The sample calibration map is coarsely calibrated using the coordinates of the feature points. In response to the completion of coarse calibration, fine calibration based on axial conjugation is performed, taking into account the relative positions of the DMD and the CMOS imaging component, to establish the conjugation relationship between the DMD and the CMOS imaging component.

6. The method according to claim 3, characterized in that, The construction of the point light source array corresponding to the DMD includes: Obtain the modulation parameters corresponding to the DMD; Based on the modulation parameters, a point light source array corresponding to the DMD is constructed.

7. The method according to claim 3, characterized in that, The process of controlling the electric displacement platform to scan the point light source array and obtaining the scanning results includes: The electric displacement platform is controlled to scan the point light source array to obtain scanning data; Based on the scanned data, a scan result is generated using a data processing method, which includes at least one of the centroid method, maximum value search method, and data fitting method.

8. The method according to claim 3, characterized in that, The generation of confocal super-resolution measurement raw data based on the image data includes: Combining the principle of fringe projection, a four-step phase-shifting sine and cosine fringe pattern is loaded; The original confocal super-resolution measurement data is generated based on the sine and cosine fringe pattern.

9. The method according to claim 3, characterized in that, The process of reconstructing the original confocal super-resolution measurement data to obtain confocal super-resolution measurement data includes: The confocal super-resolution measurement data is obtained by extracting intrinsic signals from the confocal super-resolution measurement data using the low-rank decomposition method, which is a reconstruction method based on self-supervised deep learning.

10. The method according to claim 9, characterized in that, The method further includes: The accuracy was verified using the structured light illumination confocal super-resolution measurement results, and the accuracy verification results were obtained. Adjustment parameters are generated based on the accuracy verification results, and the adjustment parameters are obtained using a modified self-supervised deep learning method.

Images