A microscope rapid focusing method based on controllable multi-value aperture and color encoding oblique illumination

Abstract

The application discloses a microscope rapid focusing method based on controllable multi-value aperture and color coding oblique illumination, and belongs to the technical field of optical microscopic imaging and automatic focusing. The application relies on a programmable illumination light path of a microscope, and the size and form of the illumination numerical aperture are dynamically controlled by electricity to adapt to different focusing stages. Color coding oblique illumination is used to encode the symmetrical two-way oblique illumination to the spectral channel of a color camera, and the axial defocus amount is calculated based on the transverse translation amount of the channel. A virtual structure texture is projected for a non-textured high-reflective sample, and the defocus is solved by the contrast ratio under high and low numerical apertures. A light-weight frequency domain neural network is matched to construct a coarse and fine two-stage focusing process. The application can realize single-frame snapshot focusing, has high ranging accuracy and low calculation time consumption, is suitable for smooth mirror surfaces and textured samples, has low hardware modification cost, and is suitable for scenes such as semiconductor industry detection, high-throughput pathological section scanning and super-resolution microscopic imaging.

Description

Technical Field

[0001] This invention relates to the field of optical microscopy imaging and autofocus technology, specifically to a rapid focusing method for microscopes based on controllable multi-numerical aperture and color-coded oblique illumination. Background Technology

[0002] In high-precision micro-nano observation fields such as semiconductor industrial inspection, high-throughput pathological section scanning, and super-resolution microscopy, rapid and accurate positioning of the sample focal plane is fundamental to ensuring image quality. Traditional microscope autofocusing techniques mainly rely on physical scanning along the axial direction (z-axis) and finding the focal peak by calculating image contrast or gradient operators (such as Brenner gradient and Laplace energy). However, this "hill-climbing" focusing process involves a large amount of mechanical reciprocating motion, which is not only time-consuming and inefficient, but also prone to sample damage and system mechanical wear.

[0003] To overcome the limitations of mechanical scanning, computational imaging has introduced specialized hardware such as polarization multidimensional sensors, spatial light modulator (SLM) wavefront coding, and event cameras. While these methods have achieved microsecond-level or single-frame snapshot-like focusing, their high hardware modification costs, material limitations due to sensitivity to polarization characteristics, and low spatial resolution of event sensors have limited their widespread adoption in existing commercial microscopes. Summary of the Invention

[0004] The purpose of this invention is to provide a rapid focusing method for microscopes based on controllable multi-numerical aperture and color-coded oblique illumination. It relies solely on the native programmable illumination optical path, and features coarse and fine focusing stages. It is compatible with both textured and non-textured mirror samples, and completes defocusing calculation in a single frame. It boasts fast calculation speed, high measurement accuracy, and extremely low hardware modification costs.

[0005] To achieve the above objectives, the present invention employs the following technical solution: A rapid focusing method for microscopes based on controllable multi-numerical aperture and color-coded oblique illumination is implemented using a programmable illumination optical path and a color imaging acquisition system for the microscope. The size and distribution of the lighting numerical aperture are dynamically adjusted by electronic control to adapt to different stages of the focusing process. A color-coded oblique illumination mechanism is adopted, in which two oblique illumination sources with symmetrical distribution and orthogonal spectrum are set on the illumination pupil plane. The imaging information of the two oblique illumination sources is encoded into different spectral channels of the color camera. The axial defocus of the sample is calculated based on the lateral relative translation of the images of different spectral channels. For textureless, high-reflectivity samples, a virtual structural texture is projected onto the sample surface using a programmable light source, and the axial defocus is calculated based on the image contrast ratio under different illumination numerical apertures. A lightweight frequency domain neural network is used to predict the defocus amount of the defocused image. Combined with a two-stage focusing process of coarse focusing and fine focusing, the automatic focusing and three-dimensional depth estimation of the sample are completed.

[0006] Furthermore, the color-coded oblique illumination mechanism is specifically as follows: A pair of biased semicircular or point light sources with orthogonal and symmetrical spectral distribution are projected onto the illumination pupil, corresponding to the red and green channels of the color camera, respectively. When the sample is at the focal plane, the imaging features of the two spectral channels completely overlap. When the sample is out of focus, the two spectral channel images produce lateral relative translations in opposite directions, and the amount of translation and the amount of axial defocusing satisfy a geometric perspective linear relationship.

[0007] Furthermore, the calculation process for axial defocusing is as follows: Fast Fourier transform is performed on the images of the red and green channels respectively. Normalized cross-correlation is performed in the frequency domain to obtain the correlation peak. Quadratic surface fitting is performed on the extreme points of the correlation peak and their 3×3 neighborhood to extract the subpixel level lateral translation. The absolute axial defocus amount and adjustment direction are then obtained by linear conversion through pre-calibrated coefficients.

[0008] Furthermore, the virtual structure texture is a two-dimensional high-frequency grid pattern in the form of squares, stripes, or sinusoidal stripes, which is projected onto the sample surface through the original coaxial illumination path of the microscope; The imaging degradation model of the sample satisfies the virtual texture pattern. With system defocus PSF operator The relationship between convolutional additive noise and its superposition: , in, It is additive noise; Two images were acquired by rapidly switching between high and low illumination numerical apertures, and the gradient contrast ratio of the two images was calculated. This ratio is used to eliminate the interference of non-uniform reflectivity of the sample surface and is only related to the axial defocus amount.

[0009] Furthermore, the illumination numerical aperture is divided into at least four settings, each adapted to different focusing stages: Extremely low numerical aperture configuration, less than 0.2, high coherence and wide depth of field, used for wide-range coarse focusing and single-frame coherent frequency domain defocus prediction; Medium numerical aperture configuration, with a value range of [0.2, 0.7], is used for virtual structure texture projection and focus transition positioning; High numerical aperture configuration, with a value greater than 0.7 and matching the numerical aperture of the objective lens, provides strong axial tomography capability for high-precision focusing and peak verification. The annular bias numerical aperture is used to enhance the high-frequency phase gradient, reduce the frequency loss in the three-dimensional Fourier space, suppress anisotropic measurement artifacts, and achieve high-precision focus locking.

[0010] Furthermore, lightweight frequency domain neural networks include fully connected Fourier neural networks and modified U-Net networks; The fully connected Fourier neural network is used in the coarse focusing stage. It extracts the single-pass defocusing frequency band spectrum magnitude value centered on the oblique illumination frequency shift in the oblique illumination image as the input feature, and obtains the coarse defocusing amount through regression of the fully connected layer. The improved U-Net network prunes deep skip connections, blocks the direct transmission of low-frequency macroscopic features, and forces the network to extract high-frequency diffraction details corresponding to defocus, which are used for high-resolution phase and depth residual prediction.

[0011] Furthermore, the two-stage focusing process of coarse focusing and fine focusing is as follows: Coarse focusing stage: Using extremely low numerical aperture coherent illumination, the coarse defocusing amount is calculated through a lightweight frequency domain neural network, and the axial motor is driven to move in one direction to converge the defocusing error to a set threshold. Precision focusing stage: Switch to high numerical aperture color-coded oblique illumination, acquire a single frame color image and calculate the precise defocus amount, and complete precise focusing through a closed-loop fine-tuning with extremely small step size; After focusing, switch to high numerical aperture illumination of the matching objective lens to perform conventional microscopic imaging and three-dimensional depth estimation.

[0012] Furthermore, for textureless, high-reflectivity samples, a virtual structure texture projection mode is automatically enabled before the coarse focusing stage to provide image contrast support for focusing calculations.

[0013] The advantages of this invention are: It offers faster focusing speed and eliminates mechanical wear. Both working modes utilize single-frame snapshot acquisition. The color-coded fine focusing mode algorithm takes less than 8ms, while the fully connected Fourier neural network coarse focusing mode takes less than 1ms, far superior to traditional multi-frame scanning focusing solutions. Furthermore, it eliminates the need for axial reciprocating scanning, requiring only unidirectional motor feeding, significantly reducing the risk of mechanical wear and sample damage.

[0014] The hardware modification cost is extremely low; only the existing programmable illumination control signals need to be reconstructed to upgrade the functionality of commercial microscopes. Therefore, it has a wide range of sample compatibility. Combined with a virtual structure texture projection scheme, it can achieve micron-level precise focusing on highly reflective, textureless mirror-like samples such as semiconductor dies and wafers, solving the problem of traditional passive focusing and event camera focusing failing on such samples.

[0015] The measurement performance is well-balanced, with an axial ranging accuracy of approximately 80nm and a measurement range of ±15μm in fine focusing mode; and a ranging accuracy of approximately 200nm and a measurement range of ±40μm in coarse focusing mode. This two-stage approach balances the needs for wide-range defocus capture and high-precision focusing. Meanwhile, ring illumination reduces the missing cone of the three-dimensional Fourier spatial frequency, ensuring isotropic depth measurement, effectively suppressing step artifacts, and a lightweight network overcomes spectral bias issues, preserving complete edge details in the depth map. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the illumination pupil surface of the present invention; Figure 2 This is a schematic diagram of the focusing optical path for the textureless sample of the present invention; Figure 3 The flowchart of the improved depth prediction network of this invention is shown below; Figure 4 This is the system interaction logic of the present invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0018] Example 1 This embodiment is designed for samples with natural textures and uses a ring-shaped high numerical aperture color-coded oblique illumination mode to achieve single-frame precision focusing.

[0019] A pair of orthogonally oriented and symmetrically distributed offset semi-circular light sources are projected onto the illumination pupil plane of the microscope, emitting red and green light respectively. The two light sources are symmetrically arranged about the optical axis, forming a ring-shaped color-coded oblique illumination structure, with the illumination numerical aperture matched to a high numerical aperture setting. Its physical basis is the parallax mapping effect: when a collimated coherent beam is incident on the sample surface at a specific angle off the optical principal axis, if the sample deviates from the objective lens focal plane, its reflected or transmitted image on the sensor will produce a lateral translation linearly related to the defocus distance.

[0020] like Figure 1 As shown, when the sample is at the ideal focal plane, the imaging features of the red channel and the green channel completely overlap; when the sample is out of focus, the two spectral channel images produce lateral relative translations in opposite directions. The amount of translation and the amount of axial defocus satisfy a geometric perspective linear relationship, that is: , in This represents the lateral translation amount of the two-channel image. Where is the axial defocusing amount, and M is the lateral magnification of the microscope objective. The equivalent offset tilt angle is determined by the numerical aperture of the illumination.

[0021] A color camera can acquire a single frame of RGB image in a single exposure. The red channel image corresponds to the sample projection under left oblique illumination, and the green channel image corresponds to the sample projection under right oblique illumination. The imaging time for a single exposure is usually less than 10ms.

[0022] When calculating the defocus amount, a Fast Fourier Transform is performed on the images of the red and green channels respectively, and the normalized cross power spectrum is calculated in the frequency domain: , in, and denoted as Fast Fourier Transform and its inverse transform, respectively, and * denotes the conjugate operator.

[0023] After obtaining the correlation spectrum through inverse Fourier transform, quadratic surface fitting is performed on the correlation peaks and their 3×3 neighborhoods, achieving sub-pixel-level lateral translation positioning with a resolution better than 0.1 pixels. Then, using pre-calibrated coefficients... K Linear regression was used to calculate the absolute axial defocusing amount and adjustment direction: .

[0024] The calculated defocus amount drives the Z-axis motor to perform closed-loop fine-tuning with extremely small steps until the defocus amount meets the focusing threshold. This embodiment employs a ring-shaped bias illumination mode, which can reduce the frequency missing cone in the three-dimensional Fourier space, maintain the isotropic nature of the depth gradient at each lateral azimuth angle, eliminate the staircase artifacts in irregular shape depth estimation, and suppress anisotropic measurement artifacts.

[0025] This embodiment requires only one frame of image for a single focusing attempt, the algorithm calculation time is less than 8ms, the axial ranging accuracy is about 80nm, and the defocus range can be measured within ±15μm, making it suitable for high-precision focusing scenarios using differential phase contrast.

[0026] Example 2 This embodiment targets highly reflective, textureless samples such as semiconductor wafers, bare dies, and cover glass sheets, employing a virtual structure texture projection combined with a depth defocusing method for focusing. These samples have smooth surfaces and lack natural texture features; traditional image gradient-based focusing methods cannot extract effective features, rendering the focusing function completely ineffective.

[0027] like Figure 2 As shown, this embodiment projects a two-dimensional high-frequency grid pattern in the form of squares, stripes or sinusoidal stripes through a programmable LED array. The pattern is then projected onto the sample surface through the original coaxial illumination path of the microscope, forcibly imparting a high-frequency texture to the sample surface without the need for additional physical gratings or laser auxiliary sources.

[0028] The sample imaging degradation model can be represented as the additive noise resulting from the convolution of the virtual mesh pattern and the system's out-of-focus spread function operator, i.e.: , in, A virtual grid pattern. Let be the system's out-of-focus diffusion function. It is additive noise.

[0029] Due to the projected pattern The mathematical distribution (spatial frequency, initial phase) is known. The system acquires two images by rapidly switching between high and low illumination numerical aperture levels. and Low numerical aperture illumination corresponds to a wider depth of field, while high numerical aperture illumination corresponds to a narrower depth of field. The gradient magnitudes of the two images are calculated separately, and the gradient contrast ratio is constructed. .

[0030] This contrast ratio can eliminate the interference caused by the non-uniformity of the sample surface reflectivity. It is only related to the axial defocus amount. Then, the axial defocus amount can be solved through the calibration relationship to achieve micron-level precise focusing of smooth mirror samples.

[0031] This embodiment can effectively focus on completely smooth, highly reflective mirror samples, significantly expanding the applicability of the microscopic focusing system and solving the problem of traditional passive focusing and event camera focusing failing on this type of sample.

[0032] Example 3 This embodiment demonstrates a complete adaptive two-stage focusing process, combining multi-numerical aperture control with a lightweight frequency domain neural network, covering the entire process from wide-range defocus capture to high-precision focus locking.

[0033] The illumination numerical aperture is available in four configurations, each suitable for different stages of focusing: Extremely low numerical aperture (less than 0.2), offering high coherence and wide depth of field, used for wide-range coarse focusing and single-frame coherent frequency domain defocus prediction; medium numerical aperture (0.2 to 0.7), used for virtual structure texture projection and focus transition positioning; high numerical aperture (greater than 0.7) matched to the objective lens numerical aperture, offering strong axial tomography capabilities, used for high-precision fine focusing and peak verification; and annular offset numerical aperture, used to enhance high-frequency phase gradients, reduce frequency loss in three-dimensional Fourier space, suppress anisotropic measurement artifacts, and achieve high-precision focus locking. See the table below.

[0034] Table 1. Impact of different illumination numerical aperture settings on key imaging performance indicators of the system. like Figure 4 As shown, the system first automatically identifies the surface texture of the sample. If it is a high-reflectivity sample without texture, it first turns on the virtual high-frequency grid projection mode and switches to the medium numerical aperture setting to provide contrast support for focusing calculation. If it is a sample with texture, it directly enters the coarse focusing stage.

[0035] like Figure 3 As shown, in the coarse focusing stage, the system switches to extremely low numerical aperture coherent illumination, acquires a single frame of oblique illumination image, and uses a lightweight frequency domain neural network to predict the defocus amount. The lightweight frequency domain neural network includes a fully connected Fourier neural network and a modified U-Net network. The fully connected Fourier neural network is used in the coarse focusing stage, based on the fact that defocus information under bias illumination is highly enriched in terms of frequency shift. The characteristics of the single-shot radio frequency band centered on the radio frequency. The wavelength is used for imaging.

[0036] The FCFNN algorithm uses a one-dimensional passband frequency domain mask to extract the complex spectral magnitude within the scattered frequency band as the input feature vector: , in, It is the original image. To perform a Fourier transform and obtain the modulus, the subscript... To preserve this frequency domain vector Center frequency, half bandwidth The input dimension is compressed from millions of pixels in the original image to a few thousand feature points. A fully connected Fourier neural network, containing only three fully connected layers, can achieve high-precision linear regression of defocus, with computation time controlled within 0.5ms, and the number of parameters reduced by two to three orders of magnitude compared to traditional deep networks.

[0037] After the network outputs the coarse defocusing amount, it drives the Z-axis motor to move in one direction, reducing the axial defocusing error to 3μm.

[0038] The improved U-Net network prunes deep skip connections, blocking the direct transmission of low-frequency macroscopic features and forcing the network to extract high-frequency diffraction details corresponding to defocus, which is used for high-resolution phase and depth residual prediction. This physically blocks the coverage of depth features by low-frequency components and overcomes the spectral bias problem in deep learning.

[0039] During the fine focusing stage, the system switches to ring-shaped high numerical aperture color-coded oblique illumination, acquires a single frame of RGB image, and calculates the precise defocus amount through frequency domain normalized cross-correlation and sub-pixel fitting. If the defocus amount exceeds the threshold, a very small step size closed-loop fine adjustment is performed until focusing is completed.

[0040] After focusing, switch to high numerical aperture illumination with the matching objective lens to perform routine microscopic imaging and sample three-dimensional depth estimation.

[0041] In this embodiment, the coarse focusing mode can cover a defocus range of ±40μm, with a ranging accuracy of approximately 200nm and an algorithm execution time of less than 1ms; the fine focusing mode has a ranging accuracy of approximately 80nm, a measurement range of ±15μm, and an algorithm execution time of less than 8ms. This two-stage process balances the requirements of a large defocus capture range and high-precision focusing, with extremely low overall hardware modification costs. It is compatible with upgrades to existing commercial microscopes and can be applied to high-precision micro-nano observation scenarios such as semiconductor industrial inspection, high-throughput pathological slide scanning, and super-resolution microscopic imaging.

[0042] The table below compares the performance of the present invention's computational microscopic focusing method with the traditional Z-stack image sharpness method and existing solutions using polarization devices and event cameras on several key indicators.

[0043] Table 2 Performance Comparison of Different Algorithm Configurations The above comparison clearly demonstrates that this invention achieves decisive performance advantages in all core dimensions, including axial ranging accuracy, algorithm computation time, adaptability to textureless mirrors, and functional scalability. This provides a more advanced and faster means for the practical application of this technology in high-precision micro-nano observation fields such as semiconductor industrial inspection, high-throughput pathological slide scanning, and super-resolution microscopy.

[0044] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A rapid focusing method for a microscope based on controllable multi-numerical aperture and color-coded oblique illumination, characterized in that, Implementation of a microscope-based programmable illumination optical path and color imaging acquisition system: The size and distribution of the lighting numerical aperture are dynamically adjusted by electronic control to adapt to different stages of the focusing process. A color-coded oblique illumination mechanism is adopted, in which two oblique illumination sources with symmetrical distribution and orthogonal spectrum are set on the illumination pupil plane. The imaging information of the two oblique illumination sources is encoded into different spectral channels of the color camera. The axial defocus of the sample is calculated based on the lateral relative translation of the images of different spectral channels. For textureless, high-reflectivity samples, a virtual structural texture is projected onto the sample surface using a programmable light source, and the axial defocus is calculated based on the image contrast ratio under different illumination numerical apertures. A lightweight frequency domain neural network is used to predict the defocus amount of the defocused image. Combined with a two-stage focusing process of coarse focusing and fine focusing, the automatic focusing and three-dimensional depth estimation of the sample are completed.

2. The microscope rapid focusing method based on controllable multi-numerical aperture and color-coded oblique illumination according to claim 1, characterized in that, The color-coded oblique illumination mechanism is specifically as follows: A pair of biased semicircular or point light sources with orthogonal and symmetrical spectral distribution are projected onto the illumination pupil, corresponding to the red and green channels of the color camera, respectively. When the sample is at the focal plane, the imaging features of the two spectral channels completely overlap. When the sample is out of focus, the two spectral channel images produce lateral relative translations in opposite directions, and the amount of translation and the amount of axial defocusing satisfy a geometric perspective linear relationship.

3. The microscope rapid focusing method based on controllable multi-numerical aperture and color-coded oblique illumination according to claim 2, characterized in that, The calculation process for the axial defocusing amount is as follows: Fast Fourier transform is performed on the images of the red and green channels respectively. Normalized cross-correlation is performed in the frequency domain to obtain the correlation peak. Quadratic surface fitting is performed on the extreme points of the correlation peak and their 3×3 neighborhood to extract the subpixel level lateral translation. The absolute axial defocus amount and adjustment direction are then obtained by linear conversion through pre-calibrated coefficients.

4. The microscope rapid focusing method based on controllable multi-numerical aperture and color-coded oblique illumination according to claim 1, characterized in that, The virtual structure texture is a two-dimensional high-frequency grid pattern in the form of squares, stripes, or sinusoidal stripes, which is projected onto the sample surface through the original coaxial illumination path of the microscope. The imaging degradation model of the sample satisfies the virtual texture pattern. With system defocus PSF operator The relationship between convolutional additive noise and its superposition: ， in, It is additive noise; Two images were acquired by rapidly switching between high and low illumination numerical apertures, and the gradient contrast ratio of the two images was calculated. This ratio is used to eliminate the interference of non-uniform reflectivity of the sample surface and is only related to the axial defocus amount.

5. The microscope rapid focusing method based on controllable multi-numerical aperture and color-coded oblique illumination according to claim 1, characterized in that, The illumination numerical aperture is configured with at least four levels to suit different focusing stages: Extremely low numerical aperture configuration, less than 0.2, high coherence and wide depth of field, used for wide-range coarse focusing and single-frame coherent frequency domain defocus prediction; Medium numerical aperture configuration, with a value range of [0.2, 0.7], is used for virtual structure texture projection and focus transition positioning; High numerical aperture configuration, with a value greater than 0.7 and matching the numerical aperture of the objective lens, provides strong axial tomography capability for high-precision focusing and peak verification. The annular bias numerical aperture is used to enhance the high-frequency phase gradient, reduce the frequency loss in the three-dimensional Fourier space, suppress anisotropic measurement artifacts, and achieve high-precision focus locking.

6. The microscope rapid focusing method based on controllable multi-numerical aperture and color-coded oblique illumination according to claim 1, characterized in that, The lightweight frequency domain neural network includes a fully connected Fourier neural network and a modified U-Net network; The fully connected Fourier neural network is used in the coarse focusing stage. It extracts the single-pass defocusing frequency band spectrum magnitude value centered on the oblique illumination frequency shift in the oblique illumination image as the input feature, and obtains the coarse defocusing amount through regression of the fully connected layer. The improved U-Net network prunes deep skip connections, blocks the direct transmission of low-frequency macroscopic features, and forces the network to extract high-frequency diffraction details corresponding to defocus, which are used for high-resolution phase and depth residual prediction.

7. The rapid focusing method for a microscope based on controllable multi-numerical aperture and color-coded oblique illumination according to claim 1, characterized in that, The two-stage focusing process of coarse focusing and fine focusing is as follows: Coarse focusing stage: Using extremely low numerical aperture coherent illumination, the coarse defocusing amount is calculated through a lightweight frequency domain neural network, and the axial motor is driven to move in one direction to converge the defocusing error to a set threshold. Precision focusing stage: Switch to high numerical aperture color-coded oblique illumination, acquire a single frame color image and calculate the precise defocus amount, and complete precise focusing through a closed-loop fine-tuning with extremely small step size; After focusing, switch to high numerical aperture illumination of the matching objective lens to perform conventional microscopic imaging and three-dimensional depth estimation.

8. The rapid focusing method for a microscope based on controllable multi-numerical aperture and color-coded oblique illumination according to claim 7, characterized in that, For samples with no texture and high reflectivity, a virtual structure texture projection mode is automatically enabled before the coarse focusing stage to provide image contrast support for focusing calculations.

Images