Machine learning assisted zoom-distortion coupled coaxial optical three-dimensional reconstruction method and system

Abstract

The application discloses a machine learning assisted zoom-distortion coupling coaxial optical three-dimensional reconstruction method and system. The application constructs a zoom-distortion evolution parameterization model based on a deep neural network (DNN), learns the complex nonlinear distortion law of a microscope in a continuous zooming process through AI, and realizes secondary calibration-free measurement under full magnification. In combination with a coaxial optical path design and a hardware trigger synchronization technology, the application solves the problems of many shadow obstructions, complicated manual calibration and low decoding accuracy in a high-reflectivity environment in micro three-dimensional reconstruction. The application is suitable for high-precision rapid detection of micro-electro-mechanical systems and biological microstructures.

Description

Technical Field

[0001] This invention belongs to the field of optical measurement technology, and relates to the cross-application technology of optical precision measurement and artificial intelligence, especially to the coaxial optical three-dimensional reconstruction method and system of machine learning-assisted zoom-distortion coupling. Background Technology

[0002] With the development of industries such as semiconductors, MEMS, micro-nano fabrication, precision optics, and biochips, the size of microstructures has decreased to the micrometer and nanometer scale. The sidewall height, trench depth, surface roughness, and three-dimensional contour of devices directly determine product performance. However, two-dimensional microscopic imaging alone cannot quantitatively detect geometric dimensions. Therefore, microscopic three-dimensional morphology measurement has become an essential technology for precision quality inspection and materials characterization. Existing microscopic three-dimensional measurement equipment, such as confocal microscopes, suffers from slow measurement speed and high cost. Although traditional structured light technology is fast, it faces two major bottlenecks in the microscopic field: first, the projection shadows caused by physical occlusion; and second, the extremely complex distortion rules during microscope zooming, making it difficult for traditional mathematical modeling to maintain high accuracy across the entire magnification range, and manual calibration is extremely time-consuming each time.

[0003] With the development of AI technology, using the powerful nonlinear fitting ability of neural networks to optimize optical system calibration has become a new possibility, but there is currently a lack of mature AI calibration solutions for coaxial microscopic 3D systems. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a coaxial optical 3D reconstruction method and system with machine learning-assisted zoom-distortion coupling, which solves the problems of numerous projection shadows and unstable zoom calibration accuracy in the 3D measurement of small objects.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method includes the following steps:

[0007] Step 1: Couple the illumination optical axis of the projector into the common optical axis of the microscope's main objective lens to achieve coaxial or near-coaxial coupling;

[0008] Step 2: Acquire images of the micro calibration plate at multiple discrete magnification nodes through the microscope eyepiece, identify and extract the sub-pixel corner features of the micro calibration plate at different magnifications, calculate the camera's intrinsic parameter matrix K, nonlinear distortion coefficient matrix D, and binocular relative extrinsic parameter matrices R and T at each magnification node, and construct a full magnification calibration dataset.

[0009] Step 3: Construct a zoom-distortion evolution calibration model and train it using a full magnification calibration dataset. The zoom value z fed back by the mechanical scale of the microscope zoom mechanism or the encoder is used as the input feature, and the corresponding calibration parameter set is used as the output label.

[0010] Step 4: During actual measurement, multi-step phase-shift sinusoidal fringes and auxiliary unwrapping patterns projected by the projector onto the surface of the microscopic sample are collected to obtain a complete deformed fringe image sequence; the polar line correction of the image is completed using the calibration parameters obtained in the current state from the zoom-distortion evolution calibration model, and sub-pixel-level stereo matching is completed under the absolute phase consistency constraint; microscopic high-precision three-dimensional point cloud reconstruction is completed.

[0011] Furthermore, the coupling is achieved using a beam splitter prism, which is positioned between the microscope's main objective lens and the object under test.

[0012] Furthermore, dual industrial cameras are mounted at the binocular interface of the microscope, and in step 4, images are acquired through the dual industrial cameras.

[0013] Furthermore, the dual industrial cameras achieve microsecond-level synchronization with the projected frames of the projector under hard trigger signal control.

[0014] Furthermore, prior to step 4, the method includes: pre-projecting a grayscale image from the projector to extract a full-field reflective mask to adaptively adjust the projection brightness.

[0015] Furthermore, in step 4, completing the epipolar correction of the image using the calibration parameters obtained from the zoom-distortion evolution calibration model in the current state includes:

[0016] Based on the current zoom scale or zoom value, the zoom-distortion evolution calibration model outputs the calibration parameters in real time under the current state, and uses the calibration parameters to complete the epipolar correction and distortion removal of the image.

[0017] Furthermore, in step 4, after obtaining the complete deformed stripe image sequence, the absolute phase solution is also included: using the acquired phase-shifted deformed stripe image, the wrapped phase map with the principal value truncated is calculated by the phase-shifting algorithm; the stripe level of each pixel is obtained by combining the synchronously acquired auxiliary pattern decoding, and the stripe level is superimposed with the wrapped phase to complete the phase unfolding operation, thereby obtaining the continuous absolute phase maps of the left and right views respectively.

[0018] Furthermore, in step 4, completing sub-pixel-level stereo matching under absolute phase consistency constraints includes:

[0019] In the left and right views after epipolar correction, corresponding pixels are found based on epipolar constraints and absolute phase consistency constraints; sub-pixel level stereo matching is achieved through linear interpolation algorithm to obtain parallax.

[0020] Furthermore, in step 4, the microscopic high-precision three-dimensional point cloud reconstruction includes: combining the camera intrinsic and extrinsic parameter data output by the zoom-distortion evolution calibration model, using the multi-frequency heterodyne phase shift algorithm to obtain the wrapping phase, combining the calibration parameters, and accurately determining the three-dimensional spatial coordinates of each matching point through the spatial triangulation principle to reconstruct the three-dimensional point cloud of the microscopic sample.

[0021] This invention also discloses a machine learning-assisted zoom-distortion coupling coaxial optical 3D reconstruction system, including a stereo microscope, a beam splitter, an industrial camera, a DLP projector, and a microcontroller. The beam splitter is positioned between the stereo microscope objective and the object under test, achieving coaxial or near-coaxial coupling between the optical axis of the DLP projector and the binocular observation optical axis of the microscope. The industrial camera is mounted on the eyepiece of the microscope objective. The microcontroller is connected to the industrial camera and the DLP projector, and implements steps 2-4 of the machine learning-assisted zoom-distortion coupling coaxial optical 3D reconstruction method.

[0022] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0023] 1. This invention ingeniously utilizes a beam splitter to coaxially or nearly coaxially integrate the illumination path of a micro-projector with the main objective lens of a stereomicroscope. This avoids the severe physical obstruction and projection shadows that are easily generated by traditional off-axis structured light in close-range measurements of tiny objects, enabling shadow-free full-field-of-view measurements in microscopic scenes. This ensures that high-frequency structured light can be projected perpendicularly and without obstruction onto the surface of complex microscopic samples. Combined with the unobstructed and blind-zone-free synchronous acquisition by a binocular industrial camera, full-field-of-view topography acquisition is achieved.

[0024] 2. An AI-based zoom-distortion evolution calibration model is proposed, achieving one-time calibration and real-time dynamic compensation at full magnification. With the introduction of the AI ​​evolution model, users no longer need to recalibrate when switching microscope magnifications; the system automatically compensates for distortion, achieving true full-magnification dynamic reconstruction and significantly improving operational efficiency. This not only eliminates tedious repetitive calibration work and greatly lowers the professional threshold for operators, but also enables the system to dynamically adapt to continuous 3D measurements during continuous zooming.

[0025] 3. Based on high-precision calibration, this invention can correct the nonlinear distortion caused by the magnification optical path of the microscope, and achieve accurate phase calculation by combining high-frequency fringe projection at the microscale.

[0026] 4. The method of the present invention can dynamically adjust the projector output to ensure that high-quality stripe images can be obtained on reflective metal or semi-transparent biological samples, and can cope with the overexposure problem caused by complex microscopic surface materials.

[0027] 5. The system architecture of this invention has low modification costs and strong versatility. The integrated optical path design does not require disruption of the existing core optical structure of the stereomicroscope; the system upgrade from two-dimensional observation to three-dimensional measurement can be achieved simply by adding a beam-splitting component and a standard industrial camera to the external optical path. This solution significantly reduces the hardware threshold and modification costs of high-precision microscopic three-dimensional measurement equipment, possesses extremely high flexibility and portability, and is easily promoted and applied on a large scale in scientific research laboratories and micro-mechanical manufacturing enterprises. Attached Figure Description

[0028] Figure 1 A schematic diagram of a coaxial optical 3D reconstruction system for machine learning-assisted zoom-distortion coupling;

[0029] Figure 2 This is a diagram of the internal architecture of the zoom-distortion evolution calibration model.

[0030] Explanation of reference numerals in the attached figures:

[0031] 1-Stereomicroscope, 2-Beam splitter prism, 3-Beam splitter prism holder, 4-Industrial camera, 5-DLP projector, 6-Projector adapter, 7-Projector support frame. Detailed Implementation

[0032] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0033] This application discloses a machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method, utilizing a stereomicroscope binocular structure to establish a 3D reconstruction system. The main working principle of this system is based on the refraction of structured light projected by a 50:50 beam-splitter with an anti-reflection coating onto the workpiece, causing distortion of the structured light. The image can then be transmitted through the prism to two industrial cameras connected to the binocular lens tubes, thus enabling 3D reconstruction of the workpiece using images acquired from different perspectives. Figure 1 As shown, the hardware in the system consists of a stereo microscope 1, a 50:50 beam splitter 2 with an antireflective coating, a beam splitter frame 3, an industrial camera 4, a DLP projector 5, a projector adapter 6, a projector support frame 7, and a microcontroller. The DLP projector 5, the industrial camera 4, and the microcontroller have signal / electrical connections.

[0034] The system utilizes a stereomicroscope 1 as a key unit, available in CMO and Greenau types. The former uses a shared primary objective lens, through which light is then split into two independent eyepiece systems. The latter consists of two independent objectives and eyepieces. Due to the need for microscopic dual-scope calibration, the latter introduces significant computational errors and calibration difficulties; therefore, the CMO type is used in this system. A 50:50 beam splitter 2 with an anti-reflection coating is used because ordinary beam splitters are flat plates. When placed at a 45-degree angle in the optical path, they produce monochromatic aberrations such as astigmatism, leading to decreased image sharpness. A beam splitter prism, however, is composed of two right-angled prisms cemented together. Light passes perpendicularly through the cubic surface, introducing almost no additional aberrations and maximizing the original image quality. High-quality beam splitters have a high-performance anti-reflection coating on their transmission surface and a precision beam-splitting coating on their internal inclined surfaces. This not only ensures the light transmittance of the optical path but, more importantly, minimizes stray light and glare. For 3D reconstruction, this means that the projected Gray code or phase-shifted fringe pattern can be imaged with higher contrast, significantly improving the accuracy of decoding and the precision of point clouds.

[0035] The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method provided by this invention has the following process:

[0036] First, without disrupting the original binocular observation optical path of the stereomicroscope, a 50:50 broadband depolarizing beam-splitter (NPBS) with an antireflection coating is seamlessly embedded between the primary objective lens of the CMO-type stereomicroscope 1 and the object under test. Utilizing the reflection and transmission characteristics of the beam-splitter, the optical axis of a DLP miniature digital projector fixed to the side of the beam-splitter is coaxially or nearly coaxially coupled with the binocular observation optical axis of the microscope. Two global shutter industrial cameras are then mounted at the microscope's binocular interface. The DLP projector is fixed to the support frame via an adapter. Traditional off-axis structured light is prone to severe physical obstruction and projection shadows during close-range measurement of small objects. The coaxial design of this invention ensures that high-frequency structured light can be projected perpendicularly onto the surface of the microscopic object under test (such as deep holes or micro-grooves). The two industrial cameras synchronously acquire images of deformed fringes highly modulated by the object's surface through the microscope's binocular eyepiece optical path without blind spots.

[0037] At the acquisition and control end, an STM32 microcontroller is used to capture the synchronization pulses of the DLP projection frame and send nanosecond-level hardware trigger signals to the dual cameras to ensure strict timing synchronization. Since the intrinsic parameters and distortion parameters of the CMO microscope do not change linearly during zooming, traditional polynomial fitting is insufficient to cover minute nonlinear fluctuations. Therefore, the algorithm and reconstruction stage of this invention are implemented in the microcontroller, including the following steps:

[0038] (1) Feature extraction and construction of full magnification calibration dataset. Using a high-precision micro calibration plate, images were acquired at multiple discrete magnification nodes of the microscope's continuous zoom mechanism. The sub-pixel corner coordinates of the micro calibration plate at different magnifications were automatically identified and extracted using a convolutional neural network (CNN). Based on the corner coordinates and the known world coordinates of the calibration plate, the intrinsic parameter matrix K, nonlinear distortion coefficient matrix D, and binocular relative extrinsic parameter matrices R and T of the camera at each node were calculated using the Zhang Zhengyou calibration method to construct a full magnification calibration dataset.

[0039] (2) Evolutionary Modeling and Training. A parameter prediction model based on a deep neural network (DNN) was constructed. The zoom value z fed back by the mechanical scale of the microscope zoom mechanism or the encoder was used as the input feature, and the corresponding calibration parameter set (K,D,R,T) was used as the output label for network training. This network, through a multi-layer fully connected structure, deeply learns and fits the complex and nonlinear spatial mapping law generated by the movement of internal optical elements of the microscope during continuous zooming. A zoom-distortion evolution deep neural network (Param-Net) was constructed with the microscope zoom mechanism scale (or the current zoom scale z) as input. This network was pre-trained with a large amount of calibration data at discrete magnifications to learn the deep evolution law of the intrinsic parameter matrix K(z) and the distortion coefficient D(z).

[0040] (3) In actual measurement, in order to obtain high-precision point clouds on microscopic, extremely shallow depth-of-field and complex material surfaces, this invention employs a high-frequency digital fringe combined with stereo matching technology to solve the high-frequency fringe projection and precise phase at the microscopic scale. Specifically, the process includes the following:

[0041] a) Stripe Image Acquisition: The system controls the DLP projector to pre-project a grayscale image to extract a full-field reflective mask, adaptively adjusting the projection brightness (specifically including: after pre-projecting a uniform grayscale image, acquiring the image through a camera and segmenting overexposed / high reflective areas according to a threshold, generating a binary mask, and reducing the projection grayscale value of the mask area). Subsequently, under hard triggering, it synchronously acquires multi-frequency heterodyne and phase-shifted stripe sequences modulated by the sample. Specifically, this invention introduces an adaptive brightness adjustment mechanism, which can analyze the grayscale distribution acquired by the camera in real time through grayscale thresholding or a lightweight neural network, predict the optimal projection stripe grayscale mask, and dynamically adjust the projector output to ensure high-quality stripe images can be acquired on metallic reflective or biological semi-transparent samples. Subsequently, the computer controls the DLP projector to project multi-step phase-shifted sinusoidal stripes and auxiliary unwrapping patterns (such as Gray code or multi-frequency heterodyne patterns) onto the surface of the microscopic sample. Under hard triggering signal control, dual industrial cameras achieve microsecond-level synchronous acquisition with the DLP projection frame, acquiring a complete deformed stripe image sequence.

[0042] b) Absolute phase calculation: Using the acquired phase-shifted deformed fringe image, the wrapped phase map with the principal value truncated is calculated using a phase-shifting algorithm. Subsequently, the fringe level of each pixel is obtained by decoding the synchronously acquired auxiliary pattern. The fringe level is superimposed with the wrapped phase to complete the phase unfolding operation, and continuous absolute phase maps of the left and right views are obtained respectively.

[0043] c) Subpixel stereo matching under epipolar constraints: The host computer Python program reads the current zoom scale in real time and inputs it into Param-Net. The zoom-distortion evolution calibration model trained in the previous steps outputs dynamic high-precision intrinsic parameters, extrinsic parameters, and nonlinear distortion coefficients in milliseconds. Based on this, distortion correction and epipolar correction are performed on the stripe image to align the epipolar lines in the left and right views. In the epipolar-corrected left and right views, the search for corresponding points is restricted to the same horizontal row based on epipolar constraints, and the equality of absolute phase value is used as the criterion for determining corresponding points (i.e., pixels with equal absolute phase in the left and right views are corresponding points). Subpixel-level stereo matching is achieved through linear interpolation of phase values ​​to obtain subpixel parallax.

[0044] d) 3D point cloud reconstruction: Finally, the camera intrinsic and extrinsic parameter data output by the zoom-distortion evolution calibration model are combined with the multi-frequency heterodyne phase shift algorithm to obtain the wrapping phase. Combined with the calibration parameters, the 3D spatial coordinates of each matching point are accurately determined through the spatial triangulation principle to reconstruct a high-fidelity 3D point cloud of the micro sample.

[0045] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method, characterized in that, Includes the following steps: Step 1: Couple the illumination optical axis of the projector into the common optical axis of the microscope's main objective lens to achieve coaxial or near-coaxial coupling; Step 2: Acquire images of the micro calibration plate at multiple discrete magnification nodes through the microscope eyepiece, identify and extract the sub-pixel corner features of the micro calibration plate at different magnifications, calculate the camera's intrinsic parameter matrix K, nonlinear distortion coefficient matrix D, and binocular relative extrinsic parameter matrices R and T at each magnification node, and construct a full magnification calibration dataset. Step 3: Construct a zoom-distortion evolution calibration model and train it using a full magnification calibration dataset. The zoom value z fed back by the mechanical scale of the microscope zoom mechanism or the encoder is used as the input feature, and the corresponding calibration parameter set is used as the output label. Step 4: During actual measurement, multi-step phase-shift sinusoidal fringes and auxiliary unwrapping patterns projected by the projector onto the surface of the microscopic sample are collected to obtain a complete deformed fringe image sequence; the polar line correction of the image is completed using the calibration parameters obtained in the current state from the zoom-distortion evolution calibration model, and sub-pixel-level stereo matching is completed under the absolute phase consistency constraint; microscopic high-precision three-dimensional point cloud reconstruction is completed.

2. The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method according to claim 1, characterized in that, The coupling is achieved using a beam splitter prism, which is positioned between the microscope's main objective lens and the object under test.

3. The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method according to claim 1, characterized in that, The microscope is equipped with dual industrial cameras at the binocular interface, and in step 4, images are acquired through the dual industrial cameras.

4. The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method according to claim 3, characterized in that, The dual industrial cameras achieve microsecond-level synchronization with the projected frames of the projector under hard trigger signal control.

5. The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method according to claim 1, characterized in that, Before step 4, the method further includes: pre-projecting a grayscale image from the projector to extract a full-field reflective mask to adaptively adjust the projection brightness.

6. The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method according to claim 1, characterized in that, In step 4, the epipolar correction of the image is completed using the calibration parameters obtained from the zoom-distortion evolution calibration model under the current state, including: Based on the current zoom scale or zoom value, the zoom-distortion evolution calibration model outputs the calibration parameters in real time under the current state, and uses the calibration parameters to complete the epipolar correction and distortion removal of the image.

7. The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method according to claim 1, characterized in that, In step 4, after obtaining the complete deformed stripe image sequence, the absolute phase solution is also included: using the acquired phase-shifted deformed stripe image, the wrapped phase map with the principal value truncated is calculated by the phase-shift algorithm. By combining the synchronously acquired auxiliary pattern decoding to obtain the stripe level of each pixel, the stripe level is superimposed with the wrapped phase to complete the phase unfolding operation, and continuous absolute phase maps of the left and right views are obtained respectively.

8. The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method according to claim 7, characterized in that, Step 4, in which sub-pixel-level stereo matching is performed under absolute phase consistency constraints, includes: In the left and right views after epipolar correction, corresponding pixels are found based on epipolar constraints and absolute phase consistency constraints; sub-pixel level stereo matching is achieved through linear interpolation algorithm to obtain parallax.

9. The machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction method according to claim 1, characterized in that, In step 4, the microscopic high-precision 3D point cloud reconstruction includes: combining the camera intrinsic and extrinsic parameter data output by the zoom-distortion evolution calibration model, using the multi-frequency heterodyne phase shift algorithm to obtain the wrapping phase, combining the calibration parameters, and using the spatial triangulation principle to accurately determine the 3D spatial coordinates of each matching point, thereby reconstructing the 3D point cloud of the microscopic sample.

10. A machine learning-assisted zoom-distortion coupled coaxial optical 3D reconstruction system, characterized in that, The device includes a stereo microscope, a beam splitter, an industrial camera, a DLP projector, and a microcontroller. The beam splitter is positioned between the stereo microscope objective and the object under test, enabling coaxial or near-coaxial coupling between the optical axis of the DLP projector and the binocular observation optical axis of the microscope. The industrial camera is mounted on the eyepiece of the microscope objective. The microcontroller is connected to the industrial camera and the DLP projector, and performs steps 2-4 of the coaxial optical 3D reconstruction method with machine learning-assisted zoom-distortion coupling as described in any one of claims 1-9.

Images