Three-dimensional shape microscopic measurement method and microscopic device based on array light spots

Abstract

The invention relates to a three-dimensional shape microscopic measurement method and microscopic device based on array light spots, and the method comprises the steps: carrying out the scanning of the microscopic device from bottom to top according to a set sampling interval, so as to collect the dot matrix images of a to-be-measured object on different focal planes; according to the optical center position of each light spot calibrated in advance, calculating brightness values at the optical center positions on different focal planes by adopting Gaussian curved surface function fitting; by means of brightness information of each light spot at the optical center on different focal planes, a Lorenz function is adopted to fit a focus position where each light spot is located when the brightness value of each light spot is maximum, the depth value of each light spot is calculated, and a sparse depth map is obtained; and complementing the sparse depth map into a dense depth map to complete the three-dimensional shape measurement of the to-be-measured object. Compared with the prior art, the system has the advantages that dot matrix image acquisition is realized by using a microscopic device with low cost and simple structure, and the system is suitable for wide-range three-dimensional shape microscopic measurement.

Description

Technical Field

[0001] The invention relates to the technical field of three-dimensional reconstruction, and in particular to a three-dimensional morphology microscopic measurement method and a microscopic device based on array light spots. Background Art

[0002] Three-dimensional topography microscopy is crucial in the three-dimensional topography measurement of industrial samples. It can accurately observe the surface microstructure of the sample and measure the size, shape and surface roughness. When measuring the topographic features, the traditional three-dimensional topography microscopy decomposes the sample into a large number of three-dimensional spatial points, and uses a fine laser beam (point light source) to scan point by point to perform imaging. There are two main methods for confocal point measurement: laser scanning and disc scanning. Laser scanning is achieved by scanning optical devices (such as resonant mirrors, servo mirrors and multi-angle mirrors), which scan the laser beam by mechanically changing the mirror angle. The scanning frequency of mechanical scanners is usually in the range of 10kHz to 100kHz. Disc scanning uses a Nipkow disk system with multiple pinholes to generate multiple focal points on the object, thereby covering the entire field of view. By scanning multiple focal points simultaneously, the imaging speed of the focal plane is greatly accelerated. However, both confocal point measurement methods require high-speed moving parts, which not only limits the scanning speed of the mechanical scanner, but also affects the measurement accuracy due to the vibration generated by the parts.

[0003] In order to overcome the limitations of traditional 3D topography microscopy in terms of measurement speed and accuracy, researchers have carried out a lot of research work. By removing high-speed moving parts and using fixed electronic optical components instead, higher stability of system accuracy and repeatability is ensured, avoiding distortion or degradation of measurement capabilities caused by long-term use. For example, Sensofar has developed Microdisplay confocal scanning technology, which uses the fast optical modulation capability of FLCOS to generate programmable gratings and dynamic optical patterns to achieve fast confocal scanning without mechanical movement. Leica TCS SP8 DLS uses digital light sheet technology, using very thin light sheets to illuminate the entire plane of the sample at once, rather than scanning point by point. The optical modulator can dynamically control the shape, position and illumination angle of the light sheet, avoiding the physical movement and delay caused by traditional mechanical methods. Confocal microscopes based on digital micromirror devices control the illumination and reflection of the light beam through a programmable micromirror array, selectively illuminating different areas of the sample, and avoiding mechanical movement. However, these methods still have some problems: 1) For texture-free, smooth or highly reflective surfaces, the system may have difficulty in generating sufficient optical contrast, affecting accurate measurement. 2) Complex electronic control systems and high-precision optical components are required, resulting in high design and manufacturing costs for the entire system.

[0004] In view of the limitations of the current mainstream confocal microscopes in measuring textureless or highly reflective surfaces, as well as the problems of complex system design and high cost, a high-performance confocal three-dimensional morphology microscopy measurement method was developed, which has the characteristics of wide application range, low system complexity and low cost.

[0005] How to achieve three-dimensional morphology microscopy measurement with a wide range of applications, low system complexity and low cost has become a technical problem that needs to be solved. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a three-dimensional morphology microscopic measurement method and a microscopic device based on array light spots.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to one aspect of the present invention, a three-dimensional topography microscopic measurement method based on array light spots is provided, the method comprising the following steps:

[0009] Step S1, the microscope device scans from bottom to top at a set sampling interval to collect dot matrix images on different focal planes of the object to be measured;

[0010] Step S2, marking the optical center position of each light spot in advance based on the dot matrix image, and using Gaussian surface function fitting to calculate the brightness value at the optical center position;

[0011] Step S3: using the brightness information of each light spot at the optical center position on different focal planes, the Lorentz function is used to fit the focal position where the brightness value of each light spot is maximum, and the depth value of each light spot is calculated to obtain a sparse depth map;

[0012] Step S4: Completing the sparse depth map into a dense depth map to complete the three-dimensional shape measurement of the object to be measured.

[0013] Preferably, the sampling interval is set according to the actual height of the object being measured.

[0014] Preferably, marking the optical center position of each light spot in step S2 includes:

[0015] Step S201, collecting focal stacks of dot images at different focal planes under the mirror to synthesize a fully focused image;

[0016] Step S202, extracting the green channel of the all-focus image and calculating the first-order gradient map of the green channel;

[0017] Step S203, making a first-order gradient map of a Gaussian kernel that obeys a normal distribution as a template;

[0018] Step S204: locating the optical center position of each light spot using a normalized cross-correlation algorithm.

[0019] More preferably, the fully focused image uses the Tenengrad function as a clarity operator to calculate the clarity value of each pixel in the focal stack, thereby selecting the best focused pixel to generate the fully focused image.

[0020] More preferably, the first-order gradient map of the green channel is obtained by convolving the all-focus image with a Sobel operator.

[0021] Preferably, the Gaussian surface function is specifically:

[0022]

[0023] Among them, M(x i ,y i ) is the coordinate point (x i ,y i ) is the gray value of the light spot; (u, v) is the coordinate of the center of the light spot; σ is the standard deviation of the Gaussian function, which is related to the object distance and focal length; B is the degree of reflection of the background; T is the total energy of the light spot projected onto the CMOS, which is related to the brightness of the light source and the exposure time of the camera.

[0024] Preferably, the Lorenz function is specifically:

[0025]

[0026] Among them, C(D i ) is the gray value of the light spot on different focal planes; d is the center position of the Lorentz peak, that is, the true depth of the target point; γ is the half width of the Lorentz function, that is, the width of the Lorentz peak; B represents the degree of reflection of the background; T represents the size of the gray value of the light spot.

[0027] Preferably, the sparse depth map is completed into a dense depth map according to a circular triangular interpolation algorithm, including:

[0028] Step S401, triangulate all sub-pixel points in the sparse depth map to construct a triangle network;

[0029] Step S402, calculating the barycentric coordinates of each triangle in the triangle network, and estimating its depth value according to the barycentric coordinate method;

[0030] Step S403, adding the barycenter coordinates and depth values ​​calculated in step S402 to the original sparse depth map to expand it into a new sparse depth map, and performing triangulation again on all sub-pixel points in the new depth map to construct a triangle network;

[0031] Step S404, repeating step S402 and step S403 m times to obtain an extended depth map, where m>1.

[0032] According to another aspect of the present invention, there is provided a microscope device, the device comprising a projection system, a first semi-transparent and semi-reflective mirror, a tube lens, a second semi-transparent and semi-reflective mirror and an objective lens; the projection system comprises a laser, a plano-convex lens and a mask;

[0033] The green light emitted by the laser passes through a plano-convex lens and a mask plate, and then passes through a first semi-transparent and semi-reflective mirror, a tube lens, a second semi-transparent and semi-reflective mirror and an objective lens in sequence before reaching a focal plane.

[0034] Preferably, the device also includes an illumination system and a camera, the white light emitted by the illumination system passes through the objective lens and then reaches the focal plane; the light emitted by the focal plane passes through the objective lens, the second semi-transparent and semi-reflective mirror and the tube lens, and then is emitted to the camera through the first semi-transparent and semi-reflective mirror.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) The present invention uses a specially designed microscope to scan and collect dot images, calibrates the optical center position of each light spot in advance, fits and calculates the brightness value at the optical center position, uses the Lorentz function to fit the focal position where the brightness value of each light spot is maximum, calculates the depth value to obtain a sparse depth map, and completes the sparse depth map into a dense depth map, so as to quickly complete high-precision three-dimensional morphology measurement of the object to be measured in different industrial scenarios.

[0037] 2) The present invention imparts features to the object being measured by actively projecting array light spots, making it suitable for measuring texture-free or highly reflective surfaces, thereby improving the measurement accuracy.

[0038] 3) The microscope device designed in the present invention does not require high-speed rotating parts and complex electronic components, and has a simple structure and low cost.

[0039] 4) The present invention uses a circular triangular interpolation algorithm, which can more smoothly and stably expand from a sparse depth map to a dense depth map, and obtain richer three-dimensional morphological features. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the process of the three-dimensional topography microscopic measurement method based on array light spots in the present invention;

[0041] FIG2( a ) is a schematic diagram of the structure and principle of a microscope device based on array light spots in the present invention;

[0042] FIG2( b ) is a schematic diagram of a mask in a microscope device according to the present invention;

[0043] Figure 3It is a schematic diagram of the process of locating the optical center of a light spot in an image in the present invention;

[0044] Figure 4 Schematic diagram of Gaussian surface and Lorentz curve fitted by a single light spot on different focal planes in the present invention;

[0045] Figure 5 It is a schematic diagram of the process flow of implementing the common focusing measurement in the present invention.

[0046] Figure 6 A schematic diagram of the effect of performing three-dimensional topography measurement on the surfaces of some samples in the present invention;

[0047] In the accompanying drawings, 1 is a projection system, 2 is a first semi-transparent and semi-reflective mirror, 3 is a tube lens, 4 is a second semi-transparent and semi-reflective mirror, 5 is an illumination system, 6 is an objective lens, 7 is a camera, 8 is a focal plane, 11 is a laser, 12 is a plano-convex lens, 13 is a mask, 51 is a light source, and 52 is a lens. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0049] This embodiment relates to a three-dimensional morphology microscopic measurement method based on array light spots to solve technical problems such as low photon utilization rate in measurement of existing confocal microscopes, limitations in measuring textureless or highly reflective surfaces, and complex system design and high cost.

[0050] like Figure 1 , the method comprises the following steps:

[0051] Step S1: The microscope device scans from bottom to top at a specific sampling interval to collect dot matrix images on different focal planes of the object to be measured;

[0052] Step S2: according to the optical center position of each light spot in the image (the image taken by the camera in the microscope device) which has been calibrated in advance, a Gaussian surface function is used to fit the brightness value at the optical center;

[0053] Step S3: using the brightness information of each light spot on different focal planes, the Lorentz function is used to fit the focal position where the brightness value of each light spot is maximum, thereby calculating the depth value of each light spot to obtain a sparse depth map;

[0054] Step S4: According to the circular triangular interpolation algorithm, the sparse depth map is completed into a dense depth map to complete the three-dimensional shape measurement of the object to be measured.

[0055] In step S1, the sampling interval is determined according to the actual height of the object to be measured, and the number of dot matrix images collected is generally between 200 and 300.

[0056] The microscope device designed in this embodiment is shown in Figures 2(a) and 2(b), and the device includes a projection system 1, a first semi-transparent mirror 2, a tube lens 3, a second semi-transparent mirror 4, an illumination system 5, an objective lens 6, and a camera 7. The projection system 1 includes a laser 11, a plano-convex lens 12, and a mask 13, and the illumination system 5 includes a light source 51 and a lens 52. The green light emitted by the laser 11 passes through the plano-convex lens 12 and the mask 13, and then passes through the first semi-transparent mirror 2, the tube lens 3, the second semi-transparent mirror 4, and the objective lens 6 in sequence to reach the focal plane 8. The white light emitted by the illumination system 5 passes through the objective lens 6 to reach the focal plane 8. The light reflected from the focal plane 8 passes through the objective lens 6, the second semi-transparent mirror 4, and the tube lens 3, and then is emitted to the camera 7 through the first semi-transparent mirror 2. The camera 7 takes a picture of the reflected light to monitor the scanned dot matrix image.

[0057] In step S2, according to the pre-calibrated positions of the light spots in the image, Gaussian surface fitting is used to calculate the brightness value at the optical center. The optical center position of each light spot needs to be calibrated in advance, such as Figure 3 ,include:

[0058] Step S201, collect the focal stack of the dot image under the mirror (the mirror is placed at the focal plane 8 in FIG2 ) and synthesize the fully focused image: the fully focused image is obtained by using the Tenengrad function as a sharpness operator to calculate the sharpness value of each pixel in the focal stack in a 7×7 window, thereby selecting the best focused pixel to generate the fully focused image. The Tenengrad function is expressed as:

[0059]

[0060] Among them, G x and G y are all convolution kernels of the Sobel operator, S(x,y) is the gradient of the image H(x,y) at the point (x,y), n is the total number of pixels, and Ten is the Tenengrad function value of the image.

[0061] Step S202, extracting the green channel of the fully focused image, and calculating the first-order gradient map of the green channel; wherein the first-order gradient map is obtained by performing a convolution operation on the original image (fully focused image) and the Sobel operator.

[0062] Step S203, making a first-order gradient image of a Gaussian kernel with a size of 5×5 and obeying a normal distribution as a template;

[0063] Step S204: locating the optical center position of each light spot using a normalized cross-correlation algorithm.

[0064] In step S3, a variable for representing the degree of object reflection is added to both the Gaussian surface function and the Lorentz function to better characterize the physical characteristics of the light spot imaging. The Gaussian surface function is expressed as:

[0065]

[0066] Among them, M(x i ,y i ) is the coordinate point (x i ,y i ) is the gray value of the light spot; (u,v) is the coordinate of the center of the light spot; σ is the standard deviation of the Gaussian function, which is related to the object distance and focal length; B is the degree of reflection of the background; T is the total energy of the light spot projected onto the CMOS, which is related to the brightness of the light source and the exposure time of the camera.

[0067] The Lorenz function is expressed as:

[0068]

[0069] Among them, C(D i ) is the gray value of the light spot on different focal planes; d is the center position of the Lorentz peak, that is, the true depth of the target point; γ is the half width of the Lorentz function, that is, the width of the Lorentz peak; B represents the degree of reflection of the background; T represents the size of the gray value of the light spot. The Gaussian surface and Lorentz curve fitted by a single light spot on different focal planes are shown in Figure 1. Figure 4 shown.

[0070] In step S4, the sparse depth map is completed into a dense depth map according to the circular triangular interpolation algorithm to complete the three-dimensional morphology measurement of the sample, including:

[0071] Step S401: triangulate all sub-pixel points in the sparse depth map to construct a triangle network;

[0072] Step S402: Calculate the barycentric coordinates of each triangle in the network, and estimate its depth value according to the barycentric coordinate method;

[0073] Step S403: adding the barycenter coordinates and depth values ​​calculated in step S402 to the original sparse depth map to expand it into a new sparse depth map, and performing triangulation again on all sub-pixel points in the new depth map to construct a triangle network;

[0074] Step S404: Repeat steps S402 and S403 to obtain an extended depth map. This embodiment repeats this step twice. If the distance between the two designed points is far, it can be repeated multiple times. According to all sub-pixel points in the extended sparse depth map, the depth value corresponding to each pixel in the image is calculated by triangular interpolation. The depth value D P It is expressed as:

[0075] D P =W v1 *D v1 +W v2 *D v2 +W v3 *D v3

[0076] Among them, D v1 ,D v2 ,D v3 is the depth value corresponding to the three vertices, W v1 ,W v2 ,W v3 is the corresponding weight value. The weight value of each vertex is calculated according to the barycentric coordinate method:

[0077]

[0078] W v3 =1-W v1 -W v2

[0079] Among them, (X v1 ,Y v1 ),(X v2 ,Y v2 ),(X v3 ,Y v3 ) are the coordinates of the three sub-pixel vertices of the triangle. x ,P y ) is the pixel coordinate that needs to be solved.

[0080] This embodiment also relates to a three-dimensional morphology microscopic measurement method based on array light spots, using Daheng's 5-megapixel industrial camera MER2-503-23GC as an image acquisition device, and the laser 11 uses a high-brightness green point light source (OPT-PIG0803-G) as a projection imaging light source to achieve three-dimensional morphology measurement; the lighting system 5 uses a direct white circular light source (OPT-LDS50-A15-W) as a color imaging light source for observing actual three-dimensional scenes.

[0081] like Figure 5 , the focal stack of the dot matrix images collected on different focal planes of the object to be measured is input into the algorithm of the present invention, and the three-dimensional shape measurement of the object to be measured is output.

[0082] High-precision equipment from home and abroad was selected for measurement accuracy comparison: China Library Confocal Probe (CHOTEST), Sensofar white light interferometer, Sensofar confocal microscope and focus recovery shape (Shape from focus) algorithm. The height of the standard gauge block was tested under 10x and 20x objective lenses respectively: cross groove block and step block. The comparison results are shown in Table 1. As can be seen from Table 1, for the cross groove block made of transparent glass, the method of the present invention still performs well, especially under the 20x objective lens, and performs better than other equipment. Since transparent glass has a high refractive index and transmittance, at low magnification, light may be more obviously scattered or refracted when passing through the glass, resulting in weak image signals, making the measurement accuracy at 10x significantly lower than that at 20x. For highly reflective surfaces such as metal step blocks, the method of the present invention shows stronger stability than other equipment. At different objective lens magnifications, its measurement accuracy is about 0.1μm. Overall, the accuracy of the present invention is slightly lower than that of Sensofar, but better than that of CHOTEST and Shape from focus, proving that the present invention has achieved industrial-grade measurement accuracy and is suitable for most industrial measurements.

[0083] Table 1

[0084]

[0085] In summary, the present invention only needs one mask to quickly complete high-precision three-dimensional shape measurement in different industrial scenarios, without the need for high-speed rotating parts and complex electronic components, and has a simple structure and low cost. And by actively projecting array light spots, the measured object is given features, making it suitable for measuring textureless or highly reflective surfaces. The circular triangular interpolation algorithm is used to more smoothly and stably expand from a sparse depth map to a dense depth map, and obtain richer three-dimensional shape features of the sample.

[0086] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A three-dimensional microscopic measurement method based on array light spots, characterized in that: The method comprises the following steps: Step S1, the microscope device scans from bottom to top at a set sampling interval to collect dot matrix images on different focal planes of the object to be measured; Step S2, marking the optical center position of each light spot in advance based on the dot matrix image, and using Gaussian surface function fitting to calculate the brightness value at the optical center position; Step S3: using the brightness information of each light spot at the optical center position on different focal planes, the Lorentz function is used to fit the focal position where the brightness value of each light spot is maximum, and the depth value of each light spot is calculated to obtain a sparse depth map; Step S4: Completing the sparse depth map into a dense depth map to complete the three-dimensional shape measurement of the object to be measured.

2. A three-dimensional microscopic measurement method based on array light spots according to claim 1, characterized in that: The sampling interval is set according to the actual height of the object being measured.

3. The three-dimensional microscopic measurement method based on array light spots according to claim 1, characterized in that: The step S2 of marking the optical center position of each light spot includes: Step S201, collecting focal stacks of dot images at different focal planes under the mirror to synthesize a fully focused image; Step S202, extracting the green channel of the all-focus image and calculating the first-order gradient map of the green channel; Step S203, making a first-order gradient map of a Gaussian kernel that obeys a normal distribution as a template; Step S204: locating the optical center position of each light spot using a normalized cross-correlation algorithm.

4. The three-dimensional microscopic measurement method based on array light spots according to claim 3, characterized in that: The fully focused image uses the Tenengrad function as a clarity operator to calculate the clarity value of each pixel in the focus stack, thereby selecting the best focused pixel to generate a fully focused image.

5. The three-dimensional microscopic measurement method based on array light spots according to claim 3, characterized in that: The first-order gradient map of the green channel is obtained by performing a convolution operation on the fully focused image and the Sobel operator.

6. The three-dimensional microscopic measurement method based on array light spots according to claim 1, characterized in that: The Gaussian surface function is specifically: Among them, M(x i ,y i ) is the coordinate point (x i ,y i ) is the gray value of the light spot; (u, v) is the coordinate of the center of the light spot; σ is the standard deviation of the Gaussian function, which is related to the object distance and focal length; B is the degree of reflection of the background; T is the total energy of the light spot projected onto the CMOS, which is related to the brightness of the light source and the exposure time of the camera.

7. The three-dimensional microscopic measurement method based on array light spots according to claim 1, characterized in that: The Lorenz function is specifically: Among them, C(D i ) is the gray value of the light spot on different focal planes; d is the center position of the Lorentz peak, that is, the true depth of the target point; γ is the half width of the Lorentz function, that is, the width of the Lorentz peak; B represents the degree of reflection of the background; T represents the size of the gray value of the light spot.

8. The three-dimensional microscopic measurement method based on array light spots according to claim 1, characterized in that: According to the circular triangular interpolation algorithm, the sparse depth map is completed into a dense depth map, including: Step S401, triangulate all sub-pixel points in the sparse depth map to construct a triangle network; Step S402, calculating the barycentric coordinates of each triangle in the triangle network, and estimating its depth value according to the barycentric coordinate method; Step S403, adding the barycenter coordinates and depth values ​​calculated in step S402 to the original sparse depth map to expand it into a new sparse depth map, and performing triangulation again on all sub-pixel points in the new depth map to construct a triangle network; Step S404, repeating step S402 and step S403 m times to obtain an extended depth map, where m>1.

9. A microscopic device for implementing the three-dimensional topography microscopic measurement method based on array light spots as described in any one of claims 1 to 2, characterized in that: The device comprises a projection system (1), a first semi-transparent and semi-reflective mirror (2), a tube lens (3), a second semi-transparent and semi-reflective mirror (4) and an objective lens (6); the projection system (1) comprises a laser (11), a plano-convex lens (12) and a mask (13); The green light emitted by the laser (11) passes through a plano-convex lens (12) and a mask plate (13), and then sequentially passes through a first semi-transparent and semi-reflective mirror (2), a tube lens (1), a second semi-transparent and semi-reflective mirror (4) and an objective lens (6) before reaching a focal plane.

10. The microscopic device according to claim 9, characterized in that The device also comprises an illumination system (5) and a camera (7); the white light emitted by the illumination system (5) reaches the focal plane after passing through the objective lens (6); the light emitted by the focal plane passes through the objective lens (6), a second semi-transparent and semi-reflective mirror (4) and a tube lens (3), and is then emitted to the camera (7) through a first semi-transparent and semi-reflective mirror (2).

Images