Rapid three-dimensional splicing method for 3D digital optical microscope
Through the fast three-dimensional microscope method, the minimum Z-axis step distance and bipolar step distance strategy are used, combined with the Laplacian operator and least squares fitting, the defocusing problem of optical microscopes when splicing large-area samples is solved, and the defocusing problem of optical microscopes is improved. Splicing speed and efficiency are suitable for samples with overall flat but with height difference.
Patent Information
- Application Number
- CN202410992641.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-26
AI Technical Summary
Existing optical microscopes are defocused due to height difference when splicing large-area samples. The traditional three-dimensional splicing method of panoramic depth is time-consuming and inefficient, and requires additional auxiliary focus modules or laser focus modules.
Fast three-dimensional microscopic stitching method is adopted to reduce the number of frames collected in the field of view by calculating the minimum Z-axis step and bipolar step strategies, combine the Laplacian operator to evaluate the clarity, use the least squares method to fit the height surface, reduce the number of measured frames and Improve splicing speed.
It achieves the clarity of three-dimensional splicing of panoramic depth without adding auxiliary focus modules, and significantly improves splicing speed. It is suitable for large-area samples with overall flatness but slight height difference.
Smart Images

Figure CN119024546A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of optical microscopy, and mainly to a three-dimensional image stitching method, and in particular to a method for reducing the number of stitching frames by predicting the sample height. The method is suitable for microscopically photographing large-area samples that are flat overall but have gentle height differences, such as patterned wafers, embedded metallographic specimens, embedded microscopic sections, etc. Background Art
[0002] Compared with other optical photography equipment, optical microscopes have small field of view and small depth of focus. In order to solve these two problems, digital microscopes use plane stitching to stitch multiple fields of view to obtain high-definition large field of view images, and take images at different depths of field by superimposing the depth of field so that each height of the sample is near the optimal depth of focus.
[0003] The distance between the objective lens and the sample surface must be equal to the depth of field of the objective lens to achieve a clear image. The higher the magnification of the optical microscope, the higher the resolution, the larger the numerical aperture of the objective lens, and the smaller the depth of focus. When there is a height difference in the sample, the image blur caused by defocus is more serious. Therefore, digital microscopes currently usually use the depth of field superposition method to take pictures at different heights, select the highest definition pictures for different areas in the field of view, and synthesize a complete and clear field of view.
[0004] At present, there are three methods for stitching optical microscope images: direct 2D stitching, assisted focus 2D stitching, and full depth 3D stitching. Direct 2D stitching has the problem of out-of-focus blur for samples with height differences. Assisted focus 2D stitching requires the addition of an auxiliary focus module or a laser focus module, and is also not suitable for samples with large height differences. Full depth 3D stitching traverses all depth heights for each field of view, which can solve the problem of sample height difference, but it requires a large number of measurement frames, is time-consuming, and has low efficiency.
[0005] The microscope fast 3D stitching method of the present invention does not require additional auxiliary focus modules or laser focus modules, but can achieve the same clarity as full-depth 3D stitching, and has a higher stitching speed than traditional full-depth 3D stitching. It is particularly suitable for large-scale microscopic photography of metallographic slices, wafers and other flat surfaces with slight height differences. Summary of the invention
[0006] The present invention aims to design a method for quickly collecting three-dimensional microscopic mosaics. In the process of three-dimensional microscopic mosaics, each field of view collects as few frames as possible.
[0007] Before measurement, calculate the minimum Z-axis step size based on the selected objective lens. The most basic parameter of the microscope objective lens is the numerical aperture NA, and the digital microscope that uses CMOS for image acquisition also has a parameter called the calibrated pixel size P, which is the actual object distance corresponding to the CMOS pixel pitch at this magnification. P = chip pixel pitch / optical magnification. The semi-aperture angle θ can be calculated based on the numerical aperture. The relationship between the two is: NA = sin(θ), that is, θ = arcsin(NA). Usually, the minimum Z-axis step size is: h = (0.5·λ / NA+P) / tan(θ), where λ is the wavelength of light. The corresponding diffuse spot radius is R = 0.5·λ / NA+P, and the theoretical diffuse spot radius pixel number is r = 0.5·λ / NA / P+1. The diffuse spot radius here is the spot radius added on the basis of the diffraction spot radius.
[0008] The samples applicable to the method of rapid acquisition of 3D microscopic stitching are divided into two categories, and there are two different field of view selection and height fitting schemes starting from the second field of view. The first category is samples with large-area overall vouchers represented by patterned wafers, and the second category is small-area incompletely flat samples represented by single metallographic or sliced samples.
[0009] For the first type of samples, there are three collection processes: first field of view collection, discrete field of view pre-collection, and continuous field of view collection. For the second type of samples, there are two collection processes: first field of view collection and continuous field of view collection.
[0010] When collecting each field of view, first estimate its possible height range based on the fitting data. If the estimated possible height range is large and exceeds 6 minimum step distances, a bipolar step method is used for the Z axis. First, move from the highest point to the lowest point from the expected Z height with a large step distance. After jumping over the clear surface, the Z axis is turned back and collected from bottom to top with a small step distance.
[0011] When collecting the first field of view, in order to protect the objective lens from collision, the objective lens is first raised and then lowered. After taking pictures at the starting point, the objective lens is raised and continuously taken pictures. At this time, the height difference of each frame takes 3 to 5 minimum steps to calculate the clear picture of each picture. If the clarity of the second frame is lower than that of the first frame, the objective lens turns back and turns down. Regardless of whether the objective lens rises or falls, when the clarity begins to decrease, the two Z heights with the highest and second highest clarity are the range of the sample height plus the optimal depth of field. After determining the height range, the objective lens turns back again and takes pictures with a small step distance.
[0012] Discrete field of view pre-collection, first pre-sample a few discrete fields of view for initial fitting. For example, for a sample of 10×10 fields of view, the 1st to 4th fields of view take the four corners of the total shooting range. For a sample of 600×400 fields of view, you can use 100 field of view distance intervals to go to (6+1)×(4+1)=35 discontinuous fields of view. When collecting each field of view, start with the height of the first field of view and use the same collection method as the first field of view.
[0013] Continuous field of view acquisition, based on the height of all points in the acquired field of view, the least squares method is used for fitting to estimate the possible height range of the field of view. If the height range exceeds 6 minimum steps, the Z axis is used in a bipolar step mode, first moving from the highest point to the lowest point at the expected Z height with a large step, and then returning to the Z axis after jumping over the clear surface, and acquiring from bottom to top with a small step.
[0014] In order to evaluate the clarity of each frame sample, the Laplacian operator is used to calculate the gradient of the image. The sample code is: img = img.astype(np.float32) dimg = cv2.Laplacian(img, -1,ksize = 5) #Laplacian operator dimg = dimg *dimg #Squaring the Laplacian result point by point dimg = cv2.GaussianBlur(img,(k, k), 0) #Gaussian blur limg = dimg [:,:,0]+dimg [:,:,1]+dimg [:,:,2] # sum of the gradients of the three layers of red, green and blue simg = np.sum(limg ) Where k is the size of the Gaussian kernel of the Gaussian blur. k is usually calculated based on the number of pixels of the theoretical diffuse spot radius r = 0.5·λ / NA / P+1, taking the integer part of r multiplied by two and adding one.
[0015] k=int(r*2)+1 When the objective moves along the Z circle, the change of simg reflects the degree of defocus. When simg increases, it means that it is close to the optimal height. When simg starts to decrease, it means that the objective is moving away from the optimal height. If the layout at this time is larger than the minimum step, it is necessary to turn back and collect with the minimum step. If it is already the minimum step, it means that the field of view collection is over.
[0016] After each field of view is acquired, the 3D data of the field of view needs to be generated to fit the overall height and predict the height of other fields of view. After calculating limg for each frame of the field of view, each pixel obtains a discrete height gradient function. The actual sample height is the maximum value of the gradient. Since the Z-axis step size takes a discrete value when taking pictures, the actual height of the pixel is between the height of the highest gradient and the height of the second highest gradient. Assume that the highest gradient and height are L0 / Z0, the gradient and height of the previous frame with the highest gradient and height are L1 / Z1, and the gradient and height of the next frame with the highest gradient and height are L2 / Z2. Since the Z circumference is equidistant. Calculate the height of the point: Z=((L0-L1)*(Z1^2-Z2^2)+(L2-L1)*(Z0^2-Z1^2)) / (2*((L0-L1)*(Z1-Z2)+(L2-L1)*(Z0-Z1))) Each frame obtains a corresponding gradient map. For each same xy coordinate, the height of the frame with the largest pixel limg in each frame is taken as the height of the pixel.
[0017] When continuously collecting fields of view, in order to reduce the number of collected frames as much as possible, it is necessary to predict the height of the field of view before collecting each field of view. The height surface of the sample can be regarded as a binary function z=h(x,y). The height of each point in the measured field of view is known, that is, the points near the surface: z1≈h(x1,y1), z1≈h(x1,y1),...zn≈h(xn,yn). Fit z=h(x,y) based on these points. Based on the fitting surface, calculate the height difference of the height of each point in the measured field of view relative to the fitting surface, and take the variance of the height difference as the height deviation. Substitute the xy coordinates of each point in the new field of view into the fitting surface to obtain the new field of view fitting height map. Add the height difference to the highest point of the fitting height map to obtain the highest point value, and subtract the height difference from the lowest point of the fitting height map to obtain the lowest point value. Then the new field of view can be collected by depth superposition.
[0018] Selection of height surface function. When fitting a function, you need to first give the form of a binary function z=h(x,y). Generally speaking, a quadratic power law function can meet the requirements, that is: h(x,y)=ax^2+by^2+cxy+dx+ey+f.
[0019] In order to increase the fitting speed, the height map of all fields of view is compressed. It is recommended to use the cv2.INTER_AREA compression algorithm to average the adjacent 2*2 pixels or 4*4 pixels. For the underlying algorithm programming, it can be further simplified from taking the average to taking the sum. After each field of view is measured, only the fitting data is added to the points in the field of view, and there is no need to re-fit.
[0020] The example python code for fitting and calculating h(x,y) is as follows: for ……: #loop once after each field of view is shot # imgz is the compressed height map, which is a width*height matrix # width and height are the width and height of the field of view in pixels, x0, y0 are the coordinates of the field of view in the entire shooting area # Generate x and y coordinates for each pixel x, y = np.meshgrid(np.arange(width), np.arange(height)) # Flatten x, y, z x = x.ravel()+x0 y = y.ravel()+y0 z = z.ravel() # Construct the design matrix X X = np.column_stack((x ** 2, y ** 2, x * y, x, y, np.ones_like(x))) # Gradually update the design matrix and target vector if X_total is None and z_total is None: X_total = X z_total = z else: X_total = np.vstack((X_total, X)) z_total = np.hstack((z_total, z)) # Fit and update coefficients coeffs = fit_and_update(X_total, z_total, coeffs if idx>0 elseNone) # Extract the final coefficients a, b, c, d, e, f = coeffs print(f"h(x, y) = {a}x^2 + {b}y^2 + {c}xy + {d}x + {e}y + {f}") After testing, when the form of h(x,y) is obtained, the standard deviation S of the deviation of the z values of all measured points relative to h(x,y) is calculated, and the coordinates of each point in the area to be measured are substituted into h(x,y) one by one to obtain the height prediction map of the area to be measured. The highest point of the height prediction map plus S is the highest value of the expected height range, and the lowest point of the height prediction map -S is the lowest value of the expected height range.
[0021] Since the calculation of h(x,y) uses compressed xyz data, the calculation of the standard deviation and the height of each point in the new area also uses the compressed xy coordinates. After testing, a 5-megapixel camera uses the above algorithm after 2*2 compression, occupies a single CPU core, and each new field of view takes less than 0.05 seconds. This time is the predicted time after each field of view ends. Considering that the microscope xy movement takes time, no additional time is consumed due to calculation.
[0022] Instruction manual drawings: Figure 1 : Relationship between semi-aperture angle θ and physical aperture and working experience. Figure 2 : The relationship between the limit value of the diffuse spot radius R and the defocus distance h. Figure 3 :Schematic diagram of Z-axis return.
Claims
1. A method for quickly stitching large-scale images with full depth of field using a 3D digital optical microscope. The method uses a digital microscope with precise movement and positioning functions in the three directions of length, width and height of XYZ to quickly obtain high-definition images and three-dimensional morphology by using depth of field superposition for different fields of view in a large observation area.
2. According to claim 1, the method of depth of field superposition, Features: Starting from the second field of view, the sample surface height is predicted first, and different Z-axis heights are used for different fields of view to reduce the number of superimposed frames for each field of view and increase the stitching speed.
3. According to claims 1 and 2, the depth of field superposition method, Features: For samples with unstable height differences, different numbers of superpositions are adopted according to the height difference of the field of view. The field of view with low height difference has a small number of superpositions and a fast speed, while the field of view with large height difference has a large number of superpositions to ensure that accurate data is collected at different heights.
4. According to claims 1 and 2, the prediction of the sample surface height, Features: For each new horizon, the height range of the new horizon is estimated by fitting based on the height information of all old horizons.
5. According to claims 1 and 2, the first field of view is collected, Features: Use a larger Z-step first, and then use a smaller layout to collect after the image is in focus.
6. According to claims 1, 2, and 3, the second field of view begins, Features: When the predicted sample height deviation is small, the Z-axis directly uses a small step size; when the predicted sample height deviation is large, a larger Z-axis step size is first used, and then a smaller layout is used for acquisition after the focus is clear.
7. According to claims 1, 2, and 4, the choice of the field of view, Features For: When stitching multiple fields of view for high-flatness samples, first select several discontinuous fields of view at a longer distance to perform Z-axis superposition to obtain height data, thereby increasing the fitting speed and accuracy of the height.
