Kalman filter based multi-view compensation three-dimensional microscopic measurement method and system

Abstract

The application relates to a Kalman filter-based multi-view compensation three-dimensional microscopic measurement method and system, which comprises the following steps: measuring the same sample at different measurement angles and adopting a three-dimensional Gaussian distribution form to describe the measurement point cloud; sequentially performing fusion iteration processing on the measurement point cloud under each view, in each round of iteration, the overlapping area of the measurement point cloud under the current view and the iteration result of the last round is obtained, the resolution of the two point clouds in the overlapping area is aligned in the form of superimposed displacement field, the resampled iteration result is obtained, the Kalman gain is calculated with the covariance matrix under the current view, the direction with lower uncertainty is given higher confidence for coordinate fusion, and the fusion point cloud is obtained as the output; and the final three-dimensional microscopic measurement result is obtained according to the result obtained through iteration optimization processing. Compared with the prior art, the application has the advantages of expanding the spatial measurement field and significantly improving the original axial measurement precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional microscopic imaging technology, and in particular to a multi-view compensation three-dimensional microscopic measurement method and system based on Kalman filtering. BACKGROUND

[0002] The extended depth of field (EDOF) technology realizes three-dimensional high-resolution imaging of complex microstructures through multi-focus imaging and topography reconstruction algorithm, and is widely used in industrial detection, biomedicine and other fields. However, the inherent characteristics of the microscope objective lens result in a trade-off relationship between the depth of field (axial resolution) and the lateral resolution. Specifically, the depth of field of the microscope objective lens is usually several to tens of times the lateral resolution, so that the axial measurement accuracy of the EDOF system is significantly lower than the lateral accuracy (i.e. anisotropic error). This defect leads to distortion of the depth information in the three-dimensional reconstruction result, and especially when analyzing samples with complex undulations or edge features, the problem of insufficient axial positioning sensitivity is particularly prominent, which seriously restricts the application range of the EDOF technology.

[0003] To improve the axial accuracy, there are mainly three types of existing improvement schemes, but there are still significant defects:

[0004] First, increase the axial sampling density, and improve the axial sampling density by introducing deformable mirrors, liquid crystal lenses or liquid lenses and other variable focus devices to improve the accuracy. However, the inherent noise in the image acquisition process limits the further improvement of the axial accuracy, and the gap between the axial and lateral accuracy is still difficult to bridge.

[0005] Second, introduce prior knowledge optimization algorithm, and enhance the accuracy of depth reconstruction by optimizing the focusing evaluation algorithm or topography reconstruction algorithm. However, this method has limited generalization ability when dealing with samples with high and rapid changes in height, and the axial accuracy is limited.

[0006] Third, the prediction model based on deep learning directly predicts the three-dimensional topography from a single or multiple images using a neural network. However, model training relies on a specific data set, and when the measured sample exceeds the training distribution, the prediction accuracy decreases significantly, making it difficult to meet the actual application requirements.

[0007] The above methods have alleviated the problem of insufficient axial accuracy to some extent, but have not fundamentally solved the axial resolution sensitivity defect caused by the excessive depth of field of the microscope objective lens. In addition, the anisotropic error characteristics of the traditional single-view EDOF system have not been effectively utilized, and the collaborative analysis and fusion strategy of multi-view data is still a technical blank. Therefore, an innovative method is needed to break through the axial accuracy limit and realize high-precision three-dimensional microscopic measurement. SUMMARY

[0008] The present application aims to overcome the defects of the prior art and provide a multi-view compensation three-dimensional microscopic measurement method and system based on Kalman filtering, which breaks through the axial precision limit and realizes high-precision three-dimensional microscopic measurement.

[0009] The object of the present application can be achieved by the following technical solutions:

[0010] A multi-view compensation three-dimensional microscopic measurement method based on Kalman filtering comprises the following steps:

[0011] An extended depth of field three-dimensional microscopic measurement system is used to measure the same sample at different measurement angles, and a three-dimensional Gaussian distribution is used to describe the measurement point cloud, and the covariance matrix corresponding to the Gaussian distribution at each angle is recorded.

[0012] The measurement point cloud at each angle is sequentially subjected to multi-view fusion iterative optimization processing, in each iteration process, the overlapping area of the measurement point cloud at the current angle and the iteration result of the last round is obtained, the resolution of the two point clouds in the overlapping area is aligned in the form of superimposed displacement field, the resampled iteration result is obtained, the Kalman gain is calculated with the covariance matrix at the current angle, the direction with lower uncertainty is given higher confidence for coordinate fusion, and the fused point cloud is obtained as the iteration output of the current round.

[0013] According to the result obtained by the iterative optimization processing, the final three-dimensional microscopic measurement result of the visible area of the sample three-dimensional topography is obtained.

[0014] Further, the measurement point cloud obtained by measuring the same sample at different measurement angles is in the form of a measurement equation:

[0015] Q k =H k ·V k (S k )+v k

[0016] In the formula, Q k is the measurement point cloud at the kth angle, S k is the system state in Kalman filtering, indicating the multi-view point cloud fusion result of the sample topography at the kth iteration, H k is a transformation matrix for mapping the world coordinate system to the k-angle extended depth of field three-dimensional microscopic measurement system coordinate system, v k is measurement noise, which is subject to 3D Gaussian distribution of the k-angle extended depth of field three-dimensional microscopic measurement system.

[0017] Further, the expression of the state equation corresponding to the processing process in each iteration process is:

[0018]

[0019] In the formula, This represents a subset of the input point cloud within the overlapping region of the first k-1 views and the kth view. For point cloud S k Corresponding subset, For point cloud S k-1 Corresponding subset, For point cloud Q k Corresponding subset, S k-1 For the results of k-1 iterations, w k For process noise, obeying the rules of S k-1 The corresponding 3D Gaussian distribution of measurement accuracy, where F(·) represents the point cloud resampling process. To transform the point cloud S resulting from k-1 iterations k-1 The measured point cloud Q from the k-th viewpoint k resampling results Modeled as a point cloud Q k and the non-rigid displacement field to be solved The superposition of elements; the viewpoint number is consistent with the iteration number.

[0020] Furthermore, the non-rigid displacement field The solution process specifically includes:

[0021] The measurement point cloud Q from the k-th viewpoint k Consider them as a cluster of equally weighted groups with uniform variance σ. 2 The centroid of the Gaussian distribution, and the point cloud S resulting from k-1 iterations. k-1 These are the data points extracted from the Gaussian distribution; the measurement point cloud Q from the k-th viewpoint... k The point cloud S resulting from k-1 iterations k-1 Non-rigid displacement field Let them be matrices Q = (q1, ..., q2) N ) T S = (s1, ..., s M ) T and Solving for the non-rigid displacement field by maximizing the corresponding likelihood function. The parameters.

[0022] Furthermore, the expression for the likelihood function is:

[0023]

[0024] In the formula, For non-rigid displacement field And equal weighted and with uniform variance σ 2 The likelihood function of the Gaussian distribution, q nand s m respectively represent the measured point cloud Q k and the iteration result point cloud S k-1 of the k-1th round m is a row vector of coordinate points, p(s m ) represents the probability of obtaining the data point s n from the Gaussian distribution cluster, I is a unit matrix, τ n is the displacement vector of the corresponding point q represents the Gaussian probability density function.

[0025] Further, the solving process of the non-rigid displacement field further comprises constraining the gradient of the non-rigid displacement field , specifically adding a weighted L2 regularization term, and the corresponding calculation expression is:

[0026]

[0027] In the formula, E reg is the weighted L2 regularization term, λ is a regularization coefficient, g(q i , q j ) is a Gaussian kernel function, G is a corresponding kernel matrix, and satisfies g ij = g(q i , q j ), and tr(·) represents the trace of a matrix.

[0028] Further, the non-rigid displacement field is iteratively solved by an EM algorithm.

[0029] Further, the measurement equation of the measured point cloud Q k under the kth view and the state equation corresponding to the processing process in each iteration round are combined to obtain a calculation expression of the multi-view point cloud fusion result S k of the sample topography under the kth iteration round:

[0030]

[0031] In the formula, P k-1 and P k are the covariance matrices of the 3D Gaussian distribution corresponding to the measurement precisions of the multi-view measurement results S k-1 and S k of the k-1th and kth rounds respectively, and satisfy P1=Cov1, Cov k is the covariance matrix corresponding to the point cloud measurement result under the V k view, is the resampling result, is the Kalman gain of the kth round.

[0032] Further, the process described in the form of three-dimensional Gaussian distribution is specifically:

[0033] The direction perpendicular to the sample surface is defined as the z-axis of the world coordinate system, the long axis of the three-dimensional Gaussian distribution ellipsoid is parallel to the z-axis, and the long axis and the short axis of the three-dimensional Gaussian distribution ellipsoid represent the measurement uncertainty of the extended depth of field three-dimensional microscopic measurement system in the axial and lateral directions, i.e. the directions with the maximum and minimum measurement uncertainty of the extended depth of field three-dimensional microscopic measurement system.

[0034] The application also provides a multi-view compensation three-dimensional microscopic measurement device based on Kalman filtering, which is structure one or structure two.

[0035] The structure one includes a rotating stage, a first controller and a single extended depth of field module, the rotating stage is used to carry and rotate the sample, the single extended depth of field module is used to obtain the microscopic image of the sample, and the first controller is used to change the observation angle of the sample through the rotating stage, cooperate with the single extended depth of field module to measure the same sample at different measurement angles, and execute the multi-view compensation three-dimensional microscopic measurement method based on Kalman filtering as described above to obtain the three-dimensional microscopic measurement result of the sample.

[0036] The structure two includes a fixed stage, a second controller and a plurality of extended depth of field modules, the fixed stage is used to place the sample, the plurality of extended depth of field modules are used to observe the sample on the fixed stage from different angles, and the second controller is used to execute the multi-view compensation three-dimensional microscopic measurement method based on Kalman filtering as described above according to the measurement results of the same sample by the plurality of extended depth of field modules at different measurement angles to obtain the three-dimensional microscopic measurement result of the sample.

[0037] Compared with the prior art, the application has the following advantages:

[0038] (1) The application provides a multi-view compensation three-dimensional microscopic measurement method based on Kalman filtering, which utilizes the anisotropy of monocular EDOF measurement precision distribution, introduces a multi-view strategy, acquires missing data under single-view measurement, and realizes compensation from high-precision direction to low-precision direction under multi-view measurement through Kalman filtering technology, thereby significantly improving the original axial measurement precision while expanding the spatial measurement field of view.

[0039] (2) In the point cloud fusion process of the application by Kalman filtering technology, an iterative optimization strategy is adopted, in each iteration process, based on the measurement results of the current view and the overlapping area of the last iteration result, the resolution of the two point clouds in the overlapping area is aligned by superimposing the displacement field, the resampled iteration result is obtained, and the Kalman gain is calculated with the measurement results of the current view, the direction with lower uncertainty is given higher confidence for coordinate fusion, and through this dynamic weighting method, the accuracy complement of different data sources is realized.

[0040] (3) The effectiveness and accuracy of the KVCM system (multi-view compensation three-dimensional microscopic measurement system based on Kalman filtering) of the application are verified through simulation and standard sample experiments, and further through the measurement of bismuth crystals with complex surface topography and metal characteristics, the correction of local overexposure and the multi-view complementary effect of complex surface samples are illustrated. Therefore, the method of the application provides a new way to effectively narrow the significant gap between axial and lateral measurement accuracy in extended depth of field microscopes, and has broad application prospects in the field of high-precision, high-resolution three-dimensional microscopic measurement. However, considering that the method relies on collecting multiple groups of image data, if the multi-view simultaneous measurement mode is not used, the overall efficiency will inevitably be reduced, and its potential in fast response scenarios needs to be further explored. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of a multi-view compensation three-dimensional microscopic measurement method based on Kalman filtering provided in an embodiment of the application;

[0042] Figure 2 A point cloud fusion process diagram based on Kalman filtering provided in an embodiment of the application, wherein (a) is a traditional single view EDOF model, (b) is a multi-view EDOF measurement process and the accuracy distribution of different views, (c) is the input of iteration, including the current measurement data and the prior iteration result, (d) is the point cloud resampling based on displacement field in the prediction step, (e) is the minimum variance fusion in the update step, and (f) is the output result of iteration;

[0043] Figure 3 A schematic diagram of the intuitive effect of the simulation experiment, wherein (a) is a KVCM simulation, including single view measurement and multi-view fusion, and (b) is the root mean square error (RMSE) and its reduction result of the simulation result;

[0044] Figure 4 An EDOF measurement result comparison diagram, wherein (a) is the texture result of single view EDOF, (b) is the texture result of KVCM, (c) is the depth result of KVCM, (d) is the root mean square error of KVCM in different areas, and (e) is the depth value of KVCM in different areas;

[0045] Figure 5 wherein (a) the root mean square error of the sphere, (b) the radius of the estimated sphere, (c) the depth result of the KVCM, (d) the texture result of the single view EDOF, (e) the texture result of the KVCM, (f) the depth result of the KVCM;

[0046] Figure 6 wherein (a) the schematic diagram of the principle of the KVCM: obtaining multi-view measurement data by rotating the sample, (b) the calibration principle of the KVCM;

[0047] Figure 7 Fig. 1 is a structural schematic diagram of a multi-view compensation three-dimensional microscopic measurement system based on Kalman filtering provided in an embodiment of the present application;

[0048] Figure 8 Fig. 2 is a structural schematic diagram of a multi-view compensation three-dimensional microscopic measurement system based on Kalman filtering provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts are within the scope of protection of the present application.

[0051] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0052] Embodiment 1

[0053] As shown in the drawings, the present embodiment provides a multi-view compensation three-dimensional microscopic measurement method based on Kalman filtering, comprising the following steps: Figure 1 S1: using an extended depth of field three-dimensional microscopic measurement system to measure the same sample at different measurement angles, and using a three-dimensional Gaussian distribution to describe the measurement point cloud, and recording the covariance matrix corresponding to the Gaussian distribution at each angle;

[0054] S2: using the Kalman filter to update the covariance matrix of the Gaussian distribution at each angle, and obtaining the updated covariance matrix of the Gaussian distribution at each angle;

[0055] Specifically, as shown in a of Figure 2 , the traditional EDOF utilizes the objective lens to conduct axial scanning along the direction perpendicular to the surface of the sample, and its axial accuracy is significantly lower than the lateral accuracy due to the limitation of the objective lens depth of field. To quantify the measurement accuracy relationship in each direction, the present embodiment adopts a three-dimensional Gaussian distribution form for description, and defines the direction perpendicular to the surface of the measured sample as the z-axis of the world coordinate system. At this time, the long axis of the corresponding three-dimensional Gaussian ellipsoid is parallel to the z-axis, and the long axis and the short axis respectively represent the measurement uncertainty of the system in the axial and lateral directions, i.e. the directions with the maximum and minimum system uncertainty.

[0056] As shown in b of Figure 2 , the present embodiment places the EDOF system at different measurement angles V1, V2, … V K , to measure the same sample. In the world coordinate system, the pose of the three-dimensional Gaussian ellipsoid representing the measurement uncertainty in different directions changes with the angle of view, and the axial direction of the system at each angle of view is no longer aligned with the z-axis. The covariance matrix Cov1-Cov K corresponding to the Gaussian distribution at each angle of view is recorded.

[0057] Considering that the multi-angle measurement data is obtained by rotating the imaging system, higher requirements are put forward for the mechanical positioning accuracy, and the system is easily affected by mechanical vibration, increasing the measurement error and the calibration difficulty. Therefore, the present embodiment adopts a high-precision rotating platform to drive the sample to reverse the above process, which is mathematically equivalent. The system schematic diagram and the space calibration method can be seen in the calibration process part of the following text.

[0058] S2: sequentially perform multi-angle fusion iterative optimization processing on the measurement point clouds at each angle of view. In each iteration process, the overlapping area of the measurement point cloud at the current angle of view and the iteration result of the last round is obtained, the resolution of the two point clouds in the overlapping area is aligned in the form of superimposed displacement field, the resampled iteration result is obtained, the Kalman gain is calculated with the covariance matrix at the current angle of view, the direction with lower uncertainty is given higher confidence for coordinate fusion, and the fused point cloud is obtained as the iteration output of the current round;

[0059] Specifically, in the point cloud fusion process based on Kalman filtering shown in c-f of Figure 2 , the present embodiment takes the point cloud measurement result Q1 at the V1 angle of view as the initial multi-angle fusion measurement result S1, and takes the measurement results Q2-Q K at the remaining angles of view and the covariance matrices Cov2-Cov K representing their accuracy distribution as subsequent inputs to iteratively optimize the multi-angle fusion measurement result.

[0060] For the kth iteration, first obtain the V kmeasurement result Q of the view angle k and the result S of the last iteration k-1 of the overlapping area (as shown by c in Figure 2 ); then the resolution of the two point clouds in the overlapping area is aligned by superimposing the displacement field, obtaining the resampled (as shown by d in Figure 2 ); then the Kalman gain is calculated according to the covariance matrix of the two, and the coordinate fusion is performed by giving higher confidence to the direction with lower uncertainty (as shown by e in Figure 2 ), and through this dynamic weighting, the accuracy of different data sources is complemented; the obtained fusion point cloud S k is taken as the output of the kth iteration (as shown by f in Figure 2 ) and enters the iteration of the next round. After K iterations, while expanding the measurement field of view, the axial measurement error of the fusion result S k is significantly reduced through multi-view accuracy complementation based on the MSE principle of Kalman filtering.

[0061] To model the above process, according to the principle of Kalman filtering, first, the measurement equation is established to describe the single measurement process:

[0062] Q k = H k · V k (S k ) + v k (1)

[0063] wherein Q k represents the measurement point cloud under the kth view angle, S k is the system state in Kalman filtering, representing the multi-view point cloud fusion result of the sample topography under the kth iteration, V k (·) represents the data points in the visible range of the input point cloud under the kth view angle, H k is the transformation matrix for mapping the world coordinate system to the kth view angle measurement system coordinate system, v k is the measurement noise, obeying the 3D Gaussian distribution of the kth view angle measurement system.

[0064] Then, the state equation is established to describe the evolution process of the multi-view measurement result with iteration:

[0065]

[0066] wherein represents the union set of the intersection of the first k-1 view angles and the visible range of the kth view angle, and the data points of the input point cloud in the overlapping area are obtained, and the sparse voxel octree method is used to realize the intersection in the embodiment, w k is the process noise, obeying the 3D Gaussian distribution of Sk-1 The corresponding 3D Gaussian distribution of the measurement accuracy. Considering that the sampling process of multi-view measurement results often has differences in the overlapping area, the scheme designs a point cloud resampling process F(·) for the state equation. The process realizes the consistency of the sampling points by modeling the resampling results of the point cloud S k-1 as the superposition of the point cloud Q and the non-rigid displacement field to be solved k .

[0067] In order to solve the displacement field , a probability estimation method is adopted, regarding the source point cloud Q k as the centroid of a cluster of Gaussian distributions with equal weights and uniform variance σ 2 , and the target point cloud S k-1 as the data points extracted from these Gaussian distributions. The two point clouds Q k , S k-1 and the displacement field are respectively denoted as matrices Q=(q1,…,q N ) T , S=(s1,…,s M ) T and By maximizing the following likelihood function, the most reasonable displacement field parameters are solved:

[0068]

[0069] Where q n and s m are row vectors representing the coordinates of the two point clouds, τ n is the displacement vector corresponding to the point q n , p(s m ) represents the probability of obtaining the data point s m from the Gaussian distribution cluster, and I is the identity matrix. At the same time, the gradient of the displacement field is constrained to avoid overfitting, obtaining the following weighted L2 regularization term:

[0070]

[0071] Where λ is the regularization coefficient, g(q i , q j ) is the Gaussian kernel function to measure the similarity between points, and G is the corresponding kernel matrix, satisfying g ij = g(q i , q h ). Subsequently, since the parameters and σ 2 are coupled in the likelihood function, the displacement field is iteratively solved by the EM algorithm to complete the resampling of the point cloud.​

[0072] By combining equations (1) and (2), the multi-view measurement result S of the k-th iteration is obtained through the following formula. k :

[0073]

[0074] Among them, P k-1 and P k The multi-view measurement results S for rounds k-1 and k are respectively. k-1 and S k The covariance matrix corresponding to the 3D Gaussian distribution of the measurement accuracy, and satisfying P1=Cov1.

[0075] S3: Based on the results obtained from the iterative optimization process, the final three-dimensional microscopic measurement results after completing the visible area of ​​the three-dimensional morphology of the sample are obtained.

[0076] That is, based on the final multi-view measurement results S obtained after K rounds of iterations K This completed the visual area of ​​the three-dimensional morphology of the sample and significantly improved the axial accuracy of the sample through multi-view precision complementarity.

[0077] Experimental procedure:

[0078] To preliminarily verify the feasibility of the proposed method, we designed the following simulation experiment. First, we constructed a set of multi-layered 3D point cloud models of revolution as ground truth. Then, we set six observation viewpoints and obtained the observation results for each viewpoint using a ray-tracing-based visibility analysis method. Gaussian noise was then added based on the error covariance matrix of each viewpoint. Since the correspondence between observation points was known in the simulation data, we used the proposed method to fuse several observation results for each point.

[0079] As Figure 3 Figure 'a' shows the intuitive effect of the simulation experiment. The blue point cloud on the left is the single-view measurement result, and the red point cloud on the right is the multi-view fusion result. It can be observed that the method of this invention suppresses the noise of the measured point cloud and significantly improves the measurement integrity. Figure 3 In step b, the RMSE value of the fused point cloud was calculated during the iteration process. As shown by the red curve, the RMSE value of the fused point cloud decreased significantly with iteration. Simultaneously, we adjusted the lateral-axial precision ratio of the input covariance matrix during the iteration process and calculated the total percentage decrease in the RMSE of the fused point cloud. As shown by the blue curve, the decrease in RMSE value was not significantly affected by the change in the precision ratio within the range of 2 to 50 times. This indicates that, within a certain range, the accuracy improvement of the method of this invention does not depend on the accurate axial-lateral uncertainty ratio of the input covariance matrix, demonstrating strong robustness.

[0080] To further verify the effectiveness and accuracy of the proposed method, standard gauge blocks and standard spheres were selected as standard samples. The nominal lengths of the standard gauge blocks were 1.000 mm, 2.000 mm, 3.000 mm, and 4.000 mm, with a length tolerance of ±0.5 μm and a surface roughness of 0.016 μm. The nominal radius of the standard sphere samples was 6 mm, with a machining accuracy of ±2 μm. Simultaneously, an experimental system was constructed using an integrated EDOF system (JUOPT) and a high-precision turntable (UPTech, ARST-100H) with a positioning accuracy of ±2 arcsec.

[0081] First, place four standard gauge blocks in the center of the field of view and set six observation angles for multi-view measurement. Figure 4 In the figures, 'a' and 'b' represent the single-view EDOF measurement result and the KVCM measurement result, respectively. It can be seen that in single-view measurement, due to occlusion between gauge blocks, topographic data for some of their sides and most of the stage surface are missing; while KVCM supplements these areas, widening the spatial measurement field of view. For example... Figure 4 As shown in c, a rectangular region within the multi-view overlapping area is selected from each of the four gauge blocks, denoted as Area1 to Area4. The root mean square error (RMSE) compared to the true depth value is calculated during the iteration process, along with the average depth of the single-view EDOF and KVCM measurements for the four regions. Figure 4 As shown in d, with the increase of the number of iterations, the RMSE of Area 1 to Area 4 decreased significantly, from 18.38 μm to 24.76 μm in single-view measurement, to 4.523 μm to 6.104 μm, with an average decrease of about 15.58 μm. This means that compared with single-view EDOF, KVCM improves the measurement accuracy by about 75%. Meanwhile, the average depth values ​​obtained from six single-view and multi-view measurements in the four regions are plotted, and the true depth values ​​are marked with dashed lines, as shown. Figure 4 As shown in e, it can be observed that the average depth result of the method of the present invention is closer to the true value than the single-view measurement.

[0082] Subsequently, multi-view measurements were performed on the standard spherical sample from six observation angles. The RMSE of the distance from the measurement result to the fitted sphere center was calculated during the iteration process, along with the fitted sphere radius from the single-view EDOF and KVCM measurement results. For example... Figure 5 As shown in 'a', the RMSE value of the spherical measurement results decreases significantly with increasing iteration number. Meanwhile, in... Figure 5 In section b, the fitted sphere radii obtained from six single-view and multi-view measurement results are plotted, and the true value of the standard sphere radius is indicated by dashed lines. It can be observed that the multi-view measurement results obtained by the method of this invention are closer to the true value than the single-view results.

[0083] Finally, bismuth crystal was selected as a non-standard sample, and its rich detail and metallic reflection properties on the crystal surface were used to verify the measurement effect of KVCM under complex three-dimensional morphology and lighting conditions. Figure 5 The data structures in the figures show the texture imaging results from a single viewpoint, the depth measurement results from the method of this invention, and the depth measurement results from the method of this invention. It can be observed that there are obvious overexposed areas on the sample surface under a single viewpoint, and the spatial field of view is limited by the single measurement viewpoint. KVCM, on the other hand, achieves the expansion of the measured spatial field of view (blue box) and the correction of overexposed areas (red box) by fusing multi-view texture and topography data.

[0084] The above experiments demonstrate that, compared to monocular EDOF, the method of this invention significantly improves measurement accuracy through the complementary fusion of multi-view data. Furthermore, it exhibits superior performance in overexposure correction and spatial field of view expansion when facing complex 3D topography and lighting conditions.

[0085] Calibration process section:

[0086] like Figure 6 As shown in a, the KVCM system consists of a high-precision turntable and an integrated EDOF system. In order to ensure sufficient field-of-view overlap between the various EDOF viewpoints and to make the field of view more balanced and comprehensive, the tilt angle of the camera optical axis is designed to be 45°.

[0087] The calibration principle of KVCM is as follows: Figure 6 As shown in b, a checkerboard pattern is used as a calibration board to collect data at different rotation angles θ, in order to obtain the transformation relationship from the world coordinate system to the camera coordinate system at any angle θ (denoted as the transformation matrix H). The establishment of the transformation matrix H consists of two steps. First, θ = 0 is set to solve for the transformation relationship from the camera coordinate system to the static world coordinate system. The origin O of the world coordinate system is set... w =(x w ,y w ,z w Move the camera to the origin O of the camera coordinate system through translation transformation. c =(x c ,y c ,z c The coordinates are then overlapped, and then rotated to align with the camera coordinate system using Euler angles. This process is denoted by matrix h. wc :

[0088]

[0089] Where α, β, γ represent Pitch, Yaw, and Roll in Euler angles, respectively. wc Let t be the synthesized rotation matrix. wc It is a translation vector.

[0090] The second step is to consider the rotation process of the world coordinate system around the axis of rotation. Since the origin of the world coordinate system is usually represented by the starting corner point of a checkerboard grid, and is related to the rotation center O′ of the axis of rotation. w If they do not coincide, the inverse rotation process of the world coordinate system from the rotation angle θ back to θ=0 can be described as follows:

[0091]

[0092] Among them, T ww′ Indicates O w Move to O w′ Translation transformation, R ⊥ This represents the rotation about the z-axis. Therefore, the transformation matrix H can be derived from H... wc and H ww′ express:

[0093]

[0094] Written in equation form:

[0095]

[0096] After obtaining the above calibration equation, solve it. Let R... ⊥ (-θ)=A,[x w ,y w ,z w ,1] T =B, [x c ,y c ,z c [1] = C, and assume:

[0097]

[0098] Formula (9) can then be expressed in the form XAYB = C. Its solution process can be divided into two parts. First, under each viewpoint, with angle θ fixed and XAY = H, we solve for the transformation matrix H through the corner points of the chessboard. θ Considering H θ It is a rigid transformation consisting of translation and rotation. The H value for each viewpoint is first solved using the Kabsch algorithm. θ First, perform singular value decomposition:

[0099]

[0100] Then calculate the rotation component R respectively. H Translation component t H :

[0101]

[0102] where B' is the non-homogeneous part of B i and C' i is the non-homogeneous part of B i and C i . Combining R H and t H , we get H θ at each view angle. In the second step, we determine the extrinsic matrix according to the obtained corresponding θ and H θ . XAY = H is a Bilinear Equation form, to solve X and Y in it, we use the external constraint method based on Kronecker product, and realize the iterative solution of X and Y by Alternating Least Squares (ALS), when it satisfies

[0103] ||XAY-H|| F <ε (14)

[0104] it is considered to have been iterated to convergence, and the matrix X and Y are obtained. At this time, the extrinsic matrix of the system at any θ angle is obtained, and the space calibration is completed.

[0105] Embodiment 2

[0106] As shown in Figure 7 , the embodiment provides a multi-view compensation three-dimensional microscopic measurement device based on Kalman filtering, which comprises a rotating stage, a first controller and a single extended depth of field module. The rotating stage is used to carry and rotate the sample. The single extended depth of field module is used to obtain a microscopic image of the sample. The first controller is used to change the observation angle of the sample by the rotating stage, cooperate with the single extended depth of field module to measure the same sample at different measurement angles, and execute a multi-view compensation three-dimensional microscopic measurement method based on Kalman filtering as in embodiment 1 to obtain a three-dimensional microscopic measurement result of the sample.

[0107] Embodiment 3

[0108] As shown in Figure 8 , the embodiment provides a multi-view compensation three-dimensional microscopic measurement device based on Kalman filtering, which comprises a fixed stage, a second controller and a plurality of extended depth of field modules. The fixed stage is used to place the sample. The plurality of extended depth of field modules are used to observe the sample on the fixed stage from different angles. The second controller is used to execute a multi-view compensation three-dimensional microscopic measurement method based on Kalman filtering as in embodiment 1 according to the measurement results of the same sample at different measurement angles by the plurality of extended depth of field modules to obtain a three-dimensional microscopic measurement result of the sample.

[0109] It should be noted that other technical means, component combinations or structural forms can be used to realize the structure for capturing different three-dimensional perspective data of the object, and the structure serves the purpose of multi-perspective extended depth-of-field three-dimensional microscopic measurement.

[0110] The preferred embodiments of the present application have been described in detail. It should be understood that those skilled in the art can make many modifications and variations without departing from the concept of the present application. Therefore, any technical solutions obtained by logical analysis, reasoning or limited experiments based on the concept of the present application and the prior art should be within the protection scope defined by the claims.

Claims

1. A multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering, characterized in that, Includes the following steps: An extended depth-of-field three-dimensional microscopic measurement system was used to measure the same sample from different perspectives. The measurement point cloud was described in the form of a three-dimensional Gaussian distribution, and the covariance matrix corresponding to the Gaussian distribution under each perspective was recorded. The measurement point cloud under each viewpoint is subjected to multi-view fusion iterative optimization processing in turn. In each iteration, the overlapping area between the measurement point cloud under the current viewpoint and the iteration result of the previous round is obtained. The resolution of the two point clouds in the overlapping area is aligned by superimposing the displacement field to obtain the resampled iteration result. The Kalman gain is calculated with the covariance matrix under the current viewpoint. The direction with lower uncertainty is given a higher confidence level for coordinate fusion, and the fused point cloud is used as the iteration output of the current round. Based on the results obtained from iterative optimization, the final three-dimensional microscopic measurement results after completing the visible area of ​​the sample's three-dimensional morphology are obtained. The expression for the state equation corresponding to the processing in each iteration is: In the formula, For the first k Point cloud measurements from various perspectives Let be the system state in Kalman filtering, representing the th k Multi-view point cloud fusion results of sample morphology under round-iteration iteration This indicates that the input point cloud comes first. k -1 perspective and the first k A subset within the overlapping regions of each viewpoint For point clouds Corresponding subset, For point clouds Corresponding subset, For point clouds Corresponding subset, for k -1 round of iteration results, For process noise, obey the rules of process noise. The corresponding 3D Gaussian distribution of measurement accuracy For point cloud resampling process, To be k Point cloud of results from -1 round of iterations With the k Measurement point cloud from one perspective resampling results Modeled as a point cloud and the non-rigid displacement field to be solved The superposition of views; the viewpoint number is consistent with the iteration number; The non-rigid displacement field The solution process specifically includes: The first k Measurement point cloud from one perspective Consider them as a cluster of equally weighted groups with uniform variance. The centroid of the Gaussian distribution, k Point cloud of results from -1 round of iterations These are the data points extracted from the Gaussian distribution; The first k Measurement point cloud from one perspective , k Point cloud of results from -1 round of iterations Non-rigid displacement field They are respectively denoted as matrices , and The non-rigid displacement field is solved by maximizing the corresponding likelihood function. The parameters.

2. The multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering according to claim 1, characterized in that, The measurement point cloud obtained from measuring the same sample from different measurement perspectives, expressed in the form of a measurement equation, is as follows: In the formula, To map the world coordinate system to k The transformation matrix of the coordinate system for the extended depth of field in a three-dimensional microscopic measurement system. To measure noise, obey k 3D Gaussian distribution of a three-dimensional microscopic measurement system with extended depth of field.

3. The multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering according to claim 1, characterized in that, The expression for the likelihood function is: In the formula, For non-rigid displacement field And equal weighted and with uniform variance The likelihood function of the Gaussian distribution, and They represent the first k Measurement point cloud from one perspective and k Point cloud of results from -1 round of iterations The row vector of the coordinate points, This indicates that data points are obtained from a Gaussian distribution cluster. The probability, As a unit array, For corresponding points The displacement vector, This represents the Gaussian probability density function.

4. The multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering according to claim 3, characterized in that, The non-rigid displacement field The solution process also includes the solution of non-rigid displacement fields. The gradient is constrained by adding a weighted L2 regularization term, and the corresponding calculation expression is: In the formula, For weighted L2 regularization terms, The regularization coefficient is . ( ) is the Gaussian kernel function. For the corresponding kernel matrix, satisfying ( ), Represents the trace of a matrix.

5. The multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering according to claim 3, characterized in that, The non-rigid displacement field was analyzed using the EM algorithm. Perform iterative solutions.

6. The multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering according to claim 1, characterized in that, United First k Measurement point cloud from one perspective The measurement equation and the state equation corresponding to the processing in each iteration are used to obtain the first... k Multi-view point cloud fusion results of sample morphology under round iteration The calculation expression is: In the formula, and The first k -1 round and the k Multi-view measurement results of the wheel and The covariance matrix corresponding to the 3D Gaussian distribution of the measurement accuracy, and satisfying , for V k The covariance matrix corresponding to the point cloud measurement results from the given viewpoint. For resampling results, For the first k Kalman gain of the wheel.

7. The multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering according to claim 1, characterized in that, The process described using a three-dimensional Gaussian distribution is as follows: The direction perpendicular to the sample surface is defined as the z-axis of the world coordinate system. The major axis of the corresponding three-dimensional Gaussian distribution ellipsoid is parallel to the z-axis. The major and minor axes of the three-dimensional Gaussian distribution ellipsoid represent the measurement uncertainty of the extended depth-of-field three-dimensional microscopy system in the axial and lateral directions, respectively, i.e., the directions in which the uncertainty of the extended depth-of-field three-dimensional microscopy system is maximum and minimum.

8. A multi-view compensated three-dimensional microscopic measurement device based on Kalman filtering, characterized in that, It can be either Structure 1 or Structure 2; The structure includes a rotating stage, a first controller, and a single extended depth-of-field module. The rotating stage is used to carry and rotate the sample. The single extended depth-of-field module is used to acquire microscopic images of the sample. The first controller is used to change the observation angle of the sample by rotating the stage. In conjunction with the single extended depth-of-field module, the same sample is measured from different measurement angles. The structure executes the multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering as described in any one of claims 1-7 to obtain the three-dimensional microscopic measurement results of the sample. The second structure includes a fixed stage, a second controller, and multiple extended depth-of-field modules. The fixed stage is used to place the sample, and the multiple extended depth-of-field modules are used to observe the sample on the fixed stage from different angles. The second controller is used to execute a multi-view compensated three-dimensional microscopic measurement method based on Kalman filtering as described in any one of claims 1-7 based on the measurement results of the same sample from different measurement perspectives by the multiple extended depth-of-field modules, so as to obtain the three-dimensional microscopic measurement results of the sample.

Images