Micro-optical element point cloud splicing method based on prior surface type, medium and equipment

Abstract

This invention relates to the field of micro-optical element surface shape generation, and particularly to a method, medium, and device for point cloud stitching of micro-optical elements based on prior surface shapes. The method acquires two frames of point clouds with overlapping regions and an ideal design surface shape of the micro-optical element. An iterative nearest-point loss function is constructed in the overlapping region, and a prior constraint loss function is constructed by projecting the transformed point cloud onto the ideal design surface shape. The two are then weighted and fused into a total loss function. The optimal rigid body transformation parameters are solved by iteratively optimizing the total loss function to complete point cloud registration. This invention introduces the ideal design surface shape as a global geometric constraint, making the registration optimization not only driven by the distance between local point pairs but also guided by the overall surface shape geometry. This effectively suppresses cumulative drift in long sequence stitching and achieves high-precision full-aperture surface shape reconstruction of micro-optical elements at the hundred-millimeter scale.

Description

Technical Field

[0001] This invention relates to the field of micro-optical element surface pattern generation, and in particular to a method, medium, and device for stitching point clouds of micro-optical elements based on prior surface patterns. Background Technology

[0002] Micro-optical components (such as calcium fluoride columnar microlens arrays) are core components of the illumination system in deep ultraviolet lithography machines, and their surface morphology precision directly affects the beam shaping quality and component performance. These components are typically manufactured using wafer-level ultra-precision grinding and polishing processes, with apertures reaching the hundreds of millimeters level, while the surface morphology must simultaneously meet stringent requirements of nanometer-level geometric accuracy, sub-nanometer-level roughness, and near-zero subsurface damage.

[0003] In topography measurement, white-light interferometers are widely used due to their sub-micron vertical resolution. However, the field of view of a single white-light interferometer is typically only sub-millimeter (e.g., 100μm × 100μm), which cannot cover the entire surface of a component in a single scan. Therefore, it is necessary to acquire multiple frames of local point clouds through multiple scans, and then fuse these frames into complete surface data using point cloud registration and stitching techniques.

[0004] Existing point cloud registration methods mostly employ the Iterative Closest Point (ICP) algorithm, which estimates the rotation matrix and translation vector by minimizing the Euclidean distance between overlapping point clouds in adjacent frames. However, in long sequence stitching (e.g., hundreds to thousands of frames), the ICP algorithm suffers from significant cumulative error: the small deviation in each registration becomes amplified with each stitch, leading to a spatial drift in the final stitched result that becomes increasingly skewed. This fails to meet the accuracy requirements for nanometer-scale full-band topography measurement, especially affecting the low-frequency surface profile of micro-optical components, i.e., the overall surface profile trend. Although some studies have attempted to introduce closed-loop detection or global optimization to suppress drift, existing methods are either computationally too complex or rely on special scanning paths, making them difficult to apply to conventional raster scanning.

[0005] Therefore, there is an urgent need for a method that can effectively suppress the cumulative drift error in the stitching of long sequence point clouds of micro-optical elements and achieve full-aperture surface reconstruction at the 100-millimeter level that meets the nanometer-level precision requirements. Summary of the Invention

[0006] To address one of the aforementioned technical problems, the present invention adopts the following technical solution:

[0007] According to one aspect of the present invention, a method for stitching point clouds of micro-optical elements based on prior surface features is provided, comprising:

[0008] Obtain the point cloud to be registered and the reference point cloud, and obtain the ideal design surface of the micro-optical element; there is an overlapping area between the point cloud to be registered and the reference point cloud;

[0009] In the overlapping region, find the nearest point between the point in the point cloud to be registered and the reference point cloud, and construct an iterative nearest point loss function. , The following conditions must be met:

[0010] ;

[0011] in, Let i be the i-th point in the point cloud to be registered. As a reference point in the cloud and The corresponding nearest point, R is the rotation matrix, t is the translation vector, and N is the number of matching point pairs in the overlapping region. This represents the points in the point cloud to be registered after transformation by a rotation matrix and a translation vector;

[0012] The points in the transformed point cloud to be registered are projected onto the ideal design surface to construct the prior constraint loss function. , The following conditions must be met:

[0013] ;

[0014] in, for The nearest projection point on the ideal design surface;

[0015] according to as well as Construct the total loss function , The following conditions must be met:

[0016] ;

[0017] in, As a balance factor;

[0018] Using the rotation matrix and translation vector as independent variables, for Iterative optimization is performed to obtain the optimal rotation matrix and optimal translation vector, and the point cloud to be registered is transformed to obtain the registered point cloud.

[0019] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the above-described method for stitching point clouds of micro-optical elements based on a priori surface types.

[0020] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for stitching micro-optical element points cloud based on a priori surface types.

[0021] This invention has at least one of the following beneficial effects:

[0022] This invention addresses the cumulative drift problem caused by the lack of overall geometric constraints in traditional ICP stitching. By introducing the ideal design surface shape of micro-optical elements as a priori constraints, it unifies local point cloud registration and global surface shape consistency within the same optimization framework, effectively suppressing error accumulation in long sequence stitching.

[0023] Specifically, the present invention has the following beneficial effects:

[0024] (1) Effectively suppresses cumulative drift in frame-by-frame stitching, achieving stable registration with nanometer-level precision. Traditional ICP algorithms only aim to minimize the distance between point pairs in overlapping regions, resulting in numerous local minima in their optimization space. When there is slight noise or imperfect overlap between adjacent frames, the rotation matrix R and translation vector t obtained in each solution may contain slight deviations. Since subsequent frames continue stitching based on the registration result of the previous frame, these deviations propagate and accumulate along the stitching chain, forming cumulative drift. This invention introduces prior constraint loss into the total loss function. The transformed point cloud to be registered is projected onto the ideal design surface S, and the distance deviation between the transformed points and the projected points on the design surface is calculated. This constraint ensures that the optimization process is driven not only by local point-to-point distances but also by the overall surface geometry—when the registration parameters of a frame deviate from the correct solution, the transformed point cloud will deviate from the design surface. As it increases, it exerts a corrective force on the optimization direction. This is achieved through the balance factor λ. and Weighted fusion, while preserving local registration accuracy, introduces global constraints, which suppresses cumulative drift in each registration, rather than correcting it after all stitching is completed, thus achieving stable stitching with nanometer-level precision.

[0025] (2) Improve the global consistency of stitching results under different scanning paths. In traditional ICP stitching, the stitching result is significantly affected by the scanning path: different frame orders and different starting positions may lead to significant differences in the spatial pose of the final stitching result, that is, the stitching result is sensitive to the path. This invention introduces the design surface shape S as a global consistency reference, so that regardless of the scanning path used, the registration parameters of each frame are constrained to the unified condition that "the transformed point cloud should fit the design surface shape". This is because The construction of the model does not rely on the relative relationships between frames, but directly measures the degree of deviation between the transformed point cloud and the designed surface shape. This deviation is independent of the scanning path. Therefore, the registration results of each frame under different paths are constrained within a reasonable range of the designed surface shape, avoiding the divergence of the stitching results under different paths and ensuring global consistency.

[0026] (3) Applicable to large-scale full-aperture reconstruction of micro-optical elements with a diameter of 100 millimeters. For micro-optical elements with a diameter of 100 millimeters, full-aperture reconstruction requires stitching hundreds or even thousands of frames of field of view, and the point cloud data volume can reach more than 10GB. At the registration method level, this invention is compatible with the stitching method that only retains the transformation relationship—that is, after each frame is registered to a unified global coordinate system, only the transformation parameters are stored. , Instead of continuously growing the global point cloud, this reduces memory and computational overhead during the stitching process. Furthermore, the prior constraint method of this invention naturally connects with the subsequent point cloud compression and global graph optimization fusion method: prior constraints suppress drift during the local registration stage, compression methods reduce data volume during the storage stage, and global graph optimization eliminates residual errors in the final stage. These three elements work together to form a complete accuracy assurance system from local to global, suitable for large-scale full-aperture reconstruction scenarios of micro-optical elements at the hundred-millimeter level. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 The flowchart illustrates a point cloud stitching method for micro-optical elements based on prior surface features, provided in an embodiment of the present invention.

[0029] Figure 2 This is a flowchart of a method for detecting surface defects in micro-optical components, provided as an embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram of the defect detection network in a surface defect detection method for micro-optical components provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] As one possible embodiment of the present invention, such as Figure 1 As shown, a point cloud stitching method for micro-optical elements based on prior surface features is provided. This method includes:

[0033] S100: Obtain the point cloud to be registered and the reference point cloud, and obtain the ideal design surface of the micro-optical element. There is an overlapping area between the point cloud to be registered and the reference point cloud.

[0034] Specifically, the ideal design surface shape is derived from the CAD design data or theoretical surface shape equation of the micro-optical element.

[0035] In this step, both the point cloud to be registered and the reference point cloud are obtained by scanning the surface of the micro-optical element using a white light interferometer. Taking a 100-millimeter-scale calcium fluoride micro-optical element as an example, the field of view of a single white light interferometer probe is only on the millimeter scale (e.g., 100μm × 100μm). Therefore, it is necessary to obtain multiple frames of point clouds by scanning each frame of view, and then reconstruct the complete surface shape by stitching them together. For two adjacent frames of point clouds, their scanning areas need to maintain a certain overlap rate (usually not less than 30%) to ensure that there are enough common feature points in the subsequent registration process. The ideal design surface shape S is the theoretical surface shape data determined in the CAD design stage of the micro-optical element. For elements with regular geometric shapes, S can be analytically given by the theoretical surface shape equation; for elements with free-form surfaces or microstructure arrays, S comes from the discrete surface shape data in the CAD design model. In the subsequent registration process, S will serve as a global geometric constraint to suppress the cumulative drift introduced by frame-by-frame stitching.

[0036] S200: In the overlapping region, find the nearest point between the point in the point cloud to be registered and the reference point cloud, and construct an iterative nearest point loss function. , The following conditions must be met:

[0037] .

[0038] in, Let i be the i-th point in the point cloud to be registered. As a reference point in the cloud and The corresponding nearest point, R is the rotation matrix, t is the translation vector, and N is the number of matching point pairs in the overlapping region. This represents the points in the point cloud to be registered after transformation by a rotation matrix and a translation vector.

[0039] This step is the core of the standard ICP algorithm. Its goal is to solve for the optimal rigid body transformation parameters (R,t) to transform the point cloud to be registered into the coordinate system of the reference point cloud by minimizing the distance between point pairs in the overlapping region. Specifically, firstly, for each point in the point cloud to be registered... For the query point, search for the point with the closest Euclidean distance in the baseline point cloud. N sets of matching point pairs are established. Then, using the rotation matrix R (3×3 orthogonal matrix) and the translation vector t (3×1 column vector) as optimization variables, the sum of squared distances of all matching point pairs after transformation is minimized.

[0040] Understandably, the traditional ICP algorithm relies solely on the aforementioned loss function. The optimization process involves numerous local minima within its optimization space. When measurement noise or surface geometric feature degradation exists between adjacent frames, the R and t obtained in each solution may contain minute deviations. Since subsequent frames continue stitching based on the registration result of the previous frame, these deviations propagate and accumulate along the stitching chain, forming a cumulative drift, resulting in a spatial offset that becomes increasingly skewed with each stitch. This is precisely the motivation behind introducing prior constraints in this invention.

[0041] S300: Project the points in the transformed point cloud to be registered onto the ideal design surface, and construct the prior constraint loss function. , The following conditions must be met:

[0042] .

[0043] in, for The nearest projection point on the ideal design surface.

[0044] This step involves obtaining the transformed points. Then, project it along the normal direction onto the ideal design surface S to obtain the corresponding nearest projection point. The specific implementation of projection depends on the form of S: when S is represented by analytical equations (such as the equation of a sphere)... When S is given, it can be obtained by solving for the shortest distance from the point to the surface; when S is given by a discrete triangular mesh, it can be obtained by searching for the nearest triangular facet on the mesh and calculating the projection point.

[0045] The physical meaning is that if the registration parameters (R,t) are correct, the transformed point cloud should fit the actual surface of the element, and the deviation between the actual surface and the ideal design surface S should be within the tolerance range (for ultra-precision machined calcium fluoride micro-optical elements, this deviation is usually on the sub-micron level). Therefore, when (R,t) deviates from the correct solution, the transformed point cloud will deviate from S. This increases accordingly, thus exerting a corrective force on the optimization direction. In other words, This provides a global constraint for the registration process that is independent of the inter-frame relative relationships, so that the optimization is driven not only by the local point-to-point distances but also by the overall surface geometry.

[0046] S400: According to as well as Construct the total loss function , The following conditions must be met:

[0047] .

[0048] in, This is the balancing factor. Specifically, the value of λ ranges from [0.1, 0.5].

[0049] This step will involve local registration constraints. With global surface constraints The fusion is achieved through a weighted balancing factor λ. The balancing factor λ adjusts the relative weights of the two constraints: when λ is small (e.g., 0.1), the total loss function is dominated by ICP local registration, with prior constraints playing a supplementary corrective role, suitable for scenarios where the deviation between the actual surface and the designed surface shape is small; when λ is large (e.g., 0.5), the influence of prior constraints is enhanced, which can more effectively suppress cumulative drift, but may sacrifice the precision of local registration.

[0050] In practical applications, the specific value of λ can be set according to the processing precision and surface quality of the micro-optical element: for elements with high processing precision and good consistency between the actual and designed surface shapes, λ can be taken as a larger value to fully utilize the drift suppression capability of the prior constraints; for elements with relatively low processing precision, λ should be taken as a smaller value to avoid excessive bias in the registration results due to the prior constraints. In addition, λ can also be determined through cross-validation: on a calibration dataset with real registration parameters, stitching experiments are conducted through different λ values, and the λ with the smallest stitching error is selected as the final value.

[0051] S500: Using the rotation matrix and translation vector as independent variables, for Iterative optimization is performed to obtain the optimal rotation matrix and optimal translation vector, and the point cloud to be registered is transformed to obtain the registered point cloud.

[0052] Specifically, the iterative optimization uses the Levenberg-Marquardt algorithm, with the termination condition being that the change in the total loss function is less than 10. -6 Or it can reach the maximum number of iterations, 50.

[0053] The Levenberg-Marquardt (LM) algorithm is a nonlinear least squares optimization algorithm that combines gradient descent and Gauss-Newton methods. It is suitable for solving nonlinear optimization problems involving rotation and translation parameters. In each iteration, the LM algorithm adaptively adjusts the step size based on the gradient information of the current loss function: when the current solution is far from the optimal solution, the algorithm tends to follow the gradient descent direction to ensure convergence; when the current solution is close to the optimal solution, the algorithm switches to the Gauss-Newton direction to accelerate convergence.

[0054] In each iteration, the algorithm updates the rotation matrix and translation vector based on the gradient information of the current loss function until the termination condition is met. After convergence, the final R and t are taken as the optimal solution. and Then use Perform a rigid body transformation on the point cloud P2 to be registered to obtain the registered point cloud P2′.

[0055] S600: For a sequence containing multiple frames of point clouds, starting from the second frame of point cloud, the current frame of point cloud is used as the point cloud to be registered, and the already registered cumulative point cloud is used as the reference point cloud for registration. Finally, all registered point clouds are merged into the complete surface data of the micro-optical element.

[0056] This step involves frame-by-frame stitching. It uses a sequence containing n frames of point clouds. For example: As the reference frame, its coordinate system is the initial global coordinate system; starting from i=2, the coordinates are sequentially... Using the point cloud to be registered and the accumulated point cloud already registered to the global coordinate system as the reference point cloud, the optimal transformation parameters are solved using the methods described in steps S200-S500. and and will Transform to the global coordinate system. This process is repeated until all frames have been processed.

[0057] In practice, one of the following two methods can be used for stitching: Method 1 is frame-by-frame cumulative stitching, where each new frame is registered to the accumulated global point cloud, and the point cloud size continues to grow. This method is suitable for scenarios with a small number of frames (within tens of frames). Method 2 is to retain only the transformation relationship, where all frames are registered to the same global coordinate system (based on the first frame), and only the transformation parameters of each frame are stored. Finally, all transformed point clouds are merged. The second method is preferred in this invention because: in the first method, the global point cloud continuously expands with the number of frames, leading to a sharp increase in the demand for memory and computing resources; while the second method only stores the transformation relationship between frames, resulting in extremely low storage overhead, making it suitable for large-scale stitching of hundreds to thousands of frames on a hundred-millimeter scale. Furthermore, the second method naturally connects with the point cloud compression method in subsequent embodiments of this invention, further reducing storage pressure.

[0058] This embodiment effectively suppresses cumulative drift in long sequence stitching by introducing an ideal design surface shape as a priori constraint into the traditional ICP loss function, thus solving the problem of "the more you stitch, the more skewed" the surface shape becomes, affecting the accuracy of low-frequency surface shapes of optical elements. Simultaneously, the Levenberg-Marquardt algorithm is used for optimization, ensuring convergence efficiency and accuracy. This method is suitable for full-aperture high-precision point cloud stitching of micro-optical elements in the hundreds of millimeters, and the stitched surface shape data can meet the requirements of nanometer-level measurements.

[0059] As a possible embodiment of the present invention, after obtaining the registered point cloud, the method further includes:

[0060] S510: The registered point cloud is compressed into a low-dimensional feature vector through a feature extraction network, and the low-dimensional feature vector is stored while the original point cloud data is discarded. The feature extraction network is a neural network.

[0061] In this step, the input to the feature extraction network is a single-frame point cloud P, which contains a large number of points (approximately 1000 × 1000 points per field of view). The feature extraction network can compress this high-dimensional point cloud data into a low-dimensional feature vector, such as a 128-dimensional feature vector T, achieving a compression ratio of up to 10. 3 That's all. After compression, only the feature vector T is retained and the original point cloud P is discarded, thereby reducing the storage requirement of a single frame of data from several MB to about 0.5KB. For full-aperture reconstruction of a hundred-millimeter-scale element (about 1000 fields of view), the total storage of the feature vector is only about 0.5MB, which greatly alleviates the storage pressure.

[0062] S520: In the subsequent stitching process, the stored low-dimensional feature vectors are acquired, and the low-dimensional feature vectors are restored to point clouds through the inverse mapping network. The restored point cloud is then used as the reference point cloud and stitched with the point cloud to be registered in the next frame. The inverse mapping network is a neural network, and the feature extraction network and the inverse mapping network have mutually symmetrical network structures.

[0063] The S510 and S520 establish a cyclical process of "compression-storage-restore-stitching". (Using three-frame point cloud data...) For example: First, Compress to And store; then, Recovered by the inverse mapping network ,by As a benchmark and Registration and splicing ;Will Compress to And store, discard the original point cloud; then Restore to ,and Registration and stitching are performed in the same way. There is a reconstruction error between the point cloud restored by the inverse mapping network and the original point cloud, but this error can be controlled to the sub-micron level through training.

[0064] It is important to note that a reconstruction error exists between the point cloud reconstructed by the inverse mapping network and the original point cloud. This error is determined by the compression-decompression accuracy of the feature extraction network and the inverse mapping network. In practical applications, by employing an autoencoder training strategy (see network structure description below), the Chamfer Distance can be controlled to the sub-micron level, ensuring that the reconstructed point cloud can still be used for high-precision registration. However, multiple compression-decompression operations may amplify this error. Therefore, this invention introduces global graph optimization in the final S530-S560 steps to eliminate such residual errors.

[0065] Specifically, feature extraction networks include:

[0066] Edge convolutional layers are used to extract local geometric features of point clouds.

[0067] The EdgeConv layer takes a single-frame point cloud P as input. For each point pi, it first searches for its k nearest neighbors using the KNN (k=16) algorithm to construct a local graph structure. Then, it performs edge convolution operations on each local graph, calculating edge features for each edge. The output of EdgeConv is an N×64-dimensional local geometric feature matrix, where N is the number of points in the point cloud (approximately 1000×1000≈10 in a single field of view). 6 Each point's 64-dimensional feature vector encodes the local geometry information of its neighborhood (such as curvature, normal vector direction, etc.). The advantage of edge convolutional layers is that they do not rely on a fixed grid structure, can operate directly on unordered point clouds, and recalculate nearest neighbor relationships at each layer through a dynamic graph update mechanism, thereby expanding the receptive field layer by layer.

[0068] A multi-head self-attention layer is used to capture the global dependencies of point clouds.

[0069] The multi-head self-attention layer employs a 4-head self-attention mechanism, with the input being the N×64-dimensional feature matrix output by EdgeConv. For each attention head, a query matrix, key matrix, and value matrix are obtained through linear transformation, and attention weights are calculated. The outputs of the four heads are concatenated and then subjected to a linear transformation to obtain the final N×128-dimensional global feature matrix.

[0070] The role of multi-head self-attention layers is to enable each point to "see" information from all other points in the point cloud, thereby capturing long-range global dependencies. For example, the columnar microstructure arrays on the surface of micro-optical elements have periodically repeating geometric features, and the self-attention mechanism can utilize this global periodicity to enhance the discriminativeness of feature representations.

[0071] Max pooling layers are used to aggregate global features into feature vectors.

[0072] The max-pooling layer performs channel-wise max-pooling on the N×128-dimensional global feature matrix along the point dimension. Specifically, for each of the 128 channels, it takes the maximum value of that channel from N points, thus compressing the N×128-dimensional matrix into a 1×256-dimensional global feature vector. The purpose of the max-pooling layer is twofold: it transforms the variable-length point set features into a fixed-length global descriptor, allowing subsequent fully connected layers to process them with a uniform dimension; simultaneously, the max-pooling operation selects the most significant response in each channel, preserving the most discriminative feature information.

[0073] Fully connected layers are used to output low-dimensional feature vectors.

[0074] The fully connected layer compresses the 1×256-dimensional global feature vector to the target dimension D=128 through a linear transformation, outputting the final feature vector T. The role of the fully connected layer is to map high-dimensional global features to a compact low-dimensional space, minimizing storage overhead while retaining sufficient information for subsequent point cloud reconstruction.

[0075] The structure of the inverse mapping network is symmetrical to that of the feature extraction network.

[0076] The inverse mapping network is the reverse process of the feature extraction network. Its structure is as follows: fully connected layer (expanding the 128-dimensional feature vector to 1×256 dimensions) → upsampling + multilayer perceptron (generating a coarse point cloud) → fine adjustment layer (EdgeConv + MLP, performing local geometric corrections on the coarse point cloud) → outputting the final reconstructed point cloud P′. The feature extraction network and the inverse mapping network constitute an autoencoder structure. During training, Chamfer Distance is used as the loss function to measure the difference between the original point cloud P and the reconstructed point cloud P′. Chamfer Distance satisfies the following condition:

[0077] ;

[0078] in, This is used to represent finding the nearest point p in the reconstructed point cloud P′ for each point p in the original point cloud P, calculating the squared Euclidean distance between them, and then summing up the squared minimum distances for all p.

[0079] For each point p' in the reconstructed point cloud P', the nearest point p in the original point cloud P is found, the squared distance is calculated, and then summed. The sum of the two terms symmetrically measures the bidirectional matching error between the two point clouds. The Chamfer Distance is small only when P and P' are very close in spatial distribution (i.e., each point can find a very close point in the other).

[0080] This loss function encourages the reconstructed point cloud to approximate the original point cloud as closely as possible in terms of spatial distribution. Training can employ the Adam optimizer with a learning rate of 1×10⁻⁶. -3 The batch size is 32.

[0081] Furthermore, after obtaining the complete surface shape data of the micro-optical element, a global fusion step is also included:

[0082] S530: Obtain all stored low-dimensional feature vectors and restore them to their corresponding local point clouds through an inverse mapping network.

[0083] All stored feature vector sequences are read from storage and sequentially reconstructed into corresponding local point clouds through an inverse mapping network. Each local point cloud corresponds to the accumulated result of a certain stage in the stitching process and exists in a different local coordinate system. Even if prior surface constraints are used in local registration, there may still be residual errors in local registration, inconsistencies in loop closure, and reconstruction errors caused by compression-decompression. Therefore, global fusion is needed to eliminate these residual errors.

[0084] S540: Construct a graph optimization problem using the coordinate systems of each local point cloud as nodes and the transformation relationships between adjacent local point clouds as edges.

[0085] Construct a graph G=(V,E), where the node set V={Node1,Node2,…,Noden}, and each node Nodei corresponds to a coordinate system of a local point cloud; each edge (i,j) in the edge set E connects adjacent nodes i and j, and the weight of the edge is the transformation relationship between adjacent local point clouds. (Derived from transformation parameters recorded during the local stitching stage). If a closed loop exists in the scanning path (i.e., non-adjacent frames i and j actually scan the same region), an additional closed loop constraint edge (i,j) is added, and the transformation relationship of this edge is... This can be achieved by performing ICP registration at the closed loop.

[0086] The core idea of ​​the graph optimization problem is to optimize a global transformation G for each node Noni. i (This represents the transformation of the local point cloud corresponding to the node from its own coordinate system to the world coordinate system), so that the transformation relationship between adjacent nodes is consistent with the local stitching record.

[0087] S550: Optimizes global transformation parameters for each local point cloud by minimizing global reprojection error.

[0088] For each edge (i,j) in the graph, define an error function that satisfies the following condition:

[0089] ;

[0090] in, and Let be the global transformation parameters for nodes i and j, respectively. This records the transformation relationship from node i to node j during the local stitching stage. ∘ denotes the Frobenius norm. ∘ denotes the composition operation of transformations.

[0091] The physical meaning of this error function is: if the global transformation parameters are correct, then through... After transforming to the world coordinate system, and then through The result of transforming back to the coordinate system of node j should be consistent with the locally stitched record. Consistent.

[0092] The goal of global optimization is to minimize the sum of errors across all edges: The optimal global transformation parameters are obtained by using the Levenberg-Marquardt algorithm or the g2o graph optimization framework.

[0093] S560: Based on the optimized global transformation parameters, transform all local point clouds to the same global coordinate system and merge them into the final full-aperture surface point cloud.

[0094] All local point clouds are transformed to the same global coordinate system using globally optimized transformation parameters, and then merged into a complete point cloud. This global point cloud is the final full-aperture surface shape data of the micro-optical element, and its accuracy is guaranteed by both local prior constraints and global graph optimization.

[0095] This embodiment compresses large-scale point clouds into low-dimensional feature vectors using deep learning networks, significantly reducing storage and computing resource requirements (compression ratios can reach over 1000 times). Simultaneously, global graph optimization eliminates residual errors introduced by local registration and compression-decompression, ensuring the accuracy of the final surface data. This method enables high-precision full-aperture reconstruction of components down to the hundreds of millimeters in size to be completed on a standard workstation.

[0096] As another possible embodiment of the present invention, such as Figure 2 As shown, a method for detecting surface defects in micro-optical components is also provided, the method comprising:

[0097] S700: Uses an optical coherence tomography system to rapidly scan the surface of a 100-millimeter-scale micro-optical element to obtain at least one suspected defect area.

[0098] Optical coherence tomography (OCT) systems are based on the principle of low-coherence interference. They acquire tomographic images of a sample surface by measuring the time-of-flight (time-domain OCT) or frequency information (spectral-domain OCT) of the interference signals. The axial resolution of an OCT system is determined by the coherence length of the light source, typically ranging from micrometers to tens of micrometers. The lateral scanning speed can reach thousands of Å scans per second, thus enabling rapid full-aperture scanning of components ranging from hundreds of millimeters (e.g., 100 mm × 100 mm) in a short time.

[0099] Specifically, the OCT system scans the surface of the component line by line to acquire the backscattered signal intensity and phase information at each location. In normal areas, the surface scattering signal of the calcium fluoride micro-optical element is relatively uniform; however, in defective areas, due to abrupt changes in surface morphology or abnormal refractive index caused by defects, the OCT signal will exhibit abnormal reflectivity (signal intensity significantly higher or lower than the surrounding normal area) or phase abrupt changes (phase difference between adjacent sampling points exceeds a threshold). By setting reflectivity and phase abrupt change thresholds, the OCT system can automatically mark suspicious defective regions ROI1, ROI2, ..., and output the center coordinates and boundary range of each ROI.

[0100] While the positioning accuracy of the OCT system (on the order of micrometers to tens of micrometers) is insufficient for quantitative analysis of the specific geometric parameters of submicrometer-level defects, it is sufficient to coarsely locate the defect position within a full-diameter area of ​​hundreds of millimeters, providing a target area for subsequent fine white light interferometry inspection. This "coarse-to-fine" strategy significantly reduces the inspection time from the order of magnitude of point-by-point white light scanning of the entire diameter to the order of magnitude of full OCT scan plus a small number of ROI white light scans, significantly improving inspection efficiency while maintaining micrometer-level detection accuracy.

[0101] S710: For each suspected defect area, a white light interferometer is used to simultaneously acquire a depth map and an intensity map, which are naturally aligned and have the same resolution.

[0102] The depth map is used to represent geometric features, which include at least one of the following: protrusions, depressions, hole depths, and edge steepness.

[0103] Intensity maps are used to represent optical scattering characteristics, which include at least one of scratches, contaminants, and subsurface damage.

[0104] White light interferometers, based on the principle of white light interference, determine the height information of each point on a surface by analyzing the peak positions of the interference fringes, achieving a vertical resolution of 0.1 μm. In a single measurement, the white light interferometer simultaneously records two types of information: the height value corresponding to each pixel (forming a depth map) and the contrast of the interference fringes or the intensity of reflected light corresponding to each pixel (forming an intensity map). The natural alignment of the two modes stems from their origin from the same optical measurement. Pixels (x, y) in the depth map and pixels (x, y) in the intensity map correspond to the exact same physical location, eliminating the need for additional image registration.

[0105] The complementarity of depth maps and intensity maps is based on the following physical principle: Depth maps reflect the geometric morphology of the surface. For defects with obvious geometric features (such as pits several micrometers deep or protrusions exceeding micrometer height), clear areas of height anomalousness will appear in the depth map. However, for defects with insignificant geometric deformation but altered optical properties (such as shallow scratches—the depth may only be submicrometers, but significant light scattering occurs at the scratch edges; or subsurface damage—the surface geometry remains almost unchanged, but the light transmission characteristics of the damaged area are abnormal), the response of the depth map is weaker. Conversely, intensity maps reflect the optical scattering intensity at various locations on the surface. Defects with obvious optical features will show significant signals in the intensity map due to scattering or transmission anomalies. Therefore, the two modes are complementary, and their combined use can reduce the false negative and false positive rates.

[0106] Specifically, the S710 acquires depth and intensity maps using a white light interferometer, including:

[0107] S711: Controls the white light interferometer to perform multiple consecutive acquisitions of the suspected defect area, obtaining multiple frames of point cloud maps with overlapping areas.

[0108] Since the field of view of a single white light interferometer probe is only on the millimeter scale, while the size of a suspected defect area may exceed the field of view of a single measurement, multiple consecutive acquisitions of the ROI area are required. A certain overlap rate (e.g., 30%) is maintained between each acquisition to ensure sufficient common feature points in the subsequent stitching process.

[0109] For multiple acquisitions within each ROI, the improved ICP stitching method based on prior surface shape constraints described in the previous embodiments needs to be used for frame-by-frame registration. Since the surfaces of micro-optical elements within the ROI also have known ideal design surface shapes S, these can be introduced as prior constraints into the ICP registration process to suppress cumulative drift during frame-by-frame stitching and ensure the accuracy of point cloud stitching within the ROI.

[0110] S712: Register and stitch together multiple point cloud images to generate a complete point cloud image of the suspected defect area.

[0111] This step can be performed using the method described in steps S100-S600 for registration and stitching. All frames are transformed to a unified coordinate system and merged into a complete point cloud map of the ROI region.

[0112] In the large-scale stitching process of point clouds within a ROI, the problem of massive data storage and computational pressure is also faced. At this time, the point cloud compression and global fusion method described in the second embodiment above can also be introduced: the registered cumulative point cloud is compressed into a low-dimensional feature vector for storage, and the original point cloud is discarded to reduce memory usage; the point cloud is restored as a reference through an inverse mapping network during subsequent frame registration; after all frames are stitched, residual errors are eliminated through global graph optimization.

[0113] S713: Converts a complete point cloud map into a depth map, where each pixel in the depth map represents the height value at that location.

[0114] The complete point cloud map is projected onto the XY plane and discretized into a grid according to a preset spatial resolution. The average or median height value of the points falling into the same grid is taken as the height value of that pixel, thus obtaining the depth map D.

[0115] S714: Synchronously records the contrast of interference fringes or the intensity of reflected light corresponding to each pixel at each acquisition, and generates an intensity map. Each pixel in the intensity map represents the optical scattering intensity at that location.

[0116] During white light interferometry, each acquisition not only records the height value of each pixel (used to construct the depth map), but also simultaneously records the interference fringe contrast V(x,y) or reflected light intensity I(x,y) of each pixel. The interference fringe contrast V reflects the optical scattering characteristics of the surface at that location: when scratches, contaminants, or subsurface damage exist on the surface, the scattering characteristics change, causing anomalies in the V value. The intensity information from each acquisition is synthesized using the same stitching transformation parameters as the depth map to generate an intensity map I spatially aligned with the depth map. The value of each pixel I(x,y) represents the optical scattering intensity at that location.

[0117] S715: Normalize the depth map and intensity map to [0,1] respectively.

[0118] This normalization step eliminates dimensional differences, making the features of the two modalities comparable within the network. The depth map normalization method is as follows: ,in and These are the minimum and maximum values ​​in the depth map, respectively; the intensity map uses the same linear normalization method. All pixel values ​​in both the normalized depth and intensity maps are within the range [0,1].

[0119] S720: Input the normalized depth map and intensity map into the first encoder and the second encoder respectively, and extract the depth map feature vector f. D and intensity map eigenvector f I .

[0120] like Figure 3 As shown, both the first and second encoders employ convolutional neural networks (such as ResNet18). They have identical structures but do not share parameters. The first encoder is dedicated to extracting geometric features from the depth map, while the second encoder is dedicated to extracting optical scattering features from the intensity map. This parameter-sharing design allows the two encoders to learn their respective modal-specific feature representations—geometric features in the depth map (such as edges and curvature variations) and optical features in the intensity map (such as scattering patterns and reflectivity gradients) have different distribution characteristics in the feature space. Independent encoding helps preserve their respective discriminative information.

[0121] S730: will f D with f I The fusion is performed to obtain the fusion feature f. fusion .

[0122] The core purpose of fusion operation is to integrate geometric topography information and optical scattering information into a unified feature space, enabling subsequent detection networks to simultaneously utilize these two complementary information sources for defect discrimination. When one mode responds weakly to a certain type of defect, the other mode can provide discriminative features, thereby reducing the false negative and false positive rates.

[0123] Specifically, f D with f I The fusion can be carried out in any of the following three ways:

[0124] Firstly, it can be integrated into a direct end-to-end splicing: .

[0125] The simplest way to merge is to directly splice the beginning and end together. and By concatenating features along the feature dimension, a double-dimensional fused feature is obtained. This method fully preserves all feature information from both modalities, introduces no additional parameters, and has the lowest computational cost. During training, subsequent YOLOv8 detection heads can adaptively assign pairs through learning. and Attention weights for each channel. This method is suitable for scenarios where the defect type is relatively simple and both modalities have strong discrimination capabilities.

[0126] Secondly, the fusion is a weighted fusion: ,in These are learnable parameters.

[0127] Weighted fusion through learnable parameters Automatically adjust the weights of the two modes. Adaptive updates are performed via backpropagation during training. When a certain type of defect becomes more prominent in the depth map... It can adaptively move towards during training. tilt( Increase); and vice versa. This method introduces only one additional parameter. The computational cost is comparable to that of direct splicing, but it can automatically adjust the modal weights according to the data distribution.

[0128] Third, fusion is achieved through attention fusion: calculating f D with f I Cross-attention is used to concatenate or add the results together.

[0129] Attention fusion through computation and Cross-attention between modalities allows the network to learn the correlation between features from two different modalities. The advantage of attention fusion is that it adaptively determines which modal information to focus on for each feature dimension, making it suitable for complex scenarios with diverse defect types and significant differences in the contributions of different modalities. However, this approach has higher computational complexity than the previous two methods.

[0130] S740: f fusion Input the YOLOv8 inspection head and output the location and type of the defect.

[0131] YOLOv8 detector head receives fused features f fusion YOLOv8 employs a Feature Pyramid Network (FPN) for multi-scale feature extraction and performs defect detection on feature maps at three different scales. It outputs the location (center coordinates, width, height), category (e.g., scratches, holes, protrusions, inclusions (or contaminants)), and confidence score for each detection box. YOLOv8 uses an Anchor-Free detection paradigm, directly predicting the target center point and bounding box size, thus avoiding the hyperparameter sensitivity issues caused by anchor box design.

[0132] During training, the dataset consists of depth and intensity map pairs actually acquired by the white light interferometry system. For each ROI, the depth-intensity map pair is manually annotated with the defect bounding box and category label. The loss function uses the default YOLOv8 combined loss (classification loss + localization loss + confidence loss), the optimizer is AdamW, and the initial learning rate is 1×10⁻⁶. -3 The batch size is 8.

[0133] During the detection phase, for detection results with a confidence score higher than a preset threshold (e.g., 0.5), the bounding box coordinates of the defect (in the ROI coordinate system) and the category label are output. Combining the position information of the ROI in the global coordinate system (given by the OCT coarse inspection phase), the defect location can be mapped back to the global coordinate system of the micro-optical element, thereby generating a full-aperture defect distribution map.

[0134] This embodiment employs a two-stage strategy of "OCT rapid coarse inspection + white light interferometry multimodal fine inspection," balancing detection speed and accuracy. By complementary fusion of depth maps (geometric features) and intensity maps (optical features), the accuracy of defect identification is significantly improved, while false negatives and missed detections are reduced. The multimodal fusion network utilizes flexible feature fusion methods (stitching, weighting, attention) to adapt to the detection needs of different defect types. This method provides an efficient and reliable solution for full-aperture defect detection of hundreds of millimeter-scale micro-optical components.

[0135] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0136] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0137] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0138] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0139] An electronic device according to this embodiment of the invention. The electronic device is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the invention.

[0140] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and buses connecting different system components (including memory and processor).

[0141] The memory stores program code that can be executed by a processor, causing the processor to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of the present invention.

[0142] The storage may include readable media in the form of volatile storage, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0143] The storage may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0144] A bus can represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0145] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0147] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0148] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0149] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0150] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0151] Furthermore, the accompanying drawings are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes shown in the above drawings do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0152] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0153] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for stitching point clouds of micro-optical elements based on prior surface features, characterized in that, include: Obtain the point cloud to be registered and the reference point cloud, and obtain the ideal design surface of the micro-optical element; There is an overlapping area between the point cloud to be registered and the reference point cloud; In the overlapping region, the nearest point between the point in the point cloud to be registered and the reference point cloud is found, and an iterative nearest point loss function is constructed. , The following conditions must be met: ； in, For the i-th point in the point cloud to be registered, For the reference point in the cloud and The corresponding nearest point, R is the rotation matrix, t is the translation vector, and N is the number of matching point pairs in the overlapping region. This represents the points in the point cloud to be registered after being transformed by the rotation matrix and the translation vector; The points in the transformed point cloud to be registered are projected onto the ideal design surface, and a prior constraint loss function is constructed. , The following conditions must be met: ； in, for The nearest projection point on the ideal design surface; according to as well as Construct the total loss function , The following conditions must be met: ； in, As a balance factor; Using the rotation matrix and the translation vector as independent variables, for Iterative optimization is performed to obtain the optimal rotation matrix and optimal translation vector, and the point cloud to be registered is transformed to obtain the registered point cloud.

2. The method according to claim 1, characterized in that, The ideal design surface shape is derived from the CAD design data or theoretical surface shape equation of the micro-optical element.

3. The method according to claim 1, characterized in that, The value range of the balance factor λ is [0.1, 0.5].

4. The method according to claim 1, characterized in that, The iterative optimization employs the Levenberg-Marquardt algorithm, with a termination condition that the change in the total loss function is less than 10%. -6 Or it can reach the maximum number of iterations, 50.

5. The method according to claim 1, characterized in that, For a sequence containing multiple frames of point clouds, starting from the second frame of point cloud, the current frame of point cloud is used as the point cloud to be registered, and the already registered cumulative point cloud is used as the reference point cloud for registration. Finally, all registered point clouds are merged into the complete surface data of the micro-optical element.

6. The method according to claim 1, characterized in that, After obtaining the registered point cloud, the method further includes: The registered point cloud is compressed into a low-dimensional feature vector through a feature extraction network, and the low-dimensional feature vector is stored while the original point cloud data is discarded; the feature extraction network is a neural network. In the subsequent stitching process, the stored low-dimensional feature vector is obtained, and the low-dimensional feature vector is restored to a point cloud through the inverse mapping network. The restored point cloud is used as the reference point cloud and stitched with the point cloud to be registered in the next frame. The inverse mapping network is a neural network, and the feature extraction network and the inverse mapping network have a mutually symmetrical network structure.

7. The method according to claim 6, characterized in that, The feature extraction network includes: Edge convolutional layers are used to extract local geometric features of the point cloud; A multi-head self-attention layer is used to capture the global dependencies of the point cloud; Max pooling layers are used to aggregate global features into feature vectors; A fully connected layer is used to output the low-dimensional feature vector; the structure of the inverse mapping network is symmetrical to the structure of the feature extraction network.

8. The method according to claim 6, characterized in that, After obtaining the complete surface data of the micro-optical element, a global fusion step is also included: All stored low-dimensional feature vectors are obtained and then restored to their corresponding local point clouds through the inverse mapping network. A graph optimization problem is constructed using the coordinate systems of each local point cloud as nodes and the transformation relationships between adjacent local point clouds as edges. The global transformation parameters of each local point cloud are optimized by minimizing the global reprojection error. Based on the optimized global transformation parameters, all local point clouds are transformed to the same global coordinate system and merged into the final full-aperture surface point cloud.

9. A non-transitory computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a point cloud stitching method for micro-optical elements based on prior surface features as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a point cloud stitching method for micro-optical elements based on prior surface features as described in any one of claims 1 to 8.

Images