An adaptive real-time focus compensation method and system based on robust plane fitting

Abstract

The present application relates to a kind of adaptive real-time focusing compensation method and system based on robust plane fitting, belong to the technical field of generic semiconductor detection.It includes: the grid division of the plane where the object to be measured is located, confirm the point position needing automatic focusing, and obtain the high-resolution image of point position;Splice the image and carry out edge detection, obtain the corresponding edge position;Edge position encryption processing is carried out, and the rest area is sparsely handled, obtain sampling point position, and automatic focusing is carried out to sampling point position, obtain the Z-axis height of corresponding point position;According to sampling point position and Z-axis height, construct three-dimensional point cloud system, calculate reference plane by the system;Wherein, the system includes that three-dimensional point cloud coordinate set is input into outer loop layer and is carried out random sampling consistency iteration cycle, and reference plane parameter is optimized using inner loop layer;Through reference plane, obtain focusing compensation amount.The present application improves detection efficiency, reduces cost, ensures full-view high-definition image, and has wider application range.

Description

Technical Field

[0001] This invention relates to the field of semiconductor detection technology, and in particular to an adaptive real-time focus compensation method and system based on robust plane fitting. Background Technology

[0002] In the production and inspection of semiconductor products, Automated Optical Inspection (AOI) technology is one of the key means to ensure product quality. With the continuous advancement of semiconductor manufacturing processes, the precision requirements for detecting surface defects are becoming increasingly stringent, especially for nanoscale surface defects such as particulate contamination and micron-level scratches. These defects have a significant impact on product performance and reliability. However, current inspection technologies face numerous challenges in balancing high resolution and depth of field.

[0003] Traditional inspection solutions using high-resolution cameras (50 megapixels or higher) combined with high-magnification optical lenses can provide sufficient resolution to identify minute defects, but this optical system significantly reduces the depth of field. When the inspected surface exhibits slight height fluctuations due to factors such as tray movement or poor fixture tilt accuracy, a single shot can only guarantee a clear image of a localized area, while other areas suffer from defocusing and loss of detail, failing to meet the inspection requirements for full-field high-definition images.

[0004] To solve this problem, the following methods are currently mainly used:

[0005] The first method is multi-focal plane scanning. For each local shooting position, different layers are divided according to the estimated height fluctuation range and depth of field range. Layered movement along the Z-axis ensures that the surface area within each height range falls within the depth of field of at least one focal plane. Then, a clear image is obtained through multi-focal plane sharpness evaluation and image fusion. While this method offers high accuracy and can handle complex surface shapes, it requires significant cost and computational resources. Frequent Z-axis movement not only increases detection time but can also lead to image misalignment issues, thus failing to meet typical project requirements.

[0006] The second method is dynamic focusing hardware compensation. For example, liquid lenses achieve dynamic focusing by changing the lens shape or the optical properties of the liquid. The principle is based on dynamically adjusting the voltage according to a sharpness evaluation function, thereby changing the liquid contact angle and adjusting the curvature of the liquid-liquid interface to make the image clear in real time. This method has a short focusing time (only a few milliseconds), making it suitable for high-speed production lines. It can also eliminate image misalignment caused by Z-axis vibration, improving imaging stability. However, it is expensive, the adjustment range of a single lens focal length is usually small, and it is ineffective for local protrusions. Furthermore, its lifespan is short due to liquid evaporation and electrode aging.

[0007] The third approach involves deep learning-based image deblurring techniques. For example, Multi-Stage Progressive Recovery Network (MPRNet) takes a blurred image as input and trains it end-to-end to obtain a clear image, utilizing various optimization methods to improve robustness. This method eliminates the need for manual estimation of the blur kernel, adapts to complex scenes, and can recover high-frequency textures through a multi-stage network, thereby identifying minute defects. However, it is highly data-dependent; network training requires a large amount of labeled data, which is difficult to obtain for projects. Furthermore, the training process requires GPU cluster support, resulting in high inference latency on edge devices, making it difficult to apply efficiently in real-world production environments.

[0008] While existing technical solutions can resolve the conflict between high resolution and depth of field to some extent, they all suffer from high cost, low efficiency, and limited applicability, making it difficult to meet the demand for high-efficiency, low-cost, full-view high-definition images for the surface inspection of semiconductor products. Summary of the Invention

[0009] Therefore, the technical problem to be solved by the present invention is to overcome the high cost, low efficiency, and limited applicability of the existing technology, which makes it difficult to meet the demand of surface inspection of semiconductor products for high efficiency, low cost, and full-view high-definition images.

[0010] In a first aspect, to solve the above-mentioned technical problems, the present invention provides an adaptive real-time focus compensation method based on robust plane fitting, comprising:

[0011] The plane containing the object to be measured is divided into grids to identify the points that require autofocus and to acquire high-resolution images of those points.

[0012] The high-resolution image is stitched together and edge detection processing is performed to obtain the edge positions of the high-resolution image;

[0013] The remaining area outside the edge position is sparsed to obtain sampling points; the sampling points are automatically focused to obtain the Z-axis height of the corresponding points;

[0014] A three-dimensional point cloud system is constructed based on the sampling points and the Z-axis height, and a reference plane is calculated using the three-dimensional point cloud system. The three-dimensional point cloud system includes a three-dimensional point cloud coordinate set, an outer loop layer, and an inner loop layer. The three-dimensional point cloud coordinate set is input into the outer loop layer for random sampling consistency iterative loop, and the parameters of the reference plane are optimized using the inner loop layer.

[0015] The focus compensation amount is obtained through the reference plane.

[0016] In one embodiment of the present invention, the step of calculating the reference plane using the three-dimensional point cloud system is as follows:

[0017] S210. Input the three-dimensional point cloud coordinate set into the outer loop layer;

[0018] S220. Determine whether the number of outer loops in the outer loop layer is less than a set value; when the number of outer loops is less than the set value, randomly select n points from the three-dimensional point cloud coordinate set, and set the weight parameters of the n points to be non-zero; where n is a positive integer;

[0019] S230. Input the point cloud coordinates of the n points into the weighted least squares plane fitting subprocess to obtain the fitting plane;

[0020] S240. Input the fitted plane into the inner loop layer, and use the inner loop layer to optimize the plane parameters until the change value of the plane parameters is less than a preset threshold.

[0021] S250. Calculate the sum of residual values ​​of the inner loop layer in this iteration; determine whether the sum of residual values ​​in this iteration is less than the sum of residual values ​​of the inner loop layer in the previous iteration. If it is less, increment the outer loop count by 1 and return to S220 until the sum of residual values ​​in this iteration is not less than the sum of residual values ​​of the inner loop layer in the previous iteration.

[0022] When the number of external loops is not less than a set value, the optimized reference plane is obtained directly.

[0023] In one embodiment of the present invention, in step S250, when the sum of the residual values ​​in this step is not less than the sum of the residual values ​​in the previous step of the inner loop layer, the change value of the plane parameter of the inner loop layer is judged; when the change value of the plane parameter remains unchanged for K consecutive times, the optimized reference plane is directly obtained; when the change value of the plane parameter does not remain unchanged for K consecutive times, the process returns to step S220; where K is a positive integer.

[0024] In one embodiment of the present invention, the step of optimizing the parameters of the reference plane using the inner loop layer is as follows:

[0025] S310. Determine whether the number of inner loop iterations is less than a set value; if so, input the current fitting plane to the inner loop layer; based on the fitting plane, calculate the residual values ​​of all point cloud coordinates in the point cloud coordinate set;

[0026] S320. Update the weight parameters according to the residual values, and input the updated weight parameters and the point cloud coordinates of n points into the weighted least squares plane fitting subprocess to refit the plane; where n is a positive integer.

[0027] S330: Calculate the change value of the plane parameters between the refitted plane and the fitted plane in S310; when the change value of the plane parameters is not less than a preset threshold, increment the number of inner loops by 1, return the refitted plane to S310 for the next iteration, until the change value of the plane parameters is less than the preset threshold.

[0028] In one embodiment of the present invention, the expression for the weight parameter is:

[0029]

[0030] Among them, W i Let c represent the weight of the i-th point, and d represent the custom threshold. i This represents the residual value at the i-th point.

[0031] In one embodiment of the present invention, the steps of the weighted least squares plane fitting subprocess are as follows:

[0032] Calculate the centroid coordinates of the point cloud coordinates of the n points; use the centroid coordinates as a point on the fitting plane;

[0033] Construct the covariance matrix of the input point cloud coordinates, and perform eigenvalue decomposition on the covariance matrix to obtain an array of eigenvalues ​​and the corresponding eigenvectors;

[0034] A fitting plane is constructed based on the centroid coordinates and the eigenvector corresponding to the smallest eigenvalue in the eigenvalue array.

[0035] In one embodiment of the present invention, the formula for calculating the centroid coordinates is:

[0036]

[0037] Where μ(x,y,z) represents the centroid coordinates, N represents the number of input points, w represents the weight parameters, and i and k both represent the numbers.

[0038] In one embodiment of the present invention, when performing sparse processing on the remaining area other than the edge position, the method further includes encrypting the initially divided area according to the edge position or a specified position.

[0039] In one embodiment of the present invention, the method for obtaining a high-resolution image of the point is: to use an automatic parameter adjustment device to automatically focus on the point to obtain a high-resolution image.

[0040] Secondly, to solve the above-mentioned technical problems, the present invention provides an adaptive real-time focus compensation system based on robust plane fitting, comprising:

[0041] The image acquisition module is used to divide the plane where the object to be measured is located into grids, identify the points that need to be automatically focused, and acquire high-resolution images of the points.

[0042] An edge location acquisition module is used to stitch the high-resolution image and perform edge detection processing to obtain the edge location of the high-resolution image;

[0043] The point acquisition module is used to perform sparse processing on the remaining area outside the edge position to obtain sampling points; and to automatically focus on the sampling points to obtain the Z-axis height of the corresponding points.

[0044] A reference plane acquisition module is used to construct a three-dimensional point cloud system based on the sampling points and the Z-axis height, and to calculate a reference plane through the three-dimensional point cloud system; wherein, the three-dimensional point cloud system includes a three-dimensional point cloud coordinate set, an outer loop layer, and an inner loop layer; the three-dimensional point cloud coordinate set is input into the outer loop layer for random sampling consistency iterative loop, and the parameters of the reference plane are optimized using the inner loop layer;

[0045] The focus compensation amount acquisition module is used to obtain the focus compensation amount through the reference plane.

[0046] Thirdly, to solve the above-mentioned technical problems, the present invention provides a general semiconductor detection device, including the above-mentioned adaptive real-time focus compensation system based on robust plane fitting.

[0047] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0048] (1) The adaptive real-time focus compensation method and system based on robust plane fitting described in this invention accurately locates key points for autofocus by dividing the plane containing the object under test into a grid, and acquires high-resolution images of these points. This process ensures accurate focus for each key area, thereby significantly improving the overall clarity and quality of the image. By stitching these high-resolution images together and combining them with edge detection processing, the system can efficiently identify and extract important edge information in the image, providing an accurate and reliable data foundation for subsequent analysis.

[0049] (2) This invention performs sparse processing on areas outside the edge positions. This design not only effectively reduces the amount of data to be processed, significantly improving processing speed and efficiency, but also obtains the Z-axis height information of the corresponding points through automatic focusing sampling, laying a solid foundation for constructing a 3D point cloud system. In the process of constructing the 3D point cloud system and calculating the reference plane, the system uses an outer loop layer for random sampling consistency iterative loop, and combines it with an inner loop layer to optimize the parameters of the reference plane. This dual loop mechanism not only improves the stability and reliability of the fitted plane, but also significantly enhances the system's robustness to noise and outliers.

[0050] (3) This invention improves the level of automation, significantly reduces reliance on manual operation, effectively reduces the risk of errors caused by human factors, and improves the efficiency of the workflow. Furthermore, this invention has excellent versatility and adaptability, capable of handling objects of various complex shapes and ensuring the acquisition of high-definition images across the entire field of view. This not only significantly shortens the detection time and improves detection efficiency but also effectively reduces the overall detection cost.

[0051] (4) By using multiple random sampling, this invention effectively avoids the problem of the initial estimate getting trapped in local optima, significantly improving the fault tolerance for outliers. Simultaneously, this invention dynamically adjusts the weights of the sampling points based on the residual values, focusing the fitting process on inliers, thereby giving the algorithm higher robustness. This design ensures that clear and accurate images can still be obtained in complex detection environments. Attached Figure Description

[0052] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0053] Figure 1 This is a flowchart of an adaptive real-time focus compensation method based on robust plane fitting in a preferred embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of sparse sampling in a preferred embodiment of the present invention;

[0055] Figure 3 This is a flowchart of robust plane fitting in a preferred embodiment of the present invention. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0057] While existing technologies can alleviate the conflict between high resolution and depth of field to some extent, they still face problems such as high cost, low efficiency, and limited applicability, making it difficult to meet the demands of surface inspection of semiconductor products for efficient, low-cost, and full-view high-definition images. To address this, this invention proposes an adaptive real-time focus compensation method based on robust plane fitting. This method achieves efficient and low-cost detection while maintaining the clarity of the image under inspection by optimizing the sampling strategy, combining an efficient automatic parameter adjustment device, and an improved algorithm.

[0058] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0059] Example 1

[0060] Reference Figure 1 As shown, this embodiment of the invention provides an adaptive real-time focus compensation method based on robust plane fitting, including but not limited to the following steps:

[0061] S1. Divide the plane containing the object to be measured into a grid, identify the points that need to be automatically focused, and acquire high-resolution images of the points.

[0062] S2. Stitch together the high-resolution images and perform edge detection processing to obtain the edge positions of the high-resolution images;

[0063] S3. Perform sparse processing on the remaining area outside the edge position to obtain the sampling points; perform automatic focusing on the sampling points to obtain the Z-axis height of the corresponding points;

[0064] S4. Construct a three-dimensional point cloud system based on the sampling points and Z-axis height, and calculate the reference plane through the three-dimensional point cloud system. The three-dimensional point cloud system includes a three-dimensional point cloud coordinate set, an outer loop layer, and an inner loop layer. Input the three-dimensional point cloud coordinate set into the outer loop layer for random sampling consistency iterative loop, and use the inner loop layer to optimize the parameters of the reference plane.

[0065] S5. Obtain the focus compensation amount through the reference plane.

[0066] This invention provides an adaptive real-time focus compensation method based on robust plane fitting. By dividing the plane containing the object under test into a grid, key autofocus points are identified, and high-resolution images of these points are acquired. This ensures accurate focus for each key region, thereby improving the overall image clarity and quality. By stitching these high-resolution images and performing edge detection processing, the system can effectively identify and process important edge information in the image, providing an accurate data foundation for subsequent analysis. Furthermore, sparse processing is applied to areas outside the edge locations, which not only reduces the amount of data to be processed, improving processing speed and efficiency, but also obtains the Z-axis height of the corresponding points through autofocus sampling, laying the foundation for constructing a 3D point cloud system. During the construction of the 3D point cloud system and the calculation of the reference plane, the system performs random sampling consistency iterative loops through the outer loop layer and optimizes the parameters of the reference plane using the inner loop layer. This not only improves the stability and reliability of the fitted plane but also enhances the system's robustness to noise and outliers. Obtaining the focus compensation amount through the reference plane allows for precise adjustment of the camera's focusing system, further improving measurement accuracy. Therefore, the method described in this embodiment of the invention improves the level of automation, reduces reliance on manual operation, thereby reducing the risk of errors caused by human factors and effectively improving workflow efficiency. Furthermore, this method possesses excellent versatility and adaptability, capable of handling objects of various complex shapes and ensuring the acquisition of high-definition images across the entire field of view. This not only significantly shortens the detection time and improves detection efficiency but also helps reduce overall detection costs. Through these advantages, the method of this invention brings significant progress and value to the field of automated detection.

[0067] Specifically, for step S1, an adaptive grid sparse sampling technique is employed. First, the plane containing the product is divided into grids to determine the key locations requiring autofocus, and then high-resolution images of these key locations are acquired. The method for acquiring the high-resolution image of this point can be: using an automatic parameter adjustment device to autofocus on the point to obtain the high-resolution image. Alternatively, other suitable image acquisition methods can be flexibly selected based on specific application requirements and scenarios.

[0068] It should be noted that the autofocus processing (also known as autofocus tracking processing) described in this embodiment of the invention differs from the autofocus processing in the prior art. Traditional autofocus processing achieves focusing by adjusting the focal length of the lens, while the autofocus processing of this invention achieves focusing by adjusting the height of the camera (i.e., the working distance).

[0069] Furthermore, in this embodiment, an automatic parameter adjustment device is preferred for acquiring high-resolution images. This device has an automatic parameter adjustment function, which can automatically adjust the camera height, and specifically includes the following key modules:

[0070] The first data acquisition module is responsible for adjusting the lens axis to focus on the product to be inspected, thereby acquiring the initial camera working distance and the initial focus image.

[0071] The first magnification value calculation module calculates the initial magnification value based on the ratio of the pixels of the OLED display panel in the initial focus image to the resolution of the OLED display panel to be detected.

[0072] The second data acquisition module is used to determine whether the initial magnification value is within the preset magnification value range. If it is not within the range, the camera axis is moved to adjust the camera working distance, and the current camera working distance and the current focus image are acquired.

[0073] The second magnification value calculation module calculates the current magnification value based on the ratio of the pixels of the OLED display panel in the current focused image to the resolution of the OLED display panel to be detected.

[0074] The target camera working distance acquisition module is used to calculate a first difference between the current camera working distance and the initial camera working distance, and a second difference between the current magnification value and the initial magnification value; obtain a first proportional coefficient based on the ratio of the first difference and the second difference; and obtain the target camera working distance based on the first proportional coefficient and a preset standard magnification value; wherein the preset standard magnification value is within the range of preset magnification values.

[0075] The first parameter adjustment module is used to move the camera axis so that the camera working distance reaches the target camera working distance and acquires the target focused image. The target camera working distance is used as the new initial camera working distance, and the target focused image is used as the new initial focused image. The system then returns to the first magnification value calculation module. This process is repeated until the initial magnification value is within the preset magnification value range, completing the adjustment of the optical inspection device's camera working distance.

[0076] This embodiment utilizes the device to quickly focus on the sampling point and rapidly obtain the corresponding Z-axis height of the camera, thereby significantly reducing the focus tracking time, further improving detection efficiency, and reducing detection costs.

[0077] Specifically, in step S2, edge detection processing is performed on the stitched, clear image to accurately locate the edge positions. Image stitching can be achieved through techniques such as feature matching and image fusion, while edge detection can be performed using methods such as the Canny algorithm, the Sobel operator, or the Prewitt operator. These methods can effectively extract edge information from the image, providing accurate data support for subsequent processing steps.

[0078] Specifically, for step S3, when performing sparse processing on the remaining areas outside the edge positions, it also includes performing densification processing (i.e., densification processing) on ​​the initially divided areas based on the edge positions or specified positions, and the sampling points include the points obtained by the densification processing.

[0079] Specifically, for new products, first divide them evenly into grids (e.g. Figure 2 As shown on the left), the center point of each grid is used as a sampling point. Autofocus is applied to each sampling point to obtain a locally clear image and its corresponding camera height (i.e., Z-axis height). All locally clear images are stitched together to form a large image of the entire product, and the Z-axis height of each sampling point is recorded. This yields a globally clear large image and the Z-axis coordinates of each clear point. Since edge areas are high-incidence areas for defects, edge detection is performed on the initial global image to obtain edge locations. Encryption processing is applied to edge locations or user-specified areas, i.e., increasing sampling points near these areas to obtain more clear images, focusing on high-incidence defect locations. For other non-high-incidence areas, the number of sampling points is reduced, and sparsification processing is performed. Through the above encryption and sparsification processing, the sampling points for subsequent region division (e.g., ...) are obtained. Figure 2 (As shown on the right). This design allows the product to perform sampling only on subsequently divided areas when inspecting similar workpieces. This optimized sampling strategy not only reduces the number of sampling points but also significantly reduces autofocus time, improving inspection efficiency. It should be noted that this design only requires encryption processing on the initially divided area; subsequent divided areas do not require encryption.

[0080] For example, refer to Figure 2 As shown, Figure 2 The points in the left image represent the initial focus points for the same product. Taking a product with dimensions of 60×60mm as an example, each area is divided into 10mm sections, and the center point of all sections is used as the focus point. An efficient and time-saving automatic parameter adjustment device is used to automatically focus on these points to obtain a clear image. Subsequently, edge detection processing is performed on the stitched clear image to accurately locate the edges. Based on the edge positions or specified locations, the initially divided areas are densified, while other areas are sparsed, ultimately resulting in the image shown. Figure 2 The right-hand image shows the point distribution. Next, these optimized points are autofocused again to obtain the corresponding camera Z-axis height.

[0081] This invention employs an optimized sampling strategy. The product surface is sparsely sampled in a grid pattern, with denser sampling at key locations (such as designated locations or edges). This strategy not only improves the defect detection rate but also further enhances detection efficiency by eliminating redundant sampling points, ensuring the high efficiency of the detection process.

[0082] Specifically, refer to Figure 3 As shown, for step S4, a 3D point cloud system is constructed based on the sampling points and corresponding Z-axis heights obtained in step S3. A robust plane fitting algorithm fusing Random Sample Consensus (RANSAC) and Iterative Weighted Least Squares (IRLS) is used to accurately calculate the surface reference plane. This 3D point cloud system includes an outer loop layer and an inner loop layer: the outer loop is a random sample consistency iterative loop used to select point sets that conform to the plane model; the inner loop is a plane parameter optimization loop, which optimizes the plane parameters based on iterative weighted least squares. The specific calculation steps for the reference plane are as follows:

[0083] S410. Input the three-dimensional point cloud coordinate set into the outer loop layer.

[0084] S420. Determine if the number of outer loop iterations is less than the set value. If the number of outer loop iterations is not less than the set value, output the result directly. If the number of outer loop iterations is less than the set value, select n points from the three-dimensional point cloud coordinate set and set the weight parameters of the n points to be non-zero. Where n is a positive integer. The standard for setting the weight parameters is: the minimum number of subsets that fit the fitted plane.

[0085] S430. Input the point cloud coordinates of n points into the weighted least squares plane fitting sub-process to obtain the fitting plane; the specific steps of the weighted least squares plane fitting sub-process are as follows:

[0086] S431. Normalize the weight parameters and calculate the centroid coordinates of the point cloud coordinates of n points; use the centroid coordinates as a point on the fitting plane; in this embodiment, the formula for calculating the centroid coordinates is:

[0087]

[0088] Where μ(x,y,z) represents the centroid coordinates, N represents the number of input points, w represents the weight parameter, and i and k both represent numbers; x,y,z represent the coordinate values ​​on the X, Y, and Z axes, respectively.

[0089] S432. Construct the covariance matrix of the input point cloud coordinates to describe the distribution direction of the point cloud data. The mathematical expression of the covariance matrix is:

[0090]

[0091] in, This represents the coordinates of the input point, where i and j represent the row and column indices of the covariance matrix, corresponding to one of the three directions X, Y, and Z, respectively, and u... i This represents the average value of all data points in the i-th dimension.

[0092] S433. The covariance matrix is ​​decomposed using the Jacobian rotation method, progressively diagonalizing it to obtain the eigenvalue array and corresponding eigenvectors of the real symmetric matrix. The mathematical expression for the eigenvalue decomposition is:

[0093]

[0094] Among them, Cov k+1 Let J represent the covariance matrix at time step k+1. k Let T represent the state transition matrix at time step k, i.e., the Jacobian matrix, and let T denote the matrix transpose.

[0095] S434. Construct a fitting plane based on the centroid coordinates and the eigenvector corresponding to the smallest eigenvalue in the eigenvalue array. The plane normal vector corresponds to the eigenvector of the smallest eigenvalue. The formula for calculating the plane normal vector n is:

[0096]

[0097] Where, n x n y and n z These represent the components of the eigenvector. The plane equation is:

[0098] n x (x-μ x )+n y (y-μ y )+n z (z-μ z ) = 0;

[0099] Where, μ x μ y and μ z Indicates the coordinates of the centroid.

[0100] S440. Determine if the number of iterations in the inner loop is less than a set value. If it is less than the set value, input the current fitted plane into the inner loop layer. Based on the fitted plane, calculate the residual values ​​of all point cloud coordinates in the point cloud coordinate set;

[0101] S450. Update the weight parameters based on the residual values. In this embodiment, considering the distribution characteristics of the point coordinates, the Huber function is selected as the loss function. This function exhibits good robustness in handling outliers and can effectively balance the impact of outliers on the overall model. The expression for the weight parameters is:

[0102]

[0103] Among them, W i Let c represent the weight of the i-th point, and d represent the custom threshold. iThis represents the residual value at the i-th point.

[0104] The updated weight parameters and the point cloud coordinates of n points are input into the weighted least squares plane fitting subprocess (this part can be referred to in steps S431 to S434, which will not be repeated here) to refit the plane and continuously optimize the plane parameters.

[0105] S460. Calculate the change value of the plane parameters between the refitted plane and the fitted plane in step S440. When the change value of the plane parameters is not less than the preset threshold, increment the number of the inner loop by 1, and return the refitted plane to step S440 for the next round of iteration until the change value of the plane parameters is less than the preset threshold.

[0106] S470. When the change in the plane parameters is less than a preset threshold, calculate the sum of the residual values ​​of the current inner loop layer. Compare the sum of the current residual values ​​with the sum of the residual values ​​of the previous inner loop layer to determine whether the sum of the current residual values ​​is less than the sum of the residual values ​​of the previous inner loop layer. If it is less, increment the outer loop count by 1 and return to step S420 until the sum of the current residual values ​​is not less than the sum of the residual values ​​of the previous inner loop layer. If it is not less, judge the change in the plane parameters of the inner loop layer. When the change in the plane parameters remains unchanged for K consecutive times, the optimized reference plane is directly obtained; when the change in the plane parameters does not remain unchanged for K consecutive times, return to step S420, where K is a positive integer.

[0107] It should be noted that if the plane parameters are not updated for K consecutive iterations, the current plane is considered to have met the robustness condition, and the outer loop is terminated early to obtain the optimized fitted plane, which becomes the baseline plane. The settings for the number of iterations of the outer loop, the number of iterations of the inner loop, and the preset threshold can all be flexibly set and adjusted according to specific application scenarios and actual needs. Furthermore, steps S440 to S460 are execution steps of the inner loop layer, which optimizes the plane parameters until the change in the plane parameters is less than the preset threshold, at which point the inner loop layer processing stops.

[0108] This invention, through a combination of weighted least squares and the Huber function, effectively handles outliers, significantly improving fitting accuracy and robustness. The Huber function, as a robust loss function, suppresses the influence of outliers during the fitting process, ensuring stable fitting even with noise or outliers. Furthermore, the algorithm employs an inner and outer loop iterative mechanism to ensure convergence. The outer loop avoids fitting failures due to improper initial point selection through multiple random samplings, effectively preventing the algorithm from getting trapped in local optima. The inner loop dynamically adjusts the weights of sampling points, focusing the fitting process on inliers based on residual values, further enhancing the algorithm's tolerance to outliers. The algorithm also exhibits good adaptability and computational efficiency. By dynamically adjusting the number of iterations and parameter settings, the algorithm can flexibly handle point cloud data of varying scales and complexities, suitable for efficient processing of large-scale point clouds while maintaining high-precision fitting on small-scale data. Efficient calculation of centroid coordinates and covariance matrices further reduces computational complexity, resulting in excellent performance in large-scale point cloud data processing and significantly improved overall computational efficiency. In complex detection environments, this algorithm, with its robustness and fault tolerance, can ensure clear and accurate fitting results.

[0109] To illustrate the processing of 3D point cloud coordinates in the outer loop layer in step S4 in more detail, the following is a specific example of this step:

[0110] Step 1: Check if the number of iterations in the outer loop is less than the set value. If yes, continue to Step 2; if no, output the result and end the process.

[0111] Step 2: Select any 3 points (n=3) from the 3D point cloud coordinates, set their weight parameter value to 1 and the weight parameter value of the remaining points to 0; input the set of point cloud coordinates with weight parameters into the weighted least squares plane fitting subprocess to obtain the initial fitting plane;

[0112] Step 3: Input the initial fitted plane into the inner loop layer and check if the number of inner loop iterations is less than the set value. If yes, continue to step 4; otherwise, proceed to step 5.

[0113] Step 4: Increment the inner loop count by 1. The inner loop calculates the residual values ​​between the initial fitted plane and the coordinates of all points. Update the weights based on the residual values, and input the updated weights and point cloud coordinates into the weighted least squares plane fitting subprocess to refit the plane. Continuously optimize the parameters, and then calculate the parameter changes between the newly fitted plane and the plane fitted in the previous step. When the plane angle change is less than the threshold, exit the inner loop.

[0114] Step 5: Check if the weight parameter is less than the threshold. If yes, proceed to step 6; otherwise, return to step 2.

[0115] Step 6: Check if the sum of residual values ​​is less than the recorded minimum value (initially, this minimum value can be set to a large reference value or the first calculated sum of residual values). If yes, update the result: update the recorded minimum value to the current sum of residual values, and update the parameters of the fitted plane; increment the outer loop count by 1, and then return to Step 1. If no, check if there are no updates for 10 consecutive iterations (i.e., K=10). If yes, return to Step 2. If no, end the process.

[0116] Specifically, in step S5, using the determined reference plane, the required focus compensation amount, i.e., the Z-axis height corresponding to each position of the product, can be calculated. During the shooting process, the camera position is moved according to the fitted Z-axis height to capture all clear images. This process involves analyzing key feature points in the image to ensure precise adjustment of the camera's focusing system, thereby optimizing image sharpness and quality.

[0117] In summary, the embodiments of the present invention, by optimizing the sampling strategy, combining an efficient automatic parameter adjustment device, and improving the algorithm, achieve the goal of significantly reducing detection costs and improving detection efficiency while ensuring image clarity. They are particularly suitable for high-precision scenarios such as surface inspection of semiconductor products.

[0118] Example 2

[0119] Based on the same inventive concept, this embodiment provides an adaptive real-time focus compensation system based on robust plane fitting. The principle of solving the problem is similar to that of the adaptive real-time focus compensation method based on robust plane fitting provided in Embodiment 1, and the repeated parts will not be described again.

[0120] This embodiment provides an adaptive real-time focus compensation system based on robust plane fitting, including:

[0121] The image acquisition module is used to divide the plane where the object to be measured is located into grids, identify the points that need to be automatically focused, and acquire high-resolution images of the points.

[0122] The edge location acquisition module is used to stitch together high-resolution images and perform edge detection processing to obtain the edge locations of the high-resolution images.

[0123] The point acquisition module is used to perform sparse processing on the remaining area outside the edge position to obtain the sampling point; and to automatically focus on the sampling point to obtain the Z-axis height of the corresponding point.

[0124] The reference plane acquisition module is used to construct a three-dimensional point cloud system based on the sampling points and Z-axis height, and calculate the reference plane through the three-dimensional point cloud system. The three-dimensional point cloud system includes a three-dimensional point cloud coordinate set, an outer loop layer, and an inner loop layer. The three-dimensional point cloud coordinate set is input into the outer loop layer for random sampling consistency iterative loop, and the parameters of the reference plane are optimized using the inner loop layer.

[0125] The focus compensation amount acquisition module is used to obtain the focus compensation amount through the reference plane.

[0126] Example 3

[0127] This embodiment provides a general semiconductor detection device, including the adaptive real-time focus compensation system based on robust plane fitting provided in Embodiment 2.

[0128] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0132] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An adaptive real-time focus compensation method based on robust plane fitting, characterized in that, include: The plane containing the object to be measured is divided into grids to identify the points that require autofocus and to acquire high-resolution images of those points. The high-resolution image is stitched together and edge detection processing is performed to obtain the edge positions of the high-resolution image; The remaining area outside the edge position is sparsed to obtain sampling points; the sampling points are automatically focused to obtain the Z-axis height of the corresponding points; A three-dimensional point cloud system is constructed based on the sampling points and the Z-axis height. A reference plane is calculated using the three-dimensional point cloud system. The three-dimensional point cloud system includes a three-dimensional point cloud coordinate set, an outer loop layer, and an inner loop layer. The three-dimensional point cloud coordinate set is input into the outer loop layer for random sampling consistency iterative looping, and the parameters of the reference plane are optimized using the inner loop layer. The steps for calculating the reference plane using the three-dimensional point cloud system are as follows: S210. Input the three-dimensional point cloud coordinate set into the outer loop layer; S220. Determine whether the number of outer loops in the outer loop layer is less than a set value; when the number of outer loops is less than the set value, randomly select n points from the three-dimensional point cloud coordinate set, and set the weight parameters of the n points to be non-zero; where n is a positive integer; S230. Input the point cloud coordinates of the n points into the weighted least squares plane fitting subprocess to obtain the fitting plane; S240. Input the fitted plane into the inner loop layer, and use the inner loop layer to optimize the plane parameters until the change value of the plane parameters is less than a preset threshold. S250. Calculate the sum of residual values ​​of the inner loop layer in this iteration; determine whether the sum of residual values ​​in this iteration is less than the sum of residual values ​​of the inner loop layer in the previous iteration. If it is less, increment the outer loop count by 1 and return to S220 until the sum of residual values ​​in this iteration is not less than the sum of residual values ​​of the inner loop layer in the previous iteration. When the number of external loops is not less than a set value, the optimized reference plane is obtained directly; The focus compensation amount is obtained through the reference plane.

2. The adaptive real-time focus compensation method based on robust plane fitting according to claim 1, characterized in that, In step S250, when the sum of the residual values ​​in this step is not less than the sum of the residual values ​​in the previous step of the inner loop layer, the change value of the plane parameter of the inner loop layer is judged; when the change value of the plane parameter remains unchanged for K consecutive times, the optimized reference plane is directly obtained; when the change value of the plane parameter does not remain unchanged for K consecutive times, the process returns to step S220; where K is a positive integer.

3. The adaptive real-time focus compensation method based on robust plane fitting according to claim 1, characterized in that, The steps for optimizing the parameters of the reference plane using the inner loop layer are as follows: S310. Determine whether the number of inner loop iterations is less than a set value; if so, input the current fitting plane to the inner loop layer; based on the fitting plane, calculate the residual values ​​of all point cloud coordinates in the point cloud coordinate set; S320. Update the weight parameters according to the residual values, and input the updated weight parameters and the point cloud coordinates of n points into the weighted least squares plane fitting subprocess to refit the plane; where n is a positive integer. S330: Calculate the change value of the plane parameters between the refitted plane and the fitted plane in S310; when the change value of the plane parameters is not less than a preset threshold, increment the number of inner loops by 1, return the refitted plane to S310 for the next iteration, until the change value of the plane parameters is less than the preset threshold.

4. The adaptive real-time focus compensation method based on robust plane fitting according to claim 3, characterized in that, The expression for the updated weight parameters is: ； in, Indicates the first The weight of each point Indicates a custom threshold. Indicates the first The residual values ​​at each point.

5. The adaptive real-time focus compensation method based on robust plane fitting according to any one of claims 1 or 3, characterized in that, The steps of the weighted least squares plane fitting subprocess are as follows: Calculate the centroid coordinates of the point cloud coordinates of the n points; use the centroid coordinates as a point on the fitting plane; Construct the covariance matrix of the input point cloud coordinates, and perform eigenvalue decomposition on the covariance matrix to obtain an array of eigenvalues ​​and the corresponding eigenvectors; A fitting plane is constructed based on the centroid coordinates and the eigenvector corresponding to the smallest eigenvalue in the eigenvalue array.

6. The adaptive real-time focus compensation method based on robust plane fitting according to claim 5, characterized in that, The formula for calculating the centroid coordinates is: ； in, Indicates the centroid coordinates. Indicates the number of input points. Represents the weight parameters. and All of these represent serial numbers.

7. The adaptive real-time focus compensation method based on robust plane fitting according to claim 1, characterized in that, When performing sparse processing on the remaining area outside the edge position, the method further includes encrypting the initially divided area based on the edge position or a specified position, and the sampling points include the points obtained through the encryption process.

8. The adaptive real-time focus compensation method based on robust plane fitting according to claim 1, characterized in that, The method for obtaining a high-resolution image of the point is as follows: the point is automatically focused using an automatic parameter adjustment device to obtain a high-resolution image.

9. An adaptive real-time focus compensation system based on robust plane fitting, used to implement the adaptive real-time focus compensation method based on robust plane fitting as described in any one of claims 1 to 8, characterized in that, include: The image acquisition module is used to divide the plane where the object to be measured is located into grids, identify the points that need to be automatically focused, and acquire high-resolution images of the points. An edge location acquisition module is used to stitch the high-resolution image and perform edge detection processing to obtain the edge location of the high-resolution image; The point acquisition module is used to perform sparse processing on the remaining area outside the edge position to obtain sampling points; and to automatically focus on the sampling points to obtain the Z-axis height of the corresponding points. A reference plane acquisition module is used to construct a three-dimensional point cloud system based on the sampling points and the Z-axis height, and to calculate a reference plane through the three-dimensional point cloud system; wherein, the three-dimensional point cloud system includes a three-dimensional point cloud coordinate set, an outer loop layer, and an inner loop layer; the three-dimensional point cloud coordinate set is input into the outer loop layer for random sampling consistency iterative loop, and the parameters of the reference plane are optimized using the inner loop layer; The focus compensation amount acquisition module is used to obtain the focus compensation amount through the reference plane.

10. A general semiconductor detection device, characterized in that, This includes the adaptive real-time focus compensation system based on robust plane fitting as described in claim 9.

Images