A defect detection method based on multi-graph joint template and residual estimation
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGDONG SOLUDA TECHNOLOGY CO LTD
- Filing Date
- 2026-07-06
- Publication Date
- 2026-08-07
AI Technical Summary
[0010]1、线扫图像局部畸变问题：解决传统对齐方法无法克服线扫描平台机械微震动带来的行级（Row-wise）空间畸变问题
[0017]1、抗线扫重度震动：抛弃全局仿射思路，使用行级二维卷积核（Row-wise 2DKernels），每一行独立求解一个对齐核，契合线形传感器在扫描推进过程中的动态物理抖动特性，且容忍度远高于传统插值。
Smart Images
Figure CN122530618A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of industrial machine vision and semiconductor defect detection technology, specifically to a method and system for detecting surface defects on wafers or periodic structures based on multi-image joint templates and residual estimation, applicable to line scan cameras. Background Technology
[0002] Semiconductor manufacturing is the core of the modern electronics industry, and its yield rate directly affects production costs and efficiency. Wafers are the fundamental material for manufacturing semiconductor chips. During the complex manufacturing process, wafer surfaces are prone to various minute defects, such as particles, scratches, and pattern losses. These defects can severely impact chip performance and reliability. Therefore, rapid and accurate defect detection of wafers during the manufacturing process is crucial.
[0003] Currently, the commonly used defect detection method is to identify randomly generated defects by comparing adjacent identical structural patterns (Die / Cell). Taking wafer die-to-die inspection as an example, if the difference between the die to be inspected and its adjacent dies exceeds a threshold, it is considered a defect, and then the defect type is further subdivided, such as foreign matter, scratches, pits, etc.
[0004] In actual production environments, the following factors are unavoidable: 1. The object under test is out of focus to a certain extent, which leads to non-uniform blurring in the image, that is, the degree of blurring is different at different positions and along different directions; 2. Due to factors such as assembly and adjustment deviations, machine vibration, and fluctuations in scanning speed, sub-pixel misalignment and distortion of independent spaces occur. 3. The period of the grain cannot be divided evenly by the pixel unit, so there is sub-pixel misalignment in images with different periods; In the above situation, directly performing differential calculations on images from different periods will result in residuals of varying degrees even in areas without defects, easily leading to numerous false alarms and masking low-contrast and minor defects (missed detections). Therefore, image alignment is necessary before performing image differential calculations to eliminate inconsistencies in the imaging process. The accuracy of detection depends heavily on the precision of image alignment.
[0005] The shortcomings of existing technologies:
[0006] 1. Global Affine Transformation / Feature Point Matching: Image registration is performed by extracting feature points and calculating the homography matrix. However, industrial inspection often uses line-scan cameras. The platform motion vibration during the scanning process can cause independent micro-geometric distortions (jitter) between rows in the image. Global transformation cannot eliminate such non-rigid deformations, resulting in high-frequency edge false alarms after differencing.
[0007] 2. Optical Flow: Calculates dense displacement fields at the pixel level. However, when there are large-area real defects in the image, the defects will seriously interfere with the calculation of optical flow, leading to registration errors in the defect area, and instead "smoothing out" or deforming the real defects. Summary of the Invention
[0008] The objective of this invention is achieved through the following technical solutions.
[0009] The technical problem to be solved by this invention is:
[0010] 1. Local distortion problem in line scan images: Solve the problem of row-wise spatial distortion caused by mechanical micro-vibration of the line scan platform, which cannot be overcome by traditional alignment methods.
[0011] 2. Defect Interference Registration Problem: This addresses the issue that existing registration algorithms are affected by real defect pixels in the image when calculating displacement parameters, leading to incorrect estimation of registration parameters.
[0012] 3. Ghost Defect and Attribution Determination: This addresses the issue of strong defects on a single image contaminating the common template during the synthesis of multiple image sets using a common template, leading to misclassification as defects on other defect-free images.
[0013] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A defect detection method for line scan cameras based on multi-image joint template and residual estimation includes: Acquire at least three identical field-of-view images captured by a line scan camera; A joint optimization model is constructed, with the row-level two-dimensional convolution kernels corresponding to each image, the clean template shared by multiple images, and the defect residuals corresponding to each image as variables. The objective function minimizes the difference between the aligned image and the "template + residual" and applies sparsity constraints to the residuals. The model is solved by alternating iterative methods. In each iteration, the convolution kernel and template are updated sequentially, and the residuals are sparsely optimized and mutually exclusive are determined. After the iteration converges, the defect residual is output as the detection result.
[0014] In the above scheme, each row of each image has an independent two-dimensional convolution kernel to simulate the local translation, blurring, and minor deformation caused by vibration in that row. During template synthesis, the residual estimated in the previous round is subtracted from the aligned image and then averaged, avoiding the mixing of defect energy into the background. The residual mutual exclusion judgment forces that the defect signal at the same pixel location is retained on at most one image, fundamentally eliminating ghosting.
[0015] The present invention also provides a defect detection system, including a data input module, a joint optimization module, and a defect output module, wherein the joint optimization module includes a convolution kernel solver, a template synthesizer, and a residual solver to execute the above-described method. Additionally, a computer-readable medium storing a computer program is also provided.
[0016] Compared with the prior art, the beneficial effects of the present invention are:
[0017] 1. Resistance to severe vibration during line scanning: Abandoning the global affine approach, row-wise 2D kernels are used, with each row independently solving for an alignment kernel. This is in line with the dynamic physical jitter characteristics of the linear sensor during the scanning process and has a much higher tolerance than traditional interpolation.
[0018] 2. "Defect immunity" alignment mechanism: for convolution The objective is to approximate the sum of the template and the residual. rather than just Due to residuals It absorbs real defect energy, so pixels with defects will not cause serious penalty interference when calculating the alignment convolution kernel, fundamentally preventing the industry pain point of "large defects causing overall registration failure".
[0019] 3. Eliminating the "ghosting" phenomenon: A three-image residual mutual exclusion mechanism is proposed. Traditional image mean or median background modeling often results in the original image being excluded. Figure 1 The black spots were diluted into light black patches in the background, resulting in a defect-free appearance. Figure 2 The difference reveals "white dots (ghosting)". The mutual exclusion mechanism forces that only one image be identified as having residuals for each spatial location, which can more cleanly remove defects. Attached Figure Description
[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0021] Figure 1 This is the overall system structure and flowchart of the present invention.
[0022] Figure 2 This is a flowchart illustrating the alternating optimization process within the algorithm of this invention.
[0023] Figure 3 This is a schematic diagram illustrating the actual operating effect of the system. Detailed Implementation
[0024] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0025] 1. System Overall Structure (Divided into Modules)
[0026] This invention proposes a multi-image joint alignment and defect detection system suitable for line scan cameras, based on row-level convolutional kernels and residual estimation models. Logically, it mainly consists of three modules: a data input module, a joint optimization module, and a defect output module. The overall system structure is as follows: Figure 1 As shown. The function of each module is as follows:
[0027] Data input module: Receives three grayscale images with the same field of view / periodic structure from the line scan camera. These images may contain unknown deformations and unknown defects.
[0028] The joint optimization module comprises three interdependent sub-solvers: a convolution kernel solver, a template synthesizer, and a residual solver. They work alternately within a closed loop, decoupling the coupled deformation error, background features, and defect signals until the estimated convolution kernel, template, and residual stabilize, at which point the optimization process ends.
[0029] Defect Output Module: Outputs the defect residuals after optimization. .
[0030] The actual operating effect of the system is shown in the figure below. Figure 3 As shown.
[0031] 2. Key Algorithm:
[0032] This invention proposes a "joint optimization model of multi-image joint template and residual estimation," which unifies image alignment (convolution kernel matrix), template reconstruction, and defect (residual) extraction into a single objective function for joint iterative solution. This model is applicable to three input images located in the same field of view and containing unknown defects. Model assumptions:
[0033] ;
[0034] The original grayscale image is a width of Gao Wei A two-dimensional array;
[0035] Row-level two-dimensional convolutional kernels (each row of each image has its own independent set of kernel parameters) are used to simulate local geometric distortions and smoothing. It is a dimension of , , A three-dimensional array, in which The height of the convolution kernel. Where is the width of the convolution kernel, so It is common The size is A set of two-dimensional convolution kernels;
[0036] The three images share a clean template (without defects), which is the shape and Same array;
[0037] The defect residual corresponding to each image is the shape and... Same array.
[0038] The optimization objective is:
[0039] ;
[0040] in , Regularization hyperparameters specified by the user are used to suppress model complexity to avoid overfitting. The larger The smaller the norm, The larger The smaller and sparser the hyperparameters, the better. Appropriate hyperparameters can lead to more reasonable results and reduce the impact of noise on the solution.
[0041] Because there are many optimization variables and the objective function contains the L1 norm, the optimal analytical solution cannot be obtained directly. Therefore, an alternating optimization approach is used. The specific algorithm execution flow is divided into two stages: initialization and alternating optimization.
[0042] Step 1: Initialization Phase
[0043] (1) Verify the three input two-dimensional grayscale images Consistent dimensions.
[0044] (2) Use the clean template Initialize to the arithmetic mean of the input image.
[0045] (3) Defect residual Initialize as an array of all zeros with the same size as the image.
[0046] Step 2: Alternating optimization phase (execute repeatedly until the maximum number of iterations, such as 15 times)
[0047] In each iteration, the following four core sub-algorithms are executed sequentially:
[0048] (1) Kernel Update Algorithm:
[0049] by Update the convolution kernel as the target. To solve the 2D convolution kernel, a local fitting problem is constructed separately for each row of each image. Wherein... The convolution kernel of the row is The original image Row pixels and top and bottom The augmented array composed of rows is denoted as The first clean template Row pixels are denoted as The solution can be obtained by solving the following optimization problem. :
[0050] ;
[0051] This problem can be solved by... The corresponding design matrix is constructed, and the optimal solution is obtained using the least squares method without iterative solution. After optimization, the calculated row-level convolution kernels are... Compared to the original image Perform convolution to obtain the currently aligned image. This convolution needs to be performed row by row and finally combined, that is... ,in express The Row pixels.
[0052] (2) Template Update:
[0053] Having the currently aligned image The residual from the previous estimation Then, the template is re-estimated. The template is directly taken as the mean of the aligned image minus the corresponding residual:
[0054] ;
[0055] (3) Mutually Exclusive Residual Update:
[0056] Optimize the residual based on the following objective function :
[0057] ;
[0058] a) of which and All are fixed, and the optimization result is denoted as Algorithms suitable for mixed norm optimization problems, such as the Alternating Direction Method of Multipliers (ADMM), can be used to solve this problem. The L1 norm is used to indicate that the model's assumptions about defects exhibit sparsity.
[0059] b) The algorithm performs a pixel-by-pixel comparison of the three residual images: for any coordinate on the image... Compare the absolute values of the residuals. :
[0060] ;
[0061] That is, the first The image with the largest estimated absolute residual value is then... The remaining residual values are set to 0. This approach suggests that the algorithm assumes the defect will typically only appear in one image.
[0062] Calculate the objective function The optimization stops when the objective function fails to decrease after multiple iterations, or when the relative change is less than the threshold, or when the maximum number of iterations is reached.
[0063] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A defect detection method based on multi-graph joint template and residual estimation, characterized in that, Includes the following steps: Acquire at least three images of the same field of view captured by a line scan camera to be detected; The at least three images to be detected with the same field of view are uniformly modeled as follows: the result of each image to be detected after being convolved by its own row-level two-dimensional convolution kernel is equal to the sum of the clean template shared by multiple images and the defect residual of that image, thereby constructing a joint optimization problem; The joint optimization problem is solved by alternating iterations, and convolution kernel update, template update and residual update are performed in sequence. In each convolution kernel update and template update, the defect interference is removed from the image by using the currently estimated defect residual to obtain adaptive image registration. After the iteration converges, the final estimated defect residual is output as the defect detection result.
2. The defect detection method based on multi-graph joint template and residual estimation according to claim 1, characterized in that, The row-level two-dimensional convolution kernel is: for each row of each image to be detected, a two-dimensional convolution kernel is independently assigned to form a row-level convolution kernel matrix; the parameters of each row convolution kernel are obtained by solving row by row with the sum of the shared template and the residual of the image as the approximation target.
3. The defect detection method based on multi-graph joint template and residual estimation according to claim 2, characterized in that, The row-by-row solution uses the least squares method, taking the augmented image patch formed by the current row and its adjacent rows as input, and the corresponding row pixels of the shared template and the sum of the image residuals as the target, to calculate the parameters of the convolution kernel for that row.
4. The defect detection method based on multi-graph joint template and residual estimation according to claim 1, characterized in that, The template update steps are as follows: The aligned image is obtained by performing row-by-row convolution on each image to be detected using the currently estimated row-level convolution kernels. The background estimate for each image is obtained by subtracting the currently estimated defect residual from each aligned image. The pixel-wise arithmetic mean of all background estimates is used as the updated shared clean template.
5. The defect detection method based on multi-graph joint template and residual estimation according to claim 1, characterized in that, The number of images to be detected is three.
6. The defect detection method based on multi-graph joint template and residual estimation according to claim 5, characterized in that, The residual update step includes: Under the conditions of fixed convolution kernel and shared template, the defect residual of each image to be detected is optimized with sparsity constraints to obtain intermediate residuals; Perform cross-image mutual exclusion processing on the intermediate residuals: For the same pixel position, compare the absolute values of the intermediate residuals of the three images at that position, retain the residual value corresponding to the image with the largest absolute value, and force the corresponding position residual values of the other two images to be zero. The result is used as the defect residual.
7. The defect detection method based on multi-graph joint template and residual estimation according to claim 6, characterized in that, The sparsity constraint is an L1 norm regularization constraint, and the optimization solution adopts the alternating direction multiplier method.
8. The defect detection method based on multi-graph joint template and residual estimation according to claim 1, characterized in that, The initialization method for the alternating iterative solution is as follows: The shared clean template is initialized as the pixel-wise arithmetic mean of all images to be detected; Initialize the defect residuals of each image to an array of all zeros.
9. The defect detection method based on multi-graph joint template and residual estimation according to any one of claims 1 to 8, characterized in that, The conditions for convergence of the iteration include: the objective function value does not decrease for multiple consecutive iterations, the relative change in the objective function value is less than a preset threshold, or the number of iterations reaches a preset maximum number.
10. A defect detection system based on multi-graph joint template and residual estimation, characterized in that, include: The data input module is used to acquire at least three identical field-of-view images of the target image captured by the line scan camera; The joint optimization module is configured to perform the method as described in any one of claims 1 to 9, and internally comprises: A convolution kernel solver is used to solve row-level 2D convolution kernels for each image row by row, with the goal of a shared template and the sum of residuals, and to generate aligned images. Template synthesizer for updating the shared clean template based on the aligned image and the current residual; A residual solver for optimizing residuals under sparse constraints and performing cross-image mutual exclusion determination; The defect output module is used to output the final estimated defect residual.