Defect detection system, method for evaluating performance thereof, and storage medium

Abstract

This application provides a defect detection system, its performance evaluation method, and a storage medium. The method includes acquiring a baseline data set and a high-confidence first detection data set and a low-confidence second detection data set from the same detection output. The two sets are spatially matched with the baseline data set, and based on the matching results, each data point in the baseline data set is divided into points that have no matching relationship with either set of detection data and points that only have a matching relationship with the second detection data set. This subdivides the general "missed detections" in traditional techniques into a first group representing insufficient hardware capabilities and a second group representing inappropriate algorithm parameters. Furthermore, based on the analysis of the grouping results, the hardware configuration parameters or algorithm configuration parameters of the detection system are adjusted when the results of the first or second group do not meet expectations. This application improves the accuracy and optimization efficiency of the detection system performance evaluation by effectively utilizing the low-confidence data discarded in traditional techniques.

Description

Technical Field

[0001] This application relates to the field of semiconductor manufacturing technology, specifically to a defect detection system and its performance evaluation method, and a storage medium. Background Technology

[0002] In semiconductor wafer manufacturing, defect detection is a core component of yield control. As feature sizes continue to shrink and process complexity increases, the performance requirements for defect detection systems become increasingly stringent. The consistency between different inspection equipment, the accuracy of inspection formulations, and the reliability of defect classification models directly affect the confidence level of defect detection results and the accuracy of subsequent yield analysis. Therefore, accurately evaluating the performance of inspection equipment, optimizing inspection formulations, and validating defect classification models have become crucial issues in the semiconductor manufacturing field.

[0003] In existing technologies, overlay analysis is commonly used to evaluate the performance of testing equipment or optimize testing formulas. Specifically, this involves spatially comparing the defect coordinates output by the current testing equipment (referred to as local data) with known reference defect coordinates (referred to as reference data, such as high-precision equipment re-judgment results or manually annotated true values). Based on the matching results, defects are categorized into three types: common defects (defects detected by both local and reference data), locally unique defects (defects detected only by local data), and missed defects (defects present only in reference data). By analyzing the quantity and distribution of these three types of defects, performance indicators such as the detection system's capture rate and false alarm rate can be evaluated.

[0004] However, existing overlay analysis schemes, which only use local and reference data for binary comparison, have revealed problems in practical applications. When a reference defect fails to find a match in the local data, it is uniformly defined as a "missed detection," but this masks two distinct physical realities: one is that the detection device's sensor simply did not capture the signal at that location, indicating insufficient hardware capability; the other is that the detection device captured the signal, but due to improperly set feature thresholds or misjudgment by the classification algorithm, the signal was filtered out as noise, indicating improper algorithm parameter settings. Because these two situations cannot be distinguished, engineers cannot determine whether a hardware upgrade is needed or only algorithm parameter adjustments are required, leading to difficulties in problem localization and low optimization efficiency.

[0005] Furthermore, in actual testing, a real reference defect may be identified by the testing equipment as multiple adjacent and independent defect points. When calculating the capture rate, existing technologies may lead to an inflated capture rate, even exceeding 100%, if the number of matched local defect points is used as the numerator. If the number of matched reference defect points is used as the numerator, the information loss caused by defect fragmentation is ignored, and the performance of the testing system cannot be truly reflected.

[0006] Meanwhile, existing technologies only use high-confidence local screening data for comparison, while a large number of low-confidence signals generated during the detection process are usually discarded directly, and the information contained in these data about the algorithm's decision-making behavior is not effectively utilized. Summary of the Invention

[0007] This application provides a defect detection system, its performance evaluation method, and storage medium, aiming to solve the technical problems in the prior art, which are that the existing technology only uses local data and reference data for binary comparison, resulting in the inability to distinguish between hardware missed detections and algorithm misclassifications, distortion of capture rate calculation in one-to-many matching scenarios, and ineffective utilization of low-confidence data.

[0008] In a first aspect, one embodiment provides a performance evaluation method for a defect detection system, comprising:

[0009] Obtain a baseline data set, a first detection data set, and a second detection data set; wherein, the baseline data set is the true value data of defect detection, the first detection data set and the second detection data set are the detection results output from the same defect detection process, and the first detection data set is the data retained after being filtered by a detection threshold or classification model, and the second detection data set is the data that has been filtered or downweighted during the detection process;

[0010] The first and second detection data groups are spatially matched with the reference data group to obtain the first matching result and the second matching result.

[0011] Based on the first matching result and the second matching result, the data points in the reference data group are grouped to obtain a first group and a second group; wherein, the first group includes reference data points in the reference data group that have no matching relationship with either the first detection data group or the second detection data group; the second group includes reference data points in the reference data group that have no matching relationship with the first detection data group but have a matching relationship with the second detection data group.

[0012] If at least one of the grouping results in the first group and the second group fails to meet expectations, the configuration parameters of the defect detection system are adjusted. The configuration parameters include hardware configuration parameters and algorithm configuration parameters.

[0013] In some embodiments, adjusting the configuration parameters of the defect detection system when at least one of the grouping results of the first group and the second group does not meet expectations includes:

[0014] Based on the grouping results, an evaluation system is constructed that includes at least a first performance index and a second performance index, wherein the first performance index characterizes the hardware capture capability of the defect detection system, and the second performance index characterizes the algorithm decision capability of the defect detection system.

[0015] The adjustment direction of the defect detection system is determined based on the first performance index and the second performance index.

[0016] In some embodiments, grouping the data points in the baseline data group according to the first matching result and the second matching result includes:

[0017] Traverse each benchmark data point in the benchmark data group and treat the benchmark data points that do not appear in the first matching result as points to be processed.

[0018] Iterate through each of the points to be processed. For each point to be processed:

[0019] If the point to be processed does not appear in the second matching result, the point to be processed is assigned to the first group; the first group is used to characterize defects that are not captured due to insufficient hardware capabilities.

[0020] When the point to be processed appears in the second matching result, the point to be processed is assigned to the second group; the second group is used to characterize defects that are mistakenly filtered due to improper algorithm parameters.

[0021] In some embodiments, after grouping the data points in the reference data set, the method further includes:

[0022] The benchmark data points appearing in the first matching result are assigned to a third group; the third group is used to characterize the defects that were correctly detected.

[0023] Traverse each reference data point in the third group and assign the first detection data point that matches it in the first detection data group to the fourth group;

[0024] The first detection data points within the fourth group are deduplicated so that the same first detection data point is recorded only once; the fourth group is used to characterize the reported defects corresponding to the correctly detected defects.

[0025] In some embodiments, after grouping the data points in the reference data set, the method further includes:

[0026] Traverse each first detection data point in the first detection data group, and assign the first detection data points that are not assigned to the fourth group to the fifth group; the fifth group is used to characterize the defects falsely reported by the defect detection system.

[0027] In some embodiments, the first performance index includes at least one of the following: the capture rate of the defect detection system, the ratio of hardware missed detections to algorithm misclassifications, the hardware missed detection rate, the hardware signal capture rate, and the hardware sensitivity index.

[0028] The second performance indicator includes at least one of the following: fragmentation index, false alarm rate, misclassification rate, classification accuracy, and duplicate detection rate of the defect detection system.

[0029] In some embodiments, determining the adjustment direction of the defect detection system based on the first performance index includes:

[0030] The capture rate of the defect detection system is calculated as the first performance index based on the number of reference data points in the third group and the total number of reference data points in the reference data group.

[0031] When the capture rate is lower than a first preset threshold, it is determined that the hardware capture capability of the defect detection system is insufficient, and an instruction is generated to adjust the hardware configuration parameters of the defect detection system.

[0032] In some embodiments, determining the adjustment direction of the defect detection system based on the second performance index includes:

[0033] The fragmentation index of the defect detection system is calculated as the second performance index based on the ratio of the number of first detection data points in the fourth group to the number of reference data points in the third group.

[0034] When the fragmentation index deviates from the preset range, it is determined that the algorithm decision-making ability of the defect detection system is insufficient, and an instruction is generated to adjust the algorithm configuration parameters of the defect detection system.

[0035] In some embodiments, adjusting the hardware configuration parameters of the defect detection system includes at least one of: adjusting optical detection parameters, replacing detection equipment, and calibrating the machine coordinates;

[0036] Adjusting the algorithm configuration parameters of the defect detection system includes at least one of adjusting the detection threshold, modifying the classification model parameters, and adjusting the clustering radius.

[0037] Secondly, one embodiment provides a defect detection system, comprising:

[0038] At least one detection module is used to perform defect detection on the object under test and output a first detection data group and a second detection data group, wherein the baseline data group is the true value data of defect detection, the first detection data group and the second detection data group are the detection results output by the same defect detection process, and the first detection data group is the data retained after being filtered by a detection threshold or classification model, and the second detection data group is the data that has been filtered or downweighted in the detection process.

[0039] A control module, connected to the detection module, includes one or more processors; the processors are configured to execute a performance evaluation method for the defect detection system as described in any embodiment of the first aspect during and / or between detection tasks performed by the detection module, so as to optimize the hardware configuration parameters and / or algorithm configuration parameters of the detection module.

[0040] Thirdly, in one embodiment, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, are used to implement the performance evaluation method of the defect detection system as described in any embodiment of the first aspect.

[0041] The defect detection system and performance evaluation method provided in this application introduce a second detection data group that is filtered or downweighted during the detection process. A baseline data group is then spatially matched with both the first and second detection data groups to obtain a first matching result and a second matching result. Based on these two matching results, each data point in the baseline data group is divided into a first group and a second group. The first group consists of baseline data points that have no matching relationship with either the first or second detection data groups, while the second group consists of baseline data points that have no matching relationship with the first group but a matching relationship with the second group. This subdivides the general "missed detections" in traditional techniques into a first group representing insufficient hardware capabilities and a second group representing inappropriate algorithm parameters, providing a precise decision-making basis for optimizing the detection system. By introducing a second detection data group and establishing a grouping mechanism based on two levels of matching results, this application effectively utilizes low-confidence data discarded in traditional techniques and eliminates the distortion in capture rate calculation caused by simply counting the number of local defect points in one-to-many matching scenarios. This significantly improves the accuracy and optimization efficiency of the detection system performance evaluation.

[0042] In addition, this application also provides a computer-readable storage medium that has the same beneficial effects as the clock error compensation method described above. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 This is a schematic diagram of the structure of a defect detection system provided in one embodiment of this application;

[0045] Figure 2 A flowchart illustrating a performance evaluation method for a defect detection system provided in one embodiment of this application;

[0046] Figure 3 A flowchart illustrating a performance evaluation method for a defect detection system provided in another embodiment of this application;

[0047] Figure 4 A flowchart of a performance evaluation method for a defect detection system provided in another embodiment of this application;

[0048] Figure 5 A flowchart of a performance evaluation method for a defect detection system provided in another embodiment of this application;

[0049] Figure 6 A flowchart of a performance evaluation method for a defect detection system provided in another embodiment of this application;

[0050] Figure 7 A flowchart of a performance evaluation method for a defect detection system provided in another embodiment of this application;

[0051] Figure 8 This is a schematic diagram of the structure of a defect detection system provided in another embodiment of this application.

[0052] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to the present invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0054] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0055] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. Unless otherwise specified, the terms "connection" and "linkage" used in this application include both direct and indirect connections (linkages).

[0056] As described in the background section, in semiconductor wafer defect detection, overlay analysis is commonly used to evaluate or determine the performance of a defect detection system. This involves spatially comparing high-confidence local data output by the current detection equipment with reference baseline data to obtain three statistical results: common defects, locally unique defects, and missed defects. However, traditional overlay analysis only performs a binary comparison between high-confidence local detection data and reference baseline data. This makes it impossible to distinguish whether the hardware failed to capture the signal (optical missed detection) or the algorithm misclassified the signal as noise and filtered it out when the reference defect was not detected (algorithm misclassification). Furthermore, in one-to-many scenarios where a real defect is detected as multiple adjacent defect points, simply counting the number of matching points will distort the capture rate. In other words, the low-confidence data (i.e., downgraded data) generated during the detection process and filtered out by thresholds or classifiers actually contains crucial information about whether the hardware captured the signal, but traditional methods discard it directly.

[0057] To address the aforementioned technical issues, this application proposes to spatially match the high-confidence first detection data group and the filtered or downweighted second detection data group output during the same detection process with the benchmark data group, and to perform cross-validation using priority exclusion logic. This decomposes the traditional single "missed detection" into a first group representing insufficient hardware capabilities and a second group representing inappropriate algorithm parameters. Simultaneously, deduplication processing eliminates statistical interference from one-to-many matching, thereby providing a precise basis for adjusting the hardware or algorithm parameters of the detection system, improving the accuracy of performance evaluation and optimization efficiency.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0059] Figure 1 This is a schematic diagram of the structure of a defect detection system provided in one embodiment of this application. Figure 1 As shown, the defect detection system provided in this embodiment includes at least a detection module 101 and a control module 102.

[0060] In this embodiment, the detection module 101 is used to perform physical scanning and signal acquisition on the object under test (such as a wafer). Its core function is to perform detection actions according to a preset detection strategy (including light source parameters, optical path, sensor gain, scanning speed, etc.) and output the original signal or the defect coordinates and attribute data after preliminary processing. The performance of the detection module 101 directly determines whether the system can effectively capture potential defect signals.

[0061] In practice, the detection module 101 can be a single machine for a specific inspection or a combination of multiple machines using different inspection principles. Each detection module 101 typically includes an optical unit (such as a light source, illumination system, lens group, and filter), a scanning mechanism (such as a stage and motion control system), a signal receiving unit (such as a CCD camera and photoelectric sensor), and a data acquisition circuit. The optical unit generates illumination light of a specific wavelength and intensity and projects it onto the surface of the object under test. The scanning mechanism controls the relative motion between the object under test and the optical system to achieve full coverage. The signal receiving unit converts reflected, scattered, or transmitted light signals into electrical signals. The data acquisition circuit amplifies, filters, and performs analog-to-digital conversion on the electrical signals to generate raw images or grayscale data. This raw data is then transmitted to the control module 102 for further processing. The number and specific composition of the detection modules 101 determine the system's detection throughput, resolution, and sensitivity to different types of defects.

[0062] The control module 102 is responsible for further processing and decision-making on the data output by the detection module 101, including executing image processing algorithms, defect detection algorithms (such as threshold judgment, connected component analysis, and feature extraction), and defect classification models, to filter out detection results considered to be real defects from the original signal, and finally output a high-confidence defect list. In addition, the control module 102 also undertakes functions such as scheduling the detection process, managing detection parameters, and data interaction with external systems, making it a key link in determining the reliability and accuracy of the detection results.

[0063] In this embodiment, in addition to performing conventional defect detection tasks, the detection module 101 and the control module 102 also work together to determine and evaluate the performance of the defect detection system itself.

[0064] In the performance determination and evaluation of the defect detection system, at least one detection module 101 is used to perform defect detection on the object under test, outputting a first detection data set and a second detection data set. The baseline data set consists of the true value data of the defect detection, while the first and second detection data sets are the detection results output from the same defect detection process. The first detection data set consists of data retained after being filtered by a detection threshold or classification model, and the second detection data set consists of data that has been filtered or downweighted during the detection process. In other words, in a single detection process, the detection module 101 not only outputs the high-confidence first detection data set (i.e., the conventional defect results retained after being filtered by a threshold or classification model), but also simultaneously outputs the filtered or downweighted second detection data set (i.e., the low-confidence signals that failed the filtering).

[0065] The control module 102 is configured to acquire the two sets of data mentioned above, as well as a reference data set serving as a benchmark for actual defects. It obtains a first matching result and a second matching result by spatially matching the first and second detection data sets with the benchmark data set, respectively. Based on priority exclusion logic, it groups the data points in the benchmark data set to generate a first group and a second group. The first group refers to benchmark data points that have no matching relationship with either the first or second detection data set, used to characterize missed defects that were not detected due to insufficient hardware capabilities. The second group refers to benchmark data points that have a matching relationship only with the second detection data set but no matching relationship with the first detection data set, used to characterize defects that were incorrectly filtered out due to improper algorithm parameters.

[0066] The control module 102 is further configured to assess whether the hardware configuration parameters or algorithm configuration parameters of the detection module 101 need to be adjusted based on whether the first group and the second group meet the expectations. Adjustments to hardware configuration parameters include changing optical detection parameters, replacing detection equipment, or calibrating the machine coordinates; adjustments to algorithm configuration parameters include modifying the detection threshold, classification model parameters, or cluster radius. The control module 102 is also configured to send adjustment commands to the detection module 101, thereby achieving self-diagnosis and closed-loop optimization of the detection system performance.

[0067] In this way, the detection module 101 and control module 102 not only complete the defect detection task but also establish a collaborative evaluation mechanism for the defect detection system. The detection module 101 simultaneously outputs high-confidence data and filtered low-confidence data in a single detection cycle. The control module 102 introduces a reference benchmark and performs two-level spatial matching and priority grouping, thereby accurately decomposing missed detections into hardware and algorithmic problems, and guiding the adjustment of hardware parameters in the detection module 101 or the optimization of algorithm parameters in the control module 102, respectively. This collaborative approach not only fully utilizes traditionally discarded low-confidence data but also eliminates the interference of fragmented defects on statistical indicators, achieving quantitative diagnosis and closed-loop optimization of the detection system's performance. This significantly improves the accuracy of the overall performance evaluation of the defect detection system and the targeted nature of parameter adjustments.

[0068] The following will further explain how the control module 102 completes the performance evaluation of the defect detection system based on the first and second detection data sets output by the detection module 101.

[0069] Figure 2This is a flowchart illustrating a performance evaluation method for a defect detection system according to one embodiment of this application. The performance evaluation method for the defect detection system provided in this embodiment is applied to the defect detection system provided in the above embodiment. The control module 102 or one or more processors evaluate the performance of the defect detection system during and / or between detection tasks performed by the detection module 101, and determine whether its parameter configuration needs adjustment based on the evaluation results. Figure 2 As shown, the performance evaluation method for the defect detection system provided in this embodiment specifically includes the following steps:

[0070] Step S201: Obtain the baseline data set, the first detection data set, and the second detection data set. The baseline data set consists of the ground truth data for defect detection. The first and second detection data sets are the detection results output from the same defect detection process. The first detection data set contains data retained after being filtered by a detection threshold or classification model, while the second detection data set contains data that was filtered or downweighted during the detection process.

[0071] To evaluate the performance of the defect detection system and differentiate between missed detections due to different causes, the control module 102 first needs to acquire three types of data—a baseline data set, a first detection data set, and a second detection data set—from the detection module 101 as the basis for analysis. The baseline data set consists of real defect data obtained through high-precision detection equipment re-judgment or manual annotation, serving as a reference value for evaluating the detection system's performance. The first and second detection data sets originate from the same defect detection process. The first detection data set comprises high-confidence defect data retained by the detection system after filtering according to preset detection thresholds or classification models, representing the standard output results. The second detection data set consists of low-confidence data that was filtered or downweighted during the same detection process; such data would typically be discarded directly in traditional methods.

[0072] Step S202: Spatial matching is performed between the first detection data group and the second detection data group and the reference data group to obtain the first matching result and the second matching result.

[0073] To establish the spatial correspondence between the baseline data and the detection data, it is necessary to spatially match the first and second detection data sets with the baseline data set, respectively. Spatial matching refers to comparing each detection point with a baseline point based on information such as the coordinate position, geometric shape, or overlapping area of ​​the defect to determine whether they belong to the same physical defect. After matching, the first matching result and the second matching result are obtained. The first matching result records the correspondence between each baseline point in the baseline data set and the detection points in the first detection data set, while the second matching result records the correspondence between each baseline point in the baseline data set and the detection points in the second detection data set.

[0074] Step S203: Based on the first matching result and the second matching result, group the data points in the benchmark data group to obtain a first group and a second group. The first group includes benchmark data points in the benchmark data group that have no matching relationship with either the first or second detection data group; the second group includes benchmark data points in the benchmark data group that have no matching relationship with the first detection data group but have a matching relationship with the second detection data group.

[0075] Based on the above matching results, it is necessary to further subdivide the traditional single "missed detection" into defect types with different physical causes. The classification process requires grouping the data points in the baseline data set. Specifically, for each baseline data point in the baseline data set, the first matching result is checked: if the baseline point has a matching first detection data point in the first matching result, it indicates that the real defect has been detected normally by the detection system and is not included in the first or second group. If the baseline point does not have a match in the first matching result, the second matching result is checked further. If there is also no match in the second matching result, it means that the real defect was neither detected with high confidence nor left any signal in the degraded data; that is, the hardware failed to capture the signal at that location. Such baseline points are classified into the first group. If there is a match in the second matching result, it means that the hardware captured the signal of the defect, but the signal was filtered out because it did not meet the threshold or classifier conditions, indicating a missed detection due to improper algorithm or parameter settings. Such baseline points are classified into the second group. Thus, the first group represents defects that were not captured due to insufficient hardware capabilities, and the second group represents defects that were mistakenly filtered out due to improper algorithm parameters.

[0076] Step S204: If at least one of the grouping results of the first group and the second group does not meet expectations, adjust the configuration parameters of the defect detection system. The configuration parameters include hardware configuration parameters and algorithm configuration parameters.

[0077] Finally, the grouping results are used to determine whether the performance of the detection system meets expectations, and accordingly, to decide whether to adjust the configuration parameters of the defect detection system and in what direction. If data points exist in either the first or second group, or if the amount of data exceeds a preset tolerance threshold, it is considered that the performance does not meet expectations. Specifically, if data points exist in the first group, it indicates a weakness in the hardware capture capability of the detection system, requiring adjustments to hardware configuration parameters, such as increasing optical resolution, adjusting light source intensity, or calibrating the machine coordinates. If data points exist in the second group, it indicates insufficient algorithmic decision-making capability, requiring adjustments to algorithm configuration parameters, such as relaxing the detection threshold, modifying the decision boundary of the classification model, or adjusting the cluster radius. If data points exist in both the first and second groups, adjustments to both hardware and algorithms are necessary simultaneously. This differentiated adjustment strategy identifies performance bottlenecks in the detection system, avoids blind adjustments, and effectively improves the overall detection performance of the defect detection system.

[0078] When the grouping results of the first group do not meet expectations, adjusting the hardware configuration parameters of the defect detection system includes at least one of the following: adjusting optical detection parameters, replacing detection equipment, and calibrating the machine coordinates.

[0079] Specifically, adjusting optical inspection parameters includes changing the light source intensity, wavelength, or illumination angle to improve the ability to capture weak defect signals; replacing inspection equipment includes upgrading from a lower-resolution optical inspection machine to a higher-resolution machine, or switching from bright-field inspection to dark-field inspection to enhance the contrast of specific types of defects; calibrating the machine coordinates corrects the spatial positioning error between the inspection module 101 and the object under test to ensure the spatial accuracy of the inspection results.

[0080] When the grouping results of the second group do not meet expectations, adjusting the algorithm configuration parameters of the defect detection system includes at least one of adjusting the detection threshold, modifying the classification model parameters, and adjusting the clustering radius.

[0081] Specifically, adjusting the detection threshold includes lowering the judgment threshold for grayscale difference or signal-to-noise ratio so that more filtered signals can enter the high-confidence output; modifying the classification model parameters includes adjusting the decision boundaries or feature weights of each category in the defect classifier to reduce the probability of real defects being misclassified as noise; adjusting the cluster radius includes increasing the merging distance of connected component analysis to solve the problem of fragmented defects being split into multiple independent points, or decreasing the cluster radius to avoid different defects being incorrectly merged.

[0082] These specific adjustments enable precise optimization of both hardware capture shortcomings and algorithm decision biases, thereby effectively improving the overall performance of the defect detection system.

[0083] In summary, the performance evaluation method for the defect detection system provided in this embodiment simultaneously acquires a baseline data group, a high-confidence first detection data group, and a filtered or downweighted second detection data group, and establishes spatial matching relationships with the baseline data group for each. Then, based on priority logic, the baseline data points are divided into a first group that does not match either of the two detection data groups and a second group that matches only the second detection data group. This achieves a refined decomposition of the traditional single "missed detection." When the first group does not meet expectations, hardware configuration parameters can be adjusted accordingly; when the second group does not meet expectations, algorithm configuration parameters can be adjusted accordingly. This provides a clear and distinguishable decision-making basis for performance optimization of the detection system. By introducing the traditionally discarded second detection data group, information in low-confidence signals is effectively utilized, avoiding data waste. Furthermore, by performing group statistics based on baseline points rather than local points, the distortion in capture rate calculation caused by a real defect being identified as multiple local defect points is eliminated. Therefore, the accuracy of defect detection system performance evaluation, the ability to locate the root cause of missed detections, and the targeted nature of parameter optimization are improved.

[0084] In some embodiments, in step S202, when spatially matching the first and second detection data groups with the reference data group, the core method is overlay comparison, which involves spatially overlaying defect coordinate layers from different sources and determining whether they belong to the same physical defect by calculating the geometric distance or degree of overlap. Depending on the available data types and accuracy requirements in actual application scenarios, overlay comparison can employ the following different specific implementation methods. Each method can effectively establish the spatial correspondence between the reference point and the detection point and generate matching results for subsequent grouping.

[0085] The first method is matching based on Euclidean distance. This method uses the spatial coordinates of the defect point as a reference, calculating the Euclidean distance between each reference point in the reference data set and each detection point in the first or second detection data set. A matching radius is pre-set (e.g., a few micrometers, determined based on the positioning accuracy of the detection equipment and the defect distribution density). For each reference point, a search is conducted within its radius to determine if a detection point exists. If at least one detection point exists, and the distance between the detection point with the smallest distance and the reference point is less than the radius, then a match is determined, and the index of that detection point is recorded. If no detection point exists within the radius, a mismatch is determined. The matching result can be a mapping table recording the detection points matched by each reference point (which may be empty). This method is simple to implement and fast to calculate, suitable for scenarios with high defect coordinate accuracy and sparse defect distribution, but requires a reasonably set matching radius to balance mismatches and missed matches.

[0086] The second method is matching based on the overlapping area of ​​defects. This approach treats each reference point and detection point as a spatial region with a certain area, rather than just point coordinates. The shape and size of the reference point's region can be extracted from a high-precision re-evaluation image (e.g., the defect's outline or minimum bounding rectangle), while the region of the detection point can be obtained from the attribute information output by the detection device (e.g., the defect's width, height, and area). Then, the overlapping area between the reference and detection regions is calculated, along with the proportion of the overlapping area to the areas of the reference and detection regions, respectively. If the overlapping area proportion exceeds a preset threshold (e.g., 50% of the reference region area and 50% of the detection region area), then a match is determined. This method fully considers the actual geometry of the defect and has a certain tolerance for coordinate positioning errors, making it particularly suitable for large or irregularly shaped defects. However, it requires prior acquisition of the defect's size information, resulting in a relatively large computational load.

[0087] The third method is matching based on image grayscale correlation. This approach is suitable for scenarios where local image data of the defect can be obtained, such as high-resolution scanning electron microscope images corresponding to the baseline data set, and optical or electron beam images acquired by the inspection equipment during inspection. For each reference point in the baseline data set, a fixed-size window image is extracted from the high-resolution image, centered on its coordinates. For each detection point in the first or second inspection data set, a window image of the same size is also extracted from the image acquired by the inspection equipment. Then, the normalized cross-correlation (NCC) coefficient or structural similarity (SSIM) index between the two window images is calculated. If the correlation coefficient exceeds a preset threshold, the two defects are considered highly similar in grayscale distribution and texture features, and a match is determined; otherwise, a match is not made. This method has high matching accuracy and can distinguish between different defects that are similar in shape but close in location, but it requires a large amount of computation and requires the inspection equipment to provide image data of the corresponding location.

[0088] The fourth method is composite matching based on multiple constraints. This approach integrates various matching criteria mentioned above to improve the robustness and accuracy of the matching. For example, firstly, a method based on Euclidean distance is used for rapid screening, selecting a set of candidate detection points whose distance to each reference point is less than a preset radius. Then, for each detection point in the candidate set, the overlap area ratio or image correlation with the reference point is further calculated. Only when all selected constraints (such as distance < radius and overlap area > threshold, or distance < radius and correlation > threshold) are met is a final match determined. Alternatively, a weighted scoring mechanism can be used: weights are assigned to indicators such as distance, overlap area, and correlation, and a comprehensive matching score is calculated. The candidate detection point with the highest score is selected as the matching object; if the highest score is lower than the set matching threshold, no match is determined. This method can adapt to complex detection environments and diverse defect types, minimizing false matches and missed matches. However, its implementation is relatively complex, requiring pre-tuning of the parameters and weights of each constraint condition according to the specific application scenario.

[0089] Figure 3 A flowchart illustrating a performance evaluation method for a defect detection system provided in another embodiment of this application. Figure 3 As shown, in some embodiments, step S203 in the above embodiments, grouping the data points in the reference data group according to the first matching result and the second matching result, includes:

[0090] Step S301: Traverse each reference data point in the reference data group and take the reference data points that do not appear in the first matching result as points to be processed.

[0091] After obtaining the first matching result between the first detection data set and the baseline data set, the first step is to use this result to filter out those defects that were not detected with high confidence by the detection system from all real defects (baseline data set). Specifically, using the baseline data set as a benchmark, for each baseline data point, check whether there is a corresponding first detection data point in the first matching result. If there is, it means that the real defect has been correctly identified and output by the detection system, and such points do not require further processing and are directly excluded from subsequent analysis. If there is no corresponding baseline data point, it is marked as a point to be processed and proceeds to the next stage of judgment. This filtering process reflects a priority logic: high-confidence detection cases are processed first, and only defects that are not detected with high confidence need further analysis to determine the reasons for their missed detection.

[0092] Step S302: Traverse each point to be processed. For each point to be processed, determine whether it appears in the second matching result.

[0093] After the initial screening, the points to be processed represent the real defects that were not output as high-confidence results by the detection system. To further distinguish whether these missed detections are due to insufficient hardware capabilities or improper algorithm parameters, a second set of detection data (i.e., filtered or downweighted low-confidence data) is needed for secondary verification. Specifically, for each point to be processed, it is checked whether there is a corresponding second detection data point in the second matching results. If there is no match in the second matching results, it means that even the low-confidence signal at the defect location was not captured by the hardware, that is, the hardware did not sense any anomaly at all, which points to a deficiency in the hardware's capture capability. If there is a match in the second matching results, it means that the hardware did capture the signal at that location, but the signal was filtered out by the algorithm because it did not meet the threshold or classification model conditions, which points to an improper algorithm decision parameter.

[0094] Step S303: If the point to be processed does not appear in the second matching result, the point to be processed is assigned to the first group; the first group is used to characterize defects that are not captured due to insufficient hardware capabilities.

[0095] Step S304: When the point to be processed appears in the second matching result, the point to be processed is assigned to the second group; the second group is used to characterize defects that are mistakenly filtered due to improper algorithm parameters.

[0096] Based on the judgment results of the second step, the points to be processed are finally categorized. If a point to be processed is neither in the first matching result nor in the second matching result, it is classified into the first group. The benchmark data points in the first group represent those defects where the hardware failed to capture any signal at all, and can be used to evaluate the hardware capture capability of the detection system. If a point to be processed is not in the first matching result but has a match in the second matching result, it is classified into the second group. The benchmark data points in the second group represent those defects where the hardware captured the signal but was filtered out by the algorithm, and can be used to evaluate the algorithm decision-making capability of the detection system. Through this two-step screening and classification mechanism, a refined breakdown of missed defects is achieved, providing a clear data basis for subsequent adjustments to hardware or algorithm parameters.

[0097] Figure 4 A flowchart illustrating a performance evaluation method for a defect detection system provided in another embodiment of this application. Based on any of the above embodiments, such as... Figure 4 As shown, in some embodiments, after grouping the data points in the reference data group according to the first matching result and the second matching result in step S203, the method further includes:

[0098] Step S401: The benchmark data points appearing in the first matching result are assigned to the third group; the third group is used to characterize the defects that were correctly detected.

[0099] After dividing the data points in the baseline data set into the first and second groups, another category of baseline data points needs to be categorized: those baseline points that have already found a matching first detection data point in the first matching result. These baseline points represent real defects that have been captured by the high-confidence output of the detection system, meaning the detection system has correctly detected these defects. They are assigned to the third group to characterize correctly detected defects. The third group can serve as a data source for constructing the first and second performance indicators and is also an important basis for evaluating the overall performance of the detection system.

[0100] Step S402: Traverse each reference data point in the third group and assign the first detection data point that matches it in the first detection data group to the fourth group.

[0101] For each baseline data point in the third group, all first detection data points with spatial matching relationships can be obtained based on the first matching result. Since in actual detection, a real defect may be identified by the detection system as multiple adjacent and independent defect points—for example, a continuous scratch may be split into multiple short scratches for reporting—one baseline point may correspond to multiple first detection data points. These matched first detection data points are collected and grouped into the fourth group. Each point in the fourth group is a reported point actually output by the detection system for a correctly detected real defect; its number is usually greater than or equal to the number of baseline points in the third group, depending on the degree of fragmentation of the detection algorithm.

[0102] Step S403: Deduplication is performed on the first detection data points in the fourth group so that the same first detection data point is recorded only once; the fourth group is used to characterize the reported defects corresponding to the correctly detected defects.

[0103] During the process of traversing the baseline points in the third group and adding the matched first detection data points to the fourth group, it is possible that the same first detection data point may have a matching relationship with multiple baseline points simultaneously. For example, two very close real defects may be merged and identified as a single detection point by the detection system. If deduplication is not performed, this first detection data point will be recorded repeatedly, resulting in an inflated data volume in the fourth group and affecting the accuracy of statistical indicators such as the fragmentation index. Therefore, it is necessary to perform deduplication on the first detection data points in the fourth group. Typically, a hash table is used to record the unique identifier of each first detection data point to ensure that the same point is retained only once. The deduplicated fourth group is used to represent the reported defects corresponding to the correctly detected defects, that is, the set of non-repeating detection points actually output by the detection system for all real defects.

[0104] In some embodiments, after grouping the data points in the reference data group in step S203, the method further includes:

[0105] Step S404: Traverse each first detection data point in the first detection data group and assign the first detection data points that are not assigned to the fourth group to the fifth group; the fifth group is used to characterize the defects falsely reported by the defect detection system.

[0106] After constructing the fourth group, to further identify false alarms (i.e., falsely reported defects) generated by the detection system, it is necessary to further process the unmatched detection points in the first detection data group. Specifically, each first detection data point in the first detection data group is iterated through to check if it has already been assigned to the fourth group. The fourth group contains all first detection data points that match the actual defects (i.e., the baseline points in the third group), representing defects correctly reported by the detection system. If a first detection data point is not assigned to the fourth group, it means that no corresponding actual defect can be found in the baseline data group, i.e., the detection system reported a defect that does not actually exist. These first detection data points not assigned to the fourth group are assigned to the fifth group. The fifth group is directly used to characterize the false alarm defects of the detection system. Its data volume reflects the false alarm rate and is also an important basis for evaluating whether the detection system algorithm parameters are too sensitive. It can serve as a quantitative reference for algorithm adjustments (such as increasing the detection threshold or optimizing the classifier).

[0107] Figure 5 A flowchart illustrating a performance evaluation method for a defect detection system provided in another embodiment of this application. Figure 5 As shown, in some embodiments, step S204, when at least one of the grouping results of the first group and the second group does not meet expectations, involves adjusting the configuration parameters of the defect detection system, including:

[0108] Step S501: Based on the grouping results, construct an evaluation system that includes at least a first performance index and a second performance index, wherein the first performance index characterizes the hardware capture capability of the defect detection system and the second performance index characterizes the algorithm decision capability of the defect detection system.

[0109] After obtaining the first and second groups, an evaluation system needs to be constructed to quantitatively assess the performance of the defect detection system. This evaluation system should include at least two performance indicators: a first performance indicator and a second performance indicator.

[0110] The first performance indicator characterizes the hardware capture capability of the detection system, that is, whether the hardware components of the detection module 101, such as the optical system and sensors, can effectively capture the signals generated by the actual defects. The first performance indicator can be derived from the data in the first group. For example, the number or proportion of reference points in the first group reflects the proportion of defects for which the hardware completely failed to capture the signal.

[0111] The second performance metric characterizes the algorithmic decision-making capability of the detection system, specifically whether the detection threshold, classification model, and other algorithmic components in the control module 102 can correctly identify the signals captured by the hardware as defects. This second performance metric can be derived from the data in the second group; for example, the number or proportion of reference points in the second group reflects the proportion of defects whose signals have been captured by the hardware but filtered out by the algorithm.

[0112] Step S502: Determine the adjustment direction of the defect detection system based on the first performance index and the second performance index.

[0113] After constructing the evaluation system and obtaining the specific values ​​of the first and second performance indicators, it is necessary to determine the optimization direction of the detection system based on these two indicators. Specifically, if the first performance indicator shows insufficient hardware capture capability, such as the data volume of the first group exceeding the preset tolerance threshold or its proportion being too high, it indicates that there is a shortcoming in the current hardware configuration of the detection system, and adjustments need to be made from the hardware side, such as upgrading optical components, adjusting light source parameters, or calibrating mechanical positioning. If the second performance indicator shows insufficient algorithm decision-making capability, such as the data volume of the second group exceeding the preset tolerance threshold or its proportion being too high, it indicates that the current detection system's algorithm parameters are improperly set, and adjustments need to be made from the algorithm side, such as relaxing the detection threshold, optimizing the classification model, or adjusting clustering parameters. If neither performance indicator meets expectations, comprehensive adjustments need to be made from both hardware and algorithm aspects simultaneously. Through this performance indicator-based decision-making mechanism, blind adjustments can be avoided, the performance bottleneck of the detection system can be accurately located, and optimization efficiency and effectiveness can be significantly improved.

[0114] After obtaining five groups of data with clear physical meaning, from the first to the fifth group, a series of quantitative indicators for evaluating the performance of the defect detection system can be constructed based on the quantitative relationships between these groups. For example, the capture rate can be calculated using the total number of the third group and the baseline data group to measure the system's overall ability to detect real defects; the ratio of the number of the first group to the number of the second group can be used to determine the relative severity of hardware missed detections and algorithm misclassifications to identify the main sources of performance bottlenecks; the fragmentation index can be calculated using the ratio of the fourth group to the third group to assess the rationality of the algorithm's clustering or segmentation parameters; and the false alarm rate can be calculated using the total number of the fifth group and the first detection data group to evaluate the system's false alarm level. These indicators complement each other, quantifying the hardware capture capability and algorithm decision-making capability of the detection system from different dimensions, providing clear data basis for subsequent hardware parameter adjustments or algorithm parameter optimization.

[0115] In some embodiments, the first performance metric includes at least one of the following: the capture rate of the defect detection system, the ratio of hardware missed detections to algorithm misclassifications, the hardware missed detection rate, the hardware signal capture rate, and the hardware sensitivity index.

[0116] The capture rate is a fundamental metric for measuring the ability of a defect detection system to detect real defects. It is defined as the proportion of correctly detected real defects to the total number of real defects. In this embodiment, correctly detected real defects correspond to the third group, i.e., the benchmark points in the benchmark data group that have a matching relationship with the first detection data group; all real defects correspond to the total number in the benchmark data group. Therefore, the capture rate can be calculated by dividing the number of benchmark points in the third group by the total number of benchmark points in the benchmark data group. A higher capture rate indicates that the detection system can detect more real defects; a lower capture rate indicates that there are more missed detections, requiring further analysis to determine whether the missed detections are due to insufficient hardware capture or improper algorithm decision-making. As a primary performance indicator, the capture rate directly reflects the combined effect of hardware capture capability and algorithm decision-making capability. However, in this embodiment, it is classified as a primary performance indicator, mainly used to reference the overall detection level in hardware adjustment decisions.

[0117] The ratio of hardware missed detections to algorithm misclassifications is defined as follows: Hardware missed detections correspond to the first group, which consists of reference points in the baseline data group that have no matching relationship with either the first or second detection data groups. These represent defects where signals were completely missed due to insufficient hardware capabilities. Algorithm misclassifications correspond to the second group, which consists of reference points in the baseline data group that have no matching relationship with the first detection data group but have a matching relationship with the second detection data group. These represent defects where signals were incorrectly filtered due to improper algorithm parameters. The ratio of hardware missed detections to algorithm misclassifications is the number of reference points in the first group divided by the number of reference points in the second group (or vice versa, depending on the actual definition). This ratio directly reflects the main source of performance bottlenecks in the detection system: if the ratio is greater than 1, the hardware missed detection problem is more serious than the algorithm misclassification problem, and hardware adjustments should be prioritized; if the ratio is less than 1, the algorithm misclassification problem is more prominent, and algorithm adjustments should be prioritized; if the ratio is close to 1, adjustments to both aspects may be necessary simultaneously.

[0118] The hardware false negative rate is defined as the number of reference data points in the first group divided by the total number of reference data points in the reference data group. The first group represents defects whose signals were not captured at all due to insufficient hardware capabilities; therefore, the hardware false negative rate directly reflects the proportion of real defects missed at the hardware level. A higher rate indicates a significant weakness in the optical unit, sensor, or scanning mechanism of the detection system, which may require hardware upgrades or parameter adjustments. Unlike the capture rate, which is an overall detection rate that combines hardware and algorithm factors, the hardware false negative rate specifically isolates the influence of algorithmic factors and evaluates hardware performance alone.

[0119] The hardware signal capture rate is defined as the proportion of defects whose signals are successfully captured by the hardware out of all real defects. Defects whose signals are successfully captured by the hardware include the third group (defects correctly retained by the algorithm) and the second group (defects incorrectly filtered by the algorithm but whose signals were captured by the hardware). Since both groups of reference points have matches in the second matching results, it indicates that the hardware has sensed the signal. Therefore, the hardware signal capture rate can be calculated by dividing the sum of the number of reference points in the third and second groups by the total number of reference points in the reference data group. This metric reflects the hardware's inherent sensing capability, is unaffected by subsequent algorithmic filtering, and can also serve as an important parameter for evaluating the core performance of the detection module 101.

[0120] The hardware sensitivity index can be defined as the ratio between the sum of the second and third groups and the first group, that is, the ratio of the number of defects whose signals were successfully captured by the hardware to the number of defects whose signals were not captured at all. A larger index indicates stronger hardware capture capability, enabling it to detect the vast majority of real defects; a smaller index indicates severe hardware inadequacy. Compared to the hardware signal capture rate, the hardware sensitivity index, presented as a ratio, more intuitively reflects the strength of the hardware's capture capability, facilitating horizontal comparisons between different detection systems or strategies.

[0121] Among them, the capture rate and the ratio of hardware missed detections to algorithm misclassifications are typical representatives of the first performance index. When constructing the first performance index, one or more of the following can be flexibly selected according to the actual evaluation needs: hardware missed detection rate, hardware signal capture rate, hardware sensitivity index, etc. Combining the capture rate and the ratio of hardware missed detections to algorithm misclassifications, the hardware capture capability of the detection system can be comprehensively evaluated from different dimensions.

[0122] In some embodiments, the second performance metric includes at least one of the following: the fragmentation index of the defect detection system, false alarm rate, misclassification rate, classification accuracy, and duplicate detection rate.

[0123] The fragmentation index measures the degree to which a detection system splits a real defect into multiple reported defect points. It is defined as the number of first detection data points in the fourth group divided by the number of baseline data points in the third group. The third group consists of correctly detected real defects, and the fourth group is the set of first detection data points matching each baseline point in the third group after deduplication. When the fragmentation index equals 1, it indicates that each real defect corresponds exactly to one reported point, a perfect match. When the fragmentation index is greater than 1, it indicates that each real defect is split into multiple reported points on average, suggesting that the cluster radius is too small or the connected component segmentation is too fine, leading to defect fragmentation. When the fragmentation index is less than 1, it indicates that multiple real defects are merged into one reported point, suggesting that the cluster radius is too large or the resolution is insufficient, leading to defect clustering. As a secondary performance indicator, the fragmentation index is specifically used to evaluate the rationality of clustering, segmentation, and merging parameter settings in the detection system algorithm, providing a clear quantitative direction for adjusting algorithm parameters.

[0124] False alarm rate (False Alarm Rate) is an indicator that measures the degree of false alarms in a detection system. It is defined as the number of first detection data points in the fifth group divided by the total number of data points in the first detection data group. The fifth group represents those first detection data points that have no matching relationship with the baseline data group, i.e., false alarms where the detection system reports defects that do not actually exist. A higher False Alarm Rate indicates that the detection system's algorithm parameters are too sensitive, such as an excessively low detection threshold or the classifier misclassifying a large amount of noise as defects, leading to an increase in the workload of ineffective re-judgment. As a secondary performance indicator, the False Alarm Rate can complement the Fragmentation Index, evaluating the rationality of the algorithm's decisions from different dimensions: the Fragmentation Index reflects the fidelity of defect morphology, while the False Alarm Rate reflects the ability to control false alarms.

[0125] The misclassification rate measures the proportion of real defects that an algorithm misclassifies as noise. It is defined as the number of baseline data points in the second group divided by the sum of the second and third groups (i.e., the total number of real defects captured by the hardware). The second group represents defects whose signals were captured by the hardware but filtered out by the algorithm, while the third group represents defects whose signals were captured by the hardware and correctly retained by the algorithm. The misclassification rate reflects the accuracy of the algorithm in distinguishing between signals and noise. A higher misclassification rate indicates that the algorithm's decision boundaries are too strict or that the classification model's discriminative power is insufficient. Unlike the fragmentation index, the misclassification rate focuses on whether the algorithm incorrectly discards real signals, while the fragmentation index focuses on whether the algorithm's segmentation of defect morphology is reasonable.

[0126] For detection systems with defect classification capabilities, classification accuracy can also be used as a secondary performance indicator. Calculating classification accuracy requires the baseline data set to contain the category label (e.g., scratches, particles, voids, etc.) for each real defect, while the first detection data set contains the defect categories output by the detection system. For each correctly detected baseline point in the third group, its category label is compared to the category output by the corresponding first detection data point. The classification accuracy is the number of correctly classified defects divided by the total number of baseline points in the third group. Classification accuracy reflects the algorithm's ability to identify defect types; a low classification accuracy indicates that the classification model needs retraining or feature parameter adjustment.

[0127] The duplicate detection rate measures the tendency for the same real defect to be reported multiple times as multiple independent defect points. It is related to but distinct from the fragmentation index. The fragmentation index calculates the average number of fragments for all split defects, while the duplicate detection rate can be defined as the difference between the number of the first detection data points in the fourth group and the number of baseline points in the third group, divided by the number of baseline points in the third group—that is, the average number of extra points reported for each real defect. A higher duplicate detection rate indicates that the clustering algorithm or connected component segmentation parameters are set too finely, resulting in a large number of redundant reports and increasing the burden on subsequent review and data analysis. This indicator can serve as a supplement to the fragmentation index, providing a more intuitive reflection of reporting efficiency.

[0128] Among them, the fragmentation index is a typical representative of the second performance index. When constructing the second performance index, one or more of the following can be flexibly selected based on the specific application scenario and optimization goal, such as false alarm rate, misclassification rate, classification accuracy, and duplicate detection rate, to comprehensively evaluate different aspects of the detection system algorithm's decision-making ability.

[0129] After determining the index types of the first and second performance indicators and completing the construction of the evaluation system including the first and second performance indicators, the next step is to proceed to step S502, which determines the adjustment direction of the defect detection system based on the first and second performance indicators.

[0130] Figure 6 A flowchart illustrating a performance evaluation method for a defect detection system provided in another embodiment of this application. Based on any of the above embodiments, such as... Figure 6 As shown, in some embodiments, the adjustment direction of the defect detection system is determined based on a first performance index, including:

[0131] Step S601: Calculate the capture rate of the defect detection system as the first performance index based on the number of reference data points in the third group and the total number of reference data points in the reference data group.

[0132] In this embodiment, after obtaining the third group (the set of correctly detected real defects) and the baseline data group (the set of all real defects), the capture rate of the defect detection system is selected as the first performance indicator to quantify the overall ability of the detection system to detect real defects. The capture rate is calculated by dividing the number of baseline data points in the third group (the set of correctly detected real defects) by the total number of baseline data points in the baseline data group (the set of all real defects). The third group represents the real defects that the detection system successfully captured with high confidence, while the baseline data group represents all real defects that should have been detected. A higher capture rate indicates that the detection system can detect more real defects; a lower capture rate indicates more missed detections.

[0133] Step S602: When the capture rate is lower than the first preset threshold, it is determined that the hardware capture capability of the defect detection system is insufficient, and an instruction is generated to indicate that the hardware configuration parameters of the defect detection system are adjusted.

[0134] After calculating the capture rate, it needs to be compared with a pre-set first threshold. This first threshold is an acceptable minimum capture rate set according to production process requirements or product yield targets, such as 95% or 98%. If the capture rate is lower than this threshold, it indicates that the detection system has failed to detect a sufficient proportion of actual defects, resulting in an overall high false negative rate. A low capture rate is usually directly related to the signal acquisition capabilities at the hardware level, such as insufficient optical resolution, inadequate light source intensity, or low sensor sensitivity. Therefore, when the capture rate does not meet expectations, it can be determined that the system's hardware capture capability is insufficient. Simultaneously, instructions are generated to adjust hardware configuration parameters, including adjusting optical detection parameters, replacing detection equipment, or calibrating machine coordinates, to improve the hardware's ability to capture defect signals, thereby improving the overall capture rate.

[0135] It's important to note that when the primary performance metric includes multiple indicators such as capture rate, hardware miss rate, hardware signal capture rate, and the ratio of hardware misses to algorithm misclassifications, determining whether the detection system's hardware capture capability is insufficient requires a comprehensive consideration of the different dimensions of information reflected by these indicators. For example, a low capture rate may stem from insufficient hardware capabilities, but it could also be due to excessive filtering by the algorithm. In such cases, it's necessary to combine the hardware miss rate (the proportion of the first group) and the hardware signal capture rate (the sum of the proportions of the second and third groups) for differentiation: if the hardware miss rate is high while the hardware signal capture rate is low, then the hardware capture capability does indeed have a weakness; if the hardware signal capture rate is normal but the capture rate is low, the problem is more likely in the algorithm's decision-making stage. Simultaneously, the ratio of hardware misses to algorithm misclassifications can help determine the severity of the hardware problem relative to the algorithm problem. Only by comprehensively evaluating these indicators can we accurately determine whether the hardware capture capability is insufficient, avoiding misjudgments due to the one-sidedness of a single indicator, thus providing a reliable basis for hardware parameter adjustments.

[0136] Figure 7 A flowchart illustrating a performance evaluation method for a defect detection system provided in another embodiment of this application. Based on any of the above embodiments, such as... Figure 7 As shown, in some embodiments, the adjustment direction of the defect detection system is determined based on a second performance index, including:

[0137] Step S701: Calculate the fragmentation index of the defect detection system as the second performance index based on the ratio of the number of first detection data points in the fourth group to the number of reference data points in the third group.

[0138] In this embodiment, after obtaining the third group (the set of correctly detected real defects) and the fourth group (the set of first detection data points that match these real defects and have undergone deduplication), the fragmentation index of the defect detection system is selected as the second performance indicator to quantify the accuracy of the detection system in restoring the defect morphology. The fragmentation index is calculated by dividing the number of first detection data points in the fourth group by the number of baseline data points in the third group. The fragmentation index reflects how many independent detection points are reported by the detection system for each real defect on average. When the fragmentation index equals 1, it indicates that each real defect corresponds to exactly one reported point, and the morphology restoration is ideal. When the fragmentation index is greater than 1, it indicates the existence of fragmentation, that is, a real defect is split into multiple reported points, usually indicating that the cluster radius is too small or the connected component segmentation is too fine. When the fragmentation index is less than 1, it indicates the existence of clustering, that is, multiple real defects are merged into one reported point, usually indicating that the cluster radius is too large or the resolution is insufficient. It can serve as an important indicator for evaluating the rationality of the clustering, segmentation, and merging parameter settings in the detection system algorithm.

[0139] Step S702: When the fragmentation index deviates from the preset range, it is determined that the algorithm decision-making ability of the defect detection system is insufficient, and an instruction is generated to indicate the adjustment of the algorithm configuration parameters of the defect detection system.

[0140] After calculating the fragmentation index, it's necessary to determine if it deviates from a preset range. The preset range typically sets an allowable deviation interval centered at 1, for example, 0.9 to 1.1, with the specific value determined based on actual process requirements. If the fragmentation index exceeds the upper limit of this range (i.e., much greater than 1), it indicates a serious fragmentation detection problem in the detection system; the algorithm is overly sensitive, splitting the same defect into multiple independent points, leading to redundant reporting. If the fragmentation index is below the lower limit of this range (i.e., much less than 1), it indicates a clustering detection problem in the detection system; the algorithm is too coarse, merging different defects into a single point, potentially causing errors in defect classification and root cause analysis. Regardless of the deviation, it indicates insufficient algorithmic decision-making capability in the detection system, requiring adjustment of algorithm configuration parameters. Therefore, after determining that the defect detection system's algorithmic decision-making capability is insufficient, instructions will be generated simultaneously to adjust the algorithm configuration parameters of the defect detection system, including adjusting the clustering radius, modifying connected component segmentation parameters, and optimizing the merging threshold, to bring the fragmentation index back to the preset range, thereby improving the detection system's accuracy in reconstructing defect morphology and its reporting efficiency.

[0141] It should be noted that when the second performance metric includes multiple indicators such as fragmentation index, false alarm rate, misclassification rate, and classification accuracy, determining whether the detection system's algorithmic decision-making capability is insufficient requires a comprehensive consideration of the different types of algorithmic defects revealed by these indicators. For example, a fragmentation index deviating from the preset range indicates inappropriate clustering or segmentation parameters; an excessively high false alarm rate indicates that the detection threshold is too low or the classifier is too sensitive; an excessively high misclassification rate indicates that the algorithm misclassifies a large proportion of real defects as noise. These indicators may exhibit abnormalities simultaneously or may mutually restrict each other (for example, lowering the threshold may reduce the misclassification rate but may increase the false alarm rate). Therefore, it is necessary to perform weighted or priority judgments based on the degree of abnormality of each indicator and the tolerance of the actual process for missed detections and false alarms in order to accurately pinpoint the specific manifestations of insufficient algorithmic decision-making capability and to adjust algorithm configuration parameters such as detection threshold, classification model parameters, or cluster radius accordingly to achieve overall performance optimization.

[0142] In summary, the defect detection system performance evaluation method provided in any of the above embodiments introduces traditionally discarded low-confidence detection data as a second detection data group, and performs two-level spatial matching and priority exclusive grouping with the high-confidence first detection data group and the benchmark data group. This refines the traditional single "missed detection" into a first group representing insufficient hardware capture capability and a second group representing inappropriate algorithm judgment capability. Simultaneously, the third, fourth, and fifth groups are used to quantify correctly detected defects, reported defects, and false alarm defects, respectively, thereby constructing multi-dimensional performance indicators such as capture rate, the ratio of hardware missed detections to algorithm misclassifications, fragmentation index, and false alarm rate. This method not only effectively utilizes the wasted data resources in the original system and eliminates the statistical distortion of the capture rate caused by defect fragmentation or clustering, but also achieves accurate positioning and differentiation of hardware shortcomings and algorithm defects in the detection system. It provides clear quantitative basis for targeted hardware parameter adjustments or algorithm parameter optimization, significantly improving the accuracy of defect detection system performance evaluation, the ability to diagnose the root causes of missed detections, and the efficiency of parameter optimization. This forms a complete technical solution from data acquisition, group diagnosis, indicator quantification to closed-loop adjustment.

[0143] Figure 8 This is a schematic diagram of the structure of a defect detection system provided in another embodiment of this application. Figure 8 As shown, the defect detection system provided in this embodiment includes at least one detection module 801 and a control module 802.

[0144] In this embodiment, the detection module 801 is used to perform defect detection on the object under test and output a first detection data group and a second detection data group. The baseline data group is the true value data of defect detection, the first detection data group and the second detection data group are the detection results output by the same defect detection process, and the first detection data group is the data retained after being filtered by the detection threshold or classification model, and the second detection data group is the data that has been filtered or downweighted in the detection process.

[0145] The control module 802 is connected to the detection module 801 and includes one or more processors. The processors are configured to perform a process of performance evaluation of the defect detection system as described in any of the above embodiments during the execution of detection tasks by the detection module 801 and / or between detection tasks, so as to optimize the hardware configuration parameters and / or algorithm configuration parameters of the detection module 801.

[0146] It should be noted that the processor of the defect detection system provided in this embodiment implements each process of any embodiment of the performance evaluation method for the defect detection system in the same way and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0147] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of any embodiment of the performance evaluation method of the above-described defect detection system and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0148] The processor can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0149] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, firmware, or any combination thereof. Furthermore, as those skilled in the art will understand, the principles herein can be reflected in a computer program product on a computer-readable storage medium pre-loaded with computer-readable program code. Any tangible, non-transitory computer-readable storage medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CDs, DVDs, Blu-ray discs, etc.), flash memory, and / or the like. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to form a machine, such that instructions executing on the computer or other programmable data processing apparatus can generate means for performing a specified function. These computer program instructions can also be stored in a computer-readable storage medium that can instruct the computer or other programmable data processing apparatus to operate in a particular manner, such that instructions stored in the computer-readable storage medium can form an article of manufacture, including means for implementing the specified function. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to perform a series of operational steps on the computer or other programmable apparatus to produce a computer-implemented process, such that instructions executing on the computer or other programmable apparatus can provide steps for implementing the specified function.

[0150] This document describes various exemplary embodiments with reference to them. However, those skilled in the art will recognize that changes and modifications can be made to the exemplary embodiments without departing from the scope of this document. For example, various operational steps and components for performing operational steps can be implemented in different ways depending on the specific application or considering any number of cost functions associated with the operation of the system (e.g., one or more steps can be deleted, modified, or combined with other steps).

[0151] While the principles herein have been illustrated in various embodiments, numerous modifications to the structures, arrangements, proportions, elements, materials, and components, particularly suited to specific environments and operational requirements, may be used without departing from the principles and scope of this disclosure. These modifications and other alterations or alterations will be included within the scope of this document. Those skilled in the art will recognize that many changes can be made to the details of the above embodiments without departing from the fundamental principles of the invention.

Claims

1. A method of performance evaluation of a defect detection system, characterized in that, include: Obtain a baseline data set, a first detection data set, and a second detection data set; wherein, the baseline data set is the true value data of defect detection, the first detection data set and the second detection data set are the detection results output from the same defect detection process, and the first detection data set is the data retained after being filtered by a detection threshold or classification model, and the second detection data set is the data that has been filtered or downweighted during the detection process; The first and second detection data groups are spatially matched with the reference data group to obtain the first matching result and the second matching result. Based on the first matching result and the second matching result, the data points in the reference data group are grouped to obtain a first group and a second group; wherein, the first group includes reference data points in the reference data group that have no matching relationship with either the first detection data group or the second detection data group; the second group includes reference data points in the reference data group that have no matching relationship with the first detection data group but have a matching relationship with the second detection data group. If at least one of the grouping results in the first group and the second group fails to meet expectations, the configuration parameters of the defect detection system are adjusted. The configuration parameters include hardware configuration parameters and algorithm configuration parameters.

2. The performance evaluation method of a defect detection system according to claim 1, wherein When at least one of the grouping results in the first group and the second group fails to meet expectations, the configuration parameters of the defect detection system are adjusted, including: Based on the grouping results, an evaluation system is constructed that includes at least a first performance index and a second performance index, wherein the first performance index characterizes the hardware capture capability of the defect detection system, and the second performance index characterizes the algorithm decision capability of the defect detection system. The adjustment direction of the defect detection system is determined based on the first performance index and the second performance index.

3. The performance evaluation method of a defect detection system according to claim 2, wherein The step of grouping the data points in the baseline data group according to the first matching result and the second matching result includes: Traverse each benchmark data point in the benchmark data group and treat the benchmark data points that do not appear in the first matching result as points to be processed. Iterate through each of the points to be processed. For each point to be processed: If the point to be processed does not appear in the second matching result, the point to be processed is assigned to the first group; the first group is used to characterize defects that are not captured due to insufficient hardware capabilities. When the point to be processed appears in the second matching result, the point to be processed is assigned to the second group; the second group is used to characterize defects that are mistakenly filtered due to improper algorithm parameters.

4. The performance evaluation method of a defect detection system according to claim 3, wherein After grouping the data points in the reference data set, the method further includes: The benchmark data points appearing in the first matching result are assigned to a third group; the third group is used to characterize the defects that were correctly detected. Traverse each reference data point in the third group and assign the first detection data point that matches it in the first detection data group to the fourth group; The first detection data points within the fourth group are deduplicated so that the same first detection data point is recorded only once; the fourth group is used to characterize the reported defects corresponding to the correctly detected defects.

5. The performance evaluation method of a defect detection system according to claim 4, wherein After grouping the data points in the reference data set, the method further includes: Traverse each first detection data point in the first detection data group, and assign the first detection data points that are not assigned to the fourth group to the fifth group; the fifth group is used to characterize the defects falsely reported by the defect detection system.

6. The performance evaluation method of a defect detection system according to claim 5, wherein The first performance index includes at least one of the following: the capture rate of the defect detection system, the ratio of hardware missed detections to algorithm misclassifications, the hardware missed detection rate, the hardware signal capture rate, and the hardware sensitivity index. The second performance indicator includes at least one of the following: fragmentation index, false alarm rate, misclassification rate, classification accuracy, and duplicate detection rate of the defect detection system.

7. The performance evaluation method of a defect detection system according to claim 5, wherein Based on the first performance index, the adjustment direction of the defect detection system is determined, including: The capture rate of the defect detection system is calculated as the first performance index based on the number of reference data points in the third group and the total number of reference data points in the reference data group. When the capture rate is lower than a first preset threshold, it is determined that the hardware capture capability of the defect detection system is insufficient, and an instruction is generated to adjust the hardware configuration parameters of the defect detection system.

8. The performance evaluation method of a defect detection system according to claim 5, wherein Based on the second performance index, the adjustment direction of the defect detection system is determined, including: The fragmentation index of the defect detection system is calculated as the second performance index based on the ratio of the number of first detection data points in the fourth group to the number of reference data points in the third group. When the fragmentation index deviates from the preset range, it is determined that the algorithm decision-making ability of the defect detection system is insufficient, and an instruction is generated to adjust the algorithm configuration parameters of the defect detection system.

9. The method of performance evaluation of a defect detection system according to claim 1, wherein, Adjusting the hardware configuration parameters of the defect detection system includes at least one of the following: adjusting optical detection parameters, replacing detection equipment, and calibrating the machine coordinates; Adjusting the algorithm configuration parameters of the defect detection system includes at least one of adjusting the detection threshold, modifying the classification model parameters, and adjusting the clustering radius.

10. A defect detection system, characterized in that, include: At least one detection module is used to perform defect detection on the object under test and output a first detection data group and a second detection data group, wherein the baseline data group is the true value data of defect detection, the first detection data group and the second detection data group are the detection results output by the same defect detection process, and the first detection data group is the data retained after being filtered by a detection threshold or classification model, and the second detection data group is the data that has been filtered or downweighted in the detection process. A control module, connected to the detection module, includes one or more processors; the processors are configured to execute the performance evaluation method of the defect detection system as described in any one of claims 1 to 9 during and / or between detection tasks performed by the detection module, so as to optimize the hardware configuration parameters and / or algorithm configuration parameters of the detection module.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program or instructions, which, when executed by a processor, are used to implement the performance evaluation method of the defect detection system as described in any one of claims 1 to 9.

Images