AOI difficult example sample identification method and device based on multi-source uncertainty

Abstract

The embodiment of the application provides an AOI difficult example sample identification method and device based on multi-source uncertainty, which comprises the following steps: obtaining multi-source data of an AOI production line site, including sample images, AI model outputs, traditional rule results and engineering context parameters, respectively calculating a plurality of risk scores capable of fully representing sample values, including uncertainty scores, classification boundary scores, divergence scores, distribution outside scores, business risk scores and noise risk scores; weighting and fusing the above scores to obtain a comprehensive difficult example score, and combining a preset access condition to screen out high-value samples and include them in a difficult example sample pool, performing deduplication, clustering and diversity selection on the samples in the pool to generate a re-judgment task list, and finally distributing the samples to a training pool, a customer arbitration pool, an engineering problem pool or a discard pool according to the re-judgment conclusion, so that the AOI production line site data difficult example sample identification efficiency and accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a method and apparatus for identifying difficult AOI samples based on multi-source uncertainty. Background Technology

[0002] In AOI (Automated Optical Inspection) production lines, the scale of on-site data is usually far greater than the scale of annotable data. Uploading and annotating all on-site images would significantly increase storage, bandwidth, labor, and management costs. Relying solely on human experience to select false positives and false negatives can easily lead to the omission of low-confidence samples, boundary samples, samples that conflict between rules and models, samples with divergent models, and novel defect samples outside the distribution.

[0003] Difficult example re-entry refers to the process in engineering inspection where samples prone to model errors are categorized as difficult examples and re-entered into a training sample pool. The model is then retrained based on the training set in this pool. Existing difficult example re-entry methods typically rely on model error results, lacking a unified consideration of model uncertainty, engineering risks, customer business risks, and image noise risks. For AOI scenarios, some abnormal samples originate from camera defocus, light source drift, motion blur, coordinate mapping errors, recipe configuration changes, or equipment malfunctions, making them unsuitable for direct addition to the defect model training set.

[0004] Therefore, there is an urgent need for a method for identifying difficult AOI samples based on multi-source uncertainty, which can improve the efficiency and accuracy of difficult sample identification in AOI production line field data. Summary of the Invention

[0005] To address the problems in the existing technology, this application provides a method and apparatus for identifying difficult AOI samples based on multi-source uncertainty, which can improve the efficiency and accuracy of difficult sample identification in AOI production line field data.

[0006] To solve at least one of the above problems, this application provides the following technical solution: Firstly, this application provides a method for identifying difficult AOI samples based on multi-source uncertainty, including: Acquire multi-source data from the AOI production line, including field samples, model output results, rule output results, and the engineering context of the field samples; Risk scores for field samples are calculated based on the multi-source data. These risk scores include model uncertainty score, classification boundary score, multi-source divergence score, out-of-distribution score, business risk score, and noise risk score. The calculated risk scores are then weighted and summed to determine the corresponding comprehensive difficult case score. Based on the risk scores, the comprehensive difficult case score, and the preset difficult case sample admission criteria, field samples that meet the difficult case sample admission criteria are included in the preset difficult case sample pool. The difficult sample in the difficult sample pool is deduplicated, clustered, and subjected to diversity selection operations to determine the corresponding re-judgment task list. The difficult sample in the re-judgment task list is re-judged and verified to determine the corresponding re-judgment conclusion. According to the re-judgment conclusion corresponding to each difficult sample, each difficult sample is assigned to the corresponding data processing pool. The data processing pool includes a training pool, a customer arbitration pool, an engineering problem pool, and a discard pool.

[0007] Furthermore, the step of calculating the risk scores of the on-site samples based on the multi-source data includes: The corresponding model uncertainty score is determined based on the information entropy of the category probability distribution of the model output. The corresponding classification boundary score is determined based on the difference between the probability of the largest class and the probability of the second largest class in the model output results. Based on the output difference between the model output and the rule output, the corresponding multi-source divergence score is determined, wherein the model includes at least one of the following: classification model, detection model, segmentation model, anomaly detection model, teacher model, and student model. The corresponding out-of-distribution score is determined based on the distance between the visual feature vector of the on-site sample and the center of the known defect category feature space. Based on the severity of defects in the field samples, the level of customer concern, the cost of missed inspections, the project stage, the customer's on-site verification status, and historical complaint records, the corresponding business risk score is determined. Based on the image quality sub-score and engineering anomaly sub-score of the field samples, the corresponding noise risk score is determined.

[0008] Further, the step of including on-site samples that meet the difficulty sample admission criteria into the preset difficulty sample pool based on the risk scores, the overall sample difficulty score, and the preset difficulty sample admission criteria includes: Pre-defined criteria for difficult sample admission include comprehensive score admission criteria, mandatory admission criteria for individual risks, and admission criteria based on confirmed facts. Determine whether the comprehensive difficulty score of the on-site sample meets the comprehensive score admission criteria. If it does, the on-site sample that meets the criteria is included in the preset difficult sample pool. Determine whether each risk score of the on-site sample meets the mandatory admission criteria for the single risk. If it does, the on-site sample that meets the criteria will be included in the preset difficult sample pool. Determine whether there are confirmed difficult case fact labels in the field samples. If so, include the field samples that meet the conditions into the preset difficult case sample pool.

[0009] Furthermore, the step of performing deduplication, clustering, and diversity selection operations on the difficult sample samples in the difficult sample pool to determine the corresponding re-judgment task list includes: Perform complete duplication detection and near-duplication clustering on the difficult sample samples in the difficult sample pool to determine the corresponding duplication sample groups. At least one representative sample is retained in each duplication sample group to form a set of candidate samples. Under the preset review budget constraint, diversity optimization selection is performed on the candidate sample set to determine the corresponding review task list.

[0010] Furthermore, the step of performing diversity optimization selection on the candidate sample set under a preset review budget constraint to determine the corresponding review task list includes: Obtain the available review budget within the current review period, the review budget including at least one of the maximum number of review samples, the maximum review working hours, and the review quota for each category; Under the constraint of the review budget, a diversity optimization algorithm is used to select representative samples from the candidate sample set to determine the corresponding review task list. The objective function of the diversity optimization algorithm is set as: to maximize the total information gain of the final selected sample set without exceeding the review budget.

[0011] Furthermore, the step of reviewing and verifying the difficult examples in the review task list to determine the corresponding review conclusion includes: The difficult cases in the review task list are assigned to the corresponding review channels for review and verification. The review channels include manual review channels, customer arbitration channels, and engineering verification channels. The corresponding review conclusions are obtained from each review channel. The review conclusions include OK tags, NG tags, missed detection tags, false alarm tags, disputed tags, new category tags, and engineering problem tags.

[0012] Further, the step of allocating each difficult case sample to a corresponding data processing pool based on the re-judgment conclusion corresponding to each difficult case sample includes: If the re-judgment conclusion of the difficult sample is NG label, missed detection label, or new class label, then the sample is assigned to the training pool; If the retrial conclusion of the difficult case sample is a dispute label, then the sample will be assigned to the customer arbitration pool; If the re-judgment conclusion of the difficult example sample is labeled as an engineering problem, then the sample is assigned to the engineering problem pool; Samples that do not receive a valid reassessment and whose business risk scores are below the preset risk threshold will be assigned to the discard pool.

[0013] Secondly, this application provides an AOI hard example sample identification device based on multi-source uncertainty, comprising: The data acquisition module is used to acquire multi-source data from the AOI production line, including field samples, model output results, rule output results, and the engineering context of the field samples. The difficult case sample scoring module is used to calculate the risk score of the field sample based on the multi-source data. The risk score includes model uncertainty score, classification boundary score, multi-source divergence score, out-of-distribution score, business risk score, and noise risk score. The module then performs a weighted sum of the calculated risk scores to determine the corresponding comprehensive difficult case score. Based on the risk scores, the comprehensive difficult case score, and the preset difficult case sample admission criteria, the field samples that meet the difficult case sample admission criteria are included in the preset difficult case sample pool. The difficult example sample diversion module is used to perform deduplication, clustering, and diversity selection operations on the difficult example samples in the difficult example sample pool, determine the corresponding re-judgment task list, perform re-judgment verification on the difficult example samples in the re-judgment task list, determine the corresponding re-judgment conclusion, and allocate each difficult example sample to the corresponding data processing pool according to the re-judgment conclusion of each difficult example sample. The data processing pool includes a training pool, a customer arbitration pool, an engineering problem pool, and a discard pool.

[0014] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the AOI difficult sample identification method based on multi-source uncertainty.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the AOI difficult sample identification method based on multi-source uncertainty.

[0016] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the AOI difficult sample identification method based on multi-source uncertainty.

[0017] As can be seen from the above technical solution, this application provides a method and apparatus for identifying difficult AOI samples based on multi-source uncertainty. By acquiring multi-source data from the AOI production line, including sample images, AI model outputs, traditional rule results, and engineering context parameters, multiple risk scores that can comprehensively characterize the sample value are calculated, including uncertainty score, classification boundary score, divergence score, out-of-distribution score, business risk score, and noise risk score. The above scores are weighted and fused to obtain a comprehensive difficult sample score. High-value samples are selected and included in the difficult sample pool based on preset admission criteria. The samples in the pool are deduplicated, clustered, and subjected to diversity selection to generate a re-judgment task list. After re-judgment verification, the samples are finally allocated to the training pool, customer arbitration pool, engineering problem pool, or discard pool according to the re-judgment conclusion. This can improve the efficiency and accuracy of difficult sample identification from AOI production line data. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the AOI difficult example sample identification method based on multi-source uncertainty in the embodiments of this application; Figure 2 This is a structural diagram of the AOI difficult sample identification device based on multi-source uncertainty in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0022] Considering that the criteria for reverting difficult samples in AOI (Automated Optical Inspection) production lines are too singular, and not all reverted samples are suitable for model training, leading to contamination of the model training set, this application provides a method and apparatus for identifying difficult samples in AOI based on multi-source uncertainty. By acquiring multi-source data from the AOI production line, including sample images, AI model outputs, traditional rule results, and engineering context parameters, multiple risk scores that comprehensively characterize the sample value are calculated, including uncertainty score, classification boundary score, divergence score, out-of-distribution score, business risk score, and noise risk score. These scores are weighted and fused to obtain a comprehensive difficult sample score. High-value samples are then selected and included in a difficult sample pool based on preset admission criteria. The samples in the pool undergo deduplication, clustering, and diversity selection to generate a re-judgment task list. After re-judgment verification, the samples are finally allocated to a training pool, customer arbitration pool, engineering problem pool, or discard pool based on the re-judgment conclusion. This improves the efficiency and accuracy of difficult sample identification from AOI production line data.

[0023] To improve the efficiency and accuracy of difficult example identification in AOI production line data, this application provides an embodiment of an AOI difficult example identification method based on multi-source uncertainty, see [link to embodiment]. Figure 1 The method for identifying difficult AOI samples based on multi-source uncertainty specifically includes the following: Step S101: Obtain multi-source data from the AOI production line site, including site samples, model output results, rule output results, and the engineering context of site samples; Optionally, in this embodiment, this step is the data entry point for the entire difficult sample identification method. It is responsible for collecting all the raw data required for subsequent scoring and decision-making from the AOI production line inspection system, and performing consistency verification and preprocessing on the collected data to ensure that the data used in subsequent steps correspond to each other in time, space and semantic dimensions.

[0024] AOI (Automated Optical Inspection) is an automated technology that uses image acquisition equipment and image processing algorithms to detect defects on the surface of industrial products. It is widely used in online quality inspection in the manufacturing processes of PCBs, panels, semiconductor packaging, etc.

[0025] On-site samples refer to image data that is collected in real time by AOI equipment during production line operation and identified as "suspected defects". Each sample includes a cropped ROI (Region of Interest) image of the defective area, the spatial coordinates of the sample on the entire board or piece of product, and information about the product batch to which it belongs.

[0026] Model output refers to the inference results of a deep learning model deployed on an AOI device or edge computing unit for a given sample. Output formats include: probability vectors for each category output by the classification model (e.g., OK probability 0.3, NG probability 0.7), coordinates and confidence scores of defect bounding boxes output by the object detection model, and pixel-level defect region masks output by the segmentation model.

[0027] The rule output refers to the detection conclusion given by a manually written deterministic judgment logic for a sample, which does not rely on a machine learning model. Rules are usually implemented based on traditional image processing methods such as image grayscale statistics, edge gradients, connected component area, and geometric shape matching. The output can be a binary judgment of NG / OK, or a continuous score of the degree of suspicion.

[0028] Engineering context refers to the equipment parameters, environmental configurations, and production information related to the sample collection and testing process, including but not limited to: Recipe version (test recipe version number), Camera Profile, Light Profile, Node Profile, Mapping (product coordinate mapping relationship), ModelVersion, Lot (production batch number), Job (work order number), GlassId or WaferId (substrate or wafer number), ScanPath, Alarm (equipment alarm record), and Log (system log).

[0029] In this embodiment, this step is performed according to the following process: 1. Once the AOI device completes online inspection of a sample, the system automatically triggers the data acquisition process for this step. Triggering conditions can be an event indicating the end of the inspection process, a manual review event triggered by a sample being deemed suspicious, or a scheduled event for batch data retrieval.

[0030] 2. Read the raw image data of the sample from the local cache or distributed storage system of the AOI equipment. The read content includes: the ROI cropped image of the defect area, the X / Y coordinates of the defect in the substrate coordinate system, the product batch number (Lot), job number (Job), substrate or wafer number (GlassId / WaferId), and the currently used scan path parameters (ScanPath). Simultaneously, read the equipment alarm records (Alarm) and system logs associated with this sample for subsequent engineering anomaly detection.

[0031] 3. Based on the current sample's identifier, the system reads the output of at least one AI model for that sample from the model inference result storage area.

[0032] 4. Read the judgment results of one or more traditional rule algorithms for the sample from the result storage area of ​​the rule engine.

[0033] 5. Based on the device ID or testing station identifier to which the sample belongs, the system retrieves the engineering context parameters corresponding to the sample at the time of acquisition from the engineering parameter database. Specific fields include at least: RecipeVersion (the version number of the currently used testing recipe), CameraProfile (camera exposure, gain, and other configurations), LightProfile (light source brightness, angle, and other configurations), ModelVersion (the version number of the AI ​​model), and Mapping (a mapping table from the defect coordinates of the current product to the physical location of the entire board). These parameters are jointly indexed using the device ID and acquisition timestamp to ensure that the retrieved engineering parameters are consistent with the actual working conditions at the time of sample acquisition.

[0034] 6. Perform consistency checks and alignment operations on the multi-source data collected above, specifically including: Field Existence Validation: Check whether each data source contains the key fields required in the above steps. For records with missing key fields, mark them with the "Field Missing" exception code, record the name of the missing field, and mark the sample as pending completion. Subsequent steps will determine whether to continue processing based on the completion status.

[0035] Value range validity check: Check whether the values ​​of each field are within a reasonable range. For example: whether the sum of the components of the category probability vector is equal to 1 within the tolerance range; whether the detection box coordinates are within the image size boundaries; whether the RecipeVersion field matches the recipe list configured for the current device.

[0036] Timestamp alignment: Check the difference between the timestamp of the model output, the timestamp of the rule output, and the timestamp of the sample image acquisition. If the difference exceeds the preset time window threshold, mark it with the "time misalignment" exception code, indicating that the model or rule result of this sample may not be based on the current sample image, and this sample will not enter the active learning sampling process in subsequent processing.

[0037] Sample identifier consistency verification: Confirm whether the sample ID, ROI coordinates, and substrate number recorded in each data source are consistent. If inconsistencies are found, mark the sample with an "identifier mismatch" exception code, and the sample will be added to the exception record pool and will not participate in subsequent scoring calculations.

[0038] 7. After the above verification and alignment, the system assembles the multi-source data into a unified standardized data structure.

[0039] This step of data acquisition and processing ensures that all data used in subsequent steps to calculate various scores are consistent and traceable across the sample, time, and operating condition dimensions.

[0040] Step S102: Calculate the risk scores of the field samples based on the multi-source data. The risk scores include model uncertainty score, classification boundary score, multi-source divergence score, out-of-distribution score, business risk score, and noise risk score. Then, perform a weighted summation of the multiple risk scores obtained after calculation to determine the corresponding comprehensive difficult case score. Based on the risk scores, the comprehensive difficult case score, and the preset difficult case sample admission criteria, the field samples that meet the difficult case sample admission criteria are included in the preset difficult case sample pool. Optionally, in this embodiment, this step performs multi-dimensional quantitative evaluation on each sample collected on-site, and selects difficult samples worthy of further processing based on the evaluation results, and categorizes them into a unified candidate pool to await subsequent processes.

[0041] Difficult sample refers to a sample that the model has difficulty in accurately identifying. This sample is used for subsequent manual annotation and backflow to further train the model.

[0042] Specifically, the model uncertainty score measures the stability of the AI ​​model's output for the current sample. When the model gives similar probabilities for multiple categories, it indicates that the model's judgment of the sample is highly ambiguous, and the uncertainty is high; conversely, when the model favors a certain category with a very high probability, the uncertainty is low. This score is obtained by reading the probability of each category output by the AI ​​model for the sample, substituting it into the information entropy calculation formula, and normalizing the entropy value to the interval of 0 to 1.

[0043] Specifically, the classification boundary score measures how close a sample's position in the model's decision space is to the classification boundary. The smaller the difference between the highest and second-highest probabilities in the class probability distribution, the closer the sample is to the decision interface that distinguishes different classes. Such samples are highly valuable for improving the classifier's boundary accuracy. This score is calculated by reading the highest and second-highest class probabilities output by the AI ​​model, calculating the difference between them, and subtracting this difference from 1 to obtain the boundary score. The smaller the difference, the higher the boundary score.

[0044] Specifically, the multi-source divergence score measures the degree of inconsistency between the judgments of different sources on the same sample. Judgment sources include, but are not limited to, AI models, traditional rule-based algorithms, and detection results under different lighting conditions. The greater the difference between the outputs of each source, the more likely the sample is to have a problem or possess special learning value. This score is calculated by collecting the judgments of multiple sources on the same sample, comparing them pairwise or calculating the consistency of the judgments as a whole; the greater the difference, the higher the divergence score.

[0045] Specifically, the out-of-distribution score measures the degree of deviation of a sample's features from the feature distributions of each category in the existing training dataset. When a sample is far from the feature centers of all known categories in the feature space, it indicates that the sample belongs to a data form that the model has never seen before, possibly corresponding to a new defect type or a new process state. This score is calculated by extracting the visual feature vector of the sample, calculating the Mahalanobis distance or cosine distance between this vector and the feature center of each category in the training set, and taking the minimum distance as the basis for the out-of-distribution score; the larger the distance, the higher the score.

[0046] Specifically, the business risk score quantifies the importance of a sample to the customer's business. The higher the severity of the defect, the closer the customer's attention to that type of defect, and the more serious the consequences of a missed defect, the higher the business risk score. The business risk score is calculated by comprehensively looking up the table or using a weighted average, based on the defect type's level in a pre-configured risk registration table, the customer's level of attention to that defect type, the current stage of the project, and the customer's historical complaint records.

[0047] Specifically, the noise risk score quantifies the degree of anomalies in the sample image quality or acquisition process. This includes issues such as image defocus, overexposure, underexposure, motion blur, image corruption, and engineering-level problems such as equipment malfunctions, light source drift, and coordinate mapping errors. The score is calculated by reading the image quality detection results (sharpness, brightness, integrity) and engineering context data (equipment alarm logs, light source parameters, coordinate mapping verification results) corresponding to the sample, obtaining image quality sub-scores and engineering anomaly sub-scores respectively. These two sub-scores are then combined to obtain the noise risk score.

[0048] Before calculation, the scores are normalized to a uniform numerical range. The corresponding weight coefficients are assigned to the risk scores of the six dimensions. The scores are multiplied by their corresponding weights and then summed to obtain the overall difficulty score of the sample.

[0049] For example, for sample x, the hardest example score can be expressed as: S(x)=λ1U(x)+λ2M(x)+λ3D(x)+λ4O(x)+λ5R(x)-λ6N(x) Where U(x) is the model uncertainty score; M(x) is the classification boundary score; D(x) is the multi-source divergence score; O(x) is the out-of-distribution score; R(x) is the business risk score; N(x) is the noise risk score; and λ1-λ6 are weighting coefficients.

[0050] The weighting coefficients are configured as follows: In the initial stage of system deployment, initial values ​​are set according to business priorities—business risk scores and out-of-distribution scores are assigned higher weights, while noise risk scores are assigned negative weights or deducted during fusion; In actual operation, the system records the correlation statistics between the scores of each dimension and the actual re-judgment conclusions, and adjusts the weights periodically accordingly.

[0051] After quantitatively evaluating all on-site samples, a determination is made based on preset admission criteria to determine whether the current sample should enter the difficult sample pool. The admission criteria employ a multi-path parallel decision logic: Comprehensive score path: When the comprehensive difficulty score of a sample reaches or exceeds a preset threshold, the sample is granted admission.

[0052] Single mandatory path: When a sample’s multi-source divergence score, out-of-distribution score, or business risk score reaches its corresponding mandatory threshold, the sample is granted admission even if the overall score does not meet the standard.

[0053] External fact path: When a sample is accompanied by customer confirmation results, historical manual review records, or formal conclusions from the quality system that existed before the current process began, the sample is directly granted access.

[0054] Meanwhile, the system performs engineering quality gating checks. If the image quality sub-score or engineering anomaly sub-score in the noise risk score of a sample exceeds the engineering anomaly threshold, the sample is marked as an engineering anomaly. Although it is included in the difficult sample pool, it does not participate in active learning sampling in the subsequent sampling process, but instead enters the engineering problem processing channel first.

[0055] Once the criteria are met, all samples that meet the admission criteria, along with their corresponding scores for each dimension, admission reasons, timestamps, and metadata version information, are stored in a pre-defined difficult sample pool.

[0056] Through the above steps, the system can automatically select samples with training value, business importance, or anomaly analysis value from the massive amount of field samples continuously produced by the production line and enter them into the candidate pool. It can also eliminate duplicate, low-value, and engineering anomaly samples that may interfere with subsequent processes, thus providing a data foundation for subsequent review tasks and model iterations.

[0057] Step S103: Perform deduplication, clustering, and diversity selection operations on the difficult sample samples in the difficult sample sample pool to determine the corresponding re-judgment task list. Perform re-judgment verification on the difficult sample samples in the re-judgment task list to determine the corresponding re-judgment conclusion. Based on the re-judgment conclusion corresponding to each difficult sample, allocate each difficult sample to the corresponding data processing pool. The data processing pool includes a training pool, a customer arbitration pool, an engineering problem pool, and a discard pool.

[0058] Optionally, in this embodiment, this step involves pre-splitting, manually processing, and selecting the flow direction for difficult sample pools.

[0059] Specifically, re-inspection verification refers to a second inspection of samples in the re-inspection task list by human quality inspectors, engineers, or customers to confirm the true defect category of the sample, whether the label is correct, whether the equipment status is abnormal, etc., and form a traceable confirmation conclusion.

[0060] Specifically, since manual verification requires a lot of effort, the difficult sample samples in the difficult sample pool are first pre-sorted, including deduplication, clustering, and diversity selection, and valuable difficult sample samples are selected to enter the re-judgment and verification stage.

[0061] The deduplication operation includes complete duplicate removal. The system compares the file hash value, overall image hash value, or ROI region hash value of the samples. When all verification fields are consistent, only one sample record is retained, and the remaining records are removed from the hard sample pool. At the same time, the index information and duplicate count of the removed samples are retained for subsequent statistics.

[0062] After deduplication, the system clusters the remaining samples. First, it stratifies the samples by product model, recipe version, or initial defect classification to ensure that samples under similar conditions are grouped together. Then, within each stratum, it calculates the distance using the samples' visual feature vectors and uses a clustering algorithm to group samples that are close to each other in the feature space into the same cluster. After clustering, the system records the cluster number of each sample and the distance of that sample within its cluster to the cluster center, serving as a representative basis for subsequent diversity selection.

[0063] The purpose of diversity selection is to choose the sample subset with the broadest coverage and greatest information gain from various clusters within a given review budget. The system first calculates the selection value for each candidate sample, which comprehensively considers the sample's overall difficulty score, business risk score, out-of-distribution degree, representativeness of its cluster, and feature differences from already selected samples. Then, samples are selected sequentially in descending order of value. After each selection, the coverage of the selected sample set is updated. Selection stops when the coverage of the selected samples in the overall feature space reaches saturation or the budget limit is reached. For samples with extremely high business risk, extremely high out-of-distribution degree, or extremely severe multi-source divergence, the system is not subject to clustering quota restrictions and directly includes them in the review task list.

[0064] After the above three operations, the selected samples form a review task list. Each record in the list includes the sample number, scores for each dimension, preliminary flow indications, selection reason code, and an index of the evidence package required for review, so that the reviewers can understand the basis for the selection of the sample and the key points that need to be focused on.

[0065] Next, the review task list is distributed to either the manual reviewer or the client confirmation panel. Reviewers verify each sample on the list, examining supporting evidence such as the original image, ROI region, model output, and rule output, and provide a definitive conclusion for each sample. Conclusion types include OK, NG, missed detection, false detection, label dispute, engineering anomaly, and new category pending confirmation. All review conclusions are recorded and linked to the reviewer's identifier and review timestamp.

[0066] Based on the definitive conclusions obtained from the review and verification, the system allocates samples to the corresponding data processing pools according to preset mapping rules. The mapping rules are as follows: confirmed missed samples, confirmed boundary samples, and confirmed new class samples enter the training pool for subsequent model iteration training; samples where customer conclusions are inconsistent with internal conclusions enter the customer arbitration pool for further confirmation; abnormal samples caused by image quality, equipment status, or engineering configuration enter the engineering problem pool for engineers to diagnose the root cause; samples that are completely duplicated with no new information, have low business risk, and zero coverage gain enter the discard pool or low-cost archiving pool.

[0067] This step reduces the waste of review resources by deduplicating duplicate samples. Through clustering and diversity selection, representative samples covering different defect types and working conditions are retained within the limited review budget. After obtaining credible labels through review verification, the samples are diverted to the corresponding data processing pools according to their respective attributes. This ultimately achieves effective screening from a large number of candidate samples to high-value review tasks, as well as reasonable diversion of sample ownership after review.

[0068] This example demonstrates how this embodiment comprehensively utilizes multi-source uncertainty data to identify and triage difficult cases from AOI production line samples, effectively improving the accuracy of difficult case identification.

[0069] As described above, the AOI difficult sample identification method based on multi-source uncertainty provided in this application can acquire multi-source data from the AOI production line, including sample images, AI model outputs, traditional rule results, and engineering context parameters. It can then calculate multiple risk scores that comprehensively characterize the sample value, including uncertainty score, classification boundary score, divergence score, out-of-distribution score, business risk score, and noise risk score. These scores are weighted and fused to obtain a comprehensive difficult sample score. High-value samples are then selected and included in the difficult sample pool based on preset admission criteria. The pool is then deduplicated, clustered, and diversity-selected to generate a review task list. After review and verification, the samples are ultimately allocated to the training pool, customer arbitration pool, engineering problem pool, or discard pool based on the review conclusion. This improves the efficiency and accuracy of difficult sample identification from AOI production line data.

[0070] In one embodiment of the AOI difficult example sample identification method based on multi-source uncertainty in this application, it may further include the following: Step S201: Determine the corresponding model uncertainty score based on the information entropy of the category probability distribution of the model output results; Step S202: Determine the corresponding classification boundary score based on the difference between the probability of the largest class and the probability of the second largest class in the model output results; Step S203: Determine the corresponding multi-source divergence score based on the output difference between the model output and the rule output, wherein the model includes at least one of the following: classification model, detection model, segmentation model, anomaly detection model, teacher model, and student model; Step S204: Determine the corresponding out-of-distribution score based on the distance between the visual feature vector of the on-site sample and the center of the known defect category feature space; Step S205: Determine the corresponding business risk score based on the severity of sample defects, customer concern level, cost of missed detection, project stage, customer on-site verification status, and historical complaint records of the field samples; Step S206: Determine the corresponding noise risk score based on the image quality sub-score and engineering anomaly sub-score of the on-site sample.

[0071] Optionally, in this embodiment, model uncertainty and classification boundary score are discussed in this step.

[0072] 1. Suppose the model outputs K class probabilities p1, p2, ..., p... K Uncertainty can be represented by normalized entropy:

[0073] 2. When the probabilities of multiple classes are close, the uncertainty is high. Let the maximum class probability be p. top1 The probability of the second largest category is p. top2 The classification boundary score can be expressed as: M(x) = 1 - (p top1 -p top2 ) When the probabilities of the largest and second-largest categories are close, it indicates that the sample is near the model's decision boundary and is suitable for boundary reinforcement training.

[0074] Optionally, in this embodiment, the multi-source divergence score and out-of-distribution detection are performed in this step.

[0075] 3. The system can simultaneously read the outputs of traditional rules, CNN / ViT classification models, detection models, segmentation models, anomaly detection models, teacher models, and student models. When there are significant differences among multiple decision sources, the sample has high difficult example value.

[0076] For judgment sources with the same output format (e.g., multiple classification models all output category probabilities), the system calculates the degree of dispersion between each output. The higher the degree of dispersion, the higher the divergence score.

[0077] For judgment sources with different output formats (e.g., the classification model outputs "NG" while the rule algorithm outputs "OK", or the detection model detects a defect while the segmentation model does not), the system will convert the different output formats into judgment conclusions before comparison. The more sources with inconsistent conclusions, the higher the divergence score.

[0078] When comparing the teacher model and the student model, we focus on the difference in their outputs on the same input. The teacher model has higher accuracy, while the student model has faster inference speed after compression, but its accuracy may decrease. When there is a significant deviation between the two outputs, it indicates that the sample is in a region where the student model has not yet learned sufficiently, corresponding to a higher divergence score.

[0079] 4. For out-of-distribution detection, the system maps samples to the feature space and compares their distances to the feature centers of known classes. When a sample deviates from all known class distributions, the system marks it as an out-of-distribution sample and sends it to the new class discovery pool or the engineering attribution process.

[0080] The known category features are obtained by extracting visual feature vectors from all samples of each known defect type in the historical training set during the model training phase. The out-of-distribution score is calculated by comparing the sample's feature vector with the feature center of each known category and calculating the distance between them. Euclidean distance or cosine distance can be used as the distance metric. The closest distance among all categories is taken as the baseline distance for the sample. Simultaneously, the system also calculates the deviation of the sample's feature vector from the overall feature distribution of the training set, including the sample's local density in the feature space (the number of surrounding neighboring samples) and global outlier status. After normalizing the distance and deviation values, a larger distance and lower local density result in a higher out-of-distribution score. For samples with high out-of-distribution scores, the system prioritizes labeling them as potential new category candidates rather than classifying them into existing defect types.

[0081] Optionally, in this embodiment, business risk and noise risk are integrated in this step.

[0082] 5. Business risk is used to prioritize critical defects, defects of high customer concern, and samples with high failure rates. Its purpose is to quantify the importance of samples from a business and customer perspective, ensuring that samples with high business value receive higher priority in subsequent sampling processes.

[0083] Read multiple input fields when calculating the business risk score: The system includes several key performance indicators (KPIs): Defect severity is categorized into pre-defined severity levels (e.g., critical, severe, minor, catastrophic) based on defect size, depth, location, and impact on product functionality. Customer concern level is determined by varying levels of concern different customers have for different defect types. The system maintains a customer concern level configuration table, allowing users to retrieve the corresponding concern level by customer ID and defect type ID. The cost of missed detection is defined as the potential subsequent losses incurred when a defect goes undetected, including rework costs, scrap costs, and potential customer claims. The project stage is considered, indicating whether the production line is in trial production, ramp-up, or mass production phases, as different stages require different defect detection rates. Customer on-site verification status indicates whether the defect type has been formally confirmed at the customer's site; confirmed defect types correspond to higher business confidence. Historical complaint records indicate whether the customer has previously filed complaints due to missed detection of similar defects; defect types with complaint records are assigned a higher risk weight.

[0084] After mapping the values ​​of each field to a unified numerical range, a weighted sum is calculated according to pre-configured weights to obtain a business risk score. A higher score indicates that the potential business consequences of a missed detection or misjudgment of the sample are more severe, and it should be prioritized in subsequent processes.

[0085] 6. Noise risk is used to reduce the training priority of out-of-focus, overexposed, motion-blurred, corrupted, and disputed-label samples. Understandably, when samples have quality issues during image acquisition or the engineering process, these issues render the samples unsuitable for direct model training and should have their training priority reduced or be addressed through engineering investigation. The overall noise risk N can be broken down into two sub-scores: image quality sub-score N. q With engineering anomaly sub-fraction N e .

[0086] N(x) = α q ·N q (x)+α e ·N e (x) Where, α q and α e The weighting coefficients are adjustable.

[0087] Image quality sub-fraction N q The system reads the sharpness evaluation indicators (such as sharpness score based on gradient energy), brightness histogram distribution (to determine whether there is overexposure or underexposure), and image integrity verification results (whether there are dropped frames, screen tearing, or truncation). The greater the deviation of the above indicators from the normal range, the higher the image quality sub-score.

[0088] Engineering anomaly sub-fraction N eThe system reads the engineering context data corresponding to the sample, including the device status register record (whether there are alarms or out-of-tolerance errors), the deviation between the current value and the standard value of the light source parameters, the coordinate mapping verification result (whether the sample position matches the device coordinates), and whether the Recipe version is consistent with the current product configuration. If any of the above items is abnormal, the engineering abnormality sub-score will increase accordingly.

[0089] By leveraging the opposing effects of business risk and noise risk, the system can simultaneously focus on business value and data quality.

[0090] Through step S206, this embodiment has completed a quantitative evaluation of each sample from six independent dimensions, covering the main factors to be considered in sample screening from different perspectives, and providing numerical basis for subsequent comprehensive scoring and admission determination in each dimension.

[0091] In one embodiment of the AOI difficult example sample identification method based on multi-source uncertainty in this application, it may further include the following: Step S301: Preset the admission conditions for difficult case samples, including the admission conditions for comprehensive scores, the mandatory admission conditions for individual risks, and the admission conditions for confirmed facts; Step S302: Determine whether the comprehensive difficulty score of the on-site sample meets the comprehensive score admission criteria. If it does, include the on-site sample that meets the criteria into the preset difficult sample pool. Step S303: Determine whether each risk score of the on-site sample meets the mandatory admission criteria for the single risk. If it does, the on-site sample that meets the criteria is included in the preset difficult sample pool. Step S304: Determine whether there are confirmed difficult case fact labels in the field samples. If so, include the field samples that meet the conditions into the preset difficult case sample pool.

[0092] Optionally, in this embodiment, this step defines the rules for determining whether a sample enters the difficult sample pool, and filters the on-site samples one by one according to the rules. The three admission conditions are related by "OR", that is, a sample can be included in the difficult sample pool as long as it meets any one of the conditions.

[0093] Specifically, the overall score admission criteria include a threshold for the overall difficult case score. This threshold is determined as follows: During the system deployment phase, the distribution of the overall difficult case score is replayed using confirmed missed, false, and general samples from historical batches. Based on the target recall rate (e.g., requiring the recall rate of known missed samples to reach a specific percentage), the corresponding score quantile is selected as the initial threshold. After the system is running, the threshold is periodically calibrated based on the actual re-judgment results.

[0094] Specifically, the mandatory admission criteria for individual risks include three independent thresholds, corresponding to the multi-source divergence score, the out-of-distribution score, and the business risk score, respectively. Mandatory admission is triggered when any score exceeds its corresponding threshold. The individual thresholds are calibrated in the same way as the overall score thresholds, based on historical batch playback. The purpose of this condition is to capture samples with low overall scores but extremely prominent individual risks. For example, when the model makes an incorrect judgment with high confidence, the overall score may be low, but the divergence score may be high.

[0095] Specifically, the confirmed fact-based admission criteria do not include numerical thresholds, but rather a set of judgment rules used to identify whether a sample carries a credible conclusion that already existed before the current processing flow was initiated. Data sources for these criteria include customer feedback systems, historical review record databases, and quality management systems.

[0096] Optionally, in this embodiment, a comprehensive score admission determination step is included.

[0097] Read the overall difficulty score of the current sample and compare it with the preset overall score threshold. If the overall difficulty score is greater than or equal to the threshold, the sample is determined to meet the overall score admission criteria. The sample identifier, overall difficulty score, risk scores of each dimension, judgment time, and current model version number of the sample are written into the difficult sample pool. If the score is lower than the threshold, the admission of this path is not triggered, and the subsequent judgment steps are continued.

[0098] Optionally, in this embodiment, there is a single-risk mandatory access determination step.

[0099] The system reads the risk scores of the current sample, including the multi-source divergence score, the out-of-distribution score, and the business risk score, and compares them with the corresponding preset mandatory thresholds. If the multi-source divergence score, the out-of-distribution score, or the business risk score reaches or exceeds its mandatory threshold, the sample is determined to meet the mandatory admission condition for a single risk. The system writes the sample's information into the difficult case sample pool and marks the specific score item that triggered admission in the record (if multiple scores exceed the threshold simultaneously, all are recorded). If all three scores are below their respective mandatory thresholds, admission for that path is not triggered, and the system proceeds to the next step.

[0100] Optionally, in this embodiment, the factual access determination step has been confirmed.

[0101] The system checks whether the current sample has a confirmed difficult case fact tag. The query is performed on an external data source, including but not limited to: customer complaint records associated with the sample's image ID or batch number; records in historical review tasks marked as "missed detection" or "false positive" with valid conclusions; and defect conclusions formally confirmed by the quality management system. If at least one matching record is found and its validity period has not expired (customer complaints and review conclusions usually have expiration dates, and conclusions may become invalid due to process changes after the expiration date), the sample is determined to have a confirmed difficult case fact tag. The system writes the sample's information into the difficult case sample pool, recording the fact source and confirmation time. If no matching record is found or the matching record has expired, the sample does not meet the admission criteria for this path.

[0102] Through step S304, this embodiment uses three independent paths to screen samples in parallel, which ensures that common difficult cases with high comprehensive scores are included, and also covers marginal cases with low comprehensive scores but prominent individual risks or existing external confirmation conclusions. This reduces the probability of omissions caused by relying solely on a single comprehensive score, and allows engineers to independently adjust and optimize the thresholds of each path based on the actual review results.

[0103] In one embodiment of the AOI difficult example sample identification method based on multi-source uncertainty in this application, it may further include the following: Step S401: Perform complete duplication detection and near duplication clustering on the difficult sample in the difficult sample pool to determine the corresponding duplication sample group. At least one representative sample is retained in each duplication sample group to form a set of candidate samples. Step S402: Under the preset review budget constraint, perform diversity optimization selection on the candidate sample set to determine the corresponding review task list.

[0104] Optionally, in this embodiment, for complete duplication detection, the source image file and ROI region image of each sample in the hard sample pool are read, and the MD5 or SHA checksum of the file, the perceptual hash value of the whole image, and the difference hash value of the ROI region are calculated respectively. Samples with completely identical checksums are marked as completely duplicate samples and grouped into the same completely duplicate group.

[0105] Optionally, in this embodiment, near-repetition clustering is performed on the remaining samples that were not marked as complete repetitions. The system extracts the visual feature vector of each sample, which can be obtained from the output of the penultimate layer of the classification model, the output of the ROI pooling layer of the detection model, or a dedicated embedding model. Cosine similarity is calculated between each pair of feature vectors of all samples. When the similarity exceeds a preset threshold, the corresponding samples are grouped into the same near-repetition group. Optional clustering methods include the density-based DBSCAN method or the hierarchical clustering method based on hierarchical agglomeration. The clustering radius or distance threshold is calibrated by replay experiments of historical samples.

[0106] After each duplicate sample group is formed, the system selects the top-ranked sample within the group, prioritizing it in descending order of overall difficulty score, image quality score, and metadata completeness. If the number of samples within a group is large and there are internal differences in feature distribution, two to three representative samples can be selected to ensure coverage. The representative sample is retained, while the remaining samples within the group are removed from the difficult sample pool and transferred to the duplicate archive record. The index information, source group number, and duplicate count of the removed samples are retained. All representative samples are then aggregated to form the candidate sample set.

[0107] Optionally, in this embodiment, the system performs diversity optimization selection on the candidate sample set. After the selection is completed, the selected samples form a review task list according to their respective source groups, selection order, and priority. Each record in the list includes a sample number, scores for each dimension, the number of the corresponding duplicate group and cluster group, the selection reason code, and the evidence package index required for review. Samples that are not selected are retained in the difficult sample pool and marked as deferred candidates for re-participation in subsequent batches.

[0108] Through step S402, this embodiment successfully selects a sample set, ensuring that the distribution of samples in the feature space is sufficiently dispersed to cover different defect types and working conditions, thereby maximizing the amount of information valuable for model iteration.

[0109] In one embodiment of the AOI difficult example sample identification method based on multi-source uncertainty in this application, it may further include the following: Step S501: Obtain the available review budget within the current review period, wherein the review budget includes at least one of the maximum number of review samples, the maximum review working hours, and the review quota for each category; Step S502: Under the constraint of the review budget, a diversity optimization algorithm is used to select representative samples from the candidate sample set and determine the corresponding review task list. The objective function of the diversity optimization algorithm is set as: maximizing the total information gain of the final selected sample set without exceeding the review budget.

[0110] Optionally, in this embodiment, a diversity optimization algorithm is used to select a subset of samples under limited resources. The core idea is to maximize the overall benefit while ensuring that the selected samples cover different defect morphologies, operating conditions, and weak areas of the model.

[0111] Information gain measures the amount of effective information added to the system after a sample is included in the re-judgment set. Samples with high information gain typically have one or more of the following characteristics: they are far away from the selected samples in the feature space, they are out-of-distribution samples, they correspond to a region of high uncertainty in the model, or they have high business risk.

[0112] The total information gain is the sum of the information gains of each sample in the final selected set of re-judged samples. The optimization objective of the diversity optimization algorithm is to maximize this sum under budget constraints.

[0113] Optionally, in this embodiment, at the beginning of each review cycle, pre-configured or dynamically calculated budget parameters are read. The maximum number of review samples is determined by the number of currently available review personnel, the average processing capacity per person per cycle, and the cycle duration; the system directly reads this value as a hard upper limit. The maximum review time is calculated as the total working hours of all available review personnel within this cycle. Some samples, due to their complex defect morphology or the need for cross-light source comparison, take longer to review than ordinary samples; the system needs to accumulate and verify the estimated working hours during subsequent sample selection. Review quotas for each category are dynamically allocated by the system based on business priorities or pre-set by management personnel; for example, higher quotas are allocated to product models that customers focus on, and minimum quotas are allocated to sample categories with high distribution rates. After obtaining at least one of the above budget constraints, the system uses them as input conditions for subsequent sample selection steps.

[0114] Optionally, in this embodiment, a diversity optimization algorithm is performed on the candidate sample set. All samples in the candidate set are traversed, and the minimum feature distance between each sample and the currently selected sample set is calculated. This distance is based on the sample's visual feature vector. Simultaneously, the basic information gain of each sample is calculated. The basic information gain is jointly determined by the comprehensive hard case score, business risk score, out-of-distribution score, and the sample's representativeness within the cluster. The greater the difference between a sample and the selected set, and the higher its basic information gain, the higher the priority for selection.

[0115] The selection process follows a greedy strategy, iteratively choosing the sample with the largest information gain increment from the unselected candidates in each round. After addition, the coverage of the selected sample set is updated, and the marginal gain of each remaining candidate sample relative to the updated selected set is recalculated. This process continues until any of the following termination conditions are met: the total number of selected samples reaches the maximum number of samples to be reviewed; the cumulative estimated review time reaches the maximum review time limit; the review quota for a certain category is exhausted and all remaining candidate samples belong to that category; or the marginal gain of the remaining candidate samples relative to the current selected set is lower than a set threshold.

[0116] For example, an implementable sampling value function is as follows: V(x|A)=a·S(x)+b·R(x)+c·O(x)+d·Rep(x)+e·Cov(x|A) f·N q (x) g·Cost(x) Where A is the current set of candidate samples, Rep represents the representativeness of the sample in its cluster, Cov represents the coverage gained after adding the sample, and Cost represents the re-judgment cost. The algorithm selects the sample with the largest V from the remaining candidates each time, until the re-judgment budget for this round is reached.

[0117] In practice, for samples whose business risk scores, out-of-distribution scores, or multi-source divergence scores exceed the mandatory retention threshold, the algorithm relaxes at least some of the constraints in the aforementioned budget constraints during the sampling process. Specifically, when such high-risk samples exist, the algorithm first selects a preset number of these samples from the candidate pool and then includes them in the optimized candidate set. The remaining samples are selected using a diversity optimization algorithm within the remaining budget. Finally, the selected sample set output by the algorithm constitutes the current review task list. Each sample in the list is accompanied by its selection reason code and estimated review time, for reference during review task scheduling.

[0118] Through step S502, this embodiment enables the system to clearly define the upper limit of available manual review resources before the start of each review cycle, avoiding the generation of tasks exceeding processing capacity. It also uses a diversity optimization algorithm to screen samples under budget constraints, ensuring that the final selected review task list covers as many defect types, operating conditions, and weak areas of the model as possible within limited resources. This avoids a large number of high-scoring but homogeneous samples crowding out review resources. At the same time, a forced retention mechanism ensures that high-risk samples are not screened out due to budget constraints.

[0119] In one embodiment of the AOI difficult example sample identification method based on multi-source uncertainty in this application, it may further include the following: Step S601: Assign the difficult case samples in the review task list to the corresponding review channels for review and verification, wherein the review channels include manual review channels, customer arbitration channels and engineering verification channels; Step S602: Obtain the corresponding review conclusion according to each review channel. The review conclusion includes OK label, NG label, missed detection label, false alarm label, dispute label, new category label and engineering problem label.

[0120] Optionally, in this embodiment, each sample record in the review task list is accompanied by a preliminary flow indication field, which is generated during the active learning diversity selection stage and is used to indicate which party is more suitable to review the sample.

[0121] Read the initial flow direction prompts and, based on the various dimensions of the sample's scoring characteristics, assign the sample to the corresponding review channel: Manual review channel: This channel accepts samples with high overall difficulty scores, located near classification boundaries, exhibiting significant multi-source discrepancies but not yet reaching the engineering anomaly threshold, and not involving customer business disputes. These samples require qualified quality inspectors to review the images and evidence packages before providing a labeling assessment.

[0122] Customer Arbitration Channel: This channel receives samples with high business risk scores, those involving defects of key concern to the customer, or those that conflict with the customer's historical judgments. The judgment result of these samples directly affects the customer's acceptance or the determination of quality standards, and requires final adjudication by the customer's representative or quality liaison.

[0123] Engineering Verification Channel: Receives samples where the image quality sub-score or engineering anomaly sub-score reaches the engineering anomaly threshold, or where there are anomaly records in the device logs or Recipe configuration. These samples require diagnosis by equipment engineers or process engineers to determine whether the root cause of the anomaly stems from the acquisition link, optical system, coordinate mapping, or Recipe configuration error.

[0124] After allocation, each channel is processed independently and in parallel. The reviewers for each channel can only view the samples allocated to their channel and their corresponding evidence packages.

[0125] Optionally, in this embodiment, after each review channel completes verification, the system obtains a structured review conclusion from each channel. The conclusion types and corresponding judgment criteria are as follows: The OK label indicates that the reviewer has confirmed that the sample has no defects and that the original judgment by the model was correct. The NG label indicates that the sample has defects, and the original judgment by the model was correct. The label was missed, and the reviewer confirmed that the sample had defects, but the original judgment output of the model or rule was OK or not detected. False positive labeling occurs when the reviewer confirms that the sample has no defects, but the original judgment output of the model or rule is NG. Disputed labels include inconsistencies in conclusions among different reviewers, or inconsistencies between the reviewers' and the system's initial judgment. The new category label indicates that the reviewer confirmed the sample had a defect, but the defect did not belong to any of the existing defect category systems. The engineering problem label indicates that the reviewers confirmed that the root cause of the sample's abnormality stemmed from image acquisition, equipment status, optical environment, or engineering configuration, rather than a defect in the sample itself.

[0126] Through step S602, this embodiment diverts samples according to their attributes to three dedicated channels: manual review, customer arbitration, and engineering verification. This ensures that different types of difficult samples are reviewed by the corresponding responsible parties, avoiding the problem of mismatched professional capabilities when a single role handles all types of samples. Each channel returns structured label conclusions, providing a clear basis for subsequent decisions on the final flow of samples.

[0127] In one embodiment of the AOI difficult example sample identification method based on multi-source uncertainty in this application, it may further include the following: Step S701: If the re-judgment conclusion of the difficult sample is NG label, missed detection label, or new class label, then the sample is assigned to the training pool. Step S702: If the retrial conclusion of the difficult case sample is a dispute label, then the sample is assigned to the customer arbitration pool; Step S703: If the re-judgment conclusion of the difficult example sample is an engineering problem label, then the sample is assigned to the engineering problem pool; Step S704: Assign samples that have not obtained a valid reassessment conclusion and whose business risk scores are lower than the preset risk threshold to the discard pool.

[0128] Optionally, in this embodiment, NG tags, missed detection tags, and new class tag samples are allocated to the training pool.

[0129] Specifically, when re-judging NG labels, the model's original judgment was correct, but the defective sample data corresponding to the NG labels still have value in entering the training pool for model training.

[0130] It's understandable that the model's output is a probability output, that is, the probability of being correct and the probability of being incorrect. When the probability distinction is small, it indicates that the model's understanding of the sample is not deep enough. Even if the output is correct, the model needs further training to strengthen its understanding of such samples. Therefore, under the NG label, samples are further subdivided based on the classification boundary score and the model uncertainty score. Specifically, samples with high classification boundary scores and high model uncertainty scores are placed into the training pool - boundary sample sub-pool for targeted training to strengthen OK / NG or class decision boundaries.

[0131] Specifically, when reviewing missed detection labels, it indicates that the AI ​​model misclassified defective samples as qualified samples. This is the most direct training material for improving model performance, so it is given priority to be included in the training pool - confirmed missed detection sub-pool.

[0132] Specifically, when a new class label is re-evaluated, it indicates that the defect morphology of the sample does not belong to any known category in the existing training data. After the sample is included in the training pool - new class confirmation sub-pool, it needs to be combined with new class definitions and labeling specifications to expand the model's classification system.

[0133] Before performing the allocation, the system also needs to confirm that the image quality sub-score of the sample does not exceed the noise threshold to ensure that the samples entering the training pool meet the basic training requirements in terms of imaging conditions.

[0134] Optionally, in this embodiment, when reviewing a disputed label, it indicates that the sample has multiple inconsistencies, which may include: inconsistent judgments by different reviewers on the same sample, inconsistencies between the output of the system's internal AI model and the human review conclusion, or conflicts between the internal review conclusion and the client's previous judgment conclusion. Such samples cannot be resolved internally. The system packages the sample and all supporting evidence and allocates it to the client's arbitration pool, awaiting the client's final ruling. After the client's arbitration is completed, the arbitration conclusion is sent back to the system, which then reassigns the sample to the corresponding final processing pool based on the arbitration result.

[0135] Optionally, in this embodiment, when reviewing an engineering problem label, it indicates that the reviewer has determined that the abnormal behavior of the sample does not originate from a genuine defect in the workpiece itself, but rather from an anomaly in the data acquisition or equipment operation process. Typical engineering problems include camera defocus leading to image blurring, light source brightness drift leading to abnormal exposure, coordinate mapping errors leading to detection area positioning deviation, Recipe version mismatch leading to criterion failure, and abnormal data collected during equipment alarms. The system allocates such samples to an engineering problem pool, allowing engineering or equipment personnel to retrieve associated equipment logs, alarm records, light source parameters, and Recipe version information for root cause analysis and repair. After repair, if the sample is restored to a readable state and has a high business risk, the system can reinstate it into the candidate process; if it cannot be repaired or has low value, it is ultimately transferred to the discard pool.

[0136] The system iterates through all samples that have undergone the review and verification process. When a sample fails to obtain any valid review conclusion within the specified review window, the system reads the sample's business risk score and compares it with a preset risk threshold. If the business risk score is below the preset threshold, it indicates that even if the sample is discarded, it will not cause significant missed detection risk or customer impact, and the system allocates it to the discard pool. If the business risk score reaches or exceeds the preset threshold, even if the sample lacks a valid review conclusion, the system should not discard it directly, but rather mark it as pending supplementary review or enter the postponement candidate state, waiting to participate in the sampling selection again in the next round of review budget allocation.

[0137] Through step S704, this embodiment successfully achieved automated sample diversion based on the re-judgment conclusion.

[0138] To improve the efficiency and accuracy of difficult sample identification in AOI production line data, this application provides an embodiment of an AOI difficult sample identification device based on multi-source uncertainty, which implements all or part of the aforementioned method for identifying difficult AOI samples based on multi-source uncertainty. See [link to embodiment]. Figure 2 The AOI difficult example sample identification device based on multi-source uncertainty specifically includes the following components: The data acquisition module 10 is used to acquire multi-source data from the AOI production line site, including site samples, model output results, rule output results, and the engineering context of site samples. The difficult case sample scoring module 20 is used to calculate the risk score of the field sample according to the multi-source data. The risk score includes model uncertainty score, classification boundary score, multi-source divergence score, out-of-distribution score, business risk score and noise risk score. The module also performs a weighted sum of the calculated risk scores to determine the corresponding comprehensive difficult case score. Based on the risk scores, the comprehensive difficult case score and the preset difficult case sample admission criteria, the field sample that meets the difficult case sample admission criteria is included in the preset difficult case sample pool. The difficult example sample diversion module 30 is used to perform deduplication, clustering, and diversity selection operations on the difficult example samples in the difficult example sample pool, determine the corresponding re-judgment task list, perform re-judgment verification on the difficult example samples in the re-judgment task list, determine the corresponding re-judgment conclusion, and allocate each difficult example sample to the corresponding data processing pool according to the re-judgment conclusion of each difficult example sample. The data processing pool includes a training pool, a customer arbitration pool, an engineering problem pool, and a discard pool.

[0139] As described above, the AOI difficult sample identification device based on multi-source uncertainty provided in this application embodiment can acquire multi-source data from the AOI production line site, including sample images, AI model outputs, traditional rule results, and engineering context parameters, and calculate multiple risk scores that can comprehensively characterize the sample value, including uncertainty score, classification boundary score, divergence score, out-of-distribution score, business risk score, and noise risk score. The above scores are weighted and fused to obtain a comprehensive difficult sample score, and high-value samples are selected and included in the difficult sample pool based on preset admission conditions. The samples in the pool are deduplicated, clustered, and subjected to diversity selection to generate a re-judgment task list. After re-judgment verification, the samples are finally allocated to the training pool, customer arbitration pool, engineering problem pool, or discard pool according to the re-judgment conclusion. This can improve the efficiency and accuracy of difficult sample identification of AOI production line site data.

[0140] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described AOI difficult sample identification method based on multi-source uncertainty.

[0141] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described AOI difficult sample identification method based on multi-source uncertainty.

[0142] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described AOI difficult sample identification method based on multi-source uncertainty.

[0143] In this embodiment of the invention, by acquiring multi-source data from the AOI production line, including sample images, AI model outputs, traditional rule results, and engineering context parameters, multiple risk scores that can comprehensively characterize the value of the samples can be calculated, including uncertainty score, classification boundary score, divergence score, out-of-distribution score, business risk score, and noise risk score. The above scores are weighted and fused to obtain a comprehensive difficult case score, and high-value samples are selected and included in the difficult case sample pool based on preset admission criteria. The samples in the pool are deduplicated, clustered, and subjected to diversity selection to generate a re-judgment task list. After re-judgment verification, the samples are finally allocated to the training pool, customer arbitration pool, engineering problem pool, or discard pool according to the re-judgment conclusion. This can improve the efficiency and accuracy of difficult case identification of AOI production line data.

[0144] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying difficult examples in AOI based on multi-source uncertainty, characterized in that, The method includes: Acquire multi-source data from the AOI production line, including field samples, model output results, rule output results, and the engineering context of the field samples; Risk scores for field samples are calculated based on the multi-source data. These risk scores include model uncertainty score, classification boundary score, multi-source divergence score, out-of-distribution score, business risk score, and noise risk score. The calculated risk scores are then weighted and summed to determine the corresponding comprehensive difficult case score. Based on the risk scores, the comprehensive difficult case score, and the preset difficult case sample admission criteria, field samples that meet the difficult case sample admission criteria are included in the preset difficult case sample pool. The difficult sample in the difficult sample pool is deduplicated, clustered, and subjected to diversity selection operations to determine the corresponding re-judgment task list. The difficult sample in the re-judgment task list is re-judged and verified to determine the corresponding re-judgment conclusion. According to the re-judgment conclusion corresponding to each difficult sample, each difficult sample is assigned to the corresponding data processing pool. The data processing pool includes a training pool, a customer arbitration pool, an engineering problem pool, and a discard pool.

2. The AOI hard example sample identification method based on multi-source uncertainty according to claim 1, characterized in that, The calculation of risk scores for on-site samples based on the multi-source data includes: The corresponding model uncertainty score is determined based on the information entropy of the category probability distribution of the model output. The corresponding classification boundary score is determined based on the difference between the probability of the largest class and the probability of the second largest class in the model output results. Based on the output difference between the model output and the rule output, the corresponding multi-source divergence score is determined, wherein the model includes at least one of the following: classification model, detection model, segmentation model, anomaly detection model, teacher model, and student model. The corresponding out-of-distribution score is determined based on the distance between the visual feature vector of the on-site sample and the center of the known defect category feature space. Based on the severity of defects in the field samples, the level of customer concern, the cost of missed inspections, the project stage, the customer's on-site verification status, and historical complaint records, the corresponding business risk score is determined. Based on the image quality sub-score and engineering anomaly sub-score of the field samples, the corresponding noise risk score is determined.

3. The AOI hard example sample identification method based on multi-source uncertainty according to claim 1, characterized in that, The step of including on-site samples that meet the pre-set difficult case sample admission criteria into the pre-set difficult case sample pool based on the risk scores, the overall sample difficulty score, and the pre-set difficult case sample admission criteria includes: Pre-defined criteria for difficult sample admission include comprehensive score admission criteria, mandatory admission criteria for individual risks, and admission criteria based on confirmed facts. Determine whether the comprehensive difficulty score of the on-site sample meets the comprehensive score admission criteria. If it does, the on-site sample that meets the criteria is included in the preset difficult sample pool. Determine whether each risk score of the on-site sample meets the mandatory admission criteria for the single risk. If it does, the on-site sample that meets the criteria will be included in the preset difficult sample pool. Determine whether there are confirmed difficult case fact labels in the field samples. If so, include the field samples that meet the conditions into the preset difficult case sample pool.

4. The AOI hard example sample identification method based on multi-source uncertainty according to claim 1, characterized in that, The process of performing deduplication, clustering, and diversity selection operations on the difficult sample pool to determine the corresponding re-judgment task list includes: Perform complete duplication detection and near-duplication clustering on the difficult sample samples in the difficult sample pool to determine the corresponding duplication sample groups. At least one representative sample is retained in each duplication sample group to form a set of candidate samples. Under the preset review budget constraint, diversity optimization selection is performed on the candidate sample set to determine the corresponding review task list.

5. The AOI hard example sample identification method based on multi-source uncertainty according to claim 4, characterized in that, The step of performing diversity optimization selection on the candidate sample set under a preset review budget constraint to determine the corresponding review task list includes: Obtain the available review budget within the current review period, the review budget including at least one of the maximum number of review samples, the maximum review working hours, and the review quota for each category; Under the constraint of the review budget, a diversity optimization algorithm is used to select representative samples from the candidate sample set to determine the corresponding review task list. The objective function of the diversity optimization algorithm is set as: to maximize the total information gain of the final selected sample set without exceeding the review budget.

6. The AOI hard example sample identification method based on multi-source uncertainty according to claim 1, characterized in that, The step of reviewing and verifying the difficult examples in the review task list and determining the corresponding review conclusion includes: The difficult cases in the review task list are assigned to the corresponding review channels for review and verification. The review channels include manual review channels, customer arbitration channels, and engineering verification channels. The corresponding review conclusions are obtained from each review channel. The review conclusions include OK tags, NG tags, missed detection tags, false alarm tags, disputed tags, new category tags, and engineering problem tags.

7. The method for identifying difficult AOI samples based on multi-source uncertainty according to claim 1, characterized in that, The step of allocating each difficult case sample to a corresponding data processing pool based on the re-judgment conclusion corresponding to each difficult case sample includes: If the re-judgment conclusion of the difficult sample is NG label, missed detection label, or new class label, then the sample is assigned to the training pool; If the retrial conclusion of the difficult case sample is a dispute label, then the sample will be assigned to the customer arbitration pool; If the re-judgment conclusion of the difficult example sample is labeled as an engineering problem, then the sample is assigned to the engineering problem pool; Samples that do not receive a valid reassessment and whose business risk scores are below the preset risk threshold will be assigned to the discard pool.

8. An AOI hard example sample identification device based on multi-source uncertainty, characterized in that, The device includes: The data acquisition module is used to acquire multi-source data from the AOI production line, including field samples, model output results, rule output results, and the engineering context of the field samples. The difficult case sample scoring module is used to calculate the risk score of the field sample based on the multi-source data. The risk score includes model uncertainty score, classification boundary score, multi-source divergence score, out-of-distribution score, business risk score, and noise risk score. The module then performs a weighted sum of the calculated risk scores to determine the corresponding comprehensive difficult case score. Based on the risk scores, the comprehensive difficult case score, and the preset difficult case sample admission criteria, the field samples that meet the difficult case sample admission criteria are included in the preset difficult case sample pool. The difficult example sample diversion module is used to perform deduplication, clustering, and diversity selection operations on the difficult example samples in the difficult example sample pool, determine the corresponding re-judgment task list, perform re-judgment verification on the difficult example samples in the re-judgment task list, determine the corresponding re-judgment conclusion, and allocate each difficult example sample to the corresponding data processing pool according to the re-judgment conclusion of each difficult example sample. The data processing pool includes a training pool, a customer arbitration pool, an engineering problem pool, and a discard pool.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the AOI difficult sample identification method based on multi-source uncertainty as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the AOI difficult sample identification method based on multi-source uncertainty as described in any one of claims 1 to 7.

Images