Defect fusion method, device and equipment based on visual inspection and storage medium

Abstract

The invention discloses a defect fusion method and device based on visual detection, equipment and a storage medium, and relates to the field of image detection. Obtaining original defect points of the to-be-detected image, and performing defect filtering according to the defect confidence and the defect size to obtain a defect set; determining a position relationship between every two defect points in all the filtered defect points based on the defect coordinates, and performing matching fusion on every two defect points meeting the position relationship according to the defect detection model to which the defect points belong; acquiring a reverse defect set extracted from the to-be-detected image through SVD reverse operation, performing redundancy filtering detection on the forward defect set and the reverse defect set, and outputting a target fusion defect; the forward defect set is established based on the filtered non-fused defects and the fused defects. According to the scheme, through defect point filtering, defect fusion based on the position relation and redundancy filtering by using SVD reverse operation, results of multiple defect detection models are effectively integrated, redundant defect storage is reduced, and the method has the advantage of reducing manual reinspection burden.

Description

Technical Field

[0001] This application relates to image detection, and more particularly to a defect fusion method, apparatus, device, and storage medium based on visual detection. Background Technology

[0002] In the semiconductor and integrated circuit manufacturing industry, various defects or flaws inevitably appear on the circuit boards after they are produced on the assembly line. To ensure product yield, it is necessary to scan and inspect each batch of circuit boards for defects.

[0003] Traditional technical solutions rely on single or multiple defect detection models, including AI-driven defect detection models and computer vision-driven CV defect detection models. As model resolution increases, panoramic scan images generate a large number of randomly distributed or densely clustered defect data. Inspection systems typically store screenshots of each defect in a database for subsequent manual review and analysis. However, with increasingly sophisticated manufacturing processes, high-precision inspection leads to a dramatic increase in the number of defects, significantly increasing the burden of manual review. A key issue is that most defects do not directly lead to the scrapping of the material board; their impact needs to be comprehensively assessed considering functional areas, location, size, and type. For example, minor defects in non-critical areas such as metal pads or substrate areas, if they do not affect core functions, should be appropriately filtered out rather than reported. Similarly, manufacturers often employ area masking strategies, allowing defective areas to operate at downgraded specifications, while the remaining functional parts can still be sold at different levels. Existing technologies lack effective means to integrate the detection results of AI and CV models, failing to intelligently identify redundant defects or fuse overlapping detection points. This results in massive amounts of defect images being stored indiscriminately, wasting storage resources and slowing down the inspection process. Especially in multi-model collaborative detection scenarios, duplicate reporting and conflict judgments occur frequently, further reducing detection efficiency. Therefore, there is an urgent need for a method to systematically integrate and filter raw defect data in order to accurately extract key defect information and reduce unnecessary re-inspection workload.

[0004] Traditional technologies urgently need improvement to address these issues. Summary of the Invention

[0005] This application provides a defect fusion method, apparatus, device, and storage medium based on visual inspection, which has the advantages of reducing the number of defect images and reducing the burden of manual re-inspection.

[0006] On the one hand, this application provides a defect fusion method based on visual detection, the method comprising: Obtain the original defect points of the image to be inspected under at least two defect detection models, and filter the defects according to the defect confidence and defect size to obtain a defect set; Based on the defect coordinates, determine the positional relationship between each pair of defect points in all filtered defect points, and match and fuse the pairs of defect points that satisfy the positional relationship according to their respective defect detection models to output the fused defect. The reverse defect set is obtained by reversing the singular value decomposition (SVD) operation on the image to be inspected. The forward defect set and the reverse defect set are subjected to redundancy filtering detection, and the target fusion defect is output. The forward defect set is constructed based on the filtered unfused defects and the fusion defect.

[0007] Specifically, the positional relationships between pairs of defect points include overlapping, intersecting, adjacent, and irrelevant relationships; at least two defect detection models are AI defect models and CV defect models; pairwise matching relationships include matching two AI defect points, matching two CV defect points, and matching AI defect points with CV defect points. When two defect contours are in an inclusive relationship, they are determined to be in a defect overlap state; When two defect profiles intersect, it is determined to be a defect intersection state; When two defect contours do not touch and the nearest distance is less than the distance threshold, they are determined to be in an adjacent defect state. When two defect profiles do not touch and the nearest distance is not less than the distance threshold, they are determined to be in a defect incoherent state.

[0008] Specifically, when two AI defect points are matched and fused, or when an AI defect point is matched and fused with a CV defect point, the area and confidence of the two defect contours are extracted. The defect with the largest contour area is selected as the fused defect, and the one with the highest confidence value is selected as the confidence of the fused defect. When two CV defect points are matched and fused under the condition of overlapping defects, the defect with the largest contour area is selected as the fused defect, and the confidence level of the corresponding CV defect is determined as the confidence level of the fused defect.

[0009] Specifically, when two AI defect points are matched and fused, or when an AI defect point is matched and fused with a CV defect point, the area and defect confidence of the two defect contours are extracted. The area enclosed by the two defect contours is taken as the fused defect, and the one with the highest confidence value is selected as the confidence of the fused defect. When two CV defect points are matched and fused under the condition of overlapping defects, the area enclosed by the two defect contours is taken as the fused defect, and the confidence level of the CV defect corresponding to the defect with the largest contour area is determined as the confidence level of the fused defect.

[0010] Specifically, when defects are adjacent, determine whether adjacent defects are in the same layer partition; When adjacent defects are in the same partition and the two defects have the same label type, the CV defect point matching and fusion is performed. The two adjacent defect contours are connected and fused. The area enclosed by the connection area and the two contour areas is determined as the fused defect. The highest confidence value is selected as the confidence value of the fused defect. When adjacent defects are in the same layer partition and two AI defect points are matched and merged, the outlines of the two adjacent defects are connected and merged. The area enclosed by the connection area and the two outline areas is determined as the merged defect, and the one with the highest confidence value is selected as the confidence value of the merged defect.

[0011] Specifically, constructing the positive defect set includes: For the matching fusion output, which contains fusion defects including AI defect points, the first AI scale caliper filtering is performed based on the fused contour area. For the matching fusion output, which does not contain fusion defects with AI defects, no scale filtering is performed; For defects that are directly output without matching and fusion, a second AI scale caliper filter is performed based on the original defect contour; The positive defect set is obtained by summing all defects that meet the first AI scale caliper filter, the second AI scale caliper filter, and those that do not undergo AI scale filtering.

[0012] Specifically, the process of performing redundant filtering detection on the positive defect set and the negative defect set, and outputting the target fused defect, includes: The positive defect set and the negative defect set are matched according to the defect coordinates, and defects whose defect contour area overlap exceeds the target ratio are filtered out to eliminate redundant detections. The remaining defects in the positive defect set are identified as the target fusion defects.

[0013] Specifically, the SVD reverse operation extracts the reverse defect set process, and the abnormal regions in the image to be inspected are identified based on the reconstruction error of the SVD algorithm; the SVD reverse operation and the forward defect detection target the same target region, and the SVD reverse operation is not limited to the same layer partition.

[0014] Specifically, the defect filtering based on defect confidence and defect size to obtain a defect set includes: The unchecked areas of the image to be tested are determined, and the original defect points are initially filtered based on the unchecked areas. The unchecked areas include AI unchecked areas, CV unchecked areas, and completely unchecked areas. AI defect points in the AI ​​unchecked areas, CV defect points in the CV unchecked areas, and all defect points in the completely unchecked areas are filtered out. Set a minimum confidence threshold for AI defects and a minimum size gauge threshold for CV defects, and perform secondary filtering on the initially filtered AI defects and CV defects to obtain the defect set.

[0015] Specifically, after outputting the target fusion defect, the method further includes: The confirmed target fusion defects are directly extracted into defect images one by one according to the set size and stored, and each defect image contains only one defect; the defect image contains defect labels and confidence information; Alternatively, the confirmed target fusion defects can be directly extracted into defect images at a set size, and a single defect image contains several defects; the defect image contains defect labels and confidence information for all defects.

[0016] In another aspect, this application provides a visual inspection-based defect fusion apparatus, the apparatus being used for the visual inspection-based defect fusion described in any of the foregoing aspects, the apparatus comprising: The defect filtering module is used to obtain the original defect points output by the image to be inspected under at least two defect detection models, and to filter defects according to the defect confidence and defect size to obtain a defect set. The defect fusion module is used to determine the positional relationship between any two defect points among all filtered defect points based on the defect coordinates, and to match and fuse the pair of defect points that satisfy the positional relationship according to their respective defect detection models, and output the fused defect. A redundancy filtering module is used to obtain the inverse defect set extracted from the image to be inspected through the inverse operation of singular value decomposition (SVD), perform redundancy filtering detection on the forward defect set and the inverse defect set, and output the target fusion defect; the forward defect set is constructed based on the filtered unfused defects and the fusion defect.

[0017] In another aspect, this application provides a computer device including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the visual detection-based defect fusion method described above.

[0018] In another aspect, this application provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the visual detection-based defect fusion method described above.

[0019] The beneficial effects of the technical solution provided in this application include at least the following: Preliminary filtering of the original defect points output by the image under inspection under various defect detection models ensures more comprehensive coverage of the initial defect data. Simultaneously, confidence and size filtering initially eliminates low-value noise. Determining the positional relationship between pairs of defect points based on defect coordinates and performing matching and fusion identifies and integrates spatially related defects, merging multiple original detection results that may point to the same physical defect into a single fused defect, significantly reducing the number of defects. Introducing SVD reverse operation to extract a reverse defect set and redundant filtering detection further improves the robustness and accuracy of defect identification, ensuring that the final output target fused defect is multi-verified and highly streamlined. This significantly reduces the number of defect images, optimizes subsequent manual re-inspection and analysis processes, thereby improving overall inspection efficiency and the yield rate of the material board, and maximizing resource utilization. Attached Figure Description

[0020] Figure 1 A schematic diagram of a local image under inspection containing defect points is shown in one possible form; Figure 2 This is a flowchart of the defect fusion method based on visual detection provided in the embodiments of this application; Figure 3 This diagram illustrates different defect distributions under one possible form. Figure 4 This is a schematic diagram of the fusion rules specified under different positional states and model attributes; Figure 5 A schematic diagram of defect fusion based on the outer contour tangent circle between adjacent defects is shown; Figure 6 This diagram illustrates the distribution of uninspected regions in an image to be inspected under one possible form. Figure 7 This is a structural block diagram of the defect fusion device based on visual detection provided in the embodiments of this application; Figure 8 A structural block diagram of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0022] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0023] In the semiconductor and integrated circuit manufacturing industry, after the PCBs are produced on the assembly line, defect detection is required to ensure yield. Traditional solutions employ various AI or CV defect detection models to analyze panoramic scan images. When the model resolution is high, it outputs a large amount of randomly or densely distributed defect data, such as... Figure 1 The diagram illustrates a partial image of a component to be inspected, containing defects. These defects need to be comprehensively assessed based on the material board's function, defect location, size, and type to determine whether they affect product functionality. Specifically, defects in a particular area that do not reach a functional impact threshold do not need to be reported, similar to a manufacturer's masking operation of defective areas, allowing the material board to still function under downgraded specifications. Furthermore, this assessment process requires manual review of the defect screenshots to classify product levels. However, the redundant defect data generated by high-precision detection increases the burden on the manual review process, leading to reduced defect processing efficiency and resource imbalance, specifically manifested as extended review time and excessive consumption of data storage resources.

[0024] For example, in the multilayer lithography process of 300mm wafers, deep learning models and traditional image processing models are used to simultaneously perform defect detection. When the detection resolution is improved to the sub-micron level, a single panoramic scan generates more than two thousand original defect coordinates. Among them, dense defect groups located in the core area of ​​logic circuits need to be manually analyzed point by point to analyze their topological relationship with functional units, while tiny defects in the edge areas are judged as invalid reports because their size is below the process tolerance. Furthermore, in this scenario, defect labels include coordinate offset, confidence threshold, and type encoding information, but manual cross-validation of duplicate reports output by different models is required, causing the re-inspection process to stall at the data screening stage. This prolongs the inspection cycle of the entire batch of materials and causes the storage system to continuously receive unnecessary defect images.

[0025] If the above problems are not addressed, the defect detection process will continuously reduce system throughput due to the accumulation of redundant data. Decision delays in the manual re-inspection stage will further propagate to subsequent grading processes, causing temporary production line shutdowns. Furthermore, unfiltered raw defect data consumes database storage resources and interferes with the identification priority of core defects, thereby increasing the probability of usable materials being misclassified as scrap, ultimately affecting the effective allocation of overall production resources.

[0026] Therefore, this application provides a defect fusion method based on visual detection. Figure 2 This is a flowchart of a visual detection-based defect fusion method provided in an embodiment of this application, including the following steps: S1. Obtain the original defect points output by the image to be inspected under at least two defect detection models, and filter the defects according to the defect confidence and defect size to obtain a defect set; The image to be inspected is the raw image data that needs to be used for defect detection. It usually comes from panoramic images of production objects such as material boards taken by scanning equipment.

[0027] Defect detection models are algorithms or systems used to identify defects in images. In this application, at least two defect detection models may include deep learning models based on artificial intelligence (AI) and models based on traditional computer vision (CV) rules. The original defect points are the total number of defect points detected and output by each model.

[0028] Defect confidence is a numerical assessment given by a defect detection model of the probability that an identified defect is a real defect. This value typically ranges from 0 to 1; a higher value indicates a higher probability that the defect is real. Examples include the AI ​​confidence score and CV confidence score output by AI and CV defect models, respectively. Furthermore, original defect points can be stored as records in the form of defect labels (e.g., scratches, wear, foreign objects), confidence scores, and defect coordinates (contours).

[0029] The defect filtering in this step can refer to screening the original defect points according to preset conditions, such as defect confidence threshold and defect size threshold, to remove defects with low confidence or excessively small size, thereby reducing the burden of subsequent processing and ultimately forming a preliminary defect set.

[0030] S2. Determine the positional relationship between any two defect points after filtering based on the defect coordinates, and match and fuse the pairs of defect points that satisfy the positional relationship according to their respective defect detection models to output the fused defect. In this embodiment, the defect points are actually the areas covered by the outer contour of the defect. The positional relationship between any two defect contours can be calculated by the intersection over union (IoU) between any two defect contours or their bounding boxes. When the IoU value exceeds a certain preset threshold, or when the contours overlap, the two defect points can be considered to have an overlapping relationship. Other relationships include adjacency and containment relationships.

[0031] For example, when the bounding boxes of two defect points do not overlap but are very close, they can be considered to be adjacent. For pairwise defect points that satisfy specific positional relationships, matching and fusion are performed based on their respective defect detection models to leverage the model's advantages. For instance, if two overlapping defect points are both identified by an AI model, the one with the higher confidence level can be simply selected as the fused defect; if one defect point is identified by an AI model and the other by a CV model, and they overlap, the one with the larger area can also be selected as the fused defect. In this way, multiple original defect points that may point to the same physical defect can be merged into a single fused defect, thereby reducing the number of defects.

[0032] S3. Obtain the inverse defect set extracted from the image to be inspected through the inverse operation of Singular Value Decomposition (SVD). Perform redundancy filtering detection on the forward defect set and the inverse defect set, and output the target fused defect. The forward defect set is constructed based on the filtered unfused defects and fused defects.

[0033] Based on this, a reverse defect set is obtained. Specifically, the image to be inspected is decomposed using SVD, and then abnormal regions are identified by reconstructing the image and analyzing the differences between the original and reconstructed images. These abnormal regions are considered reverse defects. This reverse detection mechanism provides a complementary perspective to forward detection, helping to discover defects that may be missed or misjudged by forward detection. Subsequently, the forward and reverse defect sets are subjected to redundancy filtering detection. The forward defect set consists of defect points that have been initially filtered but not fused, as well as all fused defects that have been formed. For example, the coordinates and contours of each defect in the forward defect set can be compared with those in the reverse defect set. If significant overlap is found, one of them is considered a redundant detection and is removed. Thus, the final target fused defect can be output, ensuring the conciseness and accuracy of the defect report.

[0034] In summary, this application effectively solves this problem by introducing a multi-model collaborative detection and intelligent fusion mechanism. First, it acquires the original defect points output by the image to be inspected under at least two defect detection models and performs preliminary filtering. This ensures more comprehensive coverage of the initial defect data. Simultaneously, low-value noise is initially eliminated through confidence and size filtering. Compared to traditional methods that may rely on only a single model or simple thresholds, the preliminary filtering mechanism in this application is more refined.

[0035] Secondly, determining the positional relationships between pairs of defect points based on defect coordinates and then matching and fusing them is one of the core innovations of this application. This step can identify and integrate spatially related defects, merging multiple original detection results that may point to the same physical defect into a single fused defect. For example, in the example above, "AI scratch" and "CV edge damage" were successfully fused, avoiding duplicate reporting of the same physical defect. This contrasts sharply with the traditional approach of simply listing all detection results, significantly reducing the number of defects.

[0036] Finally, a singular value decomposition (SVD) inverse operation is introduced to extract a reverse defect set, which is then subjected to redundancy filtering with the forward defect set, further improving the robustness and accuracy of defect identification. The SVD inverse operation provides an independent verification perspective, helping to discover blind spots in forward detection or verify its results. Through redundancy filtering of the forward and reverse sets, it is ensured that the final output target fusion defect is multi-verified and highly streamlined. Therefore, the method of this application can significantly reduce the number of defect images, optimize the subsequent manual re-inspection and analysis process, thereby improving the overall inspection efficiency and the yield of the material board, and maximizing the utilization of resources.

[0037] Regarding the specific implementation of defect fusion, this application defines the positional relationships between pairs of defect points as including coincidence, intersection, adjacency, and incoherence. At least two defect detection models (this application uses two as examples) are used: an AI defect model and a CV defect model; pairwise matching relationships include matching two AI defect points, matching two CV defect points, and matching AI defect points with CV defect points. Figure 3 A schematic diagram showing different defect distributions is provided. When two defect contours are in an inclusive relationship, they are determined to be in a defect coincidence state; corresponding to Figure 3 Defects D and E in; When two defect contours intersect, it is determined to be a defect intersection state; corresponding to Figure 3 Defects A and B in the text; When two defect contours do not touch and the nearest distance is less than a distance threshold (e.g., 30 pixels), they are determined to be in an adjacent defect state; corresponding to defects C and D. When two defect contours do not touch, and the nearest distance is not less than a distance threshold (e.g., more than 30 pixels), the defect is determined to be in an incoherent state; corresponding to Figure 3 Defects A and D are shown in the image. Typically, when two defects are far apart, they are not fused; the image is output directly, awaiting subsequent extraction to generate two separate defect images.

[0038] In some of the solutions described above in this application, the matching and fusion of defect points under defect overlap conditions is proposed to optimize the defect fusion process. However, in its implementation, the lack of specific rules for selecting the fused defects and confidence levels may lead to inconsistent fusion results or inappropriate confidence level selection, affecting the accuracy and efficiency of defect detection.

[0039] Therefore, this application will discuss the various positional state relationships separately. Figure 4 The following is a diagram illustrating the fusion rules specified under different positional states and model attributes: 1. Defect overlap state AI+AI, AI+CV: When matching and fusing two AI defect points, or when matching and fusing AI defect points and CV defect points, extract the area and defect confidence of the two defect contours. The defect with the largest contour area is selected as the fusion defect, and the one with the highest confidence value is selected as the fusion defect confidence value.

[0040] In both cases, the system selects the defect with the largest contour area as the final fused defect by comparison, ensuring that the physically more significant defect is preserved. Simultaneously, the system selects the defect with the highest confidence value as the fused defect's confidence level. This allows the fused result to inherit the strongest confirmation of the defect from the detection model, thereby improving the reliability of the fused defect.

[0041] CV+CV: When matching and fusing two CV defect points, the defect with the largest contour area is selected as the fusion defect, and the confidence level of the corresponding CV defect is determined as the confidence level of the fusion defect.

[0042] This approach simplifies the confidence level selection logic. While still selecting the defect with the largest contour area as the fusion defect, its corresponding CV defect confidence level is directly determined as the fusion defect's confidence level. This approach is based on the fact that CV defect models may have different confidence level evaluation mechanisms than AI models. By directly adopting the confidence level of the defect with the largest area, it maintains area-based priority while avoiding the necessity of complex confidence level comparisons between CV defects, thereby improving fusion efficiency and consistency.

[0043] As a specific implementation method, suppose that after performing defect detection on an image to be inspected, the system identifies two defect points that are in a state of overlapping defects.

[0044] Scenario 1: Matching and fusing AI defect points with AI defect points or AI defect points with CV defect points. For example, a scratch defect identified by an AI defect model is detected, with an outline area of ​​A1 and a confidence level of 0.8; simultaneously, a wear defect identified by an AI defect model or CV defect model is detected, with an outline area of ​​A2 and a confidence level of 0.6. These two defect outlines overlap. In this case, the system extracts the area A1 and confidence level of 0.8 of the scratch defect, and the area A2 and confidence level of 0.6 of the wear defect. First, A1 and A2 are compared; assuming A1 is greater than A2, the scratch defect is selected as the fusion defect. Second, the confidence levels of 0.8 and 0.6 are compared, and the highest confidence value of 0.8 is selected as the confidence level of the fusion defect. Finally, the output fusion defect will be the scratch defect with a confidence level of 0.8.

[0045] Scenario 2: Matching and fusing two CV defect points. For example, two defects identified by the CV defect model are detected: a bubble defect with an area of ​​B1 and a confidence level of 0.7; and a foreign object defect with an area of ​​B2 and a confidence level of 0.5. These two defect contours overlap. In this case, the system extracts the area B1 and confidence level 0.7 of the bubble defect, and the area B2 and confidence level 0.5 of the foreign object defect. First, B1 and B2 are compared; assuming B1 is greater than B2, the bubble defect is selected as the fusion defect. Second, the confidence level 0.7 corresponding to the bubble defect is directly determined as the confidence level of the fusion defect. Finally, the output fusion defect will be the bubble defect with a confidence level of 0.7.

[0046] Through the above technical solutions, this application provides specific and differentiated fusion rules for the defect matching and fusion process under defect overlap conditions, effectively solving the problems of inconsistent fusion results or inappropriate confidence level selection caused by the lack of clear rules in traditional methods. By prioritizing the selection of defects with the largest contour area as fusion defects, it ensures that physically more significant defect information is preserved, avoiding important defects being covered or ignored by secondary defects. Simultaneously, for fusion cases involving AI defect points, the highest confidence value is selected as the confidence level of the fusion defect, significantly improving the reliability and accuracy of the fusion defects, enabling subsequent defect analysis and decision-making to be based on more reliable data. For the fusion of pure CV defect points, the confidence level of the defect with the largest area is directly adopted, simplifying the processing flow while maintaining the rationality of the fusion results. Overall, the introduction of these rules makes the defect fusion process more standardized and automated, reducing the need for manual intervention, improving the efficiency and data quality of the defect detection system, thereby optimizing the yield management of material boards. 2. Defect Intersection State AI+AI, AI+CV: When matching and fusing two AI defect points, or when matching and fusing AI defect points and CV defect points, extract the area and defect confidence of the two defect contours; take the area enclosed by the two defect contours as the fused defect, and select the one with the highest confidence value as the fused defect confidence. In both cases, when AI defect points are involved in fusion, the system extracts the area and confidence score of the two defect contours, and uses the region enclosed by the two contours as the fused defect. Simultaneously, it selects the region with the highest confidence score as the fused defect confidence score. This mechanism ensures the accuracy of the fused region and prioritizes the detection results deemed most reliable by the model, thereby improving the reliability of the fused defect.

[0047] CV+CV: When matching and fusing two CV defect points, the area enclosed by the two defect contours is taken as the fusion defect, and the confidence level of the CV defect corresponding to the defect with the largest contour area is determined as the confidence level of the fusion defect.

[0048] In this case, the system selects the CV defect confidence score corresponding to the defect with the largest contour area as the fused defect confidence score, and outputs the defect with the largest area as the label. This processing method can highlight more significant defect features when CV defects overlap, avoiding the problem of information ambiguity caused by multiple small defects overlapping.

[0049] The following is a concrete example: Suppose two defect points are detected in the image to be inspected, namely defect A (detected by an AI defect model, confidence level 0.8, contour area 500 pixels) and defect B (detected by a CV defect model, confidence level 0.6, contour area 400 pixels). By analyzing the defect coordinates, it is determined that the contours of defect A and defect B intersect, i.e., they are in a defect intersection state. At this time, due to the matching and fusion of AI defect point (defect A) and CV defect point (defect B), the system will extract the contour areas of defect A and defect B (500 and 400 pixels) and the defect confidence levels (0.8 and 0.6). Subsequently, the region jointly enclosed by the contours of defect A and defect B (i.e., their intersection region) is determined as the contour of the fused defect. When determining the confidence level of the fused defect, the system will compare the confidence level of defect A (0.8) and defect B (0.6), and select the higher one (0.8) as the confidence level of the fused defect.

[0050] For another example, if two CV defect points are detected in the image to be inspected, namely defect C (detected by the CV defect model, confidence level 0.7, contour area 600 pixels) and defect D (detected by the CV defect model, confidence level 0.75, contour area 550 pixels), by analyzing the defect coordinates, it is determined that the contours of defect A and defect B intersect, i.e., they are in a defect intersection state. In this case, since it is a matching and fusion of two CV defect points, the system will use the area enclosed by the contours of defect C and defect D as the contour of the fused defect. When determining the confidence level of the fused defect, the system will compare the contour area of ​​defect C (600 pixels) and the contour area of ​​defect D (550 pixels), selecting defect C with the largest area, and determining its corresponding CV defect confidence level of 0.7 as the confidence level of the fused defect. Simultaneously, the label of defect C (e.g., "scratch") will also be output as the label of the fused defect.

[0051] Through the above technical solutions, this application effectively solves the problems of inaccurate fusion region definition or unreasonable confidence level selection in the case of intersecting defects. Specifically, in the intersection fusion involving AI defect points, by extracting the area and confidence level of the defect contour and using the jointly enclosed area as the fusion defect, while selecting the highest confidence level, the accuracy and reliability of the fusion result are ensured, avoiding redundancy or information loss caused by simple merging. In the intersection fusion of CV defect points, by using the jointly enclosed area as the fusion defect and using the confidence level of the defect with the largest area as the fusion confidence level, the fused defect is more representative and can highlight more significant defect features, thereby reducing interference with subsequent manual re-inspection and improving detection efficiency and accuracy. These refined fusion rules make the defect fusion process more intelligent and efficient, significantly reducing the number of redundant defect points, optimizing the overall detection process, and improving the accuracy and efficiency of defect detection. 3. Adjacent state of defects In some of the solutions described above in this application, matching and fusion are proposed to reduce redundant defect points when defects are adjacent. However, this process does not consider the layer partition where the defect is located and the defect label type, which may lead to incorrect fusion when the labels are inconsistent in the same partition, or to ignoring fusion opportunities when the defects are in different partitions, thereby increasing redundant defect points and reducing detection efficiency. Therefore, this application further proposes to optimize according to layer partitions.

[0052] When defects are adjacent, it is determined whether the adjacent defects belong to the same layer partition. If the adjacent defects do not belong to the same layer partition, they are directly output without merging. Layer partitions can be pre-defined based on the material board design drawings or CAD data. For example, different functional areas such as metal pad areas, solder mask areas, substrate areas, and through holes on the material board can be marked. During the determination, it can be checked whether the coordinates or outline areas of two defect points are completely or mainly located within the same predefined layer partition. If two defect points belong to different layer partitions, they are considered not to be in the same layer partition. When adjacent defects do not belong to the same layer partition, no merging operation is performed. Instead, these two defect points are directly output as independent defects to avoid incorrectly merging physically unrelated defects.

[0053] When adjacent defects are in the same partition, the following fusion operation is performed: CV+CV: Matching and fusing two CV defect points with the same defect label type, connecting and fusing two adjacent defect contours, defining the area enclosed by the connecting area and the two contour areas as the fused defect, and selecting the one with the highest confidence value as the confidence of the fused defect; For example, the convex hull of two defect contours can be calculated, or the two contours can be connected through morphological dilation, and then the connected region can be calculated; alternatively, the region enclosed by the connecting area and the two contour areas can be defined as the new fusion defect by using the circumscribed circles of adjacent contours. See also Figure 5 The diagram illustrates the fusion defect between the circumcircles of contours C and D. It should be noted that this adjacent fusion aims to combine the two contours into one. In some embodiments, rectangular frames or other forms of connection may also be used; this application does not impose any limitations on this.

[0054] The confidence score for merging defects is selected from the confidence scores of the two original CV defect points, with the higher score being used as the confidence score for the merged defect to ensure high reliability of the merged defect information. However, when the defect labels of two CV defect points are inconsistent, even if they are in the same layer partition and adjacent, no merging operation is performed. Instead, they are output as independent defects to avoid incorrectly merging defects of different natures, thereby maintaining the accuracy of defect classification.

[0055] AI+AI: Two AI defect points are matched and fused. The contours of two adjacent defects are connected and fused. The area enclosed by the connection area and the two contour areas is defined as the fused defect. The highest confidence value is selected as the confidence of the fused defect. Similar to CV defect fusion, the first step is to confirm that both defects were detected by the AI ​​defect model. Under this condition, the contours of these two adjacent AI defects are connected and fused. This can be done using methods such as calculating the convex hull, morphological operations, or the circumscribed circle. The region enclosed by the connecting area and the two contour areas is defined as the new fused defect. The confidence score of the fused defect is then selected from the confidence scores of the two original AI defect points, with the highest value chosen as the confidence score of the fused defect.

[0056] AI+CV: When matching and fusing AI and CV defects, this solution stipulates that no fusion operation is performed. Instead, the two defects are output as independent defects. This is to avoid mixing and fusing defects detected by different detection models (AI model and CV model), because different models may perform detection based on different principles and features, and the fusion of their results may introduce uncertainty or reduce the accuracy of the fusion result.

[0057] As a specific implementation, suppose that during defect detection on a semiconductor wafer, a series of raw defect points are detected using both an AI defect model and a CV defect model. After initial filtering of these defect points, the system identifies two defect points A and B as being adjacent. At this point, the system first queries the wafer's layout data to determine whether defect points A and B are located in the same layer partition. For example, if defect point A is located in the "metal pad" area and defect point B is located in the "solder resist area," even if they are very close in the image, the system will determine that they are not in the same layer partition and therefore will not merge them, directly outputting defect points A and B as independent defects.

[0058] On the other hand, if both defect point A and defect point B are determined to be located in the same layer partition, the "substrate area," the system will further examine their model type and defect label. For example, if defect point A is a "scratch" defect detected by a CV model with a confidence level of 0.8, and defect point B is also a "scratch" defect detected by a CV model with a confidence level of 0.7, the system will perform a connection fusion because they have the same model type (CV) and the same label type (scratch). Specifically, the circumcircle of the two defect contours can be calculated, and the area enclosed by the circumcircle can be taken as the new fused defect, with 0.8 selected as the confidence level for the fused defect. If defect point A is a "scratch" defect detected by a CV model, and defect point B is a "foreign object" defect detected by a CV model, the system will not perform fusion because of the inconsistent label types, even if they are in the same partition and adjacent, but will output them separately. For example, if defect A is a "wear" defect detected by an AI model with a confidence level of 0.9, and defect B is also a "wear" defect detected by an AI model with a confidence level of 0.85, the system will perform connection fusion because they have the same model type (AI) and the same label type (wear), and will select 0.9 as the confidence level for the fused defect. However, if defect A is a "scratch" defect detected by an AI model, and defect B is a "scratch" defect detected by a CV model, even if they are in the same partition and adjacent, the system will not fuse them, but will output them separately to avoid mixing the detection results of different models.

[0059] Through the above technical solution, this application introduces layer partitioning and defect label type judgment in the fusion process of adjacent defects, effectively solving the problems of erroneous fusion or ignoring fusion opportunities that may occur in traditional solutions. By performing fusion only within the same layer partition, it avoids the incorrect merging of physically unrelated defects. Simultaneously, refined processing based on defect detection model type and defect label type ensures the logicality and accuracy of fusion. For example, CV defect points with the same label type can be effectively merged, while defects with different label types or different model types are prevented from being unreasonably mixed. This significantly reduces the number of redundant defect points, improves the accuracy and reliability of defect detection results, and thus improves overall detection efficiency and the yield rate of material boards. 4. Non-adjacent state In this state, no fusion is performed; the output is performed directly.

[0060] In some of the solutions mentioned above in this application, a positive defect set is proposed to summarize the fused defects. However, in this process, if the types of fused defects are not distinguished and filtered, redundant defect points may be included, increasing the burden of subsequent processing. In particular, there is a lack of targeted filtering mechanism for fused defects from different sources (such as AI or CV model output), which makes it impossible to effectively optimize the number of defect images.

[0061] Therefore, this application proposes a method for constructing a positive defect set, including the following: For the matching fusion output, which includes fusion defects with AI defect points (i.e., AI+AI, AI+CV fusion in overlapping and intersecting states, and AI+AI fusion in adjacent states), the first AI scale caliper filtering is performed based on the fused contour area. For the matching fusion output, and which does not contain fusion defects with AI defects (i.e., CV+CV fusion in overlapping, intersecting, and adjacent states), no scale filtering is performed; For defects that are directly output without matching and fusion, a second AI scale caliper filter is performed based on the original defect contour; The positive defect set is obtained by summing all defects that meet the first AI scale caliper filter, the second AI scale caliper filter, and those that do not undergo AI scale filtering.

[0062] Different scale caliper filtering is used to filter redundancy. The first AI scale caliper filtering is a size screening mechanism based on the area of ​​the fused contour, targeting fused defects that contain AI defect points after matching and fusion. Its function is that, since AI defect models may be quite sensitive to small or insignificant flaws, this filtering can effectively remove redundant defects that are still small in area after fusion and have no practical reporting value, thereby avoiding the inclusion of too many minor defects in the positive defect set.

[0063] This filtering can be implemented in the following ways: One way is to set a minimum area threshold, and only retain the defect if the outline area of ​​the fusion defect is greater than the threshold. Another way is to define one or more preset caliper shapes (such as rectangles, circles, etc.) and set corresponding size parameters. If the outline of the fusion defect is completely contained within the caliper shape or its area is less than a certain proportion of the caliper shape, then the defect is considered not to need to be reported; otherwise, it is retained.

[0064] Not performing scale filtering means that for fused defects that do not contain AI defect points (i.e., those formed solely by the fusion of CV defect points), no size or area filtering operations are performed. This is because CV defect models typically have high stability and reliability, their detection results are often more practically meaningful, and the fused CV defects usually represent more definitive defect regions. Therefore, directly retaining these fused defects reduces unnecessary computational overhead and ensures that valid information detected by the CV model is not misfiltered, thus maintaining the integrity of the detection.

[0065] The second AI-scale caliper filter is a size screening mechanism based on the original defect contours of defects that are output directly without matching and fusion. These unfused defects may not meet any matching and fusion conditions (e.g., they are in an incoherent state), but may still contain minor or insignificant flaws detected by the AI ​​model. The purpose of this filter is to ensure that these independently existing defects also meet the minimum reporting criteria, preventing excessively small and meaningless defects from being included in the positive defect set.

[0066] This filtering can be implemented in the following ways: one way is to set a minimum size or area threshold, and only retain the original defect profile if its size or area is greater than the threshold; another way is to define one or more preset caliper shapes and their size parameters, and if the original defect profile is completely contained within the caliper shape or its size is less than a certain proportion of the caliper shape, then the defect is considered not to need to be reported, otherwise it is retained.

[0067] All defects processed through the three different filtering paths described above (i.e., fusion defects filtered by the first AI scale caliper, CV+CV fusion defects without scale filtering, and non-fusion defects filtered by the second AI scale caliper) are collected and integrated to form a final, filtered, and optimized defect set. This step is the core of building a positive defect set. It ensures that defects from different sources and in different fusion states are appropriately processed according to their characteristics, ultimately forming a defect list that contains key information while eliminating redundancy, providing an accurate and concise data foundation for subsequent defect analysis and processing.

[0068] In some of the solutions described above in this application, redundant filtering detection is performed on the forward defect set and the reverse defect set to output the target fusion defect. However, in this process, redundant detections with high defect contour overlap may not be effectively filtered out, leading to an increase in the number of subsequent defect images and a decrease in detection efficiency. To address this, this application selects the forward and reverse set cross-matching principle for further screening to obtain the final target fusion defect. The process can be refined as follows: 1. Match the positive defect set and the negative defect set according to the defect coordinates, filter out defects whose defect contour area overlap exceeds the target ratio, and eliminate redundant detections. 2. The remaining defects in the positive defect set are identified as target fusion defects.

[0069] The Singular Value Decomposition (SVD) reverse operation, which extracts a reverse defect set, aims to identify and extract potential defect regions from the image under inspection in a way that complements traditional forward defect detection, forming a "reverse" defect set. Its role is to provide an independent defect detection perspective for subsequent comparison and redundancy filtering with the forward defect set, thereby improving the accuracy and reliability of defect detection. This process can be implemented in various ways. For example, after decomposing the image using SVD, abnormal regions can be identified by analyzing the reconstruction error; alternatively, image processing techniques can be combined to preprocess or post-process the image before and after SVD processing to enhance defect features. Based on the reconstruction error of the SVD algorithm, abnormal regions in the image under inspection can be identified. Singular Value Decomposition (SVD) is a powerful matrix factorization technique that can decompose image data into a set of singular values ​​and corresponding singular vectors. By selecting some singular values ​​for image reconstruction, an approximate image can be obtained.

[0070] Reconstruction error, the difference between the original image and the reconstructed image, effectively reflects abnormal or irregular regions in the image. When defects exist in an image, these defective regions typically lead to larger reconstruction errors and are thus identified as anomalous regions. For example, the norm difference (such as the Frobenius norm) between the original image matrix and the SVD reconstructed image matrix can be calculated, and pixels or regions with differences exceeding a preset threshold can be marked as anomalous; alternatively, threshold segmentation can be performed on the reconstruction error image to extract the contours of anomalous regions.

[0071] It's important to note that SVD inverse operation and forward defect detection target the same area, and SVD inverse operation is not limited to the same layer partition, nor is it restricted by predefined layer partitions (such as metal pads, solder mask areas, substrate areas, and vias). Traditional defect detection may be optimized for specific layers, but this characteristic of SVD inverse operation allows it to detect defects that span different layers or do not belong to any specific layer, thus improving the comprehensiveness of defect detection. For example, SVD inverse operation can be applied directly to the entire image under inspection, regardless of different layer regions; or, SVD decomposition and reconstruction error calculation can be performed on the entire image, and then abnormal regions can be identified based on the magnitude of the reconstruction error, regardless of which layer partition these regions are located in.

[0072] Based on this, the forward and reverse defect sets are matched according to defect coordinates to establish spatial correspondences between defects in the two sets, providing a basis for subsequent redundancy assessment. Precise coordinate matching can identify defects that are close to or overlap in the image, thus determining whether they might point to the same physical defect. Alternatively, by traversing the defects in one set and calculating the distance between its center point or boundary and all defects in the other set, defect pairs with distances less than a preset threshold are matched.

[0073] The core step of redundancy filtering is filtering defects whose contour area overlap exceeds a target ratio. By quantifying the degree of overlap between two matching defect contours, it can be determined whether they are duplicate detections of the same physical defect. When the overlap reaches a preset target ratio, it indicates that the two defects are likely pointing to the same entity, and one of them should be considered redundant. For example, the intersection and union areas of the two defect contours can be calculated, and then the intersection area can be divided by the union area to obtain the Jaccard similarity coefficient. If this coefficient exceeds the target ratio (e.g., 85%), the overlap is considered high. Alternatively, the ratio of the intersection area of ​​the two defect contours to the area of ​​either defect contour can be calculated. If this ratio exceeds the target ratio, the overlap is considered high.

[0074] Subsequently, based on the overlap filtering results, defects deemed redundant are removed from the defect set. The purpose of this step is to reduce duplicate reports, simplify the defect list, and improve the efficiency of subsequent processing. For example, after identifying defect pairs with high overlap, one can be selected to be retained and the other removed based on preset priority rules (e.g., retaining defects with higher confidence, or retaining defects from a specific detection model). Alternatively, all defects deemed redundant can be directly marked or deleted from the positive defect set to ensure they are not included in the final target fusion defect set.

[0075] Finally, after matching, overlap calculation, and redundancy removal, the defects remaining in the positive defect set are officially confirmed as the final target fusion defects. These defects are considered real and non-redundant physical defects. For example, a new defect set can be created, and the defects that were not removed from the positive defect set can be added to this new set one by one as the final target fusion defects. Alternatively, the positive defect set can be directly updated, removing all removed defects, so that the set itself represents the target fusion defects.

[0076] Through the above technical solution, this application effectively solves the problem of redundant detections with high defect contour overlap not being effectively filtered. By introducing precise defect coordinate matching and a quantization filtering mechanism based on contour area overlap, duplicate detections between the forward and reverse defect sets can be accurately identified and eliminated. This significantly reduces the number of defects in the final output, avoids an increase in the number of subsequent defect images due to redundant information, and thus greatly reduces the burden of manual review and analysis. At the same time, since only true and non-redundant defects are retained, the accuracy and reliability of defect reports are improved, thereby enhancing the efficiency and quality of the overall defect detection process.

[0077] In some embodiments, defect filtering is performed based on defect confidence and defect size to obtain a defect set, which may further include the following steps: 1. Determine the uninspected areas of the image to be tested, and perform initial filtering of the original defect points based on the uninspected areas; Determining non-inspection areas in an image under test refers to pre-defining regions within the image that do not require defect detection, based on factors such as product design, functional requirements, or manufacturing processes. Any flaws or anomalies within these regions, even if detected, will not be considered defects affecting product performance or quality. These non-inspection areas can be imported from CAD design drawings, marking specific functional areas (such as pad edges or specific identifier areas) as non-inspection areas; alternatively, they can be identified through manual annotation or machine learning training, identifying unchanging background or structural areas in the image that do not affect product functionality and defining them as non-inspection areas.

[0078] like Figure 6 As shown, this application can divide the entire image into unchecked regions according to function, including AI unchecked regions, CV unchecked regions, and complete (AI+CV) unchecked regions. Figure 6 The square box represents the CV (CV) unchecked area, the diamond box represents the AI ​​(AI) unchecked area, and the center of both boxes represents the completely unchecked area. Similarly, the rectangle in the lower right corner also represents the completely unchecked area. The initial filtering will remove AI defects from the AI ​​unchecked area, CV defects from the CV unchecked area, and all defects from the completely unchecked area. 2. Set the minimum confidence threshold for AI defects and the minimum size gauge threshold for CV defects, and perform secondary filtering on the initially filtered AI defects and CV defects to obtain the defect set.

[0079] Setting a minimum confidence threshold for AI defects aims to filter out defects with low confidence levels output by the AI ​​model. These defects are typically false alarms or uncertain detection results, thereby improving the accuracy of subsequent processing. This threshold can be set to a fixed confidence value (e.g., 0.7 or 0.8) based on historical data analysis or expert experience, filtering out all AI defects below this value. Alternatively, an adaptive threshold setting method can be used to dynamically adjust the minimum confidence threshold based on the overall defect distribution of the current batch of products or the detection environment.

[0080] The minimum size caliper threshold for CV defects aims to filter out defects that are too small in the CV model output. These defects may be image noise or insignificant minor flaws, thus avoiding false positives. This threshold can be set as a minimum area or minimum side length threshold; all CV defects with a defect contour area or minimum bounding rectangle side length smaller than this threshold will be filtered out. Alternatively, one or more size calipers (e.g., minimum width, minimum height, minimum area) can be defined, and CV defects must meet all caliper requirements to be retained.

[0081] After two stages of filtering, a set of high-quality, high-confidence defect points located within the effective detection area is obtained, providing reliable input for subsequent defect fusion steps. This defect set can be formed by merging all AI and CV defect points that have passed the secondary filtering; alternatively, the filtered defect points can be stored in a data structure, such as a list or database table, with each defect point containing its label, confidence level, coordinates, and other information.

[0082] Through the above technical solution, this application effectively solves the problems of insufficient filtering accuracy and inadequate removal of redundant defects in traditional defect filtering methods. By introducing the concept of uninspected areas and performing model-specific segmentation, it achieves accurate exclusion of defects in non-critical areas, avoiding ineffective processing. Simultaneously, by combining the confidence threshold of AI defect points and the size gauge threshold of CV defect points for secondary filtering, it fully considers the characteristics of different detection models, further improving the accuracy and specificity of filtering. This results in a more refined and reliable defect set, significantly reducing the data volume and processing burden of subsequent defect fusion steps, thereby optimizing the efficiency of the entire defect detection process and providing high-quality input for subsequent defect fusion, ultimately improving the yield rate of the material board.

[0083] In some of the solutions described above in this application, a defect fusion method is proposed to reduce the number of defect points and output the target fused defect. However, if the storage method is not optimized when storing these defect images, it may lead to wasted storage space, inconvenient image management, or information loss. For example, storing dense defects individually increases the number of files, while merging storage may confuse label and confidence information, thereby increasing the burden of subsequent review and analysis. Therefore, after outputting the target fused defect, the method in this application further includes: The confirmed target fusion defects are directly extracted and stored one by one according to the set size, and each defect image contains only one defect; the defect image contains defect label and confidence information; Alternatively, the confirmed target fusion defects can be directly extracted into defect images at a set size, and a single defect image contains several defects; the defect image contains defect labels and confidence information for all defects.

[0084] As a specific implementation method, assume that after defect fusion and redundancy filtering are completed, the system identifies a series of target fusion defects. First, the system defines a standard cropping size, such as 256x256 pixels. For a confirmed scratch defect, its defect label is "scratch," with a confidence level of 0.95, and the defect is relatively independent in the image. Then, the system uses the center of the scratch defect as a reference to crop a 256x256 pixel region from the original image to be inspected and saves it as a separate image file, for example, named "scratch_001.jpg." Simultaneously, this image file includes metadata recording "defect label: scratch" and "confidence level: 0.95."

[0085] On the other hand, if the system detects three closely adjacent defects in a certain area, such as a "wear" defect (confidence 0.8), a "bubble" defect (confidence 0.75), and a "foreign object" defect (confidence 0.88), to optimize storage, the system calculates the minimum bounding rectangle of these three defects and, using this as the center, crops a 256x256 pixel image region, saving it as "cluster_001.jpg". In the image's metadata, the system records all information about these three defects, such as "Defect 1: Label - Wear, Confidence -0.8; Defect 2: Label - Bubble, Confidence -0.75; Defect 3: Label - Foreign Object, Confidence -0.88", and may even include their relative coordinates within the 256x256 pixel image.

[0086] Through the above technical solution, this application effectively solves the problems of wasted storage space, inconvenient image management, and information loss caused by suboptimal storage methods in traditional defect detection. By providing two flexible defect image storage modes—single-defect image and multi-defect image—and ensuring the complete recording of defect labels and confidence information in each mode, this solution significantly improves the efficiency and ease of management of defect image storage. This not only reduces the occupation of storage resources but also greatly simplifies the subsequent manual re-inspection and analysis process, making the utilization of defect data more efficient and accurate, thereby improving the overall level of defect detection and quality control.

[0087] Figure 7 This is a structural block diagram of a visual inspection-based defect fusion device provided in an embodiment of this application. The device includes: The defect filtering module 710 is used to acquire the original defect points output by the image to be inspected under at least two defect detection models, and to filter defects according to the defect confidence and defect size to obtain a defect set. The defect fusion module 720 is used to determine the positional relationship between any two defect points among all filtered defect points based on the defect coordinates, and to match and fuse the pair of defect points that satisfy the positional relationship according to their respective defect detection models, and output the fused defect. The redundancy filtering module 730 is used to obtain the reverse defect set extracted from the image to be inspected through the inverse operation of singular value decomposition (SVD), perform redundancy filtering detection on the forward defect set and the reverse defect set, and output the target fusion defect; the forward defect set is constructed based on the filtered unfused defects and the fusion defect.

[0088] It should be noted that the visual detection-based defect fusion device provided in this embodiment is only an example of the above-described division of functional modules / units. In practical applications, the above functions can be assigned to different functional modules / units as needed, that is, the internal structure of the visual detection-based defect fusion device can be divided into different functional modules / units to complete all or part of the functions described above. Furthermore, the implementation methods of the visual detection-based defect fusion method provided in the above method embodiments and the implementation methods of the visual detection-based defect fusion device provided in this embodiment belong to the same concept. For details of the specific implementation process of the visual detection-based defect fusion device provided in this embodiment, please refer to the above method embodiments, which will not be repeated here.

[0089] Figure 8This application provides a structural block diagram of a computer device according to an exemplary embodiment. The computer device can be a desktop computer, laptop computer, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory. The processor and memory can be connected via a bus or other means. The processor can be a central processing unit (CPU) or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a graphics processing unit (GPU), an embedded neural network processor (NPU) or other dedicated deep learning coprocessor, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0090] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0091] Those skilled in the art will understand that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0092] This application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, which, when executed by a processor, implements the methods described in the above-described method embodiments. Those skilled in the art will understand that implementing all or part of the processes in the methods described above can be accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0093] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A defect fusion method based on visual inspection, characterized in that, The method comprises: Obtaining original defect points output by at least two defect detection models respectively from a to-be-inspected image, and performing defect filtering according to defect confidence and defect size to obtain a defect set; Determine the positional relationship between each two defect points in all filtered defect points based on defect coordinates, and match and fuse each two defect points satisfying the positional relationship according to the defect detection model to which the defect points belong, and output a fused defect; Obtain a reverse defect set extracted by a singular value decomposition (SVD) reverse operation on the to-be-inspected image, and perform redundancy filtering detection on the forward defect set and the reverse defect set, and output a target fused defect; the forward defect set is composed based on the filtered unfused defect and the fused defect.

2. The method of claim 1, wherein, The positional relationship between each two defect points includes coincidence, intersection, adjacency and irrelevant relationship; the at least two defect detection models are AI defect model and CV defect model; the matching relationship between each two includes two AI defect point matching, two CV defect point matching, and AI defect point and CV defect point matching; When the two defect contours are in a containing relationship, it is determined that the defect is in a coincidence state; When the two defect contours have a crossing relationship, it is determined that the defect is in an intersection state; When the two defect contours do not contact and the closest distance is less than a distance threshold, it is determined that the defect is in an adjacent state; When the two defect contours do not contact and the closest distance is not less than the distance threshold, it is determined that the defect is in an irrelevant state.

3. The method of claim 2, wherein, In the defect coincidence state, when two AI defect points are matched and fused, or an AI defect point and a CV defect point are matched and fused, the area and defect confidence of the two defect contours are extracted; the defect with the largest contour area is selected as the fused defect, and the defect confidence with the highest confidence value is selected as the fused defect confidence; When two CV defect points are matched and fused, the defect with the largest contour area is selected as the fused defect, and the corresponding CV defect confidence is determined as the fused defect confidence.

4. The method of claim 2, wherein, In the defect intersection state, when two AI defect points are matched and fused, or an AI defect point and a CV defect point are matched and fused, the area and defect confidence of the two defect contours are extracted; the area enclosed by the two defect contours is taken as the fused defect, and the defect confidence with the highest confidence value is selected as the fused defect confidence; When two CV defect points are matched and fused, the area enclosed by the two defect contours is taken as the fused defect, and the CV defect confidence corresponding to the defect with the largest contour area is determined as the fused defect confidence.

5. The method of claim 2, wherein, In the defect adjacent state, it is determined whether the adjacent defects are in the same layer partition; When the adjacent defects are in the same partition and two defect label types consistent CV defect points are matched and fused, the two adjacent defect contours are connected and fused, the area enclosed by the connection region and the two contour regions is determined as the fused defect, and the defect confidence with the highest confidence value is selected as the fused defect confidence; When the adjacent defects are in the same layer partition and two AI defect points are matched and fused, the two adjacent defect contours are connected and fused, the area enclosed by the connection region and the two contour regions is determined as the fused defect, and the defect confidence with the highest confidence value is selected as the fused defect confidence.

6. The method of claim 2, wherein, The forward defect set is constructed by: For matched fusion output, and the fusion defects containing AI defect points, first AI scale calliper filtering is performed based on the area of the fusion contour; For matched fusion output, and the fusion defects not containing AI defect points, no scale filtering is performed; For defects directly output without matched fusion, second AI scale calliper filtering is performed based on the original defect contour; All defects satisfying the first AI scale calliper filtering, the second AI scale calliper filtering, and no AI scale filtering are collected to obtain the positive defect set.

7. The method of claim 1, wherein, The redundancy filtering detection of the positive defect set and the negative defect set is performed to output target fusion defects, including: The positive defect set and the negative defect set are matched according to defect coordinates, defects with an area overlap degree exceeding a target proportion are filtered out, and redundant detection is removed; The remaining defects in the positive defect set are determined as the target fusion defects.

8. A defect fusion apparatus based on visual inspection, characterized by, The device is used for the defect fusion based on visual detection in any one of claims 1-7, and the device comprises: A defect filtering module is configured to obtain original defect points output by at least two defect detection models from a to-be-inspected image, and perform defect filtering according to defect confidence and defect size to obtain a defect set; A defect fusion module is configured to determine a positional relationship between two defect points in all filtered defect points based on defect coordinates, and perform matched fusion on two defect points satisfying the positional relationship according to the defect detection model to which the two defect points belong, and output fusion defects; A redundancy filtering module is configured to obtain a negative defect set extracted from the to-be-inspected image by a singular value decomposition (SVD) inverse operation, perform redundancy filtering detection on the positive defect set and the negative defect set, and output target fusion defects; the positive defect set is composed of the filtered non-fusion defects and the fusion defects.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set, or an instruction set, which are loaded and executed by the processor to implement the defect fusion method based on visual detection in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, which are loaded and executed by the processor to implement the defect fusion method based on visual detection in any one of claims 1-7.

Images