A defect detection method, apparatus, medium, and product

Abstract

The application discloses a defect detection method, device, medium and product. The method comprises the following steps: acquiring a first image of a to-be-detected object, the first image comprising a to-be-detected region of the to-be-detected object, the to-be-detected region comprising a terminal row, and a plurality of conductive particles being distributed in the region of each terminal; performing segmentation on the first image based on an image segmentation mode corresponding to a target defect type, to obtain a plurality of sub-images; wherein the image segmentation mode corresponding to a conductive particle abnormality type of defect is to segment the terminal row by taking a single terminal as a segmentation target; and the segmentation mode corresponding to an apparent defect is to segment the first image into a plurality of local images based on a target size; and judging whether the to-be-detected object is of the target defect type based on the plurality of sub-images. The application performs corresponding segmentation processing modes on the image of the to-be-detected object based on different target defect types, so as to determine whether the to-be-detected object belongs to the target defect type, and realizes differential detection of different defects.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing technology, and in particular to a defect detection method, device, medium and product. Background Technology

[0002] The OLB (Outer Lead Bonding) process in module assembly is primarily used to bond COF (Chip on Film) or FPC (Flexible Printed Circuit) to the terminals of the display panel. At the bonding interface, the terminal area of ​​the display panel and the corresponding pins of the flexible circuit board achieve electrical conductivity through ACF (Anisotropic Conductive Film). Therefore, the adhesion quality of the terminal surface affects the overall functional yield of the module.

[0003] In actual production, the common types of defects that occur in this process section mainly include insufficient conductive particles, indentations on the terminal surface, and foreign matter contamination.

[0004] To address the aforementioned shortcomings, existing technologies often only output an overall score for the entire image, failing to pinpoint which specific terminal is defective. It is also difficult to distinguish whether the defect is due to particle abnormalities, indentations, or foreign matter. This makes it difficult for process engineers to trace the root cause of defects based on the test results and to make targeted adjustments to process parameters, which is detrimental to the rapid improvement of production line yield. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a defect detection method, device, medium and product that can perform corresponding segmentation processing on the image of the object to be tested based on different target defect types, and determine whether the object to be tested belongs to the target defect type based on the segmentation processing results, thereby realizing differentiated detection of different types of defects.

[0006] Firstly, this application provides a defect detection method, the method comprising: Acquire a first image of the object to be detected. The first image includes the area to be detected of the object to be detected. The area to be detected includes a terminal block, and multiple conductive particles are distributed in the area where each terminal is located. Based on the image segmentation method corresponding to the target defect type, the first image is segmented to obtain multiple sub-images; among them, the image segmentation method corresponding to the conductive particle abnormality type defect is: segmenting the terminal block with a single terminal as the segmentation target; the segmentation method corresponding to the appearance defect is: segmenting the first image into multiple local images based on the target size; Based on multiple sub-images, determine whether the object to be detected is of the target defect type.

[0007] In conjunction with the first aspect, in one possible implementation, if the target defect type is a conductive particle anomaly and the sub-image is a single-terminal image, then based on multiple sub-images, it is determined whether the object to be detected belongs to the target defect type, specifically as follows: For each single-terminal image, the sub-regions where multiple conductive particles are located in the corresponding terminal area are determined, and based on the sub-regions where multiple conductive particles are located, at least one conductive particle-related index for the corresponding terminal is determined. The conductive particle-related index is used to characterize the distribution state of multiple conductive particles in the corresponding terminal area. For each terminal, based on at least one conductive particle-related index of the terminal, it is determined whether the terminal has a conductive particle abnormality.

[0008] In conjunction with the first aspect, in one possible implementation, for each terminal, based on at least one conductive particle-related index of the terminal, it is determined whether the terminal has a conductive particle abnormality, specifically as follows: For each terminal, the conductive particle saturation of the terminal is determined based on at least one conductive particle-related index of the terminal; the conductive particle saturation is used to characterize the conductive particle filling density in the area where the terminal is located. For each terminal, based on the corresponding conductive particle saturation, determine whether the terminal has an abnormal conductive particle state.

[0009] In conjunction with the first aspect, in one possible implementation, for each single-terminal image, the sub-regions where multiple conductive particles are located within the corresponding terminal area are determined, specifically as follows: For each single-terminal image, feature enhancement and background suppression are performed to obtain the second image; The second image is processed for background separation and separation of adhered conductive particles to obtain multiple sub-regions where conductive particles are located.

[0010] In conjunction with the first aspect, in one possible implementation, the conductive particle-related indicators include at least one of the following: the number of conductive particles, the total area of ​​multiple conductive particles, the conductive particle coverage, the maximum blank area size, and the conductive particle distribution uniformity. The conductive particle coverage is used to characterize the area ratio of multiple conductive particles in the region where the corresponding terminal is located; the maximum blank area size is used to characterize the maximum continuous missing range of conductive particles in the region where the corresponding terminal is located; and the conductive particle distribution uniformity is used to characterize the degree of uniformity of the distribution of multiple conductive particles in the region where the corresponding terminal is located.

[0011] In conjunction with the first aspect, in one possible implementation, if the defect type to be detected is an appearance defect, then based on multiple sub-images, it is determined whether the object to be detected is of the target defect type, specifically as follows: Multiple local images are input into the target detection model to detect whether there are appearance defects in each local image, and the first coordinates of each appearance defect are obtained. For each first coordinate, a coordinate transformation is performed to obtain the corresponding second coordinate. The coordinate transformation is used to transform the first coordinate from the coordinate system of the local image to the coordinate system of the region to be detected.

[0012] In conjunction with the first aspect, in one possible implementation, the output of the object detection model also includes the confidence score of the second coordinate; therefore, the method further includes: For each terminal, if the terminal has an abnormality in conductive particles and / or a visible defect, then the terminal is unqualified. The process for determining whether a terminal has a visible defect is as follows: determine whether at least one second coordinate falls into the area where the terminal is located, and whether the confidence level of the corresponding second coordinate is greater than the first threshold; if so, then the terminal has a visible defect.

[0013] In conjunction with the first aspect, one possible implementation method also includes: Determine whether the non-compliance conditions are met. If so, the test object is non-compliance. The non-compliance conditions are: at least one local image detects an appearance defect, the number of terminals with abnormal conductive particles is greater than the second threshold, or the number of terminals at a preset position is non-compliance.

[0014] In conjunction with the first aspect, one possible implementation involves inputting multiple local images into the target detection model, specifically: Multiple local images are stitched together to obtain batch tensors; Input batch tensors into the object detection model.

[0015] In conjunction with the first aspect, in one possible implementation, the first image is segmented into multiple local images based on the target size, specifically as follows: Based on the preset overlap rate and target size, the segmentation step size is determined. The overlap rate is used to characterize the size of the overlapping region between adjacent local images; the segmentation step size is used to characterize the movement size of the segmentation positioning point in the first image during each segmentation. The first image is cropped based on the segmentation step size to obtain multiple local images.

[0016] In conjunction with the first aspect, one possible implementation method also includes: Determine the tilt angle of the terminal block, which is the angle between the extension direction of the terminal block and the horizontal axis of the first image; An affine matrix is ​​constructed based on the tilt angle, and an affine transformation is performed on the first image based on the affine matrix so that the extension direction of the terminal block is parallel to the horizontal axis of the first image.

[0017] In conjunction with the first aspect, one possible implementation method also includes: Acquire a standard image, which includes a standard detection region; The offset parameters are determined based on the relative coordinates between the first reference point and the second reference point; the first reference point is located in the standard image outside the standard detection area, and the second reference point is located in the standard detection area. Based on the offset parameters and preset size information, the region to be detected is determined in the first image.

[0018] In conjunction with the first aspect, one possible implementation method also includes: Determine the evaluation results for each terminal and / or the object under test. The evaluation results include at least one of the following: whether it is qualified, the type of target defect, the second coordinate of the apparent defect, the confidence level of the second coordinate, or the conductive particle related index. The evaluation results are labeled at the corresponding terminal of the first image or at the area where each second coordinate is located.

[0019] In a second aspect, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method of any one of the first aspects described above.

[0020] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method of any one of the first aspects above.

[0021] Fourthly, this application also provides a computer program product containing instructions that, when executed, perform any of the methods described in the first aspect above.

[0022] Compared with the prior art, this application has the following beneficial effects: This application addresses conductive particle anomaly defects by refining the segmentation granularity to the individual terminal region to determine whether a single terminal exhibits a conductive particle anomaly. For surface defects, it achieves fine-grained detection of large-area terminal block surfaces based on segmented sub-images. Through different segmentation methods, this application can simultaneously achieve fine-grained detection at the single-terminal level and at the level of surface defects in the tested area, improving the ability to identify different types of defects while avoiding the loss of minute terminal feature details caused by the forced scaling of the entire image in existing whole-image detection techniques. Attached Figure Description

[0023] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 Here is a flowchart of the defect detection method provided in this application, as one embodiment; Figure 2Here is a flowchart of a method for detecting conductive particle anomaly defects provided in this application, as one embodiment. Figure 3 Here is a flowchart of another method for detecting conductive particle abnormality defects provided in this application, as one embodiment; Figure 4 Here is a flowchart of the method for detecting apparent defects provided in this application, as one embodiment; Figure 5 This is a schematic diagram illustrating the process of segmenting the terminal block provided in this application, as one embodiment. Figure 6 This is a schematic diagram illustrating the process of processing a single-terminal image provided in this application, as one embodiment. Figure 7 This is a schematic diagram of the detection process for apparent defects provided in this application, as one embodiment. Figure 8 This application provides a schematic diagram of the process of performing an affine transformation on a first image in one embodiment. Figure 9 This is a schematic diagram of the terminal-level and test area-level evaluation result processing procedures provided in this application, as shown in one embodiment. Figure 10 As one embodiment, this application provides another flowchart of a defect detection method; Figure 11 This is a structural diagram of the computer device provided in this application, as one embodiment. Detailed Implementation

[0024] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments. Furthermore, the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The terms "first" and "second," etc., in the specification and claims of the embodiments of this application are used to distinguish different objects, not to describe a specific order of objects.

[0026] In the OLB (External Wire Bonding) process for modules, flexible circuit boards such as COF or FPC need to be connected to panel terminals to achieve electrical conductivity. This process typically uses anisotropic conductive film (ACF), where uniformly distributed conductive particles within the film are flattened under the pressure and temperature applied by the indenter, forming an electrical path that conducts through the layers. The bonding quality of the terminal surface directly determines the reliability of the conductive contact and the functional yield of the subsequent module. However, three types of defects often occur in actual production: First, insufficient number or sparse distribution of conductive particles in the terminal area may lead to poor soldering, open circuits, or excessive contact resistance; second, abnormal indenter pressure or foreign matter contamination may cause indentation or dents on the terminal surface, affecting long-term stability; and third, the adhesion of foreign matter such as dust, fibers, or residual adhesive may cause short circuits, corrosion, or poor subsequent bonding.

[0027] Currently, there are three main approaches to detecting these defects. The first is traditional manual visual inspection, relying entirely on operators to observe the terminal area under a microscope and judge its compliance based on experience. The second is image processing methods based on fixed thresholds, where parameters such as brightness and contrast are pre-set, and the computer automatically marks areas in the image that do not meet the thresholds as defects. The third is end-to-end deep learning detection, which trains a large neural network to directly analyze the entire high-resolution line scan image and output the location and category of all defects at once. These three methods represent a gradual evolution from manual to automated methods, and from shallow algorithms to intelligent models.

[0028] However, each of these three existing technologies has significant limitations. Manual visual inspection is slow, and inconsistent judgment standards among individuals make it difficult to guarantee stable inspection quality. Fixed threshold methods are too rigid; lighting conditions in the production environment drift slowly over time, and slight changes in camera angles and workpiece placement can cause the same defect to appear differently in images at different times, making the algorithm prone to false positives and false negatives. While end-to-end deep learning offers strong recognition capabilities, the extremely large image size and high resolution of line scans mean that directly feeding the entire image into the network consumes massive amounts of GPU memory, and the computation time far exceeds the production line's allowed cycle time, failing to meet the demands of real-time inspection.

[0029] Therefore, existing technologies cannot simultaneously achieve quantitative determination of the distribution of conductive particles and precise location of indentations and foreign objects while ensuring speed.

[0030] Based on this, embodiments of this application provide a defect detection method, device, medium, and product that can perform corresponding segmentation processing on the image of the object to be tested based on different target defect types, and determine whether the object to be tested belongs to the target defect type based on the segmentation processing results, thereby achieving differentiated detection of different types of defects.

[0031] like Figure 1 As shown, this application provides a defect detection method, the method comprising: S101. Obtain a first image of the object to be detected. The first image includes the area to be detected of the object to be detected. The area to be detected includes a terminal block, and multiple conductive particles are distributed in the area where each terminal is located. The object to be tested can be any one of a display panel, flexible circuit board, or flip-chip film.

[0032] In this embodiment, the first image is an original grayscale image, which is acquired by an imaging device. The imaging device can be a line scan camera or an area scan camera. To meet the high-speed acquisition requirements of continuous movement on the panel production line, a line scan camera is preferred. At the same time, to improve the accuracy of feature extraction, the imaging device can also be used in conjunction with a preset light source structure for image acquisition. The light source structure includes, but is not limited to, a coaxial light source, a strip light source, or a backlight structure, so as to enhance the image contrast of terminal trace morphology and conductive particles by using specific lighting methods.

[0033] The region to be detected refers to the region of interest (ROI) cropped from the first image. This ROI is used to precisely define the area containing the terminal blocks. All subsequent defect detection operations are limited to this ROI to reduce the amount of computational data and focus on the core detection target.

[0034] Terminal blocks refer to a collection of multiple metal conductive pads or pins arranged at regular intervals in an electronic module. One possible implementation, according to this application, is for the OLB process segment, in which the edge of the display panel glass is provided with a plurality of metal solder joints arranged in a predetermined spacing rule, and these metal solder joints constitute the terminal block to be tested.

[0035] S102. Based on the image segmentation method corresponding to the target defect type, the first image is segmented to obtain multiple sub-images; wherein, the image segmentation method corresponding to the conductive particle abnormality type defect is: segmenting the terminal block with a single terminal as the segmentation target; the segmentation method corresponding to the apparent defect is: segmenting the first image into multiple local images based on the target size; One possible implementation is that this application first crops out the region to be detected from the first image, and uses the region to be detected as the object to be segmented to obtain multiple sub-images.

[0036] Apparent defects refer to surface deformation or abnormal attachments on the surface of the terminal to be tested or the area to be tested, which can generally be perceived through visual inspection methods; including but not limited to: indentations and pits formed on the terminal surface, foreign matter attached to the terminal or adjacent area, and residual adhesive overflowing during the application or hot pressing of anisotropic conductive adhesive (ACF).

[0037] One possible implementation is, such as Figure 5 As shown, the terminal block is divided with a single terminal as the dividing target, specifically as follows: The grayscale normalization process is performed on the area to be detected to eliminate brightness differences between different panels or at different acquisition times; The normalized image is then subjected to background smoothing. Background smoothing can be achieved using any of the following methods: mean filtering, Gaussian filtering, or morphological opening operation, in order to suppress high-frequency noise and fine textures on the terminal surface. Thresholding segmentation is performed on the smoothed image. Thresholding segmentation can use any of the following: global threshold, adaptive threshold, or local threshold, to initially separate the terminal region from the background and obtain candidate terminal regions. Based on morphological closing operations, holes and fractures in candidate terminal regions are filled and connected. By combining any one of connected component analysis, horizontal projection, or contour filtering, a binary mask corresponding to each individual terminal is extracted from the candidate terminal region. In the binary mask, 1 represents the pixels of the effective area of ​​the terminal, and 0 represents the background area outside the terminal.

[0038] It is worth mentioning that, in order to adapt to the physical specifications of terminals of different models (such as terminal blocks of different sizes and spacings on different models of panels), during the process of extracting terminal masks, filtering constraints can be set on the geometric features such as the area, aspect ratio, orientation angle and adjacent spacing of the terminal connected domain to eliminate false detection areas and retain the real terminals.

[0039] Based on each terminal mask, its minimum bounding rectangle is calculated, and the terminals are sequentially numbered according to a preset numbering rule. The numbering is preferably performed in the order from left to right along the X-axis of the first image, or it can be performed according to the logical order defined by the product process, so as to provide a terminal index basis for subsequent defect attribution.

[0040] Based on the position information of the outer frame of each terminal, sub-images containing only a single terminal (i.e., single-terminal images) are segmented one by one from the grayscale image of the area to be inspected. Simultaneously, during the segmentation process, the mapped coordinates of this single-terminal image within the area to be inspected (which can be the offset of the upper left corner of the terminal's outer frame relative to the origin of the coordinate system in the area to be inspected) are preserved. This allows for the accurate mapping of inspection results or defect coordinates back to the area to be inspected, used for defect attribution determination and result visualization. In this way, the entire terminal block image can be completely segmented into multiple single-terminal images with mapped coordinates.

[0041] One possible implementation is, such as Figure 7 As shown, the first image is segmented into multiple local images based on the target size, specifically as follows: Based on the preset overlap rate and target size, the segmentation step size is determined. The overlap rate is used to characterize the size of the overlapping region between adjacent local images; the segmentation step size is used to characterize the movement size of the segmentation positioning point in the first image during each segmentation. The segmentation positioning point is specifically the reference coordinate of the starting position when capturing a local image during the segmentation process. It is usually the top left pixel of the local image to be segmented.

[0042] like Figure 7 As shown, the overlapping area can be the shaded area, and the segmentation point can be the upper left corner of the shaded area.

[0043] The first image is cropped based on the segmentation step size to obtain multiple local images.

[0044] In one possible implementation, the target size is a preset window size. This application performs fixed overlap rate sliding window clipping on the area to be tested based on the preset window size. The window size can be set according to the input specifications of the subsequent defect detection model, for example, it can be set to 512×512, 640×640, 768×768 or 1024×1024; the overlap rate is preferably 10% to 50% to avoid missed detection caused by defects falling on the edge of the window.

[0045] In one possible implementation, the segmentation step size is calculated as follows: Step size = Window size × (1 - overlap ratio) For example, when the window size is set to 640×640 pixels and the window overlap rate is 10%, the calculated segmentation step size is 640×(1-0.1)=576 pixels.

[0046] S103. Based on multiple sub-images, determine whether the object to be detected is of the target defect type.

[0047] One possible implementation is, such as Figure 2 As shown, if the target defect type is conductive particle anomaly and the sub-image is a single-terminal image, then based on multiple sub-images, it is determined whether the object to be detected belongs to the target defect type, specifically as follows: S201. For each single-terminal image, determine the sub-regions where multiple conductive particles are located in the corresponding terminal area, and based on the sub-regions where multiple conductive particles are located, determine at least one conductive particle related index for the corresponding terminal. The conductive particle related index is used to characterize the distribution state of multiple conductive particles in the corresponding terminal area. One possible implementation is, such as Figure 6 As shown, for each single-terminal image, the sub-regions where multiple conductive particles are located within the corresponding terminal area are determined, specifically: For each single-terminal image, feature enhancement and background suppression are performed to obtain the second image; Specifically, feature enhancement can employ one or more of the following methods: top-hat transform, differential Gaussian, high-pass filtering, local contrast enhancement, or frequency domain enhancement.

[0048] The second image is processed for background separation and separation of adhered conductive particles to obtain multiple sub-regions where conductive particles are located.

[0049] In one possible implementation, a binary separation operation is performed on the second image to separate the conductive particles from the terminal background. Morphological processing is then applied to the binarized result to disconnect the adhered conductive particles, thereby obtaining connected components that characterize individual conductive particles. Based on the extracted connected components, relevant indices for the conductive particles are further calculated.

[0050] One possible implementation involves conductive particle-related metrics including the number of conductive particles N and the total area A of the multiple conductive particles. p The conductive particle coverage rate (C), the maximum blank area size (G), and the conductive particle distribution uniformity (U) are at least one of the following: the conductive particle coverage rate is used to characterize the area ratio of multiple conductive particles in the region where the corresponding terminal is located; the maximum blank area size is used to characterize the maximum continuous missing range of conductive particles in the region where the corresponding terminal is located; and the conductive particle distribution uniformity is used to characterize the degree of uniformity of the distribution of multiple conductive particles in the region where the corresponding terminal is located.

[0051] One possible implementation is that the conductive particle coverage C=A p / A t ; Among them, A t This indicates the effective area of ​​the region where the terminal is located.

[0052] The uniformity of conductive particle distribution U is obtained based on the consistency of conductive particle distribution in multiple sub-regions within the terminal. The maximum blank area G is determined by the maximum connected blank area where conductive particles are missing, or by the area ratio of the maximum blank area in the effective area of ​​the terminal.

[0053] S202. For each terminal, based on at least one conductive particle-related index of the terminal, determine whether the terminal has a conductive particle abnormality.

[0054] One possible implementation is to construct a decision function to determine whether the terminal has an abnormal conductive particle pattern: F=f(N, C, U, G) Threshold-based rule-based determination: when N≥T N C≥T C U≥T U And G≤TG If the terminal is selected as OK, it is considered NG; otherwise, it is considered abnormal. N T C T U and T G All of these are preset standard thresholds for the relevant indicators of conductive particles; One possible implementation is, such as Figure 3 As shown, for each terminal, based on at least one conductive particle-related index of the terminal, it is determined whether the terminal has a conductive particle abnormality, specifically as follows: S301. For each terminal, the conductive particle saturation of the terminal is determined based on at least one conductive particle-related index of the terminal; the conductive particle saturation is used to characterize the conductive particle filling density in the area where the terminal is located. One possible implementation is that the saturation of conductive particles is in the following linear weighted form: F=W1·N'+ W2·C + W3·U - W4·G Where N' is the normalized number of conductive particles, and W1, W2, W3 and W4 are preset weight values.

[0055] S302. For each terminal, based on the corresponding conductive particle saturation, determine whether the terminal has an abnormal conductive particle state.

[0056] One possible implementation is when F≥T F When the terminal is selected, it is considered OK; otherwise, it is considered NG. F The preset fullness threshold; Based on the above judgment process, the saturation result of conductive particles at the terminal level, the relevant index values ​​of each conductive particle, and the corresponding terminal number can be output, which facilitates process personnel to judge the type and severity of the abnormality.

[0057] One possible implementation is, such as Figure 4 As shown, if the defect type to be detected is an appearance defect, then based on multiple sub-images, it is determined whether the object to be detected is of the target defect type, specifically: S401. Input multiple local images into the target detection model, detect whether there are appearance defects in each local image, and obtain the first coordinates of each appearance defect; For example, object detection models may employ RT-DETR, YOLO series networks, Mask R-CNN, segmentation-based networks, or other equivalent network structures.

[0058] The first coordinate refers to the coordinate of the detection box of the appearance defect, which corresponds to the local coordinate system of the local image; One possible implementation is, such as Figure 7As shown, multiple local images are input into the object detection model, specifically: Multiple local images are stitched together to obtain batch tensors; Input batch tensors into the object detection model.

[0059] Specifically, normalization, size adaptation, and channel transformation are performed on each local image, which is then converted into Tensor format and concatenated along the batch dimension dim=0 to obtain a batch tensor X∈R^{K×C×H×W}, where K is the number of local images.

[0060] By feeding a batch of tensors X into the object detection model for forward inference, candidate detection boxes, defect category labels, and confidence scores are obtained for each local image. This batch input method can significantly reduce the system overhead caused by individual inference for each local image while ensuring detection coverage, making it suitable for online deployment of ultra-long line scans.

[0061] S402. One possible implementation is that the target detection model also performs coordinate transformation on each first coordinate to obtain the corresponding second coordinate. The coordinate transformation is used to transform the first coordinate from the coordinate system of the local image to the coordinate system of the region to be detected.

[0062] Based on the top-left corner offset of each local image, the detection box in the coordinate system of the local image is mapped back to the coordinate system of the region to be detected. The offset refers to the coordinates (Δx) formed by the horizontal distance and vertical translation distance of the top-left corner pixel of the k-th local image relative to the origin of the top-left corner of the entire region to be detected (ROI). k , Δy k ).

[0063] For example, if the coordinates of the detection box are (x1, y1, x2, y2), then the mapped coordinates can be written as (x1 + Δx). k , y1+Δy k , x2+Δx k , y2+Δy k ).

[0064] In one possible implementation, this application also performs deduplication fusion on the detection boxes that are repeatedly detected across the sliding window. Specifically, duplicate boxes can be processed by using IoU (Intersection over Union) thresholding, non-maximum suppression, weighted box fusion, or retaining the highest confidence box.

[0065] This application uses coordinate mapping, window deduplication, and terminal attribution determination to accurately locate the detection box output by deep learning to a specific single terminal, refining the detection result from "overall abnormality" to "abnormality of a certain terminal". This makes it easier for process personnel to directly locate the defective workstation, improving the efficiency of locating apparent defects and data traceability.

[0066] One possible implementation is that the output of the object detection model also includes the confidence score of the second coordinate; in this case, the method would also include: For each terminal, if the terminal has an abnormality in conductive particles and / or a visible defect, then the terminal is unqualified. The process for determining whether a terminal has a visible defect is as follows: determine whether at least one second coordinate falls into the area where the terminal is located, and whether the confidence level of the corresponding second coordinate is greater than the first threshold; if so, then the terminal has a visible defect.

[0067] To determine whether at least one second coordinate falls within the area where the terminal is located, any one of the following methods can be used as the basis for judgment: intersection analysis, center point assignment analysis, or nearest neighbor assignment analysis. If the intersection area, intersection ratio, or center point position between the detection frame and a terminal mask meets the preset attribution rules, the defect is identified as belonging to that terminal, and the terminal number is recorded. The method for determining the terminal mask and the terminal number can refer to the corresponding description of segmentation and numbering of single terminal images above.

[0068] Based on this, for each terminal that has been judged, if the result of the conductive particle saturation test is NG, then at least one conductive particle abnormality mark is attached to the terminal; at the same time, for surface defects such as indentations, foreign objects or residual adhesive, if it is confirmed after the above attribution judgment that they do not fall into the effective area of ​​any terminal, then their conclusion can be recorded as background abnormality or boundary abnormality, which can be used as the basis for subsequent evaluation of the entire area to be tested.

[0069] One possible implementation method also includes: Determine whether the non-compliance conditions are met. If so, the test object is non-compliance. The non-compliance conditions are: at least one local image detects an appearance defect, the number of terminals with abnormal conductive particles is greater than the second threshold, or the number of terminals at a preset position is non-compliance.

[0070] One possible implementation method also includes: Determine the tilt angle of the terminal block, which is the angle between the extension direction of the terminal block and the horizontal axis of the first image; In one possible implementation, the first image also includes at least one pair of Mark tags. Taking a pair of Mark tags as an example, such as... Figure 8 As shown, the process for determining the tilt angle is as follows: Establish a local coordinate system, with the origin of which can be either the left or right Mark. The reference coordinate system is established with the direction of the line connecting the left and right Marks as the reference axis (usually the local X-axis).

[0071] When the left Mark is taken as the origin, the vector e = (x) pointing from the left Mark to the right Mark will be... R -x L , y R -y L If the x-axis is used as the reference direction of the local coordinate system, then: Inclination angle θ = arctan((y R -y L ) / (x R -x L )) An affine matrix is ​​constructed based on the tilt angle, and an affine transformation is performed on the first image based on the affine matrix so that the extension direction of the terminal block is parallel to the horizontal axis of the first image.

[0072] In the formula, the coordinates of the left Mark center (x) L , y L ) and the center coordinates of the right Mark (x) R , y R The process of determining the Mark is as follows: perform template matching within the preset search area to obtain the Mark candidate box, then binarize the candidate box area and calculate its geometric center to obtain the coordinates of the Mark center point; Another possible implementation is to calculate the Mark center coordinates based on edge feature extraction, corner detection, circle center fitting, or keypoint network prediction.

[0073] One possible implementation is that the rotation matrix M can be written as: [[cosθ, -sinθ, (1-cosθ)x] c + sinθ·y c ], [sinθ, cosθ, (1-cosθ)y c - sinθ·x c ]).

[0074] When |θ| exceeds a preset threshold, rotation correction is performed on the first image so that the ROI region can be determined based on the rotation-corrected first image.

[0075] For example, when |θ|>θmin (θ can take values ​​from 0.02° to 0.10°), construct the rotational affine matrix M: [cosθ sinθt x ] [sinθcosθty ] Among them, (t) x ,t y The rotation center is kept at the midpoint of the two Marks; an affine transformation is performed on the first image to obtain a corrected image, which makes the extension direction of the terminal block parallel to the horizontal axis of the first image.

[0076] By using the above-mentioned rotational correction method, a stable and consistent area to be inspected can be obtained without significantly improving the installation accuracy of the fixture or camera, providing a unified input for subsequent inspections.

[0077] In another possible implementation, the entire image can be left unrotated, and the coordinates of the region to be detected can be calculated directly in the local coordinate system.

[0078] Acquire a standard image, which includes a standard detection region; Specifically, the standard image is a qualified sample grayscale image that includes terminal blocks and alignment marks.

[0079] The offset parameters are determined based on the relative coordinates between the first reference point and the second reference point; the first reference point is located in the standard image outside the standard detection area, and the second reference point is located in the standard detection area. Specifically, the first reference point can be the origin of the global coordinate system of the standard image (i.e., the top-left pixel of the standard image); the second reference point can be the top-left vertex of the bounding rectangle of the standard detection area. The offset parameter is the pixel coordinate difference between the first and second reference points, calculated by subtracting the coordinate value of the second reference point from the coordinate value of the first reference point. The offset parameter can be stored and switched according to camera type, panel product model, or process formula.

[0080] The following example illustrates the process of determining the region to be detected: Let the center point of the line connecting the left and right Marks be (x... c , y c ), combined with the pre-calibrated offset (Δx, Δy) and ROI dimensions (W) of this model ROI H ROI This allows for the location and generation of the region to be detected. If an overall rotation correction strategy is adopted, the ROI is directly cropped on the corrected image; if the entire image is not rotated for computational efficiency, the tilt angle θ and the affine matrix are used to calculate the real pixel coordinates of the four corner points of the region to be detected on the first image, and the region to be detected is cropped from the first image.

[0081] This application corrects the first image using alignment marks, which can effectively eliminate the positioning deviation of the detection area caused by changes in workpiece posture. Specifically, after correction, it can ensure that the detection area of ​​each frame image is always determined based on the same reference. Whether the same panel is photographed multiple times or multiple different panels are processed continuously, the determined detection area can be stably locked in the same physical position of the terminal block. Moreover, no matter if the workpiece shifts or tilts slightly in the field of view, the ROI box can be accurately pressed onto the corresponding terminal block, thereby reducing the probability of false detection and missed detection.

[0082] One possible implementation method also includes: Determine the evaluation results for each terminal and / or the object under test. The evaluation results include at least one of the following: whether it is qualified, the type of target defect, the second coordinate of the apparent defect, the confidence level of the second coordinate, or the conductive particle related index. The evaluation results are labeled at the corresponding terminal of the first image or at the area where each second coordinate is located.

[0083] Specifically, in the defect detection of conductive particle abnormality type, the evaluation results include at least one of the following: whether the terminal is qualified, terminal number, defect type, conductive particle fullness, conductive particle related indicators, timestamp when the defect was detected, barcode of the object to be tested, machine number, and formula version. In the detection of apparent defects, the evaluation results include at least one of the following: whether the area to be detected is qualified, the type of defect, the second coordinate of the detection box corresponding to the apparent defect, the confidence level, the terminal number where the apparent defect is located, the timestamp when the defect was detected, the version of the target detection model, the barcode of the object to be tested, the machine number, and the recipe version. One possible implementation is, such as Figure 9 As shown, this application also integrates with a PLC / MES and stores the results. The integration method can be through TCP / IP (Transmission Control Protocol / Internet Protocol), OPC UA (Open Platform Communications Unified Architecture), Modbus (Modicon Bus), IO (Input / Output) hardwired triggering, or other industrial communication protocols, to interact with the PLC (Programmable Logic Controller) and MES (Manufacturing Execution System). The stored content may include original drawings, annotated drawings, defect screenshots, log files, and statistical reports.

[0084] In one possible implementation, this application also visualizes the evaluation results and archives the first image, a screenshot of the defective area, and log files.

[0085] This application, by linking with PLC / MES and storing the results, can support engineering scenarios such as production line alarms, sorting, shutdown, re-judgment, and statistical analysis while maintaining the traceability of the results.

[0086] like Figure 10 The diagram shown is another flowchart of the defect detection method of this application. The method includes: S1001. Determine the center coordinates of the left and right Mark markers; S1002. Determine the tilt angle based on the center point coordinates of the left and right Mark markers; S1003. Perform rotation correction on the first image based on the tilt angle; S1004. Determine the region to be detected in the corrected first image; If the target defect type is an abnormal conductive particle, then jump to S1005~S1010; if the target defect type is an apparent defect, then jump to S1011~S1015. S1005. Perform binarization and morphological processing on the area to be detected to obtain multiple single-terminal images; S1006. For each single-terminal image, perform binarization and morphological processing on the single-terminal image to obtain the mask of each conductive particle in the single-terminal image. S1007. Determine the corresponding minimum bounding rectangle based on the mask of each conductive particle, and determine multiple conductive particle-related indices for each terminal based on the minimum bounding rectangle or morphological fitting. S1008. Based on multiple conductive particle-related indicators corresponding to each terminal, determine the conductive particle saturation of the corresponding terminal. S1009. Determine whether the saturation of conductive particles is greater than the saturation threshold. S1010: If yes, the corresponding terminal is OK; otherwise, the corresponding terminal is NG. S1011. Based on the target size and the preset overlap rate, the region to be detected is segmented to obtain multiple local images; S1012. Convert each local image into Tensor format, and then concatenate them according to the dimension with dim=0 to obtain a batch of tensors; S1013. Input the batch tensors into the target detection model; S1014. Determine whether at least one local image has an apparent defect; S1015. If yes, the object to be tested is OK; otherwise, the object to be tested is NG.

[0087] It should be noted that although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0088] On the other hand, this application also provides a computer-readable storage medium, which may be included in a computer device or exist independently without being assembled into the computer device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the methods described in this application. For example, it may execute... Figure 1 The steps of the method shown are as follows.

[0089] In another aspect, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements... Figure 1 The steps of the method shown are as follows.

[0090] On the other hand, this application also provides a computer-readable storage medium, which may be included in a computer device or exist independently without being assembled into the computer device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the methods described in this application. For example, it may execute... Figure 1 The steps of the method shown.

[0091] This application provides a computer program product including instructions that, when executed, cause the method described in this application to be performed. For example, it can execute... Figure 1 The steps of the method shown are as follows.

[0092] The following is for reference. Figure 11 , Figure 11 A schematic diagram of the structure of a computer device suitable for implementing the embodiments of this application is shown. like Figure 11As shown, the computer system includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1102 or programs loaded from storage section 1108 into random access memory (RAM) 1103. RAM 1103 also stores various programs and data required for the system's operating instructions. CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0093] The following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed so that computer programs read from them can be installed into storage section 1108 as needed.

[0094] Specifically, according to embodiments of this application, Figure 1 The described process can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program contains program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by central processing unit (CPU) 1101, it performs the functions defined in the system of this application.

[0095] On another front, this application also provides a computer-readable storage medium, which may be included in an electronic device or exist independently without being assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the methods of this application. For example, it may execute... Figure 1 The steps of the method shown are as follows.

[0096] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0097] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A defect detection method, characterized in that, The method includes: A first image of the object to be detected is acquired. The first image includes the area to be detected of the object to be detected. The area to be detected includes a terminal block, and multiple conductive particles are distributed in the area where each terminal is located. Based on the image segmentation method corresponding to the target defect type, the first image is segmented to obtain multiple sub-images; wherein, the image segmentation method corresponding to the conductive particle abnormality type defect is: segmenting the terminal block with a single terminal as the segmentation target; the segmentation method corresponding to the appearance defect is: segmenting the first image into multiple local images based on the target size; Based on the multiple sub-images, it is determined whether the object to be detected is of the target defect type.

2. The method according to claim 1, characterized in that, If the target defect type is a conductive particle anomaly, and the sub-image is a single-terminal image, then based on the multiple sub-images, it is determined whether the object to be detected belongs to the target defect type, specifically as follows: For each single-terminal image, a sub-region is determined within the region where the corresponding terminal is located, and based on the sub-regions where the multiple conductive particles are located, at least one conductive particle-related index is determined for the corresponding terminal. The conductive particle-related index is used to characterize the distribution state of the multiple conductive particles in the region where the corresponding terminal is located. For each terminal, based on at least one conductive particle-related index of the terminal, it is determined whether the terminal has a conductive particle abnormality.

3. The method according to claim 2, characterized in that, For each terminal, based on at least one conductive particle-related index of the terminal, it is determined whether the terminal has a conductive particle abnormality, specifically as follows: For each terminal, the conductive particle saturation of the terminal is determined based on at least one conductive particle-related index of the terminal; the conductive particle saturation is used to characterize the conductive particle filling density in the region where the terminal is located. For each terminal, based on the corresponding conductive particle saturation, it is determined whether the terminal has an conductive particle abnormality.

4. The method according to claim 3, characterized in that, For each single-terminal image, determine the sub-regions where multiple conductive particles are located within the corresponding terminal area, specifically: For each single-terminal image, feature enhancement and background suppression are performed on the single-terminal image to obtain a second image; The second image is subjected to background separation and adhesion conductive particle separation processing to obtain multiple sub-regions where conductive particles are located.

5. The method according to any one of claims 2 to 4, characterized in that, The conductive particle related indicators include at least one of the following: number of conductive particles, total area of ​​multiple conductive particles, conductive particle coverage, maximum blank area size, and conductive particle distribution uniformity. The conductive particle coverage is used to characterize the area ratio of the multiple conductive particles in the region where the corresponding terminal is located; the maximum blank area size is used to characterize the maximum continuous missing range of conductive particles in the region where the corresponding terminal is located; and the conductive particle distribution uniformity is used to characterize the degree of uniformity of the distribution of multiple conductive particles in the region where the corresponding terminal is located.

6. The method according to claim 2, characterized in that, If the defect type to be detected is an appearance defect, then based on the multiple sub-images, it is determined whether the object to be detected is the target defect type, specifically as follows: The multiple local images are input into the target detection model to detect whether there are appearance defects in each local image, and the first coordinates of each appearance defect are obtained. For each first coordinate, a coordinate transformation is performed to obtain the corresponding second coordinate. The coordinate transformation is used to transform the first coordinate from the coordinate system of the local image to the coordinate system of the region to be detected.

7. The method according to claim 6, characterized in that, The output of the target detection model also includes the confidence score of the second coordinate, and the method further includes: For each terminal, if the terminal has an abnormality in conductive particles and / or the terminal has an apparent defect, then the terminal is unqualified; the process for determining whether the terminal has an apparent defect is as follows: determine whether at least one second coordinate falls into the area where the terminal is located, and the confidence level of the corresponding second coordinate is greater than a first threshold; if so, the terminal has an apparent defect.

8. The method according to claim 7, characterized in that, The method further includes: Determine whether the non-compliance condition is met. If so, the test object is non-compliance. The non-compliance condition is any one of the following: at least one of the local images detects an appearance defect, the number of terminals with abnormal conductive particles is greater than a second threshold, or the number of terminals at a preset position is non-compliance.

9. The method according to claim 6, characterized in that, The multiple local images are input into the target detection model, specifically as follows: The multiple local images are stitched together to obtain a batch tensor; The batch tensor is input into the target detection model.

10. The method according to claim 1, characterized in that, The first image is segmented into multiple local images based on the target size, specifically as follows: Based on the preset overlap rate and the target size, the segmentation step size is determined. The overlap rate is used to characterize the size of the overlapping region between adjacent local images. The segmentation step size is used to characterize the movement size of the segmentation positioning point in the first image during each segmentation. The first image is cropped based on the segmentation step size to obtain multiple local images.

11. The method according to claim 1, characterized in that, The method further includes: Determine the tilt angle of the terminal block, wherein the tilt angle is the angle between the extension direction of the terminal block and the horizontal axis of the first image; An affine matrix is ​​constructed based on the tilt angle, and an affine transformation is performed on the first image based on the affine matrix so that the extension direction of the terminal block is parallel to the horizontal axis of the first image.

12. The method according to claim 1, characterized in that, The method further includes: Acquire a standard image, wherein the standard image includes a standard detection region; The offset parameter is determined based on the relative coordinates between the first reference point and the second reference point; the first reference point is located in the standard image outside the standard detection area, and the second reference point is located in the standard detection area. Based on the offset parameters and preset size information, the region to be detected is determined in the first image.

13. The method according to claim 8, characterized in that, The method further includes: Determine the evaluation result for each terminal and / or the object under test, the evaluation result including whether it is qualified, the target defect type, the second coordinate of the apparent defect, the confidence level of the second coordinate, or at least one of the conductive particle related indicators; The evaluation results are marked at the corresponding terminal of the first image or at the region where each second coordinate is located.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the method of any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 13.

16. A computer program product comprising instructions that, when the instructions are executed, perform the method of any one of claims 1 to 13.

Images