A wire bonding defect detection and pattern recognition method based on image-point cloud information fusion

Abstract

The application provides a soldering wire defect detection and mode recognition method based on image-point cloud information fusion, which can detect defective soldering wires in a PCB and judge the defect modes of the defective soldering wires. The method belongs to the field of defect detection in the lead soldering manufacturing process and aims to solve the problems of high labor intensity and low detection efficiency in the current defect detection methods. The specific detection process comprises the following steps: obtaining a depth image of a PCB, judging the soldering wire contour in the depth image through an OBB bounding box and a MAD algorithm and extracting the depth data in the contour, converting the depth data into point clouds, and obtaining the point cloud data of the soldering wire; pre-processing the point clouds and performing surface segmentation, so that the accurate extraction of the soldering wire surface is completed; a defect classification model is constructed, the detection of the soldering wire defects and the recognition of the defect modes are completed; and finally, the curvature of the normal defective soldering wire is calculated, and the curvature evaluation of the normal soldering wire is completed.

Description

Technical Field

[0001] This invention belongs to the field of defect detection in the lead wire bonding manufacturing process. Specifically, it relates to a method for detecting and determining the defect pattern type of wire bonding structural defects using an image-point cloud information fusion method. Background Technology

[0002] In critical semiconductor manufacturing processes, wire bonding quality inspection still presents several unresolved issues. Wire bonding is a process that uses metal leads to connect a first solder joint to a corresponding second solder joint. Wire bonding structural defects are a major factor affecting product functionality and yield. Detecting structural defects in wire bonding not only reveals quality problems but also allows for defect pattern recognition, qualitatively assessing faults in the process flow and facilitating process improvements by engineers. Currently, wire bonding defect detection mainly falls into two categories: manual visual inspection of structural defects and image processing for detecting wire breakage defects. Manual visual inspection is inefficient, slow, and labor-intensive, while image processing can only detect a limited range of defect patterns. Therefore, there is an urgent need for a method that can automatically detect wire bonding structural defects and identify defect patterns.

[0003] Currently, image processing-based methods, such as threshold segmentation, region growing, and connected component detection, are used for wire bonding defect detection. These methods rely on traditional cameras to acquire planar images and can only detect wire breakage defects. They cannot detect other types of defects or identify defect patterns, such as bending, collapse, and abnormal curvature. Summary of the Invention

[0004] The main objective of this invention is to propose a method for wire bonding defect detection and pattern recognition based on image-point cloud information fusion. This method can be applied to the wire bonding manufacturing process, replacing the current manual visual inspection method, enabling automatic machine detection of wire bonding quality problems in products, and classifying defective wires based on pattern recognition. This invention aims to solve the following problems:

[0005] 1. Wire bonding defect detection is mainly done by manual visual inspection, which is inefficient, slow, and labor-intensive;

[0006] 2. Currently, wire bonding defect detection technology can only detect broken wires and cannot detect various types of defects such as wire bending and wire collapse.

[0007] 3. Existing wire bonding defect detection methods cannot identify multiple defect patterns, and cannot provide quantitative evaluation indicators for normal wire bonding, nor can they calculate the curvature of the wire bonding.

[0008] To address the above issues, this invention proposes a method for weld wire defect detection and pattern recognition based on image-point cloud information fusion. This method first performs coarse localization of the weld wire in the depth map, extracts the point cloud information of the weld wire, and then performs planar segmentation on the weld wire point cloud to obtain a complete weld wire plane. Defects such as broken wires, collapsed wires, and bends are detected on the segmented weld wire plane, and the spatial curvature and the curvature of the fitted mathematical model are calculated for normal weld wires.

[0009] The specific workflow of this invention is as follows:

[0010] Step 1: Use a 3D laser contour sensor to scan the entire PCB board and obtain a depth image;

[0011] Step 2: Using image enhancement, thresholding, and contour detection methods, the PCB board image and background image are segmented to obtain the depth image of the PCB board;

[0012] Step 3: Perform image enhancement and threshold segmentation on the PCB board depth map, perform contour detection on the threshold segmented image, traverse and calculate the OBB bounding box of each contour and obtain the aspect ratio of the bounding box, and use the absolute median difference (MAD) algorithm to distinguish the solder wire contour from other contours.

[0013] Step 4: Extract depth data within the weld wire contour, perform matrix operations to obtain point cloud data of the weld wire, and preprocess the point cloud data using statistical filtering and PCA affine transformation.

[0014] Step 5: Perform surface segmentation on the preprocessed wire bonding data to separate the wire bonding surface data and redundant data, and obtain complete wire bonding point cloud data.

[0015] Step 6: Project the weld wire point cloud data constructed in Step 5 onto the XZ axis, calculate the interval between adjacent points after projection, and use the 3-sigma algorithm to filter out broken wires within the line to obtain a non-broken weld wire point cloud dataset.

[0016] Step 7: Perform FPS downsampling on the weld wire point cloud data constructed in Step 6, estimate the normal vector of each point in the downsampled data, construct a defect classification model, and complete the identification of specific weld wire defect patterns.

[0017] Step 8: Calculate the spatial curvature of the normal bonding wire in Step 7, project the bonding wire data onto the XZ axis, fit the mathematical model of the bonding wire using the least squares method, calculate the curvature of the fitted mathematical model, and complete the evaluation of the curvature of the normal bonding wire.

[0018] In step 3, the calculation of the OBB bounding box aspect ratio and the screening of the solder wire contour are as follows: The PCB depth image is subjected to contrast enhancement and sharpening, and the enhanced image is thresholded for segmentation. Contour detection is performed on the binarized image after thresholding. Principal component analysis (PCA) is used to calculate the eigenvalues ​​and eigenvectors of each contour data set. The eigenvector corresponding to the largest eigenvalue is determined as the X-axis, and the contour data is affinely transformed onto the new principal axis. The maximum and minimum values ​​of the X-axis and Y-axis in the affinely transformed data are calculated, and the bounding rectangle of the contour is fitted. The aspect ratio of the rectangle is calculated and denoted as D. i Iterate through all contours and calculate their aspect ratios to generate an aspect ratio data set D.

[0019] Calculate the median of the dataset D, denoted as Dmedian. median Subtract the median from the data set D to obtain a new set D', and calculate the median of D', denoted as D''. median D satisfies formula (1) i This represents the aspect ratio of the weld wire outline.

[0020] D i -D median >8*D' median (1)

[0021] In step 5, the preprocessed point cloud data from step 4 is sliced ​​along the X-axis, with the slice thickness equal to the minimum interval between adjacent points on the X-axis. All spatial points belonging to the same X-scale are extracted. These spatial points are distributed in the same YZ-axis plane. Histograms of the extracted points at different heights on the Z-axis are plotted, and the point with the highest number of points at the same height is designated as the center point.

[0022] Iterate through the points and calculate the distance l between the center point Pn and all other points Pi. i Y-axis angle α i The angle θ between the vector and the normal vector i Generate a descriptor subset (l i ,α i ,θ i If any three descriptors in Pi satisfy two of the conditions in formulas (2), (3), and (4), then that point is added to the center point set. The center point set under each X-axis scale is traversed and calculated, and all center point sets are merged to obtain the final segmented weld wire surface point cloud.

[0023]

[0024] α<5° (3)

[0025] θ<5° (4)

[0026] In the formula l μ l σThese are the mean and variance of the distance descriptor, respectively.

[0027] In step 6, the weld wire point cloud is downsampled using FPS. Each set of weld wire data after downsampling has M points. A KD-Tree structure is built for the downsampled point cloud. A seed point is randomly selected from the first 5 points along the X-axis, and a seed point is selected every K points, for a total of N seed points, where N is a positive integer. The KD-Tree is used to query the K nearest neighbors of the seed point, forming N local planes. The normal vector of each point in the local plane is calculated, and the mean of the plane's normal vector is fitted and denoted as the plane normal vector. The angle between the plane normal vector and the X-axis is calculated. If the angle is greater than 90 degrees, the local plane descriptor is denoted as 1; otherwise, it is denoted as -1. After completing the calculation of the N local plane descriptors, a descriptor set Q is generated.

[0028]

[0029] The descriptive subset generated in (1) is expressed as Equation (6). After the descriptive subset is generated, the defect pattern is identified by querying Table 1.

[0030]

[0031]

[0032] The present invention has the following advantages:

[0033] This method employs image-point cloud information fusion to detect weld wire structural defects, significantly reducing manual labor costs and improving weld wire inspection efficiency. This method can also perform pattern recognition on weld wire defects, facilitating subsequent process evaluation and improvement. OBB bounding boxes and the MAD algorithm are used to perform coarse localization of the weld wire, and the localization information is used for matrix operations to obtain the weld wire's point cloud information. For obtaining the point cloud information, a surface feature descriptor for the maximum spatial density points under point cloud slices is proposed. This descriptor can perform spatial plane segmentation of the point cloud, remove redundant information, and obtain an accurate weld wire plane, making it easier to detect various types of weld wire defects. Based on weld wire point cloud data segmentation, this invention proposes a normal vector direction differentiation descriptor based on multi-local plane fitting for weld wire defect detection and pattern recognition. This invention can be applied to the field of defect detection in lead wire welding manufacturing processes, replacing manual inspection and analysis of weld wire defects. Attached Figure Description

[0034] Figure 1 Flowchart for wire bonding defect detection;

[0035] Figure 2(a) is a depth image captured by the camera, (b) is a depth image of the PCB board, (c) is an image after thresholding, and (d) is an image after contour detection and drawing of OBB bounding boxes;

[0036] Figure 3 (a) is the point cloud before affine transformation, and (b) is the point cloud after affine transformation.

[0037] Figure 4 (a) is the point cloud before surface segmentation, and (b) is the point cloud after surface segmentation.

[0038] Figure 5 (a) is a normal weld line, (b) is a bending defect, (c) is a collapsed line defect, (d) is a single-end break, (e) is an internal break, and (f) is an abnormal curvature. Detailed Implementation

[0039] The following section provides a detailed explanation of this method, with reference to the accompanying drawings and examples.

[0040] Figure 1 The flowchart illustrates a method for solder wire defect detection and pattern recognition. First, a depth image of the PCB board is acquired, and the solder wire contours are located within this depth image. The solder wire depth image is then converted into point cloud data, and the point cloud is segmented to obtain a complete solder wire plane. Defects such as broken wires, collapsed wires, and bends are detected on the segmented solder wire plane. For normal solder wires, spatial curvature and the curvature of a fitted mathematical model are calculated.

[0041] The specific steps are as follows:

[0042] 1. Acquire PCB board depth map

[0043] The image acquisition device uses Hikrobot's 3D laser contour sensor, with an X-axis resolution of 30 micrometers and a far field of view of 90 millimeters. The acquired depth images can be transmitted to a computer for processing in real time.

[0044] 2. Weld wire contour positioning

[0045] The PCB board depth map obtained is as follows Figure 2 As shown in (a), the PCB board image and the image background are first segmented to obtain a complete PCB depth map, as follows. Figure 2 As shown in (b), image enhancement and thresholding are performed on the segmented PCB image, and the segmentation effect is as follows. Figure 2 As shown in (c).

[0046] right Figure 2(c) Perform contour detection, extract each contour data point, and use Principal Component Analysis (PCA) to calculate the eigenvalues ​​and eigenvectors of each contour data set. The eigenvector corresponding to the largest eigenvalue is determined as the X-axis, and the contour data is affinely transformed onto the new principal axes. The maximum and minimum values ​​of the X-axis and Y-axis in the transformed data are calculated, and the bounding box of the contour is fitted, such as... Figure 2 As shown in (d), calculate the aspect ratio of the rectangle, denoted as D. i Iterate through all contours and calculate their aspect ratios to generate an aspect ratio data set D.

[0047] Calculate the median of the dataset D, denoted as Dmedian. median Subtract the median from the data set D to obtain a new set D', and calculate the median of D', denoted as D''. median D satisfies formula (7) i This represents the aspect ratio of the weld wire outline.

[0048] D i -D median >8*D' median (7)

[0049] Extract the data within the weld line contour and convert the depth data into a point cloud using formula (8).

[0050]

[0051] In the formula, X, Y, and Z are the coordinates of the point cloud in space, x and y are the coordinates of the pixels in the depth map, and z is the height information in the depth map. scale Y scale Z scale X offset Y offset Z offset These are parameters internal to the camera.

[0052] 3. Surface Segmentation

[0053] The extracted weld wire point cloud was first subjected to statistical filtering. The number of neighborhood points used to calculate the average distance in the filtering parameters was 200, and the standard deviation of the average distance of the point cloud was 2.0. PCA principal component analysis was performed on the filtered point cloud to calculate the eigenvalues ​​and eigenvectors of each group of point clouds. The eigenvector corresponding to the largest eigenvalue was determined as the X-axis. The spatial coordinates (X,Y,Z) of the point cloud were affinely transformed using formula (9), where θ is the angle between the new X-axis and the original X-axis, resulting in the affinely transformed point cloud (X',Y',Z'). The spatial pose of the point cloud before and after the affine transformation is as follows: Figure 3 As shown.

[0054]

[0055] The point cloud data after affine transformation is sliced ​​along the X-axis, with the slice thickness equal to the minimum interval between adjacent points on the X-axis. All spatial points belonging to the same X-scale are extracted; these points are distributed within the same YZ-axis plane. Histograms of the extracted points at different heights on the Z-axis are plotted, and the point with the highest number of points at the same height is designated as the center point. The distance l between the center point Pn and all other points Pi is calculated iteratively. i Y-axis angle α i The angle θ between the vector and the normal vector i Generate a descriptor subset (l i ,α i ,θ i If any three descriptors in Pi satisfy two of the conditions in formulas (10), (11), and (12), then that point is added to the set of central points.

[0056]

[0057] α i <5° (11)

[0058] θ i <5° (12)

[0059] In the formula l μ l σ These are the mean and variance of the distance descriptor, respectively.

[0060] The set of center points at each X-axis scale is traversed and calculated. All center point sets are then merged to obtain the final segmented weld wire surface point cloud. The effect before and after point cloud surface segmentation is shown below. Figure 4 As shown.

[0061] 4. Wire bonding defect detection

[0062] Project the segmented surface point cloud onto the XZ axis, and calculate the X-axis spacing between adjacent points using formula (13):

[0063] E i =x i -x i-1 (13)

[0064] Calculate the average spacing:

[0065]

[0066] Calculate the variance of the spacing:

[0067]

[0068] If the distance between adjacent points is E i If formula (14) is satisfied, it means that the weld wire is a broken wire inside the wire.

[0069] E i >u+3σ (16)

[0070] In equation (13), x i x i-1 Let E be the X-axis coordinate of the point cloud. i σ is the distance between adjacent point clouds along the X-axis; μ in equation (14) is the average distance; σ in equation (15) is the variance of the distance.

[0071] If the weld line is not an in-line break, the weld line point cloud is downsampled using FPS. Each downsampled weld line data set contains M points. A KD-Tree structure is built for the downsampled point cloud. A seed point is randomly selected from the first 5 points along the X-axis, and a seed point is selected every K points, for a total of N seed points, where N is a positive integer. The KD-Tree is used to query the K nearest neighbors of each seed point, forming N local planes. The normal vector of each point within a local plane is calculated, and the mean of the plane's normal vector is fitted and denoted as the plane normal vector. The angle between the plane normal vector and the X-axis is calculated. If the angle is greater than 90 degrees, the local plane descriptor is denoted as 1; otherwise, it is denoted as -1. After completing the calculation of the N local plane descriptors, a descriptor set Q is generated.

[0072]

[0073] The descriptive subset generated in (1) is written as Equation (18). After the descriptive subset is generated, the defect pattern is identified by querying Table 1.

[0074]

[0075] In the formula, N is the number of descriptors, i and j are descriptors 1 or -1, and n and m are the number of i and j, respectively.

[0076] Table 1. Descriptor-Defect Pattern Comparison Table

[0077]

[0078] 6. Calculation of wire curvature

[0079] After completing the defect detection of the weld wire, normal weld wires are obtained. For the point cloud data of normal weld wires, a KD-Tree structure is first established, and the n nearest points Pk of point Pi in the point cloud are queried to establish a set of nearest points X. The set X is then centered to construct a covariance matrix. The covariance matrix is ​​decomposed by SVD to obtain the set of eigenvalues ​​λ = {λ1, λ2, λ3}. The curvature is calculated using formula (19), and the curvature of all spatial points is calculated to obtain the curvature set.

[0080]

[0081] In equation (19), {λ1,λ2,λ3} are the eigenvalues ​​of the covariance matrix after decomposition, and they are arranged in ascending order. δ is the calculated spatial curvature.

[0082] For normal wire bonding point cloud data, the data is projected onto the XZ axes, and the projected wire bonding is fitted using the least squares method to obtain the corresponding mathematical model. The curvature of the mathematical model is then calculated. The fitted model is f(x) = w0 + w1x + w2x 2 +w3x 3 The fitting condition is given by formula (20), and the curvature is calculated by formula (21).

[0083]

[0084]

[0085] In equation (20), f(x) is the fitted model, and z i For the Z-axis data of the point cloud, w i , where K is the calculated curvature, and f' and f″ are the first and second derivatives of the model f(x).

Claims

1. A method for detecting and recognizing weld wire defects based on image-point cloud information fusion, characterized in that, The method includes the following steps: Step 1: Use a 3D laser contour sensor to scan the entire PCB board and obtain a depth image; Step 2: Using image enhancement, thresholding, and contour detection methods, the PCB board image and background image are segmented to obtain the depth image of the PCB board; Step 3: Perform image enhancement and threshold segmentation on the PCB board depth map, perform contour detection on the threshold segmented image, traverse and calculate the OBB bounding box of each contour and obtain the aspect ratio of the bounding box, and use the absolute median difference (MAD) algorithm to distinguish the solder wire contour from other contours. Step 4: Extract depth data within the weld wire contour, perform matrix operations to obtain point cloud data of the weld wire, and preprocess the point cloud data using statistical filtering and PCA affine transformation. Step 5: Perform surface segmentation on the preprocessed wire bonding data to separate the wire bonding surface data and redundant data, and obtain complete wire bonding point cloud data. Step 6: Project the weld wire point cloud data constructed in Step 5 onto the XZ axis, calculate the interval between adjacent points after projection, and use the 3-sigma algorithm to filter out broken wires within the line to obtain a non-broken weld wire point cloud dataset. Step 7: Perform FPS downsampling on the weld wire point cloud data constructed in Step 6, estimate the normal vector of each point in the downsampled data, construct a defect classification model, and complete the identification of specific weld wire defect patterns. Step 8: Calculate the spatial curvature of the normal bonding wire in Step 7, project the bonding wire data onto the XZ axis, fit the mathematical model of the bonding wire using the least squares method, calculate the curvature of the fitted mathematical model, and complete the evaluation of the curvature of the normal bonding wire.

2. The method for weld wire defect detection and pattern recognition based on image-point cloud information fusion as described in claim 1, characterized in that, Step 3 is as follows: (1) Calculation of the aspect ratio of the outer rectangle of the outline Image enhancement and thresholding segmentation are performed on the PCB depth image. Contour detection is then performed on the binarized image after thresholding segmentation. Principal component analysis (PCA) is used to calculate the eigenvalues ​​and eigenvectors of each contour data set. The eigenvector corresponding to the largest eigenvalue is determined as the X-axis, and the contour data is affinely transformed onto the new principal axis. The maximum and minimum values ​​of the X-axis and Y-axis in the transformed data are calculated, and the bounding rectangle of the contour is fitted. The aspect ratio of the rectangle is calculated and denoted as D. i ; Iterate through and calculate the aspect ratio of all contours to generate an aspect ratio data set D; (2) Positioning of wire bonding contour Calculate the median of the dataset D, denoted as Dmedian. median Subtract the median from the data set D to obtain a new set D', and calculate the median of D', denoted as D''. median ; D that satisfies formula (1) i The aspect ratio of the weld wire outline; D i -D median >8*D' median (1)。 3. The method for weld wire defect detection and pattern recognition based on image-point cloud information fusion as described in claim 1, characterized in that, Step 5 is as follows: (1) Calculation of center point Slice the point cloud data after preprocessing in step 4 along the X-axis. The slice thickness is the minimum interval between adjacent points on the X-axis. Extract all spatial points belonging to the same X-scale. These spatial points are distributed in the same YZ-axis plane. Draw histograms of the extracted points at different heights on the Z-axis. Record the point with the most distribution at the same height as the center point. (2) Point cloud descriptor calculation Iterate through the points and calculate the distance l between the center point Pn and all other points Pi. i Y-axis angle α i The angle θ between the vector and the normal vector i Generate a descriptor subset (l i ,α i ,θ i If any three descriptors in Pi satisfy two of the conditions in formulas (2), (3), and (4), then add that point to the set of central points. α i <5° (3) i i <5° (4) In the formula l μ l σ These are the mean and variance of the distance descriptors, respectively; (3) Extraction of wire bonding surface Using the methods in (1) and (2), the set of center points under each X-axis scale is traversed and calculated. All sets of center points are merged to obtain the final segmented weld wire surface point cloud.

4. The method for weld wire defect detection and pattern recognition based on image-point cloud information fusion as described in claim 1, characterized in that, Step 7 is as follows: (1) Calculation of wire bonding defect descriptor The weld wire point cloud is downsampled using FPS, resulting in M ​​points for each downsampled weld wire data set. A KD-Tree structure is built for the downsampled point cloud. A seed point is randomly selected from the first 5 points along the X-axis, and another seed point is selected every K points, for a total of N seed points, where N is a positive integer. The KD-Tree is used to query the K nearest neighbor points of each seed point, forming N local planes. The normal vector of each point within a local plane is calculated, and the mean of the plane's normal vector is fitted and denoted as the plane normal vector. The angle between the plane normal vector and the X-axis is calculated. If the angle is greater than 90 degrees, the local plane descriptor is denoted as 1; otherwise, it is denoted as -1. The calculation of the N local plane descriptors is completed, generating a descriptor set Q. (2) Defect pattern recognition The partial descriptive subset generated in (1) is written as Equation (6); after the descriptive subset is generated, the defect pattern is identified by querying Table 1; In the formula, N is the number of descriptors, i and j are descriptors 1 or -1, and n and m are the number of i and j, respectively; Table 1. Descriptor-Defect Pattern Comparison Table

Images