A fast detection method for three-dimensional defects of bonding wires based on images and point clouds

Abstract

The present invention discloses a bonding wire defect detection method based on point cloud, including: detecting and positioning integrated circuit image information through YOLOV8 network; corresponding the position detected by YOLOV8 network to the point cloud range of three-dimensional point cloud marking network, and marking the point cloud within the position range as a point cloud of interest; importing the point cloud of interest into the bonding wire solder joint extraction network to detect and classify the bonding wire, solder joint and substrate; performing plane fitting on the substrate point cloud, and detecting the arch height and span of the bonding wire solder joint. The present invention directly locates the solder joint bonding wire to the point cloud within a certain range through two-dimensional positioning and three-dimensional detection technology, thereby improving the detection speed of the solder joint bonding wire; the proposed abnormal point elimination network improves the detection accuracy of the arch height and span of the bonding wire solder joint; the proposed Euclidean-based calculation model establishes a spatial rectangular coordinate system; the bonding wire arch height information is calculated according to the difference between the bonding wire vertex center and the solder joint mass center in the Z-axis direction.

Description

Technical Field

[0001] The invention relates to the field of bonding wire feature detection, and in particular to a bonding wire three-dimensional defect rapid detection method based on images and point clouds. Background Art

[0002] The main technology for detecting the height of hybrid integrated circuit bonding wires and the span of solder joints is based on point cloud detection technology. Usually, point cloud data containing depth information is collected, and point clusters of bonding wires, solder joints and substrates are detected through PointNet++ algorithm processing, and finally the point clusters are calculated to obtain the height of bonding wires and the span of solder joints.

[0003] In order to detect the height of the bonding wire and the span of the solder joint, the solder joint is usually matched with the bonding wire after detecting the point clusters of the bonding wire and the solder joint. Since the bonding wire and the solder joint present a one-to-one correspondence in the physical space, they can be matched according to the positional relationship between the two. In this way, the correspondence between the solder joint point cluster and the bonding wire point cluster is established. The matched solder joint cluster and bonding wire cluster are fitted to obtain the centroid of the solder joint and the bonding wire. The centroid is calculated to obtain the bonding wire arch height and the solder joint span.

[0004] The prior art "CN113570550 A method for detecting the size of gold wire bonds based on three-dimensional point clouds" provides a method for detecting the height and span of bonding wire solder joints based on point clouds. This patent uses the PointNet++ network to classify bonding wire point clusters, solder joint point clusters, chip point clouds and baseboard point clouds, and matches the bonding wire solder joint clusters according to the angular relationship between the bonding wire solder joints. The matched bonding wire solder joint clusters are imported into the calculation model to calculate the solder joint span and bonding wire height information; when implementing the bonding wire solder joint detection, the detected bonding wires and solder joints are matched according to the angular relationship. In actual hybrid integrated circuits, the following problems exist: 1. There are many solder joint features in hybrid integrated circuits, such as device solder joints, solder joint protrusions, magazine attachment, etc., and similar features are very likely to cause false detections; 2. The bonding wire solder joints are highly dense, and the method of matching the bonding wire solder joints based on angles is very likely to cause false matching. In the field of point cloud detection, although the PointNet++ network is relatively mature, it is not easy to detect solder joints and bonding wire clusters with high accuracy in hybrid integrated circuit scenarios with small targets, high density, high feature, high complexity and strict local information requirements. If only the method based on angle matching of bonding wire solder joints combined with the PointNet++ network is used, it is difficult to detect the height and span of solder joints and bonding wires. Summary of the invention

[0005] The technical solution of the present invention to solve the above technical problem is to provide a method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds, comprising the following steps:

[0006] Collect integrated circuit image information, and detect the location and range information of integrated circuit boards and bonding wire solder joints through the YOLOV8 network model;

[0007] Match the position detected by the YOLOV8 network model with the point cloud range of the 3D point cloud labeling network model;

[0008] The 3D point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish the conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest;

[0009] The point cloud information within the region of interest is imported into the bonding wire solder point extraction network model for point cloud classification; the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster;

[0010] The classified solder point clusters and bonding wire point clusters are processed by the Euclidean distance clustering algorithm. The solder point clusters are clustered by centers, and the bonding wire point clusters are processed by outliers removal, and valid values ​​are retained.

[0011] The plane fitting method of random sampling consensus algorithm is selected to iterate the substrate point cloud of hybrid integrated circuit for multiple times to obtain the best fitting effect.

[0012] Furthermore, the three-dimensional point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish a conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest in the step:

[0013] The conversion relationship establishment part includes:

[0014] Establish the conversion relationship between point cloud and image;

[0015] The position information of the integrated circuit board is vertex 1 (x b1 ,y b1 ), Vertex 2(x b2 ,y b2 ), Vertex 3(x b3 ,y b3 )、Vertex 4(x b4 ,y b4 ); the bonding wire solder point position information is vertex 1 (x w1 ,y w1 ), Vertex 2(x w2 ,y w2 ), Vertex 3(x w3 ,y w3 )、Vertex 4(x w4 ,y w4 ); the 3D point cloud position information is vertex 1 (xt1 ,y t1 , z t1 ), vertex 2(x t2 ,y t2 , z t2 ), vertex 3(x t3 ,y t3 , z t3 ), vertex 4(x t4 ,y t4 , z t4 );

[0016] By converting the ratio between the two diagonal points of the substrate, the conversion relationship between the two-dimensional image and the three-dimensional point cloud is obtained:

[0017]

[0018]

[0019] Among them, D b : Diagonal length of the two-dimensional image substrate; D t : The diagonal length of the 3D image substrate; K: The horizontal range conversion relationship between the 2D image and the 3D point cloud;

[0020] The position of the bonding wire solder point relative to the substrate is obtained through the range vertex coordinates;

[0021] ΔL1=x b1 -x w1 ;

[0022] ΔH1=y b1 -y w1 ;

[0023] ΔL2=x w3 --x b3

[0024] ΔH2=y w3 --y b3

[0025] L b =x b3 -x b1

[0026] H b =y b2 --y b1

[0027] Wherein, ΔL1: x-axis distance from range vertex 1 to substrate vertex 1; ΔH1: y-axis distance from range vertex 1 to substrate vertex 1; ΔL2: x-axis distance from range vertex 3 to substrate vertex 3; ΔH2: y-axis distance from range vertex 3 to substrate vertex 3; L b : Range x-axis length; Hb : Range y-axis length;

[0028] Determine the position range information of solder joints and bonding wires in the three-dimensional point cloud;

[0029] ΔL t1 =ΔL1×K+t

[0030] ΔH t1 =ΔH1×K+t

[0031] ΔL t2 =ΔL2×K+t

[0032] ΔH t2 =ΔH2×K+t

[0033] Where, ΔLt1: the relative position of diagonal point 1 in the x-axis direction on the 3D point cloud; ΔH t1 : The relative position of diagonal point 1 in the y-axis direction on the 3D point cloud; ΔL t2 : The relative position of diagonal point 3 in the x-axis direction on the 3D point cloud; ΔH t2 : The relative position of diagonal point 3 in the y-axis direction on the three-dimensional point cloud; t: dynamic parameter;

[0034] The position and range information of the bonding wire solder joint detected by YOLOV8 corresponds to the corresponding position and range of the three-dimensional point cloud;

[0035] The point cloud marking part includes:

[0036] Assume that the point set of the point cloud is P = (P1, P2, P3, ... P i ...P n ), i = 1, 2, ... n, where point P i =(x i ,y i , z i , c i ), when the point cloud is in the region of interest c i =1, otherwise c i =0.

[0037] Further, the point cloud information within the region of interest is imported into the bonding wire solder point extraction network model for point cloud classification; the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster:

[0038] The bonding wire solder joint extraction network model includes an improved Pointnet++ network and an outlier removal network;

[0039] The improved PointNet++ network is used to classify the bonding wires, solder joints and substrate point clouds within the region of interest;

[0040] The improved PointNet layer single point information encoding is:

[0041]

[0042] in, is the single point feature obtained after MLP processing, C is the feature dimension, pi is the coordinate of the sampling point at the center of the local space, is the coordinate of a neighborhood point of the sampling point in the local space, is the coordinate difference between the sampling point and the neighborhood point, is the Euclidean distance between the sampling point and the neighborhood point;

[0043] Improved single local spatial feature aggregation process, introducing spatial and channel attention mechanisms, which are activated after multi-layer perceptron processing;

[0044] The abnormal point removal network is used to remove outliers and abnormal points of bonding wire point clusters and solder point clusters.

[0045] Furthermore, in the step of selecting the plane fitting method of the random sampling consensus algorithm to perform multiple iterations on the substrate point cloud of the hybrid integrated circuit to seek the best fitting effect:

[0046] The plane fitting equation is: 0 = ax + by + cz + d;

[0047] a, b, c, d: plane equation parameters, obtained by fitting algorithm; (x, y, z): x, y, z direction coordinate values;

[0048] Construct a calculation model for bonding wire solder joints, with a cluster of 1 solder joints as W p ={w1, w2, ... w i ..., w n}, where w i is the solder point cloud; the bonding wire point cluster is B w ={b1, b2, ... b i ..., b n}, where b i It is the bonding wire point cloud;

[0049] The span of the bonding wire is expressed as the horizontal distance between solder joint 1 and solder joint 2; it is obtained by the formula:

[0050]

[0051] Among them, d b : Horizontal distance between solder joint 1 and solder joint 2; x p1: The horizontal coordinate of the projection point cloud of welding point 1 on the fitting plane; x p2 : The horizontal coordinate of the projection point cloud of welding point 2 on the fitting plane; y p1 : The ordinate of the projection point cloud of welding point 1 on the fitting plane; y p2 : The vertical coordinate of the projection point cloud of welding point 2 on the fitting plane;

[0052] By obtaining the centroid spatial position of the solder point 1 cluster and the solder point 2 cluster, and projecting the two centroids onto the fitting plane, the coordinates of the projected point cloud of solder point 1 and solder point 2 on the fitting plane can be obtained. The centroid coordinates can be obtained by traversing the horizontal and vertical coordinates of the solder point 1 cluster and the solder point 2 cluster, and performing summation and mean operations. The mathematical expression of the centroid coordinates is:

[0053]

[0054] Among them, C x : abscissa of the centroid of the cluster; C y : The vertical coordinate of the centroid of the cluster; C z : The vertical coordinate of the centroid of the point cluster; m: The number of point clouds in the point cluster; Calculate the projection coordinates projected to the fitting plane z, which can be calculated by the following formula:

[0055]

[0056] Among them, P iproj : Center of mass P i Projection on a plane.

[0057] The height of the bonding wire solder joint can be expressed as the difference between the distance between the bonding wire point cluster and the distance between the solder joint point cluster and the fitting plane; the height of the point cloud from the fitting plane can be expressed as:

[0058]

[0059] h: the height of the point cloud from the fitting plane;

[0060] To ensure the accuracy of the calculation results, the 10 points h where the bonding wire cluster is the highest from the fitting plane are obtained by traversing, and the true value of the bonding wire cluster height h is calculated according to the average formula;

[0061]

[0062] Among them, Z bmax : True value of bonding wire cluster height; h bimax : The 10 maximum heights h of the point cloud in the bonding wire cluster from the fitting plane;

[0063] The Z-axis maximum value coordinates of solder joints 1 and 2 are obtained by the formula:

[0064] h wmax =max(hp )

[0065] Among them, h wmax : The maximum value of the average height of the solder joint cluster from the fitting plane; max(): Find the maximum value function; h p : The height of the welding point cloud from the fitting plane;

[0066] Compare the Z-axis maximum values ​​of solder joint 1 and solder joint 2 to determine the positional relationship between the two solder joints;

[0067]

[0068] Among them, H wmin : The solder point has a lower average height from the simulated plane; h w1 : average height of one cluster of solder joints; h w2 : The average height of the 2-point cluster of solder joints;

[0069] When the height of the solder point cluster is lower than the fitting plane, the real value of the arch height is obtained by traversing the average height of the solder point cluster and the average height of the bonding wire cluster.

[0070]

[0071] Among them, H g is the true value of the height between the highest point g in the bonding wire and the solder joint 1;

[0072] The true value of the bonding wire arch height can be obtained according to the average formula;

[0073]

[0074] Among them, H is the bonding wire arch height. Finally, according to the size of the arch height and span and the defect standard, it is determined whether the bonding wire of the solder joint has the defects of arch height and span.

[0075] The present invention detects bonding wire solder joint pairs through mature two-dimensional target detection technology. The positioning and matching of all solder joint bonding wire pairs are completed through YOLOv8 network detection. In all three-dimensional point cloud detections, the detection of bonding wire solder joints is directly positioned, which improves the detection accuracy of bonding wire solder joints.

[0076] Since the traditional technology directly transmits the point cloud data of the hybrid integrated circuit to the PointNet++ network, it should be noted that the diameter of the bonding wire is generally between tens of microns and hundreds of microns, and the detection accuracy of the three-dimensional camera used is at the micron level. The point cloud data generated is very large, and the time used to process the point cloud of the hybrid integrated circuit will be very long. The two-dimensional positioning and three-dimensional detection technology proposed in the present invention directly locates the solder joint bonding wire to the point cloud within a certain range. Only a few points need to be detected to realize the extraction and matching of the bonding wire solder joints, which greatly improves the detection speed of the solder joint bonding wire.

[0077] Since there are outliers and abnormal points in the detected bonding wire solder point clusters, these points have a great influence on the detection results of arch height and span. The abnormal point elimination network proposed in the present invention performs clustering according to the distribution and density of the point clusters, eliminates abnormal points that have a great influence on the results, and gathers the point clusters within the centroid of the solder point and the center of the bonding wire, which greatly improves the detection accuracy of the arch height and span of the bonding wire solder point.

[0078] Since there is no standard reference coordinate system in the space for the processed bonding wire solder point cluster, it is difficult to measure the spatial information of the solder point bonding wire. The Euclidean-based calculation model proposed in the present invention fits the substrate plane according to the plane fitting algorithm, takes the substrate plane as the XY axis, and takes the normal vector perpendicular to the substrate plane as the Y axis to establish a spatial rectangular coordinate system. The solder point span information is calculated based on the Euclidean distance between the centroids of the two solder points in the XY plane, and the bonding wire arch height information is calculated based on the difference between the center of the bonding wire vertex and the centroid of the solder point in the Z-axis direction.

[0079] Compared with the prior art, the present invention has a faster detection speed. The existing detection method transmits the entire hybrid integrated circuit point cloud (millions of points) to the network for detection, which has a large amount of network calculation and slow model reasoning speed. However, the present invention only imports the point cloud of the solder joint bonding wire area into the network (tens of thousands of points), which greatly reduces the amount of network calculation and speeds up the model reasoning speed.

[0080] Compared with the prior art, the present invention has higher detection accuracy. The existing detection method matches and associates the angle relationship between the solder joint bonding wires. When faced with a large number of hybrid integrated circuits with high density, the solder joint bonding wire matching accuracy is low. However, the present invention uses image processing technology to lock only one set of solder joint bonding wires in the area, so there is no matching problem, which greatly improves the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0082] Figure 1 It is a structural schematic diagram of the method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds according to the present invention;

[0083] Figure 2 This is a schematic diagram of the point cloud marking network of the present invention;

[0084] Figure 3 A network structure diagram of a solder joint extraction network of the present invention;

[0085] Figure 4 This is a schematic diagram of an outlier removal network of the present invention;

[0086] Figure 5 This is a flow chart of the random sampling consensus algorithm of the present invention. DETAILED DESCRIPTION

[0087] The present invention proposes a method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds, aiming to.

[0088] The method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds proposed by the present invention will be described below in a specific embodiment:

[0089] Embodiment 1:

[0090] A fast detection method for three-dimensional defects of bonding wires based on images and point clouds, such as Figure 1 As shown, the following steps are included:

[0091] S10: Collect integrated circuit image information, and detect the position and range information of the integrated circuit board and bonding wire solder joints through the YOLOV8 network model;

[0092] S21: Match the position detected by the YOLOV8 network model with the point cloud range of the 3D point cloud labeling network model;

[0093] S22: The three-dimensional point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which is used to establish a conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as a region of interest;

[0094] S23: importing the point cloud information within the region of interest into the bonding wire solder point extraction network model for point cloud classification; dividing the point cloud within the range into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster;

[0095] S24: The classified solder point clusters and bonding wire clusters are processed by the Euclidean distance clustering algorithm, the solder point clusters are subjected to center clustering processing, the bonding wire clusters are subjected to outlier removal processing, and valid values ​​are retained;

[0096] S30: Selecting the plane fitting method of the random sampling consensus algorithm, performing multiple iterations on the substrate point cloud of the hybrid integrated circuit to obtain the best fitting effect;

[0097] S40: Calculate the arch height and span of the bonding wire of the solder joint and determine the defect type;

[0098] Specifically, the solder point cluster, bonding wire point cluster, and plane fitting equation are transmitted to the Euclidean calculation model. By solving operations such as centroid, projection, and spatial distance, the span and arch height information of the two solder points are calculated, and the defect type is determined based on the results.

[0099] It can be understood that the Euclidean calculation model includes the steps of the Euclidean distance calculation method, and also extends the centroid solution and projection operations. The core of the bond wire solder joint calculation model is this Euclidean calculation model.

[0100] Furthermore, the three-dimensional point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish a conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest in the step:

[0101] The conversion relationship establishment part includes:

[0102] Establish the conversion relationship between point cloud and image;

[0103] The position information of the integrated circuit board is vertex 1 (x b1 ,y b1 ), Vertex 2(x b2 ,y b2 ), Vertex 3(x b3 ,y b3 )、Vertex 4(x b4 ,y b4 ); the bonding wire solder point position information is vertex 1 (x w1 ,y w1 ), Vertex 2(x w2 ,y w2 ), Vertex 3(x w3 ,y w3 )、Vertex 4(x w4 ,y w4 ); the 3D point cloud position information is vertex 1 (x t1 ,y t1 , z t1 ), vertex 2(x t2 ,y t2 , z t2 ), vertex 3(x t3 ,y t3 , z t3 ), vertex 4(x t4 ,y t4 , z t4 );

[0104] By converting the ratio between the two diagonal points of the substrate, the conversion relationship between the 3D image and the 3D point cloud is obtained:

[0105]

[0106]

[0107] Among them, D b : Diagonal length of the two-dimensional image substrate; D t : The diagonal length of the 3D image substrate; K: The horizontal range conversion relationship between the 2D image and the 3D point cloud;

[0108] The position of the bonding wire solder point relative to the substrate is obtained through the range vertex coordinates;

[0109] ΔL1=x b1 -x w1 ;

[0110] ΔH1=y b1 -y w1 ;

[0111] ΔL2=x w3 --x b3

[0112] ΔH2=y w3 --y b3

[0113] L b =x b3 -x b1

[0114] H b =y b2 --y b1

[0115] Wherein, ΔL1: x-axis distance from range vertex 1 to substrate vertex 1; ΔH1: y-axis distance from range vertex 1 to substrate vertex 1; ΔL2: x-axis distance from range vertex 3 to substrate vertex 3; ΔH2: y-axis distance from range vertex 3 to substrate vertex 3; L b : Range x-axis length; H b : Range y-axis length;

[0116] Determine the position range information of solder joints and bonding wires in the three-dimensional point cloud;

[0117] ΔL t1 =ΔL1×K+t

[0118] ΔH t1 =ΔH1×K+t

[0119] ΔL t2 =ΔL2×K+t

[0120] ΔH t2 =ΔH2×K+t

[0121] Among them, ΔL t1 : The relative position of diagonal point 1 in the x-axis direction on the 3D point cloud; ΔH t1 : The relative position of diagonal point 1 in the y-axis direction on the 3D point cloud; ΔL t2 : The relative position of diagonal point 3 in the x-axis direction on the 3D point cloud; ΔH t2 : The relative position of diagonal point 3 in the y-axis direction on the three-dimensional point cloud; t: dynamic parameter;

[0122] The position and range information of the bonding wire solder joint detected by YOLOV8 corresponds to the corresponding position and range of the three-dimensional point cloud;

[0123] The point cloud marking part includes:

[0124] Assume that the point set of the point cloud is P = (P1, P2, P3, ... P i ...P n ), i = 1, 2, ... n, where point P i =(x i ,y i , z i , c i ), when the point cloud is in the region of interest c i =1, otherwise c i =0.

[0125] Further, the point cloud information within the region of interest is imported into the bonding wire solder point extraction network model for point cloud classification; the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster:

[0126] The bonding wire solder joint extraction network model includes an improved Pointnet++ network and an outlier removal network;

[0127] The improved PointNet++ network is used to classify the bonding wires, solder joints and substrate point clouds within the region of interest;

[0128] The improved PointNet layer single point information encoding is:

[0129]

[0130] in, is the single point feature obtained after MLP processing, pi is the coordinate of the sampling point at the center of the local space, is the coordinate of a neighborhood point of the sampling point in the local space, is the coordinate difference between the sampling point and the neighborhood point, is the Euclidean distance between the sampling point and the neighborhood point;

[0131] Improved single local spatial feature aggregation process, introducing spatial and channel attention mechanisms, which are activated after multi-layer perceptron processing;

[0132] The abnormal point removal network is used to remove outliers and abnormal points of bonding wire point clusters and solder point clusters.

[0133] Furthermore, the plane fitting method using the random sampling consensus algorithm is as follows: Figure 5 As shown, the steps of performing multiple iterations on the substrate point cloud of the hybrid integrated circuit to seek the best fitting effect are:

[0134] The plane fitting equation is: 0 = ax + by + cz + d;

[0135] a, b, c, d: plane equation parameters, obtained by fitting algorithm; (x, y, z): x, y, z direction coordinate values;

[0136] Construct a calculation model for bonding wire solder joints, with a cluster of 1 solder joints as W p ={w1, w2, ... w i ..., w m}, where w i is the solder point cloud; the bonding wire point cluster is B w ={b1, b2, ... b i ..., b n}, where b i It is the bonding wire point cloud;

[0137] The span of the bonding wire is expressed as the horizontal distance between solder joint 1 and solder joint 2; it is obtained by the formula:

[0138]

[0139] Among them, d b : Horizontal distance between solder joint 1 and solder joint 2; x p1 : The horizontal coordinate of the projection point cloud of welding point 1 on the fitting plane; x p2 : The horizontal coordinate of the projection point cloud of welding point 2 on the fitting plane; y p1 : The ordinate of the projection point cloud of welding point 1 on the fitting plane; y p2 : The vertical coordinate of the projection point cloud of welding point 2 on the fitting plane;

[0140] By obtaining the spatial position of the centroid of the solder point 1 cluster and the solder point 2 cluster, and projecting the two centroids onto the fitting plane, the coordinates of the projection point cloud of solder point 1 and solder point 2 on the fitting plane can be obtained. The centroid coordinates can be obtained by traversing the horizontal and vertical coordinates of the solder point 1 cluster and the solder point 2 cluster, and performing summation and mean operations. The mathematical expression of the centroid coordinates is:

[0141]

[0142] Among them, C x : abscissa of the centroid of the cluster; C y : The vertical coordinate of the centroid of the cluster; C z : The vertical coordinate of the centroid of the point cluster; m: The number of point clouds in the point cluster; Calculate the projection coordinates projected to the fitting plane z, which can be calculated by the following formula:

[0143]

[0144] Among them, P iproj : Center of mass P i Projection on a plane.

[0145] The height of the bonding wire solder joint can be expressed as the difference between the distance between the bonding wire point cluster and the distance between the solder joint point cluster and the fitting plane; the height of the point cloud from the fitting plane can be expressed as:

[0146]

[0147] h: the height of the point cloud from the fitting plane;

[0148] To ensure the accuracy of the calculation results, the 10 points h where the bonding wire cluster is the highest from the fitting plane are obtained by traversing, and the true value of the bonding wire cluster height h is calculated according to the average formula;

[0149]

[0150] Among them, Z bmax : True value of bonding wire cluster height; h bimax : The 10 maximum heights h of the point cloud in the bonding wire cluster from the fitting plane;

[0151] The Z-axis maximum value coordinates of solder joints 1 and 2 are obtained by the formula:

[0152] h wmax =max(h p )

[0153] Among them, h wmax : Maximum height of solder joint cluster from fitting plane; max(): maximum value function; h p : The height of the welding point cloud from the fitting plane;

[0154] Compare the Z-axis maximum values ​​of solder joint 1 and solder joint 2 to determine the positional relationship between the two solder joints;

[0155]

[0156] Among them, H wmin : The solder point has a lower average height from the simulated plane; h w1 : average height of one cluster of solder joints; hw2 : The average height of the 2-point cluster of solder joints;

[0157] When the height of the solder point cluster is lower than the fitting plane, the real value of the arch height is obtained by traversing the average height of the solder point cluster and the average height of the bonding wire cluster.

[0158]

[0159] Among them, H g : The true value of the height between the highest point g in the bonding wire and the solder joint 1;

[0160] The true value of the bonding wire arch height can be obtained according to the average formula;

[0161]

[0162] Among them, H is the bonding wire arch height. Finally, according to the size of the arch height and span and the defect standard, it is determined whether the bonding wire of the solder joint has the defects of arch height and span.

[0163] Embodiment 2:

[0164] A method for efficiently detecting bonding wires based on point cloud, characterized in that it comprises the following steps:

[0165] Step 1:

[0166] Step 1.1: Use an industrial camera to collect integrated circuit image information and transmit the image information to the improved YOLOV8 network. The YOLOV8 network detects the position and range information of the hybrid integrated circuit board and the bonding wire solder joints.

[0167] Step 1.2: The YOLOV8 network transmits the detected position and range information to the 3D point cloud labeling network in the form of coordinate information.

[0168] Step 2:

[0169] Step 2.1: The 3D point cloud labeling network is responsible for matching the position detected by the YOLOV8 network with the point cloud range. The 3D point cloud labeling network consists of two parts: the conversion relationship establishment part and the point cloud labeling part.

[0170] Conversion relationship establishment part:

[0171] The conversion relationship establishment part is responsible for establishing the conversion relationship between the point cloud and the image. The position information of the integrated circuit board is vertex 1 (x b1 ,y b1 ), Vertex 2(x b2 ,y b2 ), Vertex 3(x b3 ,y b3 )、Vertex 4(xb4 ,y b4 ); the bonding wire solder point position information is vertex 1 (x w1 ,y w1 ), Vertex 2(x w2 ,y w2 ), Vertex 3(x w3 ,y w3 )、Vertex 4(x w4 ,y w4 ); the 3D point cloud position information is vertex 1 (x t1 ,y t1 , z t1 ), vertex 2(x t2 ,y t2 , z t2 ), vertex 3(x t3 ,y t3 , z t3 ), vertex 4(x t4 ,y t4 , z t4 );

[0172] By converting the ratio between the two diagonal points of the substrate, the conversion relationship between the two-dimensional image and the three-dimensional point cloud is obtained:

[0173]

[0174] Among them, D b : Diagonal length of the two-dimensional image substrate; D t : The diagonal length of the 3D image substrate; K: The horizontal range conversion relationship between the 2D image and the 3D point cloud;

[0175] The position of the bonding wire solder point relative to the substrate is obtained through the range vertex coordinates;

[0176] ΔL1=x b1 -x w1 ;

[0177] ΔH1=y b1 -y w1 ;

[0178] ΔL2=x w3 --x b3

[0179] ΔH2=y w3 --y b3

[0180] L b =x b3 -x b1

[0181] H b =yb2 --y b1

[0182] Wherein, ΔL1: x-axis distance from range vertex 1 to substrate vertex 1; ΔH1: y-axis distance from range vertex 1 to substrate vertex 1; ΔL2: x-axis distance from range vertex 3 to substrate vertex 3; ΔH2: y-axis distance from range vertex 3 to substrate vertex 3; L b : Range x-axis length; H b : Range y-axis length;

[0183] Determine the position range information of solder joints and bonding wires in the three-dimensional point cloud;

[0184] ΔL t1 =ΔL1×K+t

[0185] ΔH t1 =ΔH1×K+t

[0186] ΔL t2 =ΔL2×K+t

[0187] ΔH t2 =ΔH2×K+t

[0188] ΔL t1 : The relative position of diagonal point 1 in the x-axis direction on the 3D point cloud;

[0189] ΔH t1 : The relative position of diagonal point 1 in the y-axis direction on the 3D point cloud;

[0190] ΔL t2 : The relative position of diagonal point 3 in the x-axis direction on the 3D point cloud;

[0191] ΔH t2 : The relative position of diagonal point 3 in the y-axis direction on the 3D point cloud;

[0192] b: dynamic parameters;

[0193] Through the above formula, we can get the position and range information of the bonding wire solder joint detected by YOLOV8 corresponding to the corresponding position and range of the three-dimensional point cloud.

[0194] Point cloud marking part:

[0195] The point cloud within the range obtained by the transformation relationship is set as the region of interest.

[0196] Assume that the point set of the point cloud is P = (P1, P2, P3, ... P i …P n ), where point P i =(x i ,y i , zi , c i ), when the point cloud is in the region of interest c i =1, otherwise c i = 0. Point cloud labeling network Figure 2 As shown, the black dots represent the original point cloud and the blue dots represent the point cloud of interest.

[0197] Step 2.2: Import the point cloud information within the region of interest into the bonding wire solder joint extraction network for point cloud classification.

[0198] The network structure of the bonding wire solder joint extraction network is as follows: Figure 3 As shown in Figure 2, the network mainly consists of two parts, namely the improved Pointnet++ network and the outlier removal network.

[0199] Improved PointNet++ network: used to classify bonding wires, solder joints and substrate point clouds within the region of interest;

[0200] Outlier point removal network: used to remove outliers and abnormal points from bonding wire point clusters and solder point clusters.

[0201] Among them, in order to solve the problem of difficulty in extracting local information of point clouds, the following improvements are made to the PointNet++ network:

[0202] A PointNet layer structure is designed to enhance local information encoding. The new local space encoding method not only retains the original coordinates of the point, but also adds the relative coordinates and Euclidean distance information of the point relative to the central sampling point. By adding local information, the network can encode local geometric and structural information in detail and accurately.

[0203] The improved single-point information encoding of the PointNet layer is as follows:

[0204]

[0205] in, is the single point feature obtained after MLP processing, p i is the coordinate of the sampling point at the center of the local space, is the coordinate of a neighborhood point of the sampling point in the local space, is the coordinate difference between the sampling point and the neighborhood point, is the Euclidean distance between the sampling point and the neighborhood point;

[0206] This design introduces a spatial and channel attention mechanism, which is activated after the multi-layer perceptron (MLP) processing, aiming to accurately select the spatial regions and channels that are most critical to improving network performance. Through this strategy, the network can focus on those features that have a significant impact on the final judgment result, while ignoring those information that is less error-inducing or does not contribute much to the result. This not only improves the detection accuracy of the network, but also optimizes the processing speed, thereby achieving more efficient learning and recognition performance.

[0207] Step 2.3:

[0208] The point cloud in the area of ​​interest is processed by the improved Pointnet++ network, and the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster.

[0209] Step 2.4:

[0210] The classified solder point clusters and bonding wire clusters are processed by the Euclidean distance clustering algorithm. The solder point clusters are clustered by centers, and the bonding wire clusters are processed by outlier removal. The outliers and outliers that have a greater impact on the detection accuracy are eliminated, and the valid values ​​are retained. Figure 4 As shown, the black points represent the original point cloud, the blue points represent the point cloud of interest, the brown points represent the point cloud of the solder joints, and the light blue points represent the point cloud of the bonding wires.

[0211] Step 3:

[0212] Step 3.1: Select the plane fitting method of random sampling consensus algorithm, perform multiple iterations on the substrate point cloud of the hybrid integrated circuit to seek the best fitting effect, and obtain the fitting plane equation after fitting: 0 = ax + by + cz + d;

[0213] a, b, c, d: plane equation parameters, obtained by fitting algorithm; (x, y, z): x, y, z direction coordinate values;

[0214] Construct a calculation model for bonding wire solder joints, with a cluster of 1 solder joints as W p ={w1, w2, ... w i ..., w m}, where w i is the solder point cloud; the bonding wire point cluster is B w ={b1, b2, ... b i ..., b p}, where b i It is the bonding wire point cloud;

[0215] The span of the bonding wire is expressed as the horizontal distance between solder joint 1 and solder joint 2; it is obtained by the formula:

[0216]

[0217] Among them, d b : Horizontal distance between solder joint 1 and solder joint 2; x p1 : The horizontal coordinate of the projection point cloud of welding point 1 on the fitting plane; x p2 : The horizontal coordinate of the projection point cloud of welding point 2 on the fitting plane; y p1 : The ordinate of the projection point cloud of welding point 1 on the fitting plane; y p2 : The vertical coordinate of the projection point cloud of welding point 2 on the fitting plane;

[0218] By obtaining the centroid spatial position of the solder point 1 cluster and the solder point 2 cluster, and projecting the two centroids onto the fitting plane, the coordinates of the projected point cloud of solder point 1 and solder point 2 on the fitting plane can be obtained. The centroid coordinates can be obtained by traversing the horizontal and vertical coordinates of the solder point 1 cluster and the solder point 2 cluster, and performing summation and mean operations. The mathematical expression of the centroid coordinates is:

[0219]

[0220] Among them, C x : abscissa of the centroid of the cluster; C y : The vertical coordinate of the centroid of the cluster; C z : The vertical coordinate of the centroid of the point cluster; m: The number of point clouds in the point cluster; Calculate the projection coordinates projected to the fitting plane z, which can be calculated by the following formula:

[0221]

[0222] Among them, P iproj : Center of mass P i Projection on a plane.

[0223] The height of the bonding wire solder joint can be expressed as the difference between the value of the bonding wire point cluster from the fitting plane and the value of the solder joint point cluster from the fitting plane; the height of the point cloud from the fitting plane can be expressed as:

[0224]

[0225] h: the height of the point cloud from the fitting plane;

[0226] To ensure the accuracy of the calculation results, the 10 points h where the bonding wire cluster is the highest from the fitting plane are obtained by traversing, and the true value of the bonding wire cluster height h is calculated according to the average formula;

[0227]

[0228] The Z-axis maximum value coordinates of solder joints 1 and 2 are obtained by the formula:

[0229] h wmax =max(h p )

[0230] Among them, h wmax : The maximum value of the average height of the solder joint cluster from the fitting plane; max(): Find the maximum value function; h p : height of bonding wire cluster from fitting plane;

[0231] Compare the Z-axis maximum values ​​of solder joint 1 and solder joint 2 to determine the positional relationship between the two solder joints;

[0232]

[0233] Among them, H wmin : The solder point has a lower average height from the simulated plane; h w1 : average height of one cluster of solder joints; h w2 : The average height of the 2-point cluster of solder joints;

[0234] When the height of the solder point cluster is lower than the fitting plane, the real value of the arch height is obtained by traversing the average height of the solder point cluster and the average height of the bonding wire cluster.

[0235]

[0236] Among them, H g : The true value of the height between the highest point g in the bonding wire and the solder joint 1;

[0237] The true value of the bonding wire arch height can be obtained according to the average formula;

[0238]

[0239] Among them, H is the bonding wire arch height. Finally, according to the size of the arch height and span and the defect standard, it is determined whether the bonding wire of the solder joint has the defects of arch height and span.

[0240] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds, characterized in that: The following steps are involved: Collect integrated circuit image information, and detect the location and range information of integrated circuit boards and bonding wire solder joints through the YOLOV8 network model; Match the position detected by the YOLOV8 network model with the point cloud range of the 3D point cloud labeling network model; The 3D point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish the conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest; The point cloud information within the region of interest is imported into the bonding wire solder point extraction network model for point cloud classification; the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster; The classified solder point clusters and bonding wire point clusters are processed by the Euclidean distance clustering algorithm. The solder point clusters are clustered by centers, and the bonding wire point clusters are processed by outliers removal, and valid values ​​are retained. The plane fitting method of random sampling consensus algorithm is used to iterate the substrate point cloud of hybrid integrated circuit for multiple times to obtain the best fitting effect; Calculate the arch height and span of the bonding wire at the solder joint and determine the defect type; The three-dimensional point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish a conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest. The conversion relationship establishment part includes: Establish the conversion relationship between point cloud and image; The position information of the integrated circuit board is vertex 1 (x b1 ,y b1 ), Vertex 2(x b2 ,y b2 ), Vertex 3(x b3 ,y b3 )、Vertex 4(x b4 ,y b4 ); the bonding wire solder point position information is vertex 1 (x w1 ,y w1 ), Vertex 2(x w2 ,y w2 ), Vertex 3(x w3 ,y w3 )、Vertex 4(x w4 ,y w4 ); the 3D point cloud position information is vertex 1 (x t1 ,y t1 , z t1 ), vertex 2(x t2 ,y t2 , z t2 ), vertex 3(x t3 ,y t3 , z t3 ), vertex 4(x t4 ,y t4 , z t4 ); By converting the ratio between the two diagonal points of the substrate, the conversion relationship between the two-dimensional image and the three-dimensional point cloud is obtained: Among them, D b : Diagonal length of the two-dimensional image substrate; D t : The diagonal length of the 3D image substrate; K: The horizontal range conversion relationship between the 2D image and the 3D point cloud; The position of the bonding wire solder point relative to the substrate is obtained through the range vertex coordinates; ΔL1=x b1 -x w1 ; ΔH1=y b1 -y w1 ; ΔL2=x w3 -x b3 ΔH2=y w3 -y b3 L b =x b3 -x b1 H b =y b2 -y b1 Wherein, ΔL1: x-axis distance from range vertex 1 to substrate vertex 1; ΔH1: y-axis distance from range vertex 1 to substrate vertex 1; ΔL2: x-axis distance from range vertex 3 to substrate vertex 3; ΔH2: y-axis distance from range vertex 3 to substrate vertex 3; L b : Range x-axis length; H b : Range y-axis length; Determine the position range information of solder joints and bonding wires in the three-dimensional point cloud; ΔL t1 =ΔL1×K+t ΔH t1 =ΔH1×K+t ΔL t2 =ΔL2×K+t ΔH t2 =ΔH2×K+t Where, ΔL t1 : The relative position of diagonal point 1 in the x-axis direction on the 3D point cloud; ΔH t1 : The relative position of diagonal point 1 in the y-axis direction on the 3D point cloud; ΔL t2 : The relative position of diagonal point 3 in the x-axis direction on the 3D point cloud; ΔH t2 : The relative position of diagonal point 3 in the y-axis direction on the three-dimensional point cloud; t: dynamic parameter; The position and range information of the bonding wire solder joint detected by YOLOV8 corresponds to the corresponding position and range of the three-dimensional point cloud; The point cloud marking part includes: Assume that the point set of the point cloud is P = (P1, P2, P3, ... P i …P n ), i = 1, 2, ... n, where point P i =(x i ,y i ,z i ,c i ), when the point cloud is in the region of interest c i =1, otherwise c i =0.

2. The method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds according to claim 1, characterized in that: The step of importing the point cloud information within the region of interest into the bonding wire solder point extraction network model for point cloud classification; and classifying the point cloud within the range into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster: The bonding wire solder joint extraction network model includes an improved Pointnet++ network and an outlier removal network; The improved PointNet++ network is used to classify the bonding wires, solder joints and substrate point clouds within the region of interest; The improved PointNet layer single point information encoding is: in, is the single point feature obtained after MLP processing, C is the feature dimension, p i is the coordinate of the sampling point in local space, is the coordinate of a neighborhood point of the sampling point in the local space, is the coordinate difference between the sampling point and the neighborhood point, is the Euclidean distance between the sampling point and the neighborhood point; Improved single local spatial feature aggregation process, introducing spatial and channel attention mechanisms, which are activated after multi-layer perceptron processing; The abnormal point removal network is used to remove outliers and abnormal points of bonding wire point clusters and solder point clusters.

3. The method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds according to claim 1, characterized in that: In the step of selecting the plane fitting method of the random sampling consensus algorithm to perform multiple iterations on the substrate point cloud of the hybrid integrated circuit to seek the best fitting effect: The plane fitting equation is: 0 = ax + by + cz + d; a, b, c, d: plane equation parameters, obtained through fitting algorithm; (x, y, z): x, y, z direction coordinate values; Construct a calculation model for bonding wire solder joints, with a cluster of 1 solder joints as W p ={w1,w2,…w i …,w m }, where w i is the solder point cloud; the bonding wire point cluster is B w ={b1,b2,…b i …,b p }, where b i It is the bonding wire point cloud; The span of the bonding wire is expressed as the horizontal distance between solder joint 1 and solder joint 2; it is obtained by the formula: Among them, d b : Horizontal distance between solder joint 1 and solder joint 2; x p1 : The horizontal coordinate of the projection point cloud of welding point 1 on the fitting plane; x p2 : The horizontal coordinate of the projection point cloud of welding point 2 on the fitting plane; y p1 : The ordinate of the projection point cloud of welding point 1 on the fitting plane; y p2 : The vertical coordinate of the projection point cloud of welding point 2 on the fitting plane; By obtaining the spatial position of the centroid of the solder point 1 cluster and the solder point 2 cluster, and projecting the two centroids onto the fitting plane, the coordinates of the projection point cloud of solder point 1 and solder point 2 on the fitting plane can be obtained. By traversing the horizontal and vertical coordinates of the solder point 1 cluster and the solder point 2 cluster, and performing summation and mean operations, the centroid coordinates can be obtained. The mathematical expression of the centroid coordinates is: Among them, C x : abscissa of the centroid of the cluster; C y : The vertical coordinate of the centroid of the cluster; C z : The vertical coordinate of the centroid of the point cluster; m: The number of point clouds in the point cluster; Calculate the projection coordinates projected to the fitting plane z, which can be calculated by the following formula: Among them, P iproj : Center of mass P i projection on a plane; The height of the bonding wire solder joint can be expressed as the difference between the distance between the bonding wire point cluster and the distance between the solder joint point cluster and the fitting plane; the height of the point cloud from the fitting plane can be expressed as: h: the height of the point cloud from the fitting plane; To ensure the accuracy of the calculation results, the 10 points h where the bonding wire cluster has the highest distance from the fitting plane are obtained by traversing, and the true value of the bonding wire cluster height h is calculated according to the average formula; Among them, Z bmax : True value of bonding wire cluster height; h bimax : The 10 maximum heights h of the point cloud in the bonding wire cluster from the fitting plane; The Z-axis maximum value coordinates of solder joints 1 and 2 are obtained by the formula: h wmax =max(h p ) Among them, h wmax : The maximum height of the solder joint from the fitting plane; max(): maximum value function; h p : The height of the welding point cloud from the fitting plane; Compare the Z-axis maximum values ​​of solder joint 1 and solder joint 2 to determine the positional relationship between the two solder joints; Among them, H wmin : The solder point with a lower average height from the simulated plane; h w1 : average height of one cluster of solder joints; h w2 : The average height of the 2-point cluster of solder joints; When the height of the solder point cluster is lower than the fitting plane, the real value of the arch height is obtained by traversing the average height of the solder point cluster and the average height of the bonding wire cluster. Among them, H g : The true value of the height between the highest point g in the bonding wire and the solder joint 1; The true value of the bonding wire arch height can be obtained according to the average formula; Among them, H is the arch height of the bonding wire. Finally, the arch height and span are compared with the defect standard to determine whether the bonding wire of the solder joint has defects in arch height and span.

Images