Bonding wire three-dimensional defect rapid detection method based on image and point cloud

Abstract

The invention discloses a bonding wire defect detection method based on point cloud. The method comprises the following steps: detecting and positioning integrated circuit image information through a YOLOV8 network; enabling the position detected by the YOLOV8 network to correspond to the point cloud range of the three-dimensional point cloud marking network, and marking the point cloud in the position range as the point cloud of interest; importing the point-of-interest cloud into a bonding wire welding spot extraction network for detection and classification of bonding wires, welding spots and substrates; and carrying out plane fitting on the point cloud of the substrate, and detecting the arch height and span of the bonding wire welding spot. According to the invention, the welding spot bonding wire is directly positioned to a point cloud within a certain range through a two-dimensional positioning and three-dimensional detection technology, so that the detection speed of the welding spot bonding wire is improved; the provided abnormal point elimination network improves the detection precision of the arch height and span of the bonding wire welding spot; a calculation model based on Euclidean is provided, and a space rectangular coordinate system is established; and calculating the arch height information of the bonding wire according to the difference between the vertex center of the bonding wire and the mass center of the welding spot in the Z-axis direction.

Description

Technical Field

[0001] The invention relates to the field of bonding wire feature detection, and in particular to a bonding wire three-dimensional defect rapid detection method based on images and point clouds. Background Art

[0002] The main technology for detecting the height of the bonding wire and the span of the solder joints of the hybrid integrated circuit is based on the point cloud detection technology. Usually, by collecting point cloud data containing depth information, the point clusters of the bonding wire, solder joint and substrate are detected through the PointNet++ algorithm, and finally the point clusters are calculated to obtain the height of the bonding wire and the span of the solder joint. In order to detect the height of the bonding wire and the span of the solder joint, the solder joint and the bonding wire are usually matched after the point clusters of the bonding wire and the solder joint are detected. Since the bonding wire and the solder joint present a one-to-one correspondence in the physical space, they can be matched according to the positional relationship between the two. In this way, the correspondence between the solder joint point cluster and the bonding wire point cluster is established. The matched solder joint cluster and the bonding wire cluster are fitted to obtain the centroid of the solder joint and the bonding wire, and the centroid is calculated to obtain the bonding wire arch height and the solder joint span. The prior art "CN113570550 A method for detecting the size of gold wire bonds based on three-dimensional point clouds" provides a method for detecting the height and span of bonding wire solder joints based on point clouds. This patent uses the PointNet++ network to classify bonding wire point clusters, solder joint point clusters, chip point clouds and baseboard point clouds, and matches the bonding wire solder joint clusters according to the angular relationship between the bonding wire solder joints. The matched bonding wire solder joint clusters are imported into the calculation model to calculate the solder joint span and bonding wire height information; when implementing the bonding wire solder joint detection, the detected bonding wires and solder joints are matched according to the angular relationship. In actual hybrid integrated circuits, the following problems exist: 1. There are many solder joint features in hybrid integrated circuits, such as device solder joints, solder joint protrusions, magazine attachment, etc., and similar features are very likely to cause false detections; 2. The bonding wire solder joints are highly dense, and the method of matching the bonding wire solder joints based on angles is very likely to cause false matching. In the field of point cloud detection, although the PointNet++ network is relatively mature, it is not easy to detect solder joints and bonding wire clusters with high accuracy in hybrid integrated circuit scenarios with small targets, high density, high feature, high complexity and strict local information requirements. If only the method based on angle matching of bonding wire solder joints combined with the PointNet++ network is used, it is difficult to detect the height and span of solder joints and bonding wires. Summary of the invention

[0003] The technical solution of the present invention to solve the above technical problem is to provide a method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds, comprising the following steps: Collect integrated circuit image information, and detect the location and range information of integrated circuit boards and bonding wire solder joints through the YOLOV8 network model; Match the position detected by the YOLOV8 network model with the point cloud range of the 3D point cloud labeling network model; The 3D point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish the conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest; The point cloud information within the region of interest is imported into the bonding wire solder point extraction network model for point cloud classification; the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster; The classified solder point clusters and bonding wire point clusters are processed by the Euclidean distance clustering algorithm. The solder point clusters are clustered by centers, and the bonding wire point clusters are processed by outliers removal, and valid values ​​are retained. The plane fitting method of random sampling consensus algorithm is selected to iterate the substrate point cloud of hybrid integrated circuit for multiple times to obtain the best fitting effect.

[0004] Furthermore, the three-dimensional point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish a conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest in the step: The conversion relationship establishment part includes: establishing a conversion relationship between the point cloud and the image; the integrated circuit board position information is vertex 1 ( , )、Vertex 2( , )、Vertex 3( , )、Vertex 4( , ); The bonding wire solder point position information is vertex 1 ( , )、Vertex 2( , )、Vertex 3( , )、Vertex 4( , ); The three-dimensional point cloud position information is vertex 1 ( , , ), Vertex 2 ( , , ), vertex 3 ( , , ), vertex 4 ( , , ); The conversion relationship between the two-dimensional image and the three-dimensional point cloud is obtained by converting the ratio between the two diagonal points of the substrate: ; ; ;in, : Diagonal length of the two-dimensional image substrate; : diagonal length of the three-dimensional image substrate; : Horizontal range conversion relationship between 2D image and 3D point cloud; the position of bonding wire solder joint relative to substrate is obtained through range vertex coordinates; ; ; ; ; ; ;in, : The x-axis distance of range vertex 1 from substrate vertex 1; : The y-axis distance between range vertex 1 and substrate vertex 1; : The x-axis distance between range vertex 2 and substrate vertex 2; : The y-axis distance between range vertex 2 and substrate vertex 2; : Range x-axis length; : Range y-axis length; determine the position range information of the solder joint and bonding wire in the three-dimensional point cloud; ; ; ; ; in, : The relative position of diagonal point 1 in the x-axis direction on the 3D point cloud; : The relative position of diagonal point 1 in the y-axis direction on the 3D point cloud; : The relative position of diagonal point 3 in the x-axis direction on the 3D point cloud; : The relative position of diagonal point 3 in the y-axis direction on the 3D point cloud; : Dynamic parameters; the position and range information of the bonding wire solder joint detected by YOLOV8 corresponds to the corresponding position and range of the three-dimensional point cloud; The point cloud marking part includes: assuming that the point set of the point cloud is , where point , when the point cloud is in the area of ​​interest ,otherwise .

[0005] Furthermore, the point cloud information within the region of interest is imported into the bonding wire solder point extraction network model for point cloud classification; the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster: the bonding wire solder point extraction network model includes an improved Pointnet++ network and an abnormal point removal network; the improved PointNet++ network: is used to classify the bonding wire, solder point and substrate point clouds within the region of interest The improved PointNet layer single point information encoding is: ;in, for The single point feature obtained after processing, C is the feature dimension, is the coordinate of the sampling point at the center of the local space, is the coordinate of a neighborhood point of the sampling point in the local space, is the coordinate difference between the sampling point and the neighborhood point, is the Euclidean distance between the sampling point and the neighborhood point; Improved single local spatial feature aggregation process, introducing spatial and channel attention mechanisms, which are activated after multi-layer perceptron processing; The abnormal point removal network is used to remove outliers and abnormal points of bonding wire point clusters and solder point clusters.

[0006] Furthermore, in the step of selecting the plane fitting method of the random sampling consensus algorithm to perform multiple iterations on the substrate point cloud of the hybrid integrated circuit to seek the best fitting effect: The plane fitting equation is: ; : Plane equation parameters, obtained through fitting algorithm; : x, y, z direction coordinate values; build a bonding wire solder joint calculation model, and the solder joint 1 point cluster is , where w is the solder point cloud; the bonding wire point cluster is , where b is the bonding wire point cloud; the span of the bonding wire is expressed as the horizontal distance between solder joint 1 and solder joint 2; obtained by the formula: ;in, : The horizontal distance between solder joint 1 and solder joint 2; : The horizontal coordinate of the projection point cloud of welding point 1 on the fitting plane; : The horizontal coordinate of the projection point cloud of welding point 2 on the fitting plane; : The vertical coordinate of the projection point cloud of welding point 1 on the fitting plane; : The vertical coordinate of the projection point cloud of solder joint 2 on the fitting plane; the coordinates of the projection point cloud of solder joint 1 and solder joint 2 on the fitting plane can be obtained by obtaining the spatial position of the centroid of solder joint 1 cluster and solder joint 2 cluster, and projecting the two centroids to the fitting plane. The centroid coordinates can be obtained by traversing the horizontal and vertical coordinates of solder joint 1 cluster and solder joint 2 cluster, and performing summation and mean operations. The mathematical expression of the centroid coordinates is: ; ; ;in, : The horizontal coordinate of the centroid of the cluster; : The vertical coordinate of the centroid of the cluster; : The vertical coordinate of the centroid of the cluster; : The number of point clouds in the point cluster; calculate the projection coordinates projected to the fitting plane z, which can be calculated by the following formula: ;in, : Centroid projection on a plane; The height of the bonding wire solder joint can be expressed as the difference between the value of the bonding wire point cluster from the fitting plane and the value of the solder joint point cluster from the fitting plane; the height of the point cloud from the fitting plane can be expressed as: ; : The height of the point cloud from the fitting plane; To ensure the accuracy of the calculation results, the 10 points h with the highest distance from the bonding wire cluster to the fitting plane are obtained by traversing, and the true value of the bonding wire cluster height h is calculated according to the average formula; ; The Z-axis maximum value coordinates of solder joints 1 and 2 are obtained by the formula: ;in, : The maximum value of the average height of the solder joint cluster from the fitting plane; max() : The maximum value function; : The height of the bonding wire cluster from the fitting plane; Compare the Z-axis maximum values ​​of solder joint 1 and solder joint 2 to determine the positional relationship between the two solder joints; ;in, : The solder joint has a lower average height from the simulated plane; : The average height of the cluster of solder joints; : The average height of the solder point 2 cluster; when the solder point 1 cluster is at a lower height from the fitting plane, the true value of the arch height is obtained by traversing the average height of the solder point 1 cluster and the average height of the bonding wire cluster; ;in, : The true value of the height between the highest point 1 of the bonding wire and the solder joint 1; the true value of the bonding wire arch height can be calculated according to the average formula; ;in, The bonding wire is arched. Finally, the arch height and span are compared with the defect standard to determine whether the bonding wire of the solder joint has defects in arch height and span.

[0007] The present invention detects bonding wire solder joint pairs through mature two-dimensional target detection technology. The positioning and matching of all solder joint bonding wire pairs are completed through YOLOv8 network detection. In all three-dimensional point cloud detections, the detection of bonding wire solder joints is directly positioned, which improves the detection accuracy of bonding wire solder joints.

[0008] Since the traditional technology directly transmits the point cloud data of the hybrid integrated circuit to the PointNet++ network, it should be noted that the diameter of the bonding wire is generally between tens of microns and hundreds of microns, and the detection accuracy of the three-dimensional camera used is at the micron level. The point cloud data generated is very large, and the time used to process the point cloud of the hybrid integrated circuit will be very long. The two-dimensional positioning and three-dimensional detection technology proposed in the present invention directly locates the solder joint bonding wire to the point cloud within a certain range. Only a few points need to be detected to realize the extraction and matching of the bonding wire solder joints, which greatly improves the detection speed of the solder joint bonding wire.

[0009] Since there are outliers and abnormal points in the detected bonding wire solder point clusters, these points have a great influence on the detection results of arch height and span. The abnormal point elimination network proposed in the present invention performs clustering according to the distribution and density of the point clusters, eliminates abnormal points that have a great influence on the results, and gathers the point clusters within the centroid of the solder point and the center of the bonding wire, which greatly improves the detection accuracy of the arch height and span of the bonding wire solder point.

[0010] Since there is no standard reference coordinate system in the space for the processed bonding wire solder point cluster, it is difficult to measure the spatial information of the solder point bonding wire. The Euclidean-based calculation model proposed in the present invention fits the substrate plane according to the plane fitting algorithm, takes the substrate plane as the XY axis, and takes the normal vector perpendicular to the substrate plane as the Y axis to establish a spatial rectangular coordinate system. The solder point span information is calculated based on the Euclidean distance between the centroids of the two solder points in the XY plane, and the bonding wire arch height information is calculated based on the difference between the center of the bonding wire vertex and the centroid of the solder point in the Z-axis direction.

[0011] Compared with the prior art, the present invention has a faster detection speed. The existing detection method transmits the entire hybrid integrated circuit point cloud (millions of points) to the network for detection, which has a large amount of network calculation and slow model reasoning speed. However, the present invention only imports the point cloud of the solder joint bonding wire area into the network (tens of thousands of points), which greatly reduces the amount of network calculation and speeds up the model reasoning speed.

[0012] Compared with the prior art, the present invention has higher detection accuracy. The existing detection method matches and associates the angle relationship between the solder joint bonding wires. When faced with a large number of hybrid integrated circuits with high density, the solder joint bonding wire matching accuracy is low. However, the present invention uses image processing technology to lock only one set of solder joint bonding wires in the area, so there is no matching problem, which greatly improves the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0014] Figure 1 It is a structural schematic diagram of the method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds according to the present invention; Figure 2 This is a schematic diagram of the point cloud marking network of the present invention; Figure 3 A network structure diagram of a solder joint extraction network of the present invention; Figure 4 This is a schematic diagram of an outlier removal network of the present invention; Figure 5 This is a flow chart of the random sampling consensus algorithm of the present invention. DETAILED DESCRIPTION

[0015] The present invention proposes a method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds, aiming to.

[0016] The method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds proposed by the present invention will be described below in a specific embodiment: Example 1: A method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds, such as Figure 1 As shown, the following steps are included:

[0017] S10: Collect integrated circuit image information, and detect the position and range information of the integrated circuit board and bonding wire solder joints through the YOLOV8 network model; S21: Match the position detected by the YOLOV8 network model with the point cloud range of the 3D point cloud labeling network model; S22: The three-dimensional point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which is used to establish a conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as a region of interest; S23: importing the point cloud information within the region of interest into the bonding wire solder point extraction network model for point cloud classification; dividing the point cloud within the range into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster; S24: The classified solder point clusters and bonding wire clusters are processed by the Euclidean distance clustering algorithm, the solder point clusters are subjected to center clustering processing, the bonding wire clusters are subjected to outlier removal processing, and valid values ​​are retained; S30: Selecting the plane fitting method of the random sampling consensus algorithm, performing multiple iterations on the substrate point cloud of the hybrid integrated circuit to obtain the best fitting effect; S40: Calculate the arch height and span of the bonding wire of the solder joint and determine the defect type; Specifically, the solder point cluster, bonding wire point cluster, and plane fitting equation are transmitted to the Euclidean calculation model. By solving operations such as centroid, projection, and spatial distance, the span and arch height information of the two solder points are calculated, and the defect type is determined based on the results.

[0018] It can be understood that the Euclidean calculation model includes the steps of the Euclidean distance calculation method, and also extends the centroid solution and projection operations. The core of the bond wire solder joint calculation model is this Euclidean calculation model.

[0019] Furthermore, the three-dimensional point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish a conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest in the step: The conversion relationship establishment part includes: establishing a conversion relationship between the point cloud and the image; the integrated circuit board position information is vertex 1 ( , )、Vertex 2( , )、Vertex 3( , )、Vertex 4( , ); The bonding wire solder point position information is vertex 1 ( , )、Vertex 2( , )、Vertex 3( , )、Vertex 4( , ); The three-dimensional point cloud position information is vertex 1 ( , , ), Vertex 2 ( , , ), vertex 3 ( , , ), vertex 4 ( , , ); The conversion relationship between the two-dimensional image and the three-dimensional point cloud is obtained by converting the ratio between the two diagonal points of the substrate: ; ; ;in, : Diagonal length of the two-dimensional image substrate; : diagonal length of the three-dimensional image substrate; : Horizontal range conversion relationship between 2D image and 3D point cloud; The position of the bonding wire solder point relative to the substrate is obtained through the range vertex coordinates; ; ; ; ; ; ; in, : The x-axis distance of range vertex 1 from substrate vertex 1; : The y-axis distance between range vertex 1 and substrate vertex 1; : The x-axis distance between range vertex 2 and substrate vertex 2; : The y-axis distance between range vertex 2 and substrate vertex 2; : Range x-axis length; : Range y-axis length; Determine the position range information of solder joints and bonding wires in the three-dimensional point cloud; ; ; ; ; in, : The relative position of diagonal point 1 in the x-axis direction on the 3D point cloud; : The relative position of diagonal point 1 in the y-axis direction on the 3D point cloud; : The relative position of diagonal point 3 in the x-axis direction on the 3D point cloud; : The relative position of diagonal point 3 in the y-axis direction on the 3D point cloud; : Dynamic parameters; the position and range information of the bonding wire solder joint detected by YOLOV8 corresponds to the corresponding position and range of the three-dimensional point cloud; The point cloud marking part includes: assuming that the point set of the point cloud is , where point , when the point cloud is in the area of ​​interest ,otherwise .

[0020] Further, the point cloud information within the region of interest is imported into the bonding wire solder point extraction network model for point cloud classification; the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster: The bonding wire solder joint extraction network model includes an improved Pointnet++ network and an abnormal point removal network; the improved PointNet++ network is used to classify the bonding wire, solder joint and substrate point cloud in the region of interest; The improved PointNet layer single point information encoding is: ;in, for The single point feature obtained after processing, C is the feature dimension, is the coordinate of the sampling point at the center of the local space, is the coordinate of a neighborhood point of the sampling point in the local space, is the coordinate difference between the sampling point and the neighborhood point, is the Euclidean distance between the sampling point and the neighborhood point; Improved single local spatial feature aggregation process, introducing spatial and channel attention mechanisms, which are activated after multi-layer perceptron processing; The abnormal point removal network is used to remove outliers and abnormal points of bonding wire point clusters and solder point clusters.

[0021] Furthermore, in the step of selecting the plane fitting method of the random sampling consensus algorithm to perform multiple iterations on the substrate point cloud of the hybrid integrated circuit to seek the best fitting effect: The plane fitting equation is: ; : Plane equation parameters, obtained through fitting algorithm; : x, y, z direction coordinate values; build a bonding wire solder joint calculation model, and the solder joint 1 point cluster is , where w is the solder point cloud; the bonding wire point cluster is , where b is the bonding wire point cloud; the span of the bonding wire is expressed as the horizontal distance between solder joint 1 and solder joint 2; obtained by the formula: ;in, : The horizontal distance between solder joint 1 and solder joint 2; : The horizontal coordinate of the projection point cloud of welding point 1 on the fitting plane; : The horizontal coordinate of the projection point cloud of welding point 2 on the fitting plane; : The vertical coordinate of the projection point cloud of welding point 1 on the fitting plane; : The vertical coordinate of the projection point cloud of solder joint 2 on the fitting plane; the coordinates of the projection point cloud of solder joint 1 and solder joint 2 on the fitting plane can be obtained by obtaining the spatial position of the centroid of solder joint 1 cluster and solder joint 2 cluster, and projecting the two centroids to the fitting plane. The centroid coordinates can be obtained by traversing the horizontal and vertical coordinates of solder joint 1 cluster and solder joint 2 cluster, and performing summation and mean operations. The mathematical expression of the centroid coordinates is: ; ; ;in, : The horizontal coordinate of the centroid of the cluster; : The vertical coordinate of the centroid of the cluster; : The vertical coordinate of the centroid of the cluster; : The number of point clouds in the point cluster; calculate the projection coordinates projected to the fitting plane z, which can be calculated by the following formula: ;in, : Centroid projection on a plane; The height of the bonding wire solder joint can be expressed as the difference between the value of the bonding wire point cluster from the fitting plane and the value of the solder joint point cluster from the fitting plane; the height of the point cloud from the fitting plane can be expressed as: ; : The height of the point cloud from the fitting plane; To ensure the accuracy of the calculation results, the 10 points h with the highest distance from the bonding wire cluster to the fitting plane are obtained by traversing, and the true value of the bonding wire cluster height h is calculated according to the average formula; ; The Z-axis maximum value coordinates of solder joints 1 and 2 are obtained by the formula: ;in, : The maximum value of the average height of the solder joint cluster from the fitting plane; max() : The maximum value function; : The height of the bonding wire cluster from the fitting plane; Compare the Z-axis maximum values ​​of solder joint 1 and solder joint 2 to determine the positional relationship between the two solder joints; ;in, : The solder joint has a lower average height from the simulated plane; : The average height of the cluster of solder joints; : The average height of the solder point 2 cluster; when the solder point 1 cluster is at a lower height from the fitting plane, the true value of the arch height is obtained by traversing the average height of the solder point 1 cluster and the average height of the bonding wire cluster; ;in, : The true value of the height between the highest point 1 of the bonding wire and the solder joint 1; the true value of the bonding wire arch height can be calculated according to the average formula; ;in, The bonding wire is arched. Finally, the arch height and span are compared with the defect standard to determine whether the bonding wire of the solder joint has defects in arch height and span.

[0022] Embodiment 2: A method for efficiently detecting bonding wires based on point cloud, characterized in that it comprises the following steps: Step 1: Step 1.1: Use an industrial camera to collect integrated circuit image information and transmit the image information to the improved YOLOV8 network. The YOLOV8 network detects the position and range information of the hybrid integrated circuit board and the bonding wire solder joints.

[0023] Step 1.2: The YOLOV8 network transmits the detected position and range information to the 3D point cloud labeling network in the form of coordinate information.

[0024] Step 2: Step 2.1: The 3D point cloud labeling network is responsible for matching the position detected by the YOLOV8 network with the point cloud range. The 3D point cloud labeling network consists of two parts: the conversion relationship establishment part and the point cloud labeling part.

[0025] Transformation relationship establishment part: The transformation relationship establishment part is responsible for establishing the transformation relationship between the point cloud and the image. The position information of the integrated circuit board is vertex 1 ( , )、Vertex 2( , )、Vertex 3( , )、Vertex 4( , ); The bonding wire solder point position information is vertex 1 ( , )、Vertex 2( , )、Vertex 3( , )、Vertex 4( , ); The three-dimensional point cloud position information is vertex 1 ( , , ), Vertex 2 ( , , ), vertex 3 ( , , ), vertex 4 ( , , ); The conversion relationship between the two-dimensional image and the three-dimensional point cloud is obtained by converting the ratio between the two diagonal points of the substrate: ; ; ;in, : Diagonal length of the two-dimensional image substrate; : diagonal length of the three-dimensional image substrate; : Horizontal range conversion relationship between 2D image and 3D point cloud; The position of the bonding wire solder point relative to the substrate is obtained through the range vertex coordinates; ; ; ; ; ; ; in, : The x-axis distance of range vertex 1 from substrate vertex 1; : The y-axis distance between range vertex 1 and substrate vertex 1; : The x-axis distance between range vertex 2 and substrate vertex 2; : The y-axis distance between range vertex 2 and substrate vertex 2; : Range x-axis length; : Range y-axis length; Determine the position range information of solder joints and bonding wires in the three-dimensional point cloud; ; ; ; ; in, : The relative position of diagonal point 1 in the x-axis direction on the 3D point cloud; : The relative position of diagonal point 1 in the y-axis direction on the 3D point cloud; : The relative position of diagonal point 3 in the x-axis direction on the 3D point cloud; : The relative position of diagonal point 3 in the y-axis direction on the 3D point cloud; : Dynamic parameters; the position and range information of the bonding wire solder joint detected by YOLOV8 corresponds to the corresponding position and range of the three-dimensional point cloud; The point cloud marking part includes: assuming that the point set of the point cloud is , where point , when the point cloud is in the area of ​​interest ,otherwise The point cloud labeling network is as follows Figure 2 As shown, the black dots represent the original point cloud and the blue dots represent the point cloud of interest.

[0026] Step 2.2: Import the point cloud information within the region of interest into the bonding wire solder joint extraction network for point cloud classification.

[0027] The network structure of the bonding wire solder joint extraction network is as follows: Figure 3 As shown in Figure 2, the network mainly consists of two parts, namely the improved Pointnet++ network and the outlier removal network.

[0028] Improved PointNet++ network: used to classify bonding wires, solder joints and substrate point clouds within the region of interest; Outlier point removal network: used to remove outliers and abnormal points from bonding wire point clusters and solder point clusters.

[0029] Among them, in order to solve the problem of difficulty in extracting local information of point clouds, the following improvements are made to the PointNet++ network: A PointNet layer structure is designed to enhance local information encoding. The new local space encoding method not only retains the original coordinates of the point, but also adds the relative coordinates and Euclidean distance information of the point relative to the central sampling point. By adding local information, the network can encode local geometric and structural information in detail and accurately.

[0030] The improved single-point information encoding of the PointNet layer is as follows: in, for The single point feature obtained after processing, C is the feature dimension, is the coordinate of the sampling point at the center of the local space, is the coordinate of a neighborhood point of the sampling point in the local space, is the coordinate difference between the sampling point and the neighborhood point, is the Euclidean distance between the sampling point and the neighborhood point; this design introduces a spatial and channel attention mechanism, which is activated after the multi-layer perceptron (MLP) processing, aiming to accurately select the spatial regions and channels that are most critical to improving network performance. Through this strategy, the network can focus on those features that have a significant impact on the final judgment result, while ignoring those information that is less error-inducing or does not contribute much to the result. This not only improves the detection accuracy of the network, but also optimizes the processing speed, thereby achieving more efficient learning and recognition performance.

[0031] Step 2.3: The point cloud in the region of interest is processed by the improved Pointnet++ network, and the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster.

[0032] Step 2.4: The classified solder point clusters and bonding wire clusters are processed by the Euclidean distance clustering algorithm. The solder point clusters are clustered by center, and the bonding wire clusters are processed by outlier removal. The outliers and outliers that have a greater impact on the detection accuracy are eliminated, and the valid values ​​are retained. Figure 4 As shown, the black points represent the original point cloud, the blue points represent the point cloud of interest, the brown points represent the point cloud of the solder joints, and the light blue points represent the point cloud of the bonding wires.

[0033] Step 3: Step 3.1: Select the plane fitting method of the random sampling consensus algorithm, perform multiple iterations on the substrate point cloud of the hybrid integrated circuit to seek the best fitting effect, and obtain the fitting plane equation after fitting: ; : Plane equation parameters, obtained through fitting algorithm; : x, y, z direction coordinate values; build a bonding wire solder joint calculation model, and the solder joint 1 point cluster is , where w is the solder point cloud; the bonding wire point cluster is , where b is the bonding wire point cloud; the span of the bonding wire is expressed as the horizontal distance between solder joint 1 and solder joint 2; obtained by the formula: ;in, : The horizontal distance between solder joint 1 and solder joint 2; : The horizontal coordinate of the projection point cloud of welding point 1 on the fitting plane; : The horizontal coordinate of the projection point cloud of welding point 2 on the fitting plane; : The vertical coordinate of the projection point cloud of welding point 1 on the fitting plane; : The vertical coordinate of the projection point cloud of solder joint 2 on the fitting plane; the coordinates of the projection point cloud of solder joint 1 and solder joint 2 on the fitting plane can be obtained by obtaining the spatial position of the centroid of solder joint 1 cluster and solder joint 2 cluster, and projecting the two centroids to the fitting plane. The centroid coordinates can be obtained by traversing the horizontal and vertical coordinates of solder joint 1 cluster and solder joint 2 cluster, and performing summation and mean operations. The mathematical expression of the centroid coordinates is: ; ; ;in, : The horizontal coordinate of the centroid of the cluster; : The vertical coordinate of the centroid of the cluster; : The vertical coordinate of the centroid of the cluster; : The number of point clouds in the point cluster; calculate the projection coordinates projected to the fitting plane z, which can be calculated by the following formula: ;in, : Centroid projection on a plane; The height of the bonding wire solder joint can be expressed as the difference between the value of the bonding wire point cluster from the fitting plane and the value of the solder joint point cluster from the fitting plane; the height of the point cloud from the fitting plane can be expressed as: ; : The height of the point cloud from the fitting plane; To ensure the accuracy of the calculation results, the 10 points h with the highest distance from the bonding wire cluster to the fitting plane are obtained by traversing, and the true value of the bonding wire cluster height h is calculated according to the average formula; ; The Z-axis maximum value coordinates of solder joints 1 and 2 are obtained by the formula: ;in, : The maximum value of the average height of the solder joint cluster from the fitting plane; max() : The maximum value function; : The height of the bonding wire cluster from the fitting plane; Compare the Z-axis maximum values ​​of solder joint 1 and solder joint 2 to determine the positional relationship between the two solder joints; ;in, : The solder joint has a lower average height from the simulated plane; : The average height of the cluster of solder joints; : The average height of the solder point 2 cluster; when the solder point 1 cluster is at a lower height from the fitting plane, the true value of the arch height is obtained by traversing the average height of the solder point 1 cluster and the average height of the bonding wire cluster; ;in, : The true value of the height between the highest point 1 of the bonding wire and the solder joint 1; the true value of the bonding wire arch height can be calculated according to the average formula; ;in, The bonding wire is arched. Finally, the arch height and span are compared with the defect standard to determine whether the bonding wire of the solder joint has defects in arch height and span.

[0034] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds, characterized in that: The following steps are involved: Collect integrated circuit image information, and detect the location and range information of integrated circuit boards and bonding wire solder joints through the YOLOV8 network model; Match the position detected by the YOLOV8 network model with the point cloud range of the 3D point cloud labeling network model; The 3D point cloud labeling network model includes a conversion relationship establishment part and a point cloud labeling part, which are used to establish the conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest; The point cloud information within the region of interest is imported into the bonding wire solder point extraction network model for point cloud classification; the point cloud within the range is divided into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster; The classified solder point clusters and bonding wire point clusters are processed by the Euclidean distance clustering algorithm. The solder point clusters are clustered by centers, and the bonding wire point clusters are processed by outliers removal, and valid values ​​are retained. The plane fitting method of random sampling consensus algorithm is selected to iterate the substrate point cloud of hybrid integrated circuit for multiple times to obtain the best fitting effect.

2. The method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds according to claim 1, characterized in that: The three-dimensional point cloud marking network model includes a conversion relationship establishment part and a point cloud marking part, which are used to establish a conversion relationship between the point cloud and the image, and set the point cloud within the range obtained by the conversion relationship as the region of interest. The conversion relationship establishment part includes: establishing a conversion relationship between the point cloud and the image; the integrated circuit board position information is vertex 1 ( , )、Vertex 2( , )、Vertex 3( , )、Vertex 4( , ); The bonding wire solder point position information is vertex 1 ( , )、Vertex 2( , )、Vertex 3( , )、Vertex 4( , ); The three-dimensional point cloud position information is vertex 1 ( , , ), Vertex 2 ( , , ), vertex 3 ( , , ), vertex 4 ( , , ); The conversion relationship between the two-dimensional image and the three-dimensional point cloud is obtained by converting the ratio between the two diagonal points of the substrate: ; ; ;in, : Diagonal length of the two-dimensional image substrate; : diagonal length of the three-dimensional image substrate; : Horizontal range conversion relationship between 2D image and 3D point cloud; The position of the bonding wire solder point relative to the substrate is obtained through the range vertex coordinates; ; ; ; ; ; ; in, : The x-axis distance of range vertex 1 from substrate vertex 1; : The y-axis distance between range vertex 1 and substrate vertex 1; : The x-axis distance between range vertex 2 and substrate vertex 2; : The y-axis distance between range vertex 2 and substrate vertex 2; : Range x-axis length; : Range y-axis length; Determine the position range information of solder joints and bonding wires in the three-dimensional point cloud; ； ； ； ； in, : The relative position of diagonal point 1 in the x-axis direction on the 3D point cloud; : The relative position of diagonal point 1 in the y-axis direction on the 3D point cloud; : The relative position of diagonal point 3 in the x-axis direction on the 3D point cloud; : The relative position of diagonal point 3 in the y-axis direction on the 3D point cloud; : Dynamic parameters; the position and range information of the bonding wire solder joint detected by YOLOV8 corresponds to the corresponding position and range of the three-dimensional point cloud; The point cloud marking part includes: assuming that the point set of the point cloud is , where point , when the point cloud is in the area of ​​interest ,otherwise .

3. The method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds according to claim 1, characterized in that: The step of importing the point cloud information within the region of interest into the bonding wire solder point extraction network model for point cloud classification; and classifying the point cloud within the range into four categories: solder point 1 cluster, solder point 2 cluster, bonding wire point cluster and substrate point cluster: The bonding wire solder joint extraction network model includes an improved Pointnet++ network and an abnormal point removal network; the improved PointNet++ network is used to classify the bonding wire, solder joint and substrate point cloud in the region of interest; The improved PointNet layer single point information encoding is: ;in, for The single point feature obtained after processing, C is the feature dimension, is the coordinate of the sampling point in local space, is the coordinate of a neighborhood point of the sampling point in the local space, is the coordinate difference between the sampling point and the neighborhood point, is the Euclidean distance between the sampling point and the neighborhood point; Improved single local spatial feature aggregation process, introducing spatial and channel attention mechanisms, which are activated after multi-layer perceptron processing; The abnormal point removal network is used to remove outliers and abnormal points of bonding wire point clusters and solder point clusters.

4. The method for rapid detection of three-dimensional defects of bonding wires based on images and point clouds according to claim 1, characterized in that: In the step of selecting the plane fitting method of the random sampling consensus algorithm to perform multiple iterations on the substrate point cloud of the hybrid integrated circuit to seek the best fitting effect: The plane fitting equation is: ; : Plane equation parameters, obtained through fitting algorithm; : x, y, z direction coordinate values; build a bonding wire solder joint calculation model, and the solder joint 1 point cluster is , where w is the solder point cloud; the bonding wire point cluster is , where b is the bonding wire point cloud; the span of the bonding wire is expressed as the horizontal distance between solder joint 1 and solder joint 2; obtained by the formula: ;in, : The horizontal distance between solder joint 1 and solder joint 2; : The horizontal coordinate of the projection point cloud of welding point 1 on the fitting plane; : The horizontal coordinate of the projection point cloud of welding point 2 on the fitting plane; : The vertical coordinate of the projection point cloud of welding point 1 on the fitting plane; : The vertical coordinate of the projection point cloud of solder joint 2 on the fitting plane; the coordinates of the projection point cloud of solder joint 1 and solder joint 2 on the fitting plane can be obtained by obtaining the spatial position of the centroid of solder joint 1 cluster and solder joint 2 cluster, and projecting the two centroids to the fitting plane. The centroid coordinates can be obtained by traversing the horizontal and vertical coordinates of solder joint 1 cluster and solder joint 2 cluster, and performing summation and mean operations. The mathematical expression of the centroid coordinates is: ; ; ;in, : The horizontal coordinate of the centroid of the cluster; : The vertical coordinate of the centroid of the cluster; : The vertical coordinate of the centroid of the cluster; : The number of point clouds in the point cluster; calculate the projection coordinates projected to the fitting plane z, which can be calculated by the following formula: ;in, : Centroid projection on a plane; The height of the bonding wire solder joint can be expressed as the difference between the distance between the bonding wire point cluster and the distance between the solder joint point cluster and the fitting plane; the height of the point cloud from the fitting plane can be expressed as: ; : The height of the point cloud from the fitting plane; To ensure the accuracy of the calculation results, the 10 points h with the highest distance from the bonding wire cluster to the fitting plane are obtained by traversing, and the true value of the bonding wire cluster height h is calculated according to the average formula; ; The Z-axis maximum value coordinates of solder joints 1 and 2 are obtained by the formula: ;in, : The maximum value of the average height of the solder joint cluster from the fitting plane; max() : The maximum value function; : The height of the bonding wire cluster from the fitting plane; Compare the Z-axis maximum values ​​of solder joint 1 and solder joint 2 to determine the positional relationship between the two solder joints; ;in, : The solder joint has a lower average height from the simulated plane; : The average height of the cluster of solder joints; : The average height of the solder point 2 cluster; when the solder point 1 cluster is at a lower height from the fitting plane, the true value of the arch height is obtained by traversing the average height of the solder point 1 cluster and the average height of the bonding wire cluster; ;in, : The true value of the height between the highest point 1 of the bonding wire and the solder joint 1; the true value of the bonding wire arch height can be calculated according to the average formula; ;in, The bonding wire is arched. Finally, the arch height and span are compared with the defect standard to determine whether the bonding wire of the solder joint has defects in arch height and span.

Images