Three-dimensional point cloud bonding wire defect detection method based on global-local importance feature fusion

Abstract

The invention provides a bonding wire defect detection method based on global-local importance feature fusion, and solves the problems that in an existing method, structure defects cannot be effectively detected, the detection precision is insufficient, and three-dimensional bonding wire defects with rough edges cannot be segmented. And the sensitive area of the input feature and the weak link in the detection process cannot be revealed. The process comprises the following steps: acquiring a depth image of a PCB, converting a bonding wire depth image point column into point cloud data through matrix operation, performing data preprocessing on bonding wire point cloud by using Cloud Compare software to obtain simplified and optimized bonding wire data, and making a marked point cloud bonding wire data set; after the data set is divided, a three-dimensional point cloud bonding wire defect detection model based on global-local importance feature fusion is trained, finally, normal bonding wire detection, radian abnormity detection, wire bending detection and the like are carried out on a segmented bonding wire plane, and instance segmentation is carried out on the eight bonding wire targets.

Description

Technical Field

[0001] The present invention belongs to the field of bonding wire defect detection in the wire bonding process of integrated circuit packaging, and specifically relates to a method for detecting bonding wire structural defects in wire welding by using a global-local importance feature fusion method and determining the bonding wire defect type. Background Art

[0002] Bonding wires play a vital role in modern electronic devices and are the key part for connecting various components. Especially in miniaturized and highly integrated electronic products, bonding wires not only ensure the effective transmission of signals, but also maintain the stable flow of current. Wire bonding technology is widely used in the industrial manufacturing of products such as printed circuit boards PCBs, lithium batteries and LED devices, and its quality directly affects the performance and stability of the products. However, bonding wire defects, such as broken wires, collapsed wires and welding misalignment, are often subtle and complex, which can lead to poor connections, thereby affecting the reliability and function of the product. Therefore, fast and accurate bonding wire defect detection is crucial to improving the overall reliability of the product, and bonding wire defect detection has become the basis and key to improving the quality of related products.

[0003] At present, the detection of bonding wire defects mainly relies on manual visual inspection and automated inspection based on two-dimensional images. Although manual visual inspection can intuitively find problems, it is inefficient, time-consuming and labor-intensive, and it is difficult to meet the speed and accuracy requirements of modern large-scale production. Image-based two-dimensional detection technology uses a machine vision system to analyze the bonding wire image to identify defects. Since two-dimensional images cannot fully reflect the spatial structure of bonding wires, especially when detecting three-dimensional structural defects such as bending and breaking of bonding wires, multi-angle shooting and analysis are often required to obtain the height information of bonding wires, which increases the complexity and cost of detection. More importantly, when the bonding wire is similar to the background color, traditional image processing technology is difficult to accurately locate the bonding wire, resulting in reduced detection accuracy. These limitations make two-dimensional image technology appear to be powerless when facing complex defects in bonding wires. Therefore, detection technology based on three-dimensional point cloud data has gradually become a new research and application hotspot.

[0004] Three-dimensional point cloud data can provide the complete three-dimensional geometric features of an object, which not only covers basic features such as shape and size, but also can intuitively reflect the spatial position information. These characteristics make point cloud data have obvious advantages in defect detection of bonding wires with complex spatial structures. In the field of industrial manufacturing, point cloud technology has been widely used in aerospace, railway maintenance, electronic component manufacturing and other fields. In these fields, point cloud technology effectively solves the problems of spatial positioning and defect detection of complex structures by capturing the three-dimensional form of objects. Despite this, most existing defect detection methods focus on the smoothness and integrity detection of object surfaces, such as cracks, depressions and protrusions. For small and curved structures such as bonding wires, especially bonding wires are usually closely connected to other components, the acquired point cloud data often faces adhesion and noise problems, which limits the segmentation effect. In addition, there are many types of defects in bonding wires, including bending, broken wires, collapsed wires, etc. The three-dimensional characteristics of these defects are different, and traditional three-dimensional detection methods are difficult to effectively capture. Therefore, the method that solely relies on surface smoothness analysis is not suitable for the detection of bonding wires. How to accurately identify the multi-scale important features of bonding wires and segment the point cloud of defective bonding wires becomes the primary problem. At present, the in-depth analysis method for the spatial structure of bonding wires is not yet mature, and the method of multi-feature learning for self-built bonding wire datasets is also scarce. Therefore, it is urgent to develop a new bonding wire defect detection method that combines the advantages of three-dimensional point cloud data. Summary of the invention

[0005] The main purpose of the present invention is to propose a three-dimensional point cloud bonding wire defect detection method based on global-local importance feature fusion. This method can be widely used in the bonding wire defect detection of integrated circuit packaging wire bonding process, replacing the traditional manual visual inspection and image-based two-dimensional detection methods, realizing the automatic detection of three-dimensional point cloud bonding wire defects based on global-local importance feature fusion, and classifying according to the detected defect type. The present invention aims to solve the following problems:

[0006] 1. The current bonding wire defect detection technology mainly relies on two-dimensional visual inspection, which is limited to the identification of broken wires and cannot effectively detect three-dimensional structural defects such as wire bending and curvature abnormalities;

[0007] 2. When dealing with sophisticated and complex bonding wire defects, the existing model lacks in-depth extraction of the importance of global-local features, resulting in low detection accuracy;

[0008] 3. Existing bonding wire defect detection methods cannot segment three-dimensional bonding wire defect targets with rough edges, and thus cannot reveal the sensitive areas of the input features and the weak links in the detection process.

[0009] In order to solve the above problems, the present invention proposes a three-dimensional point cloud bonding wire defect detection method based on global-local importance feature fusion. The method first converts the depth map into three-dimensional point cloud data, removes various environmental noises introduced in the data acquisition process, and crops the bonding wire target area to obtain a complete bonding wire plane. Secondly, the local important energy attention feature extraction module and the global information position encoding module are introduced to solve the problem that the benchmark point cloud segmentation network PointNet++ cannot extract the global and local importance features in the bonding wire data. Thirdly, for the three-dimensional point cloud bonding wire defect detection task, the model's fitting ability is further enhanced by calculating the quantization loss of the extracted high-dimensional embedded features and calculating the soft similarity and hard similarity balanced metric loss of the defect target. Finally, the three-dimensional interpolation and PointNet upsampling network are used to output the seven abnormal defects of the point cloud bonding wire data, including abnormal curvature, wire bending, long tail wire, double-layer interference, lead bridging, middle wire breakage, and single-end wire breakage, and the bonding wire with rough edges is accurately segmented, so as to better display the sensitive areas of bonding wire defects and the weak links in the welding process.

[0010] The specific working process of the present invention is as follows:

[0011] Step 1: First, use a 3D depth camera to scan the welding area and obtain the original depth image of the bonding wire. Through the information of each pixel point in the depth image, it is converted into three-dimensional point cloud data to obtain the spatial coordinates (x, y, z) of each pixel point.

[0012] Step 2: Perform data preprocessing on the acquired point cloud image. Open the point cloud image with CloudCompare software, and use the manual cropping function of the software to select irrelevant areas in the acquired data for cropping operations to obtain the bonding wire target area and obtain a complete bonding wire plane.

[0013] Step 3: Perform contour detection on the point cloud image, calculate the circumscribed rectangle of each contour, and obtain the aspect ratio of the bounding box. Use the median absolute difference (MAD) algorithm to effectively distinguish the bonding wire contour from other contours, focusing on the extraction of bonding wire features.

[0014] Step 4: extract the point cloud data within the outline of the bonding wire and perform matrix operations to obtain the point cloud data of the bonding wire. Statistical filtering and principal component analysis (PCA) are used to process the point cloud data to remove irrelevant background point clouds and noise and focus on the geometric features of the bonding wire.

[0015] Step 5: Project the bonding wire point cloud data constructed in step 4 along the XZ axis, calculate the interval size between adjacent points, and use the 3-sigma algorithm to filter out the broken wires within the wire to obtain the bonding wire point cloud data set with non-broken wires, and mark each type of defect.

[0016] Step 6: Next, the model is trained on the preprocessed 3D point cloud bonding wire data to achieve instance segmentation. The local important energy attention feature extraction module and the global information position encoding module are introduced to extract the global features and local features of the bonding wire at the same time.

[0017] Step 7: Design a compound loss during the training process to perform quantization loss calculation on the extracted high-dimensional embedding features, and perform balanced metric loss calculation on the defect targets for soft similarity and hard similarity.

[0018] Step 8. Finally, the trained model is output to output various abnormal defects of the bonding wire point cloud data, including seven defect types: abnormal curvature, wire bending, excessive tail wire length, double-layer interference, wire bridging, middle wire breakage, and single-end wire breakage.

[0019] Steps 1-5 are as follows:

[0020] The professional point cloud processing software CloudCompare is used to convert the deep bonding wire defect image obtained by the depth camera into point cloud data. By obtaining the pixel information of the depth image, CloudCompare software displays its three-dimensional point cloud form. Then, using the point cloud data editing function of CloudCompare software, the environmental noise existing in the bonding wire defect data converted into point cloud form is selected and deleted, and through the active box selection setting, other scene data except the bonding wire plane is segmented, and the point cloud data within the bonding wire plane is retained. Finally, through the annotation function of CloudCompare software, seven defective target bonding wires are annotated, including abnormal curvature, wire bending, too long tail wire, double-layer interference, lead bridging, middle wire breakage, and single-end wire breakage.

[0021] Step 6 is as follows:

[0022] The entire model is built based on the Pytorch framework and consists of three convolutional layers and four fully connected layers. The convolutional layer uses a convolution kernel size of "1×1", a step size of "1×1", and an activation function of ReLU. First, the point cloud bond wire data is sampled and grouped, and the neighborhood ball is used to define the partition. Each area is determined by the center coordinates and radius. The selection of the center coordinates is implemented using the farthest point sampling algorithm. Specifically, the input points are first sampled through the sampling layer, and several center points are selected from these points; then the point set is divided into several areas using the center points obtained in the previous step through the grouping layer; finally, each area obtained above is encoded in the Pointnet layer and converted into a feature vector.

[0023] The input of each set of extraction layers is (N, (d+C)), where N is the number of input points, d is the coordinate dimension, and C is the feature dimension. The output is (N′, (d+C′)), where N′ is the number of output points, d is the unchanged coordinate dimension, and C′ is the new feature dimension. The following is a detailed introduction to the role and implementation process of each layer.

[0024] The sampling layer uses the farthest point sampling method to select N' points. First, the first point A is selected from the point cloud as the query point, and a point B with the farthest distance is selected from the remaining points; then the points A and B taken out are used as query points, and the point F with the farthest distance is taken from the remaining points. At this time, since the number of points that have been taken out exceeds 1, all query points (A, B) need to be considered at the same time. The method is as follows: For any point P among the remaining points, calculate the distance from the point P to all points (A, B) in the selected point set; take the minimum distance from points A and B as the distance d from the point to the selected point set; after calculating the distance from each remaining point to the point set, select the point with the largest distance, which is point F. 3. Repeat step 2 until N' points are sampled.

[0025] The grouping layer uses the ball query method to divide the center point sampled by the sampling layer into several corresponding local areas, selects the adjacent points of the sampling center point, sends the selected sub-point cloud to the feature extraction layer, and encodes the information of the local area into a feature vector.

[0026] The input of the PointNet layer is a local area: (N, K, (d + C)), where N is the number of input points, d is the coordinate dimension, and C is the feature dimension. The output is (N′, (d + C′)), where N' is the number of output points, d is the unchanged coordinate dimension, and C' is the new feature dimension. The PointNet layer extracts features from each group of point clouds to obtain local features. The network passes through two T-Net and MLP operations. T-Net aligns the input points in space, rotates them to the front, and then maps them to a 1024-dimensional space through MLP. Through the maximum pooling layer, only the maximum channel is retained on all channels of 1024 dimensions, thereby obtaining the global features of N point clouds. Then, two fully connected layers are used for dimensionality reduction. The fully connected layer consists of two 1x1 convolutional layers, where the first convolutional layer is used to reduce the feature vector, and the second convolutional layer is used to enlarge the reduced feature vector. The fully connected layer uses Sigmoid as the activation function, and Dropout is added to prevent overfitting. After the second fully connected layer, a maximum pooling layer with a step size of "2×2" and "3×3" is used for downsampling. The feature vectors extracted by the previous convolution and maximum pooling layers are combined and compressed through the fully connected layer, and the feature vectors are reduced to 128 dimensions for output.

[0027] Next, the sampled and grouped feature vectors enter the global-local importance feature fusion module. After the output results are input into the sampling layer, grouping layer, and PointNet layer, the output results are fused with the three branches of the SimAM attention mechanism and the global position encoder PE. After the above operation is performed once more, the final output is the fusion feature of these three branches. Among them, energy attention is a parameter-free energy attention mechanism, and its energy function is defined as follows:

[0028]

[0029] is t and x i Linear transformation of t and x i is the input feature X∈R C×H×W The target neuron and other neurons in the single channel of y are the labels of point cloud classification, y t is the attention output, y0 is the original label, i is the index in the spatial dimension, indicating the number of the point. M = H × W is the number of neurons in a channel, H and W are the features of the x and y axes of the point cloud respectively. t and b t are the weights and biases of the linear transformation. All values ​​in (1) are scalars. Minimum capacity It is obtained by the following formula:

[0030]

[0031] and λ is the correlation coefficient of the point cloud feature vector, which is set to 0.5. The importance of the neuron is expressed by Secondly, the global position encoder generates a unique code for the pixel position of each point cloud feature by cumulative summation and sine-cosine function. Here, it is combined with the attention module to form a global-local attention module. After the input grouped point cloud features are input, a high-dimensional point cloud feature vector with global-local attention features is obtained.

[0032] Step 7 is as follows:

[0033] Before training, the 3D point cloud image processed in step 5 is standardized and normalized. Standardization limits the number of point cloud images to "60,000", and normalization limits the pixels of the three channels to (0, 1). The Adam optimization algorithm is used as the loss function during training, the learning rate η=0.001, the sample batch_size is 2, each round of iteration contains 30,000 batches, and the number of iterations is epoch=250.

[0034] In the model training process, the compound loss function Compound Loss is used to improve the original NLL_loss by combining soft label and hard label losses and quantized feature losses. The formula is defined as follows:

[0035] Compound Loss=NLL_loss+quantization_loss+(hard_loss+soft_loss) / 2(3)

[0036] Negative log-likelihood loss:

[0037]

[0038] Among them, output and label are the output of the model and the original point cloud label, and N is the number of all points. is the weight of each category, is the probability corresponding to that class.

[0039] Soft label loss:

[0040]

[0041] i is the index in the spatial dimension, indicating the number of the point, label i and output i is the label and output of the i-th point, and N is the number of all points. The loss is applied to the continuous difference between the feature value and the label, mapping the feature value through the sigmoid function and calculating the absolute difference between the label.

[0042] Hard label loss:

[0043]

[0044] Among them, the weight w i The specific form is:

[0045]

[0046] It is applicable to classification problems between feature values ​​and labels. The loss is calculated through exponential and logarithmic terms, and positive and negative samples are weighted so that the loss contribution of positive and negative samples is the same.

[0047] Quantifying the loss:

[0048]

[0049] Among them, feture i is the eigenvalue of the model output. The main purpose of this quantization loss function is to make the modulus of the eigenvector close to 1. Specifically: if feture i The modulus of is close to 1, then (|feture i |-1) is close to 0, so ((|feture i |-1) 2 ) is also very small. i The modulus length is far from 1, then (|feture i |-1) is larger, so ((|feture i |-1) 2 ) is also larger, and the loss value increases.

[0050] The present invention has the following advantages:

[0051] This method uses deep point cloud vision method to detect bonding wire defects and segment seven different types of bonding wire defects, which greatly reduces the labor cost and improves the efficiency of bonding wire defect detection. A three-dimensional point cloud network PointNet++ is used to construct a three-dimensional point cloud bonding wire defect detection method based on global-local importance feature fusion. The average segmentation MIOU index of the defect detection class after training is 86.4% in 171 defect samples, the average instance segmentation MIOU is 87.2%, and the test accuracy is 95.1%. The classification model can accurately distinguish 7 types of bonding wire defects and normal bonding wire types, namely: abnormal arc, wire bending, long tail wire, double-layer interference, lead bridging, middle broken wire, single-end broken wire, and normal bonding wire. The MIOU during the test are 83.4%, 85.0%, 87.0%, 89.0%, 81.8%, 86.4%, 86.2%, and 88.8%, respectively. The model detection effect is better than the existing algorithm. In order to obtain the depth image information, a preprocessing mechanism for converting the depth map of bonding wire defects to point cloud is proposed, which can perform spatial plane segmentation on the point cloud, delete redundant information, obtain an accurate bonding wire plane, and more conveniently complete the detection of various types of bonding wire defects. Based on the segmentation of bonding wire point cloud data, the present invention proposes a three-dimensional point cloud instance target segmentation detection method based on global-local importance feature fusion for bonding wire defect detection and pattern recognition. The present invention can be applied in the field of defect detection in the process of wire welding manufacturing and processing, replacing manual detection and analysis of bonding wire defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Bond wire defect detection flow chart;

[0053] Figure 2 (a)(b) are depth images captured by the camera, (b) is the point cloud image of the PCB board, and (d) is the annotated point cloud image;

[0054] Figure 3 (a) is abnormal curvature, (b) is the tail wire is too long, (c) is double-layer interference, (d) is lead bridging, (e) is middle wire breakage, (f) is single-end wire breakage, (g) is wire bending, and (h) is a normal bonding wire.

[0055] Figure 4 Schematic diagram of the 3D point cloud bonding wire data preprocessing process;

[0056] Figure 5 Schematic diagram of the structure of the bonding wire defect segmentation detection model;

[0057] Figure 6 Schematic diagram of the global-local importance feature fusion module structure;

[0058] Figure 7 Schematic diagram of PointNet feature extraction network. DETAILED DESCRIPTION

[0059] The method is described in detail below with reference to the accompanying drawings and examples.

[0060] Figure 1 This is a flow chart of the bonding wire defect detection and pattern recognition method. First, the depth image of the PCB board is obtained. Then, the bonding wire depth image is converted into point cloud image data through the CloudCompare software. The bonding wire point cloud is plane segmented through the preprocessing mechanism to obtain a complete bonding wire plane and create a labeled point cloud bonding wire data set. Next, after the data set is divided, the three-dimensional point cloud bonding wire defect detection model based on the global-local importance feature fusion is trained. Finally, the segmented bonding wire plane is used to detect normal bonding wires, abnormal curvature, wire bending, long tail wires, double-layer interference, lead bridging, middle wire breakage, and single-end wire breakage defects, and the above eight types of bonding wire targets are instance segmented. The specific steps are as follows:

[0061] 1. Collect PCB board depth map

[0062] The image acquisition device uses Hikvision Robotics' 3D laser profile sensor, which has an X-axis resolution of 30 microns and a far field of view of 90 mm. The acquired depth images are transmitted to the computer in real time for processing.

[0063] 2. 3D point cloud bonding wire data conversion

[0064] The depth map of the PCB board is collected as follows Figure 2 As shown in (a) and (b), the point cloud data is first converted to obtain the complete PCB three-dimensional point cloud image data, as shown in Figure 2 (c) and perform image preprocessing on the converted point cloud image. The preprocessing process is as follows: Figure 4 shown.

[0065] First, an optical inspection platform is built using a single-axis motion platform and a linear structured light vision camera to capture depth images of bonding wires, and point cloud data of bonding wires is obtained through point array transformation. Secondly, a data grouping strategy is designed to generate 3D labels and assign weights to point cloud data. Data preprocessing is formulated to finely locate point cloud defects. Data fragmentation and denoising are used to remove irrelevant noise data and unnecessary aggregation effects. Multi-scale downsampling of point clouds is performed to retain key information and balance global and local features. An RGB assignment strategy based on the height slope Z value is used to expand the feature dimension of point cloud data. At this point, the construction of the three-dimensional point cloud data set of bonding wires and the positioning of defective point clouds are completed.

[0066] After constructing the pre-processed point cloud bonding wire data, use CloudCompare software to perform eight types of bonding wire annotations, such as Figure 2 Finally, eight types of bonding wire targets are as follows Figure 3 shown.

[0067] 3. 3D point cloud bonding wire defect detection based on global-local importance feature fusion

[0068] Figure 5 The overall framework of the global-local importance feature fusion network SI-PointNet is presented. First, the SetImportance module is designed, which consists of the original SA module, namely the sampling layer, the grouping layer and the PointNet layer, and the SimAM attention module and the global position encoding module are integrated to extract local and global importance features. Secondly, a composite loss function is designed to calculate the quantized loss of the extracted high-dimensional embedded features and the balanced metric loss of the soft similarity and hard similarity of the defect targets.

[0069] Specifically, a global-local importance feature fusion network SI-PointNet is designed to extract features and classify defects from the three-dimensional point cloud data of the bonding wire. First, the Set Importance module is constructed to abstract and downsample the input three-dimensional point cloud data layer by layer to extract multi-level features. Its structure mainly includes three parts: SA module, SimAM attention mechanism, and global position encoder PE. By fusing the outputs of these three branches, the multi-level defect features combining global features and local importance features are extracted. Secondly, the upsampling module propagates the features after downsampling, uses inverse distance weighted interpolation and Unit PointNet to restore the features of the downsampled points, so that the final output result segments the entire point cloud point by point. Finally, the loss function Compound Loss is designed to improve the classification performance of the network. By calculating the quantization loss of the extracted high-dimensional embedded features and merging the balanced metric loss calculation of soft and hard similarity between the target and the output, they are used to measure continuous differences, classification performance, and feature vector constraints, respectively. Through the above steps, SI-PointNet can extract features and accurately segment each part of the bonding wire when processing the complex point cloud data of the bonding wire, and identify the defective area.

[0070] 4. Global-local importance feature fusion module

[0071] like Figure 6As shown in the figure, this is an improved SI module, which strengthens the feature extraction capability of the original SA module by combining SimAM attention and PE global position encoding. Specifically, the input is first input into an SA module, which is a module composed of the sampling layer and the grouping layer in the original point++, and then it is output to a new SA module, and simultaneously input into SimAM and PE, and then the outputs of these three modules are feature fused. Finally, the fused features are respectively input into a SimAM, SA, and PE module, and the final output is the fused features of these three branches.

[0072] 5. PointNet feature extraction module

[0073] like Figure 7 As shown in the figure, this is the feature extraction module PointNet. The network first sends the input to the feature direction conversion module transform, and then sends the output to an MLP for feature extraction. After two repetitions, the feature dimension changes from n×3 to n×1024, and then the feature is max-pooled to obtain the global feature representation. In order to better integrate local information, the n×64 features of the previous process are superimposed with the cyclic 1024-dimensional global features, and finally the n×1088 feature vector is achieved. Then, the final output vector n×m is obtained by passing the MLP features twice. Next, the dimension is reduced through two fully connected layers. The fully connected layer consists of two 1x1 convolutional layers, where the first convolutional layer is used to reduce the feature vector, and the second convolutional layer is used to amplify the reduced feature vector. The fully connected layer uses Sigmoid as the activation function, and Dropout is added to prevent overfitting. After the second fully connected layer, a maximum pooling layer with a step size of "2×2" and "3×3" is used for downsampling. The feature vectors extracted by the previous convolution and maximum pooling layers are combined and compressed through the fully connected layer, and the feature vectors are reduced to 128 dimensions for output.

Claims

1. A three-dimensional point cloud bonding wire defect detection method based on global-local importance feature fusion, characterized in that: The method comprises the following steps: Step 1: Use a 3D depth camera to scan the welding area and obtain the original depth image of the bonding wire; convert each pixel point information in the depth image into three-dimensional point cloud data to obtain the spatial coordinates (x, y, z) of each pixel point; Step 2: perform data preprocessing on the acquired point cloud image; open the point cloud image using CloudCompare software, use the manual cropping function of the software to select irrelevant areas in the acquired data for cropping operations, obtain the bonding wire target area, and obtain a complete bonding wire plane; Step 3: Perform contour detection on the point cloud image, calculate the circumscribed rectangle of each contour, and obtain the aspect ratio of the bounding box; use the absolute median difference (MAD) algorithm to effectively distinguish the bonding wire contour from other contours, and focus on the extraction of bonding wire features; Step 4: extract the point cloud data within the outline of the bonding wire, and perform matrix operations to obtain the point cloud data of the bonding wire; use statistical filtering and principal component analysis PCA to process the point cloud data, remove irrelevant background point clouds and noise, and focus on the geometric features of the bonding wire; Step 5, project the bonding wire point cloud data constructed in step 4 along the XZ axis, calculate the interval size between adjacent points, and use the 3-sigma algorithm to screen the broken wires in the wires to obtain the bonding wire point cloud data set without broken wires, and mark each type of defect; Step 6: Next, the model is trained on the preprocessed 3D point cloud bonding wire data to achieve instance segmentation; a local important energy attention feature extraction module and a global information position encoding module are introduced to simultaneously extract the global features and local features of the bonding wire; Step 7: Design a compound loss to perform quantization loss calculation on the extracted high-dimensional embedding features, and perform balanced metric loss calculation on the defect targets for soft similarity and hard similarity. Step 8. Finally, the trained model is output to output various abnormal defects of the bonding wire point cloud data, including seven defect types: abnormal curvature, wire bending, excessive tail wire length, double-layer interference, wire bridging, middle wire breakage, and single-end wire breakage.

2. A three-dimensional point cloud bonding wire defect detection method based on global-local importance feature fusion as claimed in claim 1, characterized in that: Steps 1-5 are as follows: The professional point cloud processing software CloudCompare is used to convert the depth bonding wire defect image obtained by the depth camera into point cloud data. By obtaining the pixel information of the depth image, CloudCompare software displays its three-dimensional point cloud form. Then, the point cloud data editing function of CloudCompare software is used to select and delete the environmental noise existing in the bonding wire defect data converted into point cloud form, and through the active frame selection setting, other scene data except the bonding wire plane are segmented to retain the point cloud data within the bonding wire plane. Finally, through the annotation function of CloudCompare software, seven defect target bonding wires of abnormal curvature, wire bending, too long tail wire, double-layer interference, lead bridging, middle wire breakage, and single-end wire breakage are annotated.

3. The method for detecting bonding wire defects based on three-dimensional point cloud based on global-local importance feature fusion according to claim 1, characterized in that: Step 6 is as follows: The entire model is built based on the Pytorch framework and consists of three convolutional layers and four fully connected layers. The convolutional layer uses a convolution kernel size of "1×1", a step size of "1×1", and an activation function of ReLU. First, the point cloud bonding wire data is sampled and grouped, and the neighborhood ball is used to define the partition. Each area is determined by the center coordinates and radius. The selection of the center coordinates is implemented using the farthest point sampling algorithm. Specifically, the input points are first sampled through the sampling layer, and several center points are selected from these points. Then, the point set is divided into several areas using the center points obtained in the previous step through the grouping layer. Finally, each area obtained above is encoded in the PointNet layer and converted into a feature vector. The input of each set of extraction layers is (N, (d+C)), where N is the number of input points, d is the coordinate dimension, and C is the feature dimension; The output is (N′, (d+C′)), where N′ is the number of output points, d is the unchanged coordinate dimension, and C′ is the new feature dimension. The following is a detailed introduction to the role and implementation process of each layer; The sampling layer uses the farthest point sampling method to select N' points. First, the first point A is selected from the point cloud as the query point, and a point B with the farthest distance is selected from the remaining points; then the points A and B taken out are used as query points, and the point F with the farthest distance is taken from the remaining points; at this time, since the number of points that have been taken out exceeds 1, all query points (A, B) need to be considered at the same time; the method is as follows: for any point P among the remaining points, calculate the distance from the point P to all points (A, B) in the selected point set; take the minimum distance from points A and B as the distance d from the point to the selected point set; after calculating the distance from each remaining point to the point set, select the point with the largest distance, which is point F; 3. Repeat step 2 until N' points are sampled; The grouping layer uses the ball query method to divide the center point sampled by the sampling layer into several corresponding local areas, selects the adjacent points of the sampling center point, sends the selected sub-point cloud to the feature extraction layer, and encodes the information of the local area into a feature vector; The input of the PointNet layer is a local region: (N, K, (d + C)), where N is the number of input points, d is the coordinate dimension, and C is the feature dimension; The output is (N′, (d+C′)), where N′ is the number of output points, d is the unchanged coordinate dimension, and C′ is the new feature dimension. The PointNet layer extracts features from each group of point clouds to obtain local features. The network aligns the input points in space through the T-Net in the feature direction conversion module transform, rotates them to the front, and then sends the output to the MLP for feature extraction. After two repetitions, the feature dimension is mapped to a 1024-dimensional space. The maximum pooling layer retains only the maximum channel on all 1024-dimensional channels, thereby obtaining the global features of N point clouds. The features of the previous process are then combined with the cyclic 1024-dimensional global features for feature extraction. The feature vector is finally obtained by superposition, and then the final output vector is obtained by passing the MLP feature twice; then, the dimension is reduced through two fully connected layers. The fully connected layer consists of two 1x1 convolutional layers, where the first convolutional layer is used to reduce the feature vector, and the second convolutional layer is used to enlarge the reduced feature vector; the fully connected layer uses Sigmoid as the activation function, and Dropout is added to prevent overfitting; after the second fully connected layer, the maximum pooling layer with a step size of "2×2" and "3×3" is used for downsampling, and the feature vectors extracted by the previous convolution and maximum pooling layers are combined and compressed through the fully connected layer, and the feature vector is reduced to 128 dimensions for output; Next, the sampled and grouped feature vectors enter the global-local importance feature fusion module; After the output results are input into the sampling layer, grouping layer and PointNet layer, the output results are fused with the SimAM attention mechanism and the global position encoder PE. After the above operation is performed once more, the final output is the fusion feature of these three branches. Among them, energy attention is a parameter-free energy attention mechanism, and its energy function is defined as follows: is t and x i Linear transformation of t and x i is the input feature X∈R C×H×W The target neuron and other neurons in a single channel; y is the label of the point cloud classification, y t is the attention output, y0 is the original label, i is the index in the spatial dimension, indicating the number of the point; M = H × W is the number of neurons in a channel, H and W are the features of the x and y axes of the point cloud respectively; w t and b t are the weights and biases of the linear transformation; all values ​​in (1) are scalars; the minimum capacity It is obtained by the following formula: and λ is the correlation coefficient of the point cloud feature vector, which is set to 0.

5. The importance of the neuron is expressed by Obtained; secondly, the global position encoder generates a unique code for the pixel position of each point cloud feature through cumulative summation and sine-cosine functions; here it is combined with the attention module to form a global-local attention module, and after the input grouped point cloud features are input, a high-dimensional point cloud feature vector with global-local attention features is obtained.

4. The method for detecting bonding wire defects based on three-dimensional point cloud based on global-local importance feature fusion as claimed in claim 1, wherein step 7 is as follows: Before training, the 3D point cloud image processed in step 5 is standardized and normalized. Standardization limits the number of point cloud images to "60,000", and normalization limits the pixels of the three channels to (0, 1). During training, the Adam optimization algorithm is used as the loss function, the learning rate η = 0.001, the sample batch_size is 2, each round of iteration contains 30,000 batches, and the number of iterations is epoch = 250. In the model training process, the compound loss function Compound Loss is used to improve the original NLL_loss by combining soft label and hard label losses and quantized feature losses. The formula is defined as follows: Compound Loss=NLL_loss+quantization_loss+(hard-loss+soft-loss) / 2 (3) Negative log-likelihood loss: in, output and label are the output of the model and the original point cloud label, N is the number of all points; is the weight of each category, is the probability corresponding to that class; Soft label loss: i is the index in the spatial dimension, indicating the number of the point, label i and output i is the label and output of the i-th point, and N is the number of all points; this loss applies to the continuous difference between the feature value and the label, mapping the feature value through the sigmoid function and calculating the absolute difference between the label; Hard label loss: Among them, the weight w i The specific form is: Applicable to classification problems between feature values ​​and labels. The loss is calculated through exponential and logarithmic terms, and positive and negative samples are weighted so that the loss contribution of positive and negative samples is the same. Quantifying the loss: Among them, feture i is the eigenvalue of the model output; the main purpose of this quantization loss function is to make the modulus of the eigenvector close to 1; specifically: if feture i The modulus of is close to 1, then (|feture i |-1) is close to 0, so ((|feture i |-1) 2 ) is also small; if feture i The modulus length is far from 1, then (|feture i |-1) is larger, so ((|feture i |-1) 2 ) is also larger, and the loss value increases.

Images