A three-stage gold wire bonding defect detection method

Abstract

A three-stage gold wire bonding defect detection method, characterized by comprising three stages, as follows: (1) chip and key connection area segmentation: using DeepLabV3+ model to complete semantic segmentation of the chip and key connection area; (2) interested region extraction: according to the different bonding wire connection objects and the complexity of the background, the gold wire bonding picture is divided into four categories, based on the position information extracted in stage 1, further design corresponding image processing algorithm to complete the interested region extraction; (3) defect discrimination: based on the extracted interested region, using image processing algorithm and a new classification network based on twin network architecture is proposed to complete the defect discrimination. Therefore, the present application is a three-stage defect detection method which combines deep learning method with traditional digital image processing method, and completes the defect detection of missing wire and bent wire in the gold wire bonding process of microwave assembly.

Description

TECHNICAL FIELD

[0001] The present application relates to the defect detection of gold wire bonding process in the micro-assembly process of microwave assembly, and is a three-stage gold wire bonding defect detection method based on the fusion of deep learning and traditional image processing for complex practical engineering tasks. BACKGROUND

[0002] As the core component of communication and radar equipment, microwave assembly directly affects the performance index, working efficiency and manufacturing cost of the entire equipment. Referring to Figure 1 、 Figure 2 , gold wire bonding is a kind of using gold wire, using heat, pressure and ultrasonic energy to tightly weld the metal lead with two end assemblies, to realize the electrical interconnection between assemblies. The gold wire bonding picture components refer to Figure 1 . Gold wire bonding process is widely used in the manufacture of microwave assemblies. Chip and chip, chip and microstrip line, microstrip line and microstrip line, and connector and microstrip line are often connected by bonding gold wire. The quality of gold wire bonding directly determines the reliability, stability and electrical performance of microwave multi-chip assemblies.

[0003] The micro-assembly process of microwave assembly has the characteristics of precision, smallness and complex assembly. In the actual industrial production process, 1) there are many types of microwave assemblies, the background is complex, the bonding area has different shapes, and it is difficult to extract the region of interest and features in the picture. 2) The depth of field of the image acquisition device is limited, resulting in partial object blur, and in the actual gold wire bonding process, the gold wire is allowed to have a certain deformation, which causes the gold wire in the picture to have deformation and discontinuity, further increasing the detection difficulty. 3) It is difficult to collect high-quality defect image dataset. In actual detection, the image data is huge, and a large amount of manpower is needed to screen and label the photos, and because the defect position is uncertain, it is difficult to group the defects. At present, AOI is used for detection in industrial production, but the false detection rate is high, and manual re-inspection is required, which is time-consuming and laborious. It is of great significance to design an efficient and feasible machine vision method to automatically diagnose the soldering defects.

[0004] Machine vision inspection system acquires the surface image of the product through appropriate light source and image sensor, and extracts the feature information of the image by using corresponding image processing algorithm. Then, according to the feature information, the positioning and identification of defects are carried out. In recent years, machine vision technology is more and more applied to the defect detection of industrial products, such as cloth defect detection, PCB defect detection, steel surface defect detection, etc. The traditional surface defect detection method based on machine vision often uses conventional image processing algorithm or manually designed feature and classifier method, which analyzes the texture characteristics, geometric characteristics, color characteristics, etc., extracts image features, identifies defects, and distinguishes defect product images and normal product images. In recent years, with the development of deep learning technology, it has been more and more applied to industrial defect detection. Deep learning method can extract rich high-level semantic features from image for defect discrimination. The most widely used methods include using advanced classification networks such as VGG, ResNet, DenseNet, MobileNet, etc. for defect classification or feature extraction; using single-stage or two-stage detection networks such as SSD, YOLO, Faster R_CNN, etc. for defect classification and positioning; using semantic segmentation networks such as FCN, U-Net, etc. for pixel-level segmentation of defect position; other methods include metric learning, semi-supervised and unsupervised methods, etc.

[0005] However, based on the traditional digital image processing method, certain expert knowledge is needed, and the designed scheme lacks universality and cannot adapt to a large number of components existing in real applications. Based on the deep learning method, the size of the sample number of the picture data set, the quality of the picture, and the difficulty of the detection all have certain requirements, and cannot be applied to highly complex industrial scenes in an end-to-end manner. SUMMARY

[0006] For the high-frequency defects of missing wire and bent wire in gold wire bonding process, the present application combines deep learning method and traditional digital image processing method, and proposes a three-stage gold wire bonding defect detection method.

[0007] Technical scheme

[0008] A three-stage gold wire bonding defect detection method, characterized by comprising three stages, as follows:

[0009] (1) Chip and key connection area segmentation: DeepLabV3+ model is used to complete the semantic segmentation of chip and key connection area.

[0010] (2) Extraction of region of interest: According to the different bonding wire connection objects and the complexity of the background, the gold wire bonding picture is divided into 4 categories, based on the position information extracted in stage 1, further design corresponding image processing algorithm to complete the extraction of region of interest;

[0011] (3) Defect discrimination: Based on the extracted region of interest, an image processing algorithm is used and a new classification network based on a twin network architecture is proposed to complete defect discrimination.

[0012] The detection method is characterized in that stage 1: chip and key connection region segmentation

[0013] Using the semantic segmentation model DeepLabV3+, the chip region and key connection region in the picture are located, including:

[0014] Designing a loss function

[0015] Using a pixel-wise cross-entropy loss function as the objective function of model training,

[0016]

[0017] Where y is the true label of each pixel, p is the predicted output of each pixel, and n is the number of pixels in the picture;

[0018] Designing hyperparameters

[0019] The detection method is characterized in that the optimizer uses Adam, the learning rate is set to 5e-4, the batch size is 8, and the model parameters are pre-trained on the PASCAL VOC 2012 dataset.

[0020] The detection method is characterized in that stage 2: region of interest extraction

[0021] First, according to the different connection objects at both ends of the wire and the complexity of the background, the gold wire bonding picture is divided into 4 types, including type 1: chip connected to substrate; type 2: chip connected to chip and chip connected to substrate; type 3: chip connected to various components; type 4: substrate connected to various components;

[0022] Next, based on the position information of the segmentation region extracted in stage 1, different image processing methods are designed to extract the regions of interest such as solder joints and bonding wires in the picture.

[0023] The detection method is characterized in that type 1: chip connected to substrate

[0024] For type 1 picture, the following steps are used for the extraction of the substrate pad region: ① According to the chip region and the key connection region obtained in stage 1, after morphological erosion, the minimum circumscribed rectangle region of the eroded region is selected; ② The Canny operator is applied to the entire image to obtain the edge; ③ The edge region obtained in ② is subtracted from the minimum circumscribed rectangle region obtained in ① to remove the chip edge region, and then morphological dilation, hole filling, morphological erosion and connected component segmentation are applied to obtain the region, and finally the substrate pad region is filtered out according to the area size;

[0025] For the extraction of the chip internal pad region, the following steps are used: the minimum circumscribed rectangle region of the chip obtained in ① is subtracted from the region reduced according to the pad size to obtain a ring-shaped region with chip edges containing pads, and through threshold segmentation, erosion, dilation, connected component segmentation and adjacent region merging, the pad region is obtained; The substrate pad region and the chip internal pad region will be used for defect discrimination in stage 3.

[0026] The detection method, characterized by type 2: chip connected to chip and chip connected to substrate

[0027] For type 2 picture, the following steps are used for the extraction of the bonding wire region: ① The same as step ① of type 1, the minimum circumscribed rectangle region is obtained; ② Extract the lines in the entire image to obtain dark line regions and bright line regions, respectively, and then perform dilation on the bright line regions; ③ The dark line region is separated from the bright line region after dilation and the minimum circumscribed rectangle region by subtracting the bright line region after dilation from the dark line region, and then morphological dilation, connected component segmentation and region filtering are applied to obtain the bonding wire region;

[0028] The following steps are used for the extraction of the chip internal pad region: the same as type 1, the ring-shaped region with chip edges containing pads is obtained, and the threshold segmentation is directly used to obtain the chip internal pad region;

[0029] The bonding wire region, the chip internal pad region and the minimum circumscribed rectangle region of the key connection region obtained in ① will be used for defect discrimination in stage 3.

[0030] The detection method, characterized by type 3: chip connected to various components

[0031] For type 3 pictures, the following steps are used to directly extract the complete bonding area, including the complete bonding area including the bonding wire and the solder pad area at both ends thereof: ① The same as step ① of type 1, the minimum circumscribed rectangular area is obtained; ② Threshold segmentation is applied to the minimum circumscribed rectangular area of the chip obtained in step ①, the gray histogram thereof contains two peaks, the gray of the chip area is concentrated on the second peak, the second peak gray h is calculated, and the image is segmented by using the gray value [h-10, h+10] to obtain the minimum circumscribed rectangular area of the chip; ③ According to the minimum circumscribed rectangular area obtained in ②, the inner and outer two annular areas are obtained by contraction and subtraction operation, then the bonding wire position on the two annular areas is obtained through adaptive threshold segmentation and region filtering, and finally the same bonding wire area is matched according to the principle that the straight line distance between the center points of the inner and outer two areas is the nearest, the direction of the bonding wire is obtained, and the complete bonding area is further extended outward.

[0032] The complete bonding area in ③ and the minimum circumscribed rectangular area of the key connection area obtained in ① will be used for defect discrimination in stage 3.

[0033] The detection method is characterized in that stage 3: defect discrimination

[0034] For type 4 pictures, the following steps are used to directly extract the complete bonding area:

[0035] ① The same as steps ② and ③ of type 1, the substrate solder pad area is obtained through edge detection, morphological dilation, hole filling, morphological erosion, connected domain segmentation and region filtering;

[0036] ② Threshold segmentation is applied to the substrate solder pad area to obtain the bonding wire position and direction in the solder pad area, and the complete bonding area is further extended outward;

[0037] The complete bonding area will be used for defect discrimination in stage 3.

[0038] The detection method is characterized in that stage 3: defect discrimination

[0039] 1) For defect discrimination of the solder pad inside the chip, first, a threshold segmentation algorithm is used to segment the solder pad in the region, and then the solder pad area of the test picture is compared with the solder pad area of the template picture, if the area difference exceeds 50% of the template solder pad area, it is determined as a defect;

[0040] 2) For defect discrimination of the substrate solder pad, bonding wire and the like, a twin network model is designed.

[0041] The detection method, characterized in that the twin network model is a similarity measurement method, which receives two input samples, embeds them into a high-dimensional feature space through a pair of backbone structures (feature extraction network) with the same structure and shared weights, and then evaluates the similarity of the two samples through the difference of the high-dimensional feature vectors.

[0042] The detection method, characterized in that the designed defect discrimination model based on the twin network is composed of three parts: a feature extraction network, a feature fusion network, and a prediction network.

[0043] The detection method, characterized in that the feature extraction network:

[0044] The feature extraction network uses a modified version of ResNet18 for feature extraction, removes the conv5 layer of ResNet18, and takes the output of the conv4 layer as the output of the feature extraction network,

[0045] is the output feature of the template picture after the feature extraction network, is the output feature of the test picture after the feature extraction network, is the height and width of the input picture.

[0046] The detection method, characterized in that the feature fusion network: a feature fusion network with two branches is designed, which are difference fusion module and self-attention fusion module, respectively. The difference fusion module obtains the difference feature between the template picture and the test picture, and the self-attention fusion module obtains the global interaction feature of the template picture and the test picture.

[0047] The detection method, characterized in that in the difference fusion module, first, the output and of the feature extraction network are passed through the channel attention ECA-BLOCK, which gives different weights to the channel features and selectively enhances the significant features, so that the obtained feature map has more effective information, and the output features and are obtained. and are further extracted through 3x3 convolution while the channel number remains unchanged, and and are obtained. Next, and are flattened into one-dimensional feature vectors, and the absolute value of the difference is obtained. The subtraction operation captures the difference information between the test image and the template picture. Finally, Through two fully connected layers to obtain a prediction result The relationship between the input features and the output features in the difference fusion module is as follows,

[0048]

[0049]

[0050] wherein, Conv() represents a 3x3 convolution, ECA() represents an ECA-BLOCK, and FC() represents a fully connected layer.

[0051] The detection method is characterized in that, in the self-attention fusion module, the self-attention mechanism determines which parts of the input sequence are important at each step, which helps to capture global information from the input sequence. The self-attention mechanism is applied to enhance the feature fusion between the template picture and the test picture, so as to obtain high-level features after the global information interaction of the template image features and the test image features, and overcome the problem of narrow receptive field inherent in convolution operation.

[0052] The detection method is characterized in that, since the input of the self-attention mechanism is a feature vector sequence, firstly, the feature and are flattened in the spatial dimension and . The features and are spliced to obtain the input sequence of the self-attention SA-BLOCK , and the input sequence is passed through two self-attention modules to fully obtain the global dependency relationship between the features. The relationship between the input features and the output features in the self-attention fusion module is as follows,

[0053]

[0054]

[0055] wherein, SA() represents an SA-BLOCK, and FC() represents a fully connected layer. Positional Embedding and additional class embedding are added to the input of the SA-BLOCK, the Positional Embedding is used to express the position information of the features in the original picture, and the class embedding is used to calculate the output of the module.

[0056] The detection method is characterized in that, the prediction network is:

[0057] The prediction network fuses the outputs of the two branches of the feature fusion network to obtain the final prediction result,

[0058]

[0059] wherein, , are learnable parameters. p is the prediction result of the final output of the model, which is 0 / 1, indicating whether the test image and the template image are the same category.

[0060] Loss function: binary cross-entropy loss is used as the loss function in training,

[0061]

[0062] wherein, y is the real label of each picture, and p is the prediction output of each picture.

[0063] Hyperparameters: in the feature fusion network, the number of heads of multi-head attention (MHA) in SA-BLOCK is set to 8. The model is trained using Adam, the learning rate is set to 1e-4, and the batch size is 8. The parameters of the feature extraction network are initialized using the ResNet18 pre-trained on the ImageNet dataset.

[0064] Therefore, the three-stage defect detection method combining the deep learning method and the traditional digital image processing method is proposed, and the missing wire and the warped wire defect detection in the gold wire bonding process of the microwave assembly are completed. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 Schematic diagram of metal lead connection in gold wire bonding process

[0066] Figure 2 Components of gold wire bonding picture

[0067] Figure 3 Overall architecture diagram of three-stage gold wire bonding defect detection method

[0068] Figure 4 Detailed architecture diagram of three-stage gold wire bonding defect detection method

[0069] Figure 5 Output result of semantic segmentation model

[0070] Figure 6 Extraction process of the region of interest of the type 1 picture

[0071] Figure 7 Extraction result of the region of interest of the type 1 picture

[0072] Figure 8 Extraction process of the region of interest of the type 2 picture

[0073] Figure 9 Extraction result of the region of interest of the type 2 picture

[0074] Figure 10 Region of interest extraction process for type 3 and 4 pictures

[0075] Figure 11 Region of interest extraction result for type 3 picture

[0076] Figure 12 Region of interest extraction result for type 4 picture

[0077] Figure 13 Defect discrimination model based on twin network architecture

[0078] Figure 14 ResNet18 network model structure diagram

[0079] Figure 15 Self-attention module (SA-BLOCK)

[0080] Figure 16 Gold wire bonding defect detection result DETAILED DESCRIPTION

[0081] For the detection of missing wires and bent wire defects in the gold wire bonding process of microwave components, the present application proposes a three-stage gold wire bonding defect detection method that combines a deep learning method with a traditional image processing method. To address the challenge of large differences in bonding areas, the gold wire bonding picture is divided into four categories based on the characteristics of the bonding area, and semantic segmentation and image processing methods are used to extract the region of interest. To address the challenge of complex real data sets, a classification network based on a twin network architecture is proposed for defect discrimination, and a new feature fusion network is designed.

[0082] Due to the large number of gold wire bonding pictures and complex backgrounds, it is difficult to directly detect defects on the entire picture, so the present method (see Figure 3 ) first extracts the region of interest (such as the solder joint and bonding wire) affected by the gold wire bonding process, then discriminates the defects in the region of interest, and reduces the detection difficulty through a phased approach to complete the detection task. The three main stages are as follows:

[0083] (1) Chip and key connection area segmentation: the DeepLabV3+ model is used to complete the semantic segmentation of the chip and key connection area.

[0084] (2) Region of interest extraction: based on the different bonding wire connection objects and the complexity of the background, the gold wire bonding picture is divided into four categories, and based on the position information extracted in stage 1, a corresponding image processing algorithm (one of the innovations) is designed to complete the extraction of the region of interest.

[0085] (3) Defect discrimination: based on the extracted region of interest, an image processing algorithm and a newly proposed classification network based on a twin network architecture (Innovation point two: core innovation) are used to complete defect discrimination.

[0086] The detailed architecture of the proposed method is shown in Figure 4 Due to security needs, some areas in the picture have been blurred.

[0087] Stage 1: Chip and key connection area segmentation

[0088] Using the semantic segmentation model DeepLabV3+, the chip area and key connection area in the picture are located. In the process of using DeepLabV3+, the data set needs to be selected, the loss function needs to be defined, and the hyperparameters need to be determined. At the same time, according to the requirements of model parameter quantity and detection speed in actual application, the lightweight Moblienetv2 is used as the backbone network of DeepLabV3+.

[0089] 1) Data set

[0090] The data set includes a total of 458 pictures, of which 412 are training pictures and 46 are test pictures. The categories include two categories, namely chip area and key connection area, and the initial picture is uniformly adjusted to 512x512 pixels.

[0091] 2) Loss function

[0092] The pixel-wise cross-entropy loss function is used as the objective function of model training,

[0093]

[0094] Where y is the true label of each pixel, p is the predicted output of each pixel, and n is the number of pixels in the picture.

[0095] 3) Hyperparameters

[0096] The optimizer uses Adam, the learning rate is set to 5e-4, the batch size is 8, and the model parameters are pre-trained on the PASCAL VOC2012 data set.

[0097] 4) Experimental results

[0098] Referring to the experimental results shown in Table 1 and Table 2, DeepLabv3+ performs well in segmentation accuracy evaluation standards MPA (average percentage of correctly classified pixels per class) and MIoU (average of each class Intersection over Union (IoU)) and model parameter quantity and calculation time. The semantic segmentation results of the gold wire bonding picture are shown in Figure 5 .

[0099] Table 1 semantic segmentation model experimental results

[0100]

[0101] Table 2 IoU results of different categories

[0102]

[0103] Stage 2: Region of interest extraction

[0104] Firstly, according to the different connection objects of the two ends of the wire and the complexity of the background, the wire bonding pictures are divided into 4 categories, including Type 1: chip connected to substrate; Type 2: chip connected to chip and chip connected to substrate; Type 3: chip connected to various components; Type 4: substrate connected to various components. Next, based on the position information of the segmentation region extracted in stage 1, different image processing methods are designed to complete the extraction of the region of interest such as solder joints and bonding wires in the picture. In the actual application process, according to the layout information of the microwave component, one of the designed image processing methods is selected to automatically extract the region of interest.

[0105] 1) Type 1: chip connected to substrate

[0106] For Type 1 pictures, referring to Figure 6 , the following steps are used for the extraction of the substrate solder joint region:

[0107] ① According to the chip region and key connection region obtained in stage 1, after morphological erosion, the minimum circumscribed rectangle region of the eroded region is selected, see Figure 7 (b).

[0108] ② Apply Canny operator to the entire image to obtain the edge, see Figure 7 (c).

[0109] ③ Subtract the edge region obtained in ② from the minimum circumscribed rectangle region obtained in ① to remove the chip edge region, see Figure 7 (d), then apply morphological dilation, hole filling, morphological erosion and connected component segmentation to obtain the region as Figure 7 (e), finally filter out the solder joint region on the substrate according to the area size, see Figure 7 (f).

[0110] For the extraction of the chip internal solder joint region, the following steps are used: subtract the minimum circumscribed rectangle region of the chip obtained in ① from the region reduced according to the solder joint size (the solder joint size in the collected picture is basically the same, taking 50 pixels) to obtain a ring-shaped region with solder joints on the chip edge, see Figure 7(g), the solder region is obtained by adaptive thresholding, erosion, dilation, connected component segmentation and adjacent region merging, see Figure 7 (h).

[0111] The substrate solder region and the chip internal solder region will be used for defect discrimination in phase 3.

[0112] 2) Type 2: Chip to chip and chip to substrate

[0113] For Type 2 images, refer to Figure 8 , the following steps are used for extraction of the wirebond region:

[0114] ① Same as Type 1 step 1, the minimum bounding rectangle region is obtained, see Figure 9 (b).

[0115] ② Extract the line in the whole image, get the dark line region and the bright line region respectively, the dark line region is shown in Figure 9 (c), then do dilation processing to the bright line region, see Figure 9 (d).

[0116] ③ The dark line region containing wirebond is separated by subtracting the bright line dilation region from the dark line region and the minimum bounding rectangle region, see Figure 9 (e), then apply morphological dilation, connected component segmentation and region filtering to get the wirebond region, see Figure 9 (f).

[0117] The following steps are used for extraction of the chip internal solder region: same as Type 1, the chip edge containing solder ring region is obtained, see Figure 9 (g), since its solder is more obvious, the chip internal solder region can be obtained directly by adaptive thresholding, see Figure 9 (h).

[0118] The wirebond region, the chip internal solder region and the minimum bounding rectangle region of the key connection region obtained in ① will be used for defect discrimination in phase 3.

[0119] 3) Type 3: Chip to various components

[0120] For Type 3 images, refer to Figure 10 , the following steps are used for direct extraction of the complete bonding region (the complete bonding region includes the wirebond and its two end solder regions):

[0121] ① Same as Type 1 step 1, the minimum bounding rectangle region is obtained, see Figure 11 (b).

[0122] ②Threshold segmentation is applied to the minimum bounding rectangle region of the chip obtained in step 1. Since the gray histogram of the chip region mainly contains two peaks, the gray of the chip region is concentrated on the second peak, the second peak gray h is calculated, and the image is segmented with the gray value [h-10, h+10] to obtain a more accurate minimum bounding rectangle region of the chip.

[0123] ③According to the minimum bounding rectangle region obtained in ②, two annular regions (the width of the two annular regions is 3 pixels, and the interval is 4 pixels) are obtained by shrinking and subtracting operations, as shown in Figure 11 (c), and then the position of the bonding wire on the two annular regions is obtained through adaptive threshold segmentation and region filtering, as shown in Figure 11 (d), and finally, the regions on the same bonding wire are matched according to the principle of the straight-line distance between the center points of the two regions being the closest, the direction of the bonding wire is obtained, and the complete bonding region is further extended, as shown in Figure 11 (e).

[0124] The complete bonding region described in ③ and the minimum bounding rectangle region of the key connection region obtained in 1 will be used for defect discrimination in stage 3, as shown in Figure 11 (f).

[0125] 4) Type 4: Substrate connected to various components

[0126] For type 4 pictures, refer to Figure 10 , the following steps are used to directly extract the complete bonding region:

[0127] ①The same as steps ② and ③ of type 1, the substrate pad region is obtained through edge detection, morphological dilation, hole filling, morphological erosion, connected domain segmentation, and region filtering, as shown in Figure 12 (b), Figure 12 (c).

[0128] ②Threshold segmentation is applied to the substrate pad region to obtain the position and direction of the bonding wire within the pad region, as shown in Figure 12 (d), and the complete bonding region is further extended, as shown in Figure 12 (e).

[0129] The complete bonding region will be used for defect discrimination in stage 3.

[0130] Stage 3: Defect discrimination

[0131] Based on the regions of interest extracted in stage 2, since the chip internal pad region picture only contains light background and black pads, its defect discrimination is relatively simple, and an image processing method can be directly designed to obtain the detection result. In addition to the chip internal pad, other regions of interest such as substrate pads, bonding wires, etc. are diverse in type and shape, refer to Figure 13, input to the proposed classification network based on the twin network architecture, to obtain the defect discrimination result.

[0132] 1) For the defect discrimination of the chip internal solder joints, first, an adaptive threshold segmentation algorithm is used to segment the solder joints in the region, and then the solder joint area of the test picture is compared with the solder joint area of the template picture. If the area difference exceeds 50% of the template solder joint area, it is determined to be a defect.

[0133] 2) For the defect discrimination of the substrate solder joints, bonding wires, etc., a new defect discrimination model based on the twin network framework is designed. The twin network model is a similarity measurement method, which receives two input samples (one is a sample with known classification label, and the other is a sample with unknown classification label), embeds them into a high-dimensional feature space through a pair of backbone structures (feature extraction network) with the same structure and shared weights, and then compares the difference between the high-dimensional feature vectors to evaluate the similarity of the two samples. In actual detection scenarios, normal samples and defect samples in the same region of a microwave component can be obtained. In order to improve the detection accuracy, the pictures are input in pairs, and the normal sample is regarded as a template picture to compare the similarity with the test picture to determine whether there is a defect. The defect discrimination model based on the twin network is composed of three parts: feature extraction network, feature fusion network and prediction network (see Figure 13 ).

[0134] ① Feature extraction network

[0135] The feature extraction network uses a modified version of ResNet18 for feature extraction, referring to Figure 13 , Figure 14 , removes the conv5 layer of ResNet18, and takes the output of the conv4 layer as the output of the feature extraction network (

[0136] is the output feature of the template picture after the feature extraction network, is the output feature of the test picture after the feature extraction network, is the height and width of the input picture).

[0137] ① Feature fusion network

[0138] A feature fusion network with two branches is designed, which are difference fusion module and self-attention fusion module respectively. The difference fusion module obtains the difference features between the template picture and the test picture, and the self-attention fusion module obtains the global interaction features of the template picture and the test picture.

[0139] In the difference fusion module, referring to Figure 13 , first, the outputs of the feature extraction network and , the channel attention is given to each channel feature to selectively enhance the salient features, so that the obtained feature map has more effective information, and the output feature is obtained and . and Further extract features through 3x3 convolution while keeping the number of channels unchanged, obtain and . Next, and are flattened into one-dimensional feature vectors, and the absolute value is obtained after subtraction . The subtraction operation can capture the difference information between the test image and the template picture. Finally, obtain the prediction result The relationship between the input features and the output features in the difference fusion module is as follows,

[0140]

[0141]

[0142] Among them, Conv() represents 3x3 convolution, ECA() represents ECA-BLOCK, and FC() represents full connection layer.

[0143] In the self-attention fusion module, the self-attention mechanism determines which parts of the input sequence are important at each step, which helps to capture global information from the input sequence. Applying the self-attention mechanism can enhance the feature fusion between the template picture and the test picture to obtain high-level features after the global information interaction of the template image features and the test image features, and overcome the problem of narrow receptive field inherent in convolution operation. Since the input of the self-attention mechanism is a sequence of feature vectors, referring to Figure 13 、 Figure 15 , first, flatten and in the spatial dimension to obtain features and . Concatenate features and to obtain the input sequence of the self-attention SA-BLOCK, go through 2 self-attention modules to fully obtain the global dependency relationship between features. The relationship between the input features and the output features in the self-attention fusion module is as follows,

[0144]

[0145]

[0146] where SA() represents SA-BLOCK, FC() represents the fully connected layer. Positional Embedding and additional learnable class embedding are added to the input of SA-BLOCK, the former is used to express the position information of features in the original image, and the latter is used to calculate the final output of the module .

[0147] ②Prediction network

[0148] The prediction network fuses the outputs of the two branches of the feature fusion network to obtain the final prediction result,

[0149]

[0150] where, , are learnable parameters. p is the final output of the model, which is 0 / 1, indicating whether the test image and the template image are of the same class.

[0151] Dataset: The dataset includes a total of 339 pairs of pictures (each pair of pictures contains one test picture and one corresponding template picture), of which 259 pairs are used for training and 80 pairs are used for testing. The prediction result is binary classification, i.e. normal and defect. The initial picture is uniformly adjusted to 96x96 pixels.

[0152] Loss function: Binary cross-entropy loss is used as the loss function in training,

[0153]

[0154] where y is the true label of each picture, and p is the prediction output of each picture.

[0155] Hyperparameters: In the feature fusion network, the number of heads of MHA in SA-BLOCK is set to 8. Adam is used to train the model, the learning rate is set to 1e-4, and the batch size is 8. The parameters of the feature extraction network are initialized using the ResNet18 pre-trained on the ImageNet dataset.

[0156] Experimental results: The proposed model is compared with the widely used classification network (VGG16, Res50, Moblienetv2) and the basic twin network in performance. Based on the experimental results in Table 3, in the gold wire bonding defect discrimination task, the model proposed in the application obtains the highest accuracy compared with directly using the classification model and the basic twin network, verifying the effectiveness of the designed twin network structure and feature fusion network. The twin network structure is suitable for the gold wire bonding defect detection scene in the application, and the template picture and the test picture are trained in pairs, which can reduce the dependence on the data set, and effectively capture the defect features in the image. In the two-branch feature fusion network, the difference fusion module can capture the difference information between the pictures, and the self-attention fusion module can make the pictures obtain global interaction information, improving the ability of the model to distinguish defect pictures.

[0157] Table 3 Experimental results of defect discrimination model

[0158]

[0159] The gold wire bonding defect detection results are as shown in Figure 16 The method provided by the application identifies the missing wire and the bent wire defects in the four types of gold wire bonding pictures.

Claims

1. A three-stage gold wire bonding defect detection method, characterized in that, The three stages include the following: (1) Chip and key connection area segmentation: using a DeepLabV3+ model to complete the semantic segmentation of the chip and key connection area; (2) Region of interest extraction: according to the different connection objects of the bonding wires and the complexity of the background, the gold wire bonding picture is divided into 4 categories, based on the position information extracted in stage 1, further design the corresponding image processing algorithm to complete the extraction of the region of interest; (3) Defect discrimination: based on the extracted region of interest, using image processing algorithm and a new classification network based on twin network architecture is proposed to complete the defect discrimination; The defect discrimination model based on twin network designed by the application is composed of three parts: feature extraction network, feature fusion network and prediction network. Feature extraction network: The feature extraction network uses a modified version of ResNet18 for feature extraction, removing the conv5 layer of ResNet18 and using the output of the conv4 layer as the output of the feature extraction network, is the output feature of the template picture passing through the feature extraction network, is the output feature of the test picture passing through the feature extraction network, is the height and width of the input picture; Feature fusion network: a feature fusion network with two branches is designed, which is difference fusion module and self-attention fusion module, difference fusion module obtains the difference features between template picture and test picture, and self-attention fusion module obtains the global interaction features of template picture and test picture; In the difference fusion module, first, the output of the feature extraction network and passes through the channel attention ECA-BLOCK, which gives different weights to each channel feature, selectively enhances the salient features, and makes the obtained feature map have more effective information, obtaining the output feature and ; and Further extract features through 3x3 convolution while keeping the channel number unchanged, obtain and ; Next, flatten and into one-dimensional feature vectors, and take the absolute value after subtraction to obtain ; The subtraction operation captures the difference information between the test image and the template picture; Finally, obtain the prediction result through two fully connected layers ; The relationship between the input and output features in the difference fusion module is as follows, Among them, Conv() represents 3x3 convolution, ECA() represents ECA-BLOCK, and FC() represents full connection layer. Since the input of the self-attention mechanism is a sequence of feature vectors, first, flatten in the spatial dimension and , get the feature and ; concatenate the features and to get the input sequence of the self-attention SA-BLOCK , pass through 2 self-attention modules to fully obtain the global dependency between features; the relationship between the input features and the output features in the self-attention fusion module is as follows, Wherein, SA() represents SA-BLOCK, FC() represents full connection layer; Positional Embedding and additional learnable class embedding are added to the input of SA-BLOCK, the Positional Embedding is used to express the position information of the feature in the original graph, and the class embedding is used to calculate the output of the module .

2. The detection method of claim 1, wherein, Stage 2: region of interest extraction Firstly, according to the different connection objects of the bonding wires and the complexity of the background, the gold wire bonding picture is divided into 4 categories, including, Type 1: chip connected to substrate; Type 2: chip connected to chip and chip connected to substrate; Type 3: chip connected to various components; Type 4: substrate connected to various components; Next, based on the position information of the segmented region extracted in stage 1, different image processing methods are designed to complete the extraction of the regions of interest such as solder joints and bonding wires in the picture.

3. The detection method of claim 1, wherein, Stage 3: defect discrimination 1) For the defect discrimination of chip internal solder joints, firstly, threshold segmentation algorithm is used to segment the solder joints in the region, and then the solder joint area of the test picture is compared with the solder joint area of the template picture, if the area difference exceeds 50% of the template solder joint area, it is judged as a defect; 2) For the defect discrimination of substrate solder joints and bonding wires, a twin network model is designed.

4. The method of claim 1, wherein the step of detecting is characterized by, In the self-attention fusion module, the self-attention mechanism determines which parts of the input sequence are important at each step, which helps to capture global information from the input sequence; The self-attention mechanism is applied to enhance the feature fusion between the template picture and the test picture, so as to obtain the high-level features after the global information interaction of the template image features and the test image features, and overcome the problem of narrow receptive field inherent in convolution operation.

5. The detection method of claim 1, The prediction network fuses the outputs of the two branches of the feature fusion network to obtain the final prediction result, Loss function: binary cross entropy loss is used as the loss function in training, wherein, , are learnable parameters; p is the prediction result of the final output of the model, 0 / 1, indicating whether the test image and the template image are the same category; Where y is the true label of each picture. ​ Hyperparameters: In the feature fusion network, the number of heads of multi-head attention (MHA) in SA-BLOCK is set to 8; the model is trained using Adam, the learning rate is set to 1e-4, and the batch size is 8; the parameters of the feature extraction network are initialized using the pre-trained ResNet18 of the ImageNet dataset.

Images