Bonding wire defect detection method based on deep learning

Abstract

The invention provides a bonding wire defect detection method based on deep learning, which can be used for identifying the defect type of a bonding wire and automatically detecting the bonding quality problem of the wire in an integrated circuit product, and aims to solve the problems of high labor intensity, low detection efficiency and the like. The process comprises the following steps: establishing a bonding wire acquisition physical platform, acquiring a depth image of a bonding wire through a stereo camera, preprocessing the acquired depth image to enhance the image quality, generating a normal diagram corresponding to the depth image through geometric transformation, and stacking the normal diagram and the depth diagram to obtain a final bonding wire data set. Performing data enhancement on the bonding wire data set to expand the data scale; a Depth-YOLO network model is provided, an input layer of an original YOLO model is reconstructed, and an input feature enhancement module is designed; and finally, training a Depth-YOLO model by using the self-built data set, and carrying out end-to-end bonding wire defect detection by using the model.

Description

Technical Field

[0001] The present invention belongs to the field of defect detection in the wire bonding process of integrated circuit packaging, and specifically relates to a method for detecting bonding wire defects and determining defect mode types using a deep learning method. Background Art

[0002] As a strategic, basic and leading industry for the development of modern economy and society, the semiconductor industry is a key force in leading a new round of scientific and technological revolution and industrial transformation. It is considered an important indicator for measuring national information security, scientific and technological status and comprehensive national strength, and is crucial to the country's modernization process. As the international competition in the semiconductor industry becomes increasingly complex, integrated circuits, as an important pillar for ensuring my country's economic security, national defense security and social stability, are key technical bottlenecks restricting the development of my country's integrated circuit industry, such as design, manufacturing and quality inspection. However, my country's semiconductor materials and devices have long relied on imports, and the level of localization of key core technologies in a large number of production links is insufficient. Therefore, it is urgent to accelerate the innovation of domestic semiconductor technology and enhance my country's independent guarantee capabilities for key raw and auxiliary materials for semiconductors. Based on this situation, the country has issued a series of policies to support the development of the domestic semiconductor industry and accelerate the process of localization of semiconductor technology.

[0003] Integrated circuit testing technology runs through all aspects of integrated circuit research and development, manufacturing and application, and plays a key role in product quality control and equipment reliability assurance. Since the promulgation of national policies, many technical capabilities of my country's semiconductor industry, such as design, production, packaging, and quality inspection, have been improved, but there are still some urgent problems to be solved in the quality inspection of bonding wires, one of the key production links of semiconductors. Wire bonding is a key step in the integrated circuit packaging process. Its function is to connect different components and chips to each other to ensure the normal operation of the circuit. Bonding wire defects are the main factor affecting product quality and product yield. By detecting defects in the wire bonding process, not only can the defect type of the wire be accurately identified, but also the faults in the process flow can be identified according to the defect type, which is convenient for engineers to improve the process. At present, there are two main methods for defect detection of bonding wires: detecting bonding wire defects through manual visual inspection and detecting bonding wire defects through image processing methods. Manual visual inspection is inefficient, labor-intensive, and costly. It is also easily affected by subjective factors, and the detection results are unstable. The image processing detection method can detect a single defect mode, and the processing process is complicated, and the detection efficiency does not meet the industrial assembly line standards. Therefore, a method that can automatically detect bonding wire defects is urgently needed.

[0004] The image processing methods currently used in bond wire defect detection are mainly based on two-dimensional images collected by two-dimensional cameras. Although the detection efficiency is improved, due to the lack of three-dimensional spatial geometric information, the detection ability of three-dimensional structural defects such as curvature anomalies and line bending is limited, and it is difficult to fully cover all defect types. In addition, two-dimensional images are easily affected by ambient lighting and reflections, resulting in a decrease in image quality. In this case, it is difficult for traditional image processing methods to accurately segment and detect welding wires, increasing the possibility of false detection and missed detection. Summary of the invention

[0005] The main purpose of the present invention is to propose a bonding wire defect detection method based on deep learning. The method can be applied to the production and manufacturing process of integrated circuit wire bonding, replacing the existing manual visual inspection method, completing the identification of wire defect types, and realizing automatic detection of wire bonding quality problems in integrated circuit products.

[0006] The present invention aims to solve the following problems:

[0007] 1. Bonding wire defect detection is mainly done through manual visual inspection, which is inefficient, slow, and labor-intensive;

[0008] 2. Existing lead defect detection technology can only detect a small number of defects such as lead breakage, but cannot detect other types of defects such as wire bending and curvature abnormalities;

[0009] 3. Currently, there is no public lead defect dataset that can be used for lead defect detection methods based on deep learning, and the two-dimensional RGB image dataset cannot provide the depth information required for lead defect detection.

[0010] In order to solve the above problems, the present invention proposes a bonding wire defect detection method based on deep learning. The method first collects and produces a bonding wire depth image dataset, and performs data preprocessing to enhance image quality and expand data volume. The bonding wire defect types involved in this method include: double-line interference Interference, lead bridge Bridge, middle broken wire Middle broken wire, single-ended broken wire Single broken wire, Radian anomaly, wire bend Curve, and tail wire too long Taillong. The deep learning network model selected by this method is the Depth-YOLO model improved based on YOLOv8. The bonding wire defect detection network model Depth-YOLO is trained by the constructed data set, thereby realizing end-to-end wire defect detection.

[0011] The specific working process of the present invention is as follows:

[0012] Step 1: Use a 3D stereo vision camera to photograph the bonding wires and obtain a total of 2,000 original depth images;

[0013] Step 2, using normalization, superposition denoising strategy and adaptive histogram equalization method to pre-process the original depth image to improve the quality of the bond wire depth image;

[0014] Step 3: Use Labelimg to annotate the processed depth image in txt format to build a bonding wire depth image dataset;

[0015] Step 4: Use the Sobel difference operator to calculate the gradient information of the depth image, calculate the normal vector of each pixel based on the gradient, map the three components of the normal vector in the three-dimensional space to the three RGB channels respectively, use RGB color coding to represent the direction of the normal line in the three-dimensional space, and generate a normal map;

[0016] Step 5, stacking the generated RGB three-channel normal map and the single-channel depth map, expanding the image dimension and merging them to generate a four-channel pseudo RGBD image containing three-dimensional information;

[0017] Step 6: Perform data enhancement on the stacked images generated in step 5, and use the albumentations library to enhance the depth image, normal map, and label file at the same time. A total of 20,000 images are enhanced to obtain the final bonding wire dataset;

[0018] Step 7, propose a bonding wire defect detection model Depth-YOLO. Reconstruct the input layer of the two-dimensional target detection network model YOLOv8, modify DataLoader so that it can correctly load and process 4-channel images, expand the number of data input channels to 4, and modify the initial convolution of the model input layer. Modify the model pre-trained weight loading, only load the weights of the model except the first convolution layer, and randomly initialize the weights of the first layer, and use the Depth-YOLO model to train the bonding wire dataset in step 6;

[0019] Step 8: Design an input feature enhancement module to embed the model in step 7 to enhance the extraction of depth information in the depth map and geometric information in the normal map.

[0020] In step 2, the original bond wire depth image is a single-channel depth image with a size of 3600x2682, a data type of 16-bit unsigned integer, and a format of tiff. The preprocessing process for the original depth image is as follows: first normalize the depth image, denoise the normalized depth image, and adopt a stacking denoising strategy. First, use Gaussian filtering to remove high-frequency noise, then use median filtering to remove local scattered noise, and then use bilateral filtering to continue denoising while retaining image edge information. Adaptive histogram equalization is performed on the denoised image to obtain the final processed image.

[0021] In step 4, the preprocessed depth image is geometrically transformed to generate a normal map. The normal vector N(x, y) is expressed as shown in formula (1). Z(x, y) represents the depth value of each pixel in the depth map. The Sobel filter is used to calculate the gradient of each pixel in the depth map relative to its adjacent pixels to obtain the horizontal gradient Gx and vertical gradient Gy, as shown in formula (2). Normalization is performed to obtain the unit normal vector N ′ (x, y), as shown in formula (3). Each component of the calculated normal vector is in the range of [-1, 1], which is mapped to the range of [0, 255], and the normal map is visualized through RGB color coding, as shown in formula (4).

[0022] N(x,y)=(-G x ,-G y ,1) (1)

[0023]

[0024] The generated normal map is stacked with the original depth map. The depth map is a single-channel image with a shape of (H, W). The normal map is a three-channel RGB image with a shape of (H, W, 3). The depth map is expanded and stacked with the normal map to obtain a four-channel pseudo RGBD image with a shape of (H, W, 4). The stacked pseudo RGBD image dataset is used to train the deep learning network model YOLOv8 for defect detection of bonding wires.

[0025] In step 7, the bonding wire defect detection model Depth-YOLO is built based on the YOLOv8 model. The input part of YOLOv8 is reconstructed so that the model can receive the stacked image in step 6. The model is built based on the Tensorflow framework. The specific model structure is shown in the following table:

[0026]

[0027] The convolution layer uses a 3×3 kernel size, a 1×1 stride, a ReLU activation function, and 4 input channels. A 2×2 maximum pooling layer is used after each convolution layer for downsampling. The Bottleneck layer uses a 1×1 convolution to reduce the number of channels in the feature map. The SPP layer uses multiple convolution kernels of different sizes (5×5, 9×9, 13×13) for pooling. The PAN layer fuses features at different levels through a connection operation. The Head layer uses Sigmoid as the activation function and outputs a probability value.

[0028] Before training, the images processed by data augmentation in step 6 are standardized, and the image size is limited to 640×640. The normalization limits the pixels of the four channels to (0, 1). Based on the bond wire dataset built in step 6, there are 4000 depth images and normal images of bond wires without defects, and 6000 depth images and normal images of bond wires with defects. 60% are selected as the training set, and the number of images of various defect types is Interference: 517, Bridge: 515, Middlebroken: 521, Singlebroken: 523, Radiananomaly: 514, Curve: 496, Taillong: 514. During training, the learning rate η=0.0001, the sample batch_size is 128, each round of iteration contains 583 batches, and the number of iterations is epoch=125.

[0029] In step 8, the designed input feature enhancement module includes four parts: depth enhancement, texture enhancement, feature fusion and geometric feature extraction. The specific module structure is shown in the following table:

[0030]

[0031] The first three convolutional layers use convolution kernels of size 3×3, 5×5, and 7×7 with a stride of 1×1 to extract local depth features at multiple scales from a single-channel depth map. The number of output feature maps is 16, and the activation function is ReLU. Batch normalization and residual connection are used to stabilize training and retain the original features. Conv4 uses a convolution kernel of size 3×3 and a stride of 1×1 to extract gradient information from a three-channel normal map. The Fusion layer uses a convolution kernel of size 1×1 and a stride of 1×1 to fuse gradient features on different channels and integrate different directional information in the normal map. The ChannelAttention part uses a 1×1 global average pooling and a 1×1 convolution to reduce the number of channels, while enhancing nonlinear expression. The importance weight of each channel is generated through the Sigmoid activation function. The SpatialAttention part uses a 7×7 convolution to generate a single-channel feature map to represent the weight of each position. The Sigmoid activation function is used to generate the attention weight of each pixel position to enhance the model's ability to extract geometric features. After fusing the depth features and normal features, the final convolutional layer uses a convolution kernel of size 3×3 and stride 1×1 to further extract important geometric features, and uses residual connections to ensure that geometric information is not lost during the extraction process.

[0032] The present invention has the following advantages:

[0033] The present invention adopts an end-to-end deep learning method and uses a self-built bonding wire depth image dataset to detect bonding wire structural defects, which greatly reduces the labor cost and improves the bonding wire detection efficiency. Based on the YOLOv8 model, the present invention proposes a model Depth-YOLO for bonding wire defect detection, which can focus on the three-dimensional depth information of the image. The introduced enhancement module also makes the model pay more attention to the depth features and geometric features of the image, ensuring the precision and accuracy of the detection. At the same time, the constructed bonding wire dataset also fills the gap in related research fields. The defect detection rate of the trained model in 2000 defect samples is 96.8%. The detection model can accurately distinguish 8 types of defects, namely: Interference, Bridge, Middlebroken, Singlebroken, Radiananomaly, Curve, Taillong, Normal. The accuracy rates during testing are: 99.5%, 99.5%, 99.5%, 99.5%, 97.7%, 93.1%, 85.9%, 99.5%, respectively. The model detection effect is better than the existing algorithm. The present invention can be applied to the field of defect detection in semiconductor wire bonding process, replacing manual end-to-end automatic real-time detection of bonding wires. At the same time, it is also suitable for defect detection of various other integrated circuit products such as PCB boards, wafers, solder joints, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Bond wire defect detection flow chart;

[0035] Figure 2 Construction of a physical platform for bonding wire depth image acquisition;

[0036] Figure 3 (a) is an example of a bonding wire, and (b) is a depth image of the wire captured by a stereo camera.

[0037] Figure 4 (a) is the depth image after image quality enhancement, (b) is the normal map after geometric transformation;

[0038] Figure 5 Schematic diagram of the Depth-YOLO structure of the bonding wire defect detection model;

[0039] Figure 6 (a) is the mAP curve of the model detection result, and (b) is a visualization diagram of the detection result. DETAILED DESCRIPTION

[0040] The method is described in detail below with reference to the accompanying drawings and examples.

[0041] Figure 1This is a flow chart of the bonding wire defect detection method. First, a physical platform for bonding wire acquisition is built, and a depth image of the bonding wire is obtained through a stereo camera. The acquired depth map is preprocessed to enhance the image quality, and a normal map corresponding to the depth map is generated through geometric transformation. The normal map and the depth map are stacked to obtain the final bonding wire dataset. Data enhancement is performed on the bonding wire dataset to expand the data scale, and the enhanced dataset is input into the Depth-YOLO network model for training, and an input feature enhancement module is designed. The specific steps are as follows:

[0042] 1. Collect bonding wire depth images

[0043] The image acquisition device uses Hikvision's line structured light camera (MV-DP2307-01H) to collect depth images of bonding wires. Its X-axis resolution is 30 microns, and the far field of view is 96 mm. The collected depth images can be transmitted to the computer in real time for processing. Build a physical platform for wire bonding defect detection, using a stereo camera, motion slide platform, switching power supply, motion controller, network cable, and aviation head connection cable to build a stereo vision inspection platform such as Figure 2 The bonding wires used are as shown in Figure 3 As shown in (a), the collected depth image is Figure 3 (b) as shown.

[0044] 2. Deep Image Processing

[0045] The acquired depth map is processed. First, the depth image is normalized and the depth value is normalized to [0,1]. The normalized depth image is denoised by adopting the strategy of superposition denoising. First, Gaussian filtering is used to remove high-frequency noise, and then median filtering is used to remove local scattered noise. Then, bilateral filtering is used to denoise while retaining the image edge information to obtain the final denoised image. The denoised image is subjected to adaptive histogram equalization to obtain the final processed image.

[0046] The original depth map D is Gaussian filtered to obtain the depth map G after Gaussian smoothing, as shown in formula (5). G(x,y) represents the value of the pixel (x,y) after Gaussian filtering, D(x,y) represents the depth map, k is the radius of the filter window, D(x+i,y+j) is the value of the adjacent pixel in the window, and w(i,j) is the Gaussian weight.

[0047]

[0048] Perform median filtering on the depth map G after Gaussian filtering to obtain the depth map M after median filtering, as shown in formula (6). Wherein, median{·} represents the median of the pixel values ​​in the window.

[0049] M(x,y)=median{G(x+i,y+j)∣-k2≤i,j≤k2}(6)

[0050] The median filtered depth map M is subjected to bilateral filtering to obtain the final denoised depth map B, as shown in formula (7). Where Wp is the normalization coefficient, f s (i,j) is the spatial Gaussian weight, f r (M(x+i,y+j),M(x,y)) is the Gaussian weight of pixel value similarity.

[0051]

[0052] Adaptive histogram equalization is performed on the denoised image. First, the input image is divided into multiple small sub-regions, and then each sub-region is subjected to histogram equalization separately. Finally, the equalization results of these sub-regions are combined to generate the final enhanced image. The histogram of each local region R is shown in formula (8). R (i) represents the frequency of gray level i in region R. R (i) represents the number of pixels with gray level i. The gray level of each pixel is mapped to a new gray level using the cumulative distribution function (CDF), as shown in formula (9). R (i) represents the cumulative frequency of gray level i and all previous gray levels. After equalizing each local area, the pixels in the image are interpolated and synthesized to generate the enhanced full image. Assuming that a pixel is located at the junction of multiple local areas, its new pixel value T(x,y) can be calculated by formula (10). Represents the equalization result in the local area. i is the weight of bilinear interpolation, which indicates the relative distance from the pixel to the center of the region, and the sum of the weights is 1. Finally, a contrast limit parameter is introduced to limit the maximum frequency of each histogram to prevent excessive contrast enhancement.

[0053] The processed image is Figure 4 (a) shown.

[0054] H R (i) = N R (i) (8)

[0055]

[0056] 3. Deep Image Conversion to Generate Normal Map

[0057] The processed image is geometrically transformed to generate the corresponding normal map, such as Figure 4(b) shows that the horizontal and vertical gradients of the depth map are calculated first, and then the normal vector is calculated through the gradient, and the normal vector is mapped to the RGB channel to generate the normal map. The depth map and the normal map are stacked to obtain the final bonding wire dataset.

[0058] 4. Dataset data enhancement

[0059] The built bonding wire dataset is enhanced to increase the data size. The Albumentations library is used to perform various enhancement operations on the depth image and normal image, including: horizontal flipping the image with a probability of 50%, randomly rotating the image 90 degrees with a probability of 50%, randomly cropping the image with a probability of 50%, applying Gaussian blur with a probability of 20%, with a blur level between 3 and 7, and performing Gamma correction on the image with a probability of 20%. The final enhanced dataset contains 10,000 depth images and 10,000 normal images.

[0060] 5. Bonding wire defect detection

[0061] The bonding wire defect detection model Depth-YOLO is improved based on the YOLOv8 network. The input layer of the original model is reconstructed to adapt to the self-built data set. The input feature enhancement module is designed to enhance the ability to extract depth information and geometric information. The improved Depth-YOLO model is used for bonding wire defect detection.

[0062] (1) Data preparation

[0063] The dataset used for training this method is a self-built bonding wire dataset, which contains 20,000 bonding wire images, of which 10,000 images are annotated, including 7 defective wires and a normal wire without defects. 60% is selected as the training set, 15% as the validation set, and 25% as the test set. The defect types are: Interference, Bridge, Middlebroken, Singlebroken, Radiananomaly, Curve, Taillong, Normal, and their number accounts for 8.62%, 8.58%, 8.68%, 8.72%, 8.57%, 3.39%, 8.27%, 8.57%, and 40% of the total data, respectively.

[0064] (2) Model construction

[0065] The structure of the bonding wire defect detection model Depth-YOLO is as follows Figure 5 As shown in the figure, the model is improved based on the YOLOv8 model. The original input layer of the YOLOv8 model is reconstructed to adapt to the self-built data set, and an input feature enhancement module is designed to enhance the model's ability to extract depth information and geometric information.

[0066] The model is built based on the Tensorflow framework. The convolution kernel size used in the first five convolution layers is 3×3, the stride is 1×1, the activation function is ReLU, and the number of input channels is 4. A 2×2 maximum pooling layer is used after each convolution layer for downsampling. The convolution kernel size used in the Bottleneck layer is 1×1, the stride is 1×1, and the activation function is ReLU, which reduces the number of channels in the feature map. The SPP layer uses multiple convolution kernels of different sizes (5×5, 9×9, 13×13) for pooling operations. The PAN layer fuses features at different levels through connection operations, and the number of channels in the feature map is 256, 512, and 1024, respectively. The convolution kernel size used in the Head layer is 1×1, the stride is 1×1, and Sigmoid is used as the activation function. The loss function uses a combined loss function that combines bounding box localization loss, classification loss, and target confidence loss. The expression of the loss function is shown in formula (11). Where λ loc , cls , obj They are the bounding box localization losses Classification Loss and target confidence loss The bounding box localization loss uses CIoU loss to regress the bounding box, as shown in formula (12), and IoU is the intersection over union ratio between the predicted box and the true box. The classification loss uses binary cross entropy loss to calculate the classification loss, as shown in formula (13). Where C is the total number of categories, y c Indicates whether the target belongs to the category, p c Represents the probability that the model predicts that it belongs to category c. The target confidence loss uses binary cross entropy loss to calculate the target confidence loss, as shown in formula (14). obj Indicates whether there is a target in the grid cell, p obj Represents the target confidence of the model prediction.

[0067]

[0068] (3) Input feature enhancement module

[0069] The designed input feature enhancement module includes four parts: depth enhancement, texture enhancement, feature fusion and geometric feature extraction. The depth enhancement part extracts the depth information of the depth map in the stacked image, and enhances the model's ability to extract depth information through multi-scale convolution, batch normalization and residual connection. The texture enhancement part extracts the surface geometry information and gradient information of the normal map in the stacked image to provide auxiliary information for the depth feature. The feature fusion part uses the channel attention mechanism and the spatial attention mechanism to further enhance the model's attention to important features, and fuses the extracted depth and normal features. The geometric feature extraction part further extracts the fused geometric features through convolution.

[0070] The channel attention mechanism first performs global average pooling on the features of each channel to obtain a global description of each channel, as shown in formula (15). c,i,j is the value of position (i, j) on channel c. The fully connected layer is then used to calculate the weight of each channel, as shown in formula (16). Where W1 and W2 are the weights of the fully connected layer, δ is the ReLU activation function, and σ is the Sigmoid activation function. The original feature map is weighted according to the channel weight. The spatial attention mechanism performs maximum pooling and average pooling on the input feature map in the channel dimension to obtain two feature maps, which are then concatenated. The concatenated feature map is convolved with a 7×7 convolution to obtain the spatial weight of each position, and each position of the input feature map is weighted according to the weight.

[0071]

[0072] s=σ(W2δ(W1z)) (16)

[0073] 6. Model detection results and visualization

[0074] mAP@0.5, precision P, and recall R are used as evaluation indicators, as shown in formulas (17), (18), and (19). Among them, TP is the number of correct samples, FP is the number of false positive samples, and FN is the number of missed positive samples. Average precision AP is the average value of the accuracy when the recall rate is between 0 and 1. mAP is the indicator obtained by summarizing and averaging the average precision of all categories. It comprehensively considers the accuracy and recall rate and can well measure the target detection effect. The PR curve and mAP of the model detection result are shown in Figure 6 As shown in (a), the detection results are visualized as follows Figure 6 (b) as shown.

[0075]

Claims

1. A bonding wire defect detection method based on deep learning, characterized in that: The steps include: Step 1, photograph the bonding wire using a 3D stereo vision camera to obtain multiple original depth images; Step 2, using normalization, superposition denoising strategy and adaptive histogram equalization method to pre-process the original depth image to improve the quality of the bond wire depth image; Step 3, use Labelimg to annotate the processed depth image in txt format; build a bonding wire depth image dataset; Step 4: Use the Sobel difference operator to calculate the gradient information of the depth image, calculate the normal vector of each pixel based on the gradient, map the three components of the normal vector in the three-dimensional space to the three RGB channels respectively, use RGB color coding to represent the direction of the normal line in the three-dimensional space, and generate a normal map; Step 5, stacking the generated RGB three-channel normal map and the single-channel depth map, expanding the image dimension and merging them to generate a four-channel pseudo RGBD image containing three-dimensional information; Step 6: Perform data enhancement on the stacked images generated in step 5, and use the albumentations library to enhance the depth image, normal map, and label file at the same time. A total of 20,000 images are enhanced to obtain the final bonding wire dataset; Step 7, propose a bonding wire defect detection model Depth-YOLO; reconstruct the input layer of the two-dimensional target detection network model YOLOv8, modify DataLoader so that it can correctly load and process 4-channel images, expand the number of data input channels to 4, and modify the initial convolution of the model input layer; modify the model pre-trained weight loading, only load the weights of the model except the first convolution layer, and randomly initialize the weights of the first layer, and use the Depth-YOLO model to train the bonding wire dataset in step 6; Step 8: Design an input feature enhancement module to embed the model in step 7 to enhance the extraction of depth information in the depth map and geometric information in the normal map.

2. A bonding wire defect detection method based on deep learning as claimed in claim 1, characterized in that: Step 4 is as follows: (1) Deep Image Quality Enhancement The original bond wire depth image is a single-channel depth image with a size of 3600x2682, a data type of 16-bit unsigned integer, and a format of tiff. The preprocessing process for the original depth image is as follows: first, the depth image is normalized, and the normalized depth image is denoised. The stacking denoising strategy is adopted. First, Gaussian filtering is used to remove high-frequency noise, and then median filtering is used to remove local scattered noise. Then, bilateral filtering is used to continue denoising while retaining image edge information. The denoised image is subjected to adaptive histogram equalization to obtain the final processed image. (2) Geometric transformation to generate normal map The preprocessed depth image is geometrically transformed to generate a normal map; the normal vector N(x, y) is expressed as shown in formula (1); Z(x, y) represents the depth value of each pixel in the depth map, and the Sobel filter is used to calculate the gradient of each pixel in the depth map relative to its adjacent pixels to obtain the horizontal gradient G x and the vertical gradient G y , as shown in formula (2); normalize to obtain the unit normal vector N′(x,y), as shown in formula (3); each component of the calculated normal vector is in the range of [-1,1], map it to the range of [0,255], and realize the visualization of the normal map through RGB color coding, as shown in formula (4); N(x,y)=(-G x ,-G y ,1) (1) (3) Stacking the depth map and normal map Stack the generated normal map with the original depth map; The depth map is a single-channel image with a shape of (H, W); the normal map is a three-channel RGB image with a shape of (H, W, 3); the depth map is expanded and stacked with the normal map to obtain a four-channel pseudo RGBD image with a shape of (H, W, 4); the stacked pseudo RGBD image dataset is used to train the deep learning network model YOLOv8 for defect detection of bonding wires.

3. The bonding wire defect detection method based on deep learning according to claim 1, characterized in that: Step 7 is as follows: (1) Data preparation The dataset used for training this method is a self-built bonding wire dataset, which contains 20,000 bonding wire images, some of which are annotated, including 7 defective wires and one normal wire without defects; it is divided into training set, verification set, and test set; the defect types are: Interference, Bridge, Middle Broken, Single Broken, Radian Anomaly, Curve, Tail Long, Normal; (2) Model construction The bonding wire defect detection model Depth-YOLO is improved based on the YOLOv8 model. The input layer of the two-dimensional target detection network model YOLOv8 is rebuilt to adapt to the self-built data set. The DataLoader is modified to enable it to correctly load and process 4-channel images, the number of data input channels is expanded to 4, and the initial convolution of the model input layer is modified; The model is built based on the Tensorflow framework. The convolution kernel size used in the first five convolution layers is 3×3, the step size is 1×1, the activation function is ReLU, and the number of input channels is 4. A 2×2 maximum pooling layer is used after each convolution layer for downsampling. The convolution kernel size used in the Bottleneck layer is 1×1, the step size is 1×1, and the activation function is ReLU to reduce the number of channels in the feature map. The SPP layer uses multiple convolution kernels of different sizes (5×5, 9×9, 13×13) for pooling operations. The PAN layer fuses features at different levels through connection operations, and the number of channels in the feature map is 256, 512, and 1024 respectively. The convolution kernel size used in the Head layer is 1×1, the step size is 1×1, and Sigmoid is used as the activation function. The loss function uses a combined loss function that combines bounding box positioning loss, classification loss, and target confidence loss. The expression of the loss function is shown in formula (5); where λ loc , cls , obj They are the bounding box localization losses Classification Loss and target confidence loss The weight of the bounding box positioning loss uses CIoU loss to regress the bounding box, as shown in formula (6), and IoU is the intersection-over-union ratio of the predicted box and the true box; the classification loss uses binary cross entropy loss to calculate the classification loss, as shown in formula (7); Where C is the total number of categories, y c Indicates whether the target belongs to the category, p c represents the probability that the model predicts that it belongs to category c; the target confidence loss uses the binary cross entropy loss to calculate the target confidence loss, as shown in formula (8); where y obj Indicates whether there is a target in the grid cell, p obj Represents the target confidence of the model prediction; 4. The method for welding wire defect detection and pattern recognition based on image-point cloud information fusion according to claim 1, characterized in that: Step 8 is as follows: The designed input feature enhancement module includes four parts: depth enhancement, texture enhancement, feature fusion and geometric feature extraction. The depth enhancement part extracts the depth information of the depth map in the stacked image, and enhances the model's ability to extract depth information through multi-scale convolution, batch normalization and residual connection. The texture enhancement part extracts the surface geometry information and gradient information of the normal map in the stacked image to provide auxiliary information for the depth feature. The feature fusion part uses channel attention mechanism and spatial attention mechanism to further enhance the model's attention to important features and fuse the extracted depth and normal features; The geometric feature extraction part further extracts the fused geometric features through convolution; The first three convolutional layers in the multi-scale convolution use convolution kernels with sizes of 3×3, 5×5, and 7×7 and a step size of 1×1 to extract local depth features at multiple scales from the single-channel depth map. The number of output feature maps is 16, the activation function is ReLU, and batch normalization and residual connections are used to stabilize training and retain the original features. Conv4 uses a convolution kernel with a size of 3×3 and a step size of 1×1 to extract gradient information from the three-channel normal map; the Fusion layer uses a convolution kernel with a size of 1×1 and a step size of 1×1 to fuse the gradient features on different channels and integrate the different direction information in the normal map; the ChannelAttention part uses a 1×1 global average pooling and a 1×1 convolution to reduce the number of channels, while enhancing nonlinear expression, and generates the importance weight of each channel through the Sigmoid activation function; The SpatialAttention part uses a 7×7 convolution to generate a single-channel feature map to represent the weight of each position. The Sigmoid activation function is used to generate the attention weight of each pixel position to enhance the model's ability to extract geometric features. The final convolution layer uses a 3×3 convolution kernel with a step size of 1×1 to further extract important geometric features after fusing the depth features and normal features, and uses residual connections to ensure that geometric information is not lost during the extraction process. The channel attention mechanism first performs global average pooling on the features of each channel to obtain a global description of each channel, as shown in formula (9); where x c,i,j is the value of position (i, j) on channel c; the fully connected layer is then used to calculate the weight of each channel, as shown in formula (10); Where W1 and W2 are the weights of the fully connected layer, δ is the ReLU activation function, and σ is the Sigmoid activation function; and the original feature map is weighted according to the channel weight; the spatial attention mechanism performs maximum pooling and average pooling on the input feature map in the channel dimension to obtain two feature maps, which are then concatenated, and the concatenated feature map is convolved using a 7×7 convolution to obtain the spatial weight of each position, and each position of the input feature map is weighted according to the weight; s=σ(W2δ(W1z)) (10).

Images