A PCB defect detection method and system based on multi-task learning mechanism

Abstract

The present invention relates to the technical field of industrial quality inspection and discloses a PCB defect detection method based on a multi-task learning mechanism. A defect image is input into a PCB defect detection model to obtain a defect area corresponding to the defect image. A training method for the PCB defect detection model comprises the following steps: after acquiring the defect image, data preprocessing is performed to obtain a defect sample dataset; defect types in the defect image include offset defects and missing component defects; and a PCB defect detection model based on a U-Net structure is constructed. Loss functions are set for the offset defect detection task, the missing component defect detection task, and the binary defect detection task, respectively, and different weights are set for the loss function of each task through a parameter search method. The model can utilize information complementarity between tasks, reduce the risk of overfitting, and fully utilize multi-level patch component defect information.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial quality inspection, and in particular to a PCB defect detection method and system based on a multi-task learning mechanism. Background Art

[0002] PCBs (printed circuit boards), or PCBs for short, are essential components in the electronics industry. Virtually every electronic device, from small electronic watches to large computers, communications equipment, and military weapon systems, that incorporates integrated circuits and other electronic components, requires a PCB to interconnect them. While algorithms exist to detect defects in surface mount components (SMTs) on PCBs, existing general-purpose algorithms struggle to detect defects across a wide range of SMT components. Furthermore, the types and characteristics of defects are often diverse, leading to instability and limited adaptability in traditional defect detection methods.

[0003] Currently, multi-task deep learning is gaining increasing attention. This method can improve generalization performance and reduce computational complexity through information sharing. Deep learning typically requires a large amount of training data, but collecting a large number of PCB component packaging defect samples is extremely difficult. Furthermore, among various surface defects on PCBs, the mounted components on PCBs are relatively small, often numbering in the thousands and densely distributed, making detection speeds inadequate for conventional deep learning methods. Product defect detection requires high accuracy (99%), which cannot be achieved by existing industry AI detection models.

[0004] Therefore, to address the problems of model overfitting and poor generalization performance caused by a small number of offset and missing samples, difficulty in detecting sample features, and weak features, the present invention designs a PCB defect detection method and system based on a multi-task learning mechanism. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a PCB defect detection method and system based on a multi-task learning mechanism.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A PCB defect detection method based on a multi-task learning mechanism obtains a feature map of an image through an encoder of a PCB defect detection model. The feature map is input into each decoder of the PCB defect detection model to detect whether the image is a defective image and obtain the defect area of the defective image.

[0008] The training method of the PCB defect detection model includes the following steps:

[0009] Step 1: After acquiring the defect image, perform data enhancement and data preprocessing to obtain a defect sample dataset; the types of defects in the defect image include offset defects and missing parts defects;

[0010] Step 2: Build a PCB defect detection model based on U-Net structure:

[0011] The PCB defect detection model includes a shared encoder and three decoders: an offset defect decoder, a missing part defect decoder, and a binary defect detection decoder. A U-Net structure is used between the shared encoder, the offset defect decoder, and the missing part defect decoder, while the binary defect detection decoder uses a fully connected multilayer perceptron structure. The shared encoder and the offset defect decoder are used for offset defect detection, the shared encoder and the missing part defect decoder are used for missing part defect detection, and the shared encoder and the binary defect detection decoder are used for detecting whether an image has defects.

[0012] Step 3: Set loss functions for the offset defect detection task, missing part defect detection task, and binary defect detection task respectively, and set different weights for the loss function of each task through parameter search.

[0013] Specifically, when performing data enhancement on the acquired defect image in step 1:

[0014] Performing geometric transformation on the defect image means performing geometric transformation on the image, including flipping, rotation, shifting, cropping and scaling operations, and adding Gaussian noise;

[0015] When performing data preprocessing on the acquired defect image in step 1:

[0016] The defect image is scaled proportionally, gray bars are added, and the size is adjusted to 512 pixels * 512 pixels.

[0017] Specifically, in step 3, the loss function L1 of the offset defect detection task is Dice + Focal ;

[0018] The loss function L2 of the missing part defect detection task is Dice + Focal ;

[0019] The loss function of the binary defect detection task is L3=L logic ;

[0020] Among them, L Dice is the similarity coefficient loss function, L Focal is the Focal loss function, L logic is the logistic loss function, X is the area of the predicted sample positive example, Y is the area of the true sample positive example, ε is a parameter set to prevent the denominator from being zero; L Focal =-α(1-) γ log(p), α is the parameter for reducing the weight of samples of different categories, γ is the modulation coefficient, and p is the predicted value of the sample; y i is the sample value of the label, g(x) is the real sample value;

[0021] Loss function L of PCB defect detection model Total =1L1+2L2+3L3, λ1, λ2, and λ3 are the weights of the missing part loss function, the offset loss function, and the binary defect detection loss function, respectively.

[0022] Specifically, in step 2, Resnet is used as the backbone feature extraction network of the shared encoder for feature extraction.

[0023] A PCB defect detection system based on a multi-task learning mechanism includes a PCB defect detection model; a defect image is input into the PCB defect detection model to obtain a defect area corresponding to the defect image;

[0024] The training module of the PCB defect detection model includes:

[0025] Dataset acquisition module: After acquiring defect images, data enhancement and data preprocessing are performed to obtain a defect sample dataset; the types of defects in defect images include offset defects and missing parts defects;

[0026] PCB defect detection model construction module, used to build a PCB defect detection model based on U-Net structure:

[0027] The PCB defect detection model consists of a shared encoder and three decoders: an offset defect decoder, a missing part defect decoder, and a binary defect decoder. A U-Net structure is used between the shared encoder and the first two decoders, while the latter decoder uses a fully connected multilayer perceptron structure. The shared encoder and the offset defect decoder are used for offset defect detection, the shared encoder and the missing part defect decoder are used for missing part defect detection, and the shared encoder and the binary defect decoder are used for image defect detection.

[0028] Loss training module: Set loss functions for the offset defect detection task, missing part defect detection task, and binary defect detection task respectively, and set different weights for the loss function of each task through parameter search.

[0029] The system of the present invention corresponds to the method, and the preferred solutions in the method are also applicable to the system.

[0030] Compared with the prior art, the beneficial technical effects of the present invention are:

[0031] The present invention adopts the U-Net network structure and uses Resnet as the encoder backbone feature extraction network. Different task-related networks are connected at the decoding end, including offset defect detection tasks, missing part defect detection tasks and binary classification defect detection tasks. The information complementarity between tasks is utilized to reduce the risk of overfitting and make full use of multi-level patch component defect information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a structural block diagram of the PCB defect detection model of the present invention;

[0033] Figure 2 Schematic diagram of the U-Net structure. DETAILED DESCRIPTION

[0034] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] The present invention proposes a PCB defect detection method based on a U-Net structure, which uses residual networks (ResNets) as the encoder backbone feature extraction network and connects different task-related networks at the decoding end. This method improves the defect detection effect through multi-task learning and can solve technical problems such as insufficient data and difficult feature detection in existing methods for PCB defect detection.

[0036] The PCB defect detection model training method includes the following steps

[0037] S1: Acquire defect images and perform data enhancement and data preprocessing to obtain a defect sample dataset. Specifically including:

[0038] S11: Obtain a defect sample dataset. The image data used in this invention comes from actual inspection projects, including normal images and defect images. The images are scaled proportionally, gray bars are added, and the size is adjusted to 512 pixels * 512 pixels. Finally, they are fed into the backbone feature extraction network of the PCB defect detection model.

[0039] S12: Data augmentation techniques were used to increase the amount of defect sample data. Data fusion was used to generate 1,000 PCB defect images, including 1,037 samples of component offset defects and 1,040 samples of missing parts. Different sample ratios were used to improve the detection model's performance with insufficient data. Because each defect image contains multiple defect samples, the number of defect samples is greater than the number of defect images. The defect sample dataset consists of a training set, a test set, and a validation set.

[0040] Table 1. The number of training sets, test sets, and validation sets

[0041] Sample data ratio training set Test set Validation set 25％ 180 35 35 50％ 360 70 70 75％ 540 105 105 100％ 720 140 140

[0042] S2: Build a PCB defect detection model based on U-Net structure.

[0043] like Figure 2 As shown in the figure, U-Net is a network structure based on a fully convolutional network. It uses a series of encoder and decoder blocks to perform pixel-level, dense predictions on the input image. Its structure is simple but very effective. The encoder is responsible for feature extraction, and the decoder combines feature information through 3*3 up-convolution. The U-Net structure is mainly divided into three parts: the first part is the backbone feature extraction network, which uses the ResNet network as the encoder. The backbone feature extraction network can obtain five preliminary feature maps. In the second step, these five feature maps are used for feature fusion. The second part is the enhanced feature extraction network, namely the decoder. In this part, upsampling is performed layer by layer, and the feature maps are fused with the feature maps obtained by the previous encoder, followed by 3*3 convolution. The third part is the prediction part, which uses 1*1 convolution. We will use the final valid feature layer to classify each feature point, which is equivalent to classifying each pixel.

[0044] This paper draws on the U-Net structure to design a PCB defect detection model with a multi-task network structure, such as Figure 1 As shown in Figure 1, the PCB defect detection model consists of a shared encoder and three decoders: an offset defect decoder, a missing part defect decoder, and a binary defect detection decoder. A U-Net structure is used between the shared encoder and each decoder. The shared encoder and offset defect decoder are used for offset defect detection, the shared encoder and missing part defect decoder are used for missing part defect detection, and the shared encoder and binary defect detection decoder are used for image defect detection. The encoder uses a residual network (ResNet).

[0045] S3: Loss functions are set for the offset defect detection task, the missing part defect detection task, and the binary defect detection task respectively. In order to avoid the problem that some tasks dominate the gradient descent algorithm and cannot fully optimize other tasks, the present invention sets different weights for the loss function of each task through parameter search.

[0046] Specifically, step S3 includes:

[0047] S31: For the offset semantic segmentation task, the present invention adopts the Focal loss function and the Dice coefficient loss function. The Focal loss function formula is: L Focal = -α(1-p) γ log(p), where the function of α is to reduce the weight of samples of different categories. If there are fewer positive samples, the weight of positive samples will increase. γ is the modulation coefficient. When the difference in the number of positive and negative samples is greater, γ is larger. When γ = 0, the Focal loss function is the traditional cross entropy loss. The role of γ is that when the predicted value p of the sample is relatively large, the easy sample will be relatively small. In this way, the loss of easy samples will be significantly reduced, and the model will pay more attention to the optimization of the loss of difficult samples. Dice coefficient loss function It is a loss function determined by the intersection-over-union ratio of the X and Y regions. ε is a small positive number to prevent the denominator from being zero.

[0048] S32: The inductive missing part semantic segmentation task has similar characteristics to the offset defect area extraction task, and its loss function L2 is the same as the loss function L1.

[0049] S33: In the missing binary classification problem, the loss function L3 is the cross entropy loss function. In both cases, the model needs to predict the result, and the probability of each category prediction is p and 1-p, L logic The formula is:

[0050] Final loss function: L Total =λ1L1+λ2L2+λ3L3.

[0051] It should be noted that before training the PCB defect detection model, the task weight parameters are searched to determine the optimal weight combination, which helps to increase the training speed.

[0052] The present invention utilizes a multi-scale feature pyramid structure to better learn the characteristics of tiny defects, improve detection efficiency, and is adaptable to a variety of defect detections with strong adaptability.

[0053] Evaluate the trained PCB defect detection model.

[0054] S41: mIoU (Mean Intersection over Union) is a commonly used evaluation metric in semantic segmentation:

[0055] j represents the predicted value, i is the true value, k+1 is the number of categories (including empty categories); p ii is a true positive, p ijWith p ji represent false positives and false negatives, respectively.

[0056] S42: Other evaluation metrics include correctness of response (COR), completeness of response (COM), and overall quality (Q). COR is the ratio of the correct value within the sensing area to the test result, while COM is the ratio of the detected area value within the defect area to the marked area. Overall quality (Q) is a comprehensive metric that can be calculated using COM and COR.

[0057]

[0058]

[0059]

[0060] Since defects have irregular morphological characteristics, there will be certain deviations when manually marking them, which makes the COR and COM indicators very sensitive to the detection results. In order to improve this problem, this paper also uses relaxed COR, COM and Q indicators, which are recorded as R_COR, R_COM and R_Q respectively.

[0061] Table 2 Evaluation results of PCB defect detection model

[0062]

[0063]

[0064] Figure 1 The missing part defects and offset defects are segmented out, and the binary classification network is used to determine whether the image has defects.

[0065] Experimental results using a defect sample dataset, based on varying data volume ratios, show that when the number of defective samples accounts for only 25% of the total sample size, the performance indicators (R_COR) reach 92.8% and R_COM reaches 76.5%, effectively overcoming the drawback of insufficient data samples. Furthermore, when the data ratio reaches 100%, the accuracy of PCB defect detection reaches 93.2%, and the recall rate of defective products reaches 100%, fully meeting industrial quality inspection standards.

[0066] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0067] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A PCB defect detection method based on a multi-task learning mechanism. The encoder of the PCB defect detection model obtains a feature map of the input image. The feature map is input into each decoder of the PCB defect detection model to detect whether the image is a defective image and obtain the defect area of the defective image. The training method of the PCB defect detection model includes the following steps: Step 1: After acquiring the defect image, perform data enhancement and data preprocessing to obtain a defect sample dataset; the types of defects in the defect image include offset defects and missing parts defects; Step 2: Build a PCB defect detection model based on U-Net structure: The PCB defect detection model includes a shared encoder and three decoders: an offset defect decoder, a missing part defect decoder, and a binary defect detection decoder. A U-Net structure is used between the shared encoder, the offset defect decoder, and the missing part defect decoder, while the binary defect detection decoder uses a fully connected multilayer perceptron structure. The shared encoder and the offset defect decoder are used for offset defect detection, the shared encoder and the missing part defect decoder are used for missing part defect detection, and the shared encoder and the binary defect detection decoder are used for detecting whether an image has defects. Step 3: Set loss functions for the offset defect detection task, missing part defect detection task, and binary defect detection task respectively, and set different weights for the loss function of each task through parameter search.

2. The PCB defect detection method based on the multi-task learning mechanism according to claim 1 is characterized in that: When performing data enhancement on the acquired defect image in step 1: Performing geometric transformation on the defect image means performing geometric transformation on the image, including flipping, rotation, shifting, cropping and scaling operations, and adding Gaussian noise; When performing data preprocessing on the acquired defect image in step 1: The defect image is scaled proportionally, gray bars are added, and the size is adjusted to 512 pixels * 512 pixels.

3. The PCB defect detection method based on the multi-task learning mechanism according to claim 1 is characterized in that: In step 3, the loss function of the offset defect detection task is L1 = L Dice coff +L Focal ; The loss function of the missing part defect detection task is L2=L Dice coff +L Focal ; The loss function of the binary defect detection task is L3=L logic ; Among them, L Dice coff is the similarity coefficient loss function, L Focal is the Focal loss function, L logic is the logistic loss function, X is the area of the predicted sample positive example, Y is the area of the true sample positive example, ε is a parameter set to prevent the denominator from being zero; L Focal =-α(1-) γ log(p), α is the parameter for reducing the weight of samples of different categories, γ is the modulation coefficient, and p is the predicted value of the sample; y i is the sample value of the label, g(x) is the predicted sample value; Loss function L of PCB defect detection model Total =1L1+2L2+3L3, λ1, λ2, and λ3 are the weights of the missing part loss function, the offset loss function, and the binary defect detection loss function, respectively.

4. The PCB defect detection method based on the multi-task learning mechanism according to claim 1 is characterized in that: In step 2, Resnet is used as the backbone feature extraction network of the shared encoder for feature extraction.

5. A PCB defect detection system based on a multi-task learning mechanism, characterized in that: Includes PCB defect detection model; Inputting the defect image into the PCB defect detection model can obtain the defect area corresponding to the defect image; The training module of the PCB defect detection model includes: Dataset acquisition module: After acquiring defect images, data enhancement and data preprocessing are performed to obtain a defect sample dataset; the types of defects in defect images include offset defects and missing parts defects; PCB defect detection model construction module, used to build a PCB defect detection model based on U-Net structure: The PCB defect detection model consists of a shared encoder and three decoders: an offset defect decoder, a missing part defect decoder, and a binary defect decoder. A U-Net structure is used between the shared encoder and the first two decoders, while the latter decoder uses a fully connected multilayer perceptron structure. The shared encoder and the offset defect decoder are used for offset defect detection, the shared encoder and the missing part defect decoder are used for missing part defect detection, and the shared encoder and the binary defect decoder are used for image defect detection. Loss training module: Set loss functions for the offset defect detection task, missing part defect detection task, and binary defect detection task respectively, and set different weights for the loss function of each task through parameter search.

Images