PCB defect detection method and device, storage medium and electronic equipment

Abstract

The invention relates to a PCB defect detection method, a PCB defect detection device, a storage medium and electronic equipment. The PCB defect detection method comprises the following steps: acquiring a PCB image to be subjected to defect identification; inputting the PCB image to be subjected to defect identification into a twin network, and identifying large target defect features and small target defect features of the PCB image to obtain a first feature matrix representing the large target defect features and a second feature matrix representing the small target defect features; wherein the twin network is obtained by pre-training based on a PCB sample set; pCA data dimension reduction and sparse matrix processing are carried out on the first feature matrix representing the large target defect features and the second feature matrix representing the small target defect features, and fusion features are obtained; and inputting the fusion features into a YOLO model, and carrying out defect identification to obtain the types and positions of defects existing in the PCB image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a PCB defect detection method, a PCB defect detection device, a storage medium and an electronic device. Background Art

[0002] With the development of electronic manufacturing technology, printed circuit boards (PCBs) are increasingly used in electronic devices. PCBs are prone to various defects during the manufacturing process, such as short circuits, open circuits, and foreign body insertion, which seriously affect the performance and life of electronic devices. Therefore, PCB defect detection has become a very important part of the electronic manufacturing process. However, traditional PCB defect detection methods, such as manual visual subjective judgment and instrument online detection methods, have some inherent problems. Although the manual visual judgment method is low in cost, it relies on the operator's experience and vision, has low accuracy and efficiency, and is difficult to analyze data sets. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure are intended to provide a PCB defect detection method, a PCB defect detection device, a storage medium, and an electronic device.

[0004] The technical solution of the present disclosure is achieved as follows:

[0005] In a first aspect, the present disclosure provides a PCB defect detection method.

[0006] The PCB defect detection method provided by the embodiment of the present disclosure includes:

[0007] Obtain a PCB image to be defect-identified;

[0008] The PCB image to be defect-identified is input into the twin network, and the large target defect features and small target defect features of the PCB image are identified to obtain a first feature matrix characterizing the large target defect features and a second feature matrix characterizing the small target defect features; wherein the twin network is pre-trained based on a PCB sample set; wherein the large target defect features and the small target defect features are two types of defect feature types divided according to the defect features of PCB defects;

[0009] The first feature matrix representing the large target defect characteristics and the second feature matrix representing the small target defect characteristics are subjected to PCA data dimension reduction and sparse matrix processing to obtain fusion features;

[0010] The fused features are input into the YOLO model to perform defect recognition, and the types and locations of defects in the PCB image are obtained.

[0011] In some embodiments, before inputting the PCB image to be defect-identified into the twin network, identifying large target defect features and small target defect features of the PCB image, and obtaining a first feature matrix characterizing large target defect features and a second feature matrix characterizing small target defect features, the method includes:

[0012] Acquire a PCB sample set; wherein the PCB sample set is marked with the type of PCB defects to be identified;

[0013] The PCB sample set marked with the types of PCB defects to be identified is input into the twin network, and the weight parameters of the twin network are trained to obtain the twin network for feature recognition and extraction and the training feature matrix characterizing the large target defect features and the small target defect features.

[0014] In some embodiments, before inputting the fusion feature into the YOLO model to perform defect recognition and obtain the type and location of the defect in the PCB image, the method includes:

[0015] Perform PCA data dimension reduction and sparse matrix processing on the training feature matrix representing the large target defect features and the small target defect features of the twin network to obtain training fusion features;

[0016] The training fusion features are input into the YOLO model, and the YOLO model is trained to obtain a YOLO model for PCB image defect recognition and detection.

[0017] In some embodiments, after inputting the training fusion features into a YOLO model, performing YOLO model training, and obtaining a YOLO model for PCB image defect recognition and detection, the method includes:

[0018] Add a 1*1 convolution kernel after the C3, C4, and C5 layers of the YOLO model for PCB image defect recognition and detection;

[0019] The YOLO model with a 1*1 convolution kernel is used as the YOLO model for PCB defect detection.

[0020] In some embodiments, the large target defect feature and the small target defect feature both include at least one of the following information corresponding to the PCB defect: color, size, texture, and shape.

[0021] In a second aspect, the present disclosure provides a PCB defect detection device, comprising:

[0022] An image acquisition module, used to acquire a PCB image to be identified by defects;

[0023] A feature recognition module is used to input the PCB image to be defect-identified into a twin network, identify large target defect features and small target defect features of the PCB image, and obtain a first feature matrix characterizing large target defect features and a second feature matrix characterizing small target defect features; wherein the twin network is pre-trained based on a PCB sample set; wherein the large target defect features and the small target defect features are two types of defect feature types divided according to the defect features of PCB defects;

[0024] A feature fusion module is used to perform PCA data dimension reduction and sparse matrix processing on the first feature matrix representing the large target defect characteristics and the second feature matrix representing the small target defect characteristics to obtain fusion features;

[0025] The defect recognition module is used to input the fusion features into the YOLO model to perform defect recognition and obtain the types and locations of defects in the PCB image.

[0026] In some embodiments, a model training module is included;

[0027] The PCB image to be defect-identified is input into the twin network to identify the large target defect features and the small target defect features of the PCB image, and before obtaining the first feature matrix representing the large target defect features and the second feature matrix representing the small target defect features, the model training module is used

[0028] Acquire a PCB sample set; wherein the PCB sample set is marked with the type of PCB defects to be identified;

[0029] The PCB sample set marked with the types of PCB defects to be identified is input into the twin network, and the weight parameters of the twin network are trained to obtain the twin network for feature recognition and extraction and the training feature matrix characterizing the large target defect features and the small target defect features.

[0030] In some embodiments, before the fusion feature is input into the YOLO model to perform defect recognition and obtain the defect type and location in the PCB image, the model training module is used to

[0031] Perform PCA data dimension reduction and sparse matrix processing on the training feature matrix representing the large target defect features and the small target defect features of the twin network to obtain training fusion features;

[0032] The training fusion features are input into the YOLO model, and the YOLO model is trained to obtain a YOLO model for PCB image defect recognition and detection.

[0033] In a third aspect, the present disclosure provides a computer-readable storage medium having a PCB defect detection program stored thereon. When the PCB defect detection program is executed by a processor, the PCB defect detection method described in the first aspect is implemented.

[0034] In a fourth aspect, the present disclosure provides an electronic device, comprising a memory, a processor, and a PCB defect detection program stored in the memory and executable on the processor, wherein when the processor executes the PCB defect detection program, the PCB defect detection method described in the first aspect is implemented.

[0035] According to the embodiment of the present disclosure, the PCB defect detection method includes obtaining a PCB image to be defect-identified; inputting the PCB image to be defect-identified into a twin network, performing large target defect features and small target defect features identification of the PCB image, and obtaining a first feature matrix characterizing large target defect features and a second feature matrix characterizing small target defect features; wherein the twin network is pre-trained based on a PCB sample set; performing PCA data dimension reduction and sparse matrix processing on the first feature matrix characterizing large target defect features and the second feature matrix characterizing small target defect features to obtain fusion features; inputting the fusion features into a YOLO model to perform defect identification, and obtaining the types and locations of defects existing in the PCB image. In this application, by introducing a twin network to replace the Backbone of the original YOLO model, the twin network has strong robustness to the problem of class imbalance, so that when the number of samples of certain categories in the training data is far less than that of other categories, the twin network can still effectively learn and identify these minority category samples. At the same time, the first feature matrix characterizing large target defect features and the second feature matrix characterizing small target defect features are subjected to PCA data dimension reduction and sparse matrix processing to obtain fusion features. Through feature fusion, the YOLO model can capture information of different scales at the same time, thereby improving the model's recognition ability and detection accuracy for defects of various sizes.

[0036] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of a PCB defect detection method according to an exemplary embodiment;

[0038] Figure 2 It is a schematic structural diagram of a PCB defect detection device according to an exemplary embodiment. DETAILED DESCRIPTION

[0039] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0040] With the development of electronic manufacturing technology, printed circuit boards (PCBs) are increasingly used in electronic devices. PCBs are prone to various defects during the manufacturing process, such as short circuits, open circuits, and foreign body insertion, which seriously affect the performance and life of electronic devices. Therefore, PCB defect detection has become a very important part of the electronic manufacturing process. However, traditional PCB defect detection methods, such as manual visual subjective judgment and instrument online detection methods, have some inherent problems. Although the manual visual judgment method is low in cost, it relies on the operator's experience and vision, has low accuracy and efficiency, and is difficult to analyze data sets.

[0041] In view of the above situation, the present disclosure provides a PCB defect detection method. Figure 1 FIG. 1 is a flow chart of a PCB defect detection method according to an exemplary embodiment. Figure 1 As shown, the PCB defect detection method includes:

[0042] Step 10, obtaining a PCB image to be defect-identified;

[0043] Step 11, inputting the PCB image to be defect-identified into the twin network, identifying the large target defect features and the small target defect features of the PCB image, and obtaining a first feature matrix characterizing the large target defect features and a second feature matrix characterizing the small target defect features; wherein the twin network is pre-trained based on a PCB sample set; wherein the large target defect features and the small target defect features are two types of defect feature types divided according to the defect features of PCB defects;

[0044] Step 12: Perform PCA data dimension reduction and sparse matrix processing on the first feature matrix representing the large target defect characteristics and the second feature matrix representing the small target defect characteristics to obtain fusion features;

[0045] Step 13: Input the fused features into the YOLO model to perform defect recognition to obtain the types and locations of defects in the PCB image.

[0046] In this exemplary embodiment, the defects detected by PCB can include 24 types of defects. Among them, the 24 types of defects mainly include 9 types of board surface defects, 6 types of metal surface defects, 5 types of hole defects and 4 types of line defects. Specifically, board surface defects include 9 types of defects, including copper leakage, foreign matter, scratches, unevenness, edge damage, missing printing (yellow), missing printing (black), oil accumulation and ink missing; metal surface defects include 6 types of defects, including foreign matter, scratches, unevenness, oxidation, brush marks and tin oxidation; hole defects include yellow holes, blocked holes, burrs in the holes, plugging holes and poor hole walls, totaling 5 types of defects; line defects include 4 types of defects, including short circuit gaps, line deviations, residual copper and copper wire.

[0047] Among them, large target defect features and small target defect features are two types of defect feature types divided according to the defect area characteristics of PCB defects. For example, defects with defect areas larger than a predetermined area are classified as large target defect features, and defects with defect areas smaller than a predetermined area are classified as small target defect features.

[0048] For example, large target defect features may include: foreign matter on the board surface, scratches on the board surface, damage to the board edge, missing printing (yellow), missing printing (black), oil accumulation, foreign matter on the metal surface, scratches on the metal surface, oxidation on the metal surface, oxidation and dirt on the tin surface, yellow holes, blocked holes, sunken holes, poor hole walls, and crooked lines;

[0049] Small target defect features may include: copper leakage on the board surface, board surface bumps, ink missing, metal surface bumps, metal surface brush marks, burrs in the hole, short circuit gaps in the line, residual copper in the line, and copper wire in the line.

[0050] In this exemplary embodiment, the large target defect feature and the small target defect feature both include at least one of the following information corresponding to the PCB defect: color, size, texture, and shape.

[0051] In this exemplary embodiment, by introducing a twin network to replace the original YOLOv8 Backbone, the twin network has strong robustness to the class imbalance problem. This means that even when the number of samples of some categories in the training data is much smaller than that of other categories, the twin network can still effectively learn and identify samples of these minority categories. This feature makes the twin network more flexible and effective in dealing with practical problems.

[0052] In this exemplary embodiment, when the large target defect feature and small target defect feature recognition and extraction of the PCB image are performed by the twin network, two sub-networks with different structures can be used as feature extractors. For example, DenseNet is used as one of the sub-networks, and the convolution operation of its dense block is used to improve the extraction ability of small target features; DeepLabV3 is used as another sub-network, and the ASPP spatial hole convolution module is used to obtain a larger receptive field, thereby improving the extraction ability of large target features. The input data of the twin network is input into the two sub-networks at the same time, and the two sub-networks obtain different feature maps through their respective convolution operations (weights are not shared). Further, the two sub-networks are responsible for extracting large target defect features and small target defect features respectively. After obtaining the different feature maps output by the two sub-networks, data fusion is performed at the final output end, the features extracted from different dimensions are merged, and then PCA is used to reduce the data dimension of the fused features. For example, DeepLabV3 is used as the first sub-network to extract large target defect features and obtain a first feature map containing a first feature matrix; DenseNet is used as the second sub-network to extract small target defect features and obtain a second feature map containing a second feature matrix. Then the first feature map and the second feature map are fused, and finally the fused features are subjected to data dimensionality reduction through PCA.

[0053] According to the embodiment of the present disclosure, the PCB defect detection method includes obtaining a PCB image to be defect-identified; inputting the PCB image to be defect-identified into a twin network, performing large target defect features and small target defect features identification of the PCB image, and obtaining a first feature matrix characterizing large target defect features and a second feature matrix characterizing small target defect features; wherein the twin network is pre-trained based on a PCB sample set; performing PCA data dimension reduction and sparse matrix processing on the first feature matrix characterizing large target defect features and the second feature matrix characterizing small target defect features to obtain fusion features; inputting the fusion features into a YOLO model to perform defect identification, and obtaining the types and locations of defects existing in the PCB image. In this application, by introducing a twin network to replace the Backbone of the original YOLO model, the twin network has strong robustness to the problem of class imbalance, so that when the number of samples of certain categories in the training data is far less than that of other categories, the twin network can still effectively learn and identify these minority category samples. At the same time, the first feature matrix characterizing large target defect features and the second feature matrix characterizing small target defect features are subjected to PCA data dimension reduction and sparse matrix processing to obtain fusion features. Through feature fusion, the YOLO model can capture information of different scales at the same time, thereby improving the model's recognition ability and detection accuracy for defects of various sizes.

[0054] In some embodiments, before inputting the PCB image to be defect-identified into the twin network, identifying large target defect features and small target defect features of the PCB image, and obtaining a first feature matrix characterizing large target defect features and a second feature matrix characterizing small target defect features, the method includes:

[0055] Acquire a PCB sample set; wherein the PCB sample set is marked with the type of PCB defects to be identified;

[0056] The PCB sample set marked with the types of PCB defects to be identified is input into the twin network, and the weight parameters of the twin network are trained to obtain the twin network for feature recognition and extraction and the training feature matrix characterizing the large target defect features and the small target defect features.

[0057] In this exemplary embodiment, a high-definition camera is used to take photos of PCB boards actually produced at the customer site. The captured images are then sliced ​​to form a 500*500 local slice image. Professional labelers use the labeling tool Label to label and form a labeling file in XML format, and then use a script tool to convert the XML labeling file into a YOLO labeling file.

[0058] In this exemplary embodiment, the process of dividing the PCB defect data set specifically includes: randomly dividing the prepared PCB surface defect data set into a training set, a validation set, and a test set in a ratio of 8:1:1 according to experimental requirements, and placing the divided data set images and corresponding labels into corresponding folders. Among them, the training set is used to train the twin model and the YOLO model, the test set is used to test the trained twin model and the YOLO model, and the validation set is used to verify the test results of the twin model and the YOLO model.

[0059] For example, according to the classification of large target defect features and small target defect features, the model test results are verified to verify the feature extraction and defect recognition capabilities of the twin model and the YOLO model. Whether the feature extraction results of the twin model and the defect recognition results of the YOLO model are the same as the defect feature classification determined below. Among them, large target defect features may include: foreign matter on the board surface, scratches on the board surface, damage to the board edge, missing print (yellow), missing print (black), oil accumulation, foreign matter on the metal surface, scratches on the metal surface, oxidation on the metal surface, oxidized tin surface, yellow holes, blocked holes, sunken holes, poor hole walls, and crooked lines; small target defect features may include: copper leakage on the board surface, uneven board surface, missing ink, uneven metal surface, brush marks on the metal surface, burrs on the blank in the hole, short circuit gaps on the line, residual copper on the line, and copper wire on the line. In this way, it is beneficial to effectively improve the feature recognition and extraction capabilities of the twin network.

[0060] In some embodiments, before inputting the fusion feature into the YOLO model to perform defect recognition and obtain the type and location of the defect in the PCB image, the method includes:

[0061] Perform PCA data dimension reduction and sparse matrix processing on the training feature matrix representing the large target defect features and the small target defect features of the twin network to obtain training fusion features;

[0062] The training fusion features are input into the YOLO model, and the YOLO model is trained to obtain a YOLO model for PCB image defect recognition and detection.

[0063] In this exemplary embodiment, the original high-dimensional space contains redundant information and noise information, which will introduce errors in practical applications and affect the accuracy; data dimensionality reduction can extract the essential structure inside the data, reduce the errors caused by redundant information and noise information, and improve the accuracy in the application.

[0064] In this exemplary embodiment, the specific data dimension reduction is as follows:

[0065] PCA (Principal Component Analysis) is used to find the main change directions (i.e., principal components) in the data, and the data is projected onto these directions, thereby achieving data dimensionality reduction. This allows data originally in a high-dimensional space to be represented in a low-dimensional space, reducing the complexity of the data and helping to understand and analyze the data more intuitively. In addition, there may be some noise or redundant features in the fused feature information, which do not contribute much to the overall structure and information of the data. Through PCA dimensionality reduction, these noise and redundant features can be removed, retaining the principal components that are most representative of the data, thereby improving the signal-to-noise ratio of the data.

[0066] In this exemplary embodiment, sparse matrices are used to compress data. Sparse matrix data compression is a special storage method for sparse matrices, which is used to reduce the storage space required for them. Sparse matrices refer to matrices in which the number of non-zero elements is much smaller than the total number of matrix elements, and the distribution of non-zero elements is irregular. Through sparse matrix data compression, storage space can be greatly saved, and the efficiency of processing sparse matrix related calculations can be improved. This can be applied in scenes such as image processing, network analysis, and finite element analysis, because these scenes often need to process large-scale sparse matrices.

[0067] In this exemplary embodiment, the PCA data dimension reduction algorithm and sparse matrix compression are used to fuse the features. The feature map of large target defects and the feature map of small target defects calculated by the twin network are fused through data dimension reduction and sparse matrix compression to improve the model's ability to detect large target defects and small target defects at the same time.

[0068] The step of inputting the fusion features into the YOLO model to perform defect recognition and obtain the types and locations of defects in the PCB image includes:

[0069] Input the fused features into the YOLO model to obtain a prediction box of the fused features;

[0070] The prediction box is screened according to a preset threshold, and the final prediction box is obtained after being processed by the NMS algorithm; wherein the prediction box represents the PCB defect type information and the PCB defect location information. Among them, screening by threshold means screening out defect boxes whose actual prediction confidence score is higher than the set confidence score, and then the screened defect boxes are processed using the NMS algorithm.

[0071] The feature map of the large target defect and the feature map of the small target defect extracted by the twin network are fused and input into the head network of the YOLO model. The YOLO network uses the fused feature map for training and finally obtains the prediction box of the actual defect. The principle of the NMS algorithm is as follows:

[0072] a) Score sorting: First, all detected bounding boxes are sorted based on the confidence score corresponding to each bounding box. The confidence score can be determined based on the probability or score output by the object detection model.

[0073] b) Select the highest scoring bounding box: Select the bounding box with the highest score from the sorted list of bounding boxes and add it to the final output list.

[0074] c) Calculate overlap: For the remaining bounding boxes, calculate their overlap with the currently selected highest scoring bounding box. A common method for calculating overlap is to calculate the intersection area of ​​two bounding boxes divided by their union area.

[0075] d) Eliminate overlapping bounding boxes: For other bounding boxes whose overlap with the currently selected bounding box is higher than a predetermined threshold (e.g., 0.5), remove them from the candidate list. This ensures that each object is represented by only one bounding box, thus avoiding duplicate detection.

[0076] e) Repeat step: Repeat the above steps until all bounding boxes have been processed and checked.

[0077] The present application describes inputting the fusion features into the YOLO model to perform defect recognition to obtain the types and locations of defects in the PCB image, including:

[0078] S1. Input the validation set images into the improved YOLOv8 model to obtain prediction results such as prediction box, prediction type and confidence score. Then analyze the prediction results to evaluate the prediction accuracy of the model. Furthermore, according to the analysis results, adjust the training data in a targeted manner to further improve the model detection capability;

[0079] S2. Compare the detection results with the true labels, calculate the loss value of the improved YOLOv8 model during training, and continuously adjust the model parameters to reduce the training loss of the model until the training is finally completed. If the training loss is not updated within a certain number of iterations, the training task can be terminated in advance, and the model weight file and related training data are generated at the same time, such as the training label classification, the confusion matrix diagram of the verification stage after the training is completed, etc.

[0080] In step S2, the trained detection model is used to predict the test sample set, and the prediction results are obtained and compared with the real labels. The specific process is as follows:

[0081] To verify the recognition effect of the model, preprocess the test set image data, input the trained weight file in a unified format, frame the target recognized by the model on the preprocessed test set image, and obtain the PCB defect image recognition result, which is compared with the actual annotation to verify the detection effect of the model.

[0082] In this exemplary embodiment, taking target detection as an example, many detection frames (A, B, C, D, E, F, etc.) will be generated during the target detection reasoning process, many of which are detecting the same target, but ultimately only one detection frame is needed for each target. NMS selects the detection frame with the highest score (assuming it is C), and then calculates the corresponding IOU value between C and the remaining frames. When the IOU value exceeds the set threshold (generally set to 0.5, often set to 0.7 in target detection, for reference only), the frame exceeding the threshold is suppressed, and the suppression method is to set the score of the detection frame to 0. After one round, continue to search for the highest score in the remaining detection frames, and then suppress the frame with an IOU exceeding the threshold, until almost no overlapping frames are retained. In this way, it is basically possible to have only one detection frame left for each target.

[0083] In this exemplary embodiment, large target defect features and small target defect features have different scale information. Large defects may provide more global shape or texture information, while small defects involve finer local details. Through feature fusion, information of different scales can be captured at the same time, thereby improving the model's ability to recognize defects of various sizes. At the same time, fusing multi-scale features can help the model better cope with the effects of lighting changes, angle changes, and other imaging conditions, making the model more stable and reliable, thereby enhancing robustness.

[0084] In this exemplary embodiment, through feature fusion, low-level features (features extracted by shallow networks) for small targets usually contain more location information, while high-level features (features extracted by deep networks) often carry more semantic information. Combining the two helps to accurately locate the location of the defect, which is conducive to improving positioning accuracy.

[0085] In this exemplary embodiment, by utilizing features at different levels, especially combining the bottom-level small defect features with the high-level large defect features, it is possible to reduce false positives while maintaining a high recall rate, that is, correctly identify as many real defects as possible while trying to avoid incorrectly marking normal areas as defects, which is beneficial to improving the recall rate and reducing false positives.

[0086] In this exemplary embodiment, in some cases, processing large defects and small defects separately may result in a waste of computing resources or inefficiency. Feature fusion allows efficient processing of different types of defects within a unified framework, saving computing resources and increasing processing speed, which is conducive to optimizing resource usage.

[0087] In this exemplary embodiment, there are many types of defects on the PCB, with different sizes. Through feature fusion, the YOLO model can better adapt to these diversities and achieve comprehensive coverage of multiple defect types.

[0088] In some embodiments, after inputting the training fusion features into a YOLO model, performing YOLO model training, and obtaining a YOLO model for PCB image defect recognition and detection, the method includes:

[0089] Add a 1*1 convolution kernel after the C3, C4, and C5 layers of the YOLO model for PCB image defect recognition and detection;

[0090] The YOLO model with a 1*1 convolution kernel is used as the YOLO model for PCB defect detection.

[0091] In this exemplary embodiment, a 1*1 convolution kernel is added after C3, C4, and C5 of the original YOLO model to improve the detection performance of large targets. Finally, the PCB surface defect samples are sent to the improved YOLO network model for training to obtain the optimal model and data for target detection. The model detection effect is evaluated based on the final accuracy, recall rate, and the final MAP value to improve the detection precision and accuracy.

[0092] The present disclosure provides a PCB defect detection device. Figure 2 FIG. 1 is a schematic diagram of a PCB defect detection device according to an exemplary embodiment. Figure 2 As shown, the PCB defect detection device includes:

[0093] An image acquisition module 20 is used to acquire a PCB image to be defect-identified;

[0094] The feature recognition module 21 is used to input the PCB image to be defect-identified into the twin network, identify the large target defect features and small target defect features of the PCB image, and obtain a first feature matrix characterizing the large target defect features and a second feature matrix characterizing the small target defect features; wherein the twin network is pre-trained based on a PCB sample set; wherein the large target defect features and the small target defect features are two types of defect feature types divided according to the defect features of the PCB defects;

[0095] A feature fusion module 22 is used to perform PCA data dimension reduction and sparse matrix processing on a first feature matrix representing large target defect features and a second feature matrix representing small target defect features to obtain fusion features;

[0096] The defect recognition module 23 is used to input the fusion features into the YOLO model to perform defect recognition and obtain the types and locations of defects in the PCB image.

[0097] In this exemplary embodiment, the defects detected by PCB can include 24 types of defects. Among them, the 24 types of defects mainly include 9 types of board surface defects, 6 types of metal surface defects, 5 types of hole defects and 4 types of line defects. Specifically, board surface defects include 9 types of defects, including copper leakage, foreign matter, scratches, unevenness, edge damage, missing printing (yellow), missing printing (black), oil accumulation and ink missing; metal surface defects include 6 types of defects, including foreign matter, scratches, unevenness, oxidation, brush marks and tin oxidation; hole defects include yellow holes, blocked holes, burrs in the holes, plugging holes and poor hole walls, totaling 5 types of defects; line defects include 4 types of defects, including short circuit gaps, line deviations, residual copper and copper wire.

[0098] Among them, the large target defect features may include: foreign matter on the board surface, scratches on the board surface, damage to the board edge, missing printing (yellow), missing printing (black), oil accumulation, foreign matter on the metal surface, scratches on the metal surface, oxidation on the metal surface, oxidation and dirt on the tin surface, yellow holes, blocked holes, sunken holes, poor hole walls, and crooked lines;

[0099] Small target defect features may include: copper leakage on the board surface, board surface bumps, ink missing, metal surface bumps, metal surface brush marks, burrs in the hole, short circuit gaps in the line, residual copper in the line, and copper wire in the line.

[0100] In this exemplary embodiment, the large target defect feature and the small target defect feature both include at least one of the following information corresponding to the PCB defect: color, size, texture, and shape.

[0101] In this exemplary embodiment, by introducing a twin network to replace the original YOLOv8 Backbone, the twin network has strong robustness to the class imbalance problem. This means that even when the number of samples of some categories in the training data is much smaller than that of other categories, the twin network can still effectively learn and identify samples of these minority categories. This feature makes the twin network more flexible and effective in dealing with practical problems.

[0102] In this exemplary embodiment, when the large target defect feature and small target defect feature recognition and extraction of the PCB image are performed by the twin network, two sub-networks with different structures can be used as feature extractors. For example, DenseNet is used as one of the sub-networks, and the convolution operation of its dense block is used to improve the extraction ability of small target features; DeepLabV3 is used as another sub-network, and the ASPP spatial hole convolution module is used to obtain a larger receptive field, thereby improving the extraction ability of large target features. The input data of the twin network is input into the two sub-networks at the same time, and the two sub-networks obtain different feature maps through their respective convolution operations (weights are not shared). Further, the two sub-networks are responsible for extracting large target defect features and small target defect features respectively. After obtaining the different feature maps output by the two sub-networks, data fusion is performed at the final output end, the features extracted from different dimensions are merged, and then PCA is used to reduce the data dimension of the fused features. For example, DeepLabV3 is used as the first sub-network to extract large target defect features and obtain a first feature map containing a first feature matrix; DenseNet is used as the second sub-network to extract small target defect features and obtain a second feature map containing a second feature matrix. Then the first feature map and the second feature map are fused, and finally the fused features are subjected to data dimensionality reduction through PCA.

[0103] According to the embodiment of the present disclosure, the PCB defect detection device is used to obtain a PCB image to be defect-identified; the PCB image to be defect-identified is input into a twin network, and the large target defect features and small target defect features of the PCB image are identified, so as to obtain a first feature matrix characterizing the large target defect features and a second feature matrix characterizing the small target defect features; wherein the twin network is pre-trained based on a PCB sample set; the first feature matrix characterizing the large target defect features and the second feature matrix characterizing the small target defect features are subjected to PCA data dimension reduction and sparse matrix processing to obtain fusion features; the fusion features are input into a YOLO model to perform defect identification, and the types and locations of defects existing in the PCB image are obtained. In the present application, by introducing a twin network to replace the Backbone of the original YOLO model, the twin network has strong robustness to the problem of class imbalance, so that when the number of samples of certain categories in the training data is far less than that of other categories, the twin network can still effectively learn and identify these minority category samples. At the same time, the first feature matrix characterizing the large target defect features and the second feature matrix characterizing the small target defect features are subjected to PCA data dimension reduction and sparse matrix processing to obtain fusion features. Through feature fusion, the YOLO model can capture information of different scales at the same time, thereby improving the model's recognition ability and detection accuracy for defects of various sizes.

[0104] In some embodiments, a model training module is included;

[0105] The PCB image to be defect-identified is input into the twin network to identify the large target defect features and the small target defect features of the PCB image, and before obtaining the first feature matrix representing the large target defect features and the second feature matrix representing the small target defect features, the model training module is used

[0106] Acquire a PCB sample set; wherein the PCB sample set is marked with the type of PCB defects to be identified;

[0107] The PCB sample set marked with the types of PCB defects to be identified is input into the twin network, and the weight parameters of the twin network are trained to obtain the twin network for feature recognition and extraction and the training feature matrix characterizing the large target defect features and the small target defect features.

[0108] In this exemplary embodiment, a high-definition camera is used to take photos of PCB boards actually produced at the customer site. The captured images are then sliced ​​to form a 500*500 local slice image. Professional labelers use the labeling tool Label to label and form a labeling file in XML format, and then use a script tool to convert the XML labeling file into a YOLO labeling file.

[0109] In this exemplary embodiment, the process of dividing the PCB defect data set specifically includes: randomly dividing the prepared PCB surface defect data set into a training set, a validation set, and a test set in a ratio of 8:1:1 according to experimental requirements, and placing the divided data set images and corresponding labels into corresponding folders. Among them, the training set is used to train the twin model and the YOLO model, the test set is used to test the trained twin model and the YOLO model, and the validation set is used to verify the test results of the twin model and the YOLO model.

[0110] For example, according to the classification of large target defect features and small target defect features, the model test results are verified to verify the feature extraction and defect recognition capabilities of the twin model and the YOLO model. Whether the feature extraction results of the twin model and the defect recognition results of the YOLO model are the same as the defect feature classification determined below. Among them, large target defect features may include: foreign matter on the board surface, scratches on the board surface, damage to the board edge, missing print (yellow), missing print (black), oil accumulation, foreign matter on the metal surface, scratches on the metal surface, oxidation on the metal surface, oxidized tin surface, yellow holes, blocked holes, sunken holes, poor hole walls, and crooked lines; small target defect features may include: copper leakage on the board surface, uneven board surface, missing ink, uneven metal surface, brush marks on the metal surface, burrs on the blank in the hole, short circuit gaps on the line, residual copper on the line, and copper wire on the line. In this way, it is beneficial to effectively improve the feature recognition and extraction capabilities of the twin network.

[0111] In some embodiments, before the fusion feature is input into the YOLO model to perform defect recognition and obtain the defect type and location in the PCB image, the model training module is used to

[0112] Perform PCA data dimension reduction and sparse matrix processing on the training feature matrix representing the large target defect features and the small target defect features of the twin network to obtain training fusion features;

[0113] The training fusion features are input into the YOLO model, and the YOLO model is trained to obtain a YOLO model for PCB image defect recognition and detection.

[0114] In this exemplary embodiment, the original high-dimensional space contains redundant information and noise information, which will introduce errors in practical applications and affect the accuracy; data dimensionality reduction can extract the essential structure inside the data, reduce the errors caused by redundant information and noise information, and improve the accuracy in the application.

[0115] In this exemplary embodiment, the specific data dimension reduction is as follows:

[0116] PCA (Principal Component Analysis) is used to find the main change directions (i.e., principal components) in the data, and the data is projected onto these directions, thereby achieving data dimensionality reduction. This allows data originally in a high-dimensional space to be represented in a low-dimensional space, reducing the complexity of the data and helping to understand and analyze the data more intuitively. In addition, there may be some noise or redundant features in the fused feature information, which do not contribute much to the overall structure and information of the data. Through PCA dimensionality reduction, these noise and redundant features can be removed, retaining the principal components that are most representative of the data, thereby improving the signal-to-noise ratio of the data.

[0117] In this exemplary embodiment, sparse matrices are used to compress data. Sparse matrix data compression is a special storage method for sparse matrices, which is used to reduce the storage space required for them. Sparse matrices refer to matrices in which the number of non-zero elements is much smaller than the total number of matrix elements, and the distribution of non-zero elements is irregular. Through sparse matrix data compression, storage space can be greatly saved and the efficiency of processing sparse matrix related calculations can be improved. This is particularly useful in scenes such as image processing, network analysis, and finite element analysis, because these scenes often need to process large-scale sparse matrices.

[0118] In this exemplary embodiment, the feature map of large target defects and the feature map of small target defects extracted by the twin network are input into the head network, and the features of different feature maps are fused to obtain the prediction frame. Among them, the PCA data dimensionality reduction algorithm and sparse matrix compression are used to fuse the features. The feature map of large target defects and the feature map of small target defects calculated by the twin network are fused through data dimensionality reduction and sparse matrix compression (reducing the amount of data calculation and improving), so as to improve the detection ability of the model in processing large target defects and small target defects at the same time.

[0119] In some embodiments, the inputting the training fusion features into the YOLO model, performing YOLO model training, and obtaining the YOLO model for PCB image defect recognition and detection includes:

[0120] Add a 1*1 convolution kernel after the C3, C4, and C5 layers of the YOLO model for PCB image defect recognition and detection;

[0121] The YOLO model with a 1*1 convolution kernel is used as the YOLO model for PCB defect detection.

[0122] In this exemplary embodiment, a 1*1 convolution kernel is added after C3, C4, and C5 of the original YOLO model to improve the detection performance of large targets. Finally, the PCB surface defect samples are sent to the improved YOLO network model for training to obtain the optimal model and data for target detection. The model detection effect is evaluated based on the final accuracy, recall rate, and the final MAP value to improve the detection precision and accuracy.

[0123] The present disclosure provides a computer-readable storage medium on which a PCB defect detection program is stored. When the PCB defect detection program is executed by a processor, the PCB defect detection method described in the above embodiments is implemented.

[0124] The present disclosure provides an electronic device, including a memory, a processor, and a PCB defect detection program stored in the memory and executable on the processor. When the processor executes the PCB defect detection program, the PCB defect detection method described in the above embodiments is implemented.

[0125] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0126] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0128] In the description of the present disclosure, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0129] In addition, the terms "first", "second", etc. used in the embodiments of the present disclosure are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the embodiments. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present disclosure may explicitly or implicitly indicate that at least one of the features is included in the embodiment. In the description of the present disclosure, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0130] In the present disclosure, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. appearing in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to the specific implementation situation.

[0131] In the present disclosure, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0132] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A PCB defect detection method, characterized in that: include: Obtaining a PCB image to be identified by defects; The PCB image to be defect-identified is input into the twin network, and the large target defect features and small target defect features of the PCB image are identified to obtain a first feature matrix characterizing the large target defect features and a second feature matrix characterizing the small target defect features; wherein the twin network is pre-trained based on a PCB sample set; wherein the large target defect features and the small target defect features are two types of defect feature types divided according to the defect features of PCB defects; The first feature matrix representing the large target defect characteristics and the second feature matrix representing the small target defect characteristics are subjected to PCA data dimension reduction and sparse matrix processing to obtain fusion features; The fused features are input into the YOLO model to perform defect recognition, and the types and locations of defects in the PCB image are obtained.

2. The PCB defect detection method according to claim 1, characterized in that: Before inputting the PCB image to be defect-identified into the twin network, identifying large target defect features and small target defect features of the PCB image, and obtaining a first feature matrix characterizing large target defect features and a second feature matrix characterizing small target defect features, the method includes: Acquire a PCB sample set; wherein the PCB sample set is marked with the type of PCB defects to be identified; The PCB sample set marked with the types of PCB defects to be identified is input into the twin network, and the weight parameters of the twin network are trained to obtain the twin network for feature recognition and extraction and the training feature matrix characterizing the large target defect features and the small target defect features.

3. The PCB defect detection method according to claim 1, characterized in that: Before inputting the fusion feature into the YOLO model to perform defect recognition and obtain the defect type and position in the PCB image, the method includes: Perform PCA data dimension reduction and sparse matrix processing on the training feature matrix representing the large target defect features and the small target defect features of the twin network to obtain training fusion features; The training fusion features are input into the YOLO model, and the YOLO model is trained to obtain a YOLO model for PCB image defect recognition and detection.

4. The PCB defect detection method according to claim 1, characterized in that: After inputting the training fusion features into the YOLO model, performing YOLO model training, and obtaining the YOLO model for PCB image defect recognition and detection, the method includes: Add 1*1 convolution kernel after C3, C4 and C5 layers of YOLO model for PCB image defect recognition and detection; The YOLO model with a 1*1 convolution kernel is used as the YOLO model for PCB defect detection.

5. The PCB defect detection method according to claim 1, characterized in that: The large target defect feature and the small target defect feature both include at least one of the following information corresponding to the PCB defect: color, size, texture and shape.

6. A PCB defect detection device, characterized in that: include: An image acquisition module, used to acquire a PCB image to be identified by defects; A feature recognition module is used to input the PCB image to be defect-identified into a twin network, identify large target defect features and small target defect features of the PCB image, and obtain a first feature matrix characterizing large target defect features and a second feature matrix characterizing small target defect features; wherein the twin network is pre-trained based on a PCB sample set; wherein the large target defect features and the small target defect features are two types of defect feature types divided according to the defect features of PCB defects; A feature fusion module is used to perform PCA data dimension reduction and sparse matrix processing on the first feature matrix representing the large target defect characteristics and the second feature matrix representing the small target defect characteristics to obtain fusion features; The defect recognition module is used to input the fusion features into the YOLO model to perform defect recognition and obtain the types and locations of defects in the PCB image.

7. The PCB defect detection device according to claim 6, characterized in that: Includes model training module; The PCB image to be defect-identified is input into the twin network to identify the large target defect features and the small target defect features of the PCB image, and before obtaining the first feature matrix representing the large target defect features and the second feature matrix representing the small target defect features, the model training module is used Acquire a PCB sample set; wherein the PCB sample set is marked with the type of PCB defects to be identified; The PCB sample set marked with the types of PCB defects to be identified is input into the twin network, and the weight parameters of the twin network are trained to obtain the twin network for feature recognition and extraction and the training feature matrix characterizing the large target defect features and the small target defect features.

8. The PCB defect detection device according to claim 7, characterized in that: The fusion feature is input into the YOLO model to perform defect recognition and obtain the defect type and location in the PCB image. The model training module is used to Perform PCA data dimension reduction and sparse matrix processing on the training feature matrix representing the large target defect features and the small target defect features of the twin network to obtain training fusion features; The training fusion features are input into the YOLO model, and the YOLO model is trained to obtain a YOLO model for PCB image defect recognition and detection.

9. A computer-readable storage medium, characterized in that: A PCB defect detection program is stored thereon, and when the PCB defect detection program is executed by the processor, the PCB defect detection method described in any one of claims 1 to 5 is implemented.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and a PCB defect detection program stored in the memory and executable on the processor. When the processor executes the PCB defect detection program, the PCB defect detection method according to any one of claims 1 to 5 is implemented.

Images