PCB surface defect detection method and device based on sample and feature equalization

Abstract

Embodiments of the present disclosure disclose a PCB surface defect detection method and device based on sample and feature equalization. In view of the feature level imbalance problem caused by the feature semantic loss of the feature map generated by the PCB and the sample level imbalance problem caused by the low utilization of difficult negative samples in the PCB dataset and the difference in defect scale, a sample and feature equalization defect detection algorithm is proposed. Feature balancing processing is used. The feature balancing processing algorithm is added on the basis of FPN. Not only can the feature semantics of deep and shallow layers be fused, but also the loss of non-adjacent layer feature semantics in the fusion process can be reduced, thereby alleviating the imbalance problem of the feature level. The GA-RPN of the adaptive anchor box generation can make the generated anchor box better focus on the defect, enhance the detection effect of small scale defects, introduce IoU balanced sampling, increase the utilization of difficult negative samples, and effectively alleviate the imbalance problem of the sample level.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of machine vision technology, and particularly relates to a PCB surface defect detection method and device based on sample and feature equalization. BACKGROUND

[0002] In the development of modern electronic information industry, the printed circuit board (PCB) has become an irreplaceable basic component, which has the important functions of connecting, integrating and controlling various electronic components, and has a lasting influence in the production and use of various electronic information products. For example, the smart phones and portable tablet computers used in our daily life cannot be separated from the use of PCBs. Due to the high integration of PCBs, any problem in any link of PCBs will affect the use of PCBs, so the defect detection of PCBs is particularly important.

[0003] At present, there are six types of defects in PCBs: short circuit, open circuit, mouse bite, missing hole, burr and false copper. PCB defect detection can be roughly divided into two categories: traditional manual PCB defect detection and PCB defect detection based on deep learning. The traditional PCB defect detection is detected by manual, which needs to consume a lot of manpower and financial resources, and is prone to miss and wrong detection, which makes the detection efficiency low. For the enterprises producing PCBs, it is a waste of manpower and financial resources. The non-manual detection methods mainly include machine vision and deep learning. Among them, the machine vision based method has greatly improved compared with the manual method, which saves a lot of time cost, but still has some problems such as difficulty in overcoming the interference in the image, inability to adapt to multi-scale defects, and inability to well distinguish the defects with high similarity to the background. The development of PCB defect detection based on deep learning is mainly through training neural network to detect defects. Compared with other detection methods, the deep learning based method is more efficient and simple, and is suitable for various industrial production fields.

[0004] In recent years, the development of neural networks is extremely rapid, which can quickly and efficiently perform image classification, target detection, semantic segmentation and other tasks. In the aspect of defect detection, neural networks are also widely used, such as steel defect detection, PCB defect detection and the like. However, due to different objects, different defects of the same object, and various sizes and types of the same defect, many imbalance phenomena will occur in the process of defect detection by the neural network. When facing multi-scale features, the feature map will be divided into multiple scales, the deep features will contain more semantic information, and the shallow features will have more sample content. If the deep and shallow feature information cannot be well fused, the imbalance problem at the feature level will be caused, and the fixed anchor frame is difficult to conform to all scale defects in defect detection, which will cause the imbalance between samples and samples; since random sampling mainly samples simple samples, a large number of difficult negative samples will be missed, so that the semantic information of these samples is lost, which will cause the imbalance at the sample level. SUMMARY

[0005] Therefore, the embodiment of the present disclosure provides a PCB surface defect detection method and device based on sample and feature equalization, which can solve the problems of feature level imbalance and sample sampling imbalance in the prior art.

[0006] In a first aspect, the embodiment of the present disclosure provides a PCB surface defect detection method based on sample and feature equalization, which adopts the following technical scheme: a PCB surface defect detection method based on sample and feature equalization, comprising: step 1, establishing a PCB defect image dataset; wherein the dataset includes PCB finished product images and their annotation information; step 2, extracting multi-scale feature maps of the PCB finished product images, and sequentially performing primary equalization processing and secondary equalization processing on the multi-scale feature maps to obtain equalized multi-scale feature maps; step 3, using a GA-RPN network to predict anchor frames of the equalized multi-scale feature maps to determine positive samples and negative samples therein; performing IoU balanced sampling on the negative samples to obtain equalized negative samples; step 4, performing class prediction on the positive samples and the equalized negative samples to generate a prediction result; and step 5, performing model training based on steps 2-4, and using the trained model to perform PCB surface defect detection.

[0007] Further, the primary equalization processing comprises: converting the input multi-scale feature maps into the same scale, and performing averaging processing on the feature maps converted into the same scale to obtain balanced semantic features; respectively convolving and strengthening the feature maps converted into the same scale with the balanced semantic features to obtain feature maps after feature strengthening; and restoring the obtained feature maps after feature strengthening to the original scale for output.

[0008] Further, the secondary equalization processing includes: using an FPN network to perform feature fusion from deep to shallow on the input multi-scale feature map to obtain the equalized multi-scale feature map.

[0009] Further, the using the FPN network to perform feature fusion from deep to shallow on the input multi-scale feature map to obtain the equalized multi-scale feature map includes: up-sampling the input multi-scale feature map to make it have the same scale as the feature map of the previous layer, and then performing feature fusion on the two by element addition to obtain the secondary equalized multi-scale feature map; wherein each fused feature map is processed using a 3x3 convolution.

[0010] Further, the IoU balanced sampling includes: arranging all negative samples according to the IoU size and uniformly dividing them into several regions, and then randomly extracting the same number of negative samples from each region.

[0011] Further, the GA-RPN network includes: a position prediction branch, a shape prediction branch, and a feature adaptive module.

[0012] Further, the using the GA-RPN network to perform anchor box prediction on the equalized multi-scale feature map includes: according to the input feature map, respectively passing through the position prediction branch and the shape prediction branch to predict the center position and the anchor box shape of the anchor box, and then adjusting the generated anchor box through the feature adaptive module.

[0013] Further, the annotation information at least includes: an annotation box and a defect category.

[0014] Further, the defect category is at least one of short circuit, open circuit, mouse bite, missing hole, burr, and false copper.

[0015] Further, the PCB surface defect detection method based on sample and feature equalization further includes: dividing the PCB defect image dataset into a training set, a validation set, and a test set.

[0016] Further, the model training based on steps 2-4 includes: using the training set, the validation set, and the test set to sequentially perform model training, model adjustment, and model performance evaluation based on steps 2-4.

[0017] Further, the dividing the PCB defect image dataset into a training set, a validation set, and a test set includes: dividing the PCB defect image dataset into a training set, a validation set, and a test set according to a ratio of 8:1:1.

[0018] In a second aspect, the embodiments of the present disclosure further provide a PCB surface defect detection device based on sample and feature equalization, which adopts the following technical scheme: a PCB surface defect detection device based on sample and feature equalization comprises: a collection module, configured to establish a PCB defect image dataset; wherein the dataset comprises PCB finished product images and their label information; a feature equalization extraction module, configured to extract multi-scale feature maps of the PCB finished product images, and sequentially perform primary equalization processing and secondary equalization processing on the multi-scale feature maps to obtain equalized multi-scale feature maps; a sample equalization extraction module, configured to use a GA-RPN network to perform anchor box prediction on the equalized multi-scale feature maps to determine positive samples and negative samples therein; perform IoU balanced sampling on the negative samples to obtain equalized negative samples; and a detection module, configured to perform category prediction on the positive samples and the equalized negative samples to generate a prediction result; wherein model training is performed based on the feature equalization extraction module, the sample equalization extraction module and the detection module, and the trained model is used for PCB surface defect detection.

[0019] The PCB surface defect detection method and device based on sample and feature equalization provided by the embodiments of the present disclosure can reduce the unbalanced influence caused by forgetting of non-neighbor features, and solve the problem of incomplete utilization of deep features and shallow features; meanwhile, the use of the GA-RPN can make more anchor boxes generated around target objects, ensure that more semantic information can be provided regardless of the size of the object, increase the utilization rate of different samples, and save the time consumption caused by anchor box adjustment, and further improve the detection speed. In addition, the use of IoU balanced sampling can improve the value of difficult negative samples during training, and reduce the sample imbalance problem caused by sample information omission.

[0020] The above description is only a summary of the technical solutions of the present disclosure, in order to more clearly understand the technical means of the present disclosure, the contents of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings according to these drawings without creating any creative labor.

[0022] Figure 1A Faster R-CNN network structure diagram in the prior art;

[0023] Figure 2 A Faster R-CNN network structure example based on sample and feature equalization provided by the embodiment of the present disclosure;

[0024] Figure 3 A flowchart of a PCB surface defect detection method based on sample and feature equalization provided by the embodiment of the present disclosure;

[0025] Figure 4 A flowchart of feature map equalization processing provided by the embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of feature map equalization processing provided by the embodiment of the present disclosure;

[0027] Figure 6 A GA-RPN network structure diagram provided by the embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of a prediction result obtained by a PCB surface defect detection method based on sample and feature equalization provided by the embodiment of the present disclosure;

[0029] Figure 8 A principle block diagram of a PCB surface defect detection device based on sample and feature equalization provided by the embodiment of the present disclosure;

[0030] Figure 9 A principle block diagram of an electronic device provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0032] It should be apparent that the following describes the embodiments of the present disclosure through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, not all. The present disclosure can also be implemented or applied by other different specific embodiments, and each detail in the specification can be modified or changed based on different views and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0033] It is to be appreciated that various aspects described herein are described in the context of examples that are within the scope of the appended claims. It is to be understood that the aspects described herein can be implemented in a wide variety of forms and that any particular structure and / or function described herein is merely illustrative. Based on the teachings provided herein one skilled in the art will appreciate that one or more aspects described herein can be implemented independently of any other aspects described herein. The

[0034] It is also to be understood that the drawings shown in the following embodiments are only to schematically illustrate the basic concept of the present disclosure, and only the components related to the present disclosure are shown in the drawings, not drawn according to the number, shape and size of the components when actually implemented, and the shape, number and ratio of each component when actually implemented can be a random change, and the layout of the components can be more complex.

[0035] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, one skilled in the art will understand that the aspects described can be practiced without these specific details.

[0036] Figure 1 A conventional Faster R-CNN network structure is shown; as shown in Figure 1 The conventional Faster R-CNN network first extracts a feature map in the original image by using a set of basic convolution layers (Conv layer), which is shared for subsequent RPN layer (Region Proposal Network) and classification layer. The RPN is used to generate anchor boxes according to the input feature map, judge whether the anchor box belongs to positive or negative sample, and then use bounding box regression algorithm to correct the anchor box position, but due to the huge number of negative samples in the generated anchor box, simple random sampling is generally used for negative samples as candidate regions (region proposals), so as to avoid the influence of a large number of negative samples on the performance of the network and prevent overfitting phenomenon. The region of interest pooling (Roi Pooling) is used to collect the feature map and the candidate region, and the information of the two is integrated to generate the candidate feature map (proposal feature maps); the classification layer is used to calculate the class of the candidate region (i.e. the target class in the candidate box) according to the input candidate feature map, so as to obtain the prediction result of the original image.

[0037] In the traditional Faster R-CNN, although the random sampling method is used to effectively reduce the number of negative samples, the negative samples are mainly sampled randomly, which may cause a large number of difficult negative samples to be missed, and the semantic information of these samples is lost, thereby causing the imbalance of the samples.

[0038] On the other hand, the traditional Faster R-CNN does not fully utilize the multi-scale feature information, resulting in poor robustness. Mainly because the deep features in the multi-scale feature map contain more semantic information, and the shallow features contain more sample content. If the deep and shallow feature information cannot be well integrated, the imbalance of the feature level will be caused.

[0039] Therefore, the present disclosure proposes an improved Faster R-CNN network based on sample and feature balancing to solve the above problems. Specifically, as shown in Figure 2 , a Faster R-CNN network based on sample and feature balancing is shown. Figure 2 The network structure of the Faster R-CNN network based on sample and feature balancing in the embodiment of the present disclosure is shown. The network structure includes ResNet50, a feature balancing module, an FPN network, a GA-RPN module, an IoU balanced sampling module, an ROI pooling, and a Classifier. The ResNet50 is used to extract a multi-scale feature map of an input image. The feature balancing module is used to perform semantic enhancement on the extracted multi-scale feature map. The FPN network is used to fuse the multi-scale feature map after semantic enhancement. The GA-RPN module is used to predict anchor samples. The IoU balanced sampling module is used to balance the sampling of anchor negative samples. The ROI pooling and the Classifier are used to predict the categories of the input samples and output the prediction results.

[0040] The specific processing procedure in the improved detection model is described in detail in the PCB surface defect detection method based on sample and feature balancing in the embodiment of the present disclosure.

[0041] The embodiment of the present disclosure also proposes a PCB surface defect detection method based on sample and feature balancing to solve the problems of feature imbalance and sample imbalance in the Faster R-CNN network for target detection in the prior art. Specifically, as shown in Figure 3 , a PCB surface defect detection method based on sample and feature balancing is shown. Figure 3 The flowchart of the PCB surface defect detection method based on sample and feature balancing in the embodiment of the present disclosure is shown. The PCB surface defect detection method includes:

[0042] Step S1, a PCB defect image dataset is established; wherein the dataset includes PCB finished product images and their annotation information;

[0043] Step S2, multi-scale feature maps of the PCB finished product images are extracted, and one-time equalization processing and secondary equalization processing are sequentially performed thereon to obtain equalized multi-scale feature maps;

[0044] Step S3, anchor box prediction is performed on the equalized multi-scale feature maps using a GA-RPN network to determine positive samples and negative samples therefrom; IoU balanced sampling is performed on the negative samples to obtain equalized negative samples;

[0045] Step S4, class prediction is performed on the positive samples and the equalized negative samples to generate a prediction result;

[0046] Step S5, model training is performed based on steps S2-S4, and the trained model is used for PCB surface defect detection.

[0047] The PCB surface defect detection method based on sample and feature equalization provided by the embodiments of the present disclosure can reduce the unbalanced influence caused by the forgetting of non-neighbor features by sequentially performing one-time equalization processing and secondary equalization processing on the extracted multi-scale feature maps, and can also solve the problem of incomplete utilization of deep features and shallow features; at the same time, the use of GA-RPN can make more anchor boxes generated around the target object, ensuring that more semantic information can be provided regardless of the size of the object, increasing the utilization rate of different samples, and because the size of the anchor box is basically matched with the size of the target object to be detected, the time consumption caused by anchor box adjustment can also be saved, further improving the detection speed. In addition, the use of IoU balanced sampling can improve the value of difficult negative samples during training and reduce the sample imbalance problem caused by sample information omission.

[0048] Further, step S1 in the embodiments of the present disclosure for establishing a PCB defect image dataset can include:

[0049] Step S11, batch acquisition of PCB finished product images

[0050] Wherein, step S11 is not limited to acquiring PCB finished product images produced in the assembly line, or directly extracting existing PCB finished product images, etc.

[0051] Step S12, classifying the acquired PCB finished product images according to defect categories;

[0052] Wherein, the defect categories can be at least one of short circuit, open circuit, mouse bite, missing hole, burr, and false copper.

[0053] Step S13, the defects in the PCB finished product image are labeled to generate label information and establish a PCB defect image dataset.

[0054] The label information includes a label box (i.e., a position of a labeled defect) and a defect category, and is generally set in an XML format or a TXT format label file.

[0055] In some embodiments of the present disclosure, step S12 can be omitted, and the labeling process is directly performed in step S13.

[0056] Further, as shown in FIG. 2, the multi-scale feature map of the PCB finished product image extracted in step S2 in the embodiments of the present disclosure can include: Figure 4

[0057] Step S21, generating a multi-scale feature map according to the input PCB finished product image;

[0058] The multi-scale feature map refers to a plurality of feature maps of different scales converted from the PCB finished product image, which can also be understood as a plurality of feature maps with different resolutions. Generally, the multi-scale feature map can be generated from shallow to deep through downsampling or other scaling methods, and the multi-scale feature map is generally obtained through equal scaling. Figure 1

[0059] Further, the one-time equalization processing in step S2 in the embodiments of the present disclosure can include:

[0060] Step S22, converting the input multi-scale feature map to the same scale, and performing averaging processing on the feature map converted to the same scale to obtain an equalized semantic feature;

[0061] The averaging processing can be performed according to the following formula:

[0062]

[0063] Wherein, C is the equalized semantic feature, L is the total number of layers of all feature maps, C l is the feature map of the l th layer, l is the feature map of the deepest layer, and l is the feature map of the shallowest layer. l min max

[0064] In this embodiment, by converting the multi-scale feature map to the same scale, the semantic information contained therein can be avoided from being damaged in the integration process.

[0065] Step S23, respectively convolving and strengthening the feature map converted to the same scale with the equalized semantic feature to obtain a feature map after feature strengthening;

[0066] ​​​​​Step S24: Then restore the obtained feature map after feature enhancement to its original scale and output it.

[0067] Through the above steps S22-S24, the advantage of retaining the original feature information while obtaining balanced feature semantics is achieved.

[0068] Furthermore, the secondary equalization process in step S2 of this embodiment may include: step S25, using an FPN network to perform deep-to-shallow feature fusion on the input multi-scale feature map to obtain the equalized multi-scale feature map.

[0069] Specifically, the process of using an FPN network to perform deep-to-shallow feature fusion on the input multi-scale feature map to obtain the equalized multi-scale feature map can be as follows: the input multi-scale feature map is upsampled to make it the same scale as the feature map of the previous layer (feature fusion is performed from deep to shallow, so the previous layer generally refers to the shallow layer adjacent to the upsampled feature map), and then the two are fused by element-wise addition to obtain the secondary equalized multi-scale feature map; in order to prevent aliasing during the upsampling process, a 3×3 convolution can be used on each fused feature map.

[0070] The deepest feature map does not require feature fusion processing and is directly used as the processed feature map, while the feature maps that are not the deepest layer need to be fused with the previous layer through upsampling and element-wise addition.

[0071] The upsampling factor is the same as the downsampling or scaling factor in step S21. For example, if 2x downsampling is used in step S21, then 2x upsampling is used in step S25.

[0072] For example, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the multi-scale feature map equalization process in the embodiments of this disclosure, namely the process of steps S21-S25.

[0073] First, a multi-scale feature map is obtained by downsampling the input PCB image by a factor of 2. Then, the multi-scale feature map is processed into layers from shallow to deep. The number of layers in the multi-scale feature map is denoted as L, and the feature at layer L is denoted as C. l The feature map with the smallest resolution in the deepest layer is denoted as l. min The feature map with the highest resolution in the shallowest layer is denoted as l. max ,exist Figure 6 In the given example, l max For C5, l min C2;

[0074] Next, the multi-level feature semantics will be integrated, and the feature maps of different levels {C2, C3, C4, C5} will be unified. That is, the feature maps of different levels will be adjusted to the resolution of C3 (or other resolutions can be adjusted according to needs), thereby converting the multi-scale feature maps to the same scale. Then, through averaging, balanced semantic features will be obtained.

[0075] After obtaining balanced semantic features, in order to make the original features clearer and more recognizable, these features need to be refined. Therefore, the feature maps converted to the same scale are convolved with the balanced semantic features to enhance them. Finally, these feature maps at the same scale are restored to their original scale. In this way, the original feature information is preserved while obtaining balanced semantic features. Finally, the new feature maps {P2, P3, P4, P5} are obtained.

[0076] Next, a Feature Map Pyramid Network (FPN) is used for feature fusion. The deeper, more semantically rich feature maps are upsampled by a factor of 2, resulting in feature maps with the same scale as the previous layer. These two feature maps of the same scale are then fused element-wise, achieving a top-down lateral connection and completing the feature fusion. To prevent aliasing during upsampling, each fused feature map is processed using a 3×3 convolution, ultimately yielding new feature maps denoted as {Q2, Q3, Q4, Q5}.

[0077] Furthermore, such as Figure 6 As shown, in this embodiment of the present disclosure, the GA-RPN network consists of a position prediction branch, a shape prediction branch, and a feature adaptation module. Step S3 uses the GA-RPN network to predict anchor boxes on the equalized multi-scale feature map, which may include: predicting the center position and shape of the anchor box according to the input feature map through the position prediction branch and the shape prediction branch respectively, and then adjusting the generated anchor box through the feature adaptation module.

[0078] The main function of the GA-RPN network structure is to predict the confidence level that the receptive field corresponding to each pixel in the feature map is a target, as well as the corresponding width and height, through the position prediction branch and shape prediction branch, respectively. A target is considered a target if its confidence level is greater than a specific threshold. Generally, rectangular anchor boxes are used to approximate the position and shape of the target in the original image. That is, the role of GA-RPN is to generate corresponding anchor boxes at locations where the target may appear, and the shape of the anchor boxes approximates the bounding rectangle of the target.

[0079] Specifically,

[0080] The position prediction branch first performs single-channel 1x1 convolution on the input feature image, and then uses a sigmoid function to map the result of the 1x1 convolution to [0, 1] pixel by pixel to obtain the probability of the existence of a target in each position of the feature map, thereby forming a position probability matrix; the positions with probabilities higher than a threshold value in the obtained position probability matrix are selected as the center positions of the anchor boxes (i.e., the positions where the target objects may exist).

[0081] After the target position prediction is completed, shape prediction is performed, which is realized by the shape prediction branch, and the shape of the target is represented by width w and height h. The task of shape prediction is equivalent to the prediction of w and h. However, there are cases where the sizes of targets in an image differ greatly. Experience shows that directly predicting w and h may cause instability of the network. Therefore, the shape prediction network first generates a dw and dh double-channel prediction map through a two-channel 1x1 convolution layer, and then outputs the prediction of w and h through pixel-by-pixel conversion according to w = σ·s·e dw and h = σ·s·e dh . The σ is an empirical scale factor, which can be set according to the characteristics of the data set and experience when the network is constructed. The s is the step length when the convolution slides. The values of dw and dh range from -1 to 1.

[0082] However, the widths and heights of the anchor boxes predicted by different positions differ, resulting in inconsistent receptive field sizes of the anchor boxes obtained using this method. In order to eliminate this inconsistency, a feature adaptation module N T is used to adjust the generated anchor boxes. A new feature map F i ' is obtained through deformable convolution of the original feature map F i . The feature adaptation module is a small convolutional neural network composed of a 3x3 deformable convolution. This process can be represented as F i ' = N T (fi,wi,hi), where NT represents the feature adaptation module, fi is the feature of the i-th position, and (wi, hi) is the shape of the anchor box corresponding to the i-th position. First, the output of the shape prediction branch is subjected to 1x1 convolution to obtain the shape offset, and then 3x3 deformable convolution is applied to the original feature map with the offset to obtain F i ', i.e., the adjusted feature. After that, the features in the adjusted anchor box region can be classified and the width and height of the anchor box can be regressed.

[0083] Further, the determination of the positive samples and the negative samples in step S3 in the embodiments of the present disclosure can include:

[0084] The IoU (intersection over union) between the anchor boxes predicted by the GA-RPN network and the labeled boxes is calculated, and the positive samples and the negative samples are determined according to the IoU.

[0085] Further, the IoU balanced sampling of the negative samples in step S3 in the embodiments of the present disclosure can include:

[0086] The negative samples are sequentially arranged according to IoU, and then the negative samples are evenly divided into several regions, and then the same number of negative samples are randomly selected from each region as the negative samples to be trained, i.e. balanced negative samples.

[0087] Through this sampling method, the probability of obtaining samples is:

[0088]

[0089] In the formula, Y k is the number of samples to be sampled in the interval k, which can improve the utilization of difficult negative samples.

[0090] Further, in the embodiments of the present disclosure, step S4 is performed to predict the categories of the positive samples and the balanced negative samples to generate a prediction result. The prediction result can be generated by the Classifier in the Faster R-CNN network, and the prediction result generally includes a predicted frame (i.e. anchor frame center position and height and width), a defect category, and a category confidence. The prediction result can be as shown in Figure 7 .

[0091] Further, the PCB surface defect detection method based on sample and feature balancing in the embodiments of the present disclosure can further include: dividing the PCB defect image dataset into a training set, a validation set, and a test set.

[0092] The model training based on steps S2-S4 can include: sequentially training the model, adjusting the model, and evaluating the performance of the model based on steps S2-S4 using the training set, the validation set, and the test set.

[0093] Preferably, the PCB defect image dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0094] For example, the dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1, and a Faster R-CNN network framework with a ResNet-50 backbone is used to train the PCB dataset by combining the improved method. Then, the evaluation index of the model is obtained by using the obtained model and the test set.

[0095] For PCB image defect detection, the main evaluation metrics used are AP, AP50, AP75, APS, APM, and APL. AP represents the average accuracy across all categories; AP50 and AP75 represent the accuracy of AP at IoU thresholds of 50% and 75%, respectively; and APS, APM, and APL correspond to the accuracy at small, medium, and large scales, respectively. Table 1 shows that the proposed method achieves an AP of 95.64%, which is 8.18% higher than Faster R-CNN, 4.25% higher than Mask R-CNN, 1.96% higher than the first-stage SSD, and 1.37% higher than YOLOv2. The proposed method achieves good results for PCB defect detection. After addressing sample imbalance, the detection results for small-scale target defects are also significantly improved, increasing by 8.99% compared to the original method.

[0096] Table 1 Comparative Tests of PCB Defect Detection

[0097]

[0098] This disclosure also provides a PCB surface defect detection device based on sample and feature equalization, specifically, as shown in the embodiments. Figure 7 As shown, Figure 5 This is a schematic diagram of the PCB surface defect detection device based on sample and feature equalization in this embodiment of the present disclosure. The PCB surface defect detection device 100 based on sample and feature equalization includes: an acquisition module 101 for establishing a PCB defect image dataset; wherein the dataset includes PCB finished product images and their annotation information; a feature equalization extraction module 102 for extracting multi-scale feature maps from PCB finished product images and performing first equalization processing and second equalization processing on them sequentially to obtain equalized multi-scale feature maps; an anchor box prediction module 103 for using a GA-RPN network to predict anchor boxes on the equalized multi-scale feature maps to determine positive and negative samples; an IoU balancing sampling module 103 for performing IoU balancing sampling on the negative samples to obtain equalized negative samples; and a detection module 104 for performing class prediction on the positive samples and the equalized negative samples to generate prediction results; wherein, a model is trained based on the feature equalization extraction module, the sample equalization extraction module, and the detection module, and the trained model is used to detect PCB surface defects.

[0099] Furthermore, the detection module 104 in this embodiment can be implemented using the Classifier from the Faster R-CNN network.

[0100] An electronic device according to an embodiment of the disclosure includes a memory and a processor. The memory is configured to store non-transitory computer-readable instructions. Specifically, the memory can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk, flash memory, and / or the like.

[0101] The processor can be a central processing unit (CPU) or other form of processing unit that has data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions. In an embodiment of the disclosure, the processor is configured to execute the computer-readable instructions stored in the memory, so that the electronic device performs all or part of the steps of the PCB surface defect detection method based on sample and feature equalization according to the aforementioned embodiments of the disclosure.

[0102] Those skilled in the art will understand that, in order to solve the technical problem of how to obtain a good user experience effect, the embodiment can also include well-known structures such as a communication bus, an interface, and the like, which should also be included in the protection scope of the disclosure.

[0103] As Figure 9 A structural schematic diagram of an electronic device according to an embodiment of the disclosure is provided. It shows a structural schematic diagram suitable for implementing the electronic device in the embodiment of the disclosure. Figure 9 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiment of the disclosure.

[0104] As Figure 9 As shown, the electronic device can include a processing device (such as a central processor, a graphics processor, and the like), which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) or loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0105] Generally, the following devices can be connected to the I / O interface: input devices including, for example, sensors or visual information acquisition devices; output devices including, for example, display screens; storage devices including, for example, magnetic tapes, hard disks, and the like; and communication devices. The communication devices can allow the electronic device to communicate with other devices (such as edge computing devices) to exchange data wirelessly or through wires. Although Figure 9Electronic devices with various apparatuses are shown, but it is understood that not all of the illustrated apparatuses are required or present. More or fewer apparatuses can alternatively be implemented or present.

[0106] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication apparatus, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing apparatus, all or part of the steps of the sample and feature equalization based PCB surface defect detection method of embodiments of the present disclosure are performed.

[0107] Detailed descriptions of the present embodiments can refer to the corresponding descriptions of the previous embodiments, which will not be repeated here.

[0108] A computer-readable storage medium according to embodiments of the present disclosure has non-transitory computer-readable instructions stored thereon. When the non-transitory computer-readable instructions are run by a processor, all or part of the steps of the sample and feature equalization based PCB surface defect detection method of embodiments of the present disclosure described above are performed.

[0109] The computer-readable storage medium described above includes, but is not limited to, optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or a removable hard disk), media with built-in rewritable non-volatile memory (e.g., a memory card), and media with built-in ROM (e.g., a ROM cartridge).

[0110] Detailed descriptions of the present embodiments can refer to the corresponding descriptions of the previous embodiments, which will not be repeated here.

[0111] The basic principles of the present disclosure are described above in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and are not limiting, and these advantages, benefits, effects, etc. cannot be considered as mandatory for each embodiment of the present disclosure. In addition, the above specific details of the disclosure are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present disclosure to the above specific details.

[0112] In this disclosure, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The block diagram of the devices, apparatus, equipment, systems referred to in this disclosure is merely illustrative and not intended to imply the necessity or arrangement of the connections, arrangement, configuration as shown in the block diagram. As will be appreciated by those skilled in the art, the devices, apparatus, equipment, systems can be connected, arranged, configured in any way. The words "comprising," "containing," "including," "having," and the like, are to be construed open-ended, meaning "including but not limited to," and are to be taken in their broadest context. The words "or" and "and" as used herein, mean "and / or," and are to be taken in their broadest context, unless the context clearly indicates otherwise. The word "comprising" as used herein, means "comprising but not limited to," and is to be taken in its broadest context.

[0113] Also, as used in this disclosure, "or" as used in the context of items A and B as "at least one of A or B" indicates a disjunction, such that, for example, a disjunction of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "example" is not meant to convey that a described example is preferred or better than other examples.

[0114] It is also important to note that the systems and methods of the disclosure can be embodied in a variety of forms including, but not limited to, a data processor, a computer program product, a computer, one or more components of a computer, software, and combinations of the same. In this disclosure, the term "data processor" means any processor containing one or more processors that retrieves instructions and data from, and writes information to, a memory, whether local or remote. Also, it is understood that the systems and methods of the disclosure can be located on one computer, or can be distributed among several computers or components thereof.

[0115] Various changes, modifications and improvements in the herein described technologies can be made within the teachings of the technology, as defined by the appended claims, without departing from the technical teachings of the disclosure. Further, the aspects of the claims to the disclosure are not limited to the specific aspects described hereinabove. A person of ordinary skill in the art will readily recognize that the technology can be practiced with a variety of processing, machine, manufacture, event of constituent, means, methods and actions in addition to, or in place of, the ones described herein. Thus, the appended claims encompass within their scope the various alternatives and modifications of the technology.

[0116] The above description of the disclosed aspects is intended to be illustrative and not restrictive. Many variations of the technology described herein will be apparent to those of ordinary skill in the art upon reviewing the above description. The scope of the disclosure should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. The disclosures of the publications referred to herein are incorporated by reference herein in their entirety.

[0117] The foregoing description has been presented for the purposes of illustration and description. Furthermore, the description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although the various example aspects and embodiments have been described herein with regard to particular aspects and embodiments, those skilled in the art will recognize that certain modifications, changes, substitutions, additions and sub-combinations can be made without departing from the spirit of the disclosure.

Claims

1. A PCB surface defect detection method based on sample and feature equalization, characterized in that, include: Step 1: Establish a PCB defect image dataset; this dataset includes images of finished PCB products and their annotation information; Step 2: Extract the multi-scale feature map of the finished PCB image, and perform first equalization and second equalization processing on it to obtain the equalized multi-scale feature map. Step 3: Use the GA-RPN network to predict anchor boxes on the equalized multi-scale feature map to determine the positive and negative samples; perform IoU balancing sampling on the negative samples to obtain equalized negative samples. Step 4: Perform class prediction on the positive samples and the equalized negative samples to generate prediction results; Step 5: Train the model based on steps 2-4, and use the trained model to detect defects on the PCB surface. The first equalization process includes: The input multi-scale feature maps are converted to the same scale, and the converted feature maps are averaged to obtain balanced semantic features. The feature maps converted to the same scale are then convolved with the balanced semantic features to enhance them, resulting in feature maps after feature enhancement. The feature map obtained after feature enhancement is then restored to its original scale and output. The secondary equalization process includes: The input multi-scale feature map is fused from deep to shallow using an FPN network to obtain the equalized multi-scale feature map.

2. The PCB surface defect detection method based on sample and feature equalization according to claim 1, characterized in that, The process of using an FPN network to perform deep-to-shallow feature fusion on the input multi-scale feature map to obtain the equalized multi-scale feature map includes: The input multi-scale feature map is upsampled to be at the same scale as the feature map of the previous layer, and then the two are fused by element-wise addition to obtain the secondary balanced multi-scale feature map; wherein, a 3×3 convolution is used for each fused feature map.

3. The PCB surface defect detection method based on sample and feature equalization according to claim 1, characterized in that, The IoU balanced sampling includes: All negative samples are arranged according to their IoU and evenly divided into several regions. Then, the same number of negative samples are randomly drawn from each region.

4. The PCB surface defect detection method based on sample and feature equalization according to claim 1, characterized in that, The GA-RPN network consists of a location prediction branch, a shape prediction branch, and a feature adaptation module. The step of using the GA-RPN network to predict anchor boxes on the equalized multi-scale feature map includes: Based on the input feature map, the center position and shape of the anchor frame are predicted through the position prediction branch and the shape prediction branch, respectively. Then, the generated anchor frame is adjusted by the feature adaptation module.

5. The PCB surface defect detection method based on sample and feature equalization according to claim 1, characterized in that, The annotation information includes at least: the annotation box and the defect category.

6. The PCB surface defect detection method based on sample and feature equalization according to claim 1, characterized in that, Also includes: The PCB defect image dataset is divided into a training set, a validation set, and a test set. The model training based on steps 2-4 includes: The training set, validation set, and test set are used to perform model training, model tuning, and model performance evaluation in sequence based on steps 2-4.

7. The PCB surface defect detection method based on sample and feature equalization according to claim 6, characterized in that, The step of dividing the PCB defect image dataset into a training set, a validation set, and a test set includes: The PCB defect image dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

8. A PCB surface defect detection device based on sample and feature equalization, characterized in that, include: The acquisition module is used to create a PCB defect image dataset, which includes images of finished PCB products and their annotation information. The feature equalization extraction module is used to extract multi-scale feature maps from the finished PCB image and perform first and second equalization processing on them to obtain equalized multi-scale feature maps. The sample equalization extraction module is used to predict anchor boxes on the equalized multi-scale feature map using the GA-RPN network to determine positive and negative samples; and to perform IoU balancing sampling on the negative samples to obtain equalized negative samples. The detection module is used to predict the category of the positive samples and the equalized negative samples, and generate prediction results; Among them, the model is trained based on the feature equalization extraction module, the sample equalization extraction module and the detection module, and the trained model is used to detect PCB surface defects. The first equalization process includes: The input multi-scale feature maps are converted to the same scale, and the converted feature maps are averaged to obtain balanced semantic features. The feature maps converted to the same scale are then convolved with the balanced semantic features to enhance them, resulting in feature maps after feature enhancement. The feature map obtained after feature enhancement is then restored to its original scale and output. The secondary equalization process includes: The input multi-scale feature map is fused from deep to shallow using an FPN network to obtain the equalized multi-scale feature map.

Images