A small sample PCB defect detection method based on meta-feature enhancement

Abstract

The application discloses a small sample PCB defect detection method based on meta feature enhancement, comprising: inputting a PCB image to be detected into a trained defect detection model, and outputting detection frames of all detected defect targets and predicted types thereof; a global feature fusion module is introduced into a support branch of the defect detection model to improve the problem that new class and base class defects are prone to confusion, and a self-attention module is introduced into a query branch to help solve the problem of missed detection of PCB defect targets; finally, point product operation is performed on query enhancement features and support enhancement features to generate aggregation features, and the aggregation features are classified and detection frame regression are performed through a detector head. Training of the defect detection model comprises a meta training stage and a meta testing stage; all parameters of the defect detection model are optimized in the meta training stage; and only parameters of the detector head of the defect detection model are optimized in the meta testing stage. The method improves the detection recall rate and classification accuracy of PCB defect detection.

Description

Technical Field

[0001] This invention relates to the field of Internet big data technology, specifically to a small-sample PCB defect detection method based on meta-feature enhancement. Background Technology

[0002] Printed circuit boards (PCBs) are essential components in devices, connecting various electronic components. They provide circuit connections and hardware support, and are crucial for the stable and reliable operation of equipment. With advancements in integrated circuit packaging technology, the wiring on PCBs has become increasingly dense. Even minute surface defects can lead to unreliable power transmission and shorten the lifespan of the PCB.

[0003] Convolutional Neural Network (CNN)-based detection algorithms acquire semantic type and location information of specific targets in images by training on large amounts of data, and are widely used for PCB surface defect detection. However, due to industry limitations, building a large target sample library in real-world scenarios faces numerous difficulties, and in some special scenarios, only a very small number of sample images can be obtained. To address these issues, few-shot learning employs the idea of ​​generalizing prior knowledge to achieve the desired results with very few training samples. Few-shot object detection (FSOD) is a branch of few-shot learning. Unlike classification, it is a more challenging task because it requires not only distinguishing the category of the target but also locating its specific position.

[0004] Current few-shot object detection methods are mainly divided into two categories: fine-tuning-based methods and meta-learning-based methods. Fine-tuning strategies transfer knowledge learned in the source domain to the target domain, thereby improving the detection performance of the target task under few-shot conditions. Existing techniques propose a transfer learning-based method that freezes all layers before the detector head on new classes, fine-tuning only the last layer. This simple training method brings significant accuracy improvements. However, fine-tuning methods require a large number of original base class samples to achieve generalization to the source task and have high requirements for the correlation between source and target domain data. Unlike fine-tuning, meta-learning trains on multiple few-shot tasks, enabling the model to better utilize experiential knowledge from previous tasks to generalize to new scenes. For example, existing techniques have constructed a lightweight meta-model, Meta-YOLO, based on YOLOv2. It uses a feature extractor to extract meta-features that can generalize to new object classes, and then introduces reweighted vectors in the support branches to generate global features of the image to aid in the detection of new categories. Existing technologies also focus on features for each RoI, introducing a Predictive Head Reshaping Network (PRN) to infer its class attention vector and inputting the weighted attention features into the detection head to complete the detection task. Existing technologies also propose the Meta-DETR meta-learning framework, using an encoder-decoder to replace the original post-processing operations such as non-maximum suppression, achieving target classification and localization at the image level.

[0005] The applicant found that existing meta-learning-based object detection methods are mainly applicable to natural scene images. However, PCB images present unique challenges. On the one hand, defects on PCBs are typically very small, only millimeters in size. During feature extraction, the network inevitably loses image information, leading to missed detections and poor comprehensiveness in PCB defect detection, i.e., low recall. On the other hand, PCBs contain multiple defect categories with small inter-class differences. With limited data, this makes it easier for the model to falsely detect different types of defects, resulting in low classification accuracy in PCB defect detection. Therefore, improving the recall and classification accuracy of PCB defect detection is a pressing technical problem that needs to be solved. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a small-sample PCB defect detection method based on meta-feature enhancement, which improves the problem of easy confusion between new and base class defects by introducing a global feature fusion module, and helps to solve the problem of missed detection of PCB defect targets by introducing a self-attention module, thereby improving the detection recall and classification accuracy of PCB defect detection.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A small-sample PCB defect detection method based on meta-feature enhancement includes:

[0009] S1: Obtain the PCB image to be inspected;

[0010] S2: Input the PCB image to be detected into the trained defect detection model, and output the detection boxes of all detected defect targets and their prediction types;

[0011] The processing steps when training the defect detection model are as follows:

[0012] S201: Divide the training sample set into a query image set and a support image set;

[0013] S202: Extract query features and support features of the query image set and support image set respectively through the backbone networks of the query branch and support branch;

[0014] S203: The query branch enhances the query features through the self-attention module, generating enhanced query features;

[0015] S204: The support branch performs feature fusion on the support features through the global feature fusion module to generate support enhancement features;

[0016] S205: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature;

[0017] S206: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types;

[0018] The training of the defect detection model includes a meta-training phase and a meta-testing phase. The training sample set in the meta-training phase is the base class images, which are used to optimize all parameters of the defect detection model. The training sample set in the meta-testing phase is the base class images and new class images, which are used to optimize only the parameters of the detector head of the defect detection model.

[0019] S3: Use the regression boxes of all detected defects and their predicted types as the defect detection results of the PCB image to be detected.

[0020] Preferably, in step S2, the processing steps of the meta-training stage are as follows:

[0021] S211: Use the base class image set as the training sample set for the defect detection model;

[0022] S212: Divide the base class image set into a support image set and a query image set, and extract query features and support features of the support image set and query image set respectively through the backbone networks of the query branch and the support branch;

[0023] S213: The query branch enhances the query features through the self-attention module, generating enhanced query features;

[0024] S214: Supports branches to perform feature fusion on query features through the global feature fusion module to generate enhanced features;

[0025] S215: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature;

[0026] S216: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types;

[0027] S217: Calculate the model loss based on the ground truth bounding box labels and ground truth type labels corresponding to the base class image set, combined with the detection bounding boxes and predicted types of all detected defect targets, and optimize the parameters of the defect detection model.

[0028] S218: Repeat steps S211 to S217 until the model converges, and obtain the meta-trained model.

[0029] Preferably, in step S2, the processing steps of the meta-testing stage are as follows:

[0030] S221: Use a balanced set of images containing base class images and new class images as the training sample set for the meta-training model;

[0031] S222: Divide the balanced image set into a support image set and a query image set, and extract query features and support features of the support image set and query image set respectively through the backbone networks of the query branch and the support branch;

[0032] S223: The query branch enhances the query features through the self-attention module, generating enhanced query features;

[0033] S224: The support branch performs feature fusion on the support features through the global feature fusion module to generate support enhancement features;

[0034] S225: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature;

[0035] S226: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types;

[0036] S227: Calculate the model loss based on the ground truth bounding box labels and ground truth type labels corresponding to the balanced image set, combined with the detection bounding boxes and predicted types of all detected defective targets; then freeze all parameters of the meta-trained model except for the detector head, and optimize only the parameters of the detector head;

[0037] S228: Repeat steps S221 to S227 until the model converges, and you will get a trained defect detection model.

[0038] Preferably, in step S2, the backbone network is selected as ResNet-101, whose basic structural unit is a convolution-based residual block;

[0039] The processing steps for the residual blocks are as follows:

[0040] 1) The input x of the backbone network passes through the first convolutional layer and activation function to obtain the feature F(x);

[0041] 2) After the feature F(x) is processed by the weight layer, it is added to the input x to obtain the feature H(x) = F(x) + x;

[0042] 3) The feature H(x) is passed to the next layer of the network after being activated by the ReLU function.

[0043] Preferably, in step S2, the processing steps of the self-attention module are as follows:

[0044] 1) Features F input to the self-attention module q Perform a 1x1 convolution operation on the channel dimension to obtain the Query vector, Key vector, and Value vector;

[0045] 2) The attention score is calculated by performing a dot product operation between the query vector and the key vector, and the attention score is normalized using a Softmax layer;

[0046] 3) Multiply the Value vector by the attention score to obtain the self-attention feature;

[0047] The formula is expressed as:

[0048]

[0049] 4) Feature F q Adding the self-attention feature A(Q,K,V) to obtain the query enhancement feature F q′ ;

[0050] The formula is expressed as:

[0051] F q′ =A(Q,K,V)⊕F q ;

[0052] In the formula: A(Q,K,V) represents the self-attention feature; Q, K, and V represent the Query vector, Key vector, and Value vector, respectively; K T This represents the transpose of the key vector; ⊕ represents the dimension of the query vector; ⊕ represents matrix addition.

[0053] Preferably, in step S2, the processing steps of the global feature fusion module are as follows:

[0054] 1) Introduce a learnable linear layer L to support the features F of the branch input. S Perform a linear transformation, which involves weighting the features at each spatial location to generate the corresponding output ω;

[0055] The formula is expressed as:

[0056] ω=Linear(F S );

[0057] 2) After performing softmax normalization on the output ω, the weight distribution of the corresponding pixels is generated, and then the weight distribution is compared with the feature F. S Perform matrix multiplication along the channel dimension to obtain the global channel feature G;

[0058] The formula is expressed as:

[0059]

[0060] 3) Activate the global channel feature G using the leaky ReLU activation function and then add it to the feature F. S Above, generate support enhancement features F S′ ;

[0061] The formula is expressed as:

[0062] F S′ =α·LeakyReLU(G)⊕F s ;

[0063] In the formula: Linear represents the process of performing a linear transformation; α represents matrix multiplication; α represents the scaling factor.

[0064] Preferably, in step S2, the loss function used to train the defect detection model includes Faster R-CNN loss and meta-loss.

[0065] Preferably, in step S2, the Faster R-CNN loss includes classification loss and regression loss;

[0066] 1) The formula for calculating classification loss is as follows:

[0067]

[0068]

[0069] In the formula: p iThis represents the probability that anchor box i is predicted to be a detection box. If anchor box i is a detection box for a defect target, then the true bounding box label is... It is 1 if it is true, otherwise it is 0.

[0070] 2) The formula for calculating regression loss is expressed as:

[0071]

[0072] In the formula: t i Represents the coordinate parameters of the i-th detection box; The coordinate parameters of the ground truth box corresponding to the positive anchor box are represented; L1 represents the smoothing loss function. This indicates that the regression loss is only effective if the anchor box prediction is correct.

[0073] Preferably, in step S2, the formula for calculating the element loss is expressed as:

[0074]

[0075] In the formula: s i This represents the probability distribution of the i-th RoI sample belonging to each category; For true category labels.

[0076] Preferably, in step S2, the loss function calculation formula for training the defect detection model is expressed as:

[0077]

[0078] Compared with existing technologies, the small-sample PCB defect detection method based on meta-feature enhancement in this invention has the following advantages:

[0079] The training of the defect detection model in this invention includes a meta-training phase and a meta-testing phase. The meta-training phase inputs a large number of base class sample images and optimizes all parameters of the model. The meta-testing phase inputs a small number of balanced samples of base class and new class samples and optimizes only the parameters of the detector head, i.e., fine-tuning. First, this invention combines meta-learning with fine-tuning strategies. By introducing a meta-learning scheme, it fully extracts prior knowledge and performs rapid generalization on new types of defects, thereby improving the robustness and adaptability of the model in PCB defect detection. Second, this invention updates the entire model parameters during the training phase, while freezing some parameters and only fine-tuning the detector head during the meta-testing phase. This improves classification ambiguity during knowledge transfer (if the parameters of the entire network are updated directly during the meta-testing phase, it may cause classification confusion due to knowledge transfer from the base class to the new class, impairing the detection accuracy of the new class), avoiding classification confusion during meta-knowledge transfer, and thus improving the classification accuracy in PCB defect detection.

[0080] This invention addresses the issue of confusion between novel and base class defects in PCB inspection tasks by incorporating a global feature fusion module on the model's support branch. This module fuses global channel features with the original support features to distinguish different defect categories. This improves the situation of imbalanced class samples in small sample datasets in PCB surface defect detection tasks, thereby mitigating the confusion between novel and base class defects and further enhancing the classification accuracy in PCB defect detection. Simultaneously, this invention addresses the issue of small target instances in PCB image defect detection tasks by introducing a self-attention module on the model's query branch. This module captures global contextual information and learns feature representations at different levels, thus better focusing on small target defects on the PCB surface. This helps solve the problem of missed detection of PCB defects, thereby improving the comprehensiveness and recall of PCB defect detection. Attached Figure Description

[0081] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0082] Figure 1 This is a network structure diagram of the defect detection model;

[0083] Figure 2 A schematic diagram illustrating the partitioning of meta-learning tasks;

[0084] Figure 3 This is a schematic diagram of the meta-training phase;

[0085] Figure 4 This is a schematic diagram of the meta-testing phase;

[0086] Figure 5 This is the network structure diagram of the residual module;

[0087] Figure 6 This is a network structure diagram of the self-attention module;

[0088] Figure 7 This is a network structure diagram of the global feature fusion module;

[0089] Figure 8 Example of PCB defect detection data;

[0090] Figure 9 For the PCB dataset, the defect categories are: Figure 9 (a) to Figure 9 (f) represents short, missing_hole, spurious_copper, open_circuit, mouse_bite, and spur, respectively.

[0091] Figure 10 The comparison diagram shows the effect of the global feature fusion module; among which Figure 10 (a) represents the global feature fusion effect. Figure 10 (b) indicates the effect of the original feature;

[0092] Figure 11 A comparison chart of recall rates for the self-attention module;

[0093] Figure 12 Visualize the comparison chart for the new category. Detailed Implementation

[0094] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0095] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship commonly used when the product is in use. They are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. In addition, the terms "horizontal," "vertical," etc., do not indicate that the component is required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0096] The following detailed explanation illustrates the specific implementation methods:

[0097] Example:

[0098] This embodiment discloses a small-sample PCB defect detection method based on meta-feature enhancement, including:

[0099] S1: Obtain the PCB image to be inspected;

[0100] S2: Input the PCB image to be detected into the trained defect detection model, and output the detection boxes of all detected defect targets and their prediction types;

[0101] like Figure 1 As shown, the processing steps when training the defect detection model are as follows:

[0102] S201: Divide the training sample set into a query image set and a support image set;

[0103] S202: Extract query features and support features of the query image set and support image set respectively through the backbone networks of the query branch and support branch;

[0104] S203: The query branch enhances the query features through the self-attention module, generating enhanced query features;

[0105] S204: The support branch performs feature fusion on the support features through the global feature fusion module to generate support enhancement features;

[0106] S205: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature;

[0107] In this embodiment, the enhanced query and support features are aggregated by convolution and fed into the detector head for localization and classification.

[0108] S206: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types;

[0109] The training of the defect detection model includes a meta-training phase and a meta-testing phase. The training sample set in the meta-training phase consists of (a large number) base class images, which are used to optimize all parameters of the defect detection model. The training sample set in the meta-testing phase consists of base class images and (a small number) new class images, which are used to optimize only the parameters of the detector head of the defect detection model.

[0110] In this embodiment, the support images are a subset of the dataset used in the meta-task to train the model. The samples in the support set are considered known, and the features of each category learned from the support set, i.e., the class prototype vectors, are typically used as category templates to aid in the identification of new categories.

[0111] The query image is also part of the meta-task dataset used to evaluate the model's performance on new classes. It contains unseen samples, typically from new classes. After updating its parameters on the support set, the model uses the learned parameters to make predictions on the query set. The purpose of the query set is to evaluate the model's generalization ability and performance on new classes.

[0112] Novel image classes represent new defects to be detected in this paper. These defect samples are extremely rare, representing truly small samples. Novel image classes are used to evaluate the model's generalization ability when facing unknown defect targets. By training on novel image classes during the meta-testing phase, the model parameters are continuously optimized, better adapting to the detection of novel image classes.

[0113] S3: Use the regression boxes of all detected defects and their predicted types as the defect detection results of the PCB image to be detected.

[0114] The training of the defect detection model in this invention includes a meta-training phase and a meta-testing phase. The meta-training phase inputs a large number of base class sample images and optimizes all parameters of the model. The meta-testing phase inputs a small number of balanced samples of base class and new class samples and optimizes only the parameters of the detector head, i.e., fine-tuning. First, this invention combines meta-learning with fine-tuning strategies. By introducing a meta-learning scheme, it fully extracts prior knowledge and performs rapid generalization on new types of defects, thereby improving the robustness and adaptability of the model in PCB defect detection. Second, this invention updates the entire model parameters during the training phase, while freezing some parameters and only fine-tuning the detector head during the meta-testing phase. This improves classification ambiguity during knowledge transfer (if the parameters of the entire network are updated directly during the meta-testing phase, it may cause classification confusion due to knowledge transfer from the base class to the new class, impairing the detection accuracy of the new class), avoiding classification confusion during meta-knowledge transfer, and thus improving the classification accuracy in PCB defect detection.

[0115] This invention addresses the issue of confusion between novel and base class defects in PCB inspection tasks by incorporating a global feature fusion module on the model's support branch. This module fuses global channel features with the original support features to distinguish different defect categories. This improves the situation of imbalanced class samples in small sample datasets in PCB surface defect detection tasks, thereby mitigating the confusion between novel and base class defects and further enhancing the classification accuracy in PCB defect detection. Simultaneously, this invention addresses the issue of small target instances in PCB image defect detection tasks by introducing a self-attention module on the model's query branch. This module captures global contextual information and learns feature representations at different levels, thus better focusing on small target defects on the PCB surface. This helps solve the problem of missed detection of PCB defects, thereby improving the comprehensiveness and recall of PCB defect detection.

[0116] Experiments show that the PCB defect detection method of this invention is effective under extremely small sample conditions. The accuracy on 10-shot (10-shot refers to training with 10 labeled instances; for example, there may be more than 10 ground truth bounding boxes on multiple images, but only 10 samples are used for training) reaches 62.4%, which is higher than most FSOD models.

[0117] To better illustrate the technical solution of this invention, the prior art is analyzed as follows:

[0118] Existing few-shot object detection methods typically employ a two-stage training paradigm: base training and few-shot fine-tuning. Let the dataset be D = {(x,y),x∈X,y∈Y)}, where x is the input image and y is the corresponding label. This dataset has C classes. Throughout the few-shot learning task, all classes C are classified as base classes C0. b And new class Cn C b The number of samples in C is much larger than that in C. n It is worth noting that: C b ∪C n =C and Typically, rich C++ is used during training. b The samples are used for base class training, and then a small number of m-shots (m = 1, 2, 3, ...) of new class samples C are used. n Fine-tuning is performed on top of this, with the ultimate goal of learning a method that can detect new types of samples C. n The model.

[0119] The meta-learning framework follows the above FSOD setup but also has some differences. The first stage extracts meta-knowledge from a large number of base class samples; this stage is called meta-training. The second stage uses base class and new class samples for fine-tuning; this stage is called meta-testing. Meta-learning is trained by constructing meta-tasks. This represents the support set, containing m distinct targets of interest. The query set, containing one or more images and annotation information, can be represented by the formula T = {S1, ..., S...}. n In the N-way M-shot task setup, during the meta-training phase, from the base class C... b N classes and M labeled instances are randomly selected from C to form N*M support set samples. b A query set is constructed by extracting a number of samples from the dataset. Through collaborative training on the PCB defect detection dataset as a meta-task, the model acquires the ability to detect the target image in a query image with only a small number of support samples. Unlike the base class, which has a large amount of sample data, the new class has only M target samples.

[0120] This two-stage meta-learning task division method is as follows: Figure 2 As shown: the base class is marked with a yellow box, and the new class is marked with a red box. This invention constructs a meta-learning task by sampling the same number of training samples from both the base class and the new class during the meta-testing phase. The purpose of this is to avoid performance degradation on the base class while fine-tuning the new class.

[0121] Faster R-CNN (from REN SQ, HE KM, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks) is a popular two-stage object detection algorithm. Its main components include a backbone network, a Region Proposal Network (RPN), and a bounding box (BGB) head. The BGB head consists of a classifier and a regressor; the classifier classifies the object category, and the regressor predicts the bounding box coordinates. Meta R-CNN (from YAN XP, CHEN ZL, XUA, et al. Meta R-CNN: Towards general solver for instance-level low-shot learning) combines meta-learning with the RoI features extracted by the RPN in Faster R-CNN, solving the problem of meta-learning's difficulty in handling multiple instances in a single image.

[0122] In the meta-training phase, Meta R-CNN first uses the backbone network to extract base class meta-features, and then shares them with the query and support branches. The support and query branches respectively generate ROI features F for different categories through the RPN network. s and F q F s and F q Feature aggregation is performed, and the aggregated ROI features are input into the detector head to complete the detection task. During the meta-testing phase, MetaR-CNN uses balanced samples from the base class and the new class for fine-tuning. For example... Figure 3 As shown, Meta R-CNN fine-tunes the entire network to transfer the base class knowledge learned during meta-training to achieve detection of new classes. However, in Faster R-CNN, the RPN is mainly used to extract ROI features and is functionally class-agnostic; the specific classification is done by the bounding box detection head. Directly updating the parameters of the entire network may cause classification confusion due to the knowledge transfer from base classes to new classes, thereby impairing the detection accuracy of the new classes.

[0123] This invention divides feature representation learning and bounding box classification prediction learning into two steps. The first step is the same as Meta R-CNN, performing meta-training on data-rich base class samples to fully extract base class meta-features. The second step, the meta-testing stage, fine-tunes only the last layer of the detector on a small-scale balanced dataset to complete the bounding box classification prediction. Figure 4As shown: During the meta-testing phase, most parameters are frozen, with only the detector head fine-tuned. No further parameter updates are needed, and features learned from the base class are transferred to the new class. Simultaneously, to facilitate the introduction of specific modules in the next step, the query and support branches are decoupled in terms of features. For feature decoupling: a c-way m-shot (c classes, m labeled samples) labeled image is input into the backbone network, outputting c class vectors, denoted as... Then, in the RCNN network, multiple RoI features are output through the RoI pooling layer, and the feature set is denoted as . For ease of research, following the definition of meta-learning, Let it be F s That is, support feature vectors, Let it be F q That is, querying feature vectors.

[0124] Specifically, in this invention:

[0125] Combination Figure 3 As shown, the processing steps in the meta-training phase are as follows:

[0126] S211: Use the base class image set as the training sample set for the defect detection model;

[0127] S212: Divide the base class image set into a support image set and a query image set, and extract query features and support features of the support image set and query image set respectively through the backbone networks of the query branch and the support branch;

[0128] S213: The query branch enhances the query features through the self-attention module, generating enhanced query features;

[0129] S214: Supports branches to perform feature fusion on query features through the global feature fusion module to generate enhanced features;

[0130] S215: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature;

[0131] S216: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types;

[0132] S217: Calculate the model loss based on the ground truth bounding box labels and ground truth type labels corresponding to the base class image set, combined with the detection bounding boxes and predicted types of all detected defect targets, and optimize the parameters of the defect detection model.

[0133] S218: Repeat steps S211 to S217 until the model converges, and obtain the meta-trained model.

[0134] Combination Figure 4As shown, the processing steps in the meta-test phase are as follows:

[0135] S221: Use a balanced set of images containing base class images and new class images as the training sample set for the meta-training model;

[0136] In this embodiment, during the meta-testing phase, the same number of training samples are sampled from the base class and the new class to construct the meta-learning task (i.e., balancing image samples). The purpose of doing this is to avoid performance degradation of the model in the base class while fine-tuning the new class.

[0137] S222: Divide the balanced image set into a support image set and a query image set, and extract query features and support features of the support image set and query image set respectively through the backbone networks of the query branch and the support branch;

[0138] S223: The query branch enhances the query features through the self-attention module, generating enhanced query features;

[0139] S224: The support branch performs feature fusion on the support features through the global feature fusion module to generate support enhancement features;

[0140] S225: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature;

[0141] S226: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types;

[0142] S227: Calculate the model loss based on the ground truth bounding box labels and ground truth type labels corresponding to the balanced image set, combined with the detection bounding boxes and predicted types of all detected defective targets; then freeze all parameters of the meta-trained model except for the detector head, and optimize only the parameters of the detector head;

[0143] S228: Repeat steps S221 to S227 until the model converges, and you will get a trained defect detection model.

[0144] This invention combines meta-learning with fine-tuning strategies. By introducing a meta-learning scheme, it fully extracts prior knowledge and rapidly generalizes to new types of defects, thereby improving the robustness and adaptability of the model in PCB defect detection. Secondly, this invention updates the entire model parameters during the training phase, while freezing some parameters and only fine-tuning the detector head during the meta-testing phase. This improves classification ambiguity during knowledge transfer (updating the entire network parameters directly during the meta-testing phase may cause classification confusion due to knowledge transfer from the base class to the new class, impairing the detection accuracy of the new category), avoiding classification confusion during meta-knowledge transfer, and thus improving the classification accuracy in PCB defect detection.

[0145] In the specific implementation process, regarding the selection of the backbone network, by comparing the performance of various convolutional neural networks, it was found that ResNet-101 (from HE KM, ZHANG XY, REN SQ, et al. Deep residual learning for image recognition) is more efficient with the same number of parameters and is suitable for PCB defect detection with small datasets and high accuracy requirements. Therefore, this invention selects ResNet-101 for feature extraction, and its basic structural unit is a convolution-based residual block.

[0146] like Figure 5 As shown, the processing steps for the residual block are as follows:

[0147] 1) The input x of the backbone network passes through the first convolutional layer and activation function to obtain the feature F(x);

[0148] 2) After the feature F(x) is processed by the weight layer, it is added to the input x to obtain the feature H(x) = F(x) + x;

[0149] 3) The feature H(x) is passed to the next layer of the network after being activated by the ReLU function.

[0150] In practice, the original PCB image typically has 2000-3000 pixels, but the defect target to be detected usually only contains a few pixel values. If the features extracted by the backbone network are used directly for detection, the model will be unable to fully extract the target features, thus lacking the ability to discriminate the target. To solve the above problem, this invention introduces a self-attention module in the query branch for feature enhancement.

[0151] Combination Figure 6 As shown, the processing steps of the self-attention module are as follows:

[0152] 1) Features F of the query branch input q Perform a 1x1 convolution operation on the channel dimension to obtain the Query vector, Key vector, and Value vector;

[0153] 2) The attention score is calculated by performing a dot product operation between the query vector and the key vector, and the attention score is normalized using a Softmax layer;

[0154] 3) Multiply the Value vector by the attention score to obtain the final self-attention feature;

[0155] The formula is expressed as:

[0156]

[0157] 4) Feature Fq Adding the self-attention feature A(Q,K,V) to obtain the final query enhancement feature F. q′ ;

[0158] The formula is expressed as:

[0159] F q′ =A(Q,K,V)⊕F q ;

[0160] In the formula: A(Q,K,V) represents the self-attention feature; Q, K, and V represent the Query vector, Key vector, and Value vector, respectively; K T This represents the transpose of the key vector; Represents the dimension of the query vector, divided by This can prevent very small gradients from being generated after the Softmax operation, ensuring the stability of training; ⊕ represents matrix addition.

[0161] This invention addresses the issue of small target instances in PCB image defect detection tasks by introducing a self-attention module into the query branch of the model. This module captures global contextual information and learns feature representations at different levels, thereby better focusing on small target defects on the PCB surface. This helps solve the problem of missed defects in PCB detection, thus improving the comprehensiveness of PCB defect detection. The self-attention module has the following advantages: First, it does not rely on contextual information and can focus on the features of the target itself, avoiding interference from other targets; second, it establishes a connection between the target and surrounding pixels by calculating attention weights between any two positions, improving representation capabilities.

[0162] In practice, distinguishing between defect classes on PCB images is a challenging task. In the meta-learning network, the image features extracted by the query branch need to be weighted and aggregated with the support features to distinguish the category to which the PCB defect belongs. Therefore, the support features act as category templates. To improve the quality of the support features and accurately distinguish different categories, a global feature fusion module is designed here.

[0163] Combination Figure 7 As shown, the processing steps of the global feature fusion module are as follows:

[0164] 1) Introduce a learnable linear layer L∈R C×1 For the feature F that supports branch input S ∈R C×H×W Perform a linear transformation, which involves weighting the features at each spatial location to generate the corresponding output ω∈R. 1×H×W This output represents the weight of each spatial location.

[0165] The formula is expressed as:

[0166] ω=Linear(F S );

[0167] 2) After performing softmax normalization on the output ω, the weight distribution of the corresponding pixels is generated, and then the weight distribution is compared with the feature F. S Perform matrix multiplication along the channel dimension to obtain the global channel features G∈R. 1×C ;

[0168] The formula is expressed as:

[0169]

[0170] 3) Activate the global channel feature G using the leaky ReLU activation function and then add it to the feature F. S Above, the final support enhancement feature f is generated. S′ ;

[0171] The formula is expressed as:

[0172] F S′ =α·LeakyReLU(G)⊕F s ;

[0173] In the formula: Linear represents the process of performing a linear transformation; α represents matrix multiplication; α represents the scaling factor, used to adjust the proportion of global channel features. By superimposing the original support enhancement features, it helps improve the distinguishability between defect categories on the PCB image.

[0174] This invention addresses the problem of confusion between new and base class defects in PCB inspection tasks by designing a global feature fusion module on the support branch of the model. This module fuses global channel features with the original support features to distinguish different defect categories. This improves the imbalance of class samples in small sample datasets in PCB surface defect inspection tasks, thereby alleviating the problem of confusion between new and base class defects and further improving the classification accuracy in PCB defect detection.

[0175] In specific implementation, similar to Meta R-CNN, the loss function used to train the defect detection model in this invention comprises two parts. The first part is the standard Faster R-CNN loss, including classification loss and regression loss, L cls It is the cross-entropy loss of the Bbox classifier, L reg The first part is the L1 loss of the Bbox regressor. The second part is the meta-loss; the support feature vectors learned for different classes should have different selection effects on the RoI features of different classes, thus avoiding ambiguity and vagueness in the prediction process. Similar to the classification loss, L...meta This is achieved using cross-entropy loss, which causes the target vector to fall into the class to which the target belongs.

[0176] 1) Faster R-CNN loss includes classification loss and regression loss;

[0177] 1.1) The formula for calculating classification loss is as follows:

[0178]

[0179] In the formula: p i This represents the probability that anchor box i is predicted as a detection box containing a defect. If anchor box i is a detection box containing a defect (i.e., the object can be accurately predicted), then the ground truth box label is... It is 1 if it is true, otherwise it is 0.

[0180] 1.2) The formula for calculating regression loss is expressed as:

[0181]

[0182] In the formula: t i Represents the coordinate parameters of the i-th detection box; The coordinate parameters of the ground truth box corresponding to the positive anchor box are represented; L1 represents the smoothing loss function. This indicates that the regression loss is only effective if the anchor box prediction is correct, that is, when anchor box i is the detected box of the defect target, it is the true box label. It is 1 if it is true, otherwise it is 0.

[0183] 2) The formula for calculating element loss is expressed as:

[0184]

[0185] In the formula: s i This represents the probability distribution of the i-th RoI sample belonging to each category; For true category labels.

[0186] 3) The formula for calculating the loss function when training the defect detection model is as follows:

[0187]

[0188] To better illustrate the advantages of the technical solution of the present invention, the following experiment is disclosed in this embodiment.

[0189] 1. PCB Defect Detection Dataset

[0190] The experiment uses the publicly available dataset PKU-Market-PCB (from HUANG WB, WEIP.A PCB dataset for defect detection and classification). This is a public synthetic PCB dataset with 693 images and 2953 labeled instances suitable for defect detection tasks. There are six PCB defect categories: missing vias, rodent bites, open circuits, short circuits, burrs, and fake copper. For the training task, 554 images were randomly selected as the training set, and 139 images as the validation set. In the FSOD task, the rodent bite category was selected as the new class, and the other five categories were used as base classes. During the pre-training phase, only the base class data was used; during the fine-tuning phase, base class and new class data with equivalent instances were added. An example of the dataset is shown below. Figure 8 As shown, PCB defects are extremely small, and the board surface circuitry is complex. The new category of "rat bites" is easily confused with the base categories of "burrs" and "open circuits." A cropped and enlarged version of the dataset is shown below. Figure 9 It includes 6 types of defects.

[0191] 2. Performance Evaluation Indicators

[0192] The performance metric for evaluating PCB defect detection is the average AP (mAP) of each category. AP is obtained by calculating the area under the precision-recall curve, expressed by the formula:

[0193]

[0194] In the formula: TP is the number of positive examples correctly classified as positive, FP is the number of negative examples incorrectly classified as positive, and FN is the number of positive examples incorrectly classified as negative. TP and FP are determined by the Intersection over Union (IoU) between predicted and ground truth bounding boxes of the same type. If the IoU is greater than a certain threshold, the detected bounding box is TP; otherwise, it is FP. If no corresponding target is detected in the image, it is marked as FN. IoU formula definition:

[0195]

[0196] In the formula: B p For the prediction box, B gt This is the actual bounding box. `area()` represents the area. `IOU` calculates the area using `B`. p With B gt The Intersection over Union (IOU) ratio represents the degree of overlap between the predicted bounding box and the ground truth bounding box. Following the Pascal VOC criterion, the IOU threshold was set to 0.5 in this experiment to calculate the corresponding mAP value.

[0197] 3. Experimental Setup

[0198] All methods in this experiment were implemented on Mmdetection using the PyTorch framework and trained on four RTX 3090 GPUs. For fair comparison, all models underwent multi-scale training. The backbone network was a ResNet101 pre-trained on ImageNet (from DENG J, DONG W, SOCHER R, et al. ImageNet: A large-scale hierarchical image database). SGD was used as the optimizer during training, with a batch size of 4 per GPU. The global feature fusion module constant α was set to 0.1. In the meta-training phase, the learning rate was set to 0.02, momentum to 0.9, weight decay to 1e4, and the maximum number of iterations to 8000. In the meta-testing phase, the learning rate was set to 0.02. The base class and new class were given the same number of samples, and K = {1, 2, 3, 5, 10} shots corresponded to {100, 100, 200, 500, 600} iterations, while keeping other hyperparameters the same as Meta R-CNN.

[0199] 4. Ablation test

[0200] Ablation experiments were designed based on the method proposed in this invention, and the effectiveness of each module in the system was analyzed. All ablation experiments were conducted on the PKU-Market-PCB dataset, demonstrating the detection performance for the new class (mouse bite).

[0201] 1) Baseline Network Improvements. Based on Meta R-CNN, the training strategy was adjusted. Specifically, during the meta-testing phase, the backbone network and RPN network were frozen, and only the head was fine-tuned to ensure that classification accuracy was not affected during the generalization process from base class to new class knowledge. As shown in Table 1, the adjustments during the meta-testing phase significantly improved the overall accuracy of the baseline network, especially on 10-shot tests, demonstrating the effectiveness of the meta-learning combined with fine-tuning strategy. Furthermore, the improved baseline network is more memory-efficient because the entire network does not need to be retrained during the meta-testing phase.

[0202] Table 1. Improvement effect of baseline network

[0203]

[0204] 2) Impact of the Meta-Feature Enhancement Module. Table 2 verifies the effectiveness of the meta-feature enhancement module. Through comparative experiments, it can be seen that the two modules are independent in effect and can mutually promote the improvement of detection performance. Meanwhile, when the number of samples is small, the attention module improves accuracy more significantly, while at 10-shot, the global feature fusion module shows a more pronounced effect, which may be determined by the characteristics of PCB images.

[0205] Table 2 Validation of the Meta-feature Enhancement Module

[0206]

[0207] like Figure 10 As shown, when detecting new categories under the same 0.3 threshold condition, Figure 10 (a) Perform feature fusion in the support branch. Figure 10 (b) The figure shows the use of the original features. The yellow boxes show the location and confidence of the detection boxes. Obviously, there is no significant difference in the localization effect between the two methods. However, the confidence of the original supporting features in the new category is lower, which leads to ambiguity in the classification. After fusing the supporting features in the channel dimension, the classification effect is significantly improved, which further illustrates the importance of supporting features for class differentiation.

[0208] Figure 11 The study demonstrates the improvement in target recall achieved by the self-attention module. Recall is significantly improved with only 1-10 samples, and the improvement is even more pronounced when the sample size is very limited.

[0209] 5. Comparison with other methods

[0210] General object detection based on deep learning requires a large number of labeled samples for training. However, when the number of available labeled samples is limited and there is a distribution difference between the training set and the test set, the detection performance will drop significantly. Tables 3 and 4 respectively show the mAP (%) results of general object detection models such as Faster-RCNN (from REN SQ, HE KM, GIRSHICK R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks), YOLOV8, RT-DETR (from LV WY, ZHAO Y, XU SL, et al. Detrs beat yolos on real-time object detection), and object detection models based on few-shot learning such as Meta R-CNN, TFA (from WANG X, HUANG T, GONZALEZ J, et al. Frustratingly simple few-shot object detection), MPSR (from WU JX, LIU ST, HUANG D, et al. Multi-scale positive sample refinement for few-shot object detection), and VFA (from HAN JM, REN YQ, DING J, et al. Few-shot object detection via variational feature aggregation) and the method proposed in this invention on the PCB defect detection dataset. Traditional detection algorithms perform poorly in small-sample object detection tasks; with a new category containing only 10 labeled samples, the highest mAP is only 37.4%. Therefore, it is necessary to explore the prior knowledge contained in a small number of samples from multiple aspects, including models and algorithms, to improve object detection performance.

[0211] Table 3 Comparison with traditional target detection methods

[0212]

[0213] Compared with other FSOD algorithms, the model of this invention has unique advantages, indicating that the meta-feature enhancement module designed according to the characteristics of PCB images has achieved significant improvements in classification and recall, and the accuracy has been significantly improved.

[0214] Table 4 Comparison with small sample target detection methods

[0215]

[0216] Combination Figure 12 It is known that the Meta R-CNN algorithm is suitable for detection in natural scenes, but it ignores the problems existing in PCB defect detection scenarios. Therefore, it tends to incorrectly identify new classes as base classes, resulting in significant confusion between categories and a high false negative rate. The method of this invention significantly improves the false detection problem of new class defects while maintaining a high confidence level. In addition, the false negative rate of the target is also reduced to a certain extent.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A small-sample PCB defect detection method based on meta-feature enhancement, characterized in that, include: S1: Obtain the PCB image to be inspected; S2: Input the PCB image to be detected into the trained defect detection model, and output the detection boxes of all detected defect targets and their predicted types; The processing steps when training the defect detection model are as follows: S201: Divide the training sample set into a query image set and a support image set; S202: Extract query features and support features of the query image set and support image set respectively through the backbone networks of the query branch and support branch; S203: The query branch enhances the query features through the self-attention module, generating enhanced query features; S204: The support branch performs feature fusion on the support features through the global feature fusion module to generate support enhancement features; The processing steps of the global feature fusion module are as follows: 1) Introduce learnable linear layers Features that support branch inputs Perform a linear transformation, which involves weighting the features at each spatial location to generate the corresponding output. ; The formula is expressed as: ； 2) Output After performing softmax normalization, the weight distribution of the corresponding pixels is generated, and then the weight distribution is compared with the features. Perform matrix multiplication along the channel dimension to obtain global channel features. ; The formula is expressed as: ； 3) Use the LeakyReLU activation function to optimize global channel features. Activate and superimpose onto features Above, generate features that support enhancement. ; The formula is expressed as: ； In the formula: This represents the process of performing a linear transformation; This represents matrix multiplication. Indicates the scaling factor; S205: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature; S206: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types; The training of the defect detection model includes a meta-training phase and a meta-testing phase; The training sample set in the meta-training phase consists of base class images, which are used to optimize all parameters of the defect detection model. The training sample set for the meta-test phase consists of base class images and new class images, which are used to optimize the parameters of the detector head of the defect detection model only. S3: Use the detection boxes of all detected defects and their prediction types as the defect detection results of the PCB image to be inspected.

2. The small-sample PCB defect detection method based on meta-feature enhancement as described in claim 1, characterized in that: In step S2, the processing steps of the meta-training phase are as follows: S211: Use the base class image set as the training sample set for the defect detection model; S212: Divide the base class image set into a support image set and a query image set, and extract query features and support features of the support image set and query image set respectively through the backbone networks of the query branch and the support branch; S213: The query branch enhances the query features through the self-attention module, generating enhanced query features; S214: The support branch performs feature fusion on the support features through the global feature fusion module to generate support enhancement features; S215: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature; S216: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types; S217: Calculate the model loss based on the ground truth bounding box labels and ground truth type labels corresponding to the base class image set, combined with the detection bounding boxes and predicted types of all detected defect targets, and optimize the parameters of the defect detection model. S218: Repeat steps S211 to S217 until the model converges, and obtain the meta-trained model.

3. The small-sample PCB defect detection method based on meta-feature enhancement as described in claim 2, characterized in that: In step S2, the processing steps of the meta-test phase are as follows: S221: Use a balanced set of images containing base class images and new class images as the training sample set for the meta-training model; S222: Divide the balanced image set into a support image set and a query image set, and extract query features and support features of the support image set and query image set respectively through the backbone networks of the query branch and the support branch; S223: The query branch enhances the query features through the self-attention module, generating enhanced query features; S224: The support branch performs feature fusion on the support features through the global feature fusion module to generate support enhancement features; S225: Perform a dot product operation on the query augmentation feature and the supporting augmentation feature to generate the corresponding aggregated feature; S226: Classify aggregated features and regress bounding boxes using the detector head, and output the bounding boxes of all detected defect targets and their predicted types; S227: Calculate the model loss based on the ground truth bounding box labels and ground truth type labels corresponding to the balanced image set, combined with the detection bounding boxes and predicted types of all detected defective targets; then freeze all parameters of the meta-trained model except for the detector head, and optimize only the parameters of the detector head; S228: Repeat steps S221 to S227 until the model converges, and you will get a trained defect detection model.

4. The small-sample PCB defect detection method based on meta-feature enhancement as described in claim 1, characterized in that: In step S2, the backbone network is selected as ResNet-101, whose basic structural unit is a convolution-based residual block; The processing steps for the residual blocks are as follows: 1) Input to the backbone network After the first convolutional layer and activation function, the features are obtained. ; 2) Features After weighting, it is compared with the input. ; 3) Features After being activated by the ReLU function, it is passed to the next layer of the network.

5. The small-sample PCB defect detection method based on meta-feature enhancement as described in claim 1, characterized in that: In step S2, the processing steps of the self-attention module are as follows: 1) Features of the input to the self-attention module Perform a 1x1 convolution operation on the channel dimension to obtain the Query vector, Key vector, and Value vector; 2) The attention score is calculated by performing a dot product operation between the query vector and the key vector, and the attention score is normalized using a Softmax layer; 3) Multiply the Value vector by the attention score to obtain the self-attention feature; The formula is expressed as: ； 4) Features With self-attention characteristics Add them together to obtain the query enhancement features. ; The formula is expressed as: ； In the formula: Indicates self-attention characteristics; , , These represent the Query vector, Key vector, and Value vector, respectively. This represents the transpose of the key vector; Indicates the dimension of the query vector; This represents matrix addition.

6. The small-sample PCB defect detection method based on meta-feature enhancement as described in claim 1, characterized in that: In step S2, the loss function used to train the defect detection model includes Faster R-CNN loss and meta-loss.

7. The small-sample PCB defect detection method based on meta-feature enhancement as described in claim 6, characterized in that: In step S2, the Faster R-CNN loss includes classification loss and regression loss; 1) The formula for calculating classification loss is as follows: ； ； In the formula: Indicates anchor frame Predict the probability of an anchor box being a detection box. For a detection bounding box representing a defect target, the label of the ground truth bounding box is... It is 1 if it is true, otherwise it is 0. 2) The formula for calculating regression loss is expressed as: ； In the formula: Indicates the first The coordinate parameters of the secondary detection box; Represents the coordinate parameters of the actual bounding box corresponding to the positive anchor box; Represents the smoothing loss function; This indicates that the regression loss is only effective if the anchor box prediction is correct.

8. The small-sample PCB defect detection method based on meta-feature enhancement as described in claim 6, characterized in that: In step S2, the formula for calculating the meta-loss is expressed as follows: ； ； In the formula: Indicates the first The probability distribution of each RoI sample belonging to each category; For true category labels.

9. The small-sample PCB defect detection method based on meta-feature enhancement as described in claim 6, characterized in that: In step S2, the loss function calculation formula for training the defect detection model is expressed as follows: 。

Images