Small sample PCB defect detection method and device based on feature enhancement and multi-scale fusion

Abstract

The disclosed embodiment discloses a small sample PCB defect detection method and device based on feature enhancement and multi-scale fusion. Specifically, this method adds a feature enhancement module to the original meta-learning small sample target detection algorithm FSRW, highlights the key areas of the received feature map, suppresses the interference of useless information, enhances feature discriminability, and thus improves the detection effect. In view of the characteristics of PCB board defect features being small in size and difficult to detect, a BI-FU multi-scale fusion strategy is proposed to extract multiple scale feature maps of the PCB and fuse them into a high-quality feature map containing information at different scales, further improving the model's detection accuracy for small target defects. Compared with the original FSRW method, this method improves the detection accuracy of PCB defects, and compared with other small sample target detection algorithms, it also has excellent detection results, good generalization ability and robustness, and high practical application value.

Description

Technical Field

[0001] The present disclosure relates to the field of machine vision detection technology, and in particular to a small sample PCB defect detection method and device based on feature enhancement and multi-scale fusion. Background Art

[0002] Printed circuit boards (PCBs), the foundation of the modern information industry, are widely used in the manufacturing of various high-end equipment, including computers, communications electronics, and military systems. As a crucial carrier for electrical connections and support, PCB quality significantly impacts the stability and safety of these high-end products. Therefore, researching high-quality PCB surface defect detection and timely elimination is crucial.

[0003] However, PCB production lines have short turnover cycles and a relatively small number of defect samples. Traditional target detection models often suffer from overfitting or underfitting, preventing them from achieving optimal detection results. Existing small-sample detection methods are limited in their ability to mine feature information in key areas, resulting in low small-sample detection accuracy. Furthermore, due to the unique nature of PCB products, defects often have subtle features and some defects vary in size, making it difficult for models to capture and determine defect categories during the detection process. Therefore, PCB defect detection remains challenging in practice due to the complexity of the task. Summary of the Invention

[0004] In view of this, the embodiments of the present disclosure provide a small-sample PCB defect detection method and device based on feature-enhanced multi-scale fusion, which can solve the problem in current PCB board defect detection that the number of defect samples is small and the PCB board defect features are tiny, making it difficult for traditional detection models to achieve good detection results.

[0005] In a first aspect, an embodiment of the present disclosure provides a small sample PCB defect detection method based on feature enhancement multi-scale fusion, which adopts the following technical solution: A small sample PCB defect detection method based on feature enhancement multi-scale fusion, comprising: step 1, constructing a base class dataset and a new class dataset; wherein the base class dataset contains sample images and their annotation information, and the new class dataset contains PCB images and their annotation information; step 2, dividing the base class dataset and the new class dataset into task datasets consisting of query set images and support set images respectively; step 3, performing the following processing on each of the task datasets: step 31, using the DarkNet-53 network to extract the feature map of the input query set image, and then performing feature enhancement processing on the feature map to output the enhanced query set feature map; step 32, using A BI-FU multi-scale feature fusion module is used to first extract feature maps of multiple scales of the input support set image, and the module is used to fuse the feature maps of multiple scales to obtain a multi-scale fused feature map; the multi-scale fused feature map is then convolved to output a feature reweighted vector; step 33, the query set feature map and the feature reweighted vector are fused to obtain a reweighted query set feature map; step 34, a prediction analysis is performed based on the reweighted query set feature map to generate a prediction result including a predicted position and a predicted category; step 4, a two-stage training strategy is adopted, first the model is trained using the task data set of the base class data set, and then the model is fine-tuned using the task data set of the new class data set; step 5, PCB surface defects are detected based on the trained model.

[0006] Furthermore, the constructing of the base class dataset and the new class dataset includes: obtaining the FSOD dataset as the base class dataset; and establishing a PCB defect image dataset as the new class dataset using PCB finished product images with defect annotation information.

[0007] Furthermore, the defect annotation information indicates that the defect type is at least one of missing holes, rat bites, open circuits, short circuits, burrs, and excess copper.

[0008] Furthermore, the task dataset includes a query set image and multiple support set images; wherein the multiple support set images are the same number of images extracted from each category image of the base class dataset or the new class dataset.

[0009] Furthermore, performing feature enhancement processing on the feature graph and outputting the enhanced query set feature graph includes:

[0010] The intermediate feature map F′ is obtained using the following formula using the feature map F:

[0011]

[0012]

[0013] Among them, σ is the sigmiod activation function, W0 and W1 are the weights of MLP, is the average pooling feature, is the maximum pooling feature, W0∈R c / r×c , W1∈R c×c / r , R is a real number, c is the number of channels, r is the reduction rate, Represents element-wise multiplication;

[0014] Then use the intermediate feature map F′ to obtain the enhanced query set feature map F according to the following formula c :

[0015]

[0016]

[0017] Among them, σ is the sigmiod activation function, f 7×7 is a convolution operation with a convolution kernel of 7×7, is the average pooled 2D feature map, is the maximum pooled 2D feature map, Represents element-wise multiplication.

[0018] Furthermore, the BI-FU multi-scale feature fusion module is used to first extract feature maps of multiple scales of the input support set image, and the module is used to fuse the multiple feature maps of different scales to obtain a multi-scale fused feature map, including: using VGG16 and BiFPN Layer as the multi-scale feature extraction module BI, generating multiple feature maps of different scales from the input support set image through VGG16, and then extracting the multiple feature maps of different scales through BiFPN Layer, and using them as input of the feature fusion module FU; and fusing the multiple feature maps of different scales output by the multi-scale feature extraction module BI through the feature fusion module FU to obtain the multi-scale fused feature map.

[0019] Furthermore, the feature fusion module FU fuses the multiple feature maps of different scales output by the multi-scale feature extraction module BI to obtain the multi-scale fused feature map, including: performing a convolution operation on the multiple feature maps of different scales, compressing them to obtain a 1×1 tensor of the corresponding scale to form a feature fusion vector, and then using the feature fusion vector to reweight the multiple feature maps of different scales. The reweighted feature maps of different scales are fused into a multi-scale fused feature map through an element-by-element addition operation.

[0020] Furthermore, the performing convolution processing on the multi-scale fusion feature map and outputting the feature reweighted vector includes: using a lightweight convolutional network to perform a convolution operation on the input multi-scale fusion feature map to shape it into a feature reweighted vector.

[0021] Furthermore, between step 4 and step 5, the method further includes: step 4.5, testing the model using query set images of the query set divided into the new class dataset, and judging the model performance through detection indicators.

[0022] In the second aspect, the embodiment of the present disclosure further provides a small sample PCB defect detection device based on feature enhancement multi-scale fusion, which adopts the following technical solution: a small sample PCB defect detection device based on feature enhancement multi-scale fusion, comprising: an acquisition module, used to construct a base class data set and a new class data set; wherein, the base class data set contains sample images and their annotation information, and the new class data set includes PCB images and their annotation information; a task division module, used to divide the base class data set and the new class data set into task data sets consisting of query set images and support set images respectively; a task data processing module, comprising: a query set processing branch module, used to extract the feature map of the input query set image using the DarkNet-53 network, and then perform feature enhancement processing on the feature map to output the enhanced query set feature map; a support set processing module The processing branch module uses the BI-FU multi-scale feature fusion module to first extract feature maps of multiple scales of the input support set image, and uses the module to fuse the feature maps of multiple scales to obtain a multi-scale fusion feature map; then the multi-scale fusion feature map is convolved to output a feature reweighted vector; the fusion module is used to fuse the query set feature map and the feature reweighted vector to obtain a reweighted query set feature map; the detection module is used to perform prediction analysis based on the reweighted query set feature map to generate a prediction result including a predicted position and a predicted category; wherein a two-stage training strategy is adopted, firstly the model is trained through the task data set of the base class data set, and then the model is fine-tuned through the task data set of the new class data set; PCB surface defects are detected based on the trained model.

[0023] The disclosed embodiments provide a small-sample PCB defect detection method and device based on feature enhancement and multi-scale fusion. The method utilizes the DarkNet-53 network to enhance the feature extraction capability of query set images, fully utilizes the sample's own information, and alleviates the problem of insufficient sample information. Furthermore, by performing feature enhancement processing on the query set feature map, different weights are assigned within the query set feature map, highlighting key feature information in the feature map, suppressing interference from irrelevant information, enhancing the feature expression capability of the query sample, and improving the detection effect. Furthermore, in response to the tiny nature of PCB defect features, a BI-FU multi-scale feature fusion method is proposed. This method can extract multi-scale features supporting the centralized PCB and fuse the features into a high-quality feature map containing feature information at different PCB scales, thereby improving the feature extraction capability for small target defects.

[0024] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A flowchart of a small sample PCB defect detection method based on feature enhancement and multi-scale fusion provided in an embodiment of the present disclosure;

[0027] Figure 2 A schematic diagram of PCB surface defect categories provided by an embodiment of the present disclosure;

[0028] Figure 3 This is a schematic diagram of the structure of the FSRW model for small sample target detection in the prior art;

[0029] Figure 4 A schematic diagram of the structure of the small sample PCB defect detection model FAMD provided in an embodiment of the present disclosure;

[0030] Figure 5 A schematic diagram of the logical structure of the CBAM-based feature enhancement algorithm provided in an embodiment of the present disclosure;

[0031] Figure 6 A schematic diagram of the structure of the BI-FU multi-scale feature fusion module provided in an embodiment of the present disclosure;

[0032] Figure 7 A diagram showing the detection results of PCB surface defects using the detection method provided in an embodiment of the present disclosure;

[0033] Figure 8 This is a principle block diagram of a small sample PCB defect detection device based on feature enhancement and multi-scale fusion according to an embodiment of the present disclosure;

[0034] Figure 9 A block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0036] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0037] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0038] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0039] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0040] The present disclosure discloses a small sample PCB defect detection method based on feature enhancement multi-scale fusion, specifically, Figure 1 As shown, Figure 1 Flowchart of a small sample PCB defect detection method based on feature enhancement and multi-scale fusion in an embodiment of the present disclosure; the small sample PCB defect detection method based on feature enhancement and multi-scale fusion includes:

[0041] Step S1: constructing a base class dataset and a new class dataset; wherein the base class dataset includes sample images and their annotation information, and the new class dataset includes PCB images and their annotation information;

[0042] Step S2: dividing the base class dataset and the new class dataset into task datasets consisting of query set images and support set images respectively;

[0043] Step S3: Process each task data set as follows:

[0044] Step S31: Use the DarkNet-53 network to extract the feature map of the input query set image, then perform feature enhancement processing on the feature map, and output the enhanced query set feature map;

[0045] Step S32: using the BI-FU multi-scale feature fusion module to first extract multiple feature maps of different scales of the input support set image, and using the module to fuse the multiple feature maps of different scales to obtain a multi-scale fused feature map; then performing convolution processing on the multi-scale fused feature map to output a feature reweighted vector;

[0046] Among them, step S31 and step S32 are synchronous operations and can be performed in any order.

[0047] Step S33: fusing the query set feature graph and the feature reweighted vector to obtain a reweighted query set feature graph;

[0048] Exemplarily, the query set feature map and the feature reweighted vector are fused by channel-by-channel convolution, that is, there are c channels in total, and they are convolution-multiplied one by one according to the channels to obtain a reweighted query set feature map;

[0049] Step S34: performing prediction analysis based on the re-weighted query set feature graph to generate a prediction result including a predicted position and a predicted category;

[0050] Exemplarily, a prediction analysis is performed based on the reweighted query set feature graph, a bounding box is predicted using K-Means clustering, a category prediction is performed using multi-category cross entropy, a confidence level is determined using binary cross entropy, and finally a result of a predicted position and a predicted category is generated;

[0051] Step S4: adopt a two-stage training strategy, firstly perform model training using the task dataset of the base class dataset, and then perform model fine-tuning using the task dataset of the new class dataset;

[0052] Step S5: Detect PCB surface defects based on the trained model.

[0053] The disclosed embodiments provide a small-sample PCB defect detection method and device based on feature enhancement and multi-scale fusion. The method utilizes the DarkNet-53 network to enhance the feature extraction capability of query set images, fully utilizes the sample's own information, and alleviates the problem of insufficient sample information. Furthermore, by performing feature enhancement processing on the query set feature map, different weights are assigned within the query set feature map, highlighting key feature information in the feature map, suppressing interference from irrelevant information, enhancing the feature expression capability of the query sample, and improving the detection effect. Furthermore, in response to the tiny nature of PCB defect features, a BI-FU multi-scale feature fusion method is proposed. This method can extract multi-scale features supporting the centralized PCB and fuse the features into a high-quality feature map containing feature information at different PCB scales, thereby improving the feature extraction capability for small target defects.

[0054] Furthermore, in step S1 of the embodiment of the present disclosure, constructing the base class dataset and the new class dataset may specifically include:

[0055] Step S11, obtaining the FSOD dataset as a base class dataset;

[0056] Among them, the FSOD dataset is a professional small-sample target detection dataset with rich category diversity, covering many categories. Each category contains a large number of sample images and annotation information (annotation information usually contains the location data and category of the target in the image, which can generally be referred to by an annotation box and stored in XML format). Among them, the FSOD dataset is obtained directly from the Internet; for example, a FSOD dataset obtained from the Internet covers 1,000 categories in daily life, with a total of 66,502 images and 182,000 annotation boxes.

[0057] The advantage of using FSOD as the base dataset in this disclosed embodiment is that the model can better perform meta-learning, thereby improving the detection effect;

[0058] Step S12: Using the PCB finished product image with defect annotation information, a PCB defect image dataset is established as the new class dataset; wherein the defect annotation information is a labeling box of the defect in the PCB finished product image. Specifically, Figure 2 As shown, the defect category of the marked box can be at least one of missing holes, mouse bites, open circuits, short circuits, burrs, and excess copper.

[0059] Furthermore, in step S2 of the embodiment of the present disclosure, dividing the base class dataset and the new class dataset into task datasets consisting of query set images and support set images, respectively, may include: dividing the base class dataset into a task dataset consisting of query set images and support set images; and dividing the new class dataset into a task dataset consisting of query set images and support set images;

[0060] Specifically, each input data into the network (detection model) is called a task dataset T i , each task dataset consists of images in the query set Q and images in the support set S, where Q = {(I q ,L)}, Among them, I q represents an image in the query set Q, L is the label corresponding to the query set image, represents an image in the Nth category of the support set S, is the target position annotation area mask of the support set image (i.e. the corresponding label). Therefore, T i It can be expressed as:

[0061]

[0062] in is the query set image in the i-th detection task (i.e., the i-th task dataset), L i For its corresponding label, and In the i-th task, the n-th group (category) supports the set of images and corresponding masks.

[0063] Each task dataset includes a query set image and multiple support set images; wherein the multiple support set images are the same number of images extracted from each category image of the base class dataset or the new class dataset.

[0064] For example, a base dataset is used to partition several task datasets. The query set images in the task dataset are extracted from the base dataset, while the support set images are extracted from the base dataset excluding the query set images. The extraction principle is to extract the same number of k images from each category in the base dataset. Here, k is at least 1 natural number. Considering computer performance efficiency, k is generally not more than 50.

[0065] Taking the FSOD dataset covering 1000 categories used in an embodiment of the present disclosure as an example, the number of support set images in the task dataset divided from the FSOD dataset is at least 1000. For a PCB defect image dataset covering 6 defect categories in an embodiment of the present disclosure, the number of support set images in the task dataset divided from the PCB defect image dataset is at least 6.

[0066] In addition, to improve the training effect of the detection model, the algorithm can be used to avoid extracting the same query set images and the same support set images from different task datasets. Therefore, each task dataset divided by the base class dataset or the new class dataset has different query set images and different support set images.

[0067] Furthermore, if Figure 3-4 As shown, Figure 3 The following is a schematic diagram of the structure of the FSRW model for small sample target detection in the prior art. Figure 4 A schematic structural diagram of the small sample PCB defect detection model FAMD provided in an embodiment of the present disclosure is shown, corresponding to steps S31-S34 of the embodiment of the present disclosure.

[0068] In steps S31-S34 of the embodiment of the present disclosure, the query set image and the support set image are first input into the corresponding query set image branch and the support set image branch respectively:

[0069] In the query set image branch, the DarkNet-53 network is used to extract the feature map of the input query set image. The feature enhancement module is embedded to enhance the extracted feature map and output the enhanced query set feature map.

[0070] In the support set image branch, the BI-FU multi-scale feature fusion module is used to first extract multiple feature maps of different scales of the input support set image, and the module is used to fuse the multiple feature maps of different scales to obtain a multi-scale fused feature map; then the feature reweighting module is used to perform convolution processing on the multi-scale fused feature map and output a feature reweighted vector;

[0071] Then, the query set feature map and feature reweighted vector output by the two branches are fused to obtain the reweighted query set feature map;

[0072] Finally, the detection module performs prediction analysis based on the re-weighted query set feature map to generate the predicted position and predicted category results. The prediction results mainly include the predicted box (center point, size (height and width)), predicted category, category confidence, and target prediction score. For specific prediction results, please refer to Figure 7 Example.

[0073] Furthermore, in step S31 of the embodiment of the present disclosure, the DarkNet-53 network is used to extract the feature map of the input query set image, and the DarkNet-53 network is used to replace the original feature extraction module (such as DarkNet-19). In comparison, DarkNet53 borrows the idea of the feature pyramid network and introduces a residual mechanism to improve the feature extraction capability of the query set image. The main body replaces YOLOv2 with YOLOv3 as the basic detection model.

[0074] Furthermore, in step S31 of the embodiment of the present disclosure, feature enhancement processing is performed on the feature map to output the enhanced query set feature map, which may specifically include:

[0075] A feature enhancement module is embedded in the last layer of the feature extractor. This module receives the feature map of the query set image, learns the relationship between the foreground and background in the sample, and assigns different weights to the feature map to highlight important feature information and enhance the feature expression ability of the query sample.

[0076] Specifically, this module is constructed based on the attention mechanism. This example uses the CBAM attention mechanism, and its structure is as follows: Figure 5 As shown in Figure 2, the CBAM attention mechanism consists of two parts: CAM (Channel Attention Module) and SAM (Spatial Attention Module), which perform channel and spatial attention weighting respectively.

[0077] Furthermore, the input of the CBAM module is the feature map F extracted by the feature extractor, whose dimension is w×h×c. Using CBAM, we can derive the one-dimensional channel attention map: M c ∈R c×1×1 .

[0078] 2D spatial attention map: M S ∈R 1×h×w The overall process can be summarized as follows:

[0079]

[0080]

[0081] in represents element-wise multiplication, during which the attention values are propagated accordingly.

[0082] Specifically, if Figure 5 As shown, Figure 5 A schematic diagram of the logical structure of a CBAM-based feature enhancement algorithm provided in an embodiment of the present disclosure is shown.

[0083] (1) Channel information correction. For the input feature map F, global max pooling and global average pooling are performed on a channel-by-channel basis to obtain two different spatial context descriptors. and The two descriptors pass through a two-layer MLP module, and the output features are subjected to element-wise weighting operation and sigmoid activation operation to generate the final M c ∈R c×1×1 Finally, M c Element-wise with the feature map F to generate the intermediate feature F′. Channel attention calculation formula:

[0084]

[0085] Among them, σ is the sigmiod activation function, W0∈R c / r×c , W1∈R c×c / r The MLP weights W0 and W1 are shared for both inputs, and a ReLU activation function is placed before W0.

[0086] (2) Fusion of spatial information and channel information. Take the intermediate feature map F′ as the unit of its spatial position, perform global max pooling and global average pooling on its c-dimensional channel, and obtain two mappings: and The two maps are concatenated. A convolution kernel of size 7×7 is used to convolve the concatenated tensor and reduce the dimension to a w×h×1 channel. After sigmoid activation, M is generated. S ∈R 1×h×w Finally, element-wise with F′ is performed to obtain the final feature map F c . Spatial attention calculation formula:

[0087]

[0088] Among them, σ is the sigmiod activation function, f 7×7 It is a convolution operation with a convolution kernel of 7×7.

[0089] In simple terms, step S31 performs feature enhancement processing on the feature map and outputs the enhanced query set feature map. The feature map F can be calculated by the following two formulas in sequence to obtain the enhanced query set feature map F: c .

[0090] (1) Using the feature map F, we can obtain the intermediate feature map F′ according to the following formula:

[0091]

[0092]

[0093] Among them, σ is the sigmiod activation function, W0 and W1 are the weights of MLP, is the average pooling feature, is the maximum pooling feature, W0∈R c / r×c , W1∈R c×c / r , R is a real number, c is the number of channels, r is the reduction rate, Represents element-wise multiplication;

[0094] (2) Then use the intermediate feature map F′ to obtain the enhanced query set feature map F according to the following formula c :

[0095]

[0096]

[0097] Among them, σ is the sigmiod activation function, f 7×7 is a convolution operation with a convolution kernel of 7×7, is the average pooled 2D feature map, is the maximum pooled 2D feature map, Represents element-wise multiplication.

[0098] Furthermore, the BI-FU multi-scale feature fusion module is used to first extract feature maps of multiple scales of the input support set image, and the module is used to fuse the multiple feature maps of different scales to obtain a multi-scale fused feature map, which may include:

[0099] VGG16 and BiFPN Layer are used as the multi-scale feature extraction module BI, and multiple feature maps of different scales of the input support set image are generated by VGG16. The feature maps of multiple scales are then extracted by BiFPN Layer and used as the input of the feature fusion module FU; the feature maps of multiple scales output by the multi-scale feature extraction module BI are fused by the feature fusion module FU to obtain the multi-scale fused feature map.

[0100] Specifically, if Figure 6 As shown in the figure, the BI-FU multi-scale feature fusion module mainly consists of the feature extraction module BI and the adaptive feature fusion module FU. Among them, VGG16 and BiFPN layers serve as the basic building blocks of the multi-scale feature extraction BI. VGG16 consists of 13 convolutional layers and 5 pooling layers, forming 5 blocks (Block1-Block5). The feature map size in each block is reduced to 1 / 2 of the input size. Compared with the ordinary FPN, BiFPN integrates bidirectional cross-scale connections and fast normalization fusion. Its feature fusion formula is as follows:

[0101]

[0102] Among them, Resize is usually an upsampling or downsampling operation, which depends on whether it is reduced or enlarged in the multi-scale feature extraction BI. If the extraction part is completed by reduction, the fusion part is completed by upsampling, and vice versa.

[0103] Simply put, the VGG16 module extracts 5 feature maps of different scales. In the forward operation, the BiFPN Layer will support N images in the set to generate 5*N feature maps.

[0104] Furthermore, multiple feature maps of different scales are input into the FU feature fusion module, and its processing includes: performing a convolution operation on the multiple feature maps of different scales, compressing them to obtain a 1×1 tensor of the corresponding scale, forming a feature fusion vector, and then using the feature fusion vector to reweight the multiple feature maps of different scales. The reweighted feature maps of different scales are fused into a multi-scale fusion feature map through an element-by-element addition operation.

[0105] Specifically, in the feature fusion part, the network adaptively learns a feature fusion vector (w1, w2, w3, w4, w5). Each element of the feature fusion vector is compressed into a 1×1 tensor by performing a convolution operation on its corresponding feature map to obtain the feature fusion vector w k , k=1,2,3,4,5.

[0106] The five feature maps generated for each image are then reweighted using the feature fusion vector (w1, w2, w3, w4, w5). This reweighting process is then performed on the multi-scale feature maps. Finally, the five reweighted feature maps at different scales are fused into a multi-scale fused feature map through element-by-element addition. This BI-FU multi-scale feature fusion module outputs the multi-scale fused feature map. This feature map contains information at different scales and can seamlessly integrate low-resolution, strong semantic features with high-resolution, strong structural features. This allows the network to better extract the features of small defects in PCBs, improving the accuracy of the model's defect detection.

[0107] Furthermore, the performing convolution processing on the multi-scale fusion feature map and outputting the feature reweighted vector includes: using a lightweight convolutional network to perform a convolution operation on the input multi-scale fusion feature map to shape it into a feature reweighted vector.

[0108] Furthermore, the query set feature graph and the feature reweighted vector are fused to obtain a reweighted query set feature graph, including:

[0109] Specifically, the enhanced query set feature graph is fused using the feature reweighting vector according to the following formula:

[0110]

[0111] Among them, F c is the enhanced query set feature graph, ω i is the feature reweight vector, Represents channel multiplication based on 1×1 depth convolution, F i is the re-weighted query set feature map.

[0112] Furthermore, the small sample PCB defect detection method based on feature enhancement and multi-scale fusion in the embodiment of the present disclosure may further include, between step 4 and step 5:

[0113] Step 4.5: Test the model using query set images divided into query sets by the new class dataset, and judge the model performance through detection indicators.

[0114] Furthermore, in the embodiment of the present disclosure, step 4 may adopt a two-stage training strategy, firstly training the model using the task dataset of the base class dataset, and then fine-tuning the model using the task dataset of the new class dataset to update the parameters of each layer of the model.

[0115] Specifically, a two-stage training strategy is adopted during training. The first stage is the base class training stage, which uses base class data with sufficient label information and is organized in an Episode data format, with both the query set and the support set images input simultaneously. The feature extractor learns how to extract key meta-features from the query set images, and the reweighting module learns how to obtain reweighting coefficients from the support set features to reweight the query set features. The second stage is the fine-tuning stage, in which the new class data is trained using the same data organization method and follows the k-shot setting, that is, each class in the new class only contains k annotation information to fine-tune the model.

[0116] During training, the model receives both query and support set images. The query set image is fed into the meta-feature extraction module, where it is processed through the feature enhancement module to obtain an enhanced feature map. The support set image is fed into the BI-FU multi-scale feature fusion module, where it is processed through the feature reweighting module to obtain a reweighted feature vector. The reweighted feature vector is then convolved with the query set feature map channel-wise to obtain the reweighted query set feature map. This is then fed into the detection module to obtain the final defect location and classification results.

[0117] The detection model in the embodiment of the present disclosure can be adjusted and optimized through the following functions:

[0118] The model loss function L d Defined as:

[0119] L d =L c +L bbx +L obj

[0120] The classification score is corrected by the Softmax function, and the classification score of the i-th target category is c i , the actual classification score after correction The formula is:

[0121]

[0122] L c is a classification function used to determine whether the anchor box regressed at each position belongs to the current category and is defined as:

[0123]

[0124] In the formula Indicates whether the candidate box currently obtained belongs to the i-th target category.

[0125] The bounding box regression parameters are:

[0126]

[0127] Among them, λcoord is the bounding box weight coefficient, S 2 is the feature map area, B is the number of bounding boxes, It means that if there is a target in the bounding box at (i, j), its value is 1, otherwise it is 0.

[0128] L obj is the confidence loss function, defined as follows:

[0129]

[0130] Among them, λ noobj is the confidence weight coefficient, S 2 is the feature map area, B is the number of bounding boxes, It means that if there is a target in the bounding box at (i, j), its value is 1, otherwise it is 0.

[0131] The commonly used evaluation standard for target detection algorithms is mAP (Mean Average Precision), which is the average of AP under different intersection of Union (IOU) thresholds. The AP value is calculated by two indicators: precision and recall.

[0132]

[0133]

[0134]

[0135] Where area(A / B) represents the area of the A / B bounding box, TP represents true positives, FP represents false positives, and FN represents false negatives. The resulting curve, with Recall as the horizontal axis and Precision as the vertical axis, is called the PR curve, and the area under the curve is the accuracy (AP). Different PR curves are obtained at different IoU thresholds, starting from an IoU of 0.5, with a step size of 0.05, and ending at an IoU of 0.95. The mean AP is the average of 10 AP measurements.

[0136] Table 1 Detection accuracy of different small sample methods on PCB dataset (number of defects per category k = 10)

[0137]

[0138] Evaluation metrics include mAP, AP50, AP75, APS, APM, and APL. AP50 refers to the accuracy at an IoU of 0.5; AP75 refers to the accuracy at an IoU of 0.75. APS represents the average accuracy for defects smaller than 32×32 in an image; APM represents the average accuracy for defects between 32×32 and 96×96 in an image; and APL represents the average accuracy for defects larger than 96×96 in an image.

[0139] As shown in Table 1 above, compared to the original FSRW method, the proposed method achieved a 7.8% improvement in mAP when k = 10. AP50 reached 69.47% when k = 10, representing a 9.64% improvement in accuracy compared to the original FSRW model. For small, challenging-to-optimize defects, the APS improved by 4.38% in 10-shot detection compared to FSRW. Compared to several other small-sample detection methods, the proposed FAMD method also significantly outperformed several other methods, achieving superior accuracy across all evaluation metrics. These experimental results demonstrate the effectiveness and practical value of the proposed method.

[0140] The present disclosure also provides a small sample PCB defect detection device based on feature enhancement multi-scale fusion, specifically, Figure 8 As shown, the small sample PCB defect detection device 100 based on feature enhancement and multi-scale fusion includes: an acquisition module 101 for constructing a base class data set and a new class data set; wherein the base class data set includes sample images and their annotation information, and the new class data set includes PCB images and their annotation information; a task division module 102 for dividing the base class data set and the new class data set into task data sets consisting of query set images and support set images respectively; a task data processing module 103, including: a query set processing branch module 1031 for extracting a feature map of the input query set image using a DarkNet-53 network, then performing feature enhancement processing on the feature map, and outputting the enhanced query set feature map; a query set processing branch module 1031 for first extracting the input support set image using a BI-FU multi-scale feature fusion module Multiple feature maps of different scales are obtained by fusing the feature maps of different scales with the module to obtain a multi-scale fused feature map; the multi-scale fused feature map is then convolved to output a support set processing branch module 1032 of a feature reweighted vector; the query set feature map and the feature reweighted vector are fused to obtain a fusion module 1033 of a reweighted query set feature map; a prediction analysis is performed based on the reweighted query set feature map to generate a detection module 1034 of a prediction result including a predicted position and a predicted category; a two-stage training strategy is adopted, wherein the model is first trained using the task data set of the base class data set, and then the model is fine-tuned using the task data set of the new class data set; PCB surface defects are detected based on the trained model.

[0141] An electronic device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0142] The processor can be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to execute the computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of the small sample PCB defect detection method based on feature enhancement multi-scale fusion described in various embodiments of the present disclosure.

[0143] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0144] like Figure 9 The present invention provides a schematic structural diagram of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device according to an embodiment of the present invention. Figure 9 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0145] like Figure 9 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0146] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes and hard disks; and communication devices. The communication device allows the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 9 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0147] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the small sample PCB defect detection method based on feature enhancement multi-scale fusion of the embodiment of the present disclosure are performed.

[0148] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0149] According to an embodiment of the present disclosure, a computer-readable storage medium stores non-transitory computer-readable instructions. When executed by a processor, the non-transitory computer-readable instructions execute all or part of the steps of the small sample PCB defect detection method based on feature enhancement and multi-scale fusion described in each embodiment of the present disclosure.

[0150] The above-mentioned computer-readable storage media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0151] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0152] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0153] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0154] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0155] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0156] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0157] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0158] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A small sample PCB defect detection method based on feature enhancement and multi-scale fusion, characterized in that: include: Step 1: Construct a base class dataset and a new class dataset; wherein the base class dataset includes sample images and their annotation information, and the new class dataset includes PCB images and their annotation information; Step 2: Divide the base class dataset and the new class dataset into task datasets consisting of query set images and support set images respectively; Step 3: Process each task dataset as follows: Step 31: Use the DarkNet-53 network to extract the feature map of the input query set image, then perform feature enhancement processing on the feature map, and output the enhanced query set feature map; Step 32: Use the BI-FU multi-scale feature fusion module to first extract multiple feature maps of different scales of the input support set image, and use the module to fuse the multiple feature maps of different scales to obtain a multi-scale fused feature map; then perform convolution processing on the multi-scale fused feature map to output a feature reweighted vector; Step 33: fusing the query set feature graph and the feature reweighted vector to obtain a reweighted query set feature graph; Step 34: performing prediction analysis based on the re-weighted query set feature graph to generate a prediction result including a predicted position and a predicted category; Step 4: adopt a two-stage training strategy, firstly train the model using the task dataset of the base class dataset, and then fine-tune the model using the task dataset of the new class dataset; Step 5: Detect PCB surface defects based on the trained model; The performing feature enhancement processing on the feature map and outputting the enhanced query set feature map includes: The intermediate feature map F' is obtained using the following formula: Among them, σ is the sigmiod activation function, W0 and W1 are the weights of MLP, is the average pooling feature, is the maximum pooling feature, W0∈R c / r×c , W1∈R c×c / r , R is a real number, c is the number of channels, r is the reduction rate, Represents element-wise multiplication; Then use the intermediate feature map F' to obtain the enhanced query set feature map F according to the following formula c : Among them, σ is the sigmiod activation function, f 7×7 is a convolution operation with a convolution kernel of 7×7, is the average pooled 2D feature map, is the maximum pooled 2D feature map, Represents element-wise multiplication.

2. The small sample PCB defect detection method based on feature enhancement and multi-scale fusion according to claim 1 is characterized in that: The construction of the base class dataset and the new class dataset includes: The FSOD dataset is obtained as a base class dataset; and a PCB defect image dataset is established as the new class dataset using PCB finished product images with defect annotation information.

3. The small sample PCB defect detection method based on feature enhancement and multi-scale fusion according to claim 2 is characterized in that: The defect annotation information indicates that the defect type is at least one of missing holes, rat bites, open circuits, short circuits, burrs, and excess copper.

4. The small sample PCB defect detection method based on feature enhancement and multi-scale fusion according to claim 1 is characterized in that: The task dataset includes a query set image and multiple support set images; wherein the multiple support set images are the same number of images extracted from each category image of the base class dataset or the new class dataset.

5. The small sample PCB defect detection method based on feature enhancement and multi-scale fusion according to claim 1 is characterized in that: The BI-FU multi-scale feature fusion module is used to first extract multiple feature maps of different scales of the input support set image, and the module is used to fuse the multiple feature maps of different scales to obtain a multi-scale fused feature map, including: Use VGG16 and BiFPN Layer as the multi-scale feature extraction module BI, generate multiple feature maps of different scales from the input support set image through VGG16, and then use BiFPN Layer to extract multiple feature maps of different scales and use them as the input of the feature fusion module FU; The feature fusion module FU fuses the multiple feature maps of different scales output by the multi-scale feature extraction module BI to obtain the multi-scale fused feature map.

6. The small sample PCB defect detection method based on feature enhancement and multi-scale fusion according to claim 5 is characterized in that: The feature fusion module FU fuses the feature maps of multiple scales output by the multi-scale feature extraction module BI to obtain the multi-scale fused feature map, including: The multiple feature maps of different scales are convolved to obtain 1×1 tensors of corresponding scales, which are used to form feature fusion vectors. The feature fusion vectors are then used to reweight the multiple feature maps of different scales. The reweighted feature maps of different scales are fused into a multi-scale fusion feature map through element-by-element addition.

7. The small sample PCB defect detection method based on feature enhancement and multi-scale fusion according to claim 1 is characterized in that: The convolution processing is performed on the multi-scale fusion feature map to output a feature reweighted vector, including: A lightweight convolutional network is used to perform convolution operations on the input multi-scale fusion feature map and shape it into a feature reweighted vector.

8. The small sample PCB defect detection method based on feature enhancement and multi-scale fusion according to claim 1 is characterized in that: Between step 4 and step 5, it also includes: Step 4.5: Test the model using query set images divided into query sets by the new class dataset, and judge the model performance through detection indicators.

9. A small sample PCB defect detection device based on feature enhancement and multi-scale fusion, characterized in that: include: An acquisition module is used to construct a base class data set and a new class data set; wherein the base class data set includes sample images and their annotation information, and the new class data set includes PCB images and their annotation information; A task division module, configured to divide the base class dataset and the new class dataset into task datasets consisting of query set images and support set images respectively; Mission data processing module, including: The query set processing branch module is used to extract the feature map of the input query set image using the DarkNet-53 network, then perform feature enhancement on the feature map, and output the enhanced query set feature map; The support set processing branch module uses the BI-FU multi-scale feature fusion module to first extract multiple feature maps of different scales from the input support set image, and uses this module to fuse the multiple feature maps of different scales to obtain a multi-scale fused feature map; then convolution processing is performed on the multi-scale fused feature map to output a feature reweighted vector; A fusion module, configured to fuse the query set feature map and the feature reweighted vector to obtain a reweighted query set feature map; A detection module, configured to perform prediction analysis based on the reweighted query set feature graph and generate a prediction result including a predicted position and a predicted category; A two-stage training strategy is adopted, wherein the model is first trained using the task dataset of the base class dataset, and then the model is fine-tuned using the task dataset of the new class dataset; PCB surface defects are detected based on the trained model; The performing feature enhancement processing on the feature map and outputting the enhanced query set feature map includes: The intermediate feature map F' is obtained using the following formula: Among them, σ is the sigmiod activation function, W0 and W1 are the weights of MLP, is the average pooling feature, is the maximum pooling feature, W0∈R c / r×c , W1∈R c×c / r , R is a real number, c is the number of channels, r is the reduction rate, Represents element-wise multiplication; Then use the intermediate feature map F' to obtain the enhanced query set feature map F according to the following formula c : Among them, σ is the sigmiod activation function, f 7×7 is a convolution operation with a convolution kernel of 7×7, is the average pooled 2D feature map, is the maximum pooled 2D feature map, Represents element-wise multiplication.

Images