PCB defect detection method based on small target enhanced feature pyramid

Abstract

The invention discloses a PCB defect detection method based on a small target enhanced feature pyramid, and the method comprises the steps: S1, data collection and processing: downloading a PCB defect data set for screening, carrying out the data processing of a screened picture, and dividing the processed data set into a training set and a verification set; s2, constructing and training a model: constructing an improved RT-DETR-R18 model and carrying out model training to obtain the weight of the improved RT-DETR-R18 model, and then carrying out model verification; and S3, detection and visualization: deploying the verified model to a visual interface for real-time detection of PCB defects. According to the method, an improved RT-DETR-R18 model algorithm is adopted, and higher accuracy and robustness are achieved; and the improved network structure also has relatively high hardware adaptability and compatibility, and can be used in PCB defect detection tasks in different scenes.

Description

Technical Field

[0001] The present invention belongs to the field of novel artificial intelligence technology, and in particular relates to a PCB defect detection method based on a small target enhanced feature pyramid. Background Art

[0002] Printed circuit boards (PCBs) are carriers of various electronic components and play a vital role in electronic products. With the rapid progress of the PCB industry, the new generation of intelligent electronic devices are becoming smaller and more powerful, which undoubtedly puts forward stricter standards for the production and manufacturing of PCBs. If defective PCBs enter the subsequent production process, it may affect the performance and quality of the final electronic products, resulting in unnecessary waste of resources.

[0003] The traditional PCB defect detection method is mainly manual detection, which uses magnifying glasses and other instruments to assist detection and uses human eyes to identify PCB defects. However, this method is time-consuming and labor-intensive, with high labor costs. The detection accuracy is affected by personal subjective consciousness and work hours. Moreover, as PCBs become more and more complex, it is no longer practical to use human eyes to identify defects.

[0004] Electrical testing involves contacting the circuit on the PCB with a probe and then measuring the electrical quantity on the circuit to identify defects. This method is not efficient and may cause secondary damage to the product. Currently, it is not used in many scenarios. Automatic optical inspection (AOI) technology uses an optical system to capture images of the inspected object and then uses image processing technology to analyze these images to identify and classify defects. The inspection results are easily affected by factors such as light source and camera imaging quality, and AOI may not be able to detect some hidden or obscured defects.

[0005] With the development of deep learning, target detection algorithms have begun to be applied to PCB defect detection, from the two-stage algorithm represented by the R-CNN series, which is mainly divided into two processes. The first step is to propose several areas that may contain objects, and the second step is to run a classification network on the proposed areas to obtain the object category in each area. To the one-stage algorithm represented by the YOLO series, it directly separates specific categories and regresses boundaries. It is faster than the two-stage detection method, but the accuracy is slightly lower.

[0006] The above PCB defect detection methods have some disadvantages: manual detection method relies on manual inspection, but this method is time-consuming and labor-intensive, with high labor costs, and detection is subject to subjective influence of personnel. Electrical test detection efficiency is not high, and it may also cause secondary damage to the product. The detection results of automatic optical inspection technology are easily affected by factors such as light source and camera imaging quality, and some hidden or obscured defects may not be detected. The target detection algorithm based on deep learning requires a large number of pictures to learn defect features before detection. In complex or harsh environments, the performance of the algorithm may decline.

[0007] Chinese patent document CN 118587188 A discloses a PCB small target defect detection method based on improved YOLOv8s. By adding a small target detection layer and a detection head, the recognition rate of small targets is improved. S-Conv and CAFM are introduced into the backbone network to fuse multi-scale features and enhance feature representation. An efficient spatial pyramid pooling layer is designed, and deformable convolution and spatial information enhancement modules are combined to improve the target feature representation capability. The neck structure is optimized and the information aggregation-distribution mechanism is adopted. The VFWD-CIoU loss function is adopted to enhance the performance of dense small target detection. Real-time and efficient detection of PCB small target defects is achieved, laying the foundation for the application of deep learning in edge devices. By improving YOLOv8s, the recognition rate and accuracy problems in PCB defect detection are solved, and the efficiency and accuracy of detection are improved. Although the accuracy and efficiency of PCB defect detection have been improved, there is still room for improvement.

[0008] Therefore, it is necessary to provide a PCB defect detection method based on small target enhanced feature pyramid with higher accuracy and robustness. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a PCB defect detection method based on a small target enhanced feature pyramid, which adopts an improved RT-DETR-R18 model algorithm and has higher accuracy and robustness; and the improved network structure also has strong hardware adaptability and compatibility, and can be used for PCB defect detection tasks in different scenarios.

[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is: the PCB defect detection method based on small target enhanced feature pyramid specifically comprises the following steps:

[0011] S1 Data collection and processing: Download PCB defect data sets for screening, perform data processing on the screened images, and divide the processed data sets into training sets and validation sets;

[0012] S2 builds and trains the model: builds the improved RT-DETR-R18 model and performs model training to obtain the weights of the improved RT-DETR-R18 model, and then performs model verification;

[0013] S3 Detection and Visualization: Deploy the verified model to the visualization interface for real-time detection of PCB defects.

[0014] Using the above technical solution, the deep learning method adopted can automatically learn and extract features, adapt to different PCB defects and environmental conditions, and improve on the RT-DETR-R18 model. The RT-DETR-R18 end-to-end model architecture has the advantages of simple structure, easy deployment, and relatively few parameters, making its detection algorithm more accurate and robust. The improved network structure also has strong hardware adaptability and compatibility, and can be used in PCB defect detection tasks in different scenarios. Compared with other image detection algorithms and lightweight target detection algorithms, the improved algorithm in this paper demonstrates its effectiveness and feasibility.

[0015] Preferably, the specific steps of step S1 are:

[0016] S11: download the PCB defect dataset, obtain images from the PCB defect dataset, and filter the images to remove unlabeled data to obtain a processed dataset;

[0017] S12: Divide the screened images into training set and validation set;

[0018] S13: The data of the training set is expanded through data enhancement.

[0019] The data enhancement operations include affine transformation, cropping, elastic transformation, mesh distortion, or perspective transformation. The training set is expanded through a series of data enhancement methods to extract more abundant PCB defect features.

[0020] Preferably, the specific steps of step S2 are:

[0021] S21 Model building: Building an improved RT-DETR-R18 model;

[0022] S22 model training: training the improved network model using the training set processed in step S1 by setting training hyperparameters;

[0023] S23 Model Verification: Set verification parameters and use the verification set to verify the trained model.

[0024] Preferably, in step S21, the modules in the RT-DETR-R18 model are improved to obtain an improved RT-DETR-R18 model, and the specific steps of improving the modules are:

[0025] S211: First, the backbone network is replaced with the RMT network, and the prior knowledge related to the spatial distance is introduced into the model; the attention is decomposed horizontally and vertically by decomposing MasA along the two axes of the image, so that the decomposed MaSA has linear complexity in modeling global information and has the same receptive field shape as the original MaSA; a large amount of calculation is reduced;

[0026] S212: First, the S2 feature layer is subjected to SPDConv to obtain features rich in small target information, and then the process goes to step S213 for feature fusion;

[0027] S213: Replace the second Fusion module in the CCFM module with the designed CO-Fusion module, that is, add the designed CSPOKM module (CSP Omni-Kernel Module) after concat, and send the features of the S2 feature layer rich in small target information and the S3 feature layer and the previous layer features to the designed fusion module for feature fusion. The small target here usually refers to a target object with a small size and a relatively small pixel ratio. The definition of small is usually considered to be a small target with a size less than 32*32 pixels.

[0028] This operation is to fully obtain the information of small targets. The S2 feature layer is downsampled through SPDConv to retain all information in the channel dimension and the details required for small targets. For most networks, the S3, S4, and S5 feature layers are usually input for further processing. However, small targets are more difficult to detect on the S3, S4, and S5 feature layers because some details are lost. Therefore, adding the S2 detection layer and integrating it through SPDConv reduces the large amount of calculation and more time-consuming post-processing. At the same time, in order to effectively utilize the features processed by the S2 feature layer through SPDConv, the second Fusion fusion module in CCFM is replaced with the CO-Fusion fusion module designed by us, that is, the CSPOKM (CSPOmni-Kernel Module) module designed is added after concat to fuse the features obtained from S2, S3, and the previous layer, effectively learning the feature representation from global to local, and improving the model feature representation ability. Such fusion effectively learns the feature representation from global to local; it realizes the improvement of the module based on RT-DETR-R18, thereby obtaining the improved model.

[0029] Preferably, in step S211, the RMT network is constructed based on Manhattan Self-Attention MasA, and the constructed RMT network is divided into four stages, the first three stages use the decomposed MaSA, and the last stage uses the original MaSA; in each retention network combined with the visual transformer module RMT block, the input feature first passes through the DWConv residual connection, then passes through the layer normalization LN and MaSA residual connection, and finally passes through a layer normalization LN and the residual connection of the forward propagation network FFN; wherein MaSA enhances self-attention through the spatial attenuation matrix of the Manhattan distance, and the MaSA formula is:

[0030] MaSA(X)=(Softmax(QK T )⊙D 2d )V;

[0031]

[0032] Among them, softmax is a normalized exponential function, Q is the query vector of the input feature, and K T is the transpose of the input feature key vector, D 2d refers to the two-bit attenuation matrix, V is the value vector of the input feature, γ is a scalar, (x n ,y n ) is the two-dimensional coordinate of the nth marker, (x m ,y m ) is the two-dimensional coordinate of the mth marker.

[0033] The RMT network flexibly integrates explicit spatial priors into the visual backbone based on the spatial attenuation matrix and attention decomposition, and reduces a lot of computation to only linear complexity.

[0034] Preferably, in step S211, the method of decomposing MasA along two axes of the image is to calculate the attention scores in the horizontal direction and the vertical direction of the image respectively, obtain the attention weights, and then apply a one-dimensional bidirectional attenuation matrix to the attention weights, where the one-dimensional attenuation matrix represents the horizontal and vertical distances between the markers; the formula of the decomposed form of MaSA_part is:

[0035]

[0036] MaSA_part(X) = Attn H (Attn W V) T ;

[0037] Among them, Attn H represents vertical attention, QH It is a vertical query. is the transpose of the vertical key, D H is a one-dimensional bidirectional attenuation matrix in the vertical direction, where Attn W represents horizontal attention, Q W It is a horizontal query. is the transpose of the horizontal key, D W is a one-dimensional bidirectional attenuation matrix in the horizontal direction, γ is a scalar, (x n ,y n ) is the two-dimensional coordinate of the nth marker, (x m ,y m ) is the two-dimensional coordinate of the mth mark, X is the input feature, and V is the value vector of the input feature.

[0038] Preferably, the specific steps of obtaining features rich in small target information by convolution of the S2 feature layer through SPDConv in step S212 are:

[0039] For any input feature X of size S×S×C1, the feature is split into scale by the scale factor scale 2 A series of sub-features, where the scale factor scale = 2, then these 4 sub-features are expressed as:

[0040] f 0,0 =X[0:S:2,0:S:2],f 1,0 =X[1:S:2,0:S:2];

[0041] f 0,1 =X[0:S:2,1:S:2],f 1,1 =X[1:S:2,1:S:2];

[0042] Among them, S is the width and height of the input feature, and the shape of each sub-feature is Downsample the input feature X by a factor of 2 and concatenate the four sub-features along the channel dimension to obtain the feature map X ′ , the feature space dimension has a scale factor, and the channel dimension has an additional scale 2 The scaling factor of ; that is, the original input feature X is transformed from (S, S, C1) to the intermediate feature After feature conversion, a convolution with a kernel of 3×3 and stride=1 is used to make the final channel dimension C2 <sacle 2 C1. The reason for using a convolution with a stride of 1 is to retain as much discriminative feature information as possible; the SPD layer downsamples the input features but retains all information in the channel dimension, so no information is lost.

[0043] Preferably, the specific steps of step S213 are:

[0044] S2131: First, the CO-Fusion fusion module performs concat feature fusion after receiving the processed S2 feature layer, S3 feature layer and the previous layer feature, and sends the fused features to the CSPOKM module;

[0045] S2132: Then, two 1×1 convolutions are used to adjust the number of channels. One of the branches uses N repblocks for feature fusion. Finally, the two parts of the features are added together to obtain the final feature and output it.

[0046] Preferably, the CSPOKM module in step S213 combines the OKM module with the CSP structure. The input feature is divided into two parts after a 1×1 convolution, one part accounts for 0.25 and is input into the OKM module branch, and the other part accounts for 0.75. Finally, the two parts of the feature are connected by concat residual. The number n of OKM modules is taken as 1, and the fused feature is sent to a 1×1 convolution to adjust the number of output channels. Given the input feature X∈R C×H×W After 1×1 convolution, the features are sent to three branches, namely local branch, large branch and global branch, to enhance multi-scale representation; the results of the three branches are finally added and fused, and the channels are adjusted by another 1×1 convolution; among them, the large branch obtains a large receptive field by applying cheap deep convolution with kernel size K×K, and also uses 1×K and K×1 deep convolutions to obtain strip-like context information;

[0047] The global modeling function is added in the global branch through dual-domain processing; wherein the global branch includes a dual-domain channel attention module DCAM and a frequency-based spatial attention module FSAM. In the DCAM module, for the input feature X Global ∈R C×H×W , first apply the frequency channel attention module FCA to the feature, the process formula of the FCA module is:

[0048]

[0049] in, and are fast Fourier transform and its inverse operation, X FCA represents the output features after the FCA module, X Global refers to the features input to the global branch, It refers to the operation of 1×1 convolution in FCA, and GAP is global average pooling. Through Fourier processing, the global features of the image are effectively refined according to the spectral convolution theorem. The obtained features are then further sent to the spatial channel attention module SCA in the DCAM module. Therefore, the DCAM module process is finally expressed as:

[0050]

[0051] Among them, X DCAM is the output feature of the DCAM module, X FCA is the output feature of the FCA module, It is the 1×1 convolution operation in the SCA module, and GAP is the global average pooling;

[0052] The DCAM module only enhances the dual-domain features in a coarse-grained manner in the channel direction; then the frequency-based attention module is applied in the spatial dimension through the frequency-based spatial attention module FSAM to refine the spectrum at a fine-grained level. The formula of the frequency-based spatial attention module FSAM process is:

[0053]

[0054] Among them, X FSAM is the output feature of the FSAM module, and are the fast Fourier transform and its inverse operation, and They are the 1×1 convolution operations on both sides, X DCAM It is the output feature of the DCAM module. Through DCAM and FSAM, this local branch focuses on the information frequency of high-quality image reconstruction.

[0055] Preferably, the specific steps of step S23 are:

[0056] Verification parameters are set to verify the model trained in step S22 on the verification set, and mAP@0.5, Precision, Recall, Parameters, and Gflops are used as evaluation indicators, where mAP@0.5 is the average precision of the detection results of each type, Precision is the proportion of correct targets among the targets detected by the model; Recall is the proportion of targets correctly identified by the model among the total number of all real targets, Parameters is used to measure the size and complexity of the model; Gflops is used to measure the execution time of the model; the formulas of each indicator are:

[0057]

[0058] Among them, TP is the number of correctly identified targets, FP is the number of incorrectly identified targets, FN is the number of missed targets, P(r) represents the smooth precision-recall curve, and the area under the smooth curve obtained by integrating it is AP; N represents the number of detected object categories.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] (1) High accuracy: The improved network further improves the detection accuracy by improving and adding different modules;

[0061] (2) Real-time performance: RT-DETR-R18 is a target detection algorithm designed for real-time applications, and our improved model has fewer parameters, so the entire system can run efficiently in real-time scenarios and meet the needs of real-time PCB defect detection;

[0062] (3) Lightweight: By replacing the backbone network with an RMT network, the attention is decomposed horizontally and vertically, so that the decomposed MaSA has linear complexity in modeling global information, reducing a lot of calculations; and retaining spatial prior knowledge, reducing a lot of parameters, and the three improved modules have the effect of reducing parameters, making the model more lightweight; at the same time, all information in the channel dimension retains the detail information required for small targets; the designed CO-Fusion fusion module effectively learns feature representation from global to local, and CSPOKM uses the CSP structure to replace BottleNeck with OKM, effectively improving the feature representation ability of the model;

[0063] (4) Universality: The PCB defect detection method based on small target enhanced feature pyramid can be applied to PCB defect detection tasks under various types of cameras and scenarios, and has strong versatility and adaptability.

[0064] (5) Easy to deploy: RT-DETR-R18 is used as the benchmark model. Its end-to-end characteristics reduce many post-processing steps and are easy to deploy. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the overall algorithm flow in the PCB defect detection method based on the small target enhanced feature pyramid of the present invention;

[0066] Figure 2 It is a structural diagram of the improved RT-DETR-R18 model in the PCB defect detection method based on small target enhanced feature pyramid of the present invention;

[0067] Figure 3 It is a diagram of the RMT network structure in the PCB defect detection method based on the small target enhanced feature pyramid of the present invention;

[0068] Figure 4 It is a structural diagram of SPDConv in the PCB defect detection method based on small target enhanced feature pyramid of the present invention;

[0069] Figure 5 It is a structural comparison diagram of Fusion and CO-Fusion in the PCB defect detection method based on small target enhanced feature pyramid of the present invention; Figure 5 (a) is the structural diagram of the Fusion module; Figure 5 (b) is the structural diagram of the CO-Fusion fusion module;

[0070] Figure 6 It is a structural diagram of the CSPOKM module in the PCB defect detection method based on small target enhanced feature pyramid of the present invention;

[0071] Figure 7 It is a structural diagram of the OKM module in the PCB defect detection method based on small target enhanced feature pyramid of the present invention. DETAILED DESCRIPTION

[0072] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0073] The known technical content that the present invention is based on is the RT-DETR target detection algorithm.

[0074] RT-DETR technical features: RT-DETR is a single-stage target detection algorithm that combines the ideas of Transformer and DETR and adopts an end-to-end training method. It can learn target detection tasks directly from raw images without additional preprocessing steps. While maintaining high accuracy, it can achieve faster inference speed;

[0075] Benchmark network: The backbone network of RT-DETR is mainly ResNet and HGNetv2;

[0076] The basic building block of ResNet is the residual block, each of which contains two or more convolutional layers and one or more skip connections. These skip connections allow the input to directly skip one or more layers and be added to the output of the subsequent layer. This structure enables the network to learn the identity mapping, thereby simplifying the training of deep networks. The core innovation of ResNet is the introduction of the concept of "residual learning", which solves the gradient vanishing and gradient exploding problems that are prone to occur in the training process of deep neural networks, making it possible to build deeper networks. The ResNet models selected in RT-DETR are 18, 34, 50, 101;

[0077] HGNet is composed of multiple HG-Blocks, which are the core components of the network and are designed to process data in a hierarchical manner. Each HG block may process different levels of abstraction of the data, allowing the network to learn from low-level and high-level features. LDS Layer is added between blocks to perform downsampling operations, reduce the spatial dimension of the feature map, reduce the computational load and possibly increase the receptive field of subsequent layers. HGNetV2 is an improved version of HGNet, which improves the performance and generalization ability of the model by introducing probabilistic methods and multi-scale feature fusion. In RT-DETR, the l and x models use HGNetv2.

[0078] In the present invention, the features {S3, S4, S5} of the last three stages of the backbone are input into the encoder;

[0079] Efficient hybrid encoder: The encoder consists of two modules, namely, attention-based intra-scale feature interaction (AIFI) and CNN-based cross-scale feature fusion (CCFF);

[0080] AIFI further reduces computational cost by performing intra-scale interactions only on S5 using a single-scale transformer encoder. This is because applying self-attention to semantically rich high-level features captures the connections between conceptual entities, facilitating the localization and recognition of objects by subsequent modules. However, intra-scale interactions of low-level features are not performed due to the lack of semantic concepts and the risk of duplication and confusion in interacting with high-level features;

[0081] CCFF is optimized based on the cross-scale fusion module, which inserts multiple fusion blocks composed of convolutional layers in the fusion path. The function of the fusion block is to fuse two adjacent scale features into a new feature. The fusion block contains 2 1×1 convolutions to adjust the number of channels, uses N repblocks composed of RepConv for feature fusion, and fuses the two outputs by element addition;

[0082] Minimum Uncertainty Query Selection: Use minimum uncertainty query selection to select a fixed number of encoder features as the initial object query for the decoder. This scheme explicitly constructs and optimizes epistemic uncertainty to model the joint latent variables of the encoder features, thereby providing high-quality queries for the decoder.

[0083] Decoder: It uses deformable self-attention and connects several MLPs as detection heads. Deformable self-attention can significantly reduce the amount of computation. At the same time, this part also uses CUDA operators to speed up the operation. The design of the decoder allows RT-DETR to learn end-to-end directly from image pixels to bounding boxes and category predictions without additional post-processing steps such as non-maximum suppression, and is easy to deploy;

[0084] In terms of training loss, the regression loss is still GIoU loss and L1 loss, while the category loss introduces "IoU soft label", so variable focal loss is used.

[0085] Example: Figure 1 As shown, the PCB defect detection method based on small target enhanced feature pyramid specifically includes the following steps:

[0086] S1 Data collection and processing: Download PCB defect data sets for screening, perform data processing on the screened images, and divide the processed data sets into training sets and validation sets;

[0087] The specific steps of step S1 are:

[0088] S11: download the PCB defect dataset published by Peking University, obtain pictures from the PCB defect dataset, and filter the pictures to remove the unlabeled data to obtain a processed dataset; in this embodiment, 693 pictures are finally filtered, including 6 types of defects (missing holes, mouse bites, open circuits, short circuits, strays, and pseudo copper);

[0089] S12: randomly divide the screened images into a training set and a validation set in a ratio of 9:1; in this embodiment, the training set contains 554 images and the validation set contains 139 images;

[0090] S13: the data of the training set is expanded (doubled) by data enhancement, that is, the number of images in the training set is increased to 2 times; wherein the data enhancement operation includes affine transformation or cropping or elastic transformation or grid distortion or perspective transformation;

[0091] In this embodiment, the training set data is doubled through data enhancement. The data enhancement operations include applying affine transformation (including scaling, translation, rotation and shearing); performing safe random cropping; applying transformations of the D4 symmetry group, which includes rotation and reflection of square images; applying elastic transformation, which is an effect of simulating the deformation of images in space; flipping images horizontally and vertically; applying grid distortion, which is an effect of producing local distortion by randomly moving grid intersections; performing four-point perspective transformation, which is a perspective effect of simulating images in three-dimensional space. The final training set randomly performs the above data enhancement, and the final expanded training set contains 1108 images;

[0092] S2 builds and trains the model: builds the improved RT-DETR-R18 model and performs model training to obtain the weights of the improved RT-DETR-R18 model, and then performs model verification;

[0093] The specific steps of step S2 are:

[0094] S21 Model building: Building an improved RT-DETR-R18 model;

[0095] In step S21, the modules in the RT-DETR-R18 model are improved to obtain an improved RT-DETR-R18 model. The improved model structure diagram is shown in FIG. Figure 2 As shown in the figure, the specific steps to improve the module are:

[0096] S211: First, the backbone network is replaced with an RMT (Retention Network Combined with Visual Transformer) network, and the prior knowledge related to spatial distance is introduced into the model; the attention is decomposed horizontally and vertically by decomposing MasA along the two axes of the image, so that the decomposed MaSA has linear complexity in modeling global information and has the same receptive field shape as the original MaSA; reducing a lot of calculations;

[0097] In the step S211, the RMT network is constructed based on Manhattan Self-Attention MasA. The constructed RMT network is divided into four stages, the first three stages use the decomposed MaSA, and the last stage uses the original MaSA; in each RMT block (retaining the network combined with the visual transformer module), the input feature first passes through the DWConv (deep convolution) residual connection, then passes through the LN (layer normalization) and MaSA residual connection, and finally passes through a LN and FFN (forward propagation network) residual connection; the RMT network is based on the spatial attenuation matrix and the attention decomposition form, flexibly integrates the explicit spatial prior into the visual backbone, and reduces a lot of calculations, with only linear complexity;

[0098] RMT network structure diagram Figure 3 As shown; MaSA enhances self-attention through the spatial attenuation matrix of Manhattan distance, and the MaSA formula is:

[0099] MaSA(X)=(Softmax(QK T )⊙D 2d )V;

[0100]

[0101] Among them, softmax is a normalized exponential function, Q is the query vector of the input feature, and K T is the transpose of the input feature key vector, D 2d refers to the two-bit attenuation matrix, V is the value vector of the input feature, γ is a scalar, (x n ,y n ) is the two-dimensional coordinate of the nth marker, (x m ,y m ) is the 2D coordinate of the mth marker;

[0102] Because the problem of self-attention global modeling is a huge computational burden, in order to sparsely model global information without affecting the spatial attenuation matrix, a method of decomposing MaSA along two axes of the image is proposed. This decomposition method decomposes the self-attention matrix and the spatial attenuation matrix without losing prior information. The decomposed MaSA has linear complexity in modeling global information and has the same receptive field shape as the original MaSA;

[0103] In step S211, the method of decomposing MasA along two axes of the image is specifically to calculate the attention scores in the horizontal direction and the vertical direction of the image respectively, obtain the attention weights, and then apply a one-dimensional bidirectional attenuation matrix to the attention weights, where the one-dimensional attenuation matrix represents the horizontal and vertical distances between the markers; the formula of the decomposed form of MaSA_part is:

[0104]

[0105]

[0106] MaSA_part(X)=Attn H (Attn W V) T ;

[0107] Among them, Attn H represents vertical attention, Q H It is a vertical query. is the transpose of the vertical key, DH is a one-dimensional bidirectional attenuation matrix in the vertical direction, where Attn W represents horizontal attention, Q W It is a horizontal query. is the transpose of the horizontal key, D W is a one-dimensional bidirectional attenuation matrix in the horizontal direction, γ is a scalar, (x n ,y n ) is the two-dimensional coordinate of the nth marker, (x m ,y m ) is the two-dimensional coordinate of the mth mark, X is the input feature, and V is the value vector of the input feature;

[0108] S212: In order to fully obtain the information of small targets, firstly, the S2 feature layer is passed through SPDConv (space-to-depth convolution) to obtain features rich in small target information, and then transferred to step S213 and the S3 feature layer and the previous layer features are sent to the designed fusion module for feature fusion; the S2 feature layer is downsampled through SPDConv to retain all information in the channel dimension and the detail information required for small targets; the small targets here usually refer to target objects with small size and relatively small pixel proportion. The definition of small is usually considered to be small targets with a size less than 32*32 pixels;

[0109] In order to fully obtain the small target information, the specific steps of obtaining features rich in small target information by convolution of the S2 feature layer through SPDConv in step S212 are as follows:

[0110] For most networks, the S3, S4, and S5 feature layers are usually input for further processing. However, small targets are more difficult to detect on the S3, S4, and S5 feature layers because some detail information is lost. Therefore, the S2 detection layer is added in the present invention and integrated through SPDConv to reduce the large amount of calculation and the time-consuming post-processing. The SPDConv structure diagram is shown in Figure 4 As shown;

[0111] For any input feature X of size S×S×C1, the feature is split into scale by the scale factor scale 2 A series of sub-features, where the scale factor scale = 2, then these 4 sub-features are expressed as:

[0112] f 0,0 =X[0:S:2,0:S:2],f 1,0 =X[1:S:2,0:S:2];

[0113] f 0,1 =X[0:S:2,1:S:2],f 1,1=X[1:S:2,1:S:2];

[0114] Among them, S is the width and height of the input feature, and the shape of each sub-feature is The input feature X is downsampled by 2 times. Next, the four sub-features are concatenated along the channel dimension to obtain the feature map X ′ , the feature space dimension has a scale factor, and the channel dimension has an additional scale 2 The scaling factor of ; that is, the original input feature X is transformed from (S, S, C1) to the intermediate feature After feature conversion, a convolution with a kernel of 3×3 and stride=1 is used to make the final channel dimension C2 <sacle 2 C1; The reason for using a convolution with a stride of 1 in this embodiment is to retain as much discriminative feature information as possible; the SPD layer downsamples the input features but retains all information in the channel dimension, so no information is lost;

[0115] S213: Replace the second Fusion (fusion module) in CCFM (cross-scale feature fusion module) with the CO-Fusion fusion module designed by the present invention, that is, add the designed CSPOKM module (CSP Omni-Kernel Module) after the concat feature splicing, and send the features of the S2 feature layer rich in small target information and the S3 feature layer and the previous layer features into the designed fusion module for feature fusion; such fusion effectively learns feature representation from global to local; realizes module improvement based on RT-DETR-R18, thereby obtaining an improved model; the comparison diagram of the Fusion module and the CO-Fusion module is shown in FIG. Figure 5 As shown; Figure 5 (a) is the structural diagram of the Fusion module; Figure 5 (b) is the structural diagram of the CO-Fusion fusion module; Figure 5 In Concatenate stands for Concatenate (feature concatenation); Conv stands for Conv1×1+BN+SiLU, BN is batch normalization, SiLU is the activation function, Indicates Element-wise add (element-by-element addition); Expressed as Flatten;

[0116] The specific steps of step S213 are:

[0117] S2131: First, the CO-Fusion fusion module performs concat feature fusion after receiving the processed S2 feature layer, S3 feature layer and the previous layer feature, and sends the fused features to the CSPOKM module;

[0118] S2132: Then adjust the number of channels through two 1×1 convolutions, one of which uses N repblocks for feature fusion, and finally the two parts of features are added together to obtain the final feature and output;

[0119] The CSPOKM module in step S213 combines the OKM (full kernel module) with the CSP structure. The CSPOKM module structure is as follows: Figure 6 As shown, Figure 6 In Concatenate means Concatenate; Conv means Conv1×1+BN+SiLU. The input feature is divided into two parts after a 1×1 convolution. One part accounts for 0.25 and is input into the OKM module branch, and the other part accounts for 0.75. Finally, the two parts of the feature are connected through the concat residual. In this embodiment, the number of OKM modules n is taken as 1, and the fused feature is sent to a 1×1 convolution to adjust the number of output channels. The structure diagram of the OKM module is as follows: Figure 7 As shown, given the input feature X∈R C×H×W , after 1×1 convolution processing, the features are sent to three branches, namely local branch, large branch and global branch, to enhance multi-scale representation; the results of the three branches are finally added and fused, and the channel is adjusted by another 1×1 convolution; among them, the large branch obtains a large receptive field by applying cheap deep convolution with kernel size K×K, and also uses 1×K and K×1 deep convolutions to obtain strip-shaped context information; in order to avoid introducing a large amount of computational overhead caused by convolution of large kernels, only CO-Fusion replaces the second Fusion module, and only one OKM module is added to the CSPOKM module; finally based on performance, receptive feature size, computational complexity and other information, K=31 is finally selected in this embodiment;

[0120] The global branch trains the cropped preprocessed images during training, but during inference, the input images are larger than the cropped images during training, so the 31×31 kernel cannot cover the global features. To alleviate this problem, in this embodiment, a global modeling function is added to the global branch through dual-domain processing. The global branch includes a dual-domain channel attention module DCAM and a frequency-based spatial attention module FSAM. In the DCAM module, for the input feature X Global ∈R C×H×W , first apply the frequency channel attention module FCA to the feature, the process formula of the FCA module is:

[0121]

[0122] in, and are fast Fourier transform and its inverse operation, X FCA represents the output features after the FCA module, X Global refers to the features input to the global branch, It refers to the operation of 1×1 convolution in FCA, and GAP is global average pooling. Through Fourier processing, the global features of the image are effectively refined according to the spectral convolution theorem. The obtained features are then further sent to the spatial channel attention module SCA in the DCAM module. Therefore, the DCAM module process is finally expressed as:

[0123]

[0124] Among them, X DCAM is the output feature of the DCAM module, X FCA is the output feature of the FCA module, It is the 1×1 convolution operation in the SCA module, and GAP is the global average pooling;

[0125] The DCAM module only enhances the dual-domain features in a coarse-grained manner in the channel direction; then the frequency-based attention module is applied in the spatial dimension through the frequency-based spatial attention module FSAM to refine the spectrum at a fine-grained level. The formula of the frequency-based spatial attention module FSAM process is:

[0126]

[0127] Among them, X FSAM is the output feature of the FSAM module, and are the fast Fourier transform and its inverse operation, and They are the 1×1 convolution operations on both sides, X DCAM It is the output feature of the DCAM module. Through DCAM and FSAM, this local branch focuses on the information frequency of high-quality image reconstruction;

[0128] In addition to the large branch and global branch that capture a larger receptive field, local information plays an important role in image restoration, so a very simple but effective local branch is designed to perform local signal modulation by using a 1×1 deep convolutional layer; the OKM module is a full-kernel module consisting of three branches: the global branch, the large branch, and the local branch to effectively learn global-to-local feature representation;

[0129] S22 model training: training the improved network model using the training set processed in step S1 by setting training hyperparameters;

[0130] In this embodiment, for the model training hyperparameters, the training platform is BlueCloud, the test hardware platform environment is Python3.8, CUDA11.8, the GPU graphics card is NVIDIA RTX3090, and the memory is 30G; the training parameters are set as follows: the number of training times is 300 times, the BatchSize is set to 8, the AdamW optimizer is used, and the input image size is 640*640;

[0131] S23 Model Verification: Set verification parameters and use the verification set to verify the trained model.

[0132] The specific steps of step S23 are:

[0133] Verification parameters are set to verify the model trained in step S22 on the verification set, and mAP@0.5, Precision, Recall, Parameters, and Gflops are used as evaluation indicators, where mAP@0.5 is the average precision of the detection results of each type, Precision is the proportion of correct targets among the targets detected by the model; Recall is the proportion of targets correctly identified by the model among the total number of all real targets, Parameters is used to measure the size and complexity of the model; Gflops is used to measure the execution time of the model; the formulas of each indicator are:

[0134]

[0135] Wherein, TP is the number of correctly identified targets, FP is the number of incorrectly identified targets, FN is the number of missed targets, P(r) represents a smooth precision-recall curve, and the area under the smooth curve obtained by integrating it is AP; N represents the number of detected object categories, which corresponds to 6 different PCB defects in this embodiment;

[0136] S3 Detection and Visualization: Deploy the verified model to the visualization interface for real-time detection of PCB defects.

[0137] For ordinary technicians in this field, the specific embodiments are only illustrative descriptions of the present invention. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concepts and technical solutions of the present invention, or the concepts and technical solutions of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A PCB defect detection method based on small target enhanced feature pyramid, characterized in that: The specific steps include: S1 Data collection and processing: Download PCB defect data sets for screening, perform data processing on the screened images, and divide the processed data sets into training sets and validation sets; S2 builds and trains the model: builds the improved RT-DETR-R18 model and performs model training to obtain the weights of the improved RT-DETR-R18 model, and then performs model verification; S3 Detection and Visualization: Deploy the verified model to the visualization interface for real-time detection of PCB defects.

2. The PCB defect detection method based on small target enhanced feature pyramid according to claim 1 is characterized in that: The specific steps of step S1 are: S11: download the PCB defect dataset, obtain images from the PCB defect dataset, and filter the images to remove unlabeled data to obtain a processed dataset; S12: Divide the screened images into training set and validation set; S13: The data of the training set is expanded through data enhancement.

3. The PCB defect detection method based on small target enhanced feature pyramid according to claim 2 is characterized in that: The specific steps of step S2 are: S21 Model building: Building an improved RT-DETR-R18 model; S22 model training: training the improved network model using the training set processed in step S1 by setting training hyperparameters; S23 Model Verification: Set verification parameters and use the verification set to verify the trained model.

4. The PCB defect detection method based on small target enhanced feature pyramid according to claim 3 is characterized in that: In step S21, the module in the RT-DETR-R18 model is improved to obtain an improved RT-DETR-R18 model. The specific steps of improving the module are as follows: S211: First, the backbone network is replaced with the RMT network, and the prior knowledge related to spatial distance is introduced into the model; the attention is decomposed horizontally and vertically by decomposing MasA along the two axes of the image, so that the decomposed MaSA has linear complexity in modeling global information and has the same receptive field shape as the original MaSA; S212: First, the S2 feature layer is subjected to SPDConv to obtain features rich in small target information, and then the process goes to step S213 for feature fusion; S213: Replace the second Fusion fusion module in the CCFM module with the designed CO-Fusion fusion module, that is, add the designed CSPOKM module after concat, and send the features of the S2 feature layer rich in small target information and the S3 feature layer and the previous layer features to the designed fusion module for feature fusion.

5. The PCB defect detection method based on small target enhanced feature pyramid according to claim 4 is characterized in that: In the step S211, the RMT network is constructed based on Manhattan self-attention MasA. The constructed RMT network is divided into four stages, the first three stages use the decomposed MaSA, and the last stage uses the original MaSA; in each reserved network combined with the visual transformer module RMT block, the input feature first passes through the DWConv residual connection, then passes through the layer normalization LN and MaSA residual connection, and finally passes through a layer normalization LN and the forward propagation network FFN residual connection; wherein MaSA enhances self-attention through the spatial attenuation matrix of the Manhattan distance, and the MaSA formula is: MaSA(X)=(Softmax(QK T )⊙D 2d )V; Among them, softmax is a normalized exponential function, Q is the query vector of the input feature, and K T is the transpose of the input feature key vector, D 2d refers to the two-bit attenuation matrix, V is the value vector of the input feature, γ is a scalar, (x n ,y n ) is the two-dimensional coordinate of the nth marker, (x m ,y m ) is the 2D coordinate of the mth marker.

6. The PCB defect detection method based on small target enhanced feature pyramid according to claim 5 is characterized in that: In step S211, the method of decomposing MasA along two axes of the image is specifically to calculate the attention scores in the horizontal direction and the vertical direction of the image respectively, obtain the attention weights, and then apply a one-dimensional bidirectional attenuation matrix to the attention weights, where the one-dimensional attenuation matrix represents the horizontal and vertical distances between the markers; the formula of the decomposed form of MaSA_part is: MaSA_part(X)=Attn H (Attn W V) T ; Among them, Attn H represents vertical attention, Q H It is a vertical query. is the transpose of the vertical key, D H is a one-dimensional bidirectional attenuation matrix in the vertical direction, where Attn W represents horizontal attention, Q W It is a horizontal query. is the transpose of the horizontal key, D W is a one-dimensional bidirectional attenuation matrix in the horizontal direction, γ is a scalar, (x n ,y n ) is the two-dimensional coordinate of the nth marker, (x m ,y m ) is the two-dimensional coordinate of the mth mark, X is the input feature, and V is the value vector of the input feature.

7. The PCB defect detection method based on small target enhanced feature pyramid according to claim 4 is characterized in that: The specific steps of obtaining features rich in small target information by convolution of the S2 feature layer through SPDConv in step S212 are: For any input feature X of size S×S×C1, the feature is split into scale by the scale factor scale 2 A series of sub-features, where the scale factor scale = 2, then these 4 sub-features are expressed as: f 0,0 =X[0:S:2,0:S:2],f 1,0 =X[1:S:2,0:S:2]; f 0,1 =X[0:S:2,1:S:2],f 1,1 =X[1:S:2,1:S:2]; Among them, S is the width and height of the input feature, and the shape of each sub-feature is Downsample the input feature X by a factor of 2 and concatenate the four sub-features along the channel dimension to obtain the feature map X ′ , the feature space dimension has a scale factor, and the channel dimension has an additional scale 2 The scaling factor of ; that is, the original input feature X is transformed from (S, S, C1) to the intermediate feature After feature conversion, a convolution with a kernel of 3×3 and stride=1 is used to make the final channel dimension C2 <sacle 2 C1.

8. The method for detecting PCB defects based on small target enhanced feature pyramid according to claim 7, characterized in that: The specific steps of step S213 are: S2131: First, the CO-Fusion fusion module performs concat feature fusion after receiving the processed S2 feature layer, S3 feature layer and the previous layer feature, and sends the fused features to the CSPOKM module; S2132: Then, two 1×1 convolutions are used to adjust the number of channels. One of the branches uses N repblocks for feature fusion. Finally, the two parts of the features are added together to obtain the final feature and output it.

9. The PCB defect detection method based on small target enhanced feature pyramid according to claim 8, characterized in that: The CSPOKM module in step S213 combines the OKM module with the CSP structure. The input feature is divided into two parts after a 1×1 convolution. One part accounts for 0.25 and is input into the OKM module branch, and the other part accounts for 0.

75. Finally, the two parts of the feature are connected by concat residual. The number n of OKM modules is taken as 1, and the fused feature is sent to a 1×1 convolution to adjust the number of output channels. Given the input feature X∈R C×H×W After 1×1 convolution, the features are sent to three branches, namely local branch, large branch and global branch, to enhance multi-scale representation; the results of the three branches are finally added and fused, and the channels are adjusted by another 1×1 convolution; among them, the large branch obtains a large receptive field by applying cheap deep convolution with kernel size K×K, and also uses 1×K and K×1 deep convolutions to obtain strip-like context information; The global modeling function is added in the global branch through dual-domain processing; wherein the global branch includes a dual-domain channel attention module DCAM and a frequency-based spatial attention module FSAM. In the DCAM module, for the input feature X Global ∈R C×H×W , first apply the frequency channel attention module FCA to the feature, the process formula of the FCA module is: in, and are fast Fourier transform and its inverse operation, X FCA represents the output features after the FCA module, X Global refers to the features input to the global branch, It refers to the operation of 1×1 convolution in FCA, and GAP is global average pooling. Through Fourier processing, the global features of the image are effectively refined according to the spectral convolution theorem. The obtained features are then further sent to the spatial channel attention module SCA in the DCAM module. Therefore, the DCAM module process is finally expressed as: Among them, X DCAM is the output feature of the DCAM module, X FCA is the output feature of the FCA module, It is the 1×1 convolution operation in the SCA module, and GAP is the global average pooling; The DCAM module only enhances the dual-domain features in a coarse-grained manner in the channel direction; then the frequency-based attention module is applied in the spatial dimension through the frequency-based spatial attention module FSAM to refine the spectrum at a fine-grained level. The formula of the frequency-based spatial attention module FSAM process is: Among them, X FSAM is the output feature of the FSAM module, and are the fast Fourier transform and its inverse operation, and They are the 1×1 convolution operations on both sides, X DCAM It is the output feature of the DCAM module. Through DCAM and FSAM, this local branch focuses on the information frequency of high-quality image reconstruction.

10. The PCB defect detection method based on small target enhanced feature pyramid according to claim 7, characterized in that: The specific steps of step S23 are: Verification parameters are set to verify the model trained in step S22 on the verification set, and mAP@0.5, Precision, Recall, Parameters, and Gflops are used as evaluation indicators, where mAP@0.5 is the average precision of the detection results of each type, Precision is the proportion of correct targets among the targets detected by the model; Recall is the proportion of targets correctly identified by the model among the total number of all real targets, Parameters is used to measure the size and complexity of the model; Gflops is used to measure the execution time of the model; the formulas of each indicator are: Among them, TP is the number of correctly identified targets, FP is the number of incorrectly identified targets, FN is the number of missed targets, P(r) represents the smooth precision-recall curve, and the area under the smooth curve obtained by integrating it is AP; N represents the number of detected object categories.

Images