PCB surface defect detection method based on context enhancement and multi-feature fusion

Abstract

This invention discloses a PCB surface defect detection method based on context enhancement and multi-feature fusion, relating to the fields of image processing and machine vision. The method includes the following steps: inputting a preprocessed PCB image into a defect detection model; enhancing the model's feature extraction capability through an improved lightweight backbone network, reducing computational requirements and expanding its application scenarios; simultaneously establishing long-range feature dependencies to improve the recognition ability of objects difficult to detect in complex scenes; introducing a three-feature fusion module into the improved neck network and enhancing shallow features, integrating multi-scale information, reducing feature redundancy, and enhancing the ability to capture surface defects; finally, outputting the corresponding defect category and location. This invention effectively improves the detection accuracy of small and multi-scale defects on PCB surfaces, exhibiting good adaptability and robustness, especially in complex circuit backgrounds.

Description

Technical Field

[0001] This invention relates to the fields of image processing and machine vision technology, and in particular to a method for detecting PCB surface defects based on context enhancement and multi-feature fusion. Background Technology

[0002] Printed circuit boards (PCBs), as a core component of electronic products, directly affect the performance and reliability of the entire electronic device. During PCB manufacturing, due to the complexity of the process, various defects often appear on the surface, including missing holes, rodent bites, open circuits, short circuits, stray wires, and fake copper. Manual visual inspection is a common method, relying primarily on experienced inspectors using the naked eye or a magnifying glass. This method is inefficient, labor-intensive, prone to missed or false detections due to fatigue, and the results are greatly influenced by subjective factors, making it unsuitable for the demands of modern large-scale industrial production. Automated optical inspection (AOI) methods, based on traditional image processing algorithms, identify defects by comparing the differences between a standard PCB image and the image of the PCB to be inspected. While this improves inspection speed, it suffers from a high false detection rate in complex backgrounds, has limited ability to detect minute defects, and struggles to adapt to multi-scale defect variations.

[0003] In recent years, with the development of computer vision and deep learning technologies (such as some improved YOLO series algorithms), they have been introduced into the field of PCB defect detection. However, these methods still have obvious shortcomings: standard models have insufficient feature extraction capabilities when dealing with small defects on PCBs; they perform poorly in detecting small targets against complex circuit backgrounds; and their model generalization ability is limited, making it difficult to quickly and accurately detect defect types and effectively isolate defective products. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a PCB surface defect detection method based on context enhancement and multi-feature fusion, aiming to propose an improvement solution to the above shortcomings. To achieve the above objectives, this invention proposes a PCB surface defect detection method based on context enhancement and multi-feature fusion, comprising: Acquire an image of the PCB surface to be inspected, perform offline data augmentation and data preprocessing on the image, construct a dataset, and train a defect detection model; The PCB image after offline data augmentation and preprocessing is input into the backbone network of the lightweight improved defect detection model to extract multi-scale feature maps of the image; The multi-scale feature map is input into the neck network of the detection model, and the feature map at different scales is fused by the feature fusion module to obtain the fused feature map. The fused feature map is input into the head network of the detection model to obtain the image defect prediction probability map; the location coordinates of the defect region are predicted to complete the defect detection. Preferably, the steps of acquiring the surface image of the PCB board to be inspected, performing offline data augmentation and data preprocessing, and constructing the dataset include: (1) Spatial transformation: Perform horizontal and vertical flipping operations on the image to simulate the defect morphology of the PCB board under different perspectives during actual installation; (2) Enhanced robustness to illumination: Random contrast adjustment of the image is performed to enhance the model’s adaptability to changes in industrial lighting conditions; (3) Noise simulation: Gaussian noise is added to the image to simulate image degradation caused by sensor noise or surface contamination in the actual industrial environment; (4) Size unification and data partitioning: The images in the defect detection dataset are uniformly adjusted to 640 when input into the network. 640 pixels, and divided into 80% training set, 10% test set, and 10% validation set; Preferably, the lightweight improved PCB surface defect detection model backbone network is based on CSPDarkNet and includes four feature extraction blocks; the number of output channels for the four feature extraction blocks are 64, 128, 256, and 512, respectively. The improvement of the backbone network includes: (1) The BottleNeck in the C2f-CIB module has been improved by using the innovative SPConv instead of DWConv. SPConv introduces the Channel Shuffle operation on the basis of efficient partial convolution (PConv) to enhance the randomness of feature expression and cross-channel interaction, improve the poor feature performance, and retain the lightweight characteristics. (2) Design an additional residual connection component context enhancement module (CEM), which uses bidirectional depth separable strip convolution to approximate large kernel depth convolution to extract long-distance contextual relationships in different directions. Use 1x1 convolution to interact and fuse the extracted bidirectional features, thereby enhancing the representational ability of the captured contextual information.

[0005] Preferably, the neck network uses a fusion module to fuse feature maps of different scales to obtain an improved fused feature map, which includes: (1) Design a three-feature fusion module to make full use of all the features extracted by the backbone network and reduce the "redundancy" of features; (2) In view of the fact that the shallow features in the three-feature fusion module have limited semantic information but contain important texture information, an adaptive shallow feature enhancement module is designed to improve the model’s attention to the shallow feature region of interest without excessively increasing the model’s additional overhead. (3) For the fusion of other PCB image features, the corresponding adjacent low-scale feature maps and high-scale feature maps are input into the feature fusion unit for processing; (4) Input all the fused feature maps into the output layer of the neck network to obtain the final fused feature map of the PCB image.

[0006] Preferably, the adaptive shallow feature enhancement module includes two parallel branches. The first branch calculates information weights to adaptively adjust the focus on different information, while the other branch achieves lightweight and efficient shallow feature extraction through depth-separable grouped convolution.

[0007] Preferably, the expression for the loss function of the improved defect detection network model is: in, This represents the category loss function for all predicted bounding boxes in the detection model; This indicates that the predicted bounding box is obtained from the boundary regression loss function; This represents the generalized intersection-union ratio loss function.

[0008] Cross-entropy loss is typically used to measure the difference between the model's predicted class distribution and the true class distribution. The formula is: in For the sample size, No. The true class label of each sample No. The predicted class probability of each sample.

[0009] The loss is used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box. Bounding box regression loss typically uses L1 loss to calculate the distance between the predicted box and the ground truth box, which are expressed as follows: Where A is the predicted bounding box, B is the ground truth bounding box, and C is the smallest rectangular region enclosing A and B. It is the first The true bounding boxes of each sample No. Predicted bounding boxes for each sample.

[0010] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This application introduces channel-randomized hybrid convolution and a context enhancement module into the backbone network, and a three-feature fusion module and an adaptive shallow feature enhancement module into the neck network. This significantly reduces the computational cost and parameter count of the model while effectively improving the detection accuracy of PCB surface defects, solving the technical pain point of prior art where model efficiency and detection performance are difficult to balance. It enhances the detection capability for minute and multi-scale defects. The combination of the three-feature fusion module and the adaptive shallow feature enhancement module allows the model to better integrate multi-scale information and focus on shallow details, significantly enhancing the ability to capture minute defects and defects with large size variations, reducing the false negative rate, and improving robustness in complex backgrounds. The context enhancement module captures long-distance dependencies, enhancing the model's understanding of global information, enabling it to accurately distinguish defects from normal lines even in complex PCB circuit backgrounds, reducing the false positive rate and expanding application scenarios. Due to the overall lightweight design of the model, the requirements for computing resources are reduced, making it more suitable for deployment on edge computing devices next to the production line, thereby achieving real-time and efficient detection and control of PCB product quality. Attached Figure Description

[0011] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall architecture of a defect detection model provided in an embodiment of this application; Figure 2 This is a schematic diagram of the C2f-SP module structure; Figure 3 This is a schematic diagram of the SPConv module; Figure 4 This is a schematic diagram of the Context Enhancement Module (CEM); Figure 5 This is a schematic diagram of the Three-Feature Fusion Module (TFFM); Figure 6 This is a schematic diagram of the shallow feature adaptive enhancement module; Figure 7 This is a diagram comparing the detection results before and after the model improvement; Figure 8 This is a schematic diagram of the mAP50 curves during training of different models. Detailed Implementation

[0012] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0013] Example 1 Reference Figure 1 As shown, Figure 1 This is a general model diagram of a PCB surface defect detection method based on context enhancement and multi-feature fusion provided by the present invention; specifically including: The backbone network of the PCB surface defect detection model, based on CSPDarkNet and featuring lightweight improvements, comprises four feature extraction blocks. The output channels of these four feature extraction blocks are 64, 128, 256, and 512, respectively. Further improvements to the backbone network include: (1) such as Figure 2 As shown, BottleNeck in the C2f-CIB module has been improved by using the innovative SPConv instead of DWConv, represented as: in This represents the extracted features. For residual connection, Disrupt the channels in the convolution. This is a regular convolution.

[0014] SPConv introduces a channel shuffle operation on top of efficient partial convolution (PConv) to enhance the randomness of feature representation and cross-channel interaction, improving poor feature performance while retaining lightweight characteristics. Figure 3 As shown, it is represented as: in This indicates that the channels have been shuffled and rearranged. This represents one-quarter of the feature channels. This indicates the number of remaining channels that are connected by residuals.

[0015] (2) Design an additional residual connection component context enhancement module (CEM), such as Figure 4 As shown, bidirectional depthwise separable strip convolution is used to approximate large-kernel depthwise convolution to extract long-range contextual relationships in different directions. A 1x1 convolution is then used to interact and fuse the extracted bidirectional features, thereby enhancing the representational ability of the captured contextual information. This is represented as follows: in This indicates the average pooling operation. It is a 1x1 convolution.

[0016] in , These represent strip convolutions in two directions, with the value of b remaining the same as in the original text, set to 11. The sigmoid function ensures that the resulting attention map is within the range (0, 1). This is the final feature map obtained by this module.

[0017] The neck network uses a fusion module to fuse feature maps of different scales, resulting in improvements to the fused feature map, including: (1) such as Figure 5 As shown, a three-feature fusion module is designed to fully utilize all features extracted from the backbone network, reducing the "redundancy" of feature generation, as represented by: in , , These represent the shallow, middle, and deep features of the input, respectively. This is a shallow feature adaptive enhancement module. For 1x1 convolution, For 1x1 convolutional blocks, These are the features after fusion.

[0018] (2) In view of the fact that the shallow features in the three-feature fusion module have limited semantic information but contain important texture information, an adaptive shallow feature enhancement module is designed to improve the model’s attention to the shallow feature region of interest without excessively increasing the model’s additional overhead. (3) For the fusion of other PCB image features, the corresponding adjacent low-scale feature maps and high-scale feature maps are input into the feature fusion unit for processing; (4) Input all the fused feature maps into the output layer of the neck network to obtain the final fused feature map of the PCB image.

[0019] like Figure 6 As shown, the adaptive shallow feature enhancement module includes two parallel branches. The first branch calculates information weights to adaptively adjust the focus on different information. The process of adjusting the focus in this branch can be represented as follows: in This indicates splicing along the channel dimension. This indicates that average pooling is performed from the h and w directions respectively. This means rotating the features extracted in the H direction to the same direction as W to facilitate feature splicing and interaction. This represents a 1x1 convolution used to interactively concatenate the features.

[0020] Weights are then generated, and the input features are weighted and adjusted initially, which can be represented as: in This indicates that features are split along the channel dimension. is the Sigmoid activation function. Re represents channel reconstruction, which maps the height and width information of the image to the channel dimension, while changing the shape to fit the second branch features and the final output.

[0021] Another branch achieves lightweight and efficient shallow feature extraction through depthwise separable grouped convolutions, making it suitable for environments with limited computational resources and requiring real-time detection. This process can be represented as: in This indicates a depthwise separable convolution that represents a group. This represents channel reconstruction, transferring information to the channels, and finally multiplying it with the weights of the first branch to achieve feature interaction, represented as: Since the feature shape obtained from the calculation is twice the desired feature shape, the SUM function is used to reduce the feature dimensionality and obtain the desired feature size.

[0022] Specifically, recall (R), precision (P), mean average precision (mAP), number of parameters, gigabit floating-point operations (GFLOPs), and frames per second (FPS, which includes the sum of preprocessing, inference, and postprocessing) were selected as evaluation metrics for the detection model's performance to assess the model's performance from different dimensions.

[0023] The basic performance evaluation metrics, precision and recall, are used to measure the accuracy of the model's predictions of positive samples and the model's coverage of positive samples, respectively. Their calculation formulas can be expressed as follows: Where TP represents the number of correctly detected positive samples, FP represents the number of falsely detected negative samples, and FN represents the number of falsely detected positive samples.

[0024] Mean precision (MP) measures the overall detection performance of a model across the entire dataset. It is calculated by averaging the AP (Average Precision) for all classes. mAP50 represents the average precision calculated with an IoU of 0.5, while mAP50-95 is a more stringent metric, representing the average precision with an IoU range of 0.05 from 0.5 to 0.95. The specific calculation formulas are as follows: in, It is the total number of categories in the dataset. It is a category The average accuracy. The mAP50 curves for different models are as follows: Figure 8 As shown in the figure, the red curve marked "ours" represents the model of this embodiment. Throughout the training process, the average accuracy of the model in this embodiment consistently exceeded that of other comparative models, and eventually converged to a higher level. These data and images fully demonstrate that the solution of this application effectively improves detection accuracy while significantly reducing model complexity, achieving a simultaneous improvement in efficiency and performance.

[0025] Parameter count, gigabit floating-point operations (GFLOPs), and frames per second (fps) are efficiency metrics for a model. The former represents the total number of trainable parameters, measured in millions (M). A smaller parameter count results in a lighter model, making it more suitable for edge device deployment. Lower GFLOPs indicate higher computational efficiency and faster inference speed. Higher fps indicates faster processing speed and better performance. These three metrics are for reference only.

[0026] The expression for the loss function of the improved defect detection network model is as follows: in, This represents the category loss function for all predicted bounding boxes in the detection model; This indicates that the predicted bounding box is obtained from the boundary regression loss function; This represents the generalized intersection-union ratio loss function.

[0027] Cross-entropy loss is typically used to measure the difference between the model's predicted class distribution and the true class distribution. The formula is: in For the sample size, No. The true class label of each sample No. The predicted class probability of each sample.

[0028] The loss is used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box. Bounding box regression loss typically uses L1 loss to calculate the distance between the predicted box and the ground truth box. These two values ​​can be expressed as follows: Where A is the predicted bounding box, B is the ground truth bounding box, and C is the smallest rectangular region enclosing A and B. It is the first The true bounding boxes of each sample No. Predicted bounding boxes for each sample.

[0029] In summary, this invention introduces a C2f-SP module using SPConv and a context enhancement module, and a three-feature fusion module using a shallow feature adaptive enhancement module. To verify the effectiveness of the proposed modules in defect detection tasks, we conducted comparative experiments on a dataset that has already undergone offline data augmentation, comparing the performance of the original model with that of the models after adding the two modules respectively. The experimental results are shown in Table 1 below. The mAP50 value of the original model is 94.9, while after introducing the two modules respectively, the mAP50 values ​​of the model increased to 95.3 and 95.8, respectively. This indicates that both proposed modules can effectively improve the overall detection level of the model. Moreover, the model with the best overall performance is achieved when both modules are modified simultaneously, with an mAP50 value of 96.1, which is better than the original model overall. The visualization difference in detection before and after the model improvement is shown in Table 1 below. Figure 7 As shown, the model in this embodiment can successfully detect minor defects such as "open circuits," "burrs," and leaks.

[0030] Table 1 Ablation Experiment Results Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention. In particular, the FPS test here was conducted using GPU acceleration in the above experimental environment, only comparing the relative processing speeds of different models; the actual processing speed needs to be determined based on the actual scenario.

Claims

1. A PCB surface defect detection method based on context enhancement and multi-feature fusion, characterized in that, include: Acquire an image of the PCB surface to be inspected, perform offline data augmentation and data preprocessing on the image, construct a dataset, and train a defect detection model; The PCB image after offline data augmentation and preprocessing is input into the backbone network of the lightweight improved defect detection model to extract multi-scale feature maps of the image; The multi-scale feature map is input into the neck network of the detection model, and the feature map at different scales is fused by the feature fusion module to obtain the fused feature map. The fused feature map is input into the head network of the detection model to obtain the image defect prediction probability map; the location coordinates of the defect area are predicted to complete the defect detection.

2. The PCB surface defect detection method based on context enhancement and multi-feature fusion according to claim 1, characterized in that, The steps of acquiring an image of the PCB board surface to be inspected, performing offline data augmentation and data preprocessing, and constructing a dataset include: (1) Spatial transformation: Perform horizontal and vertical flipping operations on the image to simulate the defect morphology of the PCB board under different perspectives during actual installation; (2) Enhanced robustness to illumination: Random contrast adjustment of the image is performed to enhance the model’s adaptability to changes in industrial lighting conditions; (3) Noise simulation: Gaussian noise is added to the image to simulate image degradation caused by sensor noise or surface contamination in the actual industrial environment; (4) Size unification and data partitioning: The images in the defect detection dataset are uniformly adjusted to 640 when input into the network. The dataset is 640 pixels, and is divided into 80% training set, 10% test set, and 10% validation set.

3. The PCB surface defect detection method based on context enhancement and multi-feature fusion according to claim 1, characterized in that, The lightweight improved PCB surface defect detection model backbone network is based on CSPDarkNet and includes four feature extraction blocks; The four feature extraction blocks have 64, 128, 256, and 512 output channels, respectively. The improvements to the backbone network include: (1) The BottleNeck in the C2f-CIB module has been improved by using the innovative SPConv instead of DWConv. SPConv introduces the Channel Shuffle operation on the basis of efficient partial convolution (PConv) to enhance the randomness of feature expression and cross-channel interaction, improve the poor feature performance, and retain the lightweight characteristics. (2) Design an additional residual connection component context enhancement module (CEM), which uses bidirectional depth separable strip convolution to approximate large kernel depth convolution to extract long-distance contextual relationships in different directions. Use 1x1 convolution to interact and fuse the extracted bidirectional features, thereby enhancing the representational ability of the captured contextual information.

4. The PCB surface defect detection method based on context enhancement and multi-feature fusion according to claim 1, characterized in that, The neck network uses a fusion module to fuse feature maps of different scales, resulting in improvements to the fused feature map, including: (1) Design a three-feature fusion module to make full use of all the features extracted by the backbone network and reduce the "redundancy" of features; (2) In view of the fact that the shallow features in the three-feature fusion module have limited semantic information but contain important texture information, an adaptive shallow feature enhancement module is designed to improve the model’s attention to the shallow feature region of interest without excessively increasing the model’s additional overhead. (3) For the fusion of other PCB image features, the corresponding adjacent low-scale feature maps and high-scale feature maps are input into the feature fusion unit for processing; (4) Input all the fused feature maps into the output layer of the neck network to obtain the final fused feature map of the PCB image.

5. The PCB surface defect detection method based on context enhancement and multi-feature fusion according to claim 4, characterized in that, The adaptive enhancement shallow feature module includes two parallel branches. The first branch calculates information weights to adaptively adjust the focus on different information, while the other branch achieves lightweight and efficient shallow feature extraction through depth-separable grouped convolution.

6. The PCB surface defect detection method based on context enhancement and multi-feature fusion according to claim 1, characterized in that, The expression for the loss function of the improved defect detection network model is as follows: in, This represents the category loss function for all predicted bounding boxes in the detection model; This indicates that the predicted bounding box is obtained from the boundary regression loss function; Represents the generalized intersection-union ratio loss function; Cross-entropy loss is typically used to measure the difference between the model's predicted class distribution and the true class distribution. The formula is: in For the sample size, No. The true class label of each sample No. The predicted class probability of each sample; The loss is used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box. Bounding box regression loss typically uses L1 loss to calculate the distance between the predicted box and the ground truth box, which are expressed as follows: Where A is the predicted bounding box, B is the ground truth bounding box, and C is the smallest rectangular region enclosing A and B. It is the first The true bounding boxes of each sample No. Predicted bounding boxes for each sample.

Images