PCBA surface defect detection method and system based on image recognition

Abstract

The invention provides a PCBA surface defect detection method and system based on image recognition, and the method comprises the steps: collecting and constructing a multi-class PCBA overall defect image data set, and carrying out the preprocessing; constructing a pre-training neural network model, and classifying the preprocessed PCBA overall defect image data set; constructing a local target identification method based on a YOLOX network according to an overall defect classification result; and carrying out local defect detection on the PCBA surface based on a local target identification method. According to the method, the overall classification and local detection strategies are comprehensively applied to reduce the burden of the model, and the detection precision and speed are improved. And an attention mechanism is introduced to improve the performance and generalization ability of the model. Data enhancement, consistency processing, classification labeling and the like are carried out on an image data set, and the defect detection effect of the model is improved. And multiple optimization strategies such as structure improvement, loss function improvement and pruning lightweight are adopted for the network model, so that the image target detection performance is effectively improved.

Description

Technical Field

[0001] The present invention and the field of PCBA detection technology specifically relate to a PCBA surface defect detection method and system based on image recognition. Background Art

[0002] Printed Circuit Board (PCB), as a carrier for installing electronic components and making electrical connections, is an important component of various types of electronic products. The development level of its production and manufacturing has played a fundamental role in the technological progress of industries such as computers, consumer electronics, communications equipment, automotive electronics, and industrial equipment. In PCB production and manufacturing, the quality control of PCB assembly (PCBA) has an important impact on the final quality performance of electronic products. As PCBA is accelerating towards miniaturization, thinness, and density, manual visual inspection and online electrical inspection are difficult to meet the needs of the electronic circuit manufacturing industry for PCBA quality control due to limited sight and electrical contact.

[0003] PCBA is the abbreviation of the process of assembling electronic components on the bare PCB board through surface technology in electronic circuit manufacturing. It can also generally refer to the PCB with completed component assembly. The current mainstream surface processes include dual inline package (DIP) and surface mount technology (SMT). Due to the advantages of SMT in assembly density, seismic resistance, high frequency characteristics and production efficiency, it has become the most popular technology in the electronic assembly industry, promoting and facilitating the chip-based, miniaturized and lightweight electronic equipment. SMT is a process in which various surface mount components (SMD) are mounted on the surface of the PCB substrate with the help of a placement machine and solder, and the assembly is achieved through welding, cleaning and testing. SMT usually uses the reflow soldering process. Affected by factors such as the quality of solder and flux, welding temperature, substrate oxidation status, and placement machine working error, defects such as insufficient solder joints, cold solder joints, flux residue, tin beads, component skewness, and missing parts may occur during the placement process.

[0004] Currently, in the scenario of PCBA surface process defect detection, there are the following limitations: Due to the limitations of industrial computer installation space, power supply, and heat dissipation, it is difficult to use high-performance GPUs, and computing resources are limited; There are many types of components on PCBA, and the appearance characteristics of various components vary greatly. If a network is used to classify multiple components, not only will the network learning be too difficult, but it will also not conform to the extensible design concept of functional modularization and combination. Therefore, it is considered to train a classification network for each component. In actual use, the system automatically calls the corresponding model for processing according to the component type, which puts high requirements on the parameter scale and loading speed of the network model.

[0005] Many algorithms have studied PCB bare board surface defect detection methods based on traditional machine learning and deep learning. Some methods include extracting boundary pixels related to PCB defect areas and using gradient direction information as feature vectors for defect classification and identification. This method can identify defects such as scratches and burrs on the PCB surface, but it is difficult to detect when the contrast between the defect and the background environment is low. Another method uses morphology to extract PCB defect areas for defect classification, reducing the error rate of defect identification. However, when extracting defects based on shape features, similar defects with large shape differences may not be well identified.

[0006] Some researchers have proposed a multi-level PCB defect detection algorithm with improved template matching to improve the accuracy of PCB defect detection. However, this also brings about the problem of reduced detection accuracy when the defect is significantly different from the template. There are also machine learning methods based on accelerated robust features and random forests, which can detect defects on PCBs without normal images, but cannot handle defects of different scales and complex and changeable external lighting environments. In addition, the method designed based on the idea of ​​residual learning uses image downsampling methods to expand the image receptive field, improve denoising efficiency, and solve the problem of insufficient clarity in PCB image detection.

[0007] Introducing deep learning into PCB defect detection is also a mainstream direction of PCB defect detection. For example, the prior art proposes a two-stream convolutional neural network to check surface mount technology assembly defects by using two solder areas, but the complexity and computational complexity of the model are also greatly increased. Some researchers have also proposed building an enhanced feature extraction network based on a feature pyramid network for PCB defect detection, but the detection accuracy is subject to certain limitations. Although deep learning has been applied to PCB defect detection, the recognition accuracy and speed still need to be further improved. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a PCBA surface defect detection method and system based on image recognition to solve the deficiencies of the prior art described in the background technology.

[0009] To solve the above technical problems, an embodiment of the present invention provides the following technical solution: a PCBA surface defect detection method based on image recognition, comprising the following steps: Collect and construct a multi-category PCBA overall defect image dataset not limited to welding defects, component defects, PCB board defects, and other defects, and perform preprocessing; the preprocessing includes random rotation, cropping, and normalization; Construct a pre-trained neural network model to classify the pre-processed PCBA overall defect image dataset; According to the overall defect classification results, a local target recognition method based on the YOLOX network is constructed; Local defect detection on PCBA surface is performed based on local target recognition method.

[0010] Furthermore, the pre-trained neural network model is constructed to classify the pre-processed PCBA overall defect image data set, specifically: Train classification networks based on MobileNetv2 and EfficientNet-b0 according to the ImageNet dataset; Use the trained MobileNetv2 and EfficientNet-b0 networks as pre-trained models, and transfer training to the pre-trained models. Set the final classification layer to 4 categories. Then design 3 lightweight CNN branches. Use EfficientNet-b0 as the main branch to extract global feature information from PCBA images, and use two MobileNetv2 branches to extract defect features and non-defect features in local areas respectively. After using the Concatenate function to fuse the features of these regions, connect the global average pooling layer, and then add the Dropout layer; A Softmax classifier with 4 classes is used to classify PCBA images.

[0011] Furthermore, it also includes introducing an attention mechanism to the pre-trained model, specifically adding a CBAM module before and after the Bottleneck layer of MobileNetv2 to form an Attention-MobileNetV2 network.

[0012] Furthermore, the local target recognition method based on the YOLOX network is specifically constructed as follows: The YOLOX network includes the input end, the backbone feature extraction network, the neck network and the prediction end; Perform data augmentation on the original data using methods including but not limited to rotation, scaling, flipping, and cropping at the input end; The backbone feature extraction network is replaced with SDConv, which is responsible for image feature extraction; it includes multiple BConv convolutional layers, E-ELAN convolutional layers, MPConv convolutional layers, and SPPCSPC fusion layers; among them, the BConv convolutional layer extracts image features of different scales, the E-ELAN convolutional layer learns more diverse features by guiding the calculation blocks of different feature groups, the MPConv convolutional layer fuses the feature information processed by normal BConv convolution with the feature information of MaxPool expanded receptive field, and the SPPCSPC fusion layer solves the problems of image distortion and repeated feature extraction; Replace BConv with SDConv in the E-ELAN convolutional layer and MPConv convolutional layer in the neck network area; At the prediction end, the number of image channels is adjusted for the features of large, medium and small scales output by PAFPN through the REPConv convolution layer; Add the attention mechanism into the spatial pyramid pooling module SPPCSPC; Replace the Mish activation function with the Swish activation function.

[0013] Furthermore, the local defect detection on the PCBA surface based on the local target recognition method is specifically as follows: After completing the PCBA image defect classification, the images with overall defects are preprocessed again to form a detection data set, and the detection data set is divided into a detection test set and a detection training set; The detection training set and the annotation information formed after the secondary preprocessing are input into the YOLOX network for network training. The YOLOX network calculates the gradient through forward propagation, and uses the stochastic gradient descent algorithm to update the weight parameters of the YOLOX network model according to the gradient; The loss function is calculated based on the prediction results and annotations of the YOLOX network model until the specified number of iterations or convergence conditions are reached to form a basic YOLOX network model; the loss function includes three parts: confidence loss, classification probability loss, and bounding box regression loss; Perform pruning operations on the basic YOLOX network model to generate a pruned improved YOLOX network model, train the pruned improved YOLOX network model on the dataset, and readjust the number of channels or layers in the network structure; Use the parameters retained in the pruned large network to initialize the pruned network, and continue to perform several rounds of fine-tuning on the training set to further adjust the model parameters. During the fine-tuning process, sort the weights of the convolutional layer kernels, remove the convolutional kernels with the lowest weight values ​​and their corresponding feature maps, and clear the related connections; in the original pruned convolutional layer, regenerate new convolutional kernels and initialize the weights, use the remaining weights to assign values ​​to the new convolutional kernels and train them, and obtain the final pruned YOLOX network model; The detection test set is input into the pruned YOLOX network model for network testing to detect image defects after classification.

[0014] Further, the secondary preprocessing includes: Each image is automatically oriented and has its EXIF ​​orientation information removed; Perform data consistency processing, adjust the image to a unified 640x640 resolution, and adjust the image size by stretching; Enhance image contrast through adaptive equalization; Generate labels in yolo format, use LabelImg to annotate the images in the dataset with local defects under the corresponding overall defect classification, read the annotation information in the xml file after the image dataset is annotated and write it into a txt file. After running, the annotation information of all image datasets will appear in the label folder.

[0015] Furthermore, the pruning operation is performed on the basic YOLOX network model, specifically including: Determine the layer to be pruned based on the importance index of the network layer; for each network layer, calculate the sensitivity of the forward propagation of a network layer on the model. Under the condition of a given input, calculate the partial derivative of the change in the output with respect to the weight of the layer, so as to obtain the sensitivity of the layer to the output. The greater the sensitivity, the greater the impact of the layer on the output, and it should be given priority when pruning; Sort the weights of the network layers to be pruned; Determine the threshold value based on the weight ranking result and pruning rate in each pruning layer; Remove weights below the threshold in the network and retain weights above the threshold; Save the new model parameters and weights to generate the pruned improved YOLOX network model.

[0016] Furthermore, the YOLOX network adds an attention mechanism model CAM in the neck network, specifically: the input is the three feature maps output by the backbone network and the SPP structure. In the 3x3 convolution layer spliced ​​5 times, the CAM module passes through the same 1x1 convolution for the three feature maps, and outputs PCBA defect feature maps F1, F2 and F3. Feature maps F2 and F3 are matrix multiplied to obtain feature map F4, and then the Softmax operation is continued to obtain the channel attention feature map F5. Then F5 is matrix multiplied with F1 which has only undergone 1x1 convolution operation to obtain feature map F6. Finally, F6 is weighted and summed with the original image, and then the PCBA surface detail information is enlarged through scale transformation to obtain clearer feature weights.

[0017] Furthermore, it also includes 3 3, a depth-separable convolution is introduced, where the input of the network is ( M), output ( N), where the network input size is , M is the number of convolution kernels, and the input PCBA board feature map is processed Convolution, the convolution parameter is M N, the amount of convolution calculation is M N, applying depthwise separable convolution to input and output of the same size, the number of parameters is: M+ M N, the convolution is expressed as a depth-wise separable convolution, and the reduced calculation is: = + .

[0018] The present invention also proposes a PCBA surface defect detection system based on image recognition, comprising: The data acquisition module is used to collect and construct a multi-category PCBA overall defect image data set, which is not limited to welding defects, component defects, PCB board defects, and other defects, and perform preprocessing; the preprocessing includes random rotation, cropping, and normalization; The overall defect classification module builds a pre-trained neural network model to classify the pre-processed PCBA overall defect image dataset; The local recognition building module is used to build a local target recognition method based on the YOLOX network for the overall defect classification results; The local defect detection module is used to perform local defect detection on the PCBA surface based on the local target recognition method.

[0019] The beneficial effects of the above technical solution of the present invention are as follows: 1. After classifying the preprocessed PCBA overall defect image data set, local defects are detected and sorted out on the image training samples, which solves the problem of too large training samples due to the large number of PCBA defect types in reality, reduces the model training time, and thus improves the recognition accuracy and generalization ability of the model. In the overall defect image classification, the MobileNetv2 and EfficientNet-b0 models are used to build a pre-trained neural network model. Both have lightweight network structures, small model size and number of parameters, and the lightweight CNN based on ImageNet transfer learning has a strong feature extraction ability, which can better mine data features and has high classification accuracy. In addition, the CBAM module with an attention mechanism introduced in the MobileNetv2 model can significantly improve the network's feature learning ability, which helps the network to stabilize its classification performance when dealing with a variety of PCBA product models and diverse component specifications.

[0020] 2. When constructing a local target recognition method based on the YOLOX network, the input end, the backbone feature extraction network, and the neck network of the YOLOX network are optimized, and data enhancement is performed on the input end to meet the training requirements of the backbone network. The backbone feature extraction network is replaced with SDConv, because SDConv can reduce the burden of the model, improve detection accuracy, and speed up the prediction calculation speed. The improved structure of the neck network improves the detection accuracy while reducing the computational complexity, adapting to the characteristics of the PCB defect detection task. The attention mechanism is introduced in the spatial pyramid pooling module SPPCSPC, and the network's attention to targets of different scales is enhanced through the ACmix attention module with a simple structure and low computational cost, thereby significantly improving the accuracy of the model. Mish is used as the activation function, and it is replaced with a lighter activation function Swish, which can reduce the computational overhead of the model.

[0021] 3. YOLOX network adds the attention mechanism model CAM to the neck network. The channel attention module can reasonably assign weights to PCBA surface features. Since there are similarities between different types of defects in PCBA surface defect images, and the same type has different forms, it will affect the defect detection process. After adding the channel attention module, higher weights are assigned to effective feature values ​​to improve feature fusion capabilities. At the same time, in order to avoid the increase in network parameters and calculations caused by the addition of the attention mechanism, thereby affecting the detection performance, a depth-separable convolution is introduced in the 3x3 convolution of the SPP structure spliced ​​three times.

[0022] 4. When performing local defect detection on the PCBA surface based on the local target recognition method, the classified overall defect image is first preprocessed twice, including image consistency processing and adaptive equalization enhancement, to improve the model's effect on defect detection. At the same time, the images in the data set are annotated with local defects under the corresponding overall defect classification. Annotating the image data set improves the accuracy and classification ability of the model classification, which is helpful for the calculation of the loss function in the training stage and the calculation of performance indicators in the model evaluation stage.

[0023] 5. When performing local defect detection on the PCBA surface based on the local target recognition method, the pruning algorithm is used to lightweight the network model. By accurately identifying and eliminating parameters or connections that contribute little to performance, the number of model parameters and the amount of calculation are effectively reduced, the model is streamlined and the calculation speed is improved.

[0024] In summary, the present invention comprehensively applies the strategy of overall classification plus local detection to reduce the burden of the model and improve the detection accuracy and speed. The attention mechanism is introduced to improve the performance and generalization ability of the model. Data enhancement, consistency processing, classification and annotation are performed on the image data set to improve the effect of the model on defect detection. The network model adopts a variety of optimization strategies such as structural improvement, loss function improvement, and pruning lightweighting to effectively improve the image target detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The figure is an overall flow chart of the PCBA surface defect detection method based on image recognition of the present invention; Figure 2 A schematic diagram of the structure of classifying a PCBA overall defect image dataset by a pre-trained neural network model in the present invention; Figure 3 This is a flow chart of transfer learning when the present invention classifies the PCBA overall defect image data set; Figure 4 The following is a diagram of the classification network structure of MobileNetv2 of the present invention; Figure 5 This is a schematic diagram of the structure of the present invention introducing the attention mechanism into the pre-training model; Figure 6 The structural diagram of SDConv in the local target recognition method based on YOLOX network is constructed for the present invention; Figure 7 An improved neck network structure diagram in the local target recognition method based on the YOLOX network is constructed for the present invention; Figure 8 This is a model diagram of the present invention in which the attention mechanism model CAM is added to the neck network of the YOLOX network. DETAILED DESCRIPTION

[0026] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 As shown, a PCBA surface defect detection method based on image recognition includes the following steps: S1. Collect and construct a multi-category PCBA overall defect image dataset not limited to welding defects, component defects, PCB board defects, and other defects, and perform preprocessing; the preprocessing includes random rotation, cropping, and normalization; S2, build a pre-trained neural network model to classify the pre-processed PCBA overall defect image dataset; S3. Based on the overall defect classification results, a local target recognition method based on the YOLOX network is constructed; S4. Perform local defect detection on the PCBA surface based on the local target recognition method.

[0028] The specific contents of the above method steps are as follows: 1. If Figure 2 As shown, in step S2, a pre-trained neural network model is constructed to classify the pre-processed PCBA overall defect image data set, specifically: Train classification networks based on MobileNetv2 and EfficientNet-b0 according to the ImageNet dataset; Use the trained MobileNetv2 and EfficientNet-b0 networks as pre-trained models, and transfer training to the pre-trained models. Set the final classification layer to 4 categories. Then design 3 lightweight CNN branches. Use EfficientNet-b0 as the main branch to extract global feature information from PCBA images, and use two MobileNetv2 branches to extract defect features and non-defect features in local areas respectively. After using the Concatenate function to fuse the features of these regions, connect the global average pooling layer, and then add the Dropout layer; Use a Softmax classifier set to 4 categories to complete the PCBA image defect classification; Because the PCBA image datasets collected in reality are relatively small, and deep learning models often require larger datasets, in order to avoid overfitting problems in the process of network learning, it is considered to use the transfer learning method to first train the model on the large-scale ImageNet image dataset, and through the form of weight transfer, spend a shorter time on the smaller PCBA image dataset to achieve better classification results. The transfer learning method of the present invention is based on two lightweight convolutional neural networks, MobileNetv2 and EfficientNet-b0. Figure 3 As shown in the figure, firstly, by training the ImageNet dataset, a pre-trained model with 1000 classifications can be obtained, and then the model migration training is performed. The global average pooling layer is used to further input the output feature map into the fully connected layer, and Softmax is added to implement a new classifier for images that are not limited to welding defects, component defects, PCB board defects, and other defects. Then, the dataset (PCBA image dataset) is used to train this new classifier, so that the trained model and parameters are saved as the initialization parameters for PCBA image classification model training.

[0029] The MobileNetv2 network model of the present invention is composed of 17 inverted residual modules at its core, and the network structure is as follows: Figure 4As shown in the figure, Conv2d is a 2D convolution operation, Bottleneck is a reverse residual block, and Avgpool is a global average pooling operation. By adopting the MobileNetv2 model that has been trained on the ImageNet dataset and using the initial weights and structure of the pre-trained model, the network model is fully convolutional and can adapt to images of different sizes. Figure 2 As shown, the present invention adopts the MobileNetv2 model and the lightweight migrated CNN branch of the EfficientNet-b0 network to extract features of the overall defect area and the non-overall defect area of ​​the PCBA image respectively, and uses the Concatenate function to fuse the feature maps extracted from the three areas according to the channel dimension, and then obtains the corresponding feature map through a 7×7 global average pooling operation, and then inputs the feature map into the fully connected layer, and uses the Dropout layer to prevent overfitting and improve the generalization ability of the model. Finally, the Softmax layer is connected to realize the classification of PCBA images not limited to welding defects, component defects, PCB board defects, and other defects.

[0030] like Figure 5 As shown, the present invention also introduces an attention mechanism to the pre-trained model, specifically, a CBAM module is added before and after the Bottleneck layer of MobileNetv2 to form an Attention-MobileNetV2 network. The first CBAM module mainly focuses on the weighting between the underlying features of the PCBA component image and the pixels of the original image, and enhances the underlying features and key pixel areas such as colors and edges that have important influence; the second CBAM module mainly focuses on high-dimensional features and spatial pixels that have undergone a sufficient number of nonlinear transformations to further refine abstract features. The two added CBAM modules have almost no effect on the network structure of other positions of MobileNetV2, and only a small number of parameters are added, but they can significantly improve the network's feature learning ability, which helps to stabilize the network's classification performance when dealing with a variety of PCBA product models and diverse component specifications.

[0031] 2. Step S3 constructs a local target recognition method based on the YOLOX network, specifically: The YOLOX network includes the input end, the backbone feature extraction network, the neck network and the prediction end; (1) At the input end, the original data is augmented using methods including but not limited to rotation, scaling, flipping, and cropping. The input image undergoes image preprocessing operations such as pixel unification and data augmentation to meet the training requirements of the backbone network.

[0032] (2) The backbone feature extraction network is replaced by SDConv, which is responsible for image feature extraction. In the backbone network, multiple transmissions of spatial information to channel information may lead to partial loss of semantic information. In order to retain more connections between each channel, SDConv is chosen here for replacement. Since SDConv can reduce the burden of the model and improve detection accuracy, in order to speed up the calculation of prediction, SDConv is used to replace the Neck layer with smaller image width and height dimensions.

[0033] In order to solve the problems of small defect size, high model power consumption and low detection efficiency in PCB defect detection, an efficient convolution method (SDConv) is proposed. It can better balance the accuracy and speed of the model by reducing the burden of the model while maintaining the detection accuracy. The structure diagram of SDConv is shown in the figure Figure 6 As shown, the design is as follows: High-dimensional channel mapping: Use BConv to map input features to high-dimensional channels. This helps to better express feature information and improve detection accuracy.

[0034] Group channel division: Divide the feature map into two groups of channels in a ratio of 1:4. The design of this step borrows the depthwise separable convolution (DWConv) in MobileNetV2.

[0035] Depthwise separable convolution: A group of channels use depthwise separable convolution, which helps reduce the memory consumption and computational burden of the model. Depthwise separable convolution is divided into two stages: depthwise convolution and pointwise convolution.

[0036] Fusion operation: Another set of channels remain as is, and a fusion operation is performed on the two feature maps of the depthwise separable convolution. This helps to fully exploit the advantages of depthwise separable convolution.

[0037] Channel Shuffle: Finally, the channel shuffle operation is performed to achieve the interaction of channel information. This step can improve the expressiveness of the feature map.

[0038] The backbone feature extraction network includes multiple BConv convolutional layers, E-ELAN convolutional layers, MPConv convolutional layers and SPPCSPC fusion layers; among them, the BConv convolutional layers extract image features of different scales, the E-ELAN convolutional layers learn more diverse features by guiding the calculation blocks of different feature groups, the MPConv convolutional layers fuse the feature information processed by normal BConv convolution with the feature information of MaxPool expanded receptive field, and the SPPCSPC fusion layers solve the problems of image distortion and repeated feature extraction.

[0039] (3) The neck network realizes the efficient fusion of features at different levels, and efficiently transmits shallow feature information to deep layers. Since the defect size is relatively small in PCB defect detection, the detection part of the neck area for large-scale targets is redundant. In order to reduce the computational complexity of the model, the design idea of ​​the Yolo-tiny network is referenced here, and the feature map branch of large-scale detection objects is deleted. The improved structure is as follows Figure 7 As shown in the figure, the BConv in the E-ELAN convolution layer and the MPConv convolution layer in the Neck area is replaced by SDConv, and they are named E-ELAN-SD and MPConv-SD respectively to distinguish them. This improved structure improves the detection accuracy while reducing the computational complexity, which adapts to the characteristics of PCB defect detection tasks.

[0040] (4) At the prediction end, the number of image channels of the three different scales of large, medium and small features output by PAFPN is adjusted through the REPConv convolution layer to predict the detection category and confidence.

[0041] (5) The attention mechanism is introduced by adding the spatial pyramid pooling module SPPCSPC. The ACmix attention module with simple structure and low computational cost is used to enhance the network's attention to targets of different scales, thereby significantly improving the accuracy of the model. The ACmix attention module first maps the input to N sub-features through a 1×1 convolution, and then processes it in two different branches. The upper branch is the convolutional attention part, which performs channel expansion through the fully connected layer, and performs offset, aggregation and convolution processing on the generated features. The lower branch is the self-attention part, which is consistent with the traditional self-attention mechanism. Finally, the output features of the two branches are combined through the weight parameters and The fusion formula is as follows:

[0042] in, represents the output feature map, and Represent the feature maps after self-attention processing and convolution processing respectively, where and The parameter values ​​of are 0.6 and 1.2 respectively, focusing on the ability of convolutional attention to express scale information. In the SPP module, the ACmix module is introduced here and placed after the SPP structure processing. This can effectively give different weights to the receptive fields of four different scales and improve the recognition ability of small-scale targets. This improved structure improves accuracy while reducing the amount of calculation, which has obvious advantages for PCB defect detection tasks.

[0043] (6) Replace the Mish activation function with the Swish activation function. The YOLOX network uses Mish as the activation function. Replacing it with the lighter activation function Swish can reduce the computational overhead of the model.

[0044] In addition, if Figure 8 As shown in the figure, the YOLOX network adds the attention mechanism model CAM in the neck network. Specifically, the input is the three feature maps output by the backbone network and the SPP structure. In the 3x3 convolution layer spliced ​​5 times by the CAM module, the three feature maps must pass through the same 1x1 convolution, and the PCBA defect feature maps F1, F2 and F3 are output. The feature maps F2 and F3 are matrix multiplied to obtain the feature map F4, and then the Softmax operation is continued to obtain the channel attention feature map F5. Then F5 is matrix multiplied with F1 which has only undergone the 1x1 convolution operation to obtain the feature map F6. Finally, F6 is weighted and summed with the original image, and then the PCBA surface detail information is enlarged through scale transformation to obtain clearer feature weights.

[0045] The channel attention module can reasonably assign weights to PCBA surface features. Since there are similarities between different types of defects in PCBA surface defect images, and the same type has different forms, which will affect the defect detection process, after adding the channel attention module, higher weights will be assigned to effective feature values, thereby improving feature fusion capabilities.

[0046] At the same time, in order to avoid the increase in network parameters and computation caused by adding attention mechanism, which would affect the detection performance, the depth-separable convolution is introduced in the 3x3 convolution of the SPP structure. Most of the parameters and computation of the entire network are concentrated on the 3x3 standard convolution, so the structure performs 3x3 standard convolution operations, all of which use depth-separable convolution, which reduces the number of parameters and the amount of operation, while improving the nonlinearity of the entire network, thereby enhancing the generalization ability of the entire network. 3, a depth-separable convolution is introduced, where the input of the network is ( M), output ( N), where the network input size is , M is the number of convolution kernels, and the input PCBA board feature map is processed Convolution, the convolution parameter is M N, the amount of convolution calculation is M N, applying depthwise separable convolution to input and output of the same size, the number of parameters is: M+ M N, the convolution is expressed as a depth-wise separable convolution, and the reduced calculation is: = + .

[0047] 3. Step S4 performs local defect detection on the PCBA surface based on a local target recognition method, specifically: After completing the PCBA image defect classification, the images with overall defects are preprocessed for the second time to form a detection data set. Various defects are classified into several categories of overall defects, such as welding defects, component defects, PCB board defects, and other defects. The detection data set is divided into a detection test set and a detection training set. Among them, the secondary preprocessing method includes: Each image is automatically oriented to ensure the consistency of image orientation, and EXIF ​​orientation information is removed to prevent bias during training. Data consistency processing is performed to adjust the image to a uniform 640x640 resolution. The image size is adjusted by stretching, which not only maintains data consistency but also facilitates model processing. Stretching is chosen to adjust the image size instead of cropping or padding to ensure the integrity of the defect information in the image.

[0048] The contrast of the image is enhanced through adaptive equalization to highlight the defect features, making it easier for the model to learn and identify different types of defects.

[0049] Generate labels in yolo format, use LabelImg to annotate the images in the dataset under the corresponding overall defect classification, read the annotation information in the xml file after the image dataset is annotated and write it to the txt file. After running, the annotation information of all image datasets will appear in the label folder. Annotating the image dataset ensures that the model can reach a high standard in terms of accuracy and classification ability. These labels are crucial for the calculation of loss functions in the training phase and the calculation of performance indicators in the model evaluation phase.

[0050] Then, the detection training set and the annotation information formed after the secondary preprocessing are input into the YOLOX network for network training. The YOLOX network calculates the gradient through forward propagation, and uses the stochastic gradient descent algorithm to update the weight parameters of the YOLOX network model according to the gradient; The loss function is calculated based on the prediction results and annotations of the YOLOX network model until the specified number of iterations or convergence conditions are reached to form a basic YOLOX network model; the loss function consists of three parts: confidence loss, classification probability loss, and bounding box regression loss.

[0051] Perform pruning operations on the basic YOLOX network model to generate a pruned improved YOLOX network model, train the pruned improved YOLOX network model on the data set, and readjust the number of channels or layers in the network structure; wherein, perform pruning operations on the basic YOLOX network model, specifically: (1) Determine the network layers to be pruned. Before pruning the model, the network model needs to be analyzed and evaluated to determine which network layers can be pruned. This process needs to consider factors such as the number of parameters of each network layer in the model, the amount of computation, and the contribution to the overall performance of the model. This can minimize the amount of computation and size of the model while ensuring that the detection accuracy of the model does not drop too much.

[0052] (2) Sort the weights of the network layers to be pruned; weight sorting of network layers is one of the important steps in pruning. Before pruning, all parameters in the network need to be sorted to determine which parameters contribute less to the performance of the network and which parameters contribute more to the performance, and then select the network layers with less performance contribution for pruning.

[0053] (3) Determine the threshold value based on the weight. By sorting the weights of the network layers and setting the threshold value, unnecessary or redundant parts of the network can be identified and then pruned.

[0054] (4) Prune the part below the threshold to obtain new parameters. After determining the pruning threshold, prune the network parameters below the threshold. Specifically, compress the weights in the network and remove redundant and unnecessary parameters, thereby reducing the network storage and computational complexity.

[0055] (5) Save the new model parameters and weights to generate a pruned model. The number of model parameters after pruning is reduced, and the amount of calculation is also reduced accordingly, which can significantly improve the reasoning speed and efficiency of the model. It also helps to achieve real-time target detection in resource-constrained scenarios such as embedded devices.

[0056] (6) Use the parameters retained in the pruned large network to initialize the pruned network, and continue to perform several rounds of fine-tuning on the training set to further adjust the model parameters. During the fine-tuning process, sort the weights of the convolutional layer kernels, remove the convolutional kernels with the lowest weight values ​​and their corresponding feature maps, and clear the related connections; in the original pruned convolutional layer, regenerate new convolutional kernels and initialize the weights, use the remaining weight values ​​to assign values ​​to the new convolutional kernels and train them to obtain the final pruned YOLOX network model.

[0057] Finally, the test set is input into the pruned YOLOX network model for network testing to detect the image defects after classification. The image defects here correspond to the local defects under the above-mentioned overall defects, where: (1) Welding defects, including but not limited to: Poor solder joints: Poor solder joints refer to various problems that occur during the welding process, such as excessive solder, solder balls, cold solder joints, pulled tips, cracked solder joints, pad peeling, and asymmetric solder distribution, which result in substandard solder joint quality and affect the reliability and service life of electronic products.

[0058] Broken wire: A break in the wire or connection point on the PCB may cause circuit interruption and affect the normal operation of the equipment.

[0059] Short Circuit: A short circuit between different circuits on a PCB causes current to bypass its intended path, potentially causing device failure.

[0060] Empty solder joint: Empty solder joint refers to the failure of solder to successfully solder components to the substrate. This may be caused by unclean solder joints, poor solderability of components, or improper dispensing. Empty solder joints will affect the conductivity of the circuit board and the stability of components.

[0061] Cold solder joints / false solder joints: Cold solder joints refer to the lack of effective solder joints between component pins and pads, which may be caused by insufficient solder paste, insufficient soldering temperature, or too short soldering time. Cold solder joints can cause circuit instability or component malfunction.

[0062] Dirty solder joints: These include low solder paste viscosity, rough steel mesh wall, too thick coating on PAD, PCB surface contamination, and other similar PCBA surface contamination issues, which affect the appearance and reliability of the PCBA.

[0063] (2) Component defects, including but not limited to: Component misalignment: During the SMT process, components may be misaligned due to equipment failure or operating errors, resulting in misalignment with the pads on the PCB, which in turn affects the overall performance of the circuit board.

[0064] Component skew, that is, the position of components is not correct, which is caused by various factors. For example, during the reflow process, excessive flux flow can cause component displacement; Missing components means there are no components at the corresponding welding positions. The main reasons for missing PCBA components include: unclean templates will cause the solder paste to not work properly, thus affecting the placement of components; poor quality of solder paste will affect the fixation and welding of components, resulting in missing components; outdated equipment may lead to a decrease in placement accuracy, resulting in missing components.

[0065] (3) PCB board defects, including but not limited to: Scratches: Scratches or wear on the surface or internal PCB material can cause signal transmission issues. ‌

[0066] Impurities: Foreign matter or contaminants on or inside the PCB that may cause circuit failure or poor connections.

[0067] Bubbles: During the PCB manufacturing process, bubbles that are incompatible with the PCB material may appear, which may cause signal interference or reduce mechanical strength.

[0068] (4) Other defects, including but not limited to: Standing piece / tombstone: One end of the component soldered is not connected to the circuit and is lifted up, which is usually caused by improper product design or horizontal plane deviation of mounting.

[0069] Solder balls: There are round solder balls in the non-soldering area of ​​the PCB, which may be caused by solder paste resurgence, improper reflow temperature setting, etc.

[0070] Pinhole: There are pinhole holes on the surface of the solder joint, which is usually related to the rewetting of the soldering material or improper reflow temperature.

[0071] The present invention also proposes a PCBA surface defect detection system based on image recognition, comprising: The data acquisition module is used to collect and construct a multi-category PCBA overall defect image data set, which is not limited to welding defects, component defects, PCB board defects, and other defects, and perform preprocessing; the preprocessing includes random rotation, cropping, and normalization; The overall defect classification module builds a pre-trained neural network model to classify the pre-processed PCBA overall defect image dataset; The local recognition building module is used to build a local target recognition method based on the YOLOX network for the overall defect classification results; The local defect detection module is used to perform local defect detection on the PCBA surface based on the local target recognition method.

[0072] The implementation methods of the above modules will not be repeated here, and they correspond to the methods described above respectively.

[0073] The present invention comprehensively applies the strategy of overall classification plus local detection to reduce the burden of the model and improve the detection accuracy and speed. The attention mechanism is introduced to improve the performance and generalization ability of the model. Data enhancement, consistency processing, classification and annotation are performed on the image data set to improve the effect of the model on defect detection. The network model adopts multiple optimization strategies such as structural improvement, loss function improvement, and pruning lightweighting to effectively improve the image target detection performance.

[0074] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A PCBA surface defect detection method based on image recognition, characterized in that: The following steps are involved: Collect and construct a multi-category PCBA overall defect image dataset not limited to welding defects, component defects, PCB board defects, and other defects, and perform preprocessing; the preprocessing includes random rotation, cropping, and normalization; Construct a pre-trained neural network model to classify the pre-processed PCBA overall defect image dataset; According to the overall defect classification results, a local target recognition method based on the YOLOX network is constructed; Local defect detection on PCBA surface is performed based on local target recognition method.

2. The PCBA surface defect detection method based on image recognition according to claim 1, characterized in that: The pre-trained neural network model is constructed to classify the pre-processed PCBA overall defect image data set, specifically: Train classification networks based on MobileNetv2 and EfficientNet-b0 according to the ImageNet dataset; Use the trained MobileNetv2 and EfficientNet-b0 networks as pre-trained models, and transfer training to the pre-trained models. Set the final classification layer to 4 categories. Then design 3 lightweight CNN branches. Use EfficientNet-b0 as the main branch to extract global feature information from PCBA images, and use two MobileNetv2 branches to extract defect features and non-defect features in local areas respectively. After using the Concatenate function to fuse the features of these regions, connect the global average pooling layer, and then add the Dropout layer; A Softmax classifier with 4 classes is used to classify PCBA images.

3. The PCBA surface defect detection method based on image recognition according to claim 2, characterized in that: It also includes the introduction of an attention mechanism to the pre-trained model, specifically by adding a CBAM module before and after the Bottleneck layer of MobileNetv2 to form an Attention-MobileNetV2 network.

4. The PCBA surface defect detection method based on image recognition according to claim 1, characterized in that: The local target recognition method based on the YOLOX network is specifically constructed as follows: The YOLOX network includes the input end, the backbone feature extraction network, the neck network and the prediction end; Perform data augmentation on the original data using methods including but not limited to rotation, scaling, flipping, and cropping at the input end; The backbone feature extraction network is replaced with SDConv, which is responsible for image feature extraction; it includes multiple BConv convolutional layers, E-ELAN convolutional layers, MPConv convolutional layers, and SPPCSPC fusion layers; among them, the BConv convolutional layer extracts image features of different scales, the E-ELAN convolutional layer learns more diverse features by guiding the calculation blocks of different feature groups, the MPConv convolutional layer fuses the feature information processed by normal BConv convolution with the feature information of MaxPool expanded receptive field, and the SPPCSPC fusion layer solves the problems of image distortion and repeated feature extraction; Replace BConv with SDConv in the E-ELAN convolutional layer and MPConv convolutional layer in the neck network area; At the prediction end, the number of image channels is adjusted for the features of large, medium and small scales output by PAFPN through the REPConv convolution layer; Add the attention mechanism into the spatial pyramid pooling module SPPCSPC; Replace the Mish activation function with the Swish activation function.

5. The PCBA surface defect detection method based on image recognition according to claim 1, characterized in that: The local defect detection on the PCBA surface based on the local target recognition method is specifically as follows: After completing the PCBA image defect classification, the images with overall defects are preprocessed again to form a detection data set, and the detection data set is divided into a detection test set and a detection training set; The detection training set and the annotation information formed after the secondary preprocessing are input into the YOLOX network for network training. The YOLOX network calculates the gradient through forward propagation, and uses the stochastic gradient descent algorithm to update the weight parameters of the YOLOX network model according to the gradient; The loss function is calculated based on the prediction results and annotations of the YOLOX network model until the specified number of iterations or convergence conditions are reached to form a basic YOLOX network model; the loss function includes three parts: confidence loss, classification probability loss, and bounding box regression loss; Perform pruning operations on the basic YOLOX network model to generate a pruned improved YOLOX network model, train the pruned improved YOLOX network model on the dataset, and readjust the number of channels or layers in the network structure; Initialize the pruned network using the parameters retained in the pruned large network, and continue to perform several rounds of fine-tuning on the training set to further adjust the model parameters. During the fine-tuning process, sort the weights of the convolutional layer kernels, remove the convolution kernels with the lowest weight values ​​and their corresponding feature maps, and clear the related connections. In the originally pruned convolution layer, a new convolution kernel is regenerated and the weights are initialized. The remaining weights are used to assign and train the new convolution kernel to obtain the final pruned YOLOX network model. The detection test set is input into the pruned YOLOX network model for network testing to detect image defects after classification.

6. The PCBA surface defect detection method based on image recognition according to claim 5, characterized in that: The secondary pretreatment comprises: Each image is automatically oriented and has its EXIF ​​orientation information removed; Perform data consistency processing, adjust the image to a unified 640x640 resolution, and adjust the image size by stretching; Enhance image contrast through adaptive equalization; Generate labels in yolo format, use LabelImg to annotate the images in the dataset with local defects under the corresponding overall defect classification, read the annotation information in the xml file after the image dataset is annotated and write it into a txt file. After running, the annotation information of all image datasets will appear in the label folder.

7. The PCBA surface defect detection method based on image recognition according to claim 5, characterized in that: The pruning operation on the basic YOLOX network model specifically includes: Determine the layer to be pruned based on the importance index of the network layer; for each network layer, calculate the sensitivity of the forward propagation of a network layer on the model, and calculate the partial derivative of the change in the output with respect to the weight of the layer under the given input, so as to obtain the sensitivity of the layer to the output; Sort the weights of the network layers to be pruned; Determine the threshold value based on the weight ranking result and pruning rate in each pruning layer; Remove weights below the threshold in the network and retain weights above the threshold; Save the new model parameters and weights to generate the pruned improved YOLOX network model.

8. The PCBA surface defect detection method based on image recognition according to claim 4, characterized in that: The YOLOX network adds an attention mechanism model CAM in the neck network, specifically: the input is the three feature maps output by the backbone network and the SPP structure. In the 3x3 convolution layer spliced ​​5 times, the CAM module passes through the same 1x1 convolution for the three feature maps, and outputs PCBA defect feature maps F1, F2 and F3. Feature maps F2 and F3 are matrix multiplied to obtain feature map F4, and then the Softmax operation is continued to obtain the channel attention feature map F5. Then F5 is matrix multiplied with F1 that has only undergone 1x1 convolution operation to obtain feature map F6. Finally, F6 is weighted summed with the original image, and then the PCBA surface detail information is enlarged through scale transformation to obtain clearer feature weights.

9. The PCBA surface defect detection method based on image recognition according to claim 8, characterized in that: Also included in the SPP structure splicing 3 3, a depth-separable convolution is introduced, where the input of the network is ( M), output ( N), where the network input size is , M is the number of convolution kernels, and the input PCBA board feature map is processed Convolution, the convolution parameter is M N, the amount of convolution calculation is M N, applying depthwise separable convolution to input and output of the same size, the number of parameters is: M+ M N, the convolution is expressed as a depth-wise separable convolution, and the reduced calculation is: = + 。 10. The PCBA surface defect detection system based on image recognition is characterized by: include: The data acquisition module is used to collect and construct a multi-category PCBA overall defect image data set, which is not limited to welding defects, component defects, PCB board defects, and other defects, and perform preprocessing; the preprocessing includes random rotation, cropping, and normalization; The overall defect classification module is used to build a pre-trained neural network model to classify the pre-processed PCBA overall defect image dataset; The local recognition building module is used to build a local target recognition method based on the YOLOX network for the overall defect classification results; The local defect detection module is used to perform local defect detection on the PCBA surface based on the local target recognition method.

Images