PCB defect detection method based on multilayer feature fusion and transfer learning

Abstract

The invention discloses a PCB defect detection method based on multilayer feature fusion and transfer learning. The method comprises the following steps: acquiring a to-be-detected PCB image; the to-be-detected PCB diagram is input into the defect detection model subjected to transfer learning, a detection result is obtained, and the defect detection model comprises a MobileNetV2 module, an FPN module, a Faster R-CNN module and a transfer learning module; the MobileNetV2 module is used for extracting the image features to obtain a convolution feature map; the FPN module is used for enhancing the convolution feature map and obtaining a multi-scale feature map; the Faster R-CNN module is used for carrying out classification and bounding box regression on the multi-scale feature map to obtain a detection result; and the transfer learning module is used for selecting the NEU-DET data set for pre-training, obtaining the weight of a pre-training model, adjusting the weights of other modules based on the weight, and obtaining data features.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and in particular relates to a PCB defect detection method based on multi-layer feature fusion and transfer learning. Background Art

[0002] With the rapid development of electronic chips, printed circuit boards (PCBs), as the hub connecting various electronic components, are becoming a key component in modern manufacturing, providing hardware support for various types of equipment. With the increasing demand for PCBs and the increase in production speed, modern production equipment is becoming more and more compact. PCB manufacturing has also become more refined, resulting in a continuous reduction in the size of components that make up the PCB. This trend has made it increasingly difficult to detect minor defects such as missing holes, rat bites, open circuits, and short circuits during the PCB manufacturing process.

[0003] Traditional PCB defect detection mainly relies on manual work, which has low accuracy and high cost. The introduction of deep learning technology has greatly improved the accuracy of PCB defect detection, but there is still a lack of a method with high detection speed and high detection accuracy. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a PCB defect detection method based on multi-layer feature fusion and transfer learning. The present invention introduces transfer learning to realize model training of small samples, which can improve the accuracy and robustness of PCB defect detection.

[0005] To achieve the above object, the present invention provides a PCB defect detection method based on multi-layer feature fusion and transfer learning, comprising:

[0006] Get the PCB image to be tested;

[0007] Input the PCB image to be inspected into the defect detection model after transfer learning to obtain the detection result, wherein the defect detection model includes: MobileNetV2 module, FPN module, Faster R-CNN module and transfer learning module;

[0008] The MobileNetV2 module is used to extract image features and obtain a convolution feature map;

[0009] The FPN module is used to enhance the convolution feature map and obtain a multi-scale feature map;

[0010] The Faster R-CNN module is used to classify and regress the multi-scale feature map to obtain a detection result;

[0011] The transfer learning module is used to select the NEU-DET data set for pre-training, obtain the feature extraction and classifier weights of the pre-training model, the feature extraction is used to extract the features of the PCB data, and the weights are used to fine-tune the weights of other modules.

[0012] Optionally, the NEU-DET dataset includes: mill scale, plaque, cracks, pitting surfaces, inclusions and scratches.

[0013] Optionally, obtaining the PCB defect detection data set includes:

[0014] Get the initial PCB defect map;

[0015] The initial PCB defect image is subjected to histogram equalization, bilateral filtering and data enhancement processing in sequence to obtain the PCB defect detection data set.

[0016] Optionally, performing histogram equalization processing on the initial PCB defect map includes:

[0017] Based on the initial PCB defect map, the RGB color channels are separated, the histograms of the RGB color channels are adjusted respectively, the adjusted histograms of the RGB color channels are overlapped and combined, and the average histogram of the combined RGB color channels is extracted.

[0018] Optionally, the method of performing bilateral filtering on the initial PCB defect image is:

[0019]

[0020] Where I is the input image, p and q are the x and y coordinates of the current pixel to be filtered, is the spatial kernel that affects the pixel distance in coordinates, is the range kernel used to smooth intensity differences, W p is the normalized weight.

[0021] Optionally, performing data enhancement processing on the initial PCB defect image includes:

[0022] Random cropping, horizontal flipping, vertical flipping, injection of Gaussian noise and speckle noise are used for data enhancement.

[0023] Optionally, the MobileNetV2 module includes: a depth-separable convolutional layer, a linear bottleneck layer, and an inverted residual structure layer;

[0024] The depthwise separable convolutional layer is used to change the number of output channels;

[0025] The linear bottleneck layer is used to reduce information loss during feature extraction;

[0026] The inverted residual structure layer outputs the image features.

[0027] Optionally, the FPN module includes: a first convolution layer, an upsampling layer, and a second convolution layer;

[0028] The first convolutional layer is used to reduce the channel depth;

[0029] The upsampling layer combines feature maps using a nearest neighbor upsampling algorithm;

[0030] The second convolutional layer is used to reduce the aliasing effect when the merged feature map is combined with the upsampling layer.

[0031] Optionally, the FasterR-CNN module includes: a region generation network layer and a region detector;

[0032] The region generation network layer is used to slide the multi-scale feature map using a sliding window to obtain a target candidate region;

[0033] The region detector is used to classify and regress the target candidate region to obtain a detection result.

[0034] Optionally, the region detector includes: a RoIPooling layer, a fully connected layer and an output layer;

[0035] The RoIPooling layer is used to map the target candidate region to a feature map and extract fixed features;

[0036] The fully connected layer is used to classify and accurately regress the features;

[0037] The output layer is used to output the category label and bounding box coordinates of the target candidate area.

[0038] Compared with the prior art, the present invention has the following advantages and technical effects:

[0039] 1. The present invention introduces transfer learning, which reduces the time and computing resources required to train the model from scratch. Through training on the NEU-DET dataset, the model has learned the defect feature extraction capability, and can still have high accuracy when the PCB dataset is small.

[0040] 2. Based on the existing PCB defect detection data set, the present invention performs data preprocessing to enrich image samples and obtains rich image processing samples; extracts features based on MobileNetV2 and connects with FPN, and finally detects and classifies through Faster R-CNN. The model automatically captures, detects and classifies, and anchors the bounding box for detection. The present invention aims to improve the defect detection method for PCBs and strengthen industrial quality control. The model will significantly improve productivity and quality inspection, and prevent defective materials from being processed to consumers. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 It is a flow chart of a PCB defect detection method based on multi-layer feature fusion and transfer learning according to an embodiment of the present invention;

[0043] Figure 2 FIG. 4 is a structural diagram of a trap detection model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] In order to accelerate the convergence speed of the model, improve the generalization ability of the model on new data sets, and enable the deep learning model to more effectively identify small and complex defects on PCBs, this embodiment adopts the method of transfer learning. The generalized feature representations learned by the deep neural network that has been fully trained on a large, general data set (NEU-DET) are used. These feature representations usually contain rich visual information and hierarchical abstract capabilities, and can capture key information in the image. By using these pre-trained models as a starting point, we migrate them to the field of PCB defect detection, and fine-tune some layers of the model to adapt to the unique defect patterns and background noise in PCB images.

[0047] The present invention proposes a PCB defect detection method based on multi-layer feature fusion and transfer learning, such as Figure 1 As shown, the specific steps include:

[0048] Get the PCB image to be tested;

[0049] Inputting the PCB image to be detected into the defect detection model to obtain the detection result, wherein the PCB image to be detected is input into the defect detection model to obtain the detection result, wherein building the defect model includes: selecting the NEU-DET data set for pre-training to obtain the pre-trained model, and using the PCB defect detection data set to train the pre-trained model to obtain the defect detection model;

[0050] like Figure 2 As shown, the defect detection model includes: MobileNetV2 module, FPN module and Faster R-CNN module;

[0051] MobileNetV2 module, used to extract image features and obtain convolution feature maps;

[0052] FPN module, used to enhance the convolution feature map and obtain multi-scale feature maps;

[0053] The Faster R-CNN module is used to classify and regress the multi-scale feature maps to obtain the detection results.

[0054] Specifically, this embodiment combines a variety of data preprocessing techniques and an enhanced MobileNetV2 detection model, connected with FPN and Faster R-CNN, to achieve good detection accuracy without affecting real-time detection speed. The MobileNetV2 detection model is used in the backbone network to combine shallow feature information with deep feature information to prevent excessive loss of detail information during feature extraction, and dynamic upsampling is combined in the neck network to improve the model's detail perception ability.

[0055] Furthermore, the NEU-DET dataset includes: rolling scale, plaques, cracks, pitting surfaces, inclusions and scratches.

[0056] Specifically, the reason for selecting the above datasets is that the defect types of the NEU-DET dataset and PCB defects have certain similarities in the identification of surface defects, for example, both involve changes in surface texture and color, so they can be used as a starting point in transfer learning. Although the defects of hot-rolled steel strips (such as plaques, scratches, etc.) and PCB defects (such as burrs and pinholes) are different in specific manifestations, they have certain commonalities in visual features. These commonalities can help the model capture abnormal surface patterns in transfer learning. For example:

[0057] Surface anomalies: Both involve anomalies on the surface, such as scratches, spots or cracks, which usually appear as changes in the surface texture.

[0058] Texture and color changes: Defects often result in uneven surface texture or color changes. For example, scratches or patches can affect the consistency of the surface, and similarly, burrs or pinholes on the PCB can cause visual anomalies.

[0059] Edges and Shapes: Surface defects may appear as fuzzy-edged, irregular shapes, a visual feature found in defects of both materials.

[0060] Furthermore, obtaining a PCB defect detection data set includes:

[0061] Get the initial PCB defect map;

[0062] The initial PCB defect image is processed by histogram equalization, bilateral filtering and data enhancement in sequence to obtain a PCB defect detection data set.

[0063] Specifically, based on the existing PCB defect detection dataset, data preprocessing is performed to enrich image samples, that is, to ensure that there are enough PCB defect data in the training set. The initial PCB defect image is histogram equalized, and the brightness of the PCB defect image can be better distributed on the histogram. In this way, it can be used to enhance local contrast without affecting the overall contrast, and can effectively expand the commonly used brightness. Each pixel value is replaced by the weighted average of nearby pixel values ​​through bilateral filtering, while considering the change in intensity to retain the edge. Finally, in order to provide a sufficient number of training image data, the filtered images are data augmented, and more than 20 different types of images are generated from a single image source. This diversity encourages the model to learn in a deeper and wider range.

[0064] Furthermore, the histogram equalization processing of the initial PCB defect map includes:

[0065] Based on the PCB defect image, the RGB color channels are separated, the histograms of the RGB color channels are adjusted respectively, the histograms of the adjusted RGB color channels are overlapped and combined, and the average histogram of the combined RGB color channels is extracted.

[0066] Specifically, for an RGB color image, the image is first divided into three channels, and then the pixels of each channel are counted by histogram to obtain the probability density function of pixel distribution. Then the probability density function is histogram equalized according to the transformation formula to obtain a new probability density function, and then the pixels are replaced, and finally the new three-channel image is merged.

[0067] The above process relies on a probability distribution function to achieve pixel transformation.

[0068] Transformation formula:

[0069]

[0070] Among them, T(r k ) is the gray value r k The new gray value after transformation, P r (r) is the probability distribution, L is the intensity level of the image channel, n kn is the probability of occurrence of the image channel intensity level shown in formula (2), where n k and MN represent the number of intensity levels and the total number of pixels in the image, respectively.

[0071] Furthermore, the method for performing bilateral filtering on the PCB defect image is as follows:

[0072]

[0073] Where I is the input image, p and q are the x and y coordinates of the current pixel to be filtered, is the spatial kernel that affects the pixel distance in coordinates, is the range kernel used to smooth intensity differences, W p for.

[0074] Specifically, based on the existing PCB defect map, a bilateral filter is applied. This filter replaces each pixel value with the weighted average of nearby pixel values ​​while taking into account the intensity changes to preserve the edges. Mathematically, this filter can be expressed as equation (3).

[0075] Here W p The normalized term is:

[0076]

[0077] The image is improved due to noise smoothing while preserving edges. Noise is removed by bilateral filtering the image with a single pass 5×5 kernel, σ s and σ r They are 75 and 50 respectively, with a stride of 1.

[0078] Furthermore, data enhancement processing of the PCB defect map includes:

[0079] Random cropping, horizontal flipping, vertical flipping, injection of Gaussian noise and speckle noise are used for data enhancement.

[0080] Specifically, random cropping and horizontal and vertical image flipping were used. In addition, Gaussian noise and speckle noise were injected into the image data. Through this process, more than 20 different types of images were finally generated from a single image source.

[0081] Furthermore, the MobileNetV2 module includes:

[0082] 1. Depth-wise separable convolution: MobileNetV2 continues to use the depth-wise separable convolution in MobileNetV1, which decomposes the traditional convolution operation into two simpler operations: depth-wise convolution and point-wise convolution. This makes the model calculation more efficient.

[0083] 2. Linear bottleneck: This is a core innovation of MobileNetV2. It uses a linear activation function in each bottleneck layer of the network instead of the traditional ReLU activation function. The linear bottleneck can effectively reduce information loss and improve the expressiveness of the model.

[0084] 3. Inverted residual structure: MobileNetV2 uses an inverted residual structure to further improve efficiency. The inverted residual structure consists of a deep convolution layer, a point-by-point convolution layer, and a short-circuit connection. This can better capture and transmit feature information.

[0085] 4. Non-linear transformation: In the inverted residual structure, the depth convolution is followed by a non-linear activation layer (ReLU6). This transformation helps the model avoid the saturation problem of ReLU while retaining the non-linear characteristics.

[0086] Specifically, MobileNetV2 extracts image features and generates a convolutional feature map by fusing features of different scales.

[0087] Further, the FPN module includes: a first convolutional layer, an upsampling layer, and a second convolutional layer;

[0088] The first convolutional layer is used to reduce the channel depth;

[0089] The upsampling layer combines feature maps using the nearest neighbor upsampling algorithm;

[0090] The second convolutional layer is used to reduce the aliasing effect when merging feature maps and upsampling layers.

[0091] Specifically, after generating the basic convolutional feature map, FPN is connected with MobileNetV2 to generate enhanced feature maps of various resolutions. The network changes the network of the model through a top-down path and horizontal connections to effectively create a rich multi-scale feature pyramid. The output feature maps of the 19th, 14th, 7th, and 4th layers of MobileNetV2 are used. The first feature map passes through a 1×1 convolution layer to reduce the channel depth. This feature map is then deployed as the first feature layer for object detection. Using the nearest neighbor upsampling algorithm, this layer is upsampled by a factor of 2 through a top-down path. The following branch of processing involves applying a 1×1 convolution layer to the next corresponding feature map and the subsequent element-by-element addition of the upsampled feature map. In order to reduce the aliasing effect when the merged feature map is combined with the upsampling layer, a 3×3 convolution layer is applied. All layers are iterated in this way to generate maps with the highest possible resolution.

[0092] Furthermore, the Faster R-CNN module includes: a region generation network layer and a region detector;

[0093] The region generation network layer is used to slide the multi-scale feature map using a sliding window to obtain the target candidate region;

[0094] Region detector, used to classify and regress the target candidate region and obtain the detection result.

[0095] Furthermore, the region detector includes: a RoIPooling layer, a fully connected layer, and an output layer;

[0096] RoIPooling layer, used to map the target candidate region to the feature map and extract fixed features;

[0097] Fully connected layer, used for classification and precise regression of features;

[0098] The output layer is used to output the category labels and bounding box coordinates of the target candidate area.

[0099] Specifically, the region generation network (RPN) in FasterR-CNN receives the multi-scale feature map output by FPN as input. RPN is a lightweight network that slides a small window (3x3 convolution kernel is used in this paper) on each feature map and generates multiple anchor boxes for each sliding position. For each anchor box, RPN predicts the probability of whether it is a foreground (i.e., contains an object) and the offset between the anchor box and the true bounding box. Based on these predictions, RPN selects the candidate regions (region proposals) that are most likely to be the object.

[0100] Finally, these candidate regions are fed into the FastR-CNN detector in FasterR-CNN for further classification and bounding box regression. The FastR-CNN detector first uses a RoIPooling layer (or a RoIAlign layer for more precise results) to map the candidate regions to the feature map and extract fixed-size feature representations. These feature representations are then fed into a series of fully connected layers (or more modern convolutional layers plus global average pooling layers) for classification and precise bounding box regression. Ultimately, the category label and precise bounding box coordinates of each candidate region are output, completing the entire object detection process.

[0101] At the same time, in order to accelerate the convergence speed of the model, improve the generalization ability of the model on the new data set, and enable the deep learning model to more effectively identify small and complex defects on the PCB, this embodiment adopts the method of transfer learning. The generalized feature representation learned by the deep neural network that has been fully trained on a large, general data set (NEU-DET) is used. These feature representations usually contain rich visual information and hierarchical abstract capabilities, and can capture key information in the image. By taking these pre-trained models as a starting point, they are migrated to the field of PCB defect detection, and by fine-tuning some layers of the model to adapt to the unique defect patterns and background noise in PCB images.

[0102] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A PCB defect detection method based on multi-layer feature fusion and transfer learning, characterized in that: include: Get the PCB image to be tested; Input the PCB image to be detected into the defect detection model to obtain the detection result, wherein the defect detection model is constructed including: selecting the NEU-DET data set for pre-training to obtain the pre-trained model, and training the pre-trained model using the PCB defect detection data set to obtain the defect detection model; The defect detection model includes: MobileNetV2 module, FPN module and Faster R-CNN module; The MobileNetV2 module is used to extract image features and obtain a convolution feature map; The FPN module is used to enhance the convolution feature map and obtain a multi-scale feature map; The Faster R-CNN module is used to classify and regress the multi-scale feature map to obtain a detection result.

2. According to claim 1, a PCB defect detection method based on multi-layer feature fusion and transfer learning is characterized in that: The NEU-DET dataset includes: mill scale, plaque, cracks, pitting surfaces, inclusions and scratches.

3. A PCB defect detection method based on multi-layer feature fusion and transfer learning according to claim 2, characterized in that: Acquiring the PCB defect detection data set includes: Get the initial PCB defect map; The initial PCB defect image is subjected to histogram equalization, bilateral filtering and data enhancement processing in sequence to obtain the PCB defect detection data set.

4. A PCB defect detection method based on multi-layer feature fusion and transfer learning according to claim 3, characterized in that: Performing histogram equalization processing on the initial PCB defect map includes: Based on the initial PCB defect image, the RGB color channels are separated, and the pixels of each channel are statistically analyzed by histogram to obtain a probability density function of pixel distribution; Based on the probability density function, a transformation formula is used to perform histogram equalization to obtain a new probability density function; Based on the new probability density function, the histograms of the RGB color channels are overlapped and combined, and the average histogram of the combined RGB color channels is extracted.

5. The PCB defect detection method based on multi-layer feature fusion and transfer learning according to claim 3 is characterized in that: The method for performing bilateral filtering on the initial PCB defect image is as follows: Where I is the input image, p and q are the x and y coordinates of the current pixel to be filtered, is the spatial kernel that affects the pixel distance in coordinates, is the range kernel used to smooth intensity differences, W p is the normalized weight value.

6. The PCB defect detection method based on multi-layer feature fusion and transfer learning according to claim 3 is characterized in that: Performing data enhancement processing on the initial PCB defect map includes: Random cropping, horizontal flipping, vertical flipping, injection of Gaussian noise and speckle noise are used for data enhancement.

7. The PCB defect detection method based on multi-layer feature fusion and transfer learning according to claim 1 is characterized in that: The MobileNetV2 module includes: a depth-separable convolutional layer, a linear bottleneck layer, and an inverted residual structure layer; The depthwise separable convolutional layer is used to change the number of output channels; The linear bottleneck layer is used to reduce information loss during feature extraction; The inverted residual structure layer outputs the image features.

8. The PCB defect detection method based on multi-layer feature fusion and transfer learning according to claim 1 is characterized in that: The FPN module includes: a first convolutional layer, an upsampling layer and a second convolutional layer; The first convolutional layer is used to reduce the channel depth; The upsampling layer combines feature maps using a nearest neighbor upsampling algorithm; The second convolutional layer is used to reduce the aliasing effect when the merged feature map is combined with the upsampling layer.

9. The PCB defect detection method based on multi-layer feature fusion and transfer learning according to claim 1 is characterized in that: The Faster R-CNN module includes: a region generation network layer and a region detector; The region generation network layer is used to slide the multi-scale feature map using a sliding window to obtain a target candidate region; The region detector is used to classify and regress the target candidate region to obtain a detection result.

10. A PCB defect detection method based on multi-layer feature fusion and transfer learning according to claim 9, characterized in that: The region detector includes: a RoIPooling layer, a fully connected layer and an output layer; The RoIPooling layer is used to map the target candidate region to a feature map and extract fixed features; The fully connected layer is used to classify and accurately regress the features; The output layer is used to output the category label and bounding box coordinates of the target candidate area.

Images