A PCB Defect Detection Method Based on Improved Fully Convolutional Neural Network

Abstract

The present invention relates to the technical field of image processing, and particularly relates to a PCB defect detection method based on an improved fully convolutional neural network, including S1, collecting standard and to-be-detected PCB images; S2, performing grayscale transformation on the images; S3, performing Gaussian filtering, logarithmic transformation, and Otsu threshold segmentation on the images; S4, adopting normalized cross-correlation template matching; S5, performing differential image operation after binarizing the images; S6, performing morphological processing on the images; S7, inputting the images into an improved fully convolutional neural network model. The present invention introduces a continuous dilated convolution module after the fourth convolutional layer, which increases the receptive field and solves the problem of loss of information continuity in ordinary dilated convolution; due to the large upsampling rate of the traditional skip structure, the obtained image effect is not ideal. The present invention introduces an improved skip structure after the upsampling layer, which reduces the upsampling rate, thereby obtaining multi-scale fused features to improve the resolution of the images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a PCB defect detection method based on an improved fully convolutional neural network. Background Art

[0002] There are four common defect types on the bare PCB board, namely burrs, notches, short circuits, and open circuits. "A Parallel Asymmetric Atrous Convolution Module" by Zhang Zhijie et al. uses a parallel asymmetric atrous convolution. Compared with the traditional atrous convolution, it improves the feature expression ability of the network model, but the obtained output image is still not fine enough. At the same time, the traditional neural network model has 5 convolutional layers and pooling layers. Therefore, the ordinary skip structure outputs perform deconvolution operations 32 times, 16 times, and 8 times respectively, and its upsampling rate is too large, so the effect of the picture itself is not ideal. Summary of the Invention

[0003] The technical problems to be solved by the present invention are: to solve the problem of the loss of information continuity in the traditional atrous convolution; to solve the problem of too large upsampling rate and low image resolution in the skip structure.

[0004] The technical solution adopted by the present invention is: a PCB defect detection method based on an improved fully convolutional neural network includes the following steps:

[0005] S1. Image acquisition: Collect standard PCB images and PCB images to be tested.

[0006] S2. Image gray-scale transformation: Perform gray-scale transformation on the collected standard and to-be-tested PCB images.

[0007] The weighted mean method is used for image gray-scale transformation. The weighted average method is to take a weight value for each component value of the three channels of the RGB color image, and then perform weighted averaging, that is:

[0008] Gray(i, j) = ω R *R(i, j) + ω G *G(i, j) + ω B *B(i, j) (1)

[0009] Among them, ω R 、ω G 、ω B are the weights of R, G, and B respectively, and different values are taken to form different gray-scale images;

[0010] S3. Image preprocessing: First, perform Gaussian filtering on the image after gray-scale transformation in step S2, secondly, use logarithmic transformation to enhance the contrast of the image, and finally use the maximum inter-class variance method for threshold segmentation;

[0011] Use a two-dimensional Gaussian distribution function as an image smoothing filter:

[0012]

[0013] Among them, (x, y) represents the template coordinates of the pixel, the center position of the template is the origin, σ is the standard deviation of the normal distribution, and the size of the template depends on the size of the image resolution.

[0014] Secondly, logarithmic transformation is adopted to enhance the contrast of the image, that is:

[0015] s = c * log(1 + r) (3)

[0016] Among them, r is the gray value of the input image, s is the gray value of the output image, and c is a constant;

[0017] Finally, the Otsu method is used for threshold segmentation, which is a black-and-white boundary value of the picture brightness. The optimal threshold is determined by judging the maximum variance of the pixel values of the two parts, that is:

[0018] g = ω o × ω1(μ o - μ1) (4)

[0019] Among them, g is the maximum variance value, ω0 is the proportion of the number of pixels in the background to the whole image, ω1 is the proportion of the number of pixels in the foreground to the whole image, μ0 is the average gray value of the background image, and μ1 is the average gray value of the foreground image.

[0020] S4. Image registration: The normalized cross-correlation template matching method is used for image registration;

[0021] First, let g(x, y) be the searched image with a size of M × N, and f(x, y) be the template image with a size of m × n. Find the area in the searched image that matches the template image;

[0022] Secondly, in the searched image g, take a sub-image with a size of m × n with (i, j) as the upper left corner, calculate its similarity with the template, traverse the entire search image, and find the sub-image that is most similar to the template image among all the sub-images that can be taken as the final matching result;

[0023] Finally, the similarity formula of the MAD algorithm is as follows:

[0024]

[0025] Among them, \(0\leq i\leq M - m\), \(0\leq j\leq N - n\), \(f(s,t)\) represents the pixel coordinates in the template image. Formula (5) calculates the average difference of the template image on the searched image, and then moves one pixel unit from left to right and from top to bottom in turn. Each time it moves, it calculates the average difference of this area. After traversing the whole, find an area with the smallest average difference, and the smallest area is the area "most similar" to the template image.

[0026] The smaller the mean absolute difference \(D(i,j)\), the more similar it indicates. Therefore, only by finding the smallest \(D(i,j)\) can the position of the matching sub - image be determined.

[0027] S5. Difference operation processing: After binarizing the standard PCB image and the PCB image to be measured, perform a difference operation. To detect and classify the defects of the PCB image to be measured, it is necessary to extract the defect area of the PCB image. Utilize the similarity between the standard PCB and the PCB image to be measured for image segmentation, that is, after binarizing the standard PCB image and the PCB image to be measured, perform a difference operation. The difference operation is to subtract the corresponding pixel matrices of the two images to obtain the difference value of the digital matrix and display it with an image. When selecting the function for difference calculation, use the imabsdiff function.

[0028] S6. Morphological processing: Perform the processing of erosion first and then dilation on the image after the difference operation to restore the characteristics of the defects.

[0029] After obtaining the difference image, there will still be certain deviations. For the accuracy of defect detection, it is necessary to clear these misjudgment information and perform morphological processing on the graph again. Perform the processing of erosion first and then dilation on the difference image to remove those tiny differences and restore the characteristics of the real defects.

[0030] S7. Input the processed PCB image into the improved fully convolutional neural network model, and identify the defect targets of the PCB image to be measured through feature fusion.

[0031] The improved fully convolutional neural network model inserts a dilated convolution module and a skip - connection structure module respectively between the fourth convolutional layer, upsampling and softmax classifier of the fully convolutional neural network model.

[0032] The dilated convolution module is composed of consecutive dilated convolution modules with dilation rates of 1, 2, and 4 respectively. Introduce consecutive dilated convolution modules after the fourth convolutional layer to solve the problem of loss of information continuity of ordinary dilated convolution while increasing the receptive field.

[0033] The dilated convolution (i.e., atrous convolution) module injects holes into the standard convolution kernel to achieve the purpose of expanding the receptive field (the receptive field is equivalent to a field of view, and the size of the receptive field is equal to the size of the convolution kernel). Compared with traditional convolution, dilated convolution can not only preserve the internal structure of the data but also make up for the information loss caused by pooling. However, it also brings problems such as the loss of spatial hierarchy and information continuity.

[0034] The dilated convolution module used in the present invention takes three dilation rates as a group. By combining dilated convolutions with different dilation rates, it fully extracts features of different scales and makes up for the problem of the loss of information continuity in ordinary dilated convolution.

[0035] The skip connection structure module sets the transposed convolution multiples of the upsampling to 8, 4, and 2.

[0036] The fully convolutional neural network adopted in the present invention has 3 convolutional layers and pooling layers. The transposed convolution operation is reduced to 8 times, 4 times, and 2 times. Because the magnification factor of the picture itself is reduced, the clarity of the image increases, improving the accuracy of subsequent image training.

[0037] The beneficial effects of the present invention are as follows:

[0038] 1. A continuous dilated convolution module is introduced after the fourth convolutional layer, increasing the receptive field while solving the problem of the loss of information continuity in ordinary dilated convolution.

[0039] 2. Due to the relatively large upsampling rate of the traditional skip connection structure, the obtained picture effect is not ideal. The present invention introduces an improved skip connection structure after the upsampling layer, reducing the upsampling rate, thereby obtaining features of multi-scale fusion to improve the resolution of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flowchart of the PCB defect detection method based on the improved fully convolutional neural network of the present invention;

[0041] Figure 2 is the template matching effect diagram of the present invention;

[0042] Figure 3 is the improved fully convolutional neural network model of the present invention;

[0043] Figure 4 is the continuous dilated convolution module of the present invention;

[0044] Figure 5 is the effect diagram after continuous dilated convolution of the present invention;

[0045] Figure 6 is the schematic diagram of the improved skip connection structure of the present invention;

[0046] Figure 7 It is a comparison chart of the effects of the method of the present invention, FCN, and CNN. Specific embodiments

[0047] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner. Therefore, it only shows the components related to the present invention.

[0048] As Figure 1 shown, a PCB defect detection method based on an improved fully convolutional neural network includes the following steps:

[0049] S1. Image acquisition: A dataset of 474 images of four defect types, namely burrs, notches, short circuits, and open circuits, composed of the collected standard PCB images and the PCB images to be tested. Among them, there are 212 PCB short circuit defect images;

[0050] S2. Image grayscale transformation: Perform grayscale transformation on the collected standard and test PCB images;

[0051] The weighted mean method is used for image grayscale transformation. The weighted average method is to take a weight value for each component value of the three channels of the RGB color image, and then perform weighted averaging, that is:

[0052] Gray(i, j) = 0.299R(i, j) + 0.587G(i, j) + 0.114B(i, j) (1)

[0053] Among them, ω R , ω G , ω B are the weight values of R, G, and B respectively. Different values are taken to form different grayscale images. In this embodiment, the grayscale image obtained when ω R = 0.299, ω G = 0.587, and ω B = 0.114 has the best effect.

[0054] S3. Image preprocessing: First, perform Gaussian filtering on the image after grayscale transformation in step S2. Secondly, use logarithmic transformation to enhance the contrast of the image. Finally, use the maximum inter-class variance method for threshold segmentation;

[0055] Use the two-dimensional Gaussian distribution function as an image smoothing filter:

[0056]

[0057] Among them, (x, y) represents the template coordinates of the pixel. The center position of the template is the origin, σ is the standard deviation of the normal distribution, and the size of the template depends on the image resolution. In this embodiment, the image resolution is 806×725.

[0058] Secondly, logarithmic transformation is used to enhance the contrast of the image, that is:

[0059] s = c * log(1 + r) (3)

[0060] where r is the gray value of the input image, s is the gray value of the output image, and c = 1;

[0061] Finally, the Otsu method is used for threshold segmentation, which is a black-and-white boundary value for the brightness of the picture. The optimal threshold is determined by judging the maximum variance of the pixel values of the two parts, that is:

[0062] g = ω0 × ω1(μ0 - μ1) (4)

[0063] where g is the maximum variance value, ω0 is the proportion of the number of pixels in the background to the whole image, ω1 is the proportion of the number of pixels in the foreground to the whole image, μ0 = 127, and μ1 = 255.

[0064] S4. Image registration: The normalized cross-correlation template matching method is used for image registration;

[0065] Template matching is as Figure 2 shown. First, let g(x, y) be the searched image with size M×N, and f(x, y) be the template image with size m×n. Find the region in the searched image that matches the template image;

[0066] Secondly, in the searched image g, take a sub-image with size m×n with (i, j) as the upper left corner, calculate its similarity with the template, traverse the entire searched image, and find the sub-image that is most similar to the template image among all the sub-images that can be taken as the final matching result;

[0067] Finally, the similarity formula of the MAD algorithm is as follows:

[0068]

[0069] where 0 ≤ i ≤ M - m, 0 ≤ j ≤ N - n, f(s, t) represents the pixel point coordinates in the template image. Formula (5) calculates the average difference of the template image on the searched image, and then moves one pixel unit from left to right and from top to bottom in turn. Each time it moves, calculate the average difference of this region; then after traversing the whole, find a region with the smallest average difference, and the smallest region is the region "most like" the template image; the smaller the mean absolute difference D(i, j), the more similar it is. Therefore, only by finding the smallest D(i, j) can the position of the matching sub-image be determined.

[0070] S5. Subtraction operation processing: Binarize the standard PCB image and the PCB image to be measured and then perform subtraction operation;

[0071] The so-called differential imaging operation is to subtract the pixel matrices corresponding to two images to obtain the difference value of the digital matrix and display it with an image. When selecting the function for differential imaging calculation, the present invention selects the imabsdiff function, that is:

[0072] T(x) = |P(x) - Q(x)| (6)

[0073] The Imabsdiff function performs subtraction operations on digital images, but subtraction operations sometimes cause some pixel values to become negative numbers. For data of uint8 or uint16 types, the absolute value is taken after imabsdiff calculation.

[0074] S6. Morphological processing: Perform erosion first and then dilation on the image after differential imaging operation to restore the characteristics of the defect;

[0075] After obtaining the differential image, there will still be some deviations. For the accuracy of defect detection, it is necessary to remove these misjudgment information, so morphological processing is performed on the graph again. It is necessary to perform erosion first and then dilation on the differential image. Through the erosion operation, those double images and small differences that do not constitute defects are removed, and then the characteristics of real defects are restored through the dilation process. The so-called dilation process is the process of finding the local maximum value, while the erosion process is exactly the opposite, which is the process of finding the local minimum value.

[0076] S7. Input the processed PCB image into the improved fully convolutional neural network model, and identify the defect targets of the PCB image to be tested through feature fusion;

[0077] Such as Figure 3 is the improved fully convolutional neural network model. The improved fully convolutional neural network model inserts a dilated convolution module and a skip structure module between the fourth convolutional layer, upsampling, and softmax classifier of the fully convolutional neural network model;

[0078] Such as Figure 4 The dilated convolution module is composed of consecutive dilated convolution modules with dilation rates of 1, 2, and 4 respectively; After the fourth convolutional layer, consecutive dilated convolution modules are introduced, such as Figure 5 is the effect diagram after consecutive dilated convolution. The receptive field of the original convolution kernel of 3×3 becomes a receptive field of 15×15, and it also makes up for the problem of the continuity of dilated convolution, extracts clear features of the image, and prepares for the feature fusion of the skip structure;

[0079] Embed the proposed continuous dilated convolution module of the present invention into the ResNet-50 network, and compare it with the original basic network and the parallel asymmetric dilated convolution module. In this embodiment, a dataset of PCB short circuit defect maps is used, where the training set and the test set are 116 and 96 respectively. The images of all datasets are of the same size specification, with a resolution of 806×725. The learning rate is set to 0.001, and the number of epochs is set to 30. Each epoch iterates through the training set once; the initial convolution kernel size is 3×3, and dilation is performed at dilation rates of 1, 2, and 4 respectively to fully extract features of different scales. The comparison results of the obtained training accuracies are shown in Table 1:

[0080] Table 1 Comparison of training accuracies based on the ResNet-50 network

[0081]

[0082] As can be seen from Table 1, the effects of embedding the continuous dilated convolution module, parallel asymmetric dilated convolution, and basic network of the present invention in ResNet-50 are compared. The training accuracy of the continuous convolution module of the present invention is significantly improved.

[0083] As Figure 6 shown, the schematic diagram of the improved skip connection structure of the present invention. The traditional network model has 5 convolutional layers and pooling layers. Therefore, the ordinary skip connection structure outputs perform deconvolution operations at 32 times, 16 times, and 8 times respectively, and its upsampling rate is relatively large. Therefore, the effect of the picture itself is not ideal; the improved fully convolutional neural network of the present invention has 3 convolutional layers and pooling layers. Therefore, the deconvolution operation is reduced to 8 times, 4 times, and 2 times. Because the magnification factor of the picture itself is reduced, the clarity of the image is increased, and the accuracy of subsequent image training is improved.

[0084] Use the dataset of PCB short circuit defect maps to train and verify the accuracy of the improved fully convolutional neural network model of the present invention. During the training process, the batch size is set to 32, the learning rate is set to 0.001, and the number of epochs is set to 30. Each epoch iterates through the training set once.

[0085] As shown in Table 2, the evaluation results of three objective evaluation indicators for three different training methods. It can be seen that the training time, loss function, and training accuracy of the algorithm proposed in the present invention are all better than the other two methods, indicating the superiority and effectiveness of the algorithm in this paper, and it has more application value. The table is as follows:

[0086] Table 2 Results of evaluation indicators for training on the "PCB" dataset

[0087]

[0088] Figure 7 For the comparison of the result graphs of three training images, where Figure 7 (a), (b), (c), and (d) of Figure 7 are the standard defect graph, the algorithms CNN, FCN, and the method of the present invention respectively. It can be seen from the figure that the clarity of the images of the algorithm of the present invention is significantly better than that of the CNN and FCN algorithms after the improved skip connection structure.

[0089] Inspired by the ideal embodiments of the present invention described above, through the above description, relevant staff can completely make various changes and modifications without departing from the technical idea of the present invention. The technical scope of the present invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A PCB defect detection method based on an improved fully convolutional neural network, characterized in that, It includes the following steps: S1. Image acquisition: Acquire the standard PCB image and the PCB image to be measured; S2. Image gray transformation: Perform gray transformation on the acquired standard PCB image and the PCB image to be measured; S3. Image preprocessing: First, perform Gaussian filtering on the image after gray transformation in step S2, then use logarithmic transformation to enhance the contrast of the image, and finally use the maximum between-class variance method for threshold segmentation; Use the maximum between-class variance method for threshold segmentation, and determine the optimal threshold by judging the maximum variance of the pixel values of the two parts, that is: ； where g is the maximum variance value, is the proportion of the number of background pixels in the entire image, is the proportion of the number of foreground pixels in the entire image, is the average gray value of the background image, is the average gray value of the foreground image; S4. Image registration: Use the normalized cross-correlation template matching method for image registration; S5. Subtraction operation processing: Perform a subtraction operation after binarizing the standard PCB image and the PCB image to be measured; S6. Morphological processing: Perform the processing of erosion first and then dilation on the image after the subtraction operation to restore the characteristics of the defects; S7. Input the processed PCB image into the improved fully convolutional neural network model, and identify the defect targets of the PCB image to be measured through feature fusion; The improved fully convolutional neural network model inserts a dilated convolution module and a skip connection structure module between the fourth convolutional layer, upsampling, and softmax classifier of the fully convolutional neural network model respectively; The dilated convolution module is composed of consecutive dilated convolution modules with dilation rates of 1, 2, and 4 respectively; The skip connection structure module sets the deconvolution multiples of upsampling to 8, 4, and 2.

Images