Fine-grained recognition method and system for defect image of printed circuit board

Abstract

The present invention belongs to the technical field of machine vision. Provided are a fine-grained recognition method and system for a defect image of a printed circuit board. Compared with the prior art, the present invention establishes a lightweight convolutional neural network model and uses a channel pruning method to perform compression training on the network model. The channel pruning method combines the L2-norm of convolution kernel parameters and the Euclidean distance to evaluate the importance of convolution parameters, and during the network model training process, convolution parameters with lower importance are set to 0. A recognition model obtained by training the network model can accurately and quickly detect surface defects of printed circuit boards.

Description

A method and system for fine-grained recognition of defective images of printed circuit boards

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 2023114392746, filed with the Patent Office of China on October 31, 2023, entitled "A Method and System for Fine-Grained Identification of Defective Images of Printed Circuit Boards," the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention is applicable to the field of machine vision technology, and in particular relates to a method and system for fine-grained recognition of defective images of printed circuit boards. Background Art

[0004] Printed circuit boards (PCBs) are key components of electronic products and crucial carriers for integrating various electronic components. With their widespread application, PCBs are increasingly demanding higher precision, complexity, and performance, placing higher quality demands on them. Due to the complex nature of the PCB production process, various surface defects are prone to forming. These defects can severely impact circuit performance and reliability, necessitating the urgent need for better defect detection methods to improve product quality.

[0005] Due to technical reasons, most researchers currently use manual visual inspection and convolutional neural network inspection as methods for PCB surface defect detection. However, these methods have the following problems: (1) Manual visual inspection is inefficient, labor costs are high, and defect detection quality is difficult to guarantee. (2) Large-model convolutional neural networks have high computational costs, high network redundancy, slow detection speed, and small defects are easily missed.

[0006] To address the above shortcomings, the present invention provides a fine-grained recognition method and system for defective images of printed circuit boards. By improving the network structure and performing channel pruning to make it more lightweight, the system is finally deployed on a high-performance ASIC chip to detect various PCB surface defects.

[0007] Summary of the Invention

[0008] The present invention provides a fine-grained recognition method and system for defect images of printed circuit boards, aiming to solve the problems of low manual visual inspection efficiency, high labor costs, difficulty in ensuring defect detection quality, high computational costs of large-model convolutional neural networks, high network redundancy, slow detection speed, and easy omission of small defects in the existing technology for printed circuit board surface defect detection.

[0009] In a first aspect, the present invention provides a method for fine-grained recognition of defective images of printed circuit boards, the method comprising the following steps:

[0010] S1. Acquire an original defect image and preprocess the original defect image to obtain a defect image dataset;

[0011] S2. Build a network model with a deep separation convolution module with residual connection, SE attention module and relu activation function;

[0012] S3, using the defect image dataset as input of the network model to perform training through a channel pruning method to obtain a recognition model;

[0013] S4. Using the recognition model to perform fine-grained recognition on the defective image of the printed circuit board to obtain a recognition result.

[0014] Preferably, the process of preprocessing the original defect image in step S1 includes:

[0015] Cropping the original defect image;

[0016] Performing denoising on the cropped original defect image based on a median filtering method to obtain a denoised defect image;

[0017] The denoised defect image is expanded by random flipping, random splicing, contrast and brightness changes to obtain a defect image dataset.

[0018] Preferably, in step S3, the channel pruning method defines a corresponding global pruning rate based on the binomial norm and Euclidean distance of the convolution kernel parameters as the standard for measuring the convolution parameters; wherein the number of channels pruned by the binomial norm is 10% of the total number of channels, and the number of channels pruned by the Euclidean distance is 90% of the total number of channels.

[0019] Preferably, the process of training in step S3 by using the channel pruning method includes:

[0020] The convolution kernel parameters are sorted by the size of the bi-norm or the Euclidean distance and the channels with the lower sorting are pruned, the convolution kernel parameters to be pruned are set to 0 and the gradient of the convolution kernel parameters is set to 0, so that the convolution kernel parameters are not updated.

[0021] Preferably, the calculation formulas of the second norm and the Euclidean distance are as follows: weight=weight.view(C out ,C in ×K×K); weight_norm=norm(weight,2,1); weight_distance=distance(weight,euclidean);

[0022] Among them, weight represents the convolution kernel parameter, and the structure of the convolution kernel parameter is C out ×C in ×K×K four-dimensional tensor, C out is the number of output channels, C in is the number of input channels, K is the height or width of the convolution kernel; weight_norm represents the two norms; weight_distance represents the Euclidean distance, and weight.view() represents the transformation of the four-dimensional tensor of the convolution kernel parameters into a structure (C out ,C in ×K×K), norm() represents the calculation of the two-norm of the convolution kernel parameters, weight_norn represents the result of the two-norm calculation, distance() represents the Euclidean distance calculation of the convolution kernel parameters, euclidean represents the Euclidean distance, and weight_distance represents the result of the convolution kernel parameters calculated using the Euclidean distance calculation formula.

[0023] In a second aspect, the present invention further provides a fine-grained recognition system for defective images of printed circuit boards, the recognition system comprising:

[0024] An acquisition module is used to acquire an original defect image and preprocess the original defect image to obtain a defect image dataset;

[0025] A construction module for building a network model with a deep separation convolution module with residual connection, SE attention module and relu activation function;

[0026] A training module, configured to use the defect image dataset as input of the network model to perform training through a channel pruning method to obtain a recognition model;

[0027] The recognition module is used to use the recognition model to perform fine-grained recognition on the defect image of the printed circuit board to obtain a recognition result.

[0028] Preferably, the acquisition module is further used for:

[0029] Cropping the original defect image;

[0030] Performing denoising on the cropped original defect image based on a median filtering method to obtain a denoised defect image;

[0031] The denoised defect image is expanded by random flipping, random splicing, contrast and brightness changes to obtain a defect image dataset.

[0032] Preferably, the channel pruning method defines a corresponding global pruning rate based on the binomial norm and Euclidean distance of the convolution kernel parameters as the standard for measuring the convolution parameters; wherein the number of channels pruned by the binomial norm is 10% of the total number of channels, and the number of channels pruned by the Euclidean distance is 90% of the total number of channels.

[0033] Compared with existing technologies, this paper establishes a lightweight convolutional neural network model and compresses and trains it using a channel pruning method. This method combines the bi-norm of the convolution kernel parameters and the Euclidean distance to evaluate their importance. During network model training, less important convolution parameters are set to 0. The recognition model generated by training this network model can accurately and quickly detect defects on the surface of printed circuit boards. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention will be described in detail below with reference to the accompanying drawings. The above and other aspects of the present invention will become clearer and easier to understand through the detailed description made with reference to the following drawings. In the accompanying drawings:

[0035] FIG1 is a flowchart of a method for fine-grained recognition of defective images of printed circuit boards provided by an embodiment of the present invention;

[0036] FIG2 is a flowchart of a fine-grained recognition system for defective images of printed circuit boards provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] Referring to FIG1 , the present invention provides a method for fine-grained recognition of defective images of printed circuit boards, the method comprising the following steps:

[0039] S1. Acquire an original defect image and preprocess the original defect image to obtain a defect image dataset;

[0040] In an embodiment of the present invention, a 16k industrial camera is used to capture PCB defect images, and the captured original defect images are cropped to a size of 640X640. A median filtering method is first used to denoise the cropped original defect images to obtain denoised defect images. The denoised defect images are then subjected to random flipping, random splicing, contrast and brightness changes, and other processing to expand the number of images and obtain a defect image dataset.

[0041] S2. Build a network model with a deep separation convolution module with residual connection, SE attention module and relu activation function;

[0042] In an embodiment of the present invention, by making corresponding improvements to the original yolov5s network, a deep separation convolution module RDP with residual connection is constructed to replace part of the normal convolution module CBR, the downsampling module in the yolov5s network is replaced by the deep separation convolution module DPTH, and SE attention modules are added to the output paths of the three heads respectively, and all activation functions are replaced with simple relu activation functions.

[0043] S3, using the defect image dataset as input of the network model to perform training through a channel pruning method to obtain a recognition model;

[0044] In an embodiment of the present invention, the expanded dataset is divided into training, validation, and test sets in a ratio of 7:2:1. A channel pruning method is employed, using the delta-norm and Euclidean distance of the convolution kernel parameters as measures of their importance. A corresponding global pruning rate is set, where the number of channels pruned based on the delta-norm is 10% of the total number of channels, and the remaining number of channels pruned based on the Euclidean distance. The SE attention module does not participate in pruning. During pruning training, the convolution kernel parameters are sorted by delta-norm or Euclidean distance, and the lowest-ranked channels are pruned. The parameters of the pruned convolution kernels are set to 0, and the gradients of the convolution kernel parameters are also set to 0 to ensure that the parameters of the convolution kernels are not updated in subsequent training. In each round of training, pruning training is performed first, and then the accuracy of the pruned model is verified. The model weight file with the highest verified accuracy during training is retained. The pruned network structure is reconstructed, and the model weight file with the convolution kernel parameters set to 0 is loaded. The model is then fine-tuned for a certain number of rounds to obtain a recognition model. The recognition model is fine-tuned for a certain number of rounds and exported as a model file in the format required by the ASIC chip. The model is deployed on the ASIC chip and the code containing the model post-processing is constructed.

[0045] The calculation method of the second norm and Euclidean distance is as follows: weight=weight.view(C out ,C in ×K×K); weight_norm=norm(weight,2,1); weight_distance=distance(weight,euclidean);

[0046] The convolution kernel parameter weight is a four-dimensional tensor with the structure of C out ×C in ×K×K,C out is the number of output channels, C inIs the number of input channels, K is the height or width of the convolution kernel. First convert the four-dimensional tensor into a two-dimensional tensor, then calculate the two-norm and Euclidean distance, and finally output C out Values, according to C out The values ​​are sorted by size, and the corresponding channels are cut off. weight_norm represents the two-norm; weight_distance represents the Euclidean distance, and weight.view() represents the transformation of the four-dimensional tensor of the convolution kernel parameters into a structure (C out ,C in ×K×K), norm() represents the calculation of the two-norm of the convolution kernel parameters, weight_norn represents the result of the two-norm calculation, distance() represents the Euclidean distance calculation of the convolution kernel parameters, euclidean represents the Euclidean distance, and weight_distance represents the result of the convolution kernel parameters calculated using the Euclidean distance calculation formula.

[0047] S4. Using the recognition model to perform fine-grained recognition on the defective image of the printed circuit board to obtain a recognition result.

[0048] In an embodiment of the present invention, the recognition model is deployed on an ASIC chip to maximize the computing power of the ASIC chip, and fine-grained recognition of defect images of printed circuit boards is performed to accurately and quickly detect defects on the surface of the printed circuit boards.

[0049] Referring to FIG. 2 , the present invention further provides a fine-grained recognition system for defective images of printed circuit boards, the recognition system comprising:

[0050] An acquisition module is used to acquire an original defect image and preprocess the original defect image to obtain a defect image dataset;

[0051] In an embodiment of the present invention, a 16k industrial camera is used to capture PCB defect images, and the captured original defect images are cropped to a size of 640X640. A median filtering method is first used to denoise the cropped original defect images to obtain denoised defect images. The denoised defect images are then subjected to random flipping, random splicing, contrast and brightness changes, and other processing to expand the number of images and obtain a defect image dataset.

[0052] A construction module for building a network model with a deep separation convolution module with residual connection, SE attention module and relu activation function;

[0053] In an embodiment of the present invention, by making corresponding improvements to the original yolov5s network, a deep separation convolution module RDP with residual connection is constructed to replace part of the normal convolution module CBR, the downsampling module in the yolov5s network is replaced by the deep separation convolution module DPTH, and SE attention modules are added to the output paths of the three heads respectively, and all activation functions are replaced with simple relu activation functions.

[0054] A training module, configured to use the defect image dataset as input of the network model to perform training through a channel pruning method to obtain a recognition model;

[0055] In an embodiment of the present invention, the expanded defect image dataset is divided into training, validation, and test sets in a ratio of 7:2:1. A channel pruning method is employed, using the l-norm and Euclidean distance of the convolution kernel parameters as a measure of their importance. A corresponding global pruning rate is set, where the number of channels pruned based on the l-norm is 10% of the total number of channels, and the remaining number of channels pruned based on the Euclidean distance. The SE attention module does not participate in pruning. During pruning training, the convolution kernel parameters are sorted by the l-norm or Euclidean distance, and the lowest-ranked channels are pruned. The parameters of the pruned convolution kernels are set to 0, and their gradients are also set to 0 to ensure that the parameters of these convolution kernels are not updated during subsequent training. In each round of training, pruning is performed first, followed by verification of the accuracy of the pruned model. The model weight file with the highest verified accuracy during training is retained. The pruned network structure is reconstructed, and the model weight file with the convolution kernel parameters set to 0 is loaded. The model is then fine-tuned for a certain number of rounds to obtain the recognition model. The recognition model is fine-tuned for a certain number of rounds and exported as a model file in the format required by the ASIC chip. The model is deployed on the ASIC chip and the code containing the model post-processing is constructed.

[0056] The calculation method of the second norm and Euclidean distance is as follows: weight=weight.view(C out ,C in ×K×K); weight_norm=norm(weight,2,1); weight_distance=distance(weight,euclidean);

[0057] The convolution kernel parameter weight is a four-dimensional tensor with the structure of C out ×C in ×K×K,C out is the number of output channels, C in Is the number of input channels, K is the height or width of the convolution kernel. First convert the four-dimensional tensor into a two-dimensional tensor, then calculate the two-norm and Euclidean distance, and finally output C outValues, according to C out The values ​​are sorted by size, and the corresponding channels are cut off. weight_norm represents the two norms; weight_distance represents the Euclidean distance, and weight.view() represents the transformation of the four-dimensional tensor of the convolution kernel parameters into a structure (C out ,C in ×K×K), norm() represents the calculation of the two-norm of the convolution kernel parameters, weight_norn represents the result of the two-norm calculation, distance() represents the Euclidean distance calculation of the convolution kernel parameters, euclidean represents the Euclidean distance, and weight_distance represents the result of the convolution kernel parameters calculated using the Euclidean distance calculation formula.

[0058] The recognition module is used to use the recognition model to perform fine-grained recognition on the defect image of the printed circuit board to obtain a recognition result.

[0059] In an embodiment of the present invention, the recognition model is deployed on an ASIC chip to maximize the computing power of the ASIC chip, and fine-grained recognition of defect images of printed circuit boards is performed to accurately and quickly detect defects on the surface of the printed circuit boards.

[0060] Compared with existing technologies, this paper establishes a lightweight convolutional neural network model and compresses and trains it using a channel pruning method. This method combines the bi-norm of the convolution kernel parameters and the Euclidean distance to evaluate their importance. During network model training, less important convolution parameters are set to 0. The recognition model generated by training this network model can accurately and quickly detect defects on the surface of printed circuit boards.

[0061] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0062] The embodiments of the present invention are described above in conjunction with the accompanying drawings. What is disclosed is only a preferred embodiment of the present invention. However, the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms and equivalent changes without departing from the scope of protection of the purpose of the present invention and the claims, which are all within the protection of the present invention.

Claims

1. A fine-grained recognition method for defective images of printed circuit boards, characterized in that: The identification method comprises the following steps: S1. Acquire an original defect image, and preprocess the original defect image to obtain a defect image data set; S2, build a network model with deep separation convolution module with residual connection, SE attention module and relu activation function; S3, using the defect image dataset as the input of the network model to perform training through a channel pruning method to obtain a recognition model; S4. Using the recognition model to perform fine-grained recognition on the defective image of the printed circuit board to obtain a recognition result.

2. The method for fine-grained recognition of defective images of printed circuit boards according to claim 1, characterized in that: The process of preprocessing the original defect image in step S1 includes: Cropping the original defect image; Performing denoising on the cropped original defect image based on a median filtering method to obtain a denoised defect image; The denoised defect image is expanded by random flipping, random splicing, contrast and brightness changes to obtain a defect image data set.

3. The method for fine-grained recognition of defective images of printed circuit boards according to claim 1, characterized in that: In step S3, the channel pruning method defines the corresponding global pruning rate based on the binary norm and Euclidean distance of the convolution kernel parameters as the standard for measuring the convolution parameters; wherein the number of channels pruned by the binary norm is 10% of the total number of channels, and the number of channels pruned by the Euclidean distance is 90% of the total number of channels.

4. The method for fine-grained recognition of defective images of printed circuit boards according to claim 3, characterized in that: The process of training in step S3 by channel pruning method includes: The convolution kernel parameters are sorted according to the size of the binary norm or the Euclidean distance and the channels with the lowest sorting are pruned, the convolution kernel parameters to be pruned are set to 0 and the gradient of the convolution kernel parameters is set to 0, So that the convolution kernel parameters will not be updated.

5. The method for fine-grained recognition of defective images of printed circuit boards as claimed in claim 3, characterized in that: The calculation formulas of the bi-norm and the Euclidean distance are as follows: weight = weight.view(C out ,C in ×K×K); weight_norm=norm(weight,2,1); weight_distance=distance(weight,euclidean); Among them, weight represents the convolution kernel parameter, and the structure of the convolution kernel parameter is C out ×C in ×K×K four-dimensional tensor, C out is the number of output channels, C in is the number of input channels, K is the height or width of the convolution kernel; weight_norm represents the bi-norm; weight_distance represents the Euclidean distance, and weight.view() represents the transformation of the four-dimensional tensor of the convolution kernel parameters into a structure of (C out ,C in ×K×K), norm() represents the calculation of the second norm of the convolution kernel parameters, weight_norn represents the result of the second norm calculation, distance() represents the Euclidean distance calculation of the convolution kernel parameters, euclidean represents the Euclidean distance, and weight_distance represents the result of the convolution kernel parameters calculated using the Euclidean distance calculation formula.

6. A fine-grained recognition system for defective images of printed circuit boards, characterized in that: The identification system comprises: An acquisition module is used to acquire an original defect image and preprocess the original defect image to obtain a defect image data set; A construction module for building a network model with a deep separation convolution module with residual connection, SE attention module, and relu activation function; A training module, used for training the defect image dataset as the input of the network model through a channel pruning method to obtain a recognition model; The recognition module is used to use the recognition model to perform fine-grained recognition on the defective image of the printed circuit board to obtain a recognition result.

7. The fine-grained recognition system for defective images of printed circuit boards according to claim 6, characterized in that: The acquisition module is also used for: Cropping the original defect image; The cropped original defect image is denoised based on the median filtering method to obtain Denoising defective images; The denoised defect image is expanded by random flipping, random splicing, contrast and brightness changes to obtain a defect image data set.

8. The fine-grained recognition system for defective images of printed circuit boards as claimed in claim 6, characterized in that: The channel pruning method defines a corresponding global pruning rate based on the binary norm and Euclidean distance of the convolution kernel parameters as the standard for measuring the convolution parameters; wherein the number of channels pruned by the binary norm is 10% of the total number of channels, and the number of channels pruned by the Euclidean distance is 90% of the total number of channels.

Images