Deep Learning-Based Image Defect Detection Method for Industrial Printed Circuit Boards

Abstract

The present invention discloses a method for detecting defects in industrial printed circuit board images based on deep learning, specifically, designing a small data set expansion scheme to enhance data diversity; designing a deep separable convolution reconstruction parameterization module to construct the early stage of the backbone network, designing a non-adjacent window filling block to improve the multi-head self-attention module to form a lightweight neural network Transformer architecture to construct the later stage of the backbone network, designing a small target step output improvement module to construct a target detection head; designing a defect detection model that can locate industrial printed circuit board images and designing a small target detection strategy to improve the speed and accuracy of the model's learning of small target defects; designing a label deduplication scheme and an adaptive defect screening scheme to improve the accuracy of model prediction. The present invention can improve the feature learning ability of images with small detection targets, and can effectively improve the defect detection performance of industrial printed circuit board images.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to an industrial printed circuit board image defect detection method based on deep learning. Background Art

[0002] As a key link to ensure product quality and corporate competitiveness, industrial product quality inspection is facing a transition from traditional manual visual inspection to automated and intelligent inspection. Traditional quality inspection methods rely on manual vision or simple automated equipment, which are inefficient and have unstable inspection results, making it difficult to meet the high efficiency and high precision requirements of modern production.

[0003] The development of deep learning technology, especially the breakthroughs of convolutional neural networks and lightweight neural network Transformer in the field of image recognition, has provided new solutions for the quality inspection of industrial products such as printed circuit boards. Computer vision technology, including basic technologies such as target detection, key point detection and image classification, has become the key to improving the efficiency and accuracy of industrial product quality inspection. However, in the current manufacturing environment, the complexity and diversity of products have led to increasing requirements for quality inspection, and more advanced technologies need to be further explored to improve inspection performance. In addition, the field of industrial quality inspection often faces the problem of small samples, that is, the number of real defective images is limited, resulting in insufficient recognition ability of the model when facing new types of defects, affecting the overall control of product quality.

[0004] Therefore, the present invention constructs a high-quality large-scale data set by designing a combined feature expansion and multi-style data expansion scheme, and designs an efficient and reliable defect detection model by using a deep separable convolution reconstruction parameterization module, a non-adjacent window filling block improved multi-head sub-attention module, a lightweight neural network Transformer architecture and a small target step output improvement module. Post-processing schemes such as sliding window small target detection, label deduplication and adaptive defect screening are also designed to achieve higher detection efficiency and accuracy for industrial printed circuit board images, and meet the strict requirements of modern industrial production for quality inspection. Summary of the invention

[0005] The purpose of the present invention is to provide an industrial printed circuit board image defect detection method based on deep learning, which reduces labor costs, enhances the robustness and generalization of the algorithm, greatly accelerates the detection speed of large-resolution images, and improves the accuracy of small target defect detection.

[0006] To achieve the above object, the present invention provides the following solution: a method for detecting industrial printed circuit board image defects based on deep learning, the method steps are as follows:

[0007] Step S1: manually annotating a data set; making an industrial printed circuit board image into a data set, manually annotating defects on the industrial printed circuit board images in the data set to obtain a small data set, and dividing the obtained small data set into a training set, a validation set, and a test set;

[0008] Step S2: Design a scheme to expand the small data set obtained in step S1; this scheme includes two schemes: designing a combination feature expansion scheme and designing a multi-style data expansion scheme, to obtain an expanded data set;

[0009] Step S3: construct the backbone network and target detection head; it is divided into the early stage of designing the deep separable convolution reconstruction parameterization module to construct the backbone network, the late stage of designing the non-adjacent window filling block to improve the multi-head self-attention module to form a lightweight neural network Transformer architecture to construct the backbone network, and the small target step size output improvement module to build the target detection head;

[0010] Step S4: designing a defect detection model capable of identifying and locating industrial printed circuit board images; selecting the backbone network and target detection head in step S3, and combining the characteristics of industrial printed circuit board defect images to design a defect detection model for industrial printed circuit board images;

[0011] Step S5: training a defect detection model for industrial printed circuit board images and performing defect detection on industrial printed circuit board images; using the training set in the expanded data set obtained in step S2 as the input for designing a defect detection model for industrial printed circuit board images in step S4, training and learning the defect detection model for industrial printed circuit board images, verifying the validation set in the expanded data set in step S2 through the trained defect detection model for industrial printed circuit board images, and performing defect identification and location on the test set in step S1, thereby proving the detection performance and generalization of the defect detection model for industrial printed circuit board images;

[0012] Step S6: Design a sliding window small target detection strategy; in view of the problem of large resolution of the test set image in step S1, a sliding window small target strategy is designed, the image is cropped in a sliding window manner, the cropped image is input into the defect detection model of the industrial printed circuit board image trained in step S5 for detection, and the detected defect position information is restored to the corresponding position in the image before cropping;

[0013] Step S7: Design a label deduplication scheme; in order to solve the problem of repeated defect prediction in the overlapping area of ​​the sliding window in step S6, design a label deduplication scheme to filter out the labels of repeated prediction;

[0014] Step S8: Design an adaptive defect screening scheme; design an adaptive defect screening scheme according to the specific circumstances of the size and style of the target defect to maximize the defect prediction accuracy.

[0015] Furthermore, in step S1, the data set is manually labeled, and the specific steps are as follows:

[0016] Step S11: Manually mark defects on the original industrial printed circuit board image; defect categories include mouse bite, open circuit, short circuit, disconnection and miscellaneous copper;

[0017] Step S12: According to different defect categories, target defects are drawn on the original industrial printed circuit board image, and the target category and horizontal rectangular frame coordinates of the target position are marked and stored in the corresponding marking file; the marking file is in txt format, and each line of the marking file represents a marked target. The marking file has 5 values, which are , , , , ,in Indicates category, and Indicates the center coordinates of the annotation box. and Respectively represent the relative width and height of the annotation box;

[0018] Step S13: Set is an integer greater than 3, For 2 The resolution of industrial printed circuit board images is ,and and Both multiples of; all industrial printed circuit board images in the small dataset are sorted by The ratio is divided into training set, validation set and test set. There is no overlap among the training set, validation set and test set. The small data set is used as the first data set D1.

[0019] Furthermore, in step S2, a scheme for expanding the small data set obtained in step S1 is designed; it is divided into two schemes: designing combined feature expansion and designing multi-style data expansion. The specific steps are as follows:

[0020] Step S21: Fixed size cropping; the training set and validation set of the first dataset D1 manually annotated in step S1 are cropped with a resolution of Industrial printed circuit board images are cropped by fixed specifications according to Specifications, evenly cut to Size The industrial printed circuit board images are obtained to obtain the second data set D2;

[0021] Step S22: Fixed specification secondary cropping; the resolution size of the second data set D2 is The industrial printed circuit board image is cut into two pieces according to the fixed specifications. Specifications, evenly cut into 4 pieces of size The industrial printed circuit board images are obtained to obtain the third data set D3;

[0022] Step S23: Randomly shuffle and splice; all the resolutions of the third data set D3 are The industrial printed circuit board images of are classified and shuffled according to the original corresponding positions of the images in the second data set D2, and then randomly spliced ​​into blocks of size The industrial printed circuit board images are obtained to obtain the fourth data set D4;

[0023] Step S24: manual fine-tuning; deleting the industrial printed circuit board images whose defect positions are on the boundary line and whose defect features are incomplete due to cropping and splicing, and obtaining the fifth data set D5 from the remaining industrial printed circuit board images;

[0024] Step S25: The fifth data set D5 and the second data set D2 are aggregated to obtain a sixth data set D6 after combined feature expansion, and the combined feature expansion scheme is completed;

[0025] Step S26: Based on the training set and validation set of the first data set D1, the resolution size is The industrial printed circuit board image is randomly cropped and cropped to a resolution of Industrial printed circuit board images are obtained to obtain the seventh data set D7;

[0026] Step S27: Aggregate the sixth data set D6 and the seventh data set D7 to obtain an eighth data set D8;

[0027] Step S28: performing a multi-style data expansion operation on the eighth data set D8;

[0028] The step S28 of performing the diversified data expansion operation is specifically as follows:

[0029] Random flipping up and down, left and right, or up and down and left and right; distortion processing to manually re-mark the target defect position; random addition of color blocks; inversion processing; adding Gaussian blur; adding salt and pepper noise; channel conversion; overlapping; random scaling and splicing;

[0030] Finally, the ninth dataset D9 is obtained after multi-style data expansion and is used as the final dataset D.

[0031] Further, in step S3, a backbone network and a target detection head are constructed; the early stage of designing a deep separable convolutional reconstruction parameterization module to construct a backbone network, the late stage of designing a non-adjacent window filling block to improve a multi-head self-attention module to form a lightweight neural network Transformer architecture to construct a backbone network, and the small target step output improvement module to construct a target detection head; wherein, the deep separable convolutional reconstruction parameterization module is designed, and the specific steps are as follows:

[0032] Step S31: Fusing the batch normalization layer and the convolution layer;

[0033] Each branch containing the depthwise separable convolution and batch normalization layer is reconstructed, and the batch normalization layer and the depthwise separable convolution are fused into a new depthwise separable convolution layer, as shown in formula (1) and formula (2):

[0034] (1);

[0035] (2);

[0036] In the formula, represents the weight of the new convolution matrix, represents the weight of the original convolution, , and are the number of output channels, the number of input channels and the size of the convolution kernel, respectively. , , and are the scaling parameter, offset parameter, standard deviation of the current batch, and mean of the current batch in the batch normalization layer, respectively. represents the new bias, Indicates the original bias;

[0037] Step S32: merging branches;

[0038] Will be The two branches constructed by the depthwise separable convolutional layer and the batch normalization layer are merged with the branch composed of batch normalization. The corresponding elements of all relevant weight matrices and biases need to be added, as shown in formulas (3) and (4):

[0039] (3);

[0040] (4);

[0041] In the formula, and is the reconstructed weight matrix and bias, is the total number of branches, For the The weights of the new convolution matrix, For the A new bias;

[0042] Among them, the non-adjacent window filling block is designed to improve the multi-head self-attention module. The specific steps are as follows:

[0043] Step S33: Pass The size is The window is divided completely and non-overlapping into resolution sizes up to , width , the number of channels is Input feature map ;

[0044] Step S34: Select the structures with the same relative position in each window as The feature vector is composed of Filling block groups, a total of Group position distribution feature map global padding block group ;

[0045] Step S35: Group the global filling blocks of the position distribution feature map Input the neural network Transformer in sequence to calculate multi-head self-attention; calculate global correlation while optimizing computational complexity;

[0046] Among them, the lightweight neural network Transformer architecture is designed. The specific steps are as follows:

[0047] Step S36: constructing a local representation module;

[0048] use The convolutional layer extracts local features, normalizes them through the batch normalization layer and passes through the activation function, and then passes through The convolutional layer expands the feature map to an appropriate number of channels to complete the local representation;

[0049] Step S37: performing global characterization;

[0050] The output of step S36 is used to improve the multi-head self-attention module, the neural network Transformer and The convolution layer completes the extraction of global features;

[0051] Step S38: feature fusion;

[0052] The output of step S37 is densely connected with the input of step S36, and then After the convolution layer, batch normalization layer and activation function, feature fusion is completed to prevent gradient disappearance;

[0053] Among them, a small target step size output improvement module is designed, including the network layer improvement of the backbone network and the output layer detection head improvement. The specific improvements are as follows:

[0054] Step S39: the network layer connected to the backbone network is improved by transforming the network layer originally connected to the backbone network with a step length of 32, 16, 8 into a network layer connected to the backbone network with a step length of 32, 16, 8, 4;

[0055] Step S310: The output layer detection head is improved by replacing the output layer from large-scale detection head P5, medium-scale detection head P4, and small-scale detection head P3 to medium-scale detection head P4, small-scale detection head P3, and ultra-small-scale detection head P2.

[0056] Furthermore, in step S4, a defect detection model capable of identifying and locating industrial printed circuit board images is designed, and the specific steps are as follows:

[0057] Step S41, the defect detection model of the industrial printed circuit board image is composed of a backbone network and a target detection head;

[0058] Step S42, the first to fourth layers of the backbone network are constructed by a depthwise separable convolutional reconstruction parameterized module, and the fifth to seventh layers are constructed by a lightweight neural network Transformer architecture, and the input of each layer comes from the output result of the previous layer;

[0059] In step S43, the target detection head is constructed through a small target step output improvement module to realize small target defect detection; the defect detection model of the industrial printed circuit board image realizes the target defect judgment in the industrial printed circuit board image.

[0060] Furthermore, in step S5, a defect detection model for industrial printed circuit board images is trained and defect detection for industrial printed circuit board images is performed. The specific steps are as follows:

[0061] Step S51, inputting the training set of the final data set D into the defect detection model of the industrial printed circuit board image in step S4 for training. During the training process, the defect detection capability of the defect detection model of the industrial printed circuit board image is optimized by using the back propagation algorithm through the category loss, position loss and confidence loss until fitting;

[0062] Step S52: After the fitting is completed, the final data set D is used to validate the data set. The defect detection performance of the defect detection model for industrial printed circuit board images is tested;

[0063] Step S53, at the same time, using the test set of the first data set D1 with a size of The defect detection performance and generalization of the defect detection model for industrial printed circuit board images are tested.

[0064] Furthermore, in step S6, a sliding window small target detection strategy is designed, and the specific steps are as follows:

[0065] Step S61: For the first data set D1, the resolution size in the test set is The industrial printed circuit board image is cropped by sliding window; the coordinates of the upper left corner of the industrial printed circuit board image are The position of the clipping window starts sliding, and the resolution size of the clipping window is set to , the sliding step size is set to ,in , , and The best value for , or , , that is, each two windows have a size of overlapping areas;

[0066] Step S62: the sliding window slides to the right or bottom of the original image in the test set of the first data set D1. If the croppable area is smaller than the size of the sliding window, no cropping is performed.

[0067] Step S63: Crop by sliding window to obtain the common Size Industrial printed circuit board image; the size of the cropped resolution is The industrial printed circuit board image is input into the defect detection model of the industrial printed circuit board image trained in step S5 for detection, and the predicted target defect position information is restored back to the corresponding position of the original image in the test set of the first data set D1.

[0068] Furthermore, in step S7, a tag deduplication scheme is designed, and the specific steps are as follows:

[0069] Step S71: Overlapping frame determination;

[0070] Assume that the starting position coordinates of the two label frames A and B are: , ; The end position coordinates of the two label frames A and B are: , ;

[0071] When the end position of label frame A and label frame B and The minimum value is greater than the starting position The maximum value, and the end position of label box A and label box B is The minimum value is greater than the starting position and The maximum value of , label frame A and label frame B overlap, otherwise label frame A and label frame B do not overlap;

[0072] As shown in formula (5) to formula (8):

[0073] (5);

[0074] (6);

[0075] (7);

[0076] (8);

[0077] In the formula, The larger horizontal coordinate of the starting position of label box A and label box B, The smaller horizontal coordinate of the end position of label box A and label box B. The larger vertical coordinate of the starting position of label box A and label box B, The smaller vertical coordinate of the end position of label box A and label box B when the overlap condition is met and When , label frame A and label frame B overlap, and step S72 and step S73 are executed; otherwise, label frame A and label frame B do not overlap;

[0078] Step S72: merging overlapping positions;

[0079] When it is determined that the label frame A and the label frame B overlap, the positions of the label frame A and the label frame B are merged, as shown in calculation formulas (9) to (12):

[0080] (9);

[0081] (10);

[0082] (11);

[0083] (12);

[0084] In the formula, Indicates the starting position of the merge box coordinate, Indicates the starting position of the merge box coordinate, Indicates the end position of the merge box coordinate, Indicates the end position of the merge box coordinate;

[0085] Step S73: Overlapping label category selection;

[0086] When it is determined that label box A and label box B overlap, a category is selected based on the highest confidence of label box A and label box B.

[0087] Furthermore, in step S8, an adaptive defect screening scheme is designed, and the specific steps are as follows:

[0088] Step S81: initial confidence screening;

[0089] According to the overall prediction of the defect detection model of industrial printed circuit board images, an initial confidence level is set , the initial confidence The value range is , only keep the prediction confidence greater than the initial confidence Defect labels of the system are used to filter out most of the incorrectly predicted defect labels.

[0090] Step S82: small target defect screening;

[0091] The smaller the area of ​​the target defect is, the worse the prediction ability of the defect detection model of the industrial printed circuit board image is for the target defect, and the lower the confidence is; a classification discussion is conducted;

[0092] Assume the starting position of the predicted defect label box is , the terminal position of the predicted defect label box is , the area of ​​the graphic defect label box is calculated as shown in formula (13):

[0093] (13);

[0094] In the formula, The defect label area is predicted for the calculated image. The horizontal coordinates of the starting and ending positions of the predicted defect label box are , , the ordinates of the starting and ending positions of the predicted defect label box are , ;

[0095] Set an area limit value R for normal size target defects and small target defects. When the area of ​​the predicted defect is smaller than the area limit value R, it satisfies When the target defect is larger than the area limit value, the target defect is judged to be a small target defect and the target defect is retained; Defects, execute step S83;

[0096] Step S83: long impurity copper defect screening;

[0097] Step S831: Observe the characteristics of the defect type of impurity copper, which is divided into long impurity copper and short impurity copper; the defect area of ​​short impurity copper is smaller than the area limit value set in step S82 , short impurity copper defects are judged as small target defects;

[0098] Step S832: For a long copper defect, the area is greater than the area threshold value set in step S82 , the long impurity copper defect is judged as a normal size target defect. The long impurity copper defect category is a relatively rare feature among defect features, and the prediction confidence is lower than other normal size target defects, so it is processed separately;

[0099] Step S833: When the predicted target defect area is greater than the area threshold value , and when the defect category is miscellaneous copper, it is retained; for the other predicted defects, step S84 is executed;

[0100] Step S84: Screening of other defects;

[0101] For the predicted defect area greater than the area limit value , and the defect category is not a copper defect, the confidence level of the defect detection model prediction of the industrial printed circuit board image is higher; affected by the background noise in the industrial printed circuit board image, a small number of incorrect prediction labels will appear in the prediction results;

[0102] Set a higher confidence level , higher confidence The value range is , and higher confidence The value is greater than the initial confidence ; Keep predictions with confidence greater than higher confidence The defect labels with confidence less than The incorrectly predicted labels are eliminated to maximize the correct prediction rate of the defect detection model for industrial printed circuit board images.

[0103] The beneficial effects of the present invention are:

[0104] (1) The present invention designs a combined feature expansion scheme to solve the problem that manual labeling takes too much time and cost, and obtains a large number of printed circuit board images.

[0105] (2) The present invention designs a multi-style data expansion scheme to solve the problem of background diversity of printed circuit board images, enhance data diversity, and improve the robustness and generalization of the model.

[0106] (3) The present invention designs a depthwise separable convolutional reconstruction parameterization module to achieve dual optimization of the model's accuracy and speed.

[0107] (4) The present invention designs a non-adjacent window filling block to improve the multi-head self-attention module, solve the problem of redundant characteristic information contained in adjacent pixels of the feature matrix, complete the global correlation information calculation and optimize the computational complexity.

[0108] (5) The present invention designs a lightweight neural network Transformer architecture that takes into account both local and global representations.

[0109] (6) The present invention designs a small target step size output improvement module to solve the problem of information loss of small-size defects in deep networks.

[0110] (7) The present invention designs a defect detection model that can identify and locate industrial printed circuit board images, providing key technologies for improving the efficiency and accuracy of industrial product quality inspection and promoting digital innovation in the manufacturing industry.

[0111] (8) The present invention designs a sliding window small target detection strategy to solve the problem that the input image resolution is large and the model cannot quickly locate defect information.

[0112] (9) The present invention designs a label deduplication solution to solve the problem of multiple predictions of defects in overlapping areas of sliding windows.

[0113] (10) The present invention designs an adaptive defect screening scheme to improve the accuracy of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 The present invention provides a flowchart of the method for detecting image defects of industrial printed circuit boards. DETAILED DESCRIPTION

[0115] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0116] The present invention relates to the fields of deep learning, computer vision, target detection framework, reconstruction parameterization, lightweight neural network Transformer and other related technical fields, especially to the design and training of deep learning neural networks for target detection in deep learning and computer vision. The purpose of the present invention is to provide an industrial printed circuit board image defect detection method based on deep learning to solve the problems of low efficiency and unstable detection results in the quality inspection process of industrial printed circuit boards.

[0117] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0118] The present invention adopts Figure 1 The flowchart shown is a method for industrial printed circuit board image defect detection based on deep learning. The method steps are as follows:

[0119] Step S1: manually annotating a data set; making an industrial printed circuit board image into a data set, manually annotating defects on the industrial printed circuit board images in the data set to obtain a small data set, and dividing the obtained small data set into a training set, a validation set, and a test set;

[0120] Step S2: Design a scheme to expand the small data set obtained in step S1; this scheme includes two schemes: designing a combination feature expansion scheme and designing a multi-style data expansion scheme, to obtain an expanded data set;

[0121] Step S3: construct the backbone network and target detection head; it is divided into the early stage of designing the deep separable convolution reconstruction parameterization module to construct the backbone network, the late stage of designing the non-adjacent window filling block to improve the multi-head self-attention module to form a lightweight neural network Transformer architecture to construct the backbone network, and the small target step size output improvement module to build the target detection head;

[0122] Step S4: designing a defect detection model capable of identifying and locating industrial printed circuit board images; selecting the backbone network and target detection head in step S3, and combining the characteristics of industrial printed circuit board defect images to design a defect detection model for industrial printed circuit board images;

[0123] Step S5: training a defect detection model for industrial printed circuit board images and performing defect detection on industrial printed circuit board images; using the training set in the expanded data set obtained in step S2 as the input for designing a defect detection model for industrial printed circuit board images in step S4, training and learning the defect detection model for industrial printed circuit board images, verifying the validation set in the expanded data set in step S2 through the trained defect detection model for industrial printed circuit board images, and performing defect identification and location on the test set in step S1, thereby proving the detection performance and generalization of the defect detection model for industrial printed circuit board images;

[0124] Step S6: Design a sliding window small target detection strategy; in view of the problem of large resolution of the test set image in step S1, a sliding window small target strategy is designed, the image is cropped in a sliding window manner, the cropped image is input into the defect detection model of the industrial printed circuit board image trained in step S5 for detection, and the detected defect position information is restored to the corresponding position in the image before cropping;

[0125] Step S7: Design a label deduplication scheme; in order to solve the problem of repeated defect prediction in the overlapping area of ​​the sliding window in step S6, design a label deduplication scheme to filter out the labels of repeated prediction;

[0126] Step S8: Design an adaptive defect screening scheme; design an adaptive defect screening scheme according to the specific circumstances of the size and style of the target defect to maximize the defect prediction accuracy.

[0127] Among the above 8 main steps, S1 and S2 are the basis for the implementation of the invention, S3 and S4 are the core, S5 is the demonstration of the implementation of the invention, and S6 to S8 are the enhancement of the implementation of the invention; the specific implementation of the above steps is as follows:

[0128] Manually annotate the dataset in step S1;

[0129] Since the obtained printed circuit board images are limited, it is not conducive to the training of the defect detection model. Therefore, the present invention manually annotates the data set. The specific steps are as follows:

[0130] Step S11: Manually mark defects on the normal industrial printed circuit board image without defects; the defect categories include mouse bite, open circuit, short circuit, disconnection and miscellaneous copper;

[0131] Step S12: According to different defect categories, target defects are drawn for the original normal industrial printed circuit board image without defects, and the target category and horizontal rectangular frame coordinates of the target position are marked and stored in the corresponding marking file; the marking file is in txt format, and each line of the marking file represents a marked target, with 5 values ​​respectively , , , , ,in Indicates category, and Indicates the center coordinates of the annotation box. and Respectively represent the relative width and height of the annotation box;

[0132] Step S13: Set is an integer greater than 3, For 2 The resolution of industrial printed circuit board images is ,and and Both multiples of; all industrial printed circuit board images in the small dataset are sorted by The ratio of is divided into training set, validation set and test set. There is no overlap among the three sets, which is taken as the first data set D1.

[0133] Furthermore, it is assumed that the resolution of all manually annotated images is ,Right now and They are 3072 and 2048 respectively, both of which are multiples of 1024, that is, Set to 1024, which is 2 to the 10th power; a total of 5,000 defective printed circuit board images were manually annotated according to The dataset is divided into training set, validation set and test set in the ratio of , where 3500 images are divided into training set, 500 are divided into validation set and 1000 are divided into test set.

[0134] Furthermore, in step S2, a scheme for expanding the small data set obtained in step S1 is designed; since the accuracy of defect detection algorithm in identifying and locating defects in printed circuit board images is affected by the quality and quantity of the data set, and the quality and quantity of the data set are constrained by time cost and labor cost, the present invention designs a combined feature expansion scheme to reduce labor cost while obtaining a large number of data sets; the specific steps of the combined feature expansion scheme are as follows:

[0135] Step S21: Fixed size cropping: the training set and validation set of the first dataset D1 manually annotated in step S1 are of size (Right now ) images, cropped by fixed specifications, according to (Right now ) specifications, evenly cut into Zhang (i.e. Size: (Right now ) images, and obtain the second data set D2;

[0136] Step S21: Fixed size cropping: The training set D1 of the first manually labeled data set in step S1 is

[0137] Step S22: Fixed size secondary cropping: the second data set D2 is (Right now ) images, through a fixed specification secondary cropping, according to Specifications, evenly cut into 4 pieces of size (Right now ) to obtain the third data set D3;

[0138] Step S23: Randomly shuffle and splice; all the data sets D3 with a size of (Right now ) are classified and shuffled according to the original corresponding positions of the images in the second dataset D2, and then randomly spliced ​​into (Right now ) images, and obtain the fourth data set D4;

[0139] Step S24: manual fine-tuning; in the case where the manually marked defect positions in the industrial printed circuit board images are on the boundary line and the defect features are incomplete due to cropping and splicing, in order not to affect the model's learning of the defect features, such industrial printed circuit board images are deleted, and the remaining industrial printed circuit board images are used to obtain the fifth data set D5;

[0140] Step S25: The fifth data set D5 and the second data set D2 are aggregated to obtain a sixth data set D6 after combined feature expansion, and the combined feature expansion scheme is completed;

[0141] Furthermore, since the background of printed circuit board images is diverse in practical application fields, the defect detection algorithm needs to be trained with a variety of data sets so that it can adapt to more printed circuit boards with backgrounds inconsistent with the training data sets, have high-precision recognition and positioning performance, and improve the robustness and generalization of the model; therefore, the present invention designs a multi-style data expansion scheme, and the specific steps are as follows:

[0142] Step S26: Based on the training set and validation set of the first data set D1, the size is (Right now ) images, perform random sliding cropping, and crop them into a size of (Right now ) images, and obtain the seventh data set D7;

[0143] Step S27: Aggregate the sixth data set D6 and the seventh data set D7 to obtain an eighth data set D8;

[0144] Step S28: performing a multi-style data expansion operation on the eighth data set D8;

[0145] The multi-style data expansion operation in step S28 is specifically: (1) random flipping up and down, left and right, or up and down and left and right; (2) distortion processing of manually re-marking the target defect position; (3) random addition of color blocks; (4) inversion processing; (5) addition of Gaussian blur; (6) addition of salt and pepper noise; (7) channel conversion; (8) overlap; (9) random scaling and splicing;

[0146] Finally, the ninth dataset D9 is obtained after multi-style data expansion and is used as the final dataset D.

[0147] After the data construction of step S1 and step S2, the specific information of the obtained data set is shown in Table 1.

[0148] Table 1 Dataset information

[0149]

[0150] Further, in step S3, a backbone network and a target detection head are constructed; the process is divided into the early stage of designing a deep separable convolutional reconstruction parameterization module to construct a backbone network, the late stage of designing a non-adjacent window filling block to improve a multi-head self-attention module to form a lightweight neural network Transformer architecture to construct a backbone network, and the small target step size output improvement module to construct a target detection head; wherein, the deep separable convolutional reconstruction parameterization module is designed, and the specific steps are as follows:

[0151] Step S31: Fusing the batch normalization layer and the convolution layer;

[0152] Each branch containing the depthwise separable convolution and batch normalization layer is reconstructed, and the batch normalization layer and the depthwise separable convolution are fused into a new depthwise separable convolution layer, as shown in formula (1) and formula (2):

[0153] (1);

[0154] (2);

[0155] In the formula, represents the weight of the new convolution matrix, represents the weight of the original convolution, , and are the number of output channels, the number of input channels and the size of the convolution kernel, respectively. , , and are the scaling parameter, offset parameter, standard deviation of the current batch, and mean of the current batch in the batch normalization layer, respectively. represents the new bias, Indicates the original bias;

[0156] Step S32: merging branches;

[0157] Will be The two branches constructed by the depthwise separable convolutional layer and the batch normalization layer are merged with the branch composed of batch normalization. The corresponding elements of all relevant weight matrices and biases need to be added, as shown in formulas (3) and (4):

[0158] (3);

[0159] (4);

[0160] In the formula, and is the reconstructed weight matrix and bias, is the total number of branches, For the The weights of the new convolution matrix, For the A new bias;

[0161] Furthermore, due to the redundant feature information contained in adjacent pixels of the feature matrix and the high computational cost of multi-head self-attention, the present invention designs a non-adjacent window filling block to improve the multi-head self-attention module to deal with the problem of too long filling block group input into the lightweight neural network Transformer, thereby realizing the selection of filling block groups that retain global information.

[0162] Among them, the non-adjacent window filling block is designed to improve the multi-head self-attention module. The specific steps are as follows:

[0163] Step S33: Pass The size is The window is divided completely and non-overlapping into resolution sizes up to , width , the number of channels is Input feature map ;

[0164] Step S34: Select the structures with the same relative position in each window as The feature vector is composed of Filling block groups, a total of Group position distribution feature map global padding block group ;

[0165] Step S35: Group the global filling blocks of the position distribution feature map Input the neural network Transformer in sequence to calculate multi-head self-attention; calculate global correlation while optimizing computational complexity;

[0166] Since the defect extraction of printed circuit board images needs to be based on continuous line extraction across a large area of ​​the image, and local representation is required as a supplement while improving the multi-head self-attention module through non-adjacent window filling blocks for global representation, the present invention designs a lightweight neural network Transformer architecture.

[0167] Among them, the lightweight neural network Transformer architecture is designed. The specific steps are as follows:

[0168] Step S36: constructing a local representation module;

[0169] use The convolutional layer extracts local features, normalizes them through the batch normalization layer and passes through the activation function, and then passes through The convolutional layer expands the feature map to an appropriate number of channels to complete the local representation;

[0170] Step S37: performing global characterization;

[0171] The output of step S36 is used to improve the multi-head self-attention module, the neural network Transformer and The convolution layer completes the extraction of global features;

[0172] Step S38: feature fusion;

[0173] The output of step S37 is densely connected with the input of step S36, and then After the convolution layer, batch normalization layer and activation function, feature fusion is completed to prevent gradient disappearance;

[0174] Among them, a small target step size output improvement module is designed, including the network layer improvement of the backbone network and the output layer detection head improvement. The specific improvements are as follows:

[0175] Step S39: the network layer connected to the backbone network is improved by transforming the network layer originally connected to the backbone network with a step length of 32, 16, 8 into a network layer connected to the backbone network with a step length of 32, 16, 8, 4;

[0176] Step S310: The output layer detection head is improved by replacing the output layer from large-scale detection head P5, medium-scale detection head P4, and small-scale detection head P3 to medium-scale detection head P4, small-scale detection head P3, and ultra-small-scale detection head P2.

[0177] Furthermore, in step S4, a defect detection model capable of identifying and locating industrial printed circuit board images is designed, and the specific steps are as follows:

[0178] Step S41, the defect detection model of the industrial printed circuit board image is composed of a backbone network and a target detection head;

[0179] Step S42, the first to fourth layers of the backbone network are constructed by a depthwise separable convolutional reconstruction parameterized module, and the fifth to seventh layers are constructed by a lightweight neural network Transformer architecture, and the input of each layer comes from the output result of the previous layer;

[0180] In step S43, the target detection head is constructed through a small target step output improvement module to realize small target defect detection; the defect detection model of the industrial printed circuit board image realizes the target defect judgment in the industrial printed circuit board image.

[0181] In step S5, the defect detection model of the industrial printed circuit board image is trained and the defect detection of the industrial printed circuit board image is performed. The specific steps are as follows:

[0182] Step S51, inputting the training set of the final data set D into the defect detection model of the industrial printed circuit board image in step S4 for training. During the training process, the defect detection capability of the defect detection model of the industrial printed circuit board image is optimized by using the back propagation algorithm through the category loss, position loss and confidence loss until fitting;

[0183] Step S52: After the fitting is completed, the final data set D is used to validate the data set. The defect detection performance of the defect detection model for industrial printed circuit board images is tested;

[0184] Step S53, at the same time, using the test set of the first data set D1 with a size of The defect detection performance and generalization of the defect detection model for industrial printed circuit board images are tested.

[0185] Furthermore, in step S6, a sliding window small target detection strategy is designed, and the specific steps are as follows:

[0186] Since the image resolution in the test set of the first data set D1 is too large, if the printed circuit board image defect detection model trained in step S5 directly predicts the large-resolution image, it cannot quickly and accurately capture smaller defect information in the image containing too much noise and redundant features. If the large-resolution image is divided into several small-resolution images for prediction, it can not only improve the accuracy of the model's prediction of small target defects, but also speed up the prediction. In addition, if we use a non-overlapping conventional cropping method to crop the image, the target defects on the boundary may be missed or incompletely detected. Therefore, the present invention designs a sliding window small target detection strategy, and the specific steps are as follows:

[0187] Step S61: For the first data set D1, the resolution size in the test set is The industrial printed circuit board image is cropped by sliding window; the coordinates of the upper left corner of the industrial printed circuit board image are The position of the clipping window starts sliding, and the resolution size of the clipping window is set to , the sliding step size is set to ,in , , and The best value for , or , , that is, each two windows have a size of overlapping areas;

[0188] Step S62: the sliding window slides to the right or bottom of the original image in the test set of the first data set D1. If the croppable area is smaller than the size of the sliding window, no cropping is performed.

[0189] Step S63: Crop by sliding window to obtain the common Size Industrial printed circuit board image; the size of the cropped resolution is The industrial printed circuit board image is input into the defect detection model of the industrial printed circuit board image trained in step S5 for detection, and the predicted target defect position information is restored back to the corresponding position of the original image in the test set of the first data set D1.

[0190] Furthermore, in step S7, a label deduplication scheme is designed. Due to the sliding window strategy of S6, the model predicts defects in the overlapping area of ​​the window multiple times, and there are many repeated target defect information or incomplete defect detection information on the cropping boundary line in the prediction results. Therefore, the present invention designs a label deduplication scheme to filter redundant labels and merge labels with incomplete defect detection to improve the accuracy of model prediction. The specific steps of the label deduplication scheme are as follows:

[0191] Step S71: Overlapping frame determination;

[0192] Assume that the starting position coordinates of the two label frames A and B are: , ; The end position coordinates of the two label frames A and B are: , ;

[0193] When the end position of label frame A and label frame B and The minimum value is greater than the starting position The maximum value, and the end position of label box A and label box B is The minimum value is greater than the starting position and The maximum value of , label frame A and label frame B overlap, otherwise label frame A and label frame B do not overlap;

[0194] As shown in formula (5) to formula (8):

[0195] (5);

[0196] (6);

[0197] (7);

[0198] (8);

[0199] In the formula, The larger horizontal coordinate of the starting position of label box A and label box B, The smaller horizontal coordinate of the end position of label box A and label box B. The larger vertical coordinate of the starting position of label box A and label box B, The smaller vertical coordinate of the end position of label box A and label box B when the overlap condition is met and When , label frame A and label frame B overlap, and step S72 and step S73 are executed; otherwise, label frame A and label frame B do not overlap;

[0200] Step S72: merging overlapping positions;

[0201] When it is determined that the label frame A and the label frame B overlap, the positions of the label frame A and the label frame B are merged, as shown in calculation formulas (9) to (12):

[0202] (9);

[0203] (10);

[0204] (11);

[0205] (12);

[0206] In the formula, Indicates the starting position of the merge box coordinate, Indicates the starting position of the merge box coordinate, Indicates the end position of the merge box coordinate, Indicates the end position of the merge box coordinate;

[0207] Step S73: Overlapping label category selection;

[0208] When it is determined that label box A and label box B overlap, a category is selected based on the highest confidence of label box A and label box B.

[0209] Furthermore, in step S8, an adaptive defect screening scheme is designed, and the specific steps are as follows:

[0210] Step S81: initial confidence screening;

[0211] According to the overall prediction of the defect detection model of industrial printed circuit board images, an initial confidence level is set (i.e. greater than 0.35), the initial confidence The value range is , only keep the prediction confidence greater than the initial confidence Defect labels of the system are used to filter out most of the incorrectly predicted defect labels.

[0212] Step S82: small target defect screening;

[0213] The smaller the area of ​​the target defect is, the worse the prediction ability of the defect detection model of the industrial printed circuit board image is for the target defect, and the lower the confidence is; a classification discussion is conducted;

[0214] Assume the starting position of the predicted defect label box is , the terminal position of the predicted defect label box is , the area of ​​the graphic defect label box is calculated as shown in formula (13):

[0215] (13);

[0216] In the formula, The defect label area is predicted for the calculated image. The horizontal coordinates of the starting and ending positions of the predicted defect label box are , , the ordinates of the starting and ending positions of the predicted defect label box are , ;

[0217] Set a boundary value R between the area of ​​normal size target defects and small target defects. Here, according to the actual experimental data, Set to 300; when the predicted defect area is less than the area limit value R, it satisfies (i.e. satisfy When the target defect is larger than the area limit value, the target defect is determined to be a small target defect and is retained; Defects, execute step S83;

[0218] Step S83: long impurity copper defect screening;

[0219] Step S831: Observe the characteristics of the defect type of impurity copper, which is divided into long impurity copper and short impurity copper; the defect area of ​​short impurity copper is smaller than the area limit value set in step S82 , short impurity copper defects are judged as small target defects;

[0220] Step S832: For a long copper defect, the area is greater than the area threshold value set in step S82 , the long impurity copper defect is judged as a normal size target defect. The long impurity copper defect category is a relatively rare feature among defect features, and the prediction confidence is lower than other normal size target defects, so it is processed separately;

[0221] Step S833: When the predicted target defect area is greater than the area threshold value , and when the defect category is miscellaneous copper, it is retained; for the other predicted defects, step S84 is executed;

[0222] Step S84: Screening of other defects;

[0223] For the predicted defect area greater than the area limit value , and the defect category is not a copper defect, the confidence level of the defect detection model prediction of the industrial printed circuit board image is higher; affected by the background noise in the industrial printed circuit board image, a small number of incorrect prediction labels will appear in the prediction results;

[0224] Set a higher confidence level (i.e. greater than 0.45), a higher confidence level The value range is , and higher confidence The value is greater than the initial confidence ; Keep predictions with confidence greater than higher confidence The defect labels with confidence less than The incorrectly predicted labels are eliminated to maximize the correct prediction rate of the defect detection model for industrial printed circuit board images.

[0225] The present invention uses the research results of cutting-edge computer vision technology to design an industrial printed circuit board image defect detection method based on deep learning to detect defects in printed circuit board images; by manually annotating data sets and designing combined feature expansion schemes and multi-style data expansion schemes, a large data set is constructed to support the training of defect detection models; at the same time, a deep separable convolution reconstruction parameterization module, a non-adjacent window filling block to improve the multi-head self-attention to build a lightweight neural network Transformer architecture and a small target step output improvement module are designed to generate an excellent industrial printed circuit board defect detection model; in addition, post-processing schemes such as sliding window small target detection, label deduplication and adaptive defect screening are designed to improve the speed and accuracy of model detection. Through the above set of processes, a unified framework for industrial printed circuit board image defect detection is realized.

[0226] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0227] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for detecting industrial printed circuit board images based on deep learning, characterized by: Method steps as follows: Step S1: manually annotating a data set; making an industrial printed circuit board image into a data set, manually annotating defects on the industrial printed circuit board images in the data set to obtain a small data set, and dividing the obtained small data set into a training set, a validation set, and a test set; Step S2: Design a scheme to expand the small data set obtained in step S1; this scheme includes two schemes: designing a combination feature expansion scheme and designing a multi-style data expansion scheme, to obtain an expanded data set; Step S3: construct the backbone network and target detection head; it is divided into the early stage of designing the deep separable convolution reconstruction parameterization module to construct the backbone network, the late stage of designing the non-adjacent window filling block to improve the multi-head self-attention module to form a lightweight neural network Transformer architecture to construct the backbone network, and the small target step size output improvement module to build the target detection head; Step S4: designing a defect detection model capable of identifying and locating industrial printed circuit board images; selecting the backbone network and target detection head in step S3, and combining the characteristics of industrial printed circuit board defect images to design a defect detection model for industrial printed circuit board images; Step S5: training a defect detection model for industrial printed circuit board images and performing defect detection on industrial printed circuit board images; using the training set in the expanded data set obtained in step S2 as the input for designing a defect detection model for industrial printed circuit board images in step S4, training and learning the defect detection model for industrial printed circuit board images, verifying the validation set in the expanded data set in step S2 through the trained defect detection model for industrial printed circuit board images, and performing defect identification and location on the test set in step S1, thereby proving the detection performance and generalization of the defect detection model for industrial printed circuit board images; Step S6: Design a sliding window small target detection strategy; Step S7: Design a tag deduplication solution; To solve the problem of repeated defect prediction in the overlapping area of ​​the sliding window in step S6, a label deduplication scheme is designed to filter out the labels of repeated predictions; Step S8: designing an adaptive defect screening scheme; Design an adaptive defect screening solution based on the size and style of the target defect to maximize the accuracy of defect prediction; In step S3, a backbone network and a target detection head are constructed; the steps are as follows: designing a deep separable convolutional reconstruction parameterization module to construct an early stage of the backbone network, designing a non-adjacent window filling block to improve the multi-head self-attention module to form a lightweight neural network Transformer architecture to construct a late stage of the backbone network, and designing a small target step size output improvement module to construct a target detection head; wherein, designing a deep separable convolutional reconstruction parameterization module, the specific steps are as follows: Step S31: Fusing the batch normalization layer and the convolution layer; Each branch containing the depthwise separable convolution and batch normalization layer is reconstructed, and the batch normalization layer and the depthwise separable convolution are fused into a new depthwise separable convolution layer, as shown in formula (1) and formula (2): （1）； （2）； In the formula, represents the weight of the new convolution matrix, represents the weight of the original convolution, , and are the number of output channels, the number of input channels and the size of the convolution kernel, respectively. , , and are the scaling parameter, offset parameter, standard deviation of the current batch, and mean of the current batch in the batch normalization layer, respectively. represents the new bias, Indicates the original bias; Step S32: merging branches; Will be The two branches constructed by the depthwise separable convolutional layer and the batch normalization layer are merged with the branch composed of batch normalization. The corresponding elements of all relevant weight matrices and biases need to be added, as shown in formulas (3) and (4): （3）； （4）； In the formula, and is the reconstructed weight matrix and bias, is the total number of branches, For the The weights of the new convolution matrix, For the A new bias; Among them, the non-adjacent window filling block is designed to improve the multi-head self-attention module. The specific steps are as follows: Step S33: Pass The size is The window is divided completely and non-overlapping into resolution sizes up to , width , the number of channels is Input feature map ; Step S34: Select the structures with the same relative position in each window as The feature vector is composed of Filling block groups, a total of Group position distribution feature map global padding block group ; Step S35: Group the global filling blocks of the position distribution feature map Input the neural network Transformer in sequence to calculate multi-head self-attention; calculate global correlation while optimizing computational complexity; Among them, the lightweight neural network Transformer architecture is designed. The specific steps are as follows: Step S36: constructing a local representation module; use The convolutional layer extracts local features, normalizes them through the batch normalization layer and passes through the activation function, and then passes through The convolutional layer expands the feature map to an appropriate number of channels to complete the local representation; Step S37: performing global characterization; The output of step S36 is used to improve the multi-head self-attention module, the neural network Transformer and The convolution layer completes the extraction of global features; Step S38: feature fusion; The output of step S37 is densely connected with the input of step S36, and then After the convolution layer, batch normalization layer and activation function, feature fusion is completed to prevent gradient disappearance; Among them, a small target step size output improvement module is designed, including the network layer improvement of the backbone network and the output layer detection head improvement. The specific improvements are as follows: Step S39: the network layer connected to the backbone network is improved by transforming the network layer originally connected to the backbone network with a step length of 32, 16, 8 into a network layer connected to the backbone network with a step length of 32, 16, 8, 4; Step S310: the output layer detection head is improved by replacing the output layer from the large-scale detection head P5, the medium-scale detection head P4, and the small-scale detection head P3 to the medium-scale detection head P4, the small-scale detection head P3, and the ultra-small-scale detection head P2; In step S4, a defect detection model capable of identifying and locating industrial printed circuit board images is designed. The specific steps are as follows: Step S41, the defect detection model of the industrial printed circuit board image is composed of a backbone network and a target detection head; Step S42, the first to fourth layers of the backbone network are constructed by a depthwise separable convolutional reconstruction parameterized module, and the fifth to seventh layers are constructed by a lightweight neural network Transformer architecture, and the input of each layer comes from the output result of the previous layer; In step S43, the target detection head is constructed through a small target step output improvement module to realize small target defect detection; the defect detection model of the industrial printed circuit board image realizes the target defect judgment in the industrial printed circuit board image.

2. The method for detecting industrial printed circuit board images based on deep learning according to claim 1 is characterized in that: In step S1, the data set is manually labeled. The specific steps are as follows: Step S11: Manually mark defects on the original industrial printed circuit board image; defect categories include mouse bite, open circuit, short circuit, disconnection and miscellaneous copper; Step S12: According to different defect categories, target defects are drawn on the original industrial printed circuit board image, and the target category and horizontal rectangular frame coordinates of the target position are marked and stored in the corresponding marking file; the marking file is in txt format, and each line of the marking file represents a marked target. The marking file has 5 values, which are , , , , ,in Indicates category, and Indicates the center coordinates of the annotation box. and Respectively represent the relative width and height of the annotation box; Step S13: Set is an integer greater than 3, For 2 The resolution of industrial printed circuit board images is ,and and Both multiples of; all industrial printed circuit board images in the small dataset are sorted by The ratio is divided into training set, validation set and test set. There is no overlap among the training set, validation set and test set. The small data set is used as the first data set D1.

3. The method for detecting industrial printed circuit board images based on deep learning according to claim 2 is characterized in that: In step S2, a scheme for expanding the small data set obtained in step S1 is designed; it is divided into two schemes: designing combined feature expansion and designing multi-style data expansion. The specific steps are as follows: Step S21: Fixed size cropping; the training set and validation set of the first dataset D1 manually annotated in step S1 are cropped with a resolution of Industrial printed circuit board images are cropped by fixed specifications according to Specifications, evenly cut to Size The industrial printed circuit board images are obtained to obtain the second data set D2; Step S22: Fixed specification secondary cropping; the resolution size of the second data set D2 is The industrial printed circuit board image is cut into two pieces according to the fixed specifications. Specifications, evenly cut into 4 pieces of size The industrial printed circuit board images are obtained to obtain the third data set D3; Step S23: random shuffling and splicing; All the resolutions in the third dataset D3 are The industrial printed circuit board images of are classified and shuffled according to the original corresponding positions of the images in the second dataset D2, and then randomly spliced ​​into blocks of size The industrial printed circuit board images are obtained to obtain the fourth data set D4; Step S24: manual fine-tuning; deleting the industrial printed circuit board images whose defect positions are on the boundary line and whose defect features are incomplete due to cropping and splicing, and obtaining the fifth data set D5 from the remaining industrial printed circuit board images; Step S25: The fifth data set D5 and the second data set D2 are aggregated to obtain a sixth data set D6 after combined feature expansion, and the combined feature expansion scheme is completed; Step S26: Based on the training set and validation set of the first data set D1, the resolution size is The industrial printed circuit board image is randomly cropped and cropped to a resolution of Industrial printed circuit board images are obtained to obtain the seventh data set D7; Step S27: Aggregate the sixth data set D6 and the seventh data set D7 to obtain an eighth data set D8; Step S28: performing a multi-style data expansion operation on the eighth data set D8; The step S28 of performing the diversified data expansion operation is specifically as follows: Randomly flip up and down, left and right, or up and down and left and right; manually re-mark the target defect position for distortion processing; randomly add color blocks; invert color processing; add Gaussian blur; Add salt and pepper noise; channel conversion; overlap; random scaling and splicing; Finally, the ninth dataset D9 is obtained after multi-style data expansion and is used as the final dataset D.

4. The method for detecting industrial printed circuit board images based on deep learning according to claim 3 is characterized in that: In step S5, the defect detection model of the industrial printed circuit board image is trained and the defect detection of the industrial printed circuit board image is performed. The specific steps are as follows: Step S51, inputting the training set of the final data set D into the defect detection model of the industrial printed circuit board image in step S4 for training. During the training process, the defect detection capability of the defect detection model of the industrial printed circuit board image is optimized by using the back propagation algorithm through the category loss, position loss and confidence loss until fitting; Step S52: After the fitting is completed, the final data set D is used to validate the data set. The defect detection performance of the defect detection model for industrial printed circuit board images is tested; Step S53, at the same time, using the test set of the first data set D1 with a size of The defect detection performance and generalization of the defect detection model for industrial printed circuit board images are tested.

5. The method for detecting industrial printed circuit board images based on deep learning according to claim 4 is characterized in that: In step S6, a sliding window small target detection strategy is designed. The specific steps are as follows: Step S61: For the first data set D1, the resolution size in the test set is The industrial printed circuit board image is cropped by sliding window; the coordinates of the upper left corner of the industrial printed circuit board image are The position of the clipping window starts sliding, and the resolution size of the clipping window is set to , the sliding step size is set to ,in , , and The best value for , or , , that is, each two windows have a size of overlapping areas; Step S62: the sliding window slides to the right or bottom of the original image in the test set of the first data set D1. If the croppable area is smaller than the size of the sliding window, no cropping is performed. Step S63: Crop by sliding window to obtain the common Size Industrial printed circuit board image; the size of the cropped resolution is The industrial printed circuit board image is input into the defect detection model of the industrial printed circuit board image trained in step S5 for detection, and the predicted target defect position information is restored back to the corresponding position of the original image in the test set of the first data set D1.

6. The method for detecting industrial printed circuit board images based on deep learning according to claim 5 is characterized in that: In step S7, a tag deduplication scheme is designed, and the specific steps are as follows: Step S71: Overlapping frame determination; Assume that the starting position coordinates of the two label frames A and B are: , ; The end position coordinates of the two label frames A and B are: , ; When the end position of label frame A and label frame B and The minimum value is greater than the starting position The maximum value, and the end position of label box A and label box B is The minimum value is greater than the starting position and The maximum value of , label frame A and label frame B overlap, otherwise label frame A and label frame B do not overlap; As shown in formula (5) to formula (8): （5）； （6）； （7）； （8）； In the formula, The larger horizontal coordinate of the starting position of label box A and label box B, The smaller horizontal coordinate of the end position of label box A and label box B. The larger vertical coordinate of the starting position of label box A and label box B, The smaller vertical coordinate of the end position of label box A and label box B when the overlap condition is met and When , label frame A and label frame B overlap, and step S72 and step S73 are executed; otherwise, label frame A and label frame B do not overlap; Step S72: merging overlapping positions; When it is determined that the label frame A and the label frame B overlap, the positions of the label frame A and the label frame B are merged, as shown in calculation formulas (9) to (12): （9）； （10）； （11）； （12）； In the formula, Indicates the starting position of the merge box coordinate, Indicates the starting position of the merge box coordinate, Indicates the end position of the merge box coordinate, Indicates the end position of the merge box coordinate; Step S73: Overlapping label category selection; When it is determined that label box A and label box B overlap, a category is selected based on the highest confidence of label box A and label box B.

7. The method for detecting industrial printed circuit board images based on deep learning according to claim 6 is characterized in that: In step S8, an adaptive defect screening scheme is designed, and the specific steps are as follows: Step S81: initial confidence screening; According to the overall prediction of the defect detection model of industrial printed circuit board images, an initial confidence level is set , the initial confidence The value range is , only keep the prediction confidence greater than the initial confidence Defect labels of the system are used to filter out most of the incorrectly predicted defect labels. Step S82: small target defect screening; The smaller the area of ​​the target defect is, the worse the prediction ability of the defect detection model of the industrial printed circuit board image is for the target defect, and the lower the confidence is; a classification discussion is conducted; Assume the starting position of the predicted defect label box is , the terminal position of the predicted defect label box is , the area of ​​the graphic defect label box is calculated as shown in formula (13): （13）； In the formula, The defect label area is predicted for the calculated image. The horizontal coordinates of the starting and ending positions of the predicted defect label box are , , the ordinates of the starting and ending positions of the predicted defect label box are , ; Set an area limit value R for normal size target defects and small target defects. When the area of ​​the predicted defect is smaller than the area limit value R, it satisfies When , the target defect is determined to be a small target defect, and the target defect is retained; For target defect area greater than the area limit value Defects, execute step S83; Step S83: long impurity copper defect screening; Step S831: Observe the characteristics of the defect type of impurity copper, which is divided into long impurity copper and short impurity copper; the defect area of ​​short impurity copper is smaller than the area limit value set in step S82 , short impurity copper defects are judged as small target defects; Step S832: For a long copper defect, the area is greater than the area threshold value set in step S82 , the long impurity copper defect is judged as a normal size target defect. The long impurity copper defect category is a relatively rare feature among defect features, and the prediction confidence is lower than other normal size target defects, so it is processed separately; Step S833: When the predicted target defect area is greater than the area threshold value , and when the defect category is miscellaneous copper, it is retained; for the other predicted defects, step S84 is executed; Step S84: Screening of other defects; For the predicted defect area greater than the area limit value , and the defect category is not a copper defect, the confidence level of the defect detection model prediction of the industrial printed circuit board image is higher; affected by the background noise in the industrial printed circuit board image, a small number of incorrect prediction labels will appear in the prediction results; Set a higher confidence level , higher confidence The value range is , and higher confidence The value is greater than the initial confidence ; Keep predictions with confidence greater than higher confidence The defect labels with confidence less than The incorrectly predicted labels are eliminated to maximize the correct prediction rate of the defect detection model for industrial printed circuit board images.

Images