PCB surface assembly defect detection method and device based on defect perception and computer equipment

Abstract

The invention belongs to the field of image processing and deep learning, and relates to a PCB surface assembly defect detection method and device based on defect perception and computer equipment. The surface assembly defect detection method comprises the following steps: performing data enhancement on a PCB surface assembly defect data set to expand the data set; preprocessing the expanded data set, and improving an image comparison graph through histogram equalization operation so as to highlight the position and defect category of the target; constructing a multi-scale defect sensing network, extracting common features by using a feature pyramid in combination with a feature structure similarity principle, inputting the common features into a region candidate network, and performing final defect classification and positioning; and sending a to-be-detected PCB assembly image in the test set into the trained multi-scale defect sensing network to obtain a final defect detection result. According to the method, the features of the PCB assembly image are effectively extracted, the feature expression of the detection model for the structural similarity between the feature points is enhanced, and the defect detection accuracy is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the fields of image processing and deep learning, and relates to a method, apparatus and computer device for detecting surface mounting defects of a PCB based on defect perception. Background Art

[0002] PCB boards play an important role in consumer electronics products and defense aerospace electronic products. The manufacturing quality of PCB products directly affects the performance of electronic products. Therefore, defect detection of PCB boards is a key link in the PCB production and manufacturing process. AOI optical automatic detection equipment can detect defects of PCB boards after soldered parts. AOI optical automatic detection equipment is a commonly used PCB board soldering defect detection equipment, but it has problems such as high equipment purchase cost, poor program portability, long program production time, high requirements for the professional level of operators, and time-consuming and laborious manual detection, which greatly affects the efficiency and accuracy of defect detection. In addition, there are small target defects in PCB boards. Small targets have fewer pixels, lack obvious texture and structure information, and are extremely vulnerable to the interference of the surrounding environment. Therefore, there are high requirements for positioning accuracy and recognition accuracy. PCB surface mounting defect detection based on deep learning can effectively improve the automation degree of detection and ensure the detection speed and accuracy.

[0003] Defect detection algorithms are mainly divided into three categories, namely CAD template image matching method, golden standard template method, design rule detection method and quantitative feature matching method. The CAD template image matching method requires the use of complex registration algorithms and cumbersome image conversion, and has high requirements for hardware devices; the golden standard template method requires specific templates to be developed for different models of PCBs, which is time-consuming and laborious; the design rule detection method is relatively simple to implement, but the detection process is very time-consuming and requires powerful computing power; the quantitative feature matching method requires high-precision registration algorithms, is easily affected by image deformation, has high requirements for computing power, and is prone to false detection and missed detection.

[0004] In recent years, image analysis methods based on deep learning have made good progress in machine vision image segmentation tasks and have received extensive attention in the field of machine vision.

[0005] Currently, there are mainly three major challenges in the task of PCB board assembly defect detection based on deep learning:

[0006] (1) The number of data samples is unbalanced. The number of different types of defect samples is unbalanced, resulting in the deep learning model may be unstable or invalid.

[0007] (2) The number of defect data samples is small. Due to the low occurrence probability of defective PCB products, only a small number of defect data samples can be collected.

[0008] (3) The defective samples and qualified samples in the PCB board have similar shapes and are extremely vulnerable to environmental interference. Therefore, high requirements are imposed on the positioning accuracy and recognition accuracy. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to provide a method, device and computer equipment for detecting surface mounting defects of PCB based on defect perception. In particular, it relates to a method for detecting surface mounting defects of PCB based on defect perception. Among them, the feature pyramid sub-module is used to extract multi-scale feature information. Due to the diversity of feature scales in the multi-scale structure, it can better represent defect features; the structural similarity weights of global semantics and single feature point semantics are introduced, so that the feature points are weighted with context information, and the prediction of defects by the network model takes global information as a reference, avoiding the reduction of the accuracy of defect prediction due to the single local feature point information.

[0010] A method for detecting surface mounting defects of PCB based on defect perception according to the present invention, the method includes the following steps:

[0011] S1. Expand the data set by using data augmentation technology;

[0012] S2. Perform data preprocessing on the expanded data set, and obtain a contrast-enhanced image by improving the image contrast through histogram equalization operation;

[0013] S3. Construct a multi-scale defect perception network with a multi-scale defect perception module, a region proposal network, a defect classification branch and a defect localization branch;

[0014] S4. Use the contrast-enhanced image and the corresponding defect label as the input of the multi-scale defect position perception module and output common features;

[0015] S5. The region proposal network takes the common features as the input and outputs proposal boxes, and uses the binary cross-entropy loss function and the smooth L1 loss function as the first classification loss function and the first regression loss function respectively, and minimizes the above loss functions to train the region proposal network;

[0016] S6. Screen the proposal boxes and perform proposal box pooling as the input of the defect classification branch and the defect localization branch;

[0017] S7. The defect classification branch uses the multi-class cross-entropy loss function as the second classification loss function to output the defect category, and the defect localization branch uses the smooth L1 loss function as the second regression loss function to output the defect position, and minimizes the above two loss functions to train the classification and localization network;

[0018] S8. Obtain the surface-mounted image of the PCB to be detected and send it into the trained multi-scale defect perception network to obtain the final defect detection result.

[0019] Optionally, data augmentation technology is used to expand the dataset, and image processing operations can be selected according to requirements to obtain more image data, including: flipping, rotating, cropping, scaling, translating, adding noise, etc.

[0020] Optionally, in step S2, the augmented dataset is preprocessed, and the histogram equalization operation is used to improve the image contrast to obtain a contrast-enhanced image. The data preprocessing includes but is not limited to the histogram equalization operation, and other image enhancement operations can also be used as the data preprocessing operation in the present invention.

[0021] Further, the histogram equalization operation includes statistically calculating the probability of each pixel appearing in the original image to obtain the gray probability density function. Integrating the gray probability density function to obtain the gray cumulative probability distribution function. Normalizing the gray cumulative probability distribution function to [0, 255] and performing rounding operations to obtain the normalized gray cumulative probability distribution function. Inputting the original image into the normalized gray cumulative probability distribution function to obtain the contrast-enhanced image.

[0022] Further, the multi-scale defect perception module in step S3 includes 4 repeated encoding units, a feature pyramid sub-module, and a feature structural similarity calculation sub-module. Each encoding unit includes 2 convolutional layers and 1 average pooling layer. The feature pyramid sub-module includes 4 parallel feature extraction branches and 1 convolutional layer. Each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer, so as to achieve the purpose of multi-scale feature extraction.

[0023] Optionally, the 4 types of scale feature maps can be 1×1, 2×2, 3×3, 6×6, or can also be sizes such as 1×1, 2×2, 4×4, 8×8 or 1×1, 2×2, 4×4, 16×16, etc. Retain the 1×1 feature map size to introduce global features, and the other 3 feature map sizes can be selected according to needs.

[0024] Further, the multi-scale defect perception module in step S3 includes a feature pyramid sub-module and a feature structural similarity calculation sub-module. The structural similarity calculation sub-module is used to measure the structural similarity weight between the global feature vector and the feature point vector, and the calculation formula is:

[0025]

[0026]

[0027]

[0028] w mn = l(m, n)c(m, n)u(m, n),

[0029] where m represents the global feature vector of the original feature map, implemented through global average pooling, n represents the feature point vector at (x, y) in the original feature map, and μ m and μ n represent the means of the global feature vector m and the feature point vector n, and σ m and σ n represent the standard deviations of the global feature vector m and the feature point vector n. C1, C2, and C3 are constant terms. The structural similarity weight w mn is weighted to the feature point vector n at (x, y), and the calculation formula is as follows:

[0030] n' = w mn n,

[0031] where n' represents the weighted feature point vector. In addition, the original feature map is retained through skip connections.

[0032] Furthermore, the skip connection is to stack the original feature map and the weighted feature map along the feature channel direction, so as to classify and locate defects under the guidance of global semantics while retaining the original feature information.

[0033] Furthermore, the region proposal network described in step S3 includes a 3×3 convolutional layer and two parallel 1×1 convolutional layers, which are used to output the coordinate offset compensation value of the anchor box and the foreground and background probabilities of the anchor box respectively. The number of feature channels of the 3×3 convolutional layer is 512, which is used to convert the number of channels of the feature map after feature pyramid and structural similarity calculation. The number of feature channels of the two parallel 1×1 convolutional layers are 36 and 18 respectively.

[0034] Furthermore, the binary cross-entropy loss function described in step S5, the calculation formula is as follows:

[0035]

[0036] where N anchor represents the number of predicted proposal boxes, i represents the i-th predicted proposal box, n represents the n-th category, and there are two values for the foreground (n = 1) and the background (n = 2). p i,n represents the probability that the i-th predicted proposal box is predicted as the n-th category. p i,n * is the positive and negative anchor box category, which is 1 for the foreground and 0 for the background. The smooth L1 loss function described in step S5, the calculation formula is as follows:

[0037]

[0038] L reg (t i ,t i * ) = R(t i -t i * ) = L smooth (x)

[0039]

[0040] where t i -t i * = x, N anchor represents the number of predicted proposal boxes, the value of σ is 3, is the positive and negative anchor box category, 1 for foreground and 0 for background, that is, the loss is calculated for the foreground and not for the background, t i * = (t x ,t y ,t w ,t h ) represents the position parameter of the anchor box, t i = (t x ,t y ,t w ,t h ) represents the position parameter of the predicted proposal box. The overall loss function of the region proposal network is calculated as follows:

[0041] L rpn = L1 cls + L1 loc

[0042] Furthermore, the proposal box screening described in step S6 includes applying the coordinate offset compensation value of the anchor box to all anchor boxes, obtaining the foreground probability scores of all anchor boxes, sorting them from high to low, selecting the top 12,000 anchor boxes, and then performing non-maximum suppression screening to obtain 300 proposal boxes.

[0043] Optionally, the selection of the top 12,000 anchor boxes can be 11,000, 13,000, 14,000 or other values. The number of the first screening needs to be determined according to requirements. The number of the 300 proposal boxes obtained by screening can be 250, 350 or other values. The number of the second screening needs to be determined according to requirements. Both affect the speed and accuracy of the training and testing of the algorithm model at the same time.

[0044] Further, the proposed box pooling described in step S6 includes dividing each proposed box into a fixed number of sub-regions, taking the maximum value for each sub-region, and converting candidate boxes of different sizes into feature representations of a fixed size.

[0045] Optionally, taking the maximum value for each sub-region can also be taking the average value for each sub-region.

[0046] Further, the multi-class cross-entropy loss function described in step S7 has the following calculation formula:

[0047] as follows:

[0048]

[0049] where N proposal represents the number of proposed boxes, C represents the number of target categories, i represents the i-th proposed box, n represents the n-th category, and p i,n represents the probability that the i-th proposed box belongs to the n-th category. p i,n * is 1 when it corresponds to the corresponding category and 0 for other categories. The smooth L1 loss function described in step S7 has the following calculation formula:

[0050]

[0051] L reg (t i , t i * ) = R(t i - t i * ) = L smooth (x)

[0052]

[0053] where t i - t i * = x, N proposal represents the number of proposed boxes, the value of σ is 1, represents 1 when there is an object in the i-th anchor box and 0 when there is no object. t i * = (t x , t y , t w , t h ) represents the position parameters of the true bounding box, and t i = (t x , t y , t w , t h) Represents the position parameters of the predicted bounding box. The calculation formula of the overall loss function is as follows:

[0054] L fast-rcnn = L2 cls + L2 loc

[0055] The present invention also proposes a PCB surface mounting defect detection device based on defect perception, characterized in that the device includes:

[0056] An image acquisition module, used to acquire a PCB surface mounting image dataset and a PCB surface mounting defect image to be detected;

[0057] A dataset expansion module, used to perform data enhancement operations on the dataset to expand the number of images;

[0058] An image preprocessing module, used to perform histogram equalization operations on the expanded dataset to enhance the image contrast;

[0059] A network construction module, used to construct a multi-scale defect perception network with a multi-scale defect position perception module, a region proposal network, a defect classification module, and a defect localization module;

[0060] The multi-scale position perception module fully extracts common features through 4 repeated encoding units, a feature pyramid sub-module, and a feature structural similarity calculation sub-module;

[0061] A region proposal network module, used to generate proposed bounding boxes of interest;

[0062] A proposed bounding box screening module, used to perform two screenings on the proposed bounding boxes;

[0063] A proposed bounding box pooling module, used to perform fixed-size conversion on the proposed bounding boxes;

[0064] A defect classification module, used to judge the defect category according to the proposed bounding boxes provided by the region proposal network;

[0065] A defect localization module, used to finally determine the defect position according to the proposed bounding boxes provided by the region proposal network;

[0066] A defect prediction module, used to pass the PCB surface mounting image to be detected through the defect perception network to obtain a PCB surface mounting image marked with defect positions and categories.

[0067] Furthermore, the dataset expansion module includes:

[0068] A flipping unit, which flips the original image left and right to obtain a new image;

[0069] A rotation unit that rotates the original image by 45°, 90°, 135°, and 180° to obtain a new image;

[0070] A cropping unit that randomly crops the original image to obtain a new image;

[0071] A scaling unit that scales the original image proportionally to obtain a new image.

[0072] Furthermore, the image preprocessing module includes:

[0073] A grayscale probability density calculation unit that statistically calculates the probability of each pixel in the original image to obtain a grayscale probability density function;

[0074] A grayscale cumulative probability distribution calculation unit that integrates the grayscale probability density function to obtain a grayscale cumulative probability distribution function;

[0075] A normalization unit that normalizes the grayscale cumulative probability distribution function to [0, 255] and performs rounding operations to obtain a normalized grayscale cumulative probability distribution function;

[0076] A grayscale conversion unit that inputs the original image into the normalized grayscale cumulative probability distribution function to obtain a contrast-enhanced image.

[0077] Furthermore, each of the 4 repeated encoding units includes 2 convolutional layers and 1 average pooling layer.

[0078] Furthermore, the feature pyramid sub-module includes 4 parallel feature extraction branches, and each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer.

[0079] Furthermore, the feature structural similarity calculation sub-module includes:

[0080] A pixel scanning unit for traversing the feature point vectors at each feature point position of the feature map;

[0081] A global feature vector unit for calculating the global feature vector using the global pooling layer;

[0082] A structural similarity weight calculation unit for calculating the structural similarity weight between the global feature vector and the feature point vector of the feature map;

[0083] A feature map weighting unit for weighting the structural similarity weight to the corresponding feature point vectors to form a weighted feature map;

[0084] A skip connection unit for stacking the original feature map and the weighted feature map along the feature channel direction.

[0085] Furthermore, the region candidate network module includes:

[0086] The feature channel transformation unit changes the number of features through a convolutional layer with a convolution kernel of 3×3 and 512 channels;

[0087] The anchor box classification unit outputs the foreground and background probabilities of the anchor boxes through a convolutional layer with a convolution kernel of 1×1 and 18 channels;

[0088] The anchor box localization unit outputs the coordinate offset compensation value of the anchor boxes through a convolutional layer with a convolution kernel of 1×1 and 36 channels.

[0089] The first position supervision unit is used to use the positive anchor box position offset parameter as the first position supervision information;

[0090] The first position loss function unit is used to train the region proposal network according to the positive anchor box position offset parameter and the predicted proposal box position;

[0091] The first category supervision unit is used to use the positive and negative anchor boxes as the first category supervision information;

[0092] The first category loss function unit is used to train the region proposal network according to the positive and negative anchor boxes and the predicted proposal box category parameters.

[0093] Furthermore, the proposal box screening module includes:

[0094] The coordinate offset compensation unit is used to apply the coordinate offset compensation value predicted by the region proposal network module to all corresponding anchor boxes;

[0095] The pre-screening unit is used to sort the foreground probabilities of all anchor boxes predicted by the region proposal network module from high to low, and select the top 12,000 anchor boxes as proposal boxes;

[0096] The screening unit is used to perform non-maximum suppression on the remaining proposal boxes to obtain 300 final proposal boxes.

[0097] Furthermore, the proposal box pooling module includes:

[0098] The region division unit divides each input proposal box into a fixed number of sub-regions;

[0099] The region pooling unit takes the maximum value of each sub-region to obtain a feature representation of a fixed size.

[0100] Furthermore, the defect classification module and the defect localization module include:

[0101] The second position supervision unit is used to use the true bounding box position parameter as the second position supervision information;

[0102] A second position loss function unit for training a classification and localization network based on real bounding box position parameters and predicted bounding box position parameters;

[0103] A second class supervision unit for using the number of real bounding box classes as the second class supervision information;

[0104] A second class loss function unit for training a classification and localization network based on real bounding box class parameters and predicted bounding box class parameters.

[0105] The present invention also provides a computer device, including at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the method provided by the present invention by invoking the program instructions.

[0106] Advantages of the present invention:

[0107] 1) The present invention provides a method for detecting PCB surface mounting defects based on defect perception, which can improve the positioning accuracy and recognition accuracy of defect detection when the defect samples and qualified samples have similar shapes.

[0108] 2) The present invention deeply extracts multi-scale defect information, can analyze the differences between defect samples and qualified samples from different feature scales, improves the representation ability of the model from the feature level, and thus more accurately detects defects.

[0109] 3) The present invention uses global feature vectors and feature point vectors for structural similarity measurement, introduces additional structural similarity features, enables the feature point information of the model to contain global information, and effectively guides the optimization direction of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] To make the objectives, technical solutions and advantages of the present invention clearer, the present invention provides the following drawings for description:

[0111] Figure 1 Schematic diagram of the overall process of the present invention;

[0112] Figure 2 Schematic diagram of the multi-scale defect perception network structure adopted by the present invention;

[0113] Figure 3 Schematic diagram of the structure of the feature structure similarity calculation sub-module adopted by the present invention;

[0114] Figure 4 Schematic diagram of the data flow of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0115] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the following describes the technical solutions in the embodiments of the present invention clearly and completely with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0116] As Figure 1 shown, a PCB surface mounting defect detection method based on defect perception of the present method may specifically include the following steps:

[0117] S1. Expand the data set by using data augmentation techniques;

[0118] Data augmentation is a common operation in the field of deep learning. It uses common image processing operations to increase the diversity of training data, effectively alleviates the problem of insufficient data in deep learning, and at the same time can also improve the generalization of the training model. Common image processing operations include: flipping, rotating, cropping, scaling, translating, adding noise, etc. Flipping is to flip the original image in the left-right direction and then obtain a new image for the bounding box; rotating is to rotate the original image by 45°, 90°, 135°, 180° to obtain a new image; cropping is to randomly crop the original image to obtain a new image; scaling is to scale the original image proportionally to obtain a new image; translating is to translate the original image to obtain a new image.

[0119] S2. Perform data preprocessing on the expanded data set, and obtain a contrast-enhanced image by enhancing the image contrast through histogram equalization operation;

[0120] Use the histogram equalization method to enhance the contrast of the image and make the boundaries clearer. First, count the probability of each pixel appearing in the original image to obtain the gray-scale probability density function. Secondly, integrate the gray-scale probability density function to obtain the gray-scale cumulative probability distribution function. Then, normalize the gray-scale cumulative probability distribution function to [0, 255] and round it to obtain the normalized gray-scale cumulative probability distribution function. Finally, input the original image into the normalized gray-scale cumulative probability distribution function to obtain a contrast-enhanced image.

[0121] S3. Construct a multi-scale defect perception network with a multi-scale defect perception module, a region candidate network, a defect classification branch, and a defect localization branch;

[0122] In one embodiment, the multi-scale defect perception network may refer to Figure 2, this example provides an object detection network with a feature pyramid and feature structural similarity calculation. The network mainly consists of a multi-scale defect perception module, a region proposal network, a defect classification branch, and a defect localization branch. First, the training set images are input into 4 encoding units for feature extraction to obtain feature maps, and the size of the feature maps is reduced to 1 / 16 of the original image. Secondly, through pooling with specific sizes S = {1, 2, 3, 6}, the pyramid module scales the feature maps to retain feature maps of multiple scales. In each scale branch, the module will perform feature structural similarity calculation and concatenate the feature maps of each scale to obtain a common feature map. Then, the common feature map passes through the region proposal module, which compresses and fuses the features of the common feature map and outputs proposal box information, reducing the number of feature maps from 2048 to 512. Subsequently, the proposal box pooling module maps the coordinates of the proposal boxes and fixes their sizes, with the fixed size being 7×7. Finally, the proposal boxes pass through 2 consecutive fully connected layers with a length of 4096 and 2 parallel fully connected layers with lengths of 21 and 84 respectively to output predicted bounding box information.

[0123] The multi-scale defect perception module includes 4 repeated encoding units, a feature pyramid sub-module, and a feature structural similarity calculation sub-module. Each encoding unit includes 2 convolutional layers and 1 average pooling layer. The feature pyramid unit includes 4 parallel feature extraction branches, and each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer, so as to map a single-size feature map into feature maps of different scales, thereby achieving the purpose of multi-scale feature extraction.

[0124] In one embodiment, the feature structural similarity calculation sub-module can refer to Figure 3 . First, perform structural similarity weight calculation. Use global average pooling operation to obtain the global feature vector as the global information reference. At the same time, use a sliding window with a size of 1×1 and a stride of 1 to traverse the feature point vectors of each feature point on the feature map, and calculate the structural similarity weight between the feature point vector and the global feature vector through the structural similarity loss function. Finally, in each structural similarity unit, retain the original feature map through skip connection and fuse the weighted feature map of the current feature depth and the original feature map through a convolutional layer with a size of 1×1 and 512 feature channels. Among them, the structural similarity function is defined as shown in formula (1):

[0125] w mn = l(m,n)c(m,n)u(m,n) (1)

[0126] In the formula, the definitions of l(m,n), c(m,n), and u(m,n) are shown in formulas (2)-(4):

[0127]

[0128]

[0129]

[0130] Among them, m represents the global feature vector of the original feature map, which is realized by global average pooling. n represents the feature point vector at (x, y) in the original feature map, and μ m and μ n represent the means of the global feature vector m and the feature point vector n, and σ m and σ n represent the standard deviations of the global feature vector m and the feature point vector n. C1, C2, and C3 are constant terms. The structural similarity weight w mn is weighted to the feature point vector n at (x, y), and the calculation is as shown in formula (5):

[0131] n′ = w mn n (5)

[0132] Among them, n′ represents the weighted feature point vector.

[0133] S4. Use the contrast-enhanced image and the corresponding defect label as the input of the multi-scale defect location perception module and output the common features;

[0134] S5. The region candidate network takes the common features as the input and outputs the proposal boxes. Use the binary cross-entropy loss function and the smooth L1 loss function as the first classification loss function and the first regression loss function respectively, and minimize the above loss functions to train the region candidate network;

[0135] The region candidate network consists of a 3×3 convolutional layer and two parallel 1×1 convolutional layers, which are used to output the coordinate offset compensation value of the anchor box and the foreground and background probabilities of the anchor box respectively. Among them, the binary cross-entropy loss function has the following calculation formula:

[0136]

[0137] Among them, N anchor represents the number of predicted proposal boxes, i represents the i-th predicted proposal box, n represents the n-th category, and there are two values for the foreground (n = 1) and the background (n = 2). p i,n represents the probability that the i-th predicted proposal box is predicted as the n-th category, and p i,n * is the positive and negative anchor box category, which is 1 for the foreground and 0 for the background. The smooth L1 loss function described in step S5 has the calculation formula as shown in formulas (7)-(9):

[0138]

[0139] L reg (t i ,t i * ) = R(t i -t i * ) = L smooth (x) (8)

[0140]

[0141] where t i -t i * = x, N anchor represents the number of predicted proposal boxes, the value of σ is 3, is the positive and negative anchor box category, which is 1 for foreground and 0 for background, that is, the loss is calculated for foreground and not for background, t i * = (t x ,t y ,t w ,t h ) represents the position parameter of the anchor box, t i = (t x ,t y ,t w ,t h ) represents the position parameter of the predicted proposal box. The calculation formula of the overall loss function of the regional candidate network is as follows:

[0142] L rpn = L1 cls + L1 loc (10)

[0143] S6. Perform proposal box screening and proposal box pooling on the proposal boxes as the inputs of the defect classification branch and the defect localization branch;

[0144] The proposal box screening is performed twice, namely pre-screening and final screening. In the pre-screening stage, the coordinate offset compensation value of the anchor box is applied to all anchor boxes, and the foreground probability scores of all anchor boxes are obtained, sorted from high to low, and the top 12,000 anchor boxes are selected. In the final screening stage, the selected anchor boxes are screened by non-maximum suppression to obtain the most representative 300 proposal boxes. Non-maximum suppression selects the proposal box with the highest score by calculating the confidence of the proposal box and sorting the confidence, and at the same time suppresses other proposal boxes with a high overlap degree with the selected proposal box, thereby effectively reducing the number of proposal boxes.

[0145] The proposal box pooling divides each proposal box into a fixed number of sub-regions, takes the maximum value for each sub-region, and converts candidate boxes of different sizes into a fixed-size feature representation.

[0146] S7. The defect classification branch uses the multi-class cross-entropy loss function as the second classification loss function to output the defect category, and the defect localization branch uses the smooth L1 loss function as the second regression loss function to output the defect position. Minimize the above two loss functions to train the classification and localization network;

[0147] The multi-class cross-entropy loss function has the following calculation formula:

[0148]

[0149] Among them, N proposal represents the number of proposal boxes, C represents the number of target categories, i represents the i-th proposal box, n represents the n-th category, and p i,n represents the probability that the i-th proposal box belongs to the n-th category. p i,n * is 1 when it is the corresponding category and 0 for other categories. The smooth L1 loss function described in step S7 has the following calculation formula:

[0150]

[0151] L reg (t i ,t i * ) = R(t i -t i * ) = L smooth (x) (13)

[0152]

[0153] Among them, t i -t i * = x, N proposal represents that the number of proposal boxes is, the value of σ is 1, represents 1 when the i-th anchor box has an object and 0 when there is no object. t i * = (t x ,t y ,t w ,t h ) represents the position parameters of the true bounding box, and t i = (t x ,t y ,t w ,t h ) represents the position parameters of the predicted bounding box. The calculation formula of the overall loss function is as follows:

[0154] L fast-rcnn = L2 cls + L2loc (15)

[0155] As Figure 4 shown, in one embodiment, the training phase of this embodiment includes performing data augmentation and image enhancement on the original images in the training set of the PCB surface mounting image dataset to obtain training set images and validation set images. The PCB surface mounting images and defect labels are used as the input of the multi-scale defect perception network, and the defect detection results of the validation set images are used as the basis for evaluating the defect detection effect of the model. The network model is iteratively optimized, and finally a trained network model is obtained.

[0156] Finally, the overall loss function calculation formula of the multi-scale defect perception network is as follows:

[0157] L = L1 cls + L1 loc + L2 cls + L2 loc (16)

[0158] S8. Obtain the PCB surface mounting image to be detected and send it into the trained multi-scale defect perception network to obtain the final defect detection result.

[0159] As Figure 4 shown, in one embodiment, this process belongs to the test phase, and in this phase, only the image to be detected needs to be sent into the trained network model to obtain the defect detection result.

[0160] The present invention also proposes a PCB surface mounting defect detection device based on defect perception. The device includes:

[0161] An image acquisition module, configured to acquire a PCB surface mounting image dataset and a PCB surface mounting defect image to be detected;

[0162] A dataset expansion module, configured to perform data augmentation operations on the dataset to expand the number of images;

[0163] An image preprocessing module, configured to perform histogram equalization operations on the expanded dataset to enhance the image contrast;

[0164] A network construction module, configured to construct a multi-scale defect perception network with a multi-scale defect position perception module, a region proposal network, a defect classification module, and a defect localization module;

[0165] A multi-scale position perception module, which fully extracts common features through 4 repeated encoding units, a feature pyramid sub-module, and a feature structural similarity calculation sub-module;

[0166] A region proposal network module, configured to generate proposed boxes of interest;

[0167] A proposal box screening module, which is used to screen the proposal box twice;

[0168] A proposal box pooling module, which is used to perform fixed-size conversion on the proposal box;

[0169] A defect classification module, which is used to judge the defect category according to the proposal box provided by the region candidate network;

[0170] A defect localization module, which is used to finally determine the defect position according to the proposal box provided by the region candidate network;

[0171] A defect prediction module, which is used to pass the PCB surface assembly image to be detected through the defect perception network to obtain the PCB surface assembly image marked with the defect position and category.

[0172] Furthermore, the dataset augmentation module includes:

[0173] A flipping unit, which flips the original image horizontally to obtain a new image;

[0174] A rotation unit, which rotates the original image by 45°, 90°, 135°, 180° to obtain a new image;

[0175] A cropping unit, which randomly crops the original image to obtain a new image;

[0176] A scaling unit, which scales the original image proportionally to obtain a new image.

[0177] Furthermore, the image preprocessing module includes:

[0178] A gray-scale probability density calculation unit, which counts the probability of each pixel in the original image to obtain a gray-scale probability density function;

[0179] A gray-scale cumulative probability distribution calculation unit, which integrates the gray-scale probability density function to obtain a gray-scale cumulative probability distribution function;

[0180] A normalization unit, which normalizes the gray-scale cumulative probability distribution function to [0, 255] and performs rounding operations to obtain a normalized gray-scale cumulative probability distribution function;

[0181] A gray-scale conversion unit, which inputs the original image into the normalized gray-scale cumulative probability distribution function to obtain a contrast-enhanced image.

[0182] Furthermore, each of the 4 repeated encoding units includes 2 convolutional layers and 1 average pooling layer.

[0183] Furthermore, the feature pyramid sub-module includes 4 parallel feature extraction branches, and each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer.

[0184] Further, the feature structure similarity calculation sub-module includes:

[0185] A pixel scanning unit for traversing the feature point vectors at each feature point position of the feature map;

[0186] A global feature vector unit for calculating the global feature vector using the global pooling layer;

[0187] A structure similarity weight calculation unit for calculating the structure similarity weight between the global feature vector and the feature point vector of the feature map;

[0188] A feature map weighting unit for weighting the structure similarity weight to the corresponding feature point vectors to form a weighted feature map;

[0189] A skip connection unit for stacking the original feature map and the weighted feature map along the feature channel direction.

[0190] Further, the region proposal network module includes:

[0191] A feature channel transformation unit for changing the number of features through a convolutional layer with a 3×3 convolutional kernel and 512 channels;

[0192] An anchor box classification unit for outputting the foreground and background probabilities of the anchor boxes through a convolutional layer with a 1×1 convolutional kernel and 18 channels;

[0193] An anchor box localization unit for outputting the coordinate offset compensation values of the anchor boxes through a convolutional layer with a 1×1 convolutional kernel and 36 channels.

[0194] A first position supervision unit for using the positive anchor box position offset parameter as the first position supervision information;

[0195] A first position loss function unit for training the region proposal network according to the positive anchor box position offset parameter and the predicted proposal box position;

[0196] A first category supervision unit for using the positive and negative anchor boxes as the first category supervision information;

[0197] A first category loss function unit for training the region proposal network according to the positive and negative anchor boxes and the predicted proposal box category parameters.

[0198] Further, the proposal box screening module includes:

[0199] A coordinate offset compensation unit for applying the coordinate offset compensation values predicted by the region proposal network module to all corresponding anchor boxes;

[0200] A pre-screening unit for sorting the foreground probabilities of all anchor boxes predicted by the region candidate network module from high to low, and selecting the top 12,000 anchor boxes as proposed boxes;

[0201] A screening unit for performing non-maximum suppression on the remaining proposed boxes to obtain 300 final proposed boxes.

[0202] Furthermore, the proposed box pooling module includes:

[0203] A region division unit for dividing each input proposed box into a fixed number of sub-regions;

[0204] A region pooling unit for taking the maximum value of each sub-region to obtain a feature representation of a fixed size.

[0205] Furthermore, the defect classification module and the defect localization module include:

[0206] A second position supervision unit for using the real bounding box position parameters as the second position supervision information;

[0207] A second position loss function unit for training the classification and localization network according to the real bounding box position parameters and the predicted bounding box position parameters;

[0208] A second category supervision unit for using the real bounding box category number as the second category supervision information;

[0209] A second category loss function unit for training the classification and localization network according to the real bounding box category parameters and the predicted bounding box category parameters.

[0210] The present invention also proposes a computer device, including at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the method proposed by the present invention by invoking the program instructions.

[0211] Of course, it can be understood that some features of the method, device, and computer device in the present invention can be mutually referred to, and the present invention will not list them one by one in order to save space.

[0212] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: ROM, RAM, disk, or optical disc, etc.

[0213] The above-described embodiments have further elaborated on the object, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made to the present invention within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A PCB surface mounting defect detection method based on defect perception, characterized in that, The method includes the following steps: S1. Expand the dataset using data augmentation techniques; S2. Perform data preprocessing on the augmented dataset, and obtain a contrast-enhanced image by enhancing the image contrast through histogram equalization operation; S3. Construct a multi-scale defect perception network with a multi-scale defect perception module, a region proposal network, a defect classification branch, and a defect localization branch; S4. Use the contrast-enhanced image and the corresponding defect label as the input of the multi-scale defect location perception module and output common features; S5. The region proposal network takes the common features as the input and outputs proposal boxes. Use the binary cross-entropy loss function and the smooth L1 loss function as the first classification loss function and the first regression loss function respectively, and minimize the above loss functions to train the region proposal network; S6. Perform proposal box screening and proposal box pooling on the proposal boxes as the input of the defect classification branch and the defect localization branch; S7. The defect classification branch uses the multi-class cross-entropy loss function as the second classification loss function to output the defect category, and the defect localization branch uses the smooth L1 loss function as the second regression loss function to output the defect location. Minimize the above two loss functions to train the classification and localization network; S8. Obtain the surface-mounted PCB image to be detected and send it into the trained multi-scale defect perception network to obtain the final defect detection result.

2. The method for detecting surface mounting defects of a PCB based on defect perception according to claim 1, characterized in that, The data augmentation techniques described in step S1 include operations such as flipping, rotating, cropping, and scaling the original image.

3. A method for detecting surface mounting defects of a PCB based on defect perception according to claim 1, characterized in that, For the image enhancement technique described in step S2, calculate the gray probability density function and the normalized gray cumulative probability distribution function in sequence to obtain the gray conversion function, and input the original image into the restoration conversion function to obtain the enhanced image. The gray probability density function includes counting the probability of each pixel appearing in the original image. The normalized gray cumulative probability distribution function includes integrating the gray probability density function and performing normalization to [0, 255] and rounding operations.

4. A method for detecting surface mounting defects of a PCB based on defect perception according to claim 1, characterized in that, The multi-scale defect perception module described in step S3 includes 4 repeated encoding units, a feature pyramid sub-module, and a feature structural similarity calculation sub-module. Each encoding unit includes 2 convolutional layers and 1 average pooling layer. The feature pyramid sub-module includes 4 parallel feature extraction branches and 1 convolutional layer. Each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer, so as to achieve the purpose of multi-scale feature extraction. The feature structural similarity calculation sub-module calculates the similarity between the global feature vector and the feature point vector, and the calculation formula is: w mn = l(m,n)c(m,n)u(m,n), Among them, m represents the global feature vector of the original feature map, which is realized by global average pooling. n represents the feature point vector at (x, y) in the original feature map, μ m and μ n represent the means of the global feature vector m and the feature point vector n, and σ m and σ n represent the standard deviations of the global feature vector m and the feature point vector n. C1, C2, and C3 are constant terms. The structural similarity weight w mn is weighted to the feature point vector n at (x, y), and the calculation formula is as follows: n′ = w mn n, where n′ represents the weighted feature point vector. In addition, the original feature map is retained through skip connections, including stacking the original feature map and the weighted feature map along the feature channel direction, so as to classify and locate defects under the guidance of global semantics while retaining the original feature information. Among them, the region proposal network includes 1 3×3 convolutional layer and 2 parallel 1×1 convolutional layers, which are used to output the coordinate offset compensation value of the anchor box and the foreground and background probabilities of the anchor box respectively.

5. A method for detecting surface mounting defects of a PCB based on defect perception according to claim 1, characterized in that, For the binary cross-entropy loss function described in step S5, the calculation formula is as follows: Among them, N anchor represents the number of predicted proposal boxes, i represents the i-th predicted proposal box, n represents the n-th category, and there are two values, foreground (n = 1) and background (n = 2). p i,n represents the probability that the i-th predicted proposal box is predicted as the n-th category. p i,n * is the positive and negative anchor box category, which is 1 for foreground and 0 for background. The smooth L1 loss function described in step S5 has the following calculation formula: Among them, N anchor indicates that the number of predicted proposal boxes is, and the value of σ is 3. is the positive and negative anchor box category, which is 1 for foreground and 0 for background, that is, the foreground calculates the loss and the background does not calculate the loss. represents the position parameter of the anchor box, t i =(t x , t y , t w , t h ) represents the position parameter of the predicted proposal box. The calculation formula of the overall loss function of the region candidate network is as follows: L rpn = L1 cls + L1 loc .

6. A method for detecting surface mounting defects of a PCB based on defect perception according to claim 1, characterized in that, The proposed box screening described in step S6 includes applying the coordinate offset compensation value of the anchor box to all anchor boxes, obtaining the foreground probability scores of all anchor boxes, sorting them from high to low, selecting the top 12,000 anchor boxes, and then performing non-maximum suppression (NMS) screening to obtain 300 proposed boxes. The proposed box pooling includes dividing each proposed box into a fixed number of sub-regions and taking the maximum value for each sub-region to convert candidate boxes of different sizes into a fixed-size feature representation.

7. A method for detecting surface mounting defects of a PCB based on defect perception according to claim 1, characterized in that, The multi-class cross-entropy loss function described in step S7 has the following calculation formula: Among them, N proposal represents the number of proposal boxes, C represents the number of target categories, i represents the i-th proposal box, n represents the n-th category, p i,n represents the probability that the i-th proposal box belongs to the n-th category, p i,n * is 1 when it is the corresponding category and 0 for other categories. The smooth L1 loss function described in step S7 has the following calculation formula: Among them, N proposal indicates that the number of suggestion boxes is, and the value of σ is 1. indicates 1 when the i-th anchor box has an object and 0 when there is no object. indicates the position parameter of the ground truth bounding box, t i =(t x , t y , t w , t h ) indicates the position parameter of the predicted bounding box. The calculation formula of the overall loss function is as follows: L fast-rcnn = L2 cls + L2 loc 。 8. A PCB surface mounting defect detection device based on defect perception, characterized in that, The device includes: An image acquisition module, configured to acquire a PCB surface assembly image dataset and a PCB surface assembly defect image to be detected. A dataset augmentation module, configured to perform data enhancement operations on the dataset to augment the number of images, including: A flipping unit, which flips the original image left and right to obtain a new image; A rotation unit, which rotates the original image by 45°, 90°, 135°, and 180° to obtain new images; A cropping unit, which randomly crops regions of the original image to obtain new images; A scaling unit, which scales the original image proportionally to obtain new images. An image preprocessing module, configured to perform histogram equalization operations on the augmented dataset to enhance image contrast, including: A gray-scale probability density calculation unit, which statistically calculates the probability of each pixel in the original image to obtain a gray-scale probability density function; A gray-scale cumulative probability distribution calculation unit, which integrates the gray-scale probability density function to obtain a gray-scale cumulative probability distribution function; A normalization unit, which normalizes the gray-scale cumulative probability distribution function to [0, 255] and performs rounding operations to obtain a normalized gray-scale cumulative probability distribution function; A gray-scale conversion unit, which inputs the original image into the normalized gray-scale cumulative probability distribution function to obtain a contrast-enhanced image. A network construction module, configured to construct a multi-scale defect perception network with a multi-scale defect location perception module, a region proposal network, a defect classification module, and a defect localization module. The multi-scale location perception module fully extracts common features through 4 repeated encoding units, a feature pyramid sub-module, and a feature structural similarity calculation sub-module. The feature pyramid sub-module includes 4 parallel feature extraction branches, and each feature extraction branch includes 1 average pooling layer and 1 bilinear interpolation layer. The feature structural similarity calculation sub-module includes: A pixel scanning unit, configured to traverse the feature point vectors at each feature point position of the feature map; A global feature vector unit, configured to calculate the global feature vector using a global pooling layer; A structural similarity weight calculation unit, configured to calculate the structural similarity weight between the global feature vector and the feature point vector of the feature map; A feature map weighting unit, configured to weight the structural similarity weight to the corresponding feature point vector to form a weighted feature map; A skip connection unit, configured to stack the original feature map and the weighted feature map along the feature channel direction. A region proposal network module, configured to generate proposed boxes of interest, including: Feature channel transformation unit, which changes the number of features through a convolutional layer with a convolution kernel of 3×3 and 512 channels; Anchor box classification unit, which outputs the foreground and background probabilities of the anchor boxes through a convolutional layer with a convolution kernel of 1×1 and 18 channels; Anchor box localization unit, which outputs the coordinate offset compensation value of the anchor boxes through a convolutional layer with a convolution kernel of 1×1 and 36 channels; First position supervision unit, which is used to use the positive anchor box position offset parameter as the first position supervision information; First position loss function unit, which is used to train the region proposal network according to the positive anchor box position offset parameter and the predicted proposal box position; First category supervision unit, which is used to use the positive and negative anchor boxes as the first category supervision information; First category loss function unit, which is used to train the region proposal network according to the positive and negative anchor boxes and the predicted proposal box category parameters; Proposal box screening module, which is used to screen the proposal boxes twice, including three units: Coordinate offset compensation unit, which is used to apply the coordinate offset compensation value predicted by the region proposal network module to all corresponding anchor boxes; Pre-screening unit, which is used to sort the foreground probabilities of all anchor boxes predicted by the region proposal network module from high to low, and select the top 12,000 anchor boxes as proposal boxes; Screening unit, which is used to perform non-maximum suppression on the remaining proposal boxes to obtain 300 final proposal boxes. Proposal box pooling module, which is used to perform fixed-size conversion on the proposal boxes, including: Region division unit, which divides each input proposal box into a fixed number of sub-regions; Region pooling unit, which takes the maximum value of each sub-region to obtain a feature representation of a fixed size. Defect classification module, which is used to judge the defect category according to the proposal boxes provided by the region proposal network, including: Second position supervision unit, which is used to use the real bounding box position parameter as the second position supervision information; Second position loss function unit, which is used to train the classification and localization network according to the real bounding box position parameter and the predicted bounding box position parameter; Defect localization module, which is used to finally determine the defect position according to the proposal boxes provided by the region proposal network, including: Second category supervision unit, which is used to use the real bounding box category number as the second category supervision information; Second category loss function unit, which is used to train the classification and localization network according to the real bounding box category parameter and the predicted bounding box category parameter. Defect prediction module, which is used to pass the PCB surface assembly image to be detected through the defect perception network to obtain a PCB surface assembly image marked with the defect position and category.

9. A computer device, comprising at least one processor; and at least one memory communicatively connected to the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to claims 1 to 7 by calling the program instructions.

Images