A PCBA defect detection method based on multi-dimensional information fusion

Abstract

The invention discloses a PCBA defect detection method based on multi-dimensional information fusion, comprising the following steps: step 1: obtaining a PCBA visible light image and a depth image to form a data set; step 2: constructing a lightweight defect detection network model of multi-dimensional fusion learning, wherein the model is based on Yolov5m as a basic framework, and comprises a visible light feature extraction module, a depth feature extraction module, a multi-dimensional fusion module, a Neck module and a Head module; step 3: using the data set to train the lightweight defect detection network model of multi-dimensional fusion learning to obtain a trained model; inputting the required detection information into the trained model to obtain the position, category and confidence information of the defect; the invention combines the depth image with the visible light image, introduces the component height information, and improves the detection accuracy of defects related to 3D space information; the model size is relatively lightweight, and the detection accuracy is higher while ensuring the running speed.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a PCBA defect detection method based on multi-dimensional information fusion. Background Art

[0002] PCBA (Printed Circuit Board + Assembly) is a PCB with electronic components installed. During the production process, PCBA will produce component-related defects such as poor size, component offset, component missing, etc. With the accelerated advancement of industrialization, the demand for PCBA production efficiency and quality assurance is increasing day by day. The traditional manual visual defect inspection method is not suitable for PCBA defect detection in large-scale industrial production.

[0003] Existing surface defect detection methods based on deep learning have high requirements for the number of defect samples and the performance of the host computer, while PCBA generally lacks sufficient defect samples. For example, using traditional machine learning or deep learning methods based on neural network architecture for visible light images, training from a large number of defect data sets to learn defect features, and obtaining the location and category of defects in stages or directly, is particularly effective for tasks where defect types are diverse and difficult to clearly define or where product appearance may vary significantly. However, this method cannot accurately identify certain types of defects that occur in three-dimensional space, such as lifted pins or components; it is not combined with PCBA design files such as GERBER files for detection, resulting in false detection of some defects.

[0004] For example, the pattern matching method is used to compare the 2D data of visible light images with the stored reference images to find significant deviations, or feature-based methods are used to identify and quantify specific features in the image, such as edges, corners, or features of specific colors or textures, and the identified features are compared with predefined features to obtain the location and category of some defects related to the plane space information and visible light information, and 3D data such as depth maps and point clouds are used to extract the height of each component and use traditional machine vision methods to obtain the location and category of defects related to height. This method requires the processing of 2D and 3D data separately, which requires higher computing power costs and is time-consuming. Summary of the invention

[0005] The present invention aims at solving the problems in the prior art and provides a PCBA defect detection method based on multi-dimensional information fusion.

[0006] The technical solution adopted by the present invention is:

[0007] A PCBA defect detection method based on multi-dimensional information fusion includes the following steps:

[0008] Step 1: Obtain PCBA visible light image and depth image to form a data set;

[0009] Step 2: Construct a lightweight defect detection network model of multi-dimensional fusion learning. The model is based on Yolov5m framework, including a visible light feature extraction module for extracting visible light image features, a deep feature extraction module for extracting deep image features, a multi-dimensional fusion module for fusing visible light image features with deep image features, a Neck module and a Head module; the Neck module is used to further fuse the fused feature information output by the multi-dimensional fusion module; the Head module performs position masking and non-maximum suppression on the feature map to obtain the position, category and confidence information of the defect;

[0010] Step 3: Use the data set to train the lightweight defect detection network model of multi-dimensional fusion learning to obtain a trained model; input the required detection information into the trained model to obtain the defect location, category and confidence information.

[0011] Furthermore, the visible light feature extraction module in step 2 includes a CBS module and four α modules, namely α1, α2, α3 and α4; wherein the CBS module includes a convolutional layer, a normalization layer and an activation function; and the α module includes a CBS module and a C3 module.

[0012] Furthermore, the deep feature extraction module includes a CBS module and four β modules, namely β1, β2, β3 and β4; wherein the CBS module includes a convolutional layer, a normalization layer and an activation function; and the β module includes a CBS module and a C3 module.

[0013] Furthermore, the multidimensional fusion module includes three TB modules and two CBS modules arranged in sequence; the three TB modules are TB-1, TB-2, and TB-3; TB-1 fuses the feature matrix obtained by processing α2 and β2 to obtain M1, and then inputs it into the TB-2 module after passing through the CBS module; the TB-2 module fuses the TB-1 fusion information and the feature matrix obtained by processing α3 and β3 to obtain M2, and then inputs it into the TB-3 module after passing through the CBS module; the TB-3 module fuses the TB-2 fusion information and the feature matrix obtained by processing α4 and β4 to obtain M3, and then M1, M2, and M3 are input into the Neck module for processing.

[0014] Furthermore, the TB module includes a multi-head attention layer, a first sum layer, a first normalization layer and an MLP layer, a second sum layer and a second normalization layer;

[0015] The TB module processing process is as follows:

[0016] M=Norm(MLP(N)+N)

[0017] M is the fusion information after processing by the TB module, which are M1, M2, and M3 respectively; N is the fusion information after processing by the multi-head attention layer, the first sum layer, and the first normalization layer, which are N1, N2, and N3 respectively; Norm is the normalization processing, and MLP represents the multi-layer perceptron MLP processing;

[0018] N=Norm(MulityHead(Q,K,V)+Q)

[0019] Among them: MulityHead is the multi-head attention layer processing, Q, K, V are the query vector, i.e. the flattened vector of the visible light feature in the spatial dimension, the key vector, i.e. the flattened vector of the deep feature in the spatial dimension, and the value vector; the value vector is Q in the TB-1 module and is the fusion information M after sampling in the previous layer in other TB modules. i , where i is 1 and 2.

[0020] Furthermore, the Head module in step 2 includes a convolutional layer, a Position Mask module, and a non-maximum suppression layer processed by non-maximum suppression;

[0021] The convolution layer includes three convolution layers that process the output of the multi-dimensional fusion module to obtain feature maps;

[0022] The Position Mask module obtains the position mask and performs feature map adjustment.

[0023] Furthermore, the Position Mask module processing process is as follows:

[0024] Acquiring position information of electronic components to form a position mask matrix;

[0025] A confidence information matrix is ​​extracted from the detection results including position information, confidence information and category information;

[0026] The confidence information matrix and the position mask matrix are calculated by the Hamada product to obtain a new confidence matrix;

[0027] The calculated confidence information matrix replaces the confidence information matrix.

[0028] Furthermore, the detection method also includes the following steps: evaluating the model, and the evaluation indicators are precision, recall, average accuracy AP, average accuracy mAP and F1_Score.

[0029] Furthermore, the calculation process of the evaluation index is as follows:

[0030]

[0031] Among them: TP is the number of positive samples that are judged as positive samples, FP is the number of negative samples that are judged as positive samples, FN is the number of positive samples that are judged as negative samples, p(r) is the PR curve composed of precision and recall, Recall is the Precision value when it is equal to r, AP is the area enclosed by the function image, mAP is the AP value calculated for each defect category and then averaged, and n is the number of curve categories.

[0032] Furthermore, in step 1, Labeling is used to annotate the visible light images and depth images in the data set, generate a label file, and then divide the data set into a training set, a validation set, and a test set.

[0033] The beneficial effects of the present invention are:

[0034] (1) The present invention combines the depth image with the visible light image and introduces the component height information, thereby improving the detection accuracy of defects related to 3D spatial information;

[0035] (2) The present invention adopts a multi-dimensional feature fusion strategy to deeply fuse the deep image features with the visible light image features, thereby reducing the computing power cost and improving the detection speed;

[0036] (3) The model of the present invention is relatively lightweight, and has higher detection accuracy while ensuring running speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process of the present invention.

[0038] Figure 2 It is a structural diagram of the lightweight defect detection network model of multi-dimensional fusion learning in the present invention.

[0039] Figure 3 It is the Neck and Head structure in the structure of the lightweight defect detection network model of multi-dimensional fusion learning in the present invention.

[0040] Figure 4 for Figure 2 and Figure 3 Schematic diagram of the sub-module structure in, A is a schematic diagram of the sub-module structure, and B is a schematic diagram of the C3 structure.

[0041] Figure 5 This is the flowchart of the Position Mask module in the figure.

[0042] Figure 6 This is a schematic diagram of defect types in data set in Example 1 of the present invention.

[0043] Figure 7 It is a schematic diagram of the test results of Example 1 of the present invention and a comparative example.

[0044] Figure 8 This is a schematic diagram of the experimental detection effect of Example 1 of the present invention.

[0045] Fig. 9 It is the PR curve diagram of the lightweight defect detection network model of multi-dimensional fusion learning in the present invention.

[0046] Fig.10 It is the F1_Score curve graph of the lightweight defect detection network model of multi-dimensional fusion learning in the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, a PCBA defect detection method based on multi-dimensional information fusion includes the following steps:

[0049] Step 1: Obtain PCBA visible light image and depth image to form a data set;

[0050] Labeling is used to annotate the visible light images and depth images of the dataset, generate label files in PASCAL VOC format, and divide them into training set, validation set and test set in a ratio of 6:2:2.

[0051] Step 2: Construct a lightweight defect detection network model with multi-dimensional fusion learning, such as Figure 2 As shown in the figure, the model is based on Yolov5m as the framework, including a visible light feature extraction module for extracting visible light image features, a deep feature extraction module for extracting deep image features, a multidimensional fusion module, a Neck module and a Head module for fusing visible light image features and deep image features; the Neck module is used to further fuse the fused feature information output by the multidimensional fusion module; the Head module performs position masking and non-maximum suppression on the feature map to obtain the position, category and confidence information of the defect;

[0052] The visible light feature extraction module includes a CBS module and four α modules, namely α1, α2, α3 and α4; the CBS module includes a convolution layer, a normalization layer and an activation function; the α module includes a CBS module and a C3 module. The features of the visible light image are extracted, so that the semantic information of different scale contexts can be extracted during the feature fusion process.

[0053] The deep feature extraction module includes a CBS module and four β modules, namely β1, β2, β3 and β4; the CBS module includes a convolution layer, a normalization layer and an activation function; the β module includes a CBS module and a C3 module. The features of the deep image are extracted so that the semantic information of different scale contexts can be extracted during the feature fusion process.

[0054] The multidimensional fusion module includes three TB modules and two CBS modules arranged in sequence; the three TB modules are TB-1, TB-2, and TB-3; TB-1 fuses the feature matrix processed by α2 and β2 to obtain M1, which is then input into the TB-2 module after passing through the CBS module; the TB-2 module fuses the TB-1 fusion information and the feature matrix processed by α3 and β3 to obtain M2, which is then input into the TB-3 module after passing through the CBS module; the TB-3 module fuses the TB-2 fusion information and the feature matrix processed by α4 and β4 to obtain M3, and then M1, M2, and M3 are input into the Neck module for processing.

[0055] The TB module includes a multi-head attention layer, a first sum layer, a first normalization layer and an MLP layer, a second sum layer and a second normalization layer;

[0056] The TB module processing process is as follows:

[0057] M=Norm(MLP(N)+N) (1)

[0058] M is the fusion information after processing by the TB module, which are M1, M2, and M3 respectively; N is the fusion information after processing by the multi-head attention layer, the first sum layer, and the first normalization layer, which are N1, N2, and N3 respectively; Norm is the normalization processing, and MLP represents the multi-layer perceptron MLP processing;

[0059] N=Norm(MulityHead(Q,K,V)+Q) (2)

[0060] Among them: MulityHead is the multi-head attention layer processing, Q, K, V are the query vector, i.e. the flattened vector of the visible light feature in the spatial dimension, the key vector, i.e. the flattened vector of the deep feature in the spatial dimension, and the value vector; the value vector is Q in the TB-1 module and is the fusion information M after sampling in the previous layer in other TB modules. i , where i is 1 and 2.

[0061] in:

[0062] MulityHead(Q,K,V)=Concat(head1,K,head h )W O (3)

[0063] Among them: h is the number of heads, that is, the number of divisions of the original feature matrix; head i represents the output of the i-th head, W O is the output transformation matrix, Concat is the connection, and the output head of each head i It can be expressed as:

[0064] head i =Attention(QW i Q ,KW i K ,VW i V ) (4)

[0065] Where: QW i Q , KW i K 、VW i V are the query, key, and value transformation matrices of the i-th head respectively. Attention is the attention calculation function. Generally, the self-attention mechanism is used to calculate the attention. The self-attention mechanism is expressed as:

[0066]

[0067] Among them, d k is the key vector dimension, softmax is the normalization function, the weight of each key vector is calculated, and then the weight is multiplied by the value vector for weighted summation to obtain the attention output.

[0068] The multi-dimensional fusion module deeply fuses the feature information of visible light images and depth images at different scales to obtain fused feature information at three different scales.

[0069] The fused feature information of three different scales is input into the feature pyramid network Neck for multi-scale feature fusion and extraction, and the fused multi-scale features are input into the target detection head Head. The position information of defective components is screened and non-maximum suppression is performed through position masks, and finally the position, category and confidence information of the defects are output.

[0070] The structures of Neck module and Head module are as follows Figure 3As shown in the figure, in the Neck part, the feature pyramid is used to fuse the fusion feature information of three different scales together through upsampling, downsampling and concatenation operations. Among them, the operation is to concatenate two feature maps in the channel direction. The top-down part mainly realizes the fusion of features at different levels through upsampling and fusion of coarser-grained feature maps. The bottom-up part solves the problem of multi-scale object detection by using a convolution layer to fuse feature maps from different levels.

[0071] In the Head part, we first output three feature maps of different scales O1, O2, and O3 that have passed through three convolution layers with a convolution kernel size of 1×1. Then, we generate a position mask based on the gerber file and build a Position Mask module. The structure is as follows: Figure 5 , obtain the position mask and adjust the feature map. Finally, the final output is obtained through non-maximum suppression.

[0072] The Position Mask module processing process is as follows: Figure 5 As shown:

[0073] Parse the GERBER file and obtain the position information of the electronic components to form a position mask matrix M with the same size as the original image; the pixels with component positions in the matrix are set to 1, and the pixels without component positions are set to η, η∈[0,1]. After experiments, the η value is finally set to 0.5.

[0074] A confidence information matrix is ​​extracted from the detection results including position information, confidence information and category information;

[0075] The confidence information matrix and the position mask matrix are calculated by the Hamada product to obtain a new confidence matrix;

[0076] The calculated confidence information matrix is ​​replaced by the confidence information matrix to obtain the final output.

[0077] Step 3: Use the data set to train the lightweight defect detection network model of multi-dimensional fusion learning to obtain a trained model; input the required detection information into the trained model to obtain the defect location, category and confidence information.

[0078] The training process of the lightweight defect detection network model of multi-dimensional fusion learning is as follows:

[0079] The size of the input image is set to 640×640, the initial learning rate is 0.01, the batch_size is selected to 2, and SGD is used as the optimizer. The overall training process is divided into two rounds. The first round sets the epoch to 300, introduces the pre-trained model to train the visible light feature extraction module, multi-scale fusion module and output head, and saves the weights. In the second round of training, the weights obtained from the first type of training are imported, and the visible light image and deep image feature extraction network training are frozen for 100 epochs, and then all model parameters are trained for another 200 epochs.

[0080] The detection method also includes the following steps: evaluating the model, and the evaluation indicators are precision, recall, average accuracy AP, average accuracy mAP and F1_Score, where F1_Score is the harmonic mean of Precision and Recall, with a maximum value of 1 and a minimum value of 0.

[0081] The calculation process of the evaluation index is as follows:

[0082]

[0083]

[0084] Wherein: TP is the number of positive samples that are judged as positive samples, FP is the number of negative samples that are judged as positive samples, FN is the number of positive samples that are judged as negative samples, p(r) is the PR curve composed of precision and recall, Recall is the Precision value when it is equal to r, AP is the area enclosed by the function image, mAP is the AP value calculated for each defect category and then averaged, and n is the number of curve categories.

[0085] Example 1

[0086] In order to illustrate the effect of the present invention, the above-mentioned method of the present invention is used for testing, and the defect data are as follows: Figure 6 As shown, a comparative example is set for comparison, and the difference between the comparative example and the above method is as shown in Table 1.

[0087] Table 1. Ablation experiment combination table

[0088]

[0089] “√” in the table indicates that the model contains this module structure.

[0090] The results of the tested ablation experimental parameters are shown in Table 2.

[0091] Table 2. Comparison of ablation experiments

[0092]

[0093] The experimental equipment and environment are

[0094] Intel(R)Xeon(R)Gold 6226R CPU@2.90GHz, memory 32GB, NVIDIA Tesla P100with 16GB video-memory, deep learning environment is Tensorflow 2.4.0with python 3.8.19.

[0095] As can be seen from Table 2, the introduction of the depth image makes the overall defect detection accuracy higher, especially for defects related to height. At the same time, the Position Mask module is used to process the results. The comparison results are as follows: Figure 7 As shown in the figure, it can be seen that the number of positions left blank in the original design that are judged as defects is greatly reduced, and the accuracy of the most accurate result is also greatly improved.

[0096] The results of the detection by the method of the present invention are as follows Figure 8 As shown in the figure, the PR curve of the detection result obtained using the test set is as follows Fig. 9 As shown. The F1_Score curve is as follows Fig.10 shown.

[0097] The results show that the use of depth maps for PCBA defect detection improves the detection accuracy of defects related to the depth direction of components. The multi-dimensional fusion module is used to fuse depth information and visible light information, which improves the fusion results of visible light information and depth information. The spatial mask generated according to the GERBER design file information is used to process the network output, which reduces the impact of non-design areas on the final defect detection results.

Claims

1. A PCBA defect detection method based on multi-dimensional information fusion, characterized in that: The following steps are involved: Step 1: Obtain PCBA visible light image and depth image to form a data set; Step 2: Construct a lightweight defect detection network model of multi-dimensional fusion learning. The model is based on Yolov5m framework, including a visible light feature extraction module for extracting visible light image features, a deep feature extraction module for extracting deep image features, a multi-dimensional fusion module for fusing visible light image features with deep image features, a Neck module and a Head module; the Neck module is used to further fuse the fusion feature information output by the multi-dimensional fusion module; the Head module performs position masking and non-maximum suppression on the feature map to obtain the position, category and confidence information of the defect; the multi-dimensional fusion module includes three TB modules and two CBS modules set in sequence; The three TB modules are TB-1, TB-2, and TB-3; TB-1 fuses the feature matrix processed by α2 and β2 to obtain M1, which is then passed through the CBS module and input into the TB-2 module; The TB-2 module fuses the TB-1 fusion information and the feature matrix processed by α3 and β3 to obtain M2, which is then input into the TB-3 module after passing through the CBS module; The TB-3 module fuses the TB-2 fusion information and the feature matrix processed by α4 and β4 to obtain M3, and then M1, M2 and M3 are input into the Neck module for processing; The TB module includes a multi-head attention layer, a first sum layer, a first normalization layer and an MLP layer, a second sum layer and a second normalization layer; The TB module processing process is as follows: M=Norm(MLP(N)+N) M is the fusion information after processing by the TB module, which are M1, M2, and M3 respectively; N is the fusion information after processing by the multi-head attention layer, the first sum layer, and the first normalization layer, which are N1, N2, and N3 respectively; Norm is the normalization processing, and MLP represents the multi-layer perceptron MLP processing; N=Norm(MulityHead(Q,K,V)+Q) Among them: MulityHead is the multi-head attention layer processing, Q, K, V are the query vector, i.e. the flattened vector of the visible light feature in the spatial dimension, the key vector, i.e. the flattened vector of the deep feature in the spatial dimension, and the value vector; the value vector is Q in the TB-1 module and is the fusion information M after sampling in the previous layer in other TB modules. i , where i is 1 and 2; Step 3: Use the data set to train the lightweight defect detection network model of multi-dimensional fusion learning to obtain a trained model; input the required detection information into the trained model to obtain the defect location, category and confidence information.

2. A PCBA defect detection method based on multidimensional information fusion according to claim 1, characterized in that: The visible light feature extraction module in step 2 includes a CBS module and four α modules, namely α1, α2, α3 and α4; the CBS module includes a convolution layer, a normalization layer and an activation function; the α module includes a CBS module and a C3 module.

3. A PCBA defect detection method based on multi-dimensional information fusion according to claim 2, characterized in that: The deep feature extraction module includes a CBS module and four β modules, namely β1, β2, β3 and β4; the CBS module includes a convolution layer, a normalization layer and an activation function; the β module includes a CBS module and a C3 module.

4. A PCBA defect detection method based on multi-dimensional information fusion according to claim 1, characterized in that: The Head module in step 2 includes a convolutional layer, a Position Mask module, and a non-maximum suppression layer processed by non-maximum suppression; The convolution layer includes three convolution layers that process the output of the multi-dimensional fusion module to obtain feature maps; The Position Mask module obtains the position mask and makes feature map adjustments.

5. A PCBA defect detection method based on multi-dimensional information fusion according to claim 4, characterized in that: The Position Mask module processing process is as follows: Acquiring position information of electronic components to form a position mask matrix; A confidence information matrix is ​​extracted from the detection results including position information, confidence information and category information; The confidence information matrix and the position mask matrix are calculated by the Hamada product to obtain a new confidence matrix; The calculated confidence information matrix replaces the confidence information matrix.

6. The PCBA defect detection method based on multi-dimensional information fusion according to claim 1, characterized in that: The detection method also includes the following steps: evaluating the model, and the evaluation indicators are precision, recall, average accuracy AP, average accuracy mAP and F1_Score.

7. A PCBA defect detection method based on multi-dimensional information fusion according to claim 6, characterized in that: The calculation process of the evaluation index is as follows: Wherein: TP is the number of positive samples that are judged as positive samples, FP is the number of negative samples that are judged as positive samples, FN is the number of positive samples that are judged as negative samples, p(r) is the PR curve composed of precision and recall, Recall is the Precision value when it is equal to r, AP is the area m enclosed by the function graph, AP is the average of the AP values ​​calculated for each defect category, and n is the number of curve categories.

8. The PCBA defect detection method based on multi-dimensional information fusion according to claim 1, characterized in that: In step 1, Labeling is used to annotate the visible light images and depth images in the dataset, generate a label file, and then divide the dataset into a training set, a validation set, and a test set.

Images