PCB (printed circuit board) defect detection method and equipment based on 2D (two-dimensional) and 3D (three-dimensional) image visual combination

Abstract

The invention belongs to the technical field of visual inspection, and particularly relates to a PCB defect detection method and device based on 2D and 3D image visual combination, and the method comprises the steps: collecting the 2D information and 3D information of a PCB, and carrying out the defect marking; the 2D information subjected to defect labeling serves as a training set to be input into a pre-constructed yolov10 target detection network, and a yolov10 model is obtained after training is conducted through a training strategy of double label distribution and consistent matching measurement; converting the 3D information subjected to defect marking into voxel features and coordinates by using a pre-constructed seond target detection network, performing feature extraction on the voxel features and coordinates by using a sparse convolutional neural network, and classifying object directions to obtain a trained seond model; and determining the defect type of the target PCB according to the yov10 model and the seond model. By adopting the deep learning algorithm and the 3D vision technology, the detection accuracy and efficiency can be greatly improved, the maintenance cost is reduced, and the detection robustness and stability are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual inspection, and in particular relates to a PCB board defect inspection method and equipment based on the combination of 2D and 3D image vision. Background Art

[0002] With the rapid development of science and technology, electronic products play an increasingly important role in people's lives and work. With the continuous upgrading of electronic products, the demand for PCB (printed circuit board), one of the core components of electronic products, is also growing. The quality of PCB is directly related to the performance and reliability of electronic products. Therefore, it is particularly important to conduct efficient and accurate quality inspection of PCB boards.

[0003] Traditional PCB board inspection methods mainly rely on manual visual inspection and automatic optical inspection (AOI) equipment based on machine vision. However, these methods have many limitations. Manual visual inspection is not only inefficient, but also easily affected by human factors, making it difficult to ensure the accuracy and consistency of the inspection results. Although traditional AOI equipment has improved inspection efficiency, its core technology mostly relies on imported equipment, has high maintenance costs, and has strict requirements on external factors such as the environment and light source, resulting in poor stability and robustness of inspection. Summary of the invention

[0004] The purpose of the present invention is to provide a PCB board defect detection method and device based on the combination of 2D and 3D image vision to solve the problems raised in the background technology.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] In the first aspect, the present invention proposes a PCB board defect detection method based on the combination of 2D and 3D image vision, the method comprising:

[0007] Collect 2D and 3D information of PCB boards and mark defects;

[0008] The 2D information of the defect annotation is used as a training set to input into the pre-built yolov10 target detection network, and the yolov10 model is obtained after training using the training strategy of dual label assignment and consistent matching metric;

[0009] The pre-built second object detection network is used to convert the 3D information of the defect annotation into voxel features and coordinates, and a sparse convolutional neural network is used to extract features from the voxel features and coordinates, and to classify the object direction, to obtain a trained second model;

[0010] The yolov10 model is used to detect the 2D information of the target PCB board to obtain the first feature, the second model is used to detect the 3D information of the target PCB board to obtain the second feature, and the defect type of the target PCB board is determined based on the first feature and the second feature.

[0011] Furthermore, the yolov10 target detection network includes the following architecture:

[0012] Lightweight classification head: includes two 3×3 depth-wise separable convolutions and one 1×1 convolution to reduce computational overhead;

[0013] Spatial-channel decoupled downsampling module: It is used to optimize the overall computational cost by separating the spatial downsampling operation and the channel adjustment process;

[0014] Rank-based block design: Customize network blocks based on the intrinsic rank characteristics of each stage of the network to reduce structural redundancy and improve overall efficiency;

[0015] Large kernel convolution and partial self-attention module: It integrates large kernel convolution to expand the receptive field, and combines it with the partial self-attention mechanism. It applies multi-head self-attention to only some channels of the feature map, and then fuses the processed features with the remaining channel features.

[0016] Furthermore, the training strategy of the dual label assignment and consistent matching metric includes:

[0017] Assign one or more positive samples to each instance through dual label assignment;

[0018] A consistent matching metric is used to ensure the accuracy and effectiveness of the assignment of positive and negative samples.

[0019] Furthermore, the use of a sparse convolutional neural network to extract features from the voxel features and coordinates and to classify the object direction includes:

[0020] Applying two voxel feature encoding layers and a linear layer to encode the voxel features and coordinates;

[0021] Apply a sparse 3D convolutional network to perform convolution operations on the encoded features to obtain dense output features;

[0022] Mapping the dense output features to sparse output features through a pre-constructed input-output index rule matrix;

[0023] In the RPN layer, object classification and position regression branches, as well as a branch for object direction classification, are introduced.

[0024] Furthermore, the applying of the sparse 3D convolutional network to perform a convolution operation on the encoded features comprises the following steps:

[0025] The sparse input features are obtained through the gather operation to obtain dense gather features;

[0026] Use GEMM to perform convolution operations on dense gather features to obtain dense output features.

[0027] Furthermore, after extracting the voxel features and coordinates and classifying the object directions using the sparse convolutional neural network, the method further includes: optimizing the model parameters by setting a loss function and data enhancement, as follows:

[0028] Define loss functions, including classification loss, position and dimension regression loss, angle loss and direction classification loss; the classification loss adopts the form of focal loss, and the angle loss adopts the sinusoidal error loss of angle regression;

[0029] Data enhancement, by sampling and intercepting the output features after defect annotation, to generate a database or data set for model training.

[0030] In a second aspect, the present invention provides a PCB board quality inspection device, the device comprising:

[0031] Information collection module, used to collect 2D and 3D information of PCB boards and mark defects;

[0032] The first training module is used to input the 2D information of the defect annotation as a training set into the pre-built yolov10 target detection network, and obtain the yolov10 model after training using the training strategy of dual label assignment and consistent matching metric;

[0033] The second training module is used to convert the 3D information of the defect annotation into voxel features and coordinates using the pre-built second object detection network, extract the voxel features and coordinates using a sparse convolutional neural network, and classify the object direction, and then optimize the model parameters by setting a loss function and data enhancement to obtain a trained second model;

[0034] The identification module is used to detect the 2D information of the target PCB board using the yolov10 model to obtain the first feature, detect the 3D information of the target PCB board using the second model to obtain the second feature, and determine the defect type of the target PCB board based on the first feature and the second feature.

[0035] The beneficial effects of the present invention are:

[0036] 1. The detection method proposed in the present invention can make full use of the two-dimensional image and three-dimensional structure information of the PCB board to achieve all-round and high-precision detection of PCB board defects. By adopting deep learning algorithms and 3D vision technology, the accuracy and efficiency of detection can be greatly improved, maintenance costs can be reduced, and the robustness and stability of detection can be improved.

[0037] 2. The present invention utilizes the detection characteristics of the combination of the yolov10 model and the second model to solve the problem that PCB detection requires the cooperation of multiple detection equipment. It greatly improves the detection accuracy and efficiency for problems such as multiple electronic components being adjacent and detection accuracy being easily lost, and is fully ahead of the deep learning models configured in related detection equipment on the market. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of the PCB board defect detection method based on the combination of 2D and 3D image vision in the present invention.

[0039] Figure 2 The figure is a flowchart of the implementation of the PCB board defect detection method based on the combination of 2D and 3D image vision in the present invention.

[0040] Figure 3 This is an example diagram of defect detection on a PCB board using the 2D defect detection solution proposed in the present invention.

[0041] Figure 4 This is an example diagram of using AOI light source inspection equipment in the prior art to perform defect inspection on a PCB board.

[0042] Figure 5 This is a schematic diagram of the point cloud data of the chip pin part collected when the second model is used for 3D detection proposed by the present invention.

[0043] Figure 6 It is a comparative schematic diagram of the PCB board defect detection method based on the combination of 2D and 3D image vision in the present invention using data enhancement and not using data enhancement. DETAILED DESCRIPTION

[0044] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0045] Example 1

[0046] like Figure 1-4As shown, this embodiment proposes a PCB board defect detection method based on the combination of 2D and 3D image vision, and the method includes the following steps:

[0047] S1. Collect 2D information (acquired by 2D camera) and 3D information (acquired by 3D scanning device) of PCB board and mark defects.

[0048] S2. The 2D information after defect annotation is input into the pre-built yolov10 target detection network as a training set, and the yolov10 model is obtained after training using the training strategy of dual label assignment and consistent matching metric.

[0049] This embodiment proposes a non-NMS training strategy, which achieves high efficiency and competitive performance through dual label assignment and consistent matching metrics. Specifically, dual label assignment is used to assign one or more positive samples to each instance; consistent matching metrics are used to ensure the accuracy and effectiveness of positive and negative sample assignment.

[0050] Further preferably, the yolov10 target detection network includes the following architecture:

[0051] Lightweight classification head: Reduces computational overhead without significantly affecting performance. (In the YOLO series, the classification head and regression head usually share the same architecture, but the classification task has a higher computational overhead. A lightweight architecture is used for the classification head, including two 3×3 depthwise separable convolutions and one 1×1 convolution to reduce computational overhead.

[0052] Spatial-channel decoupled downsampling module: It is used to optimize the overall computational cost by separating the spatial downsampling operation and the channel adjustment process;

[0053] Rank-based block design: Customized network blocks are constructed based on the intrinsic rank characteristics of each stage of the network to reduce structural redundancy and improve overall efficiency. (Traditional downsampling methods perform spatial and channel conversions at the same time, which is computationally expensive. YOLOv10 first uses pointwise convolution to adjust the channel dimension, and then uses depthwise convolution for spatial downsampling. This can maximize information retention while reducing computational costs.)

[0054] Large kernel convolution and partial self-attention module (PSA): It integrates large kernel convolution to expand the receptive field and combines it with the partial self-attention mechanism. It applies multi-head self-attention to only some channels of the feature map and then fuses the processed features with the remaining channel features.

[0055] Self-attention is widely used in visual tasks due to its excellent global modeling ability, but its computational complexity and memory usage are high. This application introduces the PSA module, which enhances the global modeling ability and reduces the computational complexity by dividing the features into two parts by channel, applying multi-head self-attention (MHSA) to only one part, and then fusing them.

[0056] Combination Figure 3 and Figure 4 The detection example is verified by actual training tests. The 2D defect detection of PCB proposed in this embodiment is ahead of the current AOI light source detection equipment in many aspects such as detection speed, accuracy, light source requirements, and cost.

[0057] In the present embodiment, the 2D information of the target PCB board is detected and the first feature is obtained by using the yolov10 model. The yolov10 model adopts end-to-end detection technology. Compared with the existing yolov5-9 and other models, the training model converges faster, and the reasoning is more efficient without the nms part, and the speed is also faster. Compared with the previous yolo series, there is a significant improvement in system overhead, computing resources, and reasoning speed. For the detection of the PCB part, it may be necessary to adapt to the limitations of various aspects such as the production beat. The detection model has a faster reasoning speed while ensuring accuracy, and can better adapt to the faster production beat, thereby improving production efficiency. The 3D information of the target PCB board is detected and the second feature is obtained by the second model. Although the Second model is slower than pointnet (deep learning network for directly processing point cloud data) and pointpillar (deep learning model for three-dimensional target detection), it adopts double-stage detection, and the accuracy is higher. The time reduced in the yolo series can be balanced to the 3D detection part, so that in the case of comprehensive detection time, yolov10+second is the best match in terms of accuracy and speed.

[0058] S3. Use the pre-built second object detection network to convert the 3D information after defect annotation into voxel features and coordinates, use a sparse convolutional neural network to extract the voxel features and coordinates, and classify the object direction to obtain the trained second model.

[0059] Further preferably, a sparse convolutional neural network is used to extract voxel features and coordinates and classify object directions, including:

[0060] Apply two voxel feature encoding layers and a linear layer to encode voxel features and coordinates;

[0061] Apply a sparse 3D convolutional network to perform convolution operations on the encoded features to obtain dense output features;

[0062] Through the pre-built input-output index rule matrix, dense output features are mapped to sparse output features; this input-output index rule matrix is ​​the key to sparse convolution, and can also provide high detection confidence for sparse point cloud detection components produced in some areas of PCB scanning;

[0063] In the RPN layer, object classification and position regression branches, as well as a branch for object direction classification, are introduced.

[0064] In this embodiment, applying a sparse 3D convolutional network to perform a convolution operation on the encoded features includes the following steps:

[0065] The sparse input features are obtained through the gather operation to obtain dense gather features;

[0066] Use GEMM to perform convolution operations on dense gather features to obtain dense output features.

[0067] Further preferably, after extracting voxel features and coordinates using a sparse convolutional neural network and classifying the object direction, the method further includes: optimizing model parameters by setting a loss function and data enhancement, as follows:

[0068] Define loss functions, including classification loss, position and dimension regression loss, angle loss and direction classification loss; the classification loss adopts the form of focal loss, and the angle loss adopts the sinusoidal error loss of angle regression;

[0069] Data enhancement, by sampling and intercepting the output features after defect annotation, to generate a database or data set for model training. This method speeds up the convergence speed of the network and improves the network accuracy. The results are as follows Figure 6 (AP in the figure is the average precision, which measures the quality of the learned model in each category).

[0070] In this embodiment, the Loss loss function is composed of Lcls classification loss, Lreg-other position and dimension regression loss, Lreg-θ new angle loss, Ldir direction classification loss and parts, β1=1.0, β2=2.0, β3=3.0 are constant coefficients of the loss formula, and a relatively small β3 value is used to avoid the situation where the network has difficulty in identifying the direction of the object. The total loss function is as follows:

[0071] Ltotal=β1Lcls+β2(Lreg-θ+Lreg-other)+β3Ldir

[0072] In order to solve the imbalance between foreground and background classes in the sample, focal loss (loss function) is introduced. The classification loss has the following form:

[0073] FL(pt)=-αt(1-pt)γlog(pt)

[0074] Where pt is the estimated probability of the model, α and γ are the parameters of the focal loss. α = 0.25 and γ = 2 are used during training.

[0075] In order to solve the problem that the direction regression loss is too large and is not conducive to model training, the sine error loss of angle regression is introduced:

[0076] Lθ=SmoothL1(sin(θp-θt))

[0077] θp is the predicted direction angle, and θt is the actual direction angle. When the difference between θp and θt is π, the loss tends to 0, which is more conducive to model training.

[0078] S4. Use the yolov10 model to detect the 2D information of the target PCB board to obtain the first feature, use the second model to detect the 3D information of the target PCB board to obtain the second feature, and determine the defect type of the target PCB board based on the first feature and the second feature.

[0079] In the above step S4, PCB defect detection is finally carried out through yolov10 and second-based models, which is realized from multiple detection directions of 2D+3D, and has many advantages such as low cost, high speed and easy maintenance.

[0080] It can be understood that this embodiment combines 2D and 3D image vision technology to propose a new PCB board defect detection method, which can make full use of the 2D image and 3D structure information of the PCB board to achieve all-round and high-precision detection of PCB board defects. Combining 3D depth information and convolutional neural network, a multi-modal approach is adopted to achieve all-round quality inspection of PCB from 2D to 3D.

[0081] This embodiment also proposes a PCB board quality inspection device, which is used to implement the above-mentioned detection method, and the device includes:

[0082] Information collection module, used to collect 2D and 3D information of PCB boards and mark defects;

[0083] The first training module is used to input the 2D information of the defect annotation as a training set into the pre-built yolov10 target detection network, and obtain the yolov10 model after training using the training strategy of dual label assignment and consistent matching metric;

[0084] The second training module is used to convert the 3D information of the defect annotation into voxel features and coordinates using the pre-built second object detection network, extract the voxel features and coordinates using a sparse convolutional neural network, and classify the object direction. Then, the model parameters are optimized by setting the loss function and data enhancement to obtain the trained second model.

[0085] The identification module is used to detect the 2D information of the target PCB board using the yolov10 model to obtain the first feature, detect the 3D information of the target PCB board using the second model to obtain the second feature, and determine the defect type of the target PCB board based on the first feature and the second feature.

[0086] It should be noted here that each module in the above-mentioned quality inspection equipment corresponds to steps S1 to S4 in the implementation of the above-mentioned detection method, and the instances and application scenarios implemented by multiple modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0087] According to the above embodiments of the present invention, the present application combines 2D and 3D detection models to be applied in the PCB quality inspection process, which specifically includes the following specific implementation procedures:

[0088] 1.2D defect detection solution

[0089] 1.1 Data collection and annotation

[0090] Use a 2D camera to collect image data of the PCB board; use annotation tools to annotate the defects in the image and create a training data set.

[0091] 1.2YOLOv10 model selection

[0092] YOLOv10 is selected as the 2D defect detection model because it does not require NMS post-processing, has high efficiency, and excellent detection performance.

[0093] Prepare corresponding pre-trained models or train a model from scratch.

[0094] 1.3 Model training and validation

[0095] Use the labeled data set to train the YOLOv10 model and perform model verification to ensure that the model meets the requirements in terms of detection speed and accuracy.

[0096] 1.4 Model Deployment

[0097] Deploy the trained model to the above-mentioned quality inspection equipment for real-time defect detection.

[0098] 2.3D Detection Solution

[0099] 2.1 3D data collection and annotation

[0100] Use 3D scanning equipment to scan the chip pins and other areas of the PCB board to obtain point cloud data.

[0101] Annotate the point cloud data and mark the parts and locations that need to be inspected.

[0102] 2.2 second model selection

[0103] Second is selected as the 3D detection model because of its sparse convolution technology and direction estimation optimization, which is suitable for the detection of sparse point cloud data.

[0104] Prepare pre-trained models or perform model training.

[0105] 2.3 Model Training and Validation

[0106] Use the labeled point cloud data to train the second model.

[0107] Carry out model verification to ensure that the requirements in terms of detection speed, accuracy, etc. are met.

[0108] 2.4 Model Deployment

[0109] The trained second model is deployed to the above-mentioned quality inspection equipment to work together with the 2D detection model.

[0110] 3. Combining 2D and 3D detection

[0111] 3.1 Data Fusion

[0112] The 2D image data and 3D point cloud data are integrated to form more comprehensive and accurate PCB quality inspection data.

[0113] 3.2 Algorithm Design

[0114] The design algorithm combines 2D and 3D detection models to achieve more efficient and accurate PCB quality inspection. You can use the strategy of first locating the approximate area through the 2D detection model, and then perform precise detection through the 3D detection model.

[0115] 3.3 System Integration

[0116] Integrate the combined 2D and 3D inspection systems into PCB quality inspection equipment to achieve a fully automated quality inspection process.

[0117] 4. Later optimization and improvement

[0118] 4.1 Data Augmentation

[0119] Perform data augmentation on the training data set to improve the generalization ability of the model.

[0120] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0121] In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0123] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A PCB defect detection method based on the combination of 2D and 3D image vision, characterized in that: The method comprises: Collect 2D and 3D information of PCB boards and mark defects; The 2D information of the defect annotation is used as a training set to input into the pre-built yolov10 target detection network, and the yolov10 model is obtained after training using the training strategy of dual label assignment and consistent matching metric; The pre-built second object detection network is used to convert the 3D information of the defect annotation into voxel features and coordinates, and a sparse convolutional neural network is used to extract features from the voxel features and coordinates, and to classify the object direction, to obtain a trained second model; The yolov10 model is used to detect the 2D information of the target PCB board to obtain the first feature, the second model is used to detect the 3D information of the target PCB board to obtain the second feature, and the defect type of the target PCB board is determined based on the first feature and the second feature.

2. The PCB defect detection method based on the combination of 2D and 3D image vision according to claim 1 is characterized in that: The yolov10 target detection network includes the following architecture: Lightweight classification head: includes two 3×3 depth-wise separable convolutions and one 1×1 convolution to reduce computational overhead; Spatial-channel decoupled downsampling module: It is used to optimize the overall computational cost by separating the spatial downsampling operation and the channel adjustment process; Rank-based block design: Customize network blocks based on the intrinsic rank characteristics of each stage of the network to reduce structural redundancy and improve overall efficiency; Large kernel convolution and partial self-attention module: It integrates large kernel convolution to expand the receptive field, and combines it with the partial self-attention mechanism. It applies multi-head self-attention to only some channels of the feature map, and then fuses the processed features with the remaining channel features.

3. The PCB defect detection method based on the combination of 2D and 3D image vision according to claim 1 is characterized in that: The training strategy of the dual label assignment and consistent matching metric includes: Assign one or more positive samples to each instance through dual label assignment; A consistent matching metric is used to ensure the accuracy and effectiveness of the assignment of positive and negative samples.

4. The PCB defect detection method based on 2D and 3D image vision combination according to claim 1 is characterized in that: The method of extracting features of the voxel features and coordinates and classifying the object directions by using a sparse convolutional neural network includes: Applying two voxel feature encoding layers and a linear layer to encode the voxel features and coordinates; Apply a sparse 3D convolutional network to perform convolution operations on the encoded features to obtain dense output features; Mapping the dense output features to sparse output features through a pre-constructed input-output index rule matrix; In the RPN layer, object classification and position regression branches, as well as a branch for object direction classification, are introduced.

5. The PCB defect detection method based on the combination of 2D and 3D image vision according to claim 4 is characterized in that: The application of the sparse 3D convolutional network to perform a convolution operation on the encoded features comprises the following steps: The sparse input features are obtained through the gather operation to obtain dense gather features; Use GEMM to perform convolution operations on dense gather features to obtain dense output features.

6. The PCB defect detection method based on 2D and 3D image vision combination according to claim 4 is characterized in that: After extracting the voxel features and coordinates using a sparse convolutional neural network and classifying the object direction, the method further includes: optimizing the model parameters by setting a loss function and data enhancement, as follows: Define loss functions, including classification loss, position and dimension regression loss, angle loss and direction classification loss; the classification loss adopts the form of focal loss, and the angle loss adopts the sinusoidal error loss of angle regression; Data enhancement, by sampling and intercepting the output features after defect annotation, to generate a database or data set for model training.

7. A PCB board quality inspection device, characterized in that: The device comprises: Information collection module, used to collect 2D and 3D information of PCB boards and mark defects; The first training module is used to input the 2D information of the defect annotation as a training set into the pre-built yolov10 target detection network, and obtain the yolov10 model after training using the training strategy of dual label assignment and consistent matching metric; The second training module is used to convert the 3D information of the defect annotation into voxel features and coordinates using the pre-built second object detection network, extract the voxel features and coordinates using a sparse convolutional neural network, and classify the object direction, and then optimize the model parameters by setting a loss function and data enhancement to obtain a trained second model; The identification module is used to detect the 2D information of the target PCB board using the yolov10 model to obtain the first feature, detect the 3D information of the target PCB board using the second model to obtain the second feature, and determine the defect type of the target PCB board based on the first feature and the second feature.

Images