A PCB defect detection system and method based on NVIDIA Jetson TX2

Abstract

This invention discloses a PCB defect detection system and method based on NVIDIA Jetson TX2, belonging to the field of PCB defect detection technology. Addressing the problems of low accuracy and slow speed in traditional PCB defect detection methods, as well as the poor model detection performance and lack of deployment in current image processing, machine learning, and deep learning-based detection methods, this invention designs a PCB defect detection system and method based on NVIDIA Jetson TX2, effectively improving the detection accuracy and speed of PCB defects, and has broad application prospects.

Description

Technical fields:

[0001] This invention relates to the field of PCB defect detection technology, and specifically to a PCB defect detection system and method based on NVIDIA Jetson TX2. Background technology:

[0002] Printed circuit boards (PCBs) have become key components in many electronic products used in daily life, with wide applications in new energy, automobiles, computers, military, and industrial control. The PCB manufacturing process can produce a wide variety of defects in unpredictable locations. Defective PCBs can pose risks to the daily use of related electronic products, thus placing extremely high demands on the speed and accuracy of PCB defect detection in actual testing.

[0003] Early PCB defect detection methods relied on visual inspection by workers to determine the presence of defects. The inspectors' emotions, physical condition, and experience significantly impacted the detection results. This method suffered from low accuracy and slow speed, failing to meet practical production needs. Traditional PCB inspection methods are no longer sufficient for modern PCB inspection requirements. The rapid development of image processing, machine learning, and deep learning has brought new methods to PCB defect detection. These methods, with their advantages of non-contact inspection, high efficiency, and stable results, have broad application prospects in PCB defect detection. Summary of the Invention:

[0004] To overcome the problems of slow detection speed and low accuracy of traditional PCB defect detection methods, this invention provides a PCB defect detection system and method based on NVIDIA Jetson TX2, which can accurately and quickly detect common defects on the PCB surface.

[0005] The technical problem to be solved by this invention is achieved by the following technical solution:

[0006] The first objective of this invention is to provide a PCB defect detection system based on NVIDIA Jetson TX2, including a PCB image acquisition module, a defect detection module, and a human-computer interaction interface module arranged on an NVIDIA Jetson TX2 board.

[0007] The PCB image acquisition module uses an external camera connected to the NVIDIA Jetson TX2 to capture images of the PCB to be tested, and the acquired images are transmitted to the defect detection module for detection.

[0008] The defect detection module uses an improved YOLOv5 model deployed on NVIDIA Jetson TX2 to detect the PCB images transmitted by the PCB image acquisition module and outputs the detection results, which include the location and category information of each defect and the detection time for a single image.

[0009] The human-computer interaction interface module is used to summarize and analyze the detection results, displaying the total number of PCB defects detected, the number of each type of PCB defect, the total time spent in the detection process, and the PCB image after detection, including annotation information.

[0010] The second objective of this invention is to provide a PCB defect detection method based on NVIDIA Jetson TX2, comprising the following steps:

[0011] S1. The PCB image acquisition module acquires PCB images through an external camera, and performs cropping and filling processing on the images. The images are then adjusted to the size specified by the detection model, and the adjusted images are input into the defect detection module.

[0012] S2. Using the improved YOLOv5 model deployed on NVIDIA Jetson TX2, the acquired PCB images are sequentially processed through a convolutional network consisting of Focus, Conv, C3, SPP, and upsample to extract and fuse features, generating feature maps containing various semantic information of the images. Then, multi-scale detection is performed on the images through detection layers of sizes 40×40, 80×80, and 160×160.

[0013] S3. The detection results are output to the human-computer interaction interface. The detection information area displays the total number of PCB defects detected, the number of PCB defects of each type, and the total time taken for the detection process. The detection result area displays the PCB image after detection, including the annotation information.

[0014] In step S2, the improved YOLOv5 model adds a C3 module, a Conv module, and an upsample module sequentially after the second Concat module of the YOLOv5 feature fusion network. The output of the upsample module is stacked with the output of the first C3 module of the feature extraction layer in the channel dimension. The stacked feature map is then processed by the C3 and Conv modules for feature extraction. The extracted feature map is then input into the second C3 module of the YOLOv5 feature fusion network. The output of the fourth C3 module in the feature fusion network structure is expanded by upsampling and stacked with the output of the third C3 module in the feature fusion network. The output of the fifth C3 module in the feature fusion network is stacked with the output of the fourth C3 module through upsampling. The output of the sixth C3 module in the feature fusion network is stacked with the output of the fifth C3 module through upsampling. A detection layer is added to predict on a feature map of size 160×160 from the output of the third C3 module in the new feature fusion network. At the same time, the detection layer of YOLOv5 model that performs detection on a feature map of size 20×20 is removed. A hybrid attention mechanism module consisting of ECA attention and spatial attention mechanism in CBAM is added to each C3 module adjacent to the Concat module in the feature fusion network.

[0015] In step S3, the human-computer interaction interface is divided into a test area, a parameter selection area, a function button area, a test result area, and a test information area. The test area displays the PCB image to be tested; the parameter selection area allows selection of the model to be used for testing, model weights, preset confidence threshold, and IoU threshold; the function button area includes options for selecting a local image, selecting an external image, selecting a local video, selecting an external video, starting testing, saving results, deleting results, and exiting the program; the test result area displays the PCB image after testing, with the detected defect locations marked on the image using rectangles, and the defect category and confidence level marked above the rectangles. The confidence level represents the probability that the model considers the defect to belong to the marked defect category; the test information area displays the total number of detected defects, the specific number of defects in each category, and the time taken for the testing process.

[0016] The detection system of this invention includes a PCB image acquisition module, a defect detection module, and a human-computer interaction interface module. In order to improve the detection effect of the model on minute defects and reduce the false detection rate and false negative rate in the detection process, the YOLOv5 model is improved by adding a 160×160 detection head and deleting the original 20×20 detection head, thus constructing a new feature fusion network. In the feature fusion network, a hybrid attention mechanism module consisting of ECA attention and spatial attention mechanism in CBAM is added after each C3 module adjacent to the Concat module.

[0017] The beneficial effects of this invention are as follows: This invention addresses the problems of low accuracy and slow speed in traditional PCB defect detection methods, as well as the poor model detection performance and lack of deployment in current image processing, machine learning and deep learning-based detection methods. It designs a PCB defect detection system and method based on NVIDIA Jetson TX2, which effectively improves the detection accuracy and speed of PCB defect detection and has broad application prospects. Attached image description:

[0018] Figure 1 This is a flowchart of the PCB defect detection system of the present invention;

[0019] Figure 2 Here is a diagram of the network structure of an existing YOLOv5 model;

[0020] Figure 3 This is a network structure diagram of the improved YOLOv5 model of this invention;

[0021] Figure 4 The distribution and size of various defects in the PCB after normalization;

[0022] Figure 5 A human-machine interface for PCB defect detection;

[0023] Figure 6 This invention improves the detection results of the YOLOv5 model on the NVIDIA Jetson TX2. Detailed implementation method:

[0024] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific embodiments and illustrations.

[0025] This invention provides a PCB defect detection method based on NVIDIA Jetson TX2, such as... Figure 1 As shown, the specific steps include the following:

[0026] S1. Construct a PCB dataset and expand the data in the dataset. The open-source PCB defect dataset provided by the Peking University Intelligent Robotics Open Laboratory was selected as the training dataset for the model. The PCB dataset images contain six types of defects: missing holes, mouse blobs, spurious copper, short circuits, open circuits, and spurs. The number of defects in each type is shown in Table 1. To address the issue of the small sample size of the dataset, image preprocessing methods were used to expand the dataset, including image rotation, mirroring, translation, darkening, and noise addition. The number of defects in each type after expansion is shown in Table 2.

[0027] Table 1 Number of various defects in PCB

[0028]

[0029] Table 2 Number of various PCB defects after expansion

[0030]

[0031]

[0032] S2. Use Labelimg software to annotate the defects in each image. Once the annotation is complete, generate an XML file containing the annotation information.

[0033] S3. Analyze the location and size of various defect samples in the PCB dataset. The distribution information of the relative center coordinates of various PCB defects relative to the entire PCB image is as follows: Figure 4 As shown in (a), from Figure 4 As shown in (a), various defects are evenly distributed throughout the PCB image, with sparser distribution at the image edges. The relative width and height information of various PCB defects relative to the entire PCB image is as follows: Figure 4 As shown in (b), the width and height of various defects in the training set are mainly distributed within 6% of the total width and height of the image. These defects are small in size, which places high demands on the model to detect tiny defects.

[0034] S4. In the PCB defect detection task, the YOLOv5 detection algorithm is applied. The YOLOv5 detection algorithm is a convolutional neural network-based detection algorithm. The network structure diagram of the YOLOv5 model is shown below. Figure 2 As shown, it is widely used in various real-time detection tasks.

[0035] S5. Improvements to the feature fusion network: After the second Concat module in the YOLOv5 feature fusion network, C3, Conv, and upsample modules are added sequentially. The output of the upsample module is stacked with the output of the first C3 module in the feature extraction layer module along the channel dimension. The stacked feature map is then used for feature extraction through the C3 and Conv modules. The extracted feature map is then input into the second C3 module in the YOLOv5 feature fusion network. The output of the fourth C3 module in the feature fusion network structure is expanded by upsampling and stacked with the output of the third C3 module in the feature fusion network. The output of the fifth C3 module in the feature fusion network is stacked with the output of the fourth C3 module by upsampling. The output of the sixth C3 module in the feature fusion network is stacked with the output of the fifth C3 module by upsampling.

[0036] S6. Optimization of the feature detection head. A detection layer is added to predict on the 160×160 feature map output by the third C3 module in the new feature fusion network. At the same time, the detection layer of the YOLOv5 model that performs detection on the 20×20 feature map is removed to improve the detection speed of the model.

[0037] S7. The addition of a hybrid attention mechanism. A hybrid attention mechanism module, consisting of ECA attention and spatial attention from CBAM, is added to each C3 module adjacent to the Concat module in the feature fusion network. The Concat module is responsible for concatenating feature maps with the same width and height scale but from different network layers. The C3 module is responsible for integrating the feature information of the stacked feature maps. The hybrid attention mechanism module is responsible for extracting key information from the feature maps and ignoring useless information. This hybrid attention mechanism includes both the ECA network and the spatial attention network from CBAM, enabling the network to achieve cross-channel feature information interaction and acquire feature information from different spaces, thereby enhancing the model's detection accuracy. The improved YOLOv5 model network structure diagram is shown below. Figure 3 As shown.

[0038] S8. Dataset Partitioning. In the PCB defect detection task, the amount of data in the constructed PCB dataset is relatively small. In order to simplify the partitioning of the dataset, a fixed ratio partitioning method is adopted, dividing the training set, validation set, and test set in an 8:1:1 ratio, giving the training set a larger proportion to ensure sufficient training data sample size.

[0039] S9. Label Information Normalization. In the YOLO algorithm, the location information in the dataset needs to be normalized to improve the model's training speed. The data normalization process used in the YOLO algorithm is as follows:

[0040] (1) Given the width w and height h of the image, the four coordinate information of a single defect are xmin, xmax, ymin and ymax;

[0041] (2) Calculate the width w1 and height h1 of the rectangle containing the defect information;

[0042] w1 = xmax – xmin (1)

[0043] h1 = ymax – ymin (2)

[0044] (3) Calculate the midpoint positions x and y of the rectangle;

[0045]

[0046]

[0047] (4) Normalization operation yields Xout, Yout, Wout, and Hout;

[0048]

[0049]

[0050] Xout = x × dw (7)

[0051] Yout = y × dh (8)

[0052] Wout = w1 × dw (9)

[0053] Hout = h1 × dh (10)

[0054] Through the above calculations, we obtain Xout, Yout, Wout, and Hout after the normalization operation for each annotation information.

[0055] S10. Set the hyperparameters for model training. The hyperparameters used in training are shown in Table 3.

[0056] Table 3. Main hyperparameter settings for model training

[0057]

[0058] S11. Start model training; generate model weights upon completion of training. The hardware and software environments used for model training in the PCB defect detection experiment are shown in Tables 4 and 5, respectively. The weights used are those obtained from training the original model on the COCO dataset. Utilizing transfer learning, the model is then trained on the PCB dataset. After model training, a .pt file containing weight information is obtained.

[0059] Table 4 Hardware Environment

[0060]

[0061] Table 5 Software Environment

[0062]

[0063]

[0064] S12. Training Result Analysis. To verify the training results of the improved YOLOv5 model on a desktop computer, this invention lists the desktop performance of the single-stage classic object detection algorithms YOLOv3, YOLOv3-spp, RetinaNet network, and the two-stage classic object detection algorithm Faster R-CNN on the PCB dataset. The analysis mainly focuses on three aspects: model accuracy, the size of the weights generated by the model, and the detection speed of the model. Specific data are shown in Table 6.

[0065] Table 6 shows the detection performance of different detection algorithms on the PCB dataset.

[0066]

[0067] As shown in Table 6, the improved YOLOv5 model proposed in this invention achieves mAP@0.5 and mAP@0.5:0.95 of 0.986 and 0.554 respectively, with a weight file size of only 15.5M and a detection speed of 143 FPS. Compared to other models, the improved YOLOv5 model proposed in this invention has the highest detection accuracy, the fastest detection speed, and the smallest model weight.

[0068] S13. Deploy the improved YOLOv5 model trained on the desktop to the TX2 board using TensorRT.

[0069] S14. Set up the testing platform. Place the PCB sample to be tested on the testing workbench, install a Logitech Pro C920 camera above the testing platform, and connect the camera to the NVIDIA Jetson TX2 board via USB cable to take pictures of the PCB sample from above.

[0070] S15. Enable model inspection on the NVIDIA Jetson TX2 and use an external camera to photograph the PCB sample on the inspection workbench.

[0071] S16. Crop or pad the input image to adjust its size to the preset size of the detection model.

[0072] S17. The preprocessed image is input into the improved YOLOv5 model on the NVIDIA Jetson TX2. The model divides the image into grids, and each grid is responsible for detecting objects whose center falls within that grid. It needs to predict B bounding boxes. The bounding box prediction results include four location information and a confidence score. The confidence expression is shown in Equation (11). The confidence parameter contains the probability information that the predicted bounding box contains the object and the degree of overlap between this bounding box and the target's true location. The degree of overlap is represented by IoU, and its expression is shown in Equation (12). The larger the IoU parameter, the higher the degree of overlap between the predicted box and the true box. Each grid is also responsible for the category prediction task. The prediction results need to be processed by the NMS algorithm to remove redundant bounding boxes to obtain the final detection results.

[0073] Confidence=Pr(object)×IoU (11)

[0074] In the formula, Pr(Object) = 0 when there is no object to be measured in the grid, and Pr(object) = 1 when there is an object to be measured in the grid. IoU represents the intersection-union ratio between the predicted box and the ground truth box.

[0075]

[0076] In the formula, A and B represent the bounding box predicted by the model and the actual bounding box containing the object, respectively.

[0077] S18. Detection complete. The model uses rectangles to accurately mark the locations of all detected defects and marks the confidence score above the rectangles.

[0078] S19. Calculate the total number of all detected defects, the number of defects of each type, and the total time taken for the detection process.

[0079] S20. The detection results are transmitted to the human-machine interface, as shown in the human-machine interface... Figure 5 As shown, the detection results are displayed in the detection results area of ​​the interactive interface, and the specific detection information is displayed in the detection information area.

[0080] S21. Improved YOLOv5 model detection performance on NVIDIA Jetson TX2 for six types of PCB defects: Figure 6 As shown. From Figure 6As can be seen, the improved YOLOv5 model can accurately detect six types of defects in each image: missing hole, mouse bite, open circuit, short, spur, and spurious copper. Moreover, the defect location prediction is accurate, which basically meets the actual needs of PCB defect detection model deployment.

[0081] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A PCB defect detection method based on NVIDIA Jetson TX2, characterized in that, Includes the following steps: S1. The PCB image acquisition module acquires PCB images through an external camera, and performs cropping and filling processing on the images. The images are then adjusted to the size specified by the detection model, and the adjusted images are input into the defect detection module. S2. Using the improved YOLOv5 model deployed on NVIDIA Jetson TX2, the acquired PCB images are sequentially processed through a convolutional network consisting of Focus, Conv, C3, SPP, and upsample to extract and fuse features, generating feature maps containing various semantic information of the images. Then, multi-scale detection is performed on the images through detection layers of sizes 40×40, 80×80, and 160×160. S3. The detection results are output to the human-computer interaction interface. The detection information area displays the total number of PCB defects detected, the number of PCB defects of each type, and the total time taken for the detection process. The detection result area displays the PCB image after detection, including the annotation information. In step S2, the improved YOLOv5 model adds a C3 module, a Conv module, and an upsample module sequentially after the second Concat module of the YOLOv5 feature fusion network. The output of the upsample module is stacked with the output of the first C3 module of the feature extraction layer in the channel dimension. The stacked feature map is then processed by the C3 and Conv modules for feature extraction. The extracted feature map is then input into the fourth C3 module of the YOLOv5 feature fusion network. The output of the fourth C3 module in the feature fusion network structure is expanded by upsampling and then stacked with the output of the third C3 module in the feature fusion network. The output of the fifth C3 module in the feature fusion network is stacked with the output of the fourth C3 module through upsampling. The output of the sixth C3 module in the feature fusion network is stacked with the output of the fifth C3 module through upsampling. A detection layer is added to predict on a feature map of size 160×160 from the output of the third C3 module in the new feature fusion network. At the same time, the detection layer of YOLOv5 model that performs detection on a feature map of size 20×20 is removed. A hybrid attention mechanism module consisting of ECA attention and spatial attention mechanism in CBAM is added to each C3 module adjacent to the Concat module in the feature fusion network.

2. The PCB defect detection method according to claim 1, characterized in that: In step S3, the human-computer interaction interface is divided into a test area, a parameter selection area, a function button area, a test result area, and a test information area.

3. The PCB defect detection method according to claim 2, characterized in that: The test area displays the PCB image to be tested; the parameter selection area selects the model to be used for detection, model weights, preset confidence threshold, and IoU threshold; the function button area includes options to select a local image, select an external image, select a local video, select an external video, start detection, save results, delete results, and exit the program; the detection result area displays the PCB image after detection, with the detected defect locations marked on the image using rectangles, and the defect category and confidence level marked above the rectangles; the detection information area displays the total number of detected defects, the specific number of defects of each type, and the time taken for the detection process.

Images