Circuit board defect detection method based on high-resolution image

Abstract

The invention discloses a circuit board defect detection method based on a high-resolution image, and belongs to the field of image recognition. The invention designs a training and detection scheme based on a slicing mode and provides an improved circuit board defect detection method based on a YOLOv8 network in order to solve the problem of low identification accuracy of defects caused by small components in target detection of visible light image components of a circuit board based on the classic YOLOv8 network. A data set is constructed by building an experimental platform, experimental verification is carried out on the proposed scheme and method, the result shows that the detection effect of the model for a small target is improved, and the algorithm can output a defect detection result. According to the invention, the component loss and wrong welding defects of the complex circuit board can be detected.

Description

Technical Field

[0001] The invention belongs to the field of image recognition, in particular to circuit board defect detection based on high-resolution images. Background Art

[0002] At present, the detection of appearance defects of circuit boards with components mainly adopts the reference method detection scheme based on deep learning, but this detection method has the following two disadvantages: The defect detection accuracy is not high: When the existing detection method is used for complex circuit boards, due to the dense and high integration of circuit board components, the background of each device in the circuit board image is complex, and it is difficult to use conventional detection methods for defect detection, and the accuracy is not high.

[0003] The recognition accuracy of small target components is not high: When identifying components, the existing methods have difficulty in identifying small target components, especially when resistors and capacitors are densely packed. There are a large number of recognition errors and missed recognitions, which in turn affects the accuracy of small target component defect detection. Summary of the invention

[0004] By combining image processing technology and target detection technology, using high-resolution cameras to perform high-definition imaging of complex circuit boards, improving data set generation and target detection methods, and making structural improvements based on the YOLOv8 target recognition model, the model's recognition accuracy for small target components is improved, further improving the accuracy of circuit board defect detection.

[0005] The technical solution of the present invention is a circuit board defect detection method based on high-resolution images, the method comprising:

[0006] Step 1: Build a dataset;

[0007] Step 1.1: Collect the front view of the circuit board and mark the components in the circuit board. The marked components include: chips, resistors, capacitors, tantalum capacitors, electrolytic capacitors, diodes, transistors, LED lights, inductors, crystal oscillators, and fuses;

[0008] Step 1.2: Slice the front view of the circuit board, and the adjacent slices overlap;

[0009] Step 1.3: Label the slices of each component in step 1.2 according to the labeling in step 1.1;

[0010] Step 2: Establish a YOLOv8 recognition model;

[0011] The YOLOv8 recognition model includes a backbone network, a neck network, and a head network connected in series. The backbone network extracts features from the input image to obtain multiple feature maps of different sizes. The neck network first aligns the feature maps obtained by the backbone network, then fuses the feature maps to obtain global information, and then fuses the obtained global information with some feature maps obtained by the backbone network. The feature maps that have been fused with global information and the feature maps that have not been fused with global information are combined. Figure 1 It serves as the output of the neck network and is transmitted to the head network, which then outputs the recognition result.

[0012] Step 3: Use the data set obtained in step 1 to train the established YOLOv8 recognition model;

[0013] Step 4: Use the trained YOLOv8 recognition model to identify each component in the circuit board to be tested, obtain the recognition result, and compare the recognition result with the standard circuit board to determine whether there are any component errors or missing in the circuit board to be tested.

[0014] Furthermore, the specific structure of the neck network in the YOLOv8 recognition model established in step 2 is:

[0015] The backbone network outputs features of four scales, namely: feature B5, feature B4, feature B3, and feature B2. The first feature alignment module is used to align these four features, and then they are input into the first information fusion module. The information fusion module outputs the first global information, which is fused with feature B4 and feature B3 respectively to obtain feature P4 and feature P3 respectively.

[0016] Feature B5, feature P4, and feature P3 are input to the second feature alignment module. The output of the second feature alignment module is connected to the second information fusion module. The second information fusion module outputs the second global information. The second global information is fused with feature P4 and feature B5 respectively to obtain feature N4 and feature N5 accordingly. Feature N4, feature N5, and feature P3 are used as the output of the neck network.

[0017] Furthermore, the specific method of step 4 is:

[0018] Step 4.1: Slice the circuit board to be tested, and adjacent slices overlap;

[0019] Step 4.2: Use the trained YOLOv8 recognition model to identify the components in each slice, and obtain the prediction box of each component in all slices and the confidence of each prediction box;

[0020] Step 4.3: Sort all prediction boxes by confidence to obtain a sorted list;

[0021] Step 4.4: The prediction with the highest confidence is framed as S and removed from the sorted list;

[0022] Step 4.5: Calculate the overlap between S and the remaining prediction boxes, retain the prediction boxes in the sorted list whose overlap IOU is lower than the threshold, and update the sorted list; at the same time, add S to the result list;

[0023] Step 4.6: Repeat steps 4.4 and 4.5 until the sorted list is empty, and the final result list is the recognition result.

[0024] Furthermore, the specific calculation method of the overlap IOU in step 4.5 is:

[0025]

[0026] Among them, A is the detection box.

[0027] Aiming at the problem of low accuracy of defect recognition caused by small components in the target detection of components in visible light images of circuit boards based on the classic YOLOv8 network, the present invention designs a training and detection scheme based on slicing, and proposes a circuit board defect detection method based on the improved YOLOv8 network. By building an experimental platform to construct a data set, the proposed scheme and method are experimentally verified. The results show that the detection effect of the model for small targets is improved, and the algorithm can output defect detection results. The present invention can detect missing components and mis-soldering defects on complex circuit boards. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The flowchart of the circuit board defect detection method based on high-resolution images.

[0029] Figure 2 Some pictures of the dataset.

[0030] Figure 3 This is a partial picture of the sliced ​​dataset.

[0031] Figure 4 Flowchart of the greedy non-maximum suppression merging algorithm.

[0032] Figure 5 The flowchart of the circuit board defect detection method based on high-resolution images.

[0033] Figure 6 This is a graph showing the model improvement and slicing method detection results.

[0034] Figure 7 This is the flow chart of the defect determination algorithm.

[0035] Figure 8 This is the result diagram of defect detection based on this algorithm. DETAILED DESCRIPTION

[0036] Step 1: Dataset Construction

[0037] A dataset was constructed for model training and testing. A Xiongmai JZC camera with a resolution of 3840×2160 was used to collect 124 types of circuit boards, with a total of 278 circuit board front views. Figure 2 These are some pictures of the dataset.

[0038] Labelimg software was used to complete the labeling of the image targets. A total of 11 components were selected for labeling, which basically covered the common types of components on the circuit board. Table 1 shows the number of component instances in the labeled data set. From the table analysis, there are 26,901 component instances, with resistors and capacitors accounting for the largest proportion, with a number of more than 9,000, accounting for about 35%, chips and diodes accounting for about 10%, and the remaining components accounting for a small proportion.

[0039] Table 1 Number of instances of each component category in the dataset

[0040]

[0041] Step 2: Slice-based dataset generation and detection

[0042] Slice dataset generation

[0043] Because the images in the dataset are high-resolution images, the conventional detection method is to scale the image size to 640×640 and then input it into the model. This method will lose a lot of detail information of the original image, and the components account for a small proportion of the entire image, which is a small target. According to the detection mechanism of the model, it is easy to miss the detection of dense small targets. Therefore, the high-resolution image is cropped into multiple 640×640 sizes, and the detection results are merged after each detection. In this way, on the one hand, the detail information of the high-resolution image is not lost, so the detail information of the target is retained, and it is less likely to be confused in the identification of resistors and capacitors; on the other hand, in the cropped sliced ​​image, the area of ​​each component target in the slice is increased, and the small target becomes a larger target, which is easier to be detected by the network model, which can improve the detection effect of the target and reduce missed recognition.

[0044] When cropping, the target object at the edge of the slice may be cut into two parts, and there will be no complete target in both slices, resulting in the loss of target instances in the dataset. Therefore, when cropping slices, an overlap ratio of 0.2 is set so that some areas of the edges of adjacent slices are the same, thus ensuring that the edge targets can be fully retained in at least one image. Even if the overlap ratio is set, the edge targets may still be incomplete when cutting the image, and whether these incomplete targets should be retained requires careful consideration. By calculating the ratio of the remaining pixel area of ​​the object area after cropping to the object pixel area of ​​the original image before cropping, a value greater than 0.45 indicates that most of the target area is retained and the annotation labels in the slice are retained. A value less than 0.45 indicates that there are few target areas in the slice and the annotation labels are discarded.

[0045] After designing the above implementation details, crop the images in the dataset, calculate the annotation labels of the cropped slices based on the annotation labels of the original images, and save the slices and labels as files to obtain the slice dataset. Figure 3 It is a partial image file of the processed slice dataset.

[0046] Step 3: Slice detection and result merging

[0047] After training the model using the slice dataset and obtaining the trained component target detection model, when testing a high-resolution image, the high-resolution image is cropped into multiple slices, and the slices are fed into the model for detection. The overlap ratio is also set during detection to ensure that edge objects can be fully preserved in adjacent slices. After the slice detection is completed, the object label position of the slice detection result needs to be calculated as the position in the original image, which is achieved by adding the position to the position offset of the upper left corner of the slice in the original image.

[0048] In order to ensure that all objects in the image can be detected as much as possible, especially large objects that occupy several slices are fully detected, an uncropped original image detection is performed. The detection results of all slices are merged with the detection results of the original image. Because the same target object is detected multiple times in multiple slices and original images, there are duplications, and it is necessary to filter out redundant object boxes of the same target. This is achieved by designing a greedy non-maximum suppression merging algorithm (Greedy Non-MaximumMerging).

[0049] Figure 4This is the flowchart of the greedy non-maximum suppression algorithm. At the beginning of the algorithm, the result list is initialized first, and all prediction boxes are sorted according to the confidence score. If the sorted list is not empty, the prediction box with the highest confidence score is selected as S, and it is removed from the sorted list. If the sorted list is not empty, the IOU between S and the remaining prediction boxes is calculated, and the prediction boxes below the threshold are retained, and the sorted list is updated. At the same time, S is added to the result list keep_res. If the sorted list is empty, S is directly added to the sorted list. This loop is executed until the sorted list is empty, and the result list keep_res is returned.

[0050] Step 4: Neck structure based on aggregation and distribution mechanism

[0051] For the problem of small target detection, the YOLOv8 network structure is analyzed. The Backbone structure is responsible for feature extraction and expression of image data, the Neck structure is responsible for feature fusion of feature maps of different feature layers and enhancing the features of each layer, and the Head structure is responsible for prediction based on feature maps of different scales. Therefore, for the problem of small targets, improving the Neck structure is a more reasonable direction. A good Neck structure can more perfectly integrate the features of each layer, so that the low-level large-scale feature map also integrates the deeper features of the high-level small-scale feature map, so that small targets can be found more accurately during prediction;

[0052] YOLO's Neck structure has evolved from the original feature pyramid (FPN) structure to the later path aggregation network (PAN) structure. Although PAN has an additional fusion branch from low to high layers based on FPN, its feature fusion still has the disadvantages of only being able to fuse adjacent layers and not being able to directly fuse across layers, and there will be information loss.

[0053] The Neck structure is improved by introducing the Gather and Distribute Mechanism (GDM). A unified module is used to collect and fuse information at each level and distribute it to different layers. Compared with PAN, this avoids information loss and enhances the feature fusion capability of Neck.

[0054] The aggregation and distribution of features are implemented through three modules: Feature Alignment Module (FAM), Information Fusion Module (IFM), and Information Injection Module (Inject). Feature aggregation collects and aligns features of each layer through FAM, and then fuses features through IFM to generate global information. Feature distribution collects global and local features through the Inject module, and injects information into different feature layers through attention operations. For the model to detect targets of different sizes, the low-level aggregation distribution branch (Low-GD) and the high-level aggregation distribution branch (High-GD) are used to process them respectively.

[0055] Figure 5 This is the GDM structure diagram. After the image is extracted by Backbone, four feature maps of different sizes, B2, B3, B4, and B5, will be generated. The size of the deep feature map is smaller, but the feature extraction and expression are more profound. GDM uses a new Neck structure to fuse the features of each layer. First, the low-level aggregation and distribution branch Low-GD part first collects and aligns all feature layers through the Low-FAM module. Low-IFM further extracts and fuses the feature information for a deeper understanding and obtains the global features. The Low-Inject module inputs the local features from the current layer and the global features from the Low-IFM, and injects the global features into the current layer, that is, the P3 and P4 layers in the figure. The High-GD branch also includes FAM for feature collection and alignment, IFM for information fusion, and Inject for feature injection, but the internal structure of the module is different, and Inject is injected into the high-level N4 and N5 layers. The two branches enhance the model's prediction ability for large and small targets respectively.

[0056] (1) Design of a circuit board defect detection method based on high-resolution images Figure 1 As shown in the figure, the high-resolution image is sliced, and the model is trained based on the slice data set. During the inspection, the standard circuit board and the inspection circuit board images are sliced ​​for inspection, and the improved YOLOv8 model is used for recognition. The recognition results are input into the defect detection judgment algorithm for processing, and the defect detection results are output.

[0057] (2) A slicing-based dataset generation and detection method was designed to handle the problem of small target component recognition in high-resolution circuit board images, improve the recognition accuracy of small target components in high-resolution images of complex circuit boards, and reduce the problems of missed recognition and misidentification of resistors and capacitors.

[0058] (3) The clustering and distribution mechanism is introduced to improve the Neck network structure in YOLOv8, solving the problem that the features in the original structure can only be fused in adjacent layers but not across layers. The two clustering and distribution branch modules are used to improve the model's feature fusion capabilities for large and small targets respectively, thereby improving the model's accuracy in component recognition.

[0059] (4) A defect detection and judgment algorithm based on the reference method was designed. By calculating the IOU and target category of the corresponding positions of the standard board and the defective board identification results, it was determined whether there were any missing components or mis-soldering defects, and the circuit board defect detection results were obtained.

[0060] Table 1 shows the evaluation index results of the validation set test using the trained model. Analysis shows that in terms of accuracy, the four components of resistors, LED lights, chips, and diodes are all above 0.95, indicating that the model's predictions for these three types of devices are relatively accurate. In terms of recall, the four components of resistors, capacitors, inductors, and chips are all above 0.95, indicating that the model can find most of these devices. In terms of mAP50, chips, resistors, capacitors, and tantalum capacitors are all above 0.97, which is the first tier, and the model has the best detection performance for these devices; electrolytic capacitors, diodes, triodes, LED lights, and inductors are all around 0.95-0.96, which is the second tier; the remaining crystal oscillator devices are 0.906, which is significantly lower than the rest of the devices, and are the third tier. The model has a very good detection performance for the first and second tiers, and a slightly lower detection performance for the third tier. Analysis shows that the number of crystal oscillator instances is low, that is, the proportion of crystal oscillators in circuit board devices is very low, which may be caused by too few samples for learning during training. Figure 8 The figure is the result of defect detection based on the algorithm; it can be seen that the algorithm can accurately detect component loss and wrong soldering defects.

[0061] Table 1 Experimental results of the improved model validation set trained on the slice dataset

[0062]

[0063] Figure 6 The target recognition results of components detected using the improved model and slicing method. It can be seen that most components have been basically recognized, and small target components such as resistors and capacitors also have excellent recognition effects.

[0064] 1. Defect determination algorithm

[0065] After the above model improvement, the detection capability of components has been significantly improved, but defect detection still requires the design of defect determination algorithm to determine whether the circuit board has defects. Figure 7 This is the defect judgment algorithm flow designed in this paper. The algorithm is mainly used to detect two defects: missing components and wrong component soldering.

[0066] Reference Figure 7Algorithm flow, after testing the test board, input the test results and the test result files of the standard board into the algorithm. Traverse the components of the standard board, select the currently traversed component and calculate its pixel area, calculate the pixel area of ​​all components of the test board, find the intersection area and calculate the intersection area and union area, and then calculate the IOU of the current component and all components of the test board. The IOU value greater than 0.5 is judged as a match, that is, the components of the standard board and the test board correspond in this area. If the current component does not match, it is judged as a missing component. If it matches but the category of the component is different, it is judged as a component welding error. Mark the two defects in the corresponding positions in the figure and output the test result picture, so that the detection and judgment of the two circuit board defects are completed.

Claims

1. A circuit board defect detection method based on high-resolution images, the method comprising: Step 1: Build a dataset; Step 1.1: Collect the front view of the circuit board and mark the components in the circuit board. The marked components include: chips, resistors, capacitors, tantalum capacitors, electrolytic capacitors, diodes, transistors, LED lights, inductors, crystal oscillators, and fuses; Step 1.2: Slice the front view of the circuit board, and the adjacent slices overlap; Step 1.3: Label the slices of each component in step 1.2 according to the labeling in step 1.1; Step 2: Establish a YOLOv8 recognition model; The YOLOv8 recognition model includes a backbone network, a neck network, and a head network connected in series in sequence. The backbone network extracts features from an input image to obtain a plurality of feature maps of different sizes. The neck network first aligns the feature maps obtained by the backbone network, then fuses the feature maps to obtain global information, and then fuses the obtained global information with part of the feature maps obtained by the backbone network. The feature map that fuses the global information and the feature map that does not fuse the global information are used as the output of the neck network and transmitted to the head network, and the head network outputs the recognition result. Step 3: Use the data set obtained in step 1 to train the established YOLOv8 recognition model; Step 4: Use the trained YOLOv8 recognition model to identify each component in the circuit board to be tested, obtain the recognition result, and compare the recognition result with the standard circuit board to determine whether there are any component errors or missing in the circuit board to be tested.

2. A circuit board defect detection method based on high-resolution images as claimed in claim 1, characterized in that: The specific structure of the neck network in the step 2 to establish the YOLOv8 recognition model is: The backbone network outputs features of four scales, namely: feature B5, feature B4, feature B3, and feature B2. The first feature alignment module is used to align these four features, and then they are input into the first information fusion module. The information fusion module outputs the first global information, which is fused with feature B4 and feature B3 respectively to obtain feature P4 and feature P3 respectively. Feature B5, feature P4, and feature P3 are input to the second feature alignment module. The output of the second feature alignment module is connected to the second information fusion module. The second information fusion module outputs the second global information. The second global information is fused with feature P4 and feature B5 respectively to obtain feature N4 and feature N5 accordingly. Feature N4, feature N5, and feature P3 are used as the output of the neck network.

3. A circuit board defect detection method based on high-resolution images as claimed in claim 1, characterized in that: The specific method of step 4 is: Step 4.1: Slice the circuit board to be tested, and adjacent slices overlap; Step 4.2: Use the trained YOLOv8 recognition model to identify the components in each slice, and obtain the prediction box of each component in all slices and the confidence of each prediction box; Step 4.3: Sort all prediction boxes by confidence to obtain a sorted list; Step 4.4: The prediction with the highest confidence is framed as S and removed from the sorted list; Step 4.5: Calculate the overlap between S and the remaining prediction boxes, retain the prediction boxes in the sorted list whose overlap IOU is lower than the threshold, and update the sorted list; at the same time, add S to the result list; Step 4.6: Repeat steps 4.4 and 4.5 until the sorted list is empty, and the final result list is the recognition result.

4. The circuit board defect detection method based on high-resolution images according to claim 1, characterized in that: The specific calculation method of the overlap IOU in step 4.5 is: Among them, A is the detection box.

Images