PCB (Printed Circuit Board) defect detection method based on multistage feature fusion

Abstract

The invention relates to a PCB quality detection technology, and aims to provide a PCB defect detection method based on multistage feature fusion to solve the problem of high hardware requirements caused by weak background interference resistance or large calculation amount in the prior art, and the technical scheme can be summarized as follows: obtaining a PCB image to be detected; carrying out ROI extraction on a key area of the to-be-detected PCB image, excluding a substrate background area, and caching the extracted ROI; aligning the ROI of the preset standard image and the ROI of the to-be-detected PCB image by adopting a multi-scale feature registration method, and obtaining an ROI intersection region; and carrying out sliding window detection by adopting a parallel detection framework based on the ROI intersection region to obtain a full-size defect mask, and completing defect detection. The method is small in calculation amount, high in detection precision and suitable for PCB quality detection.

Description

Technical Field

[0001] This invention relates to PCB board quality inspection technology, and in particular to a PCB board defect detection method based on multi-level feature fusion. Background Technology

[0002] As the core carrier of electronic products, the manufacturing quality of PCBs directly determines the reliability and stability of downstream electronic devices. The PCB manufacturing process encompasses several key stages, including substrate pretreatment, circuit printing, etching, solder mask coating, and pad preparation. Defects can occur at any stage due to fluctuations in process parameters, equipment precision limitations, or environmental interference. Common defects in the circuit printing stage include open circuits, short circuits, and burrs. Open circuits cause conductor breakage, leading to signal transmission interruptions; short circuits cause abnormal connections between adjacent lines, resulting in circuit malfunctions; and burrs cause irregular protrusions at the edges of the circuit, easily leading to insulation failure. Common defects in the pad preparation stage include missing pads, contamination, and misalignment. Missing pads prevent component soldering; contamination causes oil or impurities to adhere to the surface, leading to poor soldering; and misalignment causes misalignment of component pins, resulting in cold solder joints. If these defects are not detected and eliminated in time, they will lead to problems such as functional failure and short service life of electronic products, and may even cause safety hazards such as overheating and short circuits. Especially in high-end fields such as automotive electronics and aerospace, PCB defects may cause significant economic losses and safety accidents.

[0003] Currently, automated inspection methods for PCB defects generally employ traditional machine vision methods and end-to-end deep learning methods. Traditional machine vision methods have relatively simple feature extraction and fusion structures, performing only single-level feature difference extraction without distinguishing between multi-scale feature levels. Furthermore, their attention and fusion modules are often poorly designed, resulting in weak background interference suppression and an inability to effectively separate defects from background features. Consequently, they are sensitive to deformation and lighting conditions, exhibiting poor robustness. End-to-end deep learning methods, on the other hand, require a large number of labeled samples and have high requirements for image registration. Failure to employ effective image registration methods can lead to positioning errors, causing detection failures. Additionally, the model inference used in both traditional machine vision and end-to-end deep learning methods generally relies on GPU (Graphics Processing Unit) hardware, and some require dedicated AOI (Automated Optical Inspection) equipment, placing high demands on hardware and resulting in higher costs.

[0004] In summary, existing technologies suffer from weak resistance to background interference or high computational demands leading to high hardware requirements. Summary of the Invention

[0005] The purpose of this invention is to provide a PCB board defect detection method based on multi-level feature fusion, so as to solve the problems of weak anti-background interference ability or high hardware requirements due to large computational load in the prior art.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems provides a PCB board defect detection method based on multi-level feature fusion, comprising the following steps:

[0007] Acquire the image of the PCB to be tested;

[0008] The Region of Interest (ROI) of key areas is extracted from the PCB image under test, the substrate background area is excluded, and the extracted ROI is cached.

[0009] A multi-scale feature registration method is used to align the ROI of the preset standard image and the ROI of the PCB image under test, and the intersection area of ​​the ROI is obtained.

[0010] Based on the intersection region of ROI, a parallel detection architecture is used to perform sliding window detection to obtain a full-size defect mask and complete the defect detection.

[0011] In some embodiments, to provide a feasible method for acquiring an image of a PCB under test, the acquisition of the PCB image under test includes:

[0012] An industrial area scan camera is used to acquire images of the PCB under test. The acquisition environment uses a dual-strip white light source. During acquisition, the vertical distance between the camera head and the PCB image under test is controlled to be 30-50cm.

[0013] In some embodiments, a feasible method is provided for extracting the Region of Interest (ROI) of a key area from a PCB image under test, excluding the substrate background area, and caching the extracted ROI data. The method includes:

[0014] A pre-defined deep learning model is used to perform semantic segmentation on the PCB image under test, predict key regions, extract the largest connected region in the segmentation results as the effective ROI contour, and exclude the substrate background region.

[0015] The minimum bounding rectangle of the effective ROI contour is calculated using the minimum bounding rectangle algorithm. The rotation angle of the rectangle is obtained, and an affine transformation matrix is ​​constructed with the center of the minimum bounding rectangle as the rotation center and the rotation angle of the rectangle as the rotation angle. Image rotation correction is then performed.

[0016] Create a binary mask for the ROI from the corrected image, scale it by a certain scaling factor, extract feature points and descriptors using the SIFT algorithm, and cache the coordinates of the scaled image, feature points, descriptors, and the corresponding ROI contours.

[0017] In some embodiments, to provide a feasible method for aligning the ROI of a preset standard image and the ROI of a PCB image under test using a multi-scale feature registration method, the method for aligning the ROI of the preset standard image and the ROI of the PCB image under test using a multi-scale feature registration method includes:

[0018] A multi-scale feature registration method using SIFT and RANSAC (Random Sample Consensus) algorithms is employed to align the ROI of a pre-defined standard image with the ROI of the PCB image under test.

[0019] In some embodiments, to provide a feasible method for aligning the ROI of a preset standard image and the ROI of a PCB image under test using a multi-scale feature registration method formed by SIFT and RANSAC algorithms, and obtaining the ROI intersection region, the method for aligning the ROI of the preset standard image and the ROI of the PCB image under test using a multi-scale feature registration method formed by SIFT and RANSAC algorithms, and obtaining the ROI intersection region, includes:

[0020] Obtain the SIFT feature descriptor of the pre-stored standard image, and use the FLANN matcher to match the SIFT feature descriptor of the standard image with the descriptor of the PCB image to be tested in the cache;

[0021] Based on the matching results, the 8-parameter perspective transformation matrix is ​​solved by robust estimation using the RANSAC algorithm. The ROI of the PCB image under test is then transformed by perspective to achieve pixel-level alignment between the ROI of the PCB image under test and the ROI of the standard image.

[0022] The ROI coordinates of the standard image are mapped to the image space of the ROI of the PCB image under test through a perspective transformation matrix. The intersection region of the two ROI bounding boxes is calculated. The overlapping region in the ROI of the PCB image under test is retained as the ROI intersection region, and the remaining regions are deleted.

[0023] In some embodiments, to provide a feasible method for performing sliding window detection based on a parallel detection architecture using ROI intersection regions to obtain a full-size defect mask and complete defect detection, the method of performing sliding window detection based on a parallel detection architecture using ROI intersection regions to obtain a full-size defect mask and complete defect detection includes:

[0024] The detection range is the intersection area of ​​ROIs. A 512×512 pixel window with a step size of 256 pixels is used for sliding. Mirror filling is used for areas where the boundary is insufficient for the window size.

[0025] Using a Siamese U-Net network with pre-defined shared weights, the ROI of the registered and aligned standard image and the ROI of the PCB image under test are used as dual-branch inputs to obtain the window defect probability map for each thread.

[0026] The window defect probability maps of each thread are stitched together according to their coordinate positions. If overlapping areas are encountered during stitching, the maximum probability is taken, and the full-size defect mask is obtained by fusion.

[0027] In some embodiments, to provide a feasible pre-defined structure for a Siamese U-Net network with shared weights, the Siamese U-Net network with shared weights includes:

[0028] A shared encoder is used to simultaneously extract features from at least two levels of dual-branch input, enabling parameter sharing during the feature extraction process;

[0029] The multi-type difference calculation module includes at least two difference calculation units corresponding to the output of the shared encoder. Each difference calculation unit is used to first calculate the absolute difference and cosine similarity of the features corresponding to the two-branch input. Here, the cosine similarity is calculated along the channel dimension and then expanded to the original number of channels to obtain the difference features. The difference features are then concatenated along the channel dimension and fused through a two-level convolutional network to output a difference feature map with the same number of input feature channels. The two-level convolutional network is composed of 3×3 convolution, batch normalization, and ReLU.

[0030] The decoder includes a DecoderBlock with skip connections and an UPConvBlock. The number of DecoderBlocks is one less than the number of difference computation units. Each pair of adjacent difference computation units corresponds to one DecoderBlock, and there is one UPConvBlock. The DecoderBlock adjusts the number of channels by combining bilinear upsampling with 1×1 convolution, and concatenates the channels with the difference feature maps corresponding to the output of the shared encoder layer. Then, the features are further fused through convolutional layers. The UPConvBlock uses upsampling and channel adjustment strategies to complete the final feature recovery.

[0031] The prediction module is used to output a window defect probability map by combining 3×3 convolution with batch normalization and ReLU, Dropout, 1×1 convolution and Sigmoid activation function. If the size of the output window defect probability map is inconsistent with the input window, it is adjusted to the input size by bilinear interpolation.

[0032] In some embodiments, to provide a feasible training method for a Siamese U-Net network with preset shared weights, the training method for the Siamese U-Net network with preset shared weights includes:

[0033] Obtain standard and defect images of the PCB board;

[0034] The standard image preprocessing includes: traversing all standard images; for any given standard image, extracting ROI point coordinates to obtain the ROI contour; rotating the standard image to a horizontal position based on the ROI contour; calculating the ROI bounding box and ROI mask; extracting SIFT feature points and descriptors; obtaining a preprocessing result containing the rotated image, ROI information, and SIFT feature points and descriptors; and storing the preprocessing result in a cache according to filename. The ROI information includes ROI point coordinates, ROI contour, ROI bounding box, and ROI mask.

[0035] Image pair matching is performed, including: traversing all defect images, matching the corresponding standard image according to the file name, collecting all valid matching image pairs, and ensuring that each image pair corresponds to different states of the same PCB board;

[0036] The processing of a single image pair includes: reading the defect image and extracting its ROI point coordinates and defect annotations to obtain the ROI contour; rotating the defect image to a horizontal position based on the ROI contour; using SIFT feature matching to register the ROI of the standard image to the coordinate system of the ROI of the defect image; calculating the intersection of the ROI bounding box of the standard image and the ROI bounding box of the defect image to obtain the ROI intersection region; if there is no intersection, skipping the image pair; transforming the coordinates of the defect annotations in the defect image to the coordinate system of the ROI of the rotated defect image; performing sliding window cropping within the ROI intersection region to obtain the cropped region; calculating the SSIM value of the cropped region and filtering out difference image pairs with a value lower than min_ssim; extracting the defect annotations within the cropped region and transforming their coordinates to the coordinate system of the cropped image; and collecting information on all valid cropped image pairs as samples for training.

[0037] The training configuration of the Siamese U-Net network with shared weights includes: initializing the optimizer, configuring the loss function, and configuring the learning rate scheduler;

[0038] The Siamese U-Net network with shared weights, after training configuration, is trained using the aforementioned samples. In some embodiments, to further improve the accuracy and quality of defect localization, the network may further include:

[0039] Morphological enhancement and contour filtering operations are performed on the acquired full-size defect mask to enhance the continuity of the defect contour and eliminate noise interference.

[0040] In some embodiments, to provide a feasible morphological enhancement and contour filtering operation, the morphological enhancement includes:

[0041] A 5×5 rectangular structural element is used to perform a closing operation on the full-size defect mask, filling the tiny holes in the corresponding defect area and enhancing the continuity of the defect outline.

[0042] The contour filtering includes:

[0043] The contour extraction algorithm is used to extract the defect contour corresponding to the full-size defect mask. The search mode of the contour extraction algorithm is set to only the outer contour, filtering out defect contours with an area of ​​less than or equal to 100 pixels. It is also verified whether the unfiltered defect contours are located in the ROI intersection area. If so, they are retained; otherwise, they are deleted.

[0044] The beneficial effects of this invention are that, in the solution of this invention, background areas are removed by ROI extraction, reducing the amount of data processed in subsequent processing by more than 60%, thereby solving the problem of large computational load in full-image detection. At the same time, the multi-scale feature registration method effectively alleviates the problem of existing models being sensitive to deformation. Furthermore, the parallel detection architecture for sliding window detection can be combined with CPU multi-threading acceleration, eliminating the need for GPU hardware support, allowing deployment on ordinary industrial control computers, and resulting in lower costs. Attached Figure Description

[0045] Figure 1 This is a schematic flowchart of a PCB board defect detection method based on multi-level feature fusion in an embodiment of the present invention. Detailed Implementation

[0046] The technical solution of the present invention will now be described in detail with reference to the embodiments and accompanying drawings.

[0047] like Figure 1 As shown, this embodiment of the invention provides a PCB board defect detection method based on multi-level feature fusion, including the following steps:

[0048] Acquire the image of the PCB to be tested;

[0049] The Region of Interest (ROI) of key areas is extracted from the PCB image under test, the substrate background area is excluded, and the extracted ROI is cached.

[0050] A multi-scale feature registration method is used to align the ROI of the preset standard image and the ROI of the PCB image under test, and the intersection area of ​​the ROI is obtained.

[0051] Based on the intersection region of ROI, a parallel detection architecture is used to perform sliding window detection to obtain a full-size defect mask and complete the defect detection.

[0052] It is understandable that in the above embodiments, background areas are removed by ROI extraction, which reduces the amount of data to be processed in subsequent processing. Actual tests have shown that the amount of data to be processed in subsequent processing is reduced by more than 60%, thereby solving the problem of large computational load in full-image detection. At the same time, the multi-scale feature registration method effectively alleviates the problem of existing models being sensitive to deformation. Furthermore, the parallel detection architecture for sliding window detection can be combined with CPU multi-threading acceleration, without the need for GPU hardware support, and can be deployed on ordinary industrial control computers at a low cost.

[0053] In some embodiments, to provide a feasible method for acquiring an image of a PCB under test, acquiring the image of the PCB under test may include:

[0054] An industrial area scan camera is used to acquire images of the PCB under test. The acquisition environment uses a dual-strip white light source. During acquisition, the vertical distance between the camera head and the PCB image under test is controlled to be 30-50cm.

[0055] It is understandable that the industrial area scan camera can be a Hikvision industrial camera with 20 megapixels, and the acquisition environment uses a dual-strip white light source. During acquisition, the vertical distance between the camera head of the industrial area scan camera and the PCB image under test is controlled to be 30-50cm, which can ensure that the image resolution meets the requirements for micron-level defect detection, that is, the actual size of a single pixel is ≤10μm.

[0056] In some embodiments, a feasible method is provided for extracting the Region of Interest (ROI) of a critical region from a PCB image under test, excluding the substrate background region, and caching the extracted ROI data. This method may include:

[0057] A pre-defined deep learning model is used to perform semantic segmentation on the PCB image under test, predict key regions, extract the largest connected region in the segmentation results as the effective ROI contour, and exclude the substrate background region.

[0058] The minimum bounding rectangle of the effective ROI contour is calculated using the minimum bounding rectangle algorithm. The rotation angle of the rectangle is obtained, and an affine transformation matrix is ​​constructed with the center of the minimum bounding rectangle as the rotation center and the rotation angle of the rectangle as the rotation angle. Image rotation correction is then performed.

[0059] Create a binary mask for the ROI from the corrected image, scale it by a certain scaling factor, extract feature points and descriptors using the SIFT algorithm, and cache the coordinates of the scaled image, feature points, descriptors, and the corresponding ROI contours.

[0060] Understandably, using deep learning models for semantic image segmentation is a common technique in existing technologies, and will not be elaborated upon here. The key regions predicted can be the regions of interest containing printed circuit lines and pads. By excluding the substrate background region, the amount of data required for subsequent processing can be reduced, thus solving the problem of high computational cost for full-image detection.

[0061] Simultaneously, performing image rotation correction can eliminate angular deviations caused by the tilt of the PCB substrate, improving the accuracy of subsequent registration and laying the foundation for subsequent registration. Here, a scaling factor of 0.25 can be used, while the scaling factor in the affine transformation matrix can be 1.0.

[0062] In addition, caching the scaled image, feature points, descriptors, and the corresponding coordinates of the corrected ROI contours involves storing the scaled image, feature points, descriptors, and the corresponding coordinates of the corrected ROI contours in a memory cache, which avoids redundant calculations and thus improves detection efficiency.

[0063] In some embodiments, to provide a feasible method for aligning the ROI of a preset standard image and the ROI of a PCB image under test using a multi-scale feature registration method, aligning the ROI of the preset standard image and the ROI of the PCB image under test using a multi-scale feature registration method may include:

[0064] A multi-scale feature registration method based on the SIFT and RANSAC algorithms is used to align the ROI of a preset standard image with the ROI of the PCB image under test.

[0065] Here, a multi-scale feature registration method based on the SIFT and RANSAC algorithms is used to align the ROI of the preset standard image and the ROI of the PCB image under test, and the intersection region of the ROIs is obtained, which may include:

[0066] Obtain the SIFT feature descriptor of the pre-stored standard image, and use the FLANN matcher to match the SIFT feature descriptor of the standard image with the descriptor of the PCB image to be tested in the cache;

[0067] Based on the matching results, the 8-parameter perspective transformation matrix is ​​solved by robust estimation using the RANSAC algorithm. The ROI of the PCB image under test is then transformed by perspective to achieve pixel-level alignment between the ROI of the PCB image under test and the ROI of the standard image.

[0068] The ROI coordinates of the standard image are mapped to the image space of the ROI of the PCB image under test through a perspective transformation matrix. The intersection region of the two ROI bounding boxes is calculated. The overlapping region in the ROI of the PCB image under test is retained as the ROI intersection region, and the remaining regions are deleted.

[0069] Understandably, matching the SIFT feature descriptors of the standard image with the descriptors of the PCB image under test in the cache using the FLANN matcher is a coarse match. The FLANN matcher can be configured with the FLANN_INDEX_KDTREE algorithm, trees=5, checks=50; in addition, a ratio test (taking 0.7 × nearest neighbor distance) can be performed to filter out high-quality matching pairs and eliminate false matches, ensuring a matching accuracy of ≥85%. The matching results required subsequently can then be the filtered high-quality matching pairs.

[0070] When using the RANSAC algorithm for robust estimation, the reprojection error threshold can be set to 5.0. By achieving pixel-level alignment between the ROI of the PCB image under test and the ROI of the standard image, the feature misalignment problem caused by deformation can be resolved.

[0071] In addition, since the overlapping areas in the ROI of the PCB image under test are retained as the ROI intersection area and the rest of the area is deleted, the subsequent detection range is only in the ROI intersection area, which further reduces redundant calculations.

[0072] In some embodiments, to provide a feasible method for performing sliding window detection based on a parallel detection architecture using ROI intersection regions to obtain a full-size defect mask and complete defect detection, the method may include:

[0073] The detection range is the intersection area of ​​ROIs. A 512×512 pixel window with a step size of 256 pixels is used for sliding. Mirror filling is used for areas where the boundary is insufficient for the window size.

[0074] Using a Siamese U-Net network with pre-defined shared weights, the ROI of the registered and aligned standard image and the ROI of the PCB image under test are used as dual-branch inputs to obtain the window defect probability map for each thread.

[0075] The window defect probability maps of each thread are stitched together according to their coordinate positions. If overlapping areas are encountered during stitching, the maximum probability is taken, and the full-size defect mask is obtained by fusion.

[0076] It is understandable that in the above embodiments, mirroring is used to fill areas where the window size is insufficient to ensure that there are no blind spots in the detection. Furthermore, if overlapping areas are encountered during stitching, the maximum probability is used to avoid missing detections at window boundaries.

[0077] Here, a feasible pre-defined structure for a Siamese U-Net network with shared weights is provided. The Siamese U-Net network with shared weights may include:

[0078] A shared encoder is used to simultaneously extract features from at least two levels of dual-branch input, enabling parameter sharing during the feature extraction process;

[0079] The multi-type difference calculation module includes at least two difference calculation units corresponding to the output of the shared encoder. Each difference calculation unit is used to first calculate the absolute difference and cosine similarity of the features corresponding to the two-branch input. Here, the cosine similarity is calculated along the channel dimension and then expanded to the original number of channels to obtain the difference features. The difference features are then concatenated along the channel dimension and fused through a two-level convolutional network to output a difference feature map with the same number of input feature channels. The two-level convolutional network is composed of 3×3 convolution, batch normalization, and ReLU.

[0080] The decoder includes a DecoderBlock with skip connections and an UPConvBlock. The number of DecoderBlocks is one less than the number of difference computation units. Each pair of adjacent difference computation units corresponds to one DecoderBlock, and there is one UPConvBlock. The DecoderBlock adjusts the number of channels by combining bilinear upsampling with 1×1 convolution, and concatenates the channels with the difference feature maps corresponding to the output of the shared encoder layer. Then, the features are further fused through convolutional layers. The UPConvBlock uses upsampling and channel adjustment strategies to complete the final feature recovery.

[0081] The prediction module is used to output a window defect probability map by combining 3×3 convolution with batch normalization and ReLU, Dropout, 1×1 convolution and Sigmoid activation function. If the size of the output window defect probability map is inconsistent with the input window, it is adjusted to the input size by bilinear interpolation to ensure that the output matches the input size.

[0082] In the shared encoder, ResNet34 can be used as the backbone network and configured to output only feature mode, loading pre-trained weights. However, with at least two feature levels, too few levels will lead to a decrease in final detection accuracy, while too many levels may result in increased false positives due to excessive detail. Therefore, 3-5 levels are recommended, with 4 levels being preferred.

[0083] In the decoder's DecoderBlock, the number of channels is adjusted through bilinear upsampling combined with 1×1 convolutions, and the scaling factor for this bilinear upsampling can be set to 2.0. Furthermore, before channel concatenation of the difference feature maps corresponding to the shared encoder layer outputs, size matching can be performed on these difference feature maps.

[0084] The decoder may also include channel attention units and spatial attention units that correspond one-to-one with the DecoderBlock. The output of each DecoderBlock is connected to the channel attention unit, and the output of the UPConvBlock is connected to the spatial attention unit.

[0085] Here, the channel attention unit obtains channel feature statistics through adaptive average pooling and adaptive max pooling, calculates channel weights through convolutional layers and assigns them to each channel, thereby strengthening key difference features;

[0086] The spatial attention unit obtains spatial feature information through average pooling and max pooling in the channel dimension, calculates spatial weights through 7×7 convolution and distributes them to each pixel, thereby focusing on the defect area.

[0087] After practical verification, the multi-type difference calculation (absolute difference + cosine similarity) and channel-space attention enhancement based on the ResNet34 encoder achieved a detection rate of over 98.5% for micron-level defects (burrs with a width ≥ 5μm and pad contamination with an area ≥ 10μm²). Compared with the Siamese network that only uses the highest layer feature difference, the recall rate of small defects was improved by 30%, and the false negative rate was reduced by 60% compared with traditional machine vision.

[0088] In some embodiments, to provide a feasible training method for a Siamese U-Net network with preset shared weights, the training method for the Siamese U-Net network with preset shared weights may include:

[0089] Obtain standard and defect images of the PCB board;

[0090] The standard image preprocessing includes: traversing all standard images; for any given standard image, extracting ROI point coordinates to obtain the ROI contour; rotating the standard image to a horizontal position based on the ROI contour; calculating the ROI bounding box and ROI mask; extracting SIFT feature points and descriptors; obtaining a preprocessing result containing the rotated image, ROI information, and SIFT feature points and descriptors; and storing the preprocessing result in a cache according to filename. The ROI information includes ROI point coordinates, ROI contour, ROI bounding box, and ROI mask.

[0091] Image pair matching is performed, including: traversing all defect images, matching the corresponding standard image according to the file name, collecting all valid matching image pairs, and ensuring that each image pair corresponds to different states of the same PCB board;

[0092] The processing of a single image pair includes: reading the defect image and extracting its ROI point coordinates and defect annotations to obtain the ROI contour; rotating the defect image to a horizontal position based on the ROI contour; using SIFT feature matching to register the ROI of the standard image to the coordinate system of the ROI of the defect image; calculating the intersection of the ROI bounding box of the standard image and the ROI bounding box of the defect image to obtain the ROI intersection region; if there is no intersection, skipping the image pair; transforming the coordinates of the defect annotations in the defect image to the coordinate system of the ROI of the rotated defect image; performing sliding window cropping within the ROI intersection region to obtain the cropped region; calculating the SSIM value of the cropped region and filtering out difference image pairs with a value lower than min_ssim; extracting the defect annotations within the cropped region and transforming their coordinates to the coordinate system of the cropped image; and collecting information on all valid cropped image pairs as samples for training.

[0093] The training configuration of the Siamese U-Net network with shared weights includes: initializing the optimizer, configuring the loss function, and configuring the learning rate scheduler;

[0094] The samples were used to train the Siamese U-Net network with shared weights after the training configuration was completed.

[0095] Here, when preprocessing the standard image to extract ROI point coordinates, one can first read the standard image file and its corresponding JSON annotation file, and then extract the ROI point coordinates from the JSON annotation file. Alternatively, an ROI extraction model can be used to extract the ROI point coordinates, in which case the JSON annotation file is not needed. Furthermore, before extracting SIFT feature points and descriptors, the image can be scaled by the scale_factor.

[0096] Similarly, when processing a single image pair, in reading the defective image and extracting its ROI coordinates and defect annotations, one can first read the image file of the defective image and its corresponding JSON annotation file, and then extract the ROI coordinates and defect annotations from the JSON annotation file. Alternatively, an ROI extraction model can be used to extract the ROI coordinates and defect annotations, in which case the JSON annotation file is not needed. When using SIFT feature matching to register the ROI of the standard image to the coordinate system of the ROI of the defective image, the align_images_with_sift algorithm can be called. If the registration fails, such as due to insufficient matching points or lack of a transformation matrix, the rotated original standard image can be used as the ROI of the standard image to avoid subsequent calculation failures.

[0097] Additionally, when performing sliding window cropping within the intersection area of ​​the ROIs, the cropping area can be obtained by generating the sliding window position according to window_size (512,512) and stride (256,256) to crop the ROI of the registered standard image and the ROI of the rotated defective image.

[0098] In the above embodiments, min_ssim can be set according to actual needs, which will not be detailed here. Furthermore, when filtering out image pairs with discrepancies below min_ssim, the filtered image pairs can be saved as negative samples for training to prevent over-detection by the model.

[0099] When configuring the training of the Siamese U-Net network with shared weights, the Adam optimizer can be used during optimizer initialization, with the initial learning rate set to args.lr, which can actually be 0.001. When configuring the loss function, the DiceFocalLoss combined loss function can be used to balance the segmentation of minor defects and class imbalance, with parameters α=0.7, β=0.3, and γ=2.0, where α is the Dice loss weight, β is the Focal loss weight, and γ is the Focal loss focusing coefficient for difficult-to-separate samples. When configuring the learning rate scheduler, the ReduceLROnPlateau scheduler can be configured, which dynamically adjusts the learning rate based on the validation set loss, improving training stability.

[0100] When training the Siamese U-Net network with shared weights after the training configuration is completed using the aforementioned samples, after each training round, if the average Dice coefficient of the validation set refreshes to the optimal value, the core parameters of the model are saved using `model.state_dict()`. The training environment for the aforementioned Siamese U-Net network with shared weights can be:

[0101] Hardware configuration: NVIDIA GeForce RTX 3090 with at least 10GB of video memory to support large-scale data training; CPU: Intel Core i9-12th generation; RAM: 64GB.

[0102] Software configuration: Operating system Windows 11; deep learning framework PyTorch 1.12; in addition, it is necessary to install dependency libraries such as OpenCV, NumPy, Matplotlib, and Scikit-learn.

[0103] Understandably, the aforementioned Siamese U-Net network with shared weights adopts a dual-branch contrastive learning mode, which only requires 20 to 50 standard images and at least 3 defect images for each type of defect as samples for training, without the need for massive amounts of labeled data; when adapting to different PCB models, it only needs to update the samples and retrain, shortening the adaptation cycle to 1 to 2 days, meeting the rapid switching requirements of flexible production.

[0104] In some embodiments, to further improve the accuracy and quality of defect location, the following may also be included:

[0105] Morphological enhancement and contour filtering operations are performed on the acquired full-size defect mask to enhance the continuity of the defect contour and eliminate noise interference.

[0106] To provide a feasible morphological enhancement and contour filtering operation, morphological enhancement may include:

[0107] A 5×5 rectangular structural element is used to perform a closing operation on the full-size defect mask, filling the tiny holes in the corresponding defect area and enhancing the continuity of the defect outline.

[0108] Contour filtering may include:

[0109] The contour extraction algorithm extracts the defect contour corresponding to the full-size defect mask. The retrieval mode of the contour extraction algorithm is set to only the outer contour, filtering out defect contours with an area of ​​less than or equal to 100 pixels. It is also verified whether the unfiltered defect contours are located in the ROI intersection area. If so, they are retained; otherwise, they are deleted. This can eliminate noise interference and improve the accuracy and quality of defect localization.

[0110] In addition, the ROI of the preset standard image mentioned in the above embodiments can be the ROI of the standard image obtained during training, or it can be the ROI of the reselected standard image. The appropriate selection can be made according to the actual situation, which will not be described in detail here.

[0111] The following example illustrates the above embodiments:

[0112] First, hardware configuration is required, including setting up the image acquisition module, computing module, and corresponding mechanical structure.

[0113] Here, the image acquisition module uses a Hikvision 20-megapixel industrial area scan camera (resolution 5472×3648) with a fixed-focus lens; the light source uses a dual-strip white LED light source, with the light source angle parallel to the PCB board surface and a height of 30cm.

[0114] Computing module: Configured with Intel Core i7-10700K CPU (8 cores and 16 threads), 32GB DDR4 memory, and 1TB NVMe SSD, no GPU acceleration module required.

[0115] Mechanical structure: A manual stage (accuracy ±0.1mm) is used. The camera is fixed directly above the stage. The vertical distance between the camera and the PCB is adjusted to 40cm by the lifting bracket to ensure that each pixel corresponds to an actual size of 10μm.

[0116] Then, the software environment was set up: the operating system was Windows 10 64-bit Professional Edition, the development framework was Python 3.8, and the dependent libraries included OpenCV 4.5.5 (image preprocessing and registration), PyTorch 1.12.1 (deep learning model training and inference), Scikit-image 0.19.3 (morphological operations), and NumPy 1.23.5 (numerical computation).

[0117] The specific testing process is as follows:

[0118] First, image acquisition and preprocessing are performed, including:

[0119] A defect-free standard PCB board is placed in the center of the stage. Dual strip light sources are turned on, and three images are acquired using an industrial area scan camera (exposure time 10ms, gain 1.0). The image with the highest clarity is selected as the standard image for subsequent use (i.e., a reselected standard image, not the ROI of the standard image obtained during training). The PCB board to be tested is then replaced with one that has been pre-fabricated with burr defects, pad scratches, and circuit break defects. Three images are acquired, and the image with the highest clarity is selected as the PCB image to be tested.

[0120] Then, key region of interest (ROI) extraction is performed. Since the standard image is a newly selected one, ROI extraction should also be performed on the standard image during key region of interest (ROI) extraction, including:

[0121] The lightweight semantic segmentation model U2Net was used to segment the standard image and the PCB image under test, respectively. The model input size was 480×704 pixels. During training, its dataset contained 100 labeled images of different PCBs (labeled circuit and pad areas). After training, the segmentation accuracy reached 99.77%. The largest connected component in the segmentation results was extracted to generate ROI contours (ROI contours of the standard image and ROI contours of the PCB image under test).

[0122] The minimum bounding rectangle of the ROI contour (both the ROI contour of the standard image and the ROI contour of the PCB image under test) is calculated using the cv2.minAreaRect() function of OpenCV, and the rotation angle θ = 2.3° is obtained. With the center of the rectangle (rect_center = (2736, 1824) pixels) as the rotation center, an affine transformation matrix is ​​constructed, and the cv2.warpAffine() function is called to perform rotation correction. After correction, the ROI tilt angle is ≤ 0.1°.

[0123] The corrected ROI is converted into a binary mask (threshold 127), scaled to 1368×912 pixels by a ratio of 0.25, and the cv2.SIFT_create() function is used to extract feature points (about 8000) and a 128-dimensional SIFT descriptor. The scaled image, feature points, SIFT descriptor and ROI coordinates (x=100, y=100, w=5272, h=3448 pixels) are stored in the memory cache. The cache is reset after switching to the standard image.

[0124] Then image registration and alignment are performed, including:

[0125] The SIFT descriptor of the standard image is read from the memory cache, and the FLANN matcher is initialized (configured as FLANN_INDEX_KDTREE=1, trees=5, checks=50). Coarse matching is performed with the SIFT descriptor extracted from the ROI of the PCB image under test, resulting in approximately 3000 matching pairs. A ratio test (using 0.7 × nearest neighbor distance) is then performed to filter out 1200 high-quality matching pairs, achieving a matching accuracy of 97%.

[0126] Based on high-quality matching pairs, the cv2.findHomography() function is called, and ransacReprojThreshold=5.0 is set to robustly estimate the 8-parameter perspective transformation matrix H (3×3 matrix); perspective transformation is then performed on the ROI of the PCB image under test.

[0127] The ROI coordinates of the standard image (e.g., x=100, y=100, w=5272, h=3448) are mapped to the image space of the ROI of the PCB image under test through matrix H, resulting in the mapped ROI coordinates of the PCB image under test (e.g., x=105, y=98, w=5268, h=3452). The intersection region of the two ROI bounding boxes is calculated as (x=105, y=100, w=5267, h=3448), which is used as the subsequent detection range.

[0128] Then perform sliding window testing, including:

[0129] Using the intersection area as the detection range, a 512×512 pixel window with a step size of 256 pixels is used for sliding, and a total of ((5267-512) / 256 +1)×((3448-512) / 256 +1)≈20×12=240 sliding windows are generated; for areas where the boundary is insufficient for the window size (such as 100 pixels remaining on the right and 80 pixels remaining at the bottom), mirror filling is used.

[0130] A pre-trained Siamese U-Net model was loaded (using the DiceFocal combined loss function during training, α=0.7, β=0.3, γ=2.0, 100 training epochs, batch size=8), and an 8-thread pool was enabled to process each sliding window in parallel. The window images of the ROI of the registered standard image and the ROI of the PCB image under test were used as dual-branch inputs. A ResNet34 encoder extracted four levels of features (64, 128, 256, and 512 channels). The absolute difference and cosine similarity features were obtained through a difference calculation module and fused. The decoder upsampled through three DecoderBlocks and one UPConvBlock, combined with channel-spatial attention enhancement. The final output was a 512×512 pixel defect probability map (with a probability threshold of 0.5 after sigmoid activation).

[0131] Collect the defect probability maps of 240 windows, stitch them together according to their coordinate positions, and take the maximum probability of the overlapping areas (256-pixel step size results in 256-pixel overlap) to fuse them, thus obtaining a full-size (5267×3448 pixels) defect mask.

[0132] Finally, post-processing and defect assessment are performed, including:

[0133] Morphological enhancement is performed by using a 5×5 rectangular structuring element (cv2.getStructuringElement(cv2.MORPH_RECT, (5,5))) to perform a closing operation, filling in tiny holes (diameter ≤ 3 pixels) within the defect area.

[0134] Contour filtering involves calling the OpenCV contour extraction function to extract defect contours, calculating the area of ​​each contour, and filtering contours with an area ≤ 100 pixels (corresponding to an actual area of ​​10000μm²). It verifies whether the center of the remaining contours is located in the intersection area of ​​the ROI, and finally detects three defects: burr defects, pad scratches, and circuit breaks, which are completely consistent with the preset defects.

Claims

1. A PCB board defect detection method based on multi-level feature fusion, characterized in that, Includes the following steps: Obtain the image of the PCB to be tested; The ROI of key areas is extracted from the PCB image to be tested, the substrate background area is excluded, and the extracted ROI is cached. A multi-scale feature registration method is used to align the ROI of the preset standard image and the ROI of the PCB image under test, and the intersection area of ​​the ROI is obtained. Based on the intersection region of ROI, a parallel detection architecture is used to perform sliding window detection to obtain a full-size defect mask and complete the defect detection.

2. The PCB board defect detection method based on multi-level feature fusion as described in claim 1, characterized in that, The acquisition of the PCB image to be tested includes: An industrial area scan camera is used to acquire images of the PCB under test. The acquisition environment uses a dual-strip white light source. During acquisition, the vertical distance between the camera head and the PCB image under test is controlled to be 30-50cm.

3. The PCB board defect detection method based on multi-level feature fusion as described in claim 1, characterized in that, The process of extracting the Region of Interest (ROI) of key areas from the PCB image under test, excluding the substrate background area, and caching the extracted ROI data includes: A pre-defined deep learning model is used to perform semantic segmentation on the PCB image under test, predict key regions, extract the largest connected region in the segmentation results as the effective ROI contour, and exclude the substrate background region. The minimum bounding rectangle of the effective ROI contour is calculated using the minimum bounding rectangle algorithm. The rotation angle of the rectangle is obtained, and an affine transformation matrix is ​​constructed with the center of the minimum bounding rectangle as the rotation center and the rotation angle of the rectangle as the rotation angle. Image rotation correction is then performed. Create a binary mask for the ROI from the corrected image, scale it by a certain factor, extract feature points and descriptors using the SIFT algorithm, and cache the coordinates of the scaled image, feature points, descriptors, and the corresponding ROI contours.

4. The PCB board defect detection method based on multi-level feature fusion as described in claim 1, characterized in that, The method of aligning the ROI of a preset standard image and the ROI of the PCB image under test using a multi-scale feature registration method includes: A multi-scale feature registration method based on the SIFT and RANSAC algorithms is used to align the ROI of a preset standard image with the ROI of the PCB image under test.

5. The PCB board defect detection method based on multi-level feature fusion as described in claim 4, characterized in that, The multi-scale feature registration method using the SIFT and RANSAC algorithms aligns the ROI of the preset standard image with the ROI of the PCB image under test, and obtains the ROI intersection region, including: Obtain the SIFT feature descriptor of the pre-stored standard image, and use the FLANN matcher to match the SIFT feature descriptor of the standard image with the descriptor of the PCB image to be tested in the cache; Based on the matching results, the 8-parameter perspective transformation matrix is ​​solved by robust estimation using the RANSAC algorithm. The ROI of the PCB image under test is then transformed by perspective to achieve pixel-level alignment between the ROI of the PCB image under test and the ROI of the standard image. The ROI coordinates of the standard image are mapped to the image space of the ROI of the PCB image under test through a perspective transformation matrix. The intersection region of the two ROI bounding boxes is calculated. The overlapping region in the ROI of the PCB image under test is retained as the ROI intersection region, and the remaining regions are deleted.

6. The PCB board defect detection method based on multi-level feature fusion as described in claim 1, characterized in that, The parallel detection architecture based on the intersection region of the ROI is used for sliding window detection to obtain a full-size defect mask, thus completing the defect detection, including: The detection range is the intersection area of ​​ROIs. A 512×512 pixel window with a step size of 256 pixels is used for sliding. Mirror filling is used for areas where the boundary is insufficient for the window size. Using a Siamese U-Net network with pre-defined shared weights, the ROI of the registered and aligned standard image and the ROI of the PCB image under test are used as dual-branch inputs to obtain the window defect probability map for each thread. The window defect probability maps of each thread are stitched together according to their coordinate positions. If overlapping areas are encountered during stitching, the maximum probability is taken, and the full-size defect mask is obtained by fusion.

7. The PCB board defect detection method based on multi-level feature fusion as described in claim 6, characterized in that, The Siamese U-Net network with shared weights includes: A shared encoder is used to simultaneously extract features from at least two levels of dual-branch input, enabling parameter sharing during the feature extraction process; The multi-type difference calculation module includes at least two difference calculation units corresponding to the output of the shared encoder. Each difference calculation unit is used to first calculate the absolute difference and cosine similarity of the features corresponding to the two-branch input. Here, the cosine similarity is calculated along the channel dimension and then expanded to the original number of channels to obtain the difference features. The difference features are then concatenated along the channel dimension and fused through a two-level convolutional network to output a difference feature map with the same number of input feature channels. The two-level convolutional network is composed of 3×3 convolution, batch normalization, and ReLU. The decoder includes a DecoderBlock and an UPConvBlock with skip connections. The number of DecoderBlocks is one less than the number of difference computation units. Each pair of adjacent difference computation units corresponds to one DecoderBlock, and there is one UPConvBlock. The DecoderBlock adjusts the number of channels by combining bilinear upsampling with 1×1 convolution, and concatenates the channels with the difference feature maps corresponding to the output of the shared encoder layer. Then, the features are further fused through convolutional layers. The UPConvBlock uses upsampling and channel adjustment strategies to complete the final feature recovery. The prediction module is used to output a window defect probability map by combining 3×3 convolution with batch normalization and ReLU, Dropout, 1×1 convolution and Sigmoid activation function. If the size of the output window defect probability map is inconsistent with the input window, it is adjusted to the input size by bilinear interpolation.

8. The PCB board defect detection method based on multi-level feature fusion as described in claim 7, characterized in that, The training method for the pre-defined Siamese U-Net network with shared weights includes: Obtain standard and defect images of the PCB board; The standard image preprocessing includes: traversing all standard images; for any given standard image, extracting ROI point coordinates to obtain the ROI contour; rotating the standard image to a horizontal position based on the ROI contour; calculating the ROI bounding box and ROI mask; extracting SIFT feature points and descriptors; obtaining a preprocessing result containing the rotated image, ROI information, and SIFT feature points and descriptors; and storing the preprocessing result in a cache according to filename. The ROI information includes ROI point coordinates, ROI contour, ROI bounding box, and ROI mask. Image pair matching is performed, including: traversing all defect images, matching the corresponding standard image according to the file name, collecting all valid matching image pairs, and ensuring that each image pair corresponds to different states of the same PCB board; The processing of a single image pair includes: reading the defect image and extracting its ROI point coordinates and defect annotations to obtain the ROI contour; rotating the defect image to a horizontal position based on the ROI contour; using SIFT feature matching to register the ROI of the standard image to the coordinate system of the ROI of the defect image; calculating the intersection of the ROI bounding box of the standard image and the ROI bounding box of the defect image to obtain the ROI intersection region; if there is no intersection, skipping the image pair; transforming the coordinates of the defect annotations in the defect image to the coordinate system of the ROI of the rotated defect image; performing sliding window cropping within the ROI intersection region to obtain the cropped region; calculating the SSIM value of the cropped region and filtering out difference image pairs with a value lower than min_ssim; extracting the defect annotations within the cropped region and transforming their coordinates to the coordinate system of the cropped image; and collecting information on all valid cropped image pairs as samples for training. The training configuration of the Siamese U-Net network with shared weights includes: initializing the optimizer, configuring the loss function, and configuring the learning rate scheduler; The samples were used to train the Siamese U-Net network with shared weights after the training configuration was completed.

9. The PCB board defect detection method based on multi-level feature fusion as described in any one of claims 1-8, characterized in that, Also includes: Morphological enhancement and contour filtering operations are performed on the acquired full-size defect mask to enhance the continuity of the defect contour and eliminate noise interference.

10. The PCB board defect detection method based on multi-level feature fusion as described in claim 9, characterized in that, The morphological enhancements include: A 5×5 rectangular structural element is used to perform a closing operation on the full-size defect mask, filling the tiny holes in the corresponding defect area and enhancing the continuity of the defect outline. The contour filtering includes: The contour extraction algorithm is used to extract the defect contour corresponding to the full-size defect mask. The search mode of the contour extraction algorithm is set to only the outer contour, filtering out defect contours with an area of ​​less than or equal to 100 pixels. It is also verified whether the unfiltered defect contours are located in the ROI intersection area. If so, they are retained; otherwise, they are deleted.

Images