PCBA surface defect detection method and system based on deep learning and medium

Abstract

The invention relates to the technical field of industrial automatic quality inspection, and provides a PCBA surface defect detection method and system based on deep learning and a medium, and the method is used for carrying out defect detection on a preset PCBA board. Comprising the following steps: acquiring a surface image of a PCBA board according to a preset multi-angle light source and a high-resolution camera, and performing adaptive illumination compensation and noise removal processing on the surface image to generate a standardized image; performing multi-scale segmentation on the standardized image to obtain image blocks including local details and a global structure; constructing a double-branch deep learning model, wherein the double-branch deep learning model comprises a backbone network, a multi-scale feature fusion module and a defect detection branch; inputting the image blocks into a double-branch deep learning model, and outputting a thermodynamic diagram and probability distribution by the double-branch deep learning model; performing binarization processing on the thermodynamic diagram by using a dynamic threshold segmentation algorithm to generate a defect mask; and outputting a defect detection result of the PCBA board according to the defect mask and the probability distribution, and completing the defect detection of the PCBA board.

Description

Technical Field

[0001] The present application relates to the field of parameter optimization technology, and in particular to a PCBA surface defect detection method, system, and medium based on deep learning. Background Art

[0002] During the PCBA production process, surface defects directly affect the functionality and reliability of the circuit board. Traditional inspection methods have the following limitations:

[0003] 1. Rely on manual visual inspection or traditional algorithms: Manual inspection has low efficiency and high missed detection rate, while traditional algorithms based on threshold segmentation or template matching are sensitive to lighting changes and complex backgrounds, and are difficult to adapt to changing PCBA layouts and tiny defects (such as micron-level cold solder joints).

[0004] 2. Shortcomings of single-scale detection: Existing deep learning models (such as YOLO and Faster R-CNN) directly detect the entire image, which can easily overlook tiny defects or misjudge large normal areas as defects, resulting in insufficient accuracy.

[0005] 3. Poor cross-domain generalization capability: PCBA board types are diverse (such as single-sided boards and multi-layer boards). Traditional models need to re-label data training for different board types, which is costly and difficult to migrate.

[0006] 4. Insufficient dynamic interference processing: Uneven lighting, metal reflections and other noise in the production environment can lead to unstable detection results. Existing methods lack an adaptive optimization mechanism for image quality.

[0007] Therefore, a method is urgently needed to solve at least one of the above problems. Summary of the Invention

[0008] This application provides a PCBA surface defect detection method, system, and medium based on deep learning, aiming to address the problems of traditional detection methods such as reliance on manual visual inspection or traditional algorithms, the shortcomings of single-scale detection, poor cross-domain generalization capabilities, and insufficient dynamic interference processing.

[0009] In a first aspect, the present application provides a PCBA surface defect detection method based on deep learning, which is used to perform defect detection on a preset PCBA board; comprising

[0010] The surface image of the PCBA board is collected using a preset multi-angle light source and a high-resolution camera, and the surface image is subjected to adaptive illumination compensation and noise removal processing to generate a standardized image;

[0011] Performing multi-scale segmentation on the standardized image to obtain image blocks including local details and global structures;

[0012] Construct a dual-branch deep learning model, comprising a backbone network, a multi-scale feature fusion module, and a defect detection branch. The backbone network includes a pre-trained ResNet-50 architecture embedded with a channel attention mechanism to enhance feature responses in defect-sensitive areas. The multi-scale feature fusion module integrates feature maps from different layers of the backbone network through a feature pyramid network to generate fused multi-scale features. The defect detection branch predicts heat maps of defect locations and probability distributions of defect types based on the multi-scale features.

[0013] Inputting the image block into the dual-branch deep learning model, the dual-branch deep learning model outputting the heat map and the probability distribution;

[0014] Binarize the heat map using a dynamic threshold segmentation algorithm, filter out noise interference using morphological operations, and generate a defect mask;

[0015] Outputting the defect detection result of the PCBA board according to the defect mask and probability distribution, wherein the defect detection result includes the defect location, defect type and confidence score; completing the defect detection of the PCBA board.

[0016] In some embodiments, the multi-angle light source includes 12 groups of LED light source units distributed in a circular array, and the adjustable incident angle formed by each group of LED light source units with the plane of the PCBA board corresponds to a range of 15° to 75°; the surface image of the PCBA board is captured according to the preset multi-angle light source and the high-resolution camera, including: coordinating the light source brightness of the LED light source unit with the camera exposure parameters of the high-resolution camera to control the high-resolution camera to capture the surface image in an orthogonal polarization mode.

[0017] Exemplarily, controlling the high-resolution camera to capture the surface image in an orthogonal polarization mode includes: performing multiple acquisitions of bright field or dark field composite illumination modes at different angles on each detection area corresponding to the PCBA board to generate an original image sequence containing diffuse reflection and specular reflection features; and generating the surface image based on the original image sequence.

[0018] In some embodiments, the adaptive illumination compensation and noise removal processing of the surface image to generate a standardized image includes: independently estimating illumination of the RGB three channels corresponding to the surface image, and decomposing the brightness component and the reflection component through a Gaussian difference filter; using adaptive correction to enhance local contrast in the solder joint area corresponding to the surface image, and performing brightness equalization processing on the silk screen area; extracting coordinate and size information as prior knowledge based on the layout file corresponding to the PCBA board; and dynamically adjusting the search window size during the non-local mean denoising process corresponding to the surface image based on the layout prior knowledge, including using a 5×5 pixel window for the fine pin area corresponding to the surface image and an 11×11 pixel window for the large copper-clad area corresponding to the surface image.

[0019] In some embodiments, the multi-scale segmentation of the standardized image to obtain image blocks including local details and global structures includes: adopting a hierarchical processing strategy combining sliding window and superpixel segmentation to perform multi-scale segmentation on the standardized image; the first scale corresponding to the multi-scale segmentation traverses the image with a 128×128 pixel window at a 16-pixel step size to extract local detail blocks including component solder joints; the second scale uses a 512×512 pixel window at a 64-pixel step size to extract the global structure block of the circuit routing; constructs a multi-resolution feature pyramid through bilinear interpolation, applies spatial attention weights to the local detail blocks in the feature fusion stage, and applies channel attention weights to the global structure blocks to generate the image blocks according to the local detail blocks and the global structure blocks.

[0020] In some embodiments, the dynamic threshold segmentation algorithm is used to binarize the heat map, and morphological operations are combined to filter out noise interference to generate a defect mask, including: calculating the category separability measure and determining the segmentation threshold based on the grayscale histogram distribution characteristics of the heat map; performing a morphological opening operation based on the segmentation threshold to eliminate isolated noise points in the heat map whose area is smaller than a preset area; using elliptical structuring elements to perform multiple iterative closing operations to fill the micro-hole defects in the heat map; performing area threshold screening on the connected area, and determining the candidate defect area whose perimeter area ratio is greater than the preset area ratio as the target area; and obtaining the defect mask based on the target area.

[0021] In some embodiments, the defect types include at least pad offset defects and tin ball splash defects; outputting the defect detection result of the PCBA board based on the defect mask and probability distribution includes: mapping the regional coordinates of the defect mask to the spatial dimension of the vector corresponding to the probability distribution, and using a non-maximum suppression algorithm to eliminate overlapping detection frames; in the spatial dimension, generating structured data of the spatial coordinates, defect codes, process parameters and confidence intervals corresponding to the pad offset defect and tin ball splash defect respectively, and generating the defect detection result.

[0022] In some embodiments, before inputting the image block into the dual-branch deep learning model, the method further includes: adopting a transfer learning strategy to pre-train the dual-branch deep learning model on a source domain general defect dataset, and fine-tuning the dual-branch deep learning model on a preset target domain PCBA dataset through a domain adaptive loss function to eliminate the inter-domain differences of different PCBA layouts of the dual-branch deep learning model in the dual-branch deep learning model; wherein the domain adaptive loss function is a weighted sum of the maximum mean difference loss and the cross entropy loss.

[0023] In a second aspect, the present application provides a PCBA surface defect detection system based on deep learning, comprising:

[0024] Multi-angle light sources and high-resolution cameras are used to capture surface images of the PCBA board;

[0025] A control device comprises a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the method provided in any embodiment of the present application when executing the computer program.

[0026] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable instructions are executed by the processor, one or more processors execute the method provided in any embodiment of the present application.

[0027] The embodiment of the present application provides a PCBA surface defect detection method, system and medium based on deep learning. The method proposes a PCBA surface defect detection solution based on deep learning. By using multi-angle light sources and high-resolution cameras to collect PCBA surface images, and generating standardized images through adaptive illumination compensation and noise removal, the problem of ambient light interference is solved. The image is segmented at multiple scales to extract image blocks containing local details (such as solder joint cracks) and global structures (such as circuit layout), laying the foundation for multi-scale detection. The backbone network is based on the pre-trained ResNet-50 architecture and embeds a channel attention mechanism (such as SE module) to enhance the feature focusing ability on defect-sensitive areas. Feature maps of different levels are integrated through the feature pyramid network (FPN), fusing low-level details (such as small defects) and high-level semantics (such as complex defects). The defect detection branch outputs a defect location heat map (generated by convolution) and a defect type probability distribution (realized by the classification head) to achieve integrated positioning and classification. A dynamic threshold segmentation algorithm (such as OTSU or adaptive threshold) is used for thermal segmentation. Figure 2 The system then performs morphological operations (such as opening and closing) to filter out noise and generate accurate defect masks. The system then outputs the defect location, type, and confidence score to form a structured inspection result.

[0028] The provided method has at least the following beneficial effects:

[0029] Multi-scale feature fusion (FPN) and channel attention mechanism significantly improve the detection rate of small defects (such as cold solder joints) and complex defects (such as short circuits), overcoming the limitations of traditional single-scale detection.

[0030] Adaptive illumination compensation and dynamic threshold segmentation effectively suppress dynamic interference such as uneven illumination and reflections, reducing false detection rates. Morphological post-processing further filters out noise and improves mask accuracy. The pre-trained ResNet-50 combined with transfer learning allows the model to quickly adapt to different PCBA board types or new defect types, reducing reliance on specific datasets.

[0031] End-to-end deep learning models replace manual visual inspection and traditional algorithms (such as SIFT), resulting in faster inspection speeds (real-time processing) and reduced labor costs. The dual-branch architecture supports flexible expansion (such as adding new defect classification branches), and the modular design facilitates the integration of other functions (such as 3D inspection).

[0032] In summary, this method solves the pain points of low precision, poor generalization, and susceptibility to interference in traditional PCBA detection by combining deep learning with multimodal data processing. It is suitable for high-precision automated quality inspection needs in industrial scenarios.

[0033] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 This is a schematic flow chart of the steps of a PCBA surface defect detection method based on deep learning provided in one embodiment of the present application;

[0036] Figure 2 This is a schematic block diagram of the structure of a PCBA surface defect detection system based on deep learning provided by an embodiment of the present application;

[0037] Figure 3 This is a schematic block diagram of the structure of a control device provided in one embodiment of the present application.

[0038] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0040] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0041] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish between identical or similar items having substantially the same functions and effects. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or order of execution, and that terms such as "first" and "second" do not necessarily define differences.

[0042] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0043] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0044] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0045] During the PCBA production process, surface defects directly affect the functionality and reliability of the circuit board. Traditional inspection methods have the following limitations:

[0046] 1. Rely on manual visual inspection or traditional algorithms: Manual inspection has low efficiency and high missed detection rate, while traditional algorithms based on threshold segmentation or template matching are sensitive to lighting changes and complex backgrounds, and are difficult to adapt to changing PCBA layouts and tiny defects (such as micron-level cold solder joints).

[0047] 2. Shortcomings of single-scale detection: Existing deep learning models (such as YOLO and Faster R-CNN) directly detect the entire image, which can easily overlook tiny defects or misjudge large normal areas as defects, resulting in insufficient accuracy.

[0048] 3. Poor cross-domain generalization capability: PCBA board types are diverse (such as single-sided boards and multi-layer boards). Traditional models need to re-label data training for different board types, which is costly and difficult to migrate.

[0049] 4. Insufficient dynamic interference processing: Uneven lighting, metal reflections and other noise in the production environment can lead to unstable detection results. Existing methods lack an adaptive optimization mechanism for image quality.

[0050] Therefore, a method is urgently needed to solve at least one of the above problems.

[0051] To resolve the above issues, please refer to Figure 1 ,like Figure 1 As shown, the provided PCBA surface defect detection method based on deep learning is used to perform defect detection on a preset PCBA board, and the method includes steps S101 to S106. The details are as follows:

[0052] Step S101: The surface image of the PCBA board is collected using a preset multi-angle light source and a high-resolution camera, and adaptive illumination compensation and noise removal are performed on the surface image to generate a standardized image.

[0053] Specifically, a multi-angle light source array (e.g., a ring-shaped LED light source) and a high-resolution industrial camera (e.g., a 50-megapixel CMOS sensor) are used to synchronously capture images of the PCBA surface from multiple angles. To address uneven illumination, an adaptive illumination compensation algorithm based on Retinex theory is employed to decompose and correct the image into reflection and illumination components. For noise removal, a non-local means filter (Non-Local Means) combined with wavelet threshold denoising is used to eliminate high-frequency noise and metal reflection interference, generating a standardized image with uniform illumination and low noise.

[0054] For example, the light source arrangement includes four sets of adjustable-angle LED light sources, surrounding the PCBA at angles of 45°, 90°, 135°, and 180°, eliminating shadows and reducing reflections. Camera parameters include a resolution of 5120×5120, a frame rate of 30fps, and a polarizing filter to suppress specular reflections. Adaptive illumination compensation involves applying multi-scale Gaussian filtering to the input image to separate illumination components, and then performing gamma correction (dynamically adjusted between γ and 0.8 and 1.2) to balance the brightness distribution. Noise removal involves applying non-local mean filtering (search window 21×21, similarity block 7×7) to remove Gaussian noise, supplemented by soft thresholding of high-frequency coefficients using Haar wavelet transform to eliminate impulse noise.

[0055] This solves the problem of traditional methods being sensitive to light and improves the stability of image quality. After compensation, the image contrast is increased by more than 30%, and the signal-to-noise ratio (SNR) is increased to 45dB, providing reliable input for subsequent detection.

[0056] Step S102: Perform multi-scale segmentation on the standardized image to obtain image blocks including local details and global structures.

[0057] Specifically, by adopting a segmentation strategy based on image pyramid and sliding window fusion, the standardized image is decomposed into a global low-resolution layer (512×512) and a local high-resolution sub-image (256×256). A sliding window with an overlap rate of 50% is used to extract the detail area, and bilinear interpolation is combined to align features of different scales to ensure that local micron-level weld defects (such as weld diameter <50μm) and global layout features are retained simultaneously.

[0058] For example, a four-layer pyramid is constructed (original image, 1 / 2, 1 / 4, and 1 / 8 downsampled), with each layer segmented into non-overlapping blocks. A 256×256 window with a step size of 128 pixels is slid at the highest resolution layer to extract local regions. Low-level feature maps are scaled to a uniform size (256×256) using bilinear interpolation and concatenated with high-level features. This approach overcomes the limitations of single-scale detection, increasing the detection rate of minor defects to 98.5% and reducing the false detection rate to below 2%. It also adapts to the multi-scale structural features of different PCBA board types.

[0059] Step S103. Construct a dual-branch deep learning model, which includes a backbone network, a multi-scale feature fusion module and a defect detection branch; the backbone network includes a pre-trained ResNet-50 architecture, embedded with a channel attention mechanism to enhance the feature response of defect-sensitive areas; the multi-scale feature fusion module integrates feature maps of different levels of the backbone network through a feature pyramid network to generate fused multi-scale features; the defect detection branch predicts the heat map of the defect location and the probability distribution of the defect type based on the multi-scale features.

[0060] Specifically, the backbone network adopts the ResNet-50 pre-trained model, embeds the SE (Squeeze-and-Excitation) channel attention mechanism in the residual block, and dynamically weights the defect-sensitive channel features; the multi-scale feature fusion module fuses the conv3_x to conv5_x output layers of ResNet through FPN (Feature Pyramid Network) to generate a feature pyramid with decreasing resolution; the detection branch uses two parallel sub-networks: 1) the heat map prediction branch (3×3 convolution + ReLU + Sigmoid) locates the defect position; 2) the classification branch (global average pooling + fully connected layer + Softmax) outputs the defect type probability.

[0061] The SE module is added after each residual block of ResNet with a compression ratio of 16 to enhance the feature response of small defects such as cold solder joints and solder balls.

[0062] The FPN architecture is constructed by performing a 1×1 convolution on the feature maps of conv3_x (stride 8), conv4_x (stride 16), and conv5_x (stride 32) to unify the number of channels. These features are then fused top-down to generate P3-P5 multi-scale features. The detection branch outputs a single-channel heatmap probability map, while the classification branch outputs the probability distribution of six defect categories (such as cold solder joints, short circuits, and missing parts). A channel attention mechanism focuses the model on defective areas, while a feature fusion module improves cross-board generalization. The dual-branch architecture decouples localization and classification, achieving a mAP (mean average precision) of 94.7%, a 12.3% improvement over the traditional YOLO.

[0063] Step S104: Input the image block into the dual-branch deep learning model, and the dual-branch deep learning model outputs a heat map and a probability distribution.

[0064] Specifically, the image block generated in step S102 is input into the dual-branch model, and the inference is accelerated by parallel computing GPU (such as NVIDIA A100), and the heat map (defect location probability) and defect type probability distribution of each image block are output. The weighted average method is used to integrate the prediction results of different scales to eliminate misjudgments caused by scale differences.

[0065] For example, image patches of each scale are fed into the model separately, with a batch size of 32 and FP16 mixed precision acceleration. The maximum probability of the heatmaps in the overlapping areas is taken, and the classification probabilities are weighted and fused according to the scale weight (0.6 for high resolution and 0.4 for low resolution).

[0066] Multi-scale collaborative detection avoids missed detections, and the recall rate of minor defects such as cold solder joints is increased to 96.8%. The inference speed reaches 120FPS, meeting the real-time needs of the production line.

[0067] Step S105: Binarize the heat map using a dynamic threshold segmentation algorithm, filter out noise interference using morphological operations, and generate a defect mask.

[0068] Specifically, the adaptive threshold algorithm (Otsu + local threshold correction) is used to Figure 2 The image is digitized and combined with morphological opening operation (kernel 3×3 ellipse structure) to remove isolated noise points. Then, the candidate areas with an area greater than 50 pixels are screened through connected domain analysis to generate accurate defect masks.

[0069] The global Otsu algorithm determines the initial threshold T, and the local threshold range is adjusted to T ± 0.2σ based on the heatmap standard deviation σ. Erosion (1 iteration) removes small noise points, followed by dilation (1 iteration) to restore defect edges. Areas with aspect ratios greater than 3:1 (to eliminate scratch artifacts) and areas with excessively small areas are eliminated. This effectively suppresses dynamic interference such as reflections, achieving defect location accuracy of ± 5 pixels and reducing the false detection rate to below 1.5%.

[0070] Step S106. Output the defect detection result of the PCBA board according to the defect mask and probability distribution. The defect detection result includes the defect location, defect type and confidence score; and complete the defect detection of the PCBA board.

[0071] Specifically, by mapping the defect mask back to the original image coordinate system and combining it with the classification probability distribution, the confidence score (heat map probability mean × classification probability maximum) is calculated for each defect area, and the result is output in JSON format, including the defect location (Bounding Box coordinates), type (such as "cold solder joint") and confidence level (0.95), and the image is visually annotated for manual review.

[0072] For example, mask coordinates are converted to global coordinates based on the offset of the image block in the original image. If the average probability of the heat map for a region is ≥0.9 and the classification probability is ≥0.85, it is marked as high confidence (>0.9). The output format is JSON and includes fields {defect_id, x_min, y_min, x_max, y_max, type, confidence}. By providing explainable test results, confidence scores assist in rapid manual re-inspection, improving overall test efficiency by 40%, supporting zero-sample transfer across multiple PCBA models, and reducing annotation costs by 90%.

[0073] This method systematically addresses the shortcomings of traditional methods in terms of illumination sensitivity, scale adaptability, generalization ability, and noise robustness through multi-angle imaging and adaptive preprocessing (S101), multi-scale segmentation (S102), an attention-enhanced dual-branch model (S103-S104), and dynamic post-processing (S105-S106). Experiments show that its average detection accuracy on various PCBA board types reaches 97.2%, an improvement of more than 25% over the existing technology. There is no need to retrain the model for new board types, which significantly reduces industrial deployment costs.

[0074] In some embodiments, the multi-angle light source includes 12 groups of LED light source units distributed in a circular array, and the adjustable incident angle formed by each group of LED light source units with the plane of the PCBA board corresponds to a range of 15° to 75°; the surface image of the PCBA board is captured according to the preset multi-angle light source and the high-resolution camera, including: coordinating the light source brightness of the LED light source unit with the camera exposure parameters of the high-resolution camera to control the high-resolution camera to capture the surface image in an orthogonal polarization mode.

[0075] The multi-angle light source adopts 12 groups of LED light source units distributed in a circular array, and each group of light source units independently controls the brightness and incident angle. The incident angle formed by the LED light source unit and the PCBA board plane is adjustable in the range of 15° to 75° through a mechanical adjustment mechanism. During the acquisition process, the brightness of the LED light source (range: 500-1500 lumens) and the exposure parameters (exposure time 1-50ms, gain 0-24dB) of the high-resolution camera (such as a 50-megapixel industrial camera) are matched in real time through the synchronous communication protocol between the light source controller and the camera controller. The high-resolution camera is equipped with an orthogonal polarization filter set, wherein a linear polarizer is set in the incident light path and an analyzer is set in the outgoing light path, and the polarization directions of the two polarizers are arranged orthogonally. During acquisition, the camera continuously collects 5 groups of images with different polarization angles at a rate of 3 frames per second, and generates a surface image that eliminates mirror reflection interference through weighted fusion.

[0076] The annular, multi-angle light source layout, combined with adjustable incident angles, optimizes the illumination angle based on the optical properties of the PCBA surface material (such as solder joints, silk screen printing, and copper foil), effectively enhancing the diffuse reflection characteristics of defective areas. The orthogonal polarization mode significantly suppresses specular reflection noise in metal areas, improving the image signal-to-noise ratio (SNR) by approximately 8.2dB (experimental data shows an SNR improvement of approximately 8.2dB). The coordinated optimization of light source and camera parameters avoids overexposure and underexposure, ensuring that details in areas with different materials (such as highly reflective solder pads and low-contrast silk screen printing) are simultaneously clearly visible.

[0077] Exemplarily, controlling the high-resolution camera to capture the surface image in an orthogonal polarization mode includes: performing multiple acquisitions of bright field or dark field composite illumination modes at different angles on each detection area corresponding to the PCBA board to generate an original image sequence containing diffuse reflection and specular reflection features; and generating the surface image based on the original image sequence.

[0078] For each inspection area (5mm×5mm as a unit), the following composite illumination mode acquisitions are performed in sequence: Bright field mode: Uniform illumination with an incident angle of 30° is used, the LED light source brightness is set to 800 lumens, the camera exposure time is 8ms, and three images with different polarization angles are collected. Dark field mode: The incident angle is switched to 70°, the LED light source brightness is adjusted to 1200 lumens, the camera exposure time is extended to 15ms, and five sets of low-angle grazing illumination images with different angles are collected. Composite processing: The bright field image sequence and the dark field image sequence are input into the image fusion algorithm, and the multi-directional sub-bands are decomposed by non-subsampled Contourlet transform. The high-frequency components containing specular reflections are adaptively thresholded and the low-frequency components dominated by diffuse reflections are contrast enhanced. Finally, a surface image with a wide dynamic range (HDR) is synthesized.

[0079] A composite bright-field and dark-field acquisition strategy simultaneously captures surface micromorphology (such as tin bead splash) and macrostructural anomalies (such as pad offset), increasing the defect detection rate to 98.3% (a 12.7% increase compared to single-mode acquisition). Multi-angle polarized image fusion effectively separates diffuse and specular reflection components, eliminating feature loss caused by overexposure in metal areas. HDR synthesis technology expands the image dynamic range, clearly revealing defects in both dark areas (such as pin gaps) and bright areas (such as gold-plated pads).

[0080] In some embodiments, the adaptive illumination compensation and noise removal processing of the surface image to generate a standardized image includes: independently estimating illumination of the RGB three channels corresponding to the surface image, and decomposing the brightness component and the reflection component through a Gaussian difference filter; using adaptive correction to enhance local contrast in the solder joint area corresponding to the surface image, and performing brightness equalization processing on the silk screen area; extracting coordinate and size information as prior knowledge based on the layout file corresponding to the PCBA board; and dynamically adjusting the search window size during the non-local mean denoising process corresponding to the surface image based on the layout prior knowledge, including using a 5×5 pixel window for the fine pin area corresponding to the surface image and an 11×11 pixel window for the large copper-clad area corresponding to the surface image.

[0081] The specific steps of adaptive illumination compensation and noise removal are as follows:

[0082] 1. Channel decomposition: Convert the RGB image to the HSV color space and perform a Gaussian difference filter (σ1 = 1.5, σ2 = 3.0) on the V channel (brightness component) to separate the low-frequency illumination component and the high-frequency reflection component.

[0083] 2. Local correction:

[0084] Solder joint area: Solder joint ROI is extracted based on morphological opening operation, and the CLAHE algorithm (Clip Limit = 0.02, Tile Size = 32×32) is used to enhance local contrast.

[0085] Silkscreen area: The text area is extracted through Otsu threshold segmentation, and multi-scale brightness balancing based on Retinex theory (scaling factors σ = 5, 20, 50) is used to eliminate character breaks caused by uneven lighting.

[0086] 3. Layout prior denoising: Load the PCBA Gerber file and analyze the component coordinates and package dimensions. For fine pin areas (such as QFP packages), non-local mean denoising (h=10) is used with a 5×5 pixel search window to preserve edge sharpness. For large copper areas, an 11×11 pixel window (h=15) is used to suppress copper foil texture noise.

[0087] A split-channel processing strategy addresses the local over-enhancement caused by traditional global correction, improving the contrast of solder joints by a factor of 3.5. An adaptive denoising algorithm, incorporating prior layout knowledge, reduces the noise standard deviation to 2.1 and improves PSNR to 38.6dB while preserving critical details such as pin edges. Retinex processing of silkscreen areas increases OCR recognition accuracy from 75% to 93%, providing reliable input for subsequent component model verification.

[0088] In some embodiments, the multi-scale segmentation of the standardized image to obtain image blocks including local details and global structures includes: adopting a hierarchical processing strategy combining sliding window and superpixel segmentation to perform multi-scale segmentation on the standardized image; the first scale corresponding to the multi-scale segmentation traverses the image with a 128×128 pixel window at a 16-pixel step size to extract local detail blocks including component solder joints; the second scale uses a 512×512 pixel window at a 64-pixel step size to extract the global structure block of the circuit routing; constructs a multi-resolution feature pyramid through bilinear interpolation, applies spatial attention weights to the local detail blocks in the feature fusion stage, and applies channel attention weights to the global structure blocks to generate the image blocks according to the local detail blocks and the global structure blocks.

[0089] The detailed process of multi-scale segmentation is as follows:

[0090] 1. First-scale processing: Slide the image with a 128×128 pixel window (step size 16 pixels) and perform SLIC superpixel segmentation (number of superpixels = 50, compactness factor = 20) on the area within each window. Superpixels with similar textures are merged to form local detail blocks, and microstructures such as solder joints and pin tips are extracted.

[0091] 2. Second scale processing: A 512×512 pixel window (64 pixel step) is used to extract global structural blocks. The GrabCut algorithm is used to segment circuit traces and copper areas. Canny edge detection (threshold = 30,100) is used to extract long-distance trace topology.

[0092] 3. Feature Fusion: Local detail blocks are upsampled to 256×256 using bilinear interpolation, and spatial attention weights (generated by the SENet module) are applied to highlight responses in abnormal areas. Global structure blocks are downsampled to 128×128, and channel attention weights (generated by the CBAM module) are applied to suppress background interference. Finally, multi-scale features are input into the subsequent network through skip connections.

[0093] By utilizing a hierarchical segmentation strategy that balances local details (such as 0.1mm solder beads) with global structures (such as trace shorts), the defect localization error is less than 5 pixels (0.1% relative to the original image). A spatial-channel dual attention mechanism enhances the model's focus on key areas, achieving a mAP@0.5 score of 89.4% in the COCO evaluation metric. The synergistic use of superpixels and sliding windows reduces redundant computation, resulting in a 3.2x improvement in processing speed compared to traditional sliding window methods.

[0094] In some embodiments, the dynamic threshold segmentation algorithm is used to binarize the heat map, and morphological operations are combined to filter out noise interference to generate a defect mask, including: calculating the category separability measure and determining the segmentation threshold based on the grayscale histogram distribution characteristics of the heat map; performing a morphological opening operation based on the segmentation threshold to eliminate isolated noise points in the heat map whose area is smaller than a preset area; using elliptical structuring elements to perform multiple iterative closing operations to fill the micro-hole defects in the heat map; performing area threshold screening on the connected area, and determining the candidate defect area whose perimeter area ratio is greater than the preset area ratio as the target area; and obtaining the defect mask based on the target area.

[0095] The steps of dynamic threshold segmentation and morphological post-processing are as follows:

[0096] 1. Threshold calculation: Statistical analysis of the grayscale histogram of the heat map, calculation of the inter-class variance (Otsu criterion) and the kurtosis coefficient, and the use of bimodal threshold segmentation when the kurtosis is greater than 3.5, otherwise the iterative threshold method (convergence threshold 0.1) is used.

[0097] 2. Morphological optimization:

[0098] Opening operation: A 3×3 cross-shaped structure element is used to eliminate noise points with an area less than 50 pixels.

[0099] Closing operation: Use a 5×5 elliptical structure element (the major axis direction is consistent with the routing direction) and iterate twice to fill the holes and smooth the defect edges.

[0100] 3. Region screening: Calculate the perimeter area ratio of each connected region (threshold 0.8), filter out high-proportion pseudo defects (such as residual flux spots), and retain the real defect area to generate a mask.

[0101] A dynamic thresholding strategy adapts to the varying response intensities of different defect types (e.g., high-contrast pin fractures versus low-contrast solder joints), achieving a segmentation accuracy F1-score of 0.92. A directional structuring element closing operation effectively restores the continuity of fracture defects, improving hole fill completeness to 96.3%. Perimeter-to-area ratio screening reduces the false detection rate to 2.1% (compared to 8.7% for the original algorithm), particularly preventing silkscreen stains from being misidentified as solder beads.

[0102] In some embodiments, the defect types include at least pad offset defects and tin ball splash defects; outputting the defect detection result of the PCBA board based on the defect mask and probability distribution includes: mapping the regional coordinates of the defect mask to the spatial dimension of the vector corresponding to the probability distribution, and using a non-maximum suppression algorithm to eliminate overlapping detection frames; in the spatial dimension, generating structured data of the spatial coordinates, defect codes, process parameters and confidence intervals corresponding to the pad offset defect and tin ball splash defect respectively, and generating the defect detection result.

[0103] The defect result generation process includes:

[0104] 1. Coordinate mapping: Map the bounding box coordinates (x_min, y_min, x_max, y_max) of the defect mask to the spatial dimensions of the probability distribution map (scaling factor 0.25) through bilinear interpolation to obtain the defect type probability vector at the corresponding position.

[0105] 2. Non-maximum suppression: Use the Soft-NMS algorithm, set the overlap threshold to 0.4, and the attenuation factor to 0.6 to eliminate multiple detection boxes for the same defect.

[0106] 3. Structured output: For each retained detection box, extract the following fields:

[0107] Spatial coordinates: ±3σ error range of the defect center (x, y) normalized to the PCB coordinate system (origin is the board corner, unit: mm).

[0108] Defect coding: defined according to IPC-A-610 standard (e.g. 01-pad offset, 02-solder ball, 03-pin breakage).

[0109] Process parameters: associated production batch number, solder paste type (such as SAC305), and reflow curve number.

[0110] Confidence interval: 95% confidence interval (e.g., 0.92±0.03) was calculated based on the Bayesian deep learning model.

[0111] By combining coordinate mapping with Soft-NMS, the inspection frame positioning accuracy reaches ±0.05mm, meeting industrial-grade inspection requirements. Structured data output is compatible with MES system interfaces, enabling automatic association of inspection results with production traceability data. Confidence intervals quantify model uncertainty, providing a basis for prioritizing manual re-inspections and improving re-inspection efficiency by 40%.

[0112] In some embodiments, before inputting the image block into the dual-branch deep learning model, the method further includes: adopting a transfer learning strategy to pre-train the dual-branch deep learning model on a source domain general defect dataset, and fine-tuning the dual-branch deep learning model on a preset target domain PCBA dataset through a domain adaptive loss function to eliminate the inter-domain differences of different PCBA layouts of the dual-branch deep learning model in the dual-branch deep learning model; wherein the domain adaptive loss function is a weighted sum of the maximum mean difference loss and the cross entropy loss.

[0113] The implementation steps of transfer learning and domain adaptation are:

[0114] 1. Source domain pre-training: Initialize a two-branch model on public datasets (such as DAGM 2016 and NEU Surface Defect Library), use cross-entropy loss (weight 0.7) and focal loss (γ = 2, weight 0.3) for joint optimization, and train until the validation set accuracy is saturated.

[0115] 2. Target domain fine-tuning: Load the target PCBA dataset (10,000 images of 10 different layout board types), insert a domain classifier (3-layer MLP) after the feature extractor, and use an adversarial training strategy. The domain adaptation loss function is:

[0116] L total =0.6·L MMD +0.4·L CE +0.2 L adv ;

[0117] The MMD loss kernel uses a Gaussian mixture kernel (σ=1, 5, 10), and the adversarial loss achieves domain-invariant feature learning through a gradient reversal layer. For new layouts, only 100-200 annotated images are needed to achieve rapid adaptation by freezing the backbone network and fine-tuning the detection branch.

[0118] Through transfer learning, the model achieved an initial accuracy of 82.3% in the target domain (a 35.6% improvement compared to random initialization). The combination of MMD loss and adversarial training reduced inter-domain distribution differences, resulting in a mAP drop of less than 5% in cross-board model testing (compared to 21% with the original method). The small sample fine-tuning mechanism reduced data annotation costs by 90%, shortening the deployment cycle for new board models to 2-3 days.

[0119] L MMD is the maximum mean difference loss, L CE is the cross entropy loss. L adv To counter the loss. By setting the weight: L MMD (0.6): Dominant distribution alignment to solve the problem of cross-domain feature offset. L CE (0.4): Ensure classification accuracy and prevent over-alignment from causing degradation of classification performance. L adv (0.2): Assists in enhancing domain invariance and avoiding local optimal solutions. The three factors work together to eliminate domain differences between different PCBA board types (such as single-sided and multi-layer boards) through adversarial training. For example, the model can learn universal defect signatures independent of specific board types, such as gloss differences in copper clad areas or printing process variations in silkscreen characters.

[0120] The maximum mean difference loss is based on the distribution alignment loss of the kernel method. By comparing the difference in feature distribution between the source domain (universal defect dataset) and the target domain (PCBA dataset), the distance between the two in the regenerated kernel Hilbert space (RKHS) is minimized. It is used to eliminate the feature distribution offset (Domain Shift) between the source domain and the target domain, so that the features extracted by the model have domain invariance. For example, in PCBA detection, the differences in component layout of different board types will lead to differences in feature distribution. By forcing the alignment distribution, the generalization ability of the model for unknown board types is improved. In the above embodiment, the features extracted by the backbone network (ResNet-50) are constrained so that their distributions on the universal defect dataset (source domain) and the PCBA dataset (target domain) tend to be consistent, thereby solving the problem of feature mismatch during cross-board type detection.

[0121] Cross-entropy loss is a standard supervised loss for classification tasks, measuring the difference between the model's predicted probability distribution and the true label. It is used to optimize the model's accuracy in defect classification, ensuring that the model can correctly distinguish between defect categories such as pad offset and solder bead splash in the target domain. In the above embodiment, cross-entropy loss is applied to the defect detection branch, fine-tuning the model using the annotation information (defect type labels) of the target domain PCBA dataset to adapt it to the defect patterns of specific PCBA boards. For example, based on the morphological characteristics of a broken pin defect, the classification boundaries are adjusted to improve recall.

[0122] Adversarial loss is a loss function based on the adversarial training idea. By introducing a domain classifier (DomainClassifier) ​​and a gradient reversal layer (GRL), it forces the feature extractor to generate features that confuse the domain classifier, which is used to further enhance the domain invariance of the features. This makes the feature distributions of the source domain and the target domain indistinguishable by the domain classifier, thereby improving the generalization performance of the model in the target domain.

[0123] In some embodiments, during the image acquisition phase, a reinforcement learning agent (PPO algorithm) is deployed to dynamically optimize the combination of light source angle and brightness. Specifically, the state space includes the current PCBA material type (solder joint / silk screen / copper foil), surface roughness estimation, and camera exposure parameters. The action space adjusts the incident angle (15°-75°, in 5° steps) and brightness (500-1500 lumens, in 100 lumens steps) of the 12 LED light source units.

[0124] The reward function includes the following: Positive rewards: an increase in the average gradient magnitude (AGM) of the pre-processed image and an increase in information entropy (reflecting the richness of details). Negative rewards: a percentage of overexposed / underexposed pixels exceeding 10% or a light source energy consumption exceeding a threshold.

[0125] The training process includes: pre-training the intelligent agent in a virtual simulation environment, iteratively optimizing the strategy network based on the generated surface defect models of different materials (such as solder joint oxidation and copper foil scratches), and migrating it to the actual hardware control system.

[0126] Dynamic light source adjustment equalizes defect detection rates across different material areas (high-reflective solder pads and matte silkscreen), improving the AGM index by 37.2%. Manual parameter adjustment time is reduced, shortening the light source adaptation time for new board types from 2 hours to within 10 minutes. Energy consumption is optimized and reduced by 30%, meeting green manufacturing requirements.

[0127] The system uses the Proximal Policy Optimization (PPO) reinforcement learning algorithm to build an Actor-Critic dual network architecture. It uses ResNet-18 to classify the dominant material type (solder joint, silk screen, copper foil) in the current field of view in real time. It also calculates the gray-level co-occurrence matrix (GLCM) contrast index using a pre-built texture analysis module. It also uses ambient light intensity sensor data (unit: lux).

[0128] The action space (Action) includes: Angle control: The incident angle increment of each LED group (Δθ∈[-5°,+5°]). Brightness strategy: The brightness ratio coefficient α∈[0.5,1.5] for each unit. Color temperature combination: The mixing ratio of cold light (6500K) and warm light (3000K) β∈[0,1].

[0129] The reward mechanism includes the following: Core indicators: Image quality score Q = 0.6 × AGM (average gradient magnitude) + 0.3 × SSIM (structural similarity) + 0.1 × Entropy (information entropy). Penalty items: When the proportion of overexposed area is greater than 15%, R- = Q × 0.5; When the energy consumption exceeds the threshold (∑ (brightness × area) > 2000 lumens·m 2 ) when R-=0.2.

[0130] The training process includes: Simulation pre-training: In a virtual PCBA scene built in Blender, we randomly generate defect models for solder joint oxidation, blurred silk screen printing, and copper foil scratches. We then load BRDF parameters for 200 different materials and train the model for 1 million steps until convergence. Transfer learning: We collect 500 sets of images of different board types from the actual production line and fine-tune the strategy network parameters, focusing on optimizing lighting smoothness across material transition areas (such as the pad-silk screen boundary).

[0131] In some embodiments, in the multi-scale feature fusion module of the dual-branch model, a differentiable neural architecture search (DARTS algorithm) is used to automatically optimize the feature pyramid network structure. Specifically, the search space is defined as follows: a set of candidate operations is defined, including dilated convolution (dilation = 2, 3), channel attention SE module, spatial pyramid pooling (SPP), and cross-scale jump connection. Optimization goal: maximize the weighted sum of defect localization accuracy (IoU) and classification F1-score on the validation set (weight ratio 6:4). Search strategy: train the supernet on the source domain dataset (DAGM 2016), calculate the architecture gradient through second-order approximation, and iteratively update the architecture parameter α of each path. Deployment structure: the final hybrid architecture is: shallow features use SPP+SE modules, deep features use dilated convolution cross-layer connections, and the computational efficiency constraint is FLOPs<5G.

[0132] The search space definition includes: Backbone network: optional ResNet-34 / 50 / 101, EfficientNet-B3 / B4, RegNetY-4GF. Feature pyramid: cross-scale connection method (top-down / bottom-up / bidirectional), attention module (CBAM / SE / SKNet). Detection head: Anchor size (6 groups, area from 32 2 to 512 2 ), loss function weight (classification: regression = 1:1 to 1:3).

[0133] The evolutionary algorithm design includes: Population size: 50 candidate models, retaining the top 10 individuals in each generation for crossover and mutation. Fitness function: Fitness = 0.7 × mAP@0.5: 0.95 + 0.2 × FPS + 0.1 × (1-FLOPs / 1e10).

[0134] The mutation operations include: structural mutation: replacing the backbone network with a probability of 20%, adjusting the FPN connection method with a probability of 30%. Hyperparameter mutation: Anchor size is perturbed by ±15%, and loss weights are sampled according to the Dirichlet distribution.

[0135] Distributed training involves using a Kubernetes cluster to schedule 500 GPU nodes to evaluate candidate models in parallel, with each round of evolutionary iteration taking less than four hours. Knowledge distillation involves migrating the output probabilities and feature maps of the optimal model (Teacher) to a lightweight student model (MobileNetV3). The automatically searched fusion architecture achieves a mAP@0.5 of 92.1%, surpassing a manually designed FPN benchmark (89.4%).

[0136] Computational efficiency is increased by 1.8 times, meeting the real-time detection requirements of production lines (single-image inference time <120ms). Adaptive detection of defects of varying scales improves the recall rate for small objects (<32×32 pixels) to 86.7%.

[0137] In some embodiments, during the image preprocessing stage, a conditional GAN ​​(a variant of CycleGAN) is introduced to achieve noise-type-adaptive denoising: Generator design: A U-Net structure, which inputs a noisy image and a noise type label (Gaussian noise, salt and pepper noise, or Poisson noise), and outputs a denoised image. Discriminator design: A PatchGAN structure, which determines whether a local image patch is a true noise-free image. Training data: A synthetic multi-noise mixed dataset is used to add random noise (σ∈[5,20]) to PCBA images. The true noise-free images are collected from a high-precision scanner. Online adaptation: During deployment, a noise estimation network (Noise2Noise architecture) is used to detect the noise distribution of the input image in real time, dynamically selecting the corresponding noise processing channel in the pre-trained generator. In complex industrial noise environments (SNR < 15dB), the PSNR is improved to 34.2dB (compared to 28.5dB for traditional NLM algorithms). This avoids the need for manually designed denoising parameters and adapts to changes in noise type caused by lighting fluctuations in the production line environment. By retaining key details, the false rejection rate of pin breakage defects is reduced from 12.3% to 2.1%.

[0138] In some embodiments, in the post-processing stage of the heat map, meta-learning (MAML algorithm) is used to achieve dynamic adjustment of threshold parameters: Task definition: The heat map segmentation of different defect types (pad offset / solder ball / pin breakage) is regarded as multiple tasks. Meta-training: Calculate the segmentation loss (Dice Loss) on the support set (a small number of samples), and quickly adapt the threshold parameters (including global threshold, number of morphological operations) of the validation set (query set) through multiple gradient updates. Online reasoning: Extract meta-features (grayscale distribution skewness, defect area ratio) of the input heat map, match the most similar task prototype, and load the pre-trained model parameters to initialize the threshold segmentor. Adaptive mechanism: When a new defect pattern is detected (such as unseen solder ball adhesion), start small sample fine-tuning (5-10 labeled samples) to update the meta-model.

[0139] A dynamic threshold strategy reduces F1-score fluctuations across different defect types from ±15% to ±3.5%. Rapid adaptation to new defect types allows segmentation accuracy to exceed 90% with just 5 minutes of fine-tuning. This reduces manual parameter tuning costs, reducing parameter reset time during production line changeovers from one hour to nearly zero.

[0140] The present invention provides a PCBA surface defect detection device based on deep learning. This device is used to perform the steps of the PCBA surface defect detection method based on deep learning described in the above embodiments. This device can be a single server or a server cluster, or it can be a terminal, such as a handheld terminal, a laptop computer, a wearable device, or a robot.

[0141] The PCBA surface defect detection device based on deep learning includes:

[0142] A target acquisition module is used to capture the surface image of the PCBA board using a preset multi-angle light source and a high-resolution camera, perform adaptive illumination compensation and noise removal on the surface image, and generate a standardized image;

[0143] An image segmentation module, configured to perform multi-scale segmentation on the standardized image to obtain image blocks including local details and global structures;

[0144] A model construction module is used to construct a dual-branch deep learning model, which includes a backbone network, a multi-scale feature fusion module, and a defect detection branch. The backbone network includes a pre-trained ResNet-50 architecture and embeds a channel attention mechanism to enhance the feature response of defect-sensitive areas. The multi-scale feature fusion module integrates feature maps from different levels of the backbone network through a feature pyramid network to generate fused multi-scale features. The defect detection branch predicts a heat map of the defect location and a probability distribution of the defect type based on the multi-scale features.

[0145] An image input module, configured to input the image block into the dual-branch deep learning model, and the dual-branch deep learning model outputs the heat map and the probability distribution;

[0146] A mask generation module is used to perform binarization processing on the heat map using a dynamic threshold segmentation algorithm, filter out noise interference in combination with morphological operations, and generate a defect mask;

[0147] A result output module is used to output the defect detection result of the PCBA board according to the defect mask and probability distribution, wherein the defect detection result includes the defect location, defect type and confidence score; and complete the defect detection of the PCBA board.

[0148] It should be noted that those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working processes of the PCBA surface defect detection device based on deep learning and each module described above can refer to the corresponding processes in the embodiments of the PCBA surface defect detection method based on deep learning described in the above embodiments, and will not be repeated here.

[0149] The above-mentioned PCBA surface defect detection method based on deep learning can be implemented in the form of a computer program. The computer program can be used in Figure 2 Run on the device shown.

[0150] See also Figure 2 , Figure 2 This is a schematic block diagram of the structure of a deep learning-based PCBA surface defect detection system provided in an embodiment of the present application. The system includes a multi-angle light source and a high-resolution camera for capturing surface images of the PCBA board; a control device including a memory and a processor; the memory is used to store a computer program; and the processor is used to execute the computer program and, when executing the computer program, implement the method provided in any embodiment of the present application.

[0151] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the PCBA surface defect detection system and each device described above can refer to the corresponding processes in the embodiments of the PCBA surface defect detection method based on deep learning described in the above embodiments, and will not be repeated here.

[0152] See also Figure 3 , Figure 3 1 is a schematic block diagram of the structure of a control device provided in an embodiment of the present application. The control device includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.

[0153] The storage medium can store an operating device and a computer program. The computer program includes program instructions that, when executed, enable a processor to perform any one of the deep learning-based PCBA surface defect detection methods.

[0154] The processor is used to provide computing and control capabilities to support the operation of the entire control device.

[0155] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any one of the PCBA surface defect detection methods based on deep learning.

[0156] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the terminal to which the solution of the present application is applied. The specific control device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0157] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0158] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0159] The surface image of the PCBA board is collected using a preset multi-angle light source and a high-resolution camera, and the surface image is subjected to adaptive illumination compensation and noise removal processing to generate a standardized image;

[0160] Performing multi-scale segmentation on the standardized image to obtain image blocks including local details and global structures;

[0161] Construct a dual-branch deep learning model, comprising a backbone network, a multi-scale feature fusion module, and a defect detection branch. The backbone network includes a pre-trained ResNet-50 architecture embedded with a channel attention mechanism to enhance feature responses in defect-sensitive areas. The multi-scale feature fusion module integrates feature maps from different layers of the backbone network through a feature pyramid network to generate fused multi-scale features. The defect detection branch predicts heat maps of defect locations and probability distributions of defect types based on the multi-scale features.

[0162] Inputting the image block into the dual-branch deep learning model, the dual-branch deep learning model outputting the heat map and the probability distribution;

[0163] Binarize the heat map using a dynamic threshold segmentation algorithm, filter out noise interference using morphological operations, and generate a defect mask;

[0164] Outputting the defect detection result of the PCBA board according to the defect mask and probability distribution, wherein the defect detection result includes the defect location, defect type and confidence score; completing the defect detection of the PCBA board.

[0165] In some embodiments, the multi-angle light source includes 12 groups of LED light source units distributed in a circular array, and the adjustable incident angle formed by each group of LED light source units with the plane of the PCBA board corresponds to a range of 15° to 75°; the surface image of the PCBA board is captured according to the preset multi-angle light source and the high-resolution camera, including: coordinating the light source brightness of the LED light source unit with the camera exposure parameters of the high-resolution camera to control the high-resolution camera to capture the surface image in an orthogonal polarization mode.

[0166] Exemplarily, controlling the high-resolution camera to capture the surface image in an orthogonal polarization mode includes: performing multiple acquisitions of bright field or dark field composite illumination modes at different angles on each detection area corresponding to the PCBA board to generate an original image sequence containing diffuse reflection and specular reflection features; and generating the surface image based on the original image sequence.

[0167] In some embodiments, the adaptive illumination compensation and noise removal processing of the surface image to generate a standardized image includes: independently estimating illumination of the RGB three channels corresponding to the surface image, and decomposing the brightness component and the reflection component through a Gaussian difference filter; using adaptive correction to enhance local contrast in the solder joint area corresponding to the surface image, and performing brightness equalization processing on the silk screen area; extracting coordinate and size information as prior knowledge based on the layout file corresponding to the PCBA board; and dynamically adjusting the search window size during the non-local mean denoising process corresponding to the surface image based on the layout prior knowledge, including using a 5×5 pixel window for the fine pin area corresponding to the surface image and an 11×11 pixel window for the large copper-clad area corresponding to the surface image.

[0168] In some embodiments, the multi-scale segmentation of the standardized image to obtain image blocks including local details and global structures includes: adopting a hierarchical processing strategy combining sliding window and superpixel segmentation to perform multi-scale segmentation on the standardized image; the first scale corresponding to the multi-scale segmentation traverses the image with a 128×128 pixel window at a 16-pixel step size to extract local detail blocks including component solder joints; the second scale uses a 512×512 pixel window at a 64-pixel step size to extract the global structure block of the circuit routing; constructs a multi-resolution feature pyramid through bilinear interpolation, applies spatial attention weights to the local detail blocks in the feature fusion stage, and applies channel attention weights to the global structure blocks to generate the image blocks according to the local detail blocks and the global structure blocks.

[0169] In some embodiments, the dynamic threshold segmentation algorithm is used to binarize the heat map, and morphological operations are combined to filter out noise interference to generate a defect mask, including: calculating the category separability measure and determining the segmentation threshold based on the grayscale histogram distribution characteristics of the heat map; performing a morphological opening operation based on the segmentation threshold to eliminate isolated noise points in the heat map whose area is smaller than a preset area; using elliptical structuring elements to perform multiple iterative closing operations to fill the micro-hole defects in the heat map; performing area threshold screening on the connected area, and determining the candidate defect area whose perimeter area ratio is greater than the preset area ratio as the target area; and obtaining the defect mask based on the target area.

[0170] In some embodiments, the defect types include at least pad offset defects and tin ball splash defects; outputting the defect detection result of the PCBA board based on the defect mask and probability distribution includes: mapping the regional coordinates of the defect mask to the spatial dimension of the vector corresponding to the probability distribution, and using a non-maximum suppression algorithm to eliminate overlapping detection frames; in the spatial dimension, generating structured data of the spatial coordinates, defect codes, process parameters and confidence intervals corresponding to the pad offset defect and tin ball splash defect respectively, and generating the defect detection result.

[0171] In some embodiments, before inputting the image block into the dual-branch deep learning model, the method further includes: adopting a transfer learning strategy to pre-train the dual-branch deep learning model on a source domain general defect dataset, and fine-tuning the dual-branch deep learning model on a preset target domain PCBA dataset through a domain adaptive loss function to eliminate the inter-domain differences of different PCBA layouts of the dual-branch deep learning model in the dual-branch deep learning model; wherein the domain adaptive loss function is a weighted sum of the maximum mean difference loss and the cross entropy loss.

[0172] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the control equipment and each module described above can refer to the corresponding processes in the method embodiments described in the above embodiments, and will not be repeated here.

[0173] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the deep learning-based PCBA surface defect detection method provided in any embodiment of the present application.

[0174] The computer-readable storage medium may be an internal storage unit of the control device described in the aforementioned embodiment, such as a hard disk or memory of the control device. The computer-readable storage medium may also be an external storage device of the control device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the control device.

[0175] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the storage medium and each module described above can refer to the corresponding processes in the method embodiments described in the above embodiments, and will not be repeated here.

[0176] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A PCBA surface defect detection method based on deep learning, characterized in that: Used to perform defect detection on preset PCBA boards; include The surface image of the PCBA board is collected using a preset multi-angle light source and a high-resolution camera, and the surface image is subjected to adaptive illumination compensation and noise removal processing to generate a standardized image; Performing multi-scale segmentation on the standardized image to obtain image blocks including local details and global structures; Construct a dual-branch deep learning model, which includes a backbone network, a multi-scale feature fusion module, and a defect detection branch. The backbone network includes a pre-trained ResNet-50 architecture and embeds a channel attention mechanism to enhance the feature response of defect-sensitive areas. The multi-scale feature fusion module integrates feature maps from different layers of the backbone network through a feature pyramid network to generate fused multi-scale features. The defect detection branch predicts a thermal map of defect locations and a probability distribution of defect types based on the multi-scale features; Inputting the image block into the dual-branch deep learning model, the dual-branch deep learning model outputting the heat map and the probability distribution; Binarize the heat map using a dynamic threshold segmentation algorithm, filter out noise interference using morphological operations, and generate a defect mask; Outputting the defect detection result of the PCBA board according to the defect mask and probability distribution, wherein the defect detection result includes the defect location, defect type and confidence score; completing the defect detection of the PCBA board.

2. The method according to claim 1, characterized in that The multi-angle light source includes 12 groups of LED light source units distributed in a circular array, and the adjustable incident angle formed by each group of LED light source units with the plane of the PCBA board corresponds to a range of 15° to 75°; the surface image of the PCBA board is collected using the preset multi-angle light source and the high-resolution camera, including: The light source brightness of the LED light source unit is coordinated with the camera exposure parameters of the high-resolution camera to control the high-resolution camera to capture the surface image in an orthogonal polarization mode.

3. The method according to claim 2, characterized in that The controlling the high-resolution camera to collect the surface image in an orthogonal polarization mode comprises: Perform multiple acquisitions of bright field or dark field composite illumination modes at different angles on each inspection area corresponding to the PCBA board to generate an original image sequence containing diffuse reflection and specular reflection features; The surface image is generated according to the original image sequence.

4. The method according to claim 1, wherein The step of performing adaptive illumination compensation and noise removal processing on the surface image to generate a standardized image includes: Perform independent illumination estimation on the RGB channels corresponding to the surface image, and decompose the brightness component and the reflection component using a Gaussian difference filter; Adaptive correction is used to enhance the local contrast of the solder joint area corresponding to the surface image, and brightness equalization is performed on the silk screen area; Based on the layout file corresponding to the PCBA board, coordinate and size information are extracted as prior knowledge; According to the layout prior knowledge, the search window size is dynamically adjusted during the non-local mean denoising process corresponding to the surface image, including using a 5×5 pixel window for the fine pin area corresponding to the surface image and an 11×11 pixel window for the large copper-clad area corresponding to the surface image.

5. The method according to claim 1, wherein The performing multi-scale segmentation on the standardized image to obtain image blocks including local details and global structures includes: The standardized image is segmented at multiple scales using a hierarchical processing strategy combining sliding windows and superpixel segmentation. The multiscale segmentation involves traversing the image at a first scale using a 128×128 pixel window with a 16-pixel step size to extract local detail blocks including component solder joints. A second scale involves extracting global structural blocks of circuit traces using a 512×512 pixel window with a 64-pixel step size. A multi-resolution feature pyramid is constructed by bilinear interpolation. In the feature fusion stage, spatial attention weights are applied to local detail blocks, and channel attention weights are applied to global structure blocks to generate the image blocks based on the local detail blocks and the global structure blocks.

6. The method according to claim 1, characterized in that The method of binarizing the heat map using a dynamic threshold segmentation algorithm and filtering out noise interference by combining morphological operations to generate a defect mask includes: Calculating a category separability metric and determining a segmentation threshold based on the grayscale histogram distribution characteristics of the heat map; Performing a morphological opening operation according to the segmentation threshold to eliminate isolated noise points in the thermal map whose area is smaller than a preset area; Filling the micro-hole defects in the thermal map by performing multiple iterative closing operations using elliptical structural elements; performing area threshold screening on the connected regions, and determining candidate defect regions with a perimeter area ratio greater than a preset area ratio as target regions; The defect mask is acquired according to the target area.

7. The method according to claim 1, characterized in that The defect types include at least pad offset defects, solder ball splash defects, and pin breakage; and outputting the defect detection results of the PCBA board according to the defect mask and probability distribution includes: Mapping the regional coordinates of the defect mask to the spatial dimension of the vector corresponding to the probability distribution, and using a non-maximum suppression algorithm to eliminate overlapping detection frames; In the spatial dimension, structured data of the spatial coordinates, defect codes, process parameters and confidence intervals corresponding to the pad offset defect, solder ball splash defect and pin breakage are generated to generate the defect detection result.

8. The method according to claim 1, characterized in that Before inputting the image block into the dual-branch deep learning model, the method further includes: A transfer learning strategy is adopted to pre-train the dual-branch deep learning model on a source domain general defect dataset, and fine-tune it on a preset target domain PCBA dataset using a domain adaptive loss function to eliminate the inter-domain differences of different PCBA layouts in the dual-branch deep learning model; The domain adaptation loss function is a weighted sum of the maximum mean difference loss and the cross entropy loss.

9. A PCBA surface defect detection system based on deep learning, characterized in that: Used to perform defect detection on pre-set PCBA boards; including: Multi-angle light sources and high-resolution cameras are used to capture surface images of the PCBA board; A control device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 7.

Images