PCB intelligent AOI defect classification and detection system based on deep learning

Abstract

This invention discloses a deep learning-based intelligent AOI defect classification and detection system for PCBs, comprising a multimodal image acquisition module that simultaneously acquires optical and infrared images of the PCB board, a hierarchical decision classification module that receives image pairs and sequentially performs defect localization, multi-branch feature fusion, defect classification, and confidence output, and a feedback optimization module that filters samples based on confidence and triggers incremental learning. By introducing an infrared thermal imaging module that is triggered synchronously with electrical performance testing and fusing it with high-resolution optical images, this invention constructs a multi-physical-dimensional detection information source. This enables the system to capture defect features that are difficult to detect using traditional pure optical methods, thereby significantly improving the detection rate and classification accuracy of hidden and complex defects, and solving the problem of missed detection and misjudgment caused by a single dimension of detection information.

Description

Technical Field

[0001] This invention relates to the field of PCB defect detection technology, and in particular to a deep learning-based intelligent AOI defect classification and detection system for PCBs. Background Technology

[0002] In the PCB manufacturing process, automated optical inspection (AOI) is a necessary step to ensure the quality of finished products. This step is used to identify various process defects such as solder defects, missing components, or substrate damage. Traditional AOI systems mainly rely on pre-set rule thresholds based on brightness, contrast, and geometry for judgment. Their detection logic is rigid, and their flexibility and adaptability are limited.

[0003] Existing AOI systems mostly rely on a single visible light image for analysis, making them insensitive to defects with low contrast to the background or those that only become apparent under specific physical conditions (such as abnormal heating during electrical operation). For example, cold solder joints may be difficult to detect when cold, or some inner-layer defects on the substrate may have weak features in the surface optical image, leading to missed detections or false positives. Furthermore, in terms of model decision-making mechanisms, most solutions use a single end-to-end network for defect localization and classification. This architecture struggles to balance detection speed and accuracy when dealing with high-density, multi-type, and dimensionally diverse defects on PCBs, and the network decision-making process lacks interpretability, hindering engineers' diagnosis and optimization.

[0004] Therefore, in response to the problems mentioned above, this invention proposes a deep learning-based intelligent AOI defect classification and detection system for PCBs. Summary of the Invention

[0005] To overcome the problems of low detection rate of specific defects due to the single dimension of detection information and the rigidity of model decision-making mechanism in complex scenarios in existing technologies, this invention proposes a deep learning-based intelligent AOI defect classification and detection system for PCBs. This system integrates multimodal imaging information, constructs a hierarchical collaborative intelligent decision-making network, and introduces a data-driven feedback optimization mechanism to achieve more comprehensive and self-evolving automated detection of PCB defects.

[0006] The technical solution of this invention is: a deep learning-based intelligent AOI defect classification and detection system for PCBs, comprising: The image acquisition module is used to simultaneously acquire standard optical images and infrared thermal images of the PCB board to be inspected. The image acquisition module includes a CCD camera, a ring LED light source, and an infrared thermal imager. The ring LED light source adopts a multi-channel independent controllable design to adjust the lighting scheme according to the surface material and component layout of different PCB boards to suppress reflections and shadows. It is worth noting that the image acquisition of the infrared thermal imager is triggered synchronously with the online electrical performance test signal of the PCB board to ensure that the dynamic heat distribution of the PCB under a preset load is captured, and the synchronization time error is less than 1 millisecond. The hierarchical decision classification module includes a pre-trained multi-branch deep neural network, which comprises a defect localization network, a multi-branch feature extraction and fusion network, and a defect classification and confidence output network. This module receives aligned optical and thermal image pairs output from the image acquisition module and executes the following classification decision process: S1. First, the optical image is preliminarily analyzed through a convolutional neural network to generate candidate boxes containing potential defect regions. The generation of these candidate boxes combines a prior probability map based on historical defect statistics. S2, for each candidate box, is synchronously input into three parallel sub-network branches. The first branch extracts fine-grained texture and shape features from the RGB information of the optical image; the second branch extracts contour and edge features from the preprocessed gradient and brightness normalized image of the optical image; the third branch extracts heat distribution and abnormal temperature rise features from the infrared thermal imaging image of the corresponding region. The high-dimensional feature vectors extracted by the three branches are weighted and concatenated through a feature fusion layer with an attention mechanism to generate a fused feature vector. S3, the fused feature vector is passed through a fully connected classification network to output the probability of the region belonging to various PCB defects. The categories include at least short solder, insufficient solder, missing components, wrong components, raised leads and substrate scratches, and a classification confidence score is also output. The feedback optimization module, connected to the hierarchical decision classification module, includes a difficult example sample library and an incremental learning unit. When the system detects a sample whose classification confidence score is lower than the first threshold (preferably 0.85) but is confirmed to be correctly classified by manual review, or a sample whose confidence score is higher than the second threshold (preferably 0.99) but is disproven by manual review, the system automatically stores the sample and its image data into the difficult example sample library. The incremental learning unit periodically (preferably after accumulating 500 new difficult example samples) uses the data in the difficult example sample library to adjust the multi-branch deep neural network and update the network parameters.

[0007] Preferably, the defect localization network in the multi-branch deep neural network adopts a key point detection network based on the Anchor-Free architecture, which directly predicts the center point heatmap and bounding box size of the defect region. The prior probability map is input into the network along with the input image in a channel overlay manner.

[0008] Preferably, in the multi-branch feature extraction and fusion network, the first branch adopts a residual network constructed with depthwise separable convolutions, the second branch adopts a custom convolutional kernel group that emphasizes high-frequency information, and the third branch adopts a small convolutional neural network; the feature fusion layer with attention mechanism generates a weight mask for each spatial location of the feature map from the three branches through a shallow neural network, and then performs weighted summation.

[0009] Preferably, the system also includes a data augmentation and synthesis module located in the training phase. This module is capable of synthesizing physically realistic defect samples on defect-free PCB images based on known defect physical models. Specifically, it includes simulating substrate scratches of different angles, depths, and widths; simulating solder bridging or cold solder joints of different volumes, wetting angles, and positions; and generating realistic images of misaligned, reversed, or offset components through 3D rendering and image fusion technology.

[0010] Preferably, the incremental learning unit in the feedback optimization module adopts an elastic weight merging algorithm, which needs to constrain the change range of important network parameters during adjustment in order to prevent catastrophic forgetting of learned knowledge.

[0011] Preferably, the system also includes a visualization interface connected to the hierarchical decision classification module, which can overlay the edge feature map extracted by the second branch, the thermal anomaly feature map extracted by the third branch, and the weight mask of the attention fusion layer in the multi-branch feature extraction and fusion network onto the original optical image in the form of a heat map, so as to indicate the key image region on which the classification decision is based.

[0012] Preferably, the image acquisition of the infrared thermal imager is triggered synchronously with the electrical test signal of the PCB board to ensure that the thermal imaging image captures the heat distribution of the PCB under specific working conditions.

[0013] Preferably, the lighting scheme of the ring LED light source is determined by a hierarchical decision classification module based on the preliminary analysis results of the first image of the current PCB board, through a pre-trained lighting strategy generator, which determines the brightness and lighting sequence of each channel LED.

[0014] Preferably, the defect classification and confidence output network adopts a multi-task learning architecture, with its main task being defect multi-classification and auxiliary tasks including defect region segmentation mask prediction and defect size regression prediction. The classification confidence score is jointly determined by the entropy value of the output probability of the main task and the consistency of the prediction results of the main and auxiliary tasks.

[0015] The beneficial effects of this invention are: 1. This invention introduces an infrared thermal imaging module that is triggered synchronously with electrical performance testing and fuses it with high-resolution optical images to construct a multi-physical-dimensional detection information source. This enables the system to capture defect features that are difficult to detect using traditional pure optical methods, such as local abnormal temperature rises caused by poor soldering or microcracks, or abnormal heat conduction caused by internal lamination problems. This significantly improves the detection rate and classification accuracy of hidden and complex defects, and solves the problem of missed defects and misjudgments caused by a single dimension of detection information.

[0016] 2. This invention employs a hierarchical decision architecture and a multi-branch feature fusion network to decouple the defect detection task into three collaborative stages: localization, multi-feature extraction, and classification. It also introduces a dual attention mechanism of channel and space for adaptive feature weighting, enabling the model to dynamically adjust the attention given to different modalities and regional features according to the defect type. This improves the recognition accuracy of multi-morphological defects in complex scenarios. At the same time, through the localization sub-network and feature fusion strategy, a balance between detection speed and accuracy is achieved under limited computing resources, overcoming the problem of rigid decision-making in traditional end-to-end single models.

[0017] 3. By integrating an online hard example sample library with an incremental learning unit based on an elastic weight merging algorithm, the system possesses continuous self-optimization and adaptability. When switching to new products or processes on the production line, it can quickly fine-tune the model using feedback data from actual production without spending a lot of time and manpower on comprehensive retraining, thereby significantly improving the system's generalization ability across different production batches and product models. Attached Figure Description

[0018] Figure 1 The diagram shown is a schematic representation of the system framework of the present invention. Figure 2 The diagram shown illustrates the system workflow of this invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 and Figure 2 This invention provides an embodiment of a deep learning-based intelligent AOI defect classification and detection system for PCBs: In this embodiment, the image acquisition module will be described in detail: The optical acquisition section of this module employs a linear scan camera, coupled with a programmable multi-channel ring LED light source. The light source is designed for independent control in four quadrants, with each quadrant containing an LED array of three different wavelengths (e.g., white, red, and blue) to address the complex reflections caused by different materials on the PCB surface, such as metal, solder mask, or silkscreen printing. The infrared acquisition section utilizes a mid-wave infrared thermal imager, whose spectral response range covers the operating temperature rise range of the PCB components.

[0021] In this implementation, the PLC coordinates the simultaneous exposure of the optical camera and image acquisition of the thermal imager after the electrical test signal reaches a steady state (e.g., 100ms after power-on), ensuring that the two images represent the same physical state. Image alignment is achieved through a pre-calibrated affine transformation matrix, which is calculated once during system installation using a calibration board (which has both high optical contrast features and infrared heating points).

[0022] In this embodiment, the hierarchical decision classification module will be described in detail: This module mainly includes a multi-branch deep neural network, which comprises a defect localization network, a multi-branch feature extraction and fusion network, and a defect classification and confidence output network. Specifically: (1) For the defect localization network, an Anchor-Free architecture based on CenterNet is adopted. Its input is a 608x608 optical image, and the output is a heat map after resolution downsampling, where each peak point corresponds to a defect center. In this embodiment, the "defect prior probability map" obtained from historical data is used as the fourth channel and stitched with the RGB three-channel image as input. This prior map is generated by smoothing historical defect coordinates with a Gaussian kernel, which can guide the network to pay more attention to areas with high incidence of defects such as solder joints or pins.

[0023] (2) For the multi-branch feature extraction and fusion network, for each candidate box output by the localization network, the corresponding optical image patch and thermal image patch are cropped and input into three branches respectively. The first branch (RGB texture branch) is used to extract rich surface color, texture and shape features. The input of the second branch (edge ​​enhancement branch) is the gradient magnitude map, followed by a shallow network composed of 3x3 and 5x5 sharpening convolution kernels, which is specifically used to enhance the boundary information between defects and background, and is very sensitive to defects such as scratches and warping. The third branch (thermal feature branch) uses a simple 4-layer CNN to process the thermal image patch and extract features such as average temperature, temperature gradient and local hotspots. After the feature maps of the three branches are adjusted to the same spatial size, they are fed into the attention fusion layer. This layer first uses global average pooling to generate channel attention vectors and performs channel recalibration on the features of each branch. Then, a spatial attention sub-network (composed of two convolutional layers) generates a unified spatial weight map based on the feature maps of the three branches. This map highlights the spatial locations that contribute the most to the classification decision (such as solder joint edges, component bodies, etc.). Finally, the double-weighted feature maps are concatenated in the channel dimension to form a fused feature vector containing multi-source information.

[0024] (3) For the defect classification and confidence output network, the main classification head outputs the defect category probability (containing 12 common defects), and the two auxiliary task heads output a low-resolution defect segmentation mask (used to refine the defect shape) and a regression value representing the approximate physical size of the defect (such as solder area and scratch length).

[0025] The network training process is divided into two phases. The first phase involves pre-training using a large-scale hybrid dataset of synthetic and ground-labeled data (approximately 500,000 samples). The second phase involves fine-tuning on the client's actual production data. Data augmentation includes random rotation, cropping, color jitter, and simulating different environmental temperature shifts in the infrared images.

[0026] In this embodiment, the feedback optimization module will be described in detail: The online difficult sample library for this module is a database with version management capabilities. During system operation, all detection results with confidence levels in the "uncertainty range" (e.g., 0.7-0.9) are temporarily cached. Quality inspectors review these results on the interactive interface and confirm or correct them. Once the confirmed result is inconsistent with the system prediction, or although consistent but with extremely low confidence, the sample, its multimodal image, corrected label, and intermediate features extracted by the network will be permanently stored in the difficult sample library, and tagged with a timestamp and product model.

[0027] After the incremental learning unit of this module starts, it first randomly samples the 500 most recently accumulated samples from the hard example sample library, and then uses the EWC algorithm to fine-tune the main network. The EWC algorithm evaluates the importance of each parameter by calculating the Fisher information matrix of the network parameters on the old task, and applies a penalty to changes in important parameters when fine-tuning new samples, thereby keeping the performance forgetting of the model on old data within 2%. After fine-tuning, the new model replaces the old model.

[0028] In this embodiment, the visual interface will be described in detail: This interface displays basic information such as defect images, categories, and locations. For any detected defect, users can select to view the "Edge Feature Map," "Thermal Anomaly Map," and "Attention Weight Map." The Edge Feature Map displays strong edge responses extracted by the second branch in red, clearly outlining the defect contour; the Thermal Anomaly Map shows the temperature deviation of the area relative to the normal area using a color overlay; and the Attention Weight Map, in the form of a semi-transparent heatmap, shows which pixels in the image the network pays the most attention to when making classification decisions.

[0029] This invention provides an embodiment in which, for a defect judged as "insufficient solder", the interface will display a high focus on the unfilled area at the edge of the pad, and the thermal anomaly map will show that the temperature rise at this point is slightly lower due to poor contact.

[0030] This invention provides Embodiment 1: This embodiment implements the system on the SMT production line of a mobile phone motherboard in a large electronics manufacturing company. The product models on this production line are relatively fixed, but the component density is high, and the defect types are mainly tombstoning, displacement and poor soldering of 0201 / 01005 micro components.

[0031] This embodiment was compared with an existing rule-based threshold-based AOI (Comparative Example 1) and a mainstream deep learning AOI based on monocular optical images (Comparative Example 2) in a parallel test lasting one month. The test dataset consisted of 50,000 motherboards produced that month, and all defects were finally confirmed by manual visual inspection and X-ray sampling.

[0032]

[0033] As shown in the table above, Example 1 performed best in terms of overall accuracy, detection rate of micro-component defects, and false alarm rate control. In particular, for defects such as "cold solder joints" which are not obvious in cold optical images, the present invention improved the detection rate from 85% in Comparative Example 2 to 98% by using thermal imaging information.

[0034] This invention provides Embodiment 2: This embodiment is applied to the testing of HDI boards in automotive electronic controllers. These HDI boards are characterized by a large number of blind vias, buried vias, and inner layer circuits, and surface defects such as substrate pits and copper foil scratches are small and have low contrast.

[0035] This embodiment optimizes the illumination angle of the ring light source to address surface defects, employing low-angle grazing light to highlight minute three-dimensional deformations. The training data significantly increases the amount of synthetic data for various substrate damages, and the second branch (edge ​​branch) of the network is given higher weights.

[0036] This embodiment is compared with Comparative Example 2 above on the same batch of HDI board test sets containing 2000 known defects (500 of which are minor substrate scratches / pits).

[0037]

[0038] As can be seen from the table above, for conventional solder joint defects, the two are not much different. However, for minor damage to the substrate, Example 2, with its edge-enhanced branches and optimized lighting, has a significantly higher detection rate than Comparative Example 2.

[0039] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A deep learning-based intelligent AOI defect classification and detection system for PCBs, characterized in that, Including: The image acquisition module is used to simultaneously acquire standard optical images and infrared thermal images of the PCB board to be inspected. The image acquisition module includes a CCD camera, a ring LED light source and an infrared thermal imager. The ring LED light source adopts a multi-channel independent controllable design to adjust the lighting scheme according to the surface material and component layout of different PCB boards to suppress reflections and shadows. The hierarchical decision classification module includes a pre-trained multi-branch deep neural network, which comprises a defect localization network, a multi-branch feature extraction and fusion network, and a defect classification and confidence output network. This module receives aligned optical and thermal image pairs output from the image acquisition module and executes the following classification decision process: S1. First, the optical image is preliminarily analyzed through a convolutional neural network to generate candidate boxes containing potential defect regions. The generation of these candidate boxes combines a prior probability map based on historical defect statistics. S2, for each candidate box, synchronous input is fed to three parallel sub-network branches. The first branch performs fine-grained texture and shape feature extraction on the RGB information of the optical image. The second branch extracts contour and edge features from the gradient and brightness normalized images after preprocessing of the optical images; the third branch extracts heat distribution and abnormal temperature rise features from the infrared thermal imaging images of the corresponding regions; the high-dimensional feature vectors extracted by the three branches are weighted and concatenated through a feature fusion layer with an attention mechanism to generate a fused feature vector. S3, the fused feature vector is passed through a fully connected classification network to output the probability of the region belonging to various PCB defects. The categories include at least short solder, insufficient solder, missing components, wrong components, raised leads and substrate scratches, and a classification confidence score is also output. The feedback optimization module, connected to the hierarchical decision classification module, includes a difficult example sample library and an incremental learning unit. When the system detects a sample whose classification confidence score is lower than the first threshold but is confirmed to be correctly classified by manual review, or a sample whose confidence score is higher than the second threshold but is disproven by manual review, the system automatically stores the sample and its image data into the difficult example sample library. The incremental learning unit periodically uses the data in the difficult example sample library to adjust the multi-branch deep neural network and update the network parameters.

2. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 1, characterized in that: The defect localization network in the multi-branch deep neural network adopts a key point detection network based on the Anchor-Free architecture, which directly predicts the center point heatmap and bounding box size of the defect region. The prior probability map is input into the network along with the input image in a channel overlay manner.

3. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 1, characterized in that: In the multi-branch feature extraction and fusion network, the first branch uses a residual network constructed with depthwise separable convolutions, the second branch uses a custom convolutional kernel group that emphasizes high-frequency information, and the third branch uses a small convolutional neural network; the feature fusion layer with attention mechanism generates a weight mask for each spatial location of the feature maps from the three branches through a shallow neural network, and then performs weighted summation.

4. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 1, characterized in that: The system also includes a data augmentation and synthesis module, which is located during the training phase and is capable of synthesizing physically realistic defect samples on defect-free PCB images based on known defect physical models.

5. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 4, characterized in that: The defect samples include simulated scratches of different directions and intensities, soldering anomalies of different volumes and shapes, and projection synthesis of faulty and missing parts based on 3D models of components.

6. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 1, characterized in that: The incremental learning unit in the feedback optimization module adopts an elastic weight merging algorithm, which needs to constrain the change range of important network parameters during adjustment in order to prevent catastrophic forgetting of learned knowledge.

7. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 1, characterized in that: The system also includes a visualization interface connected to the hierarchical decision classification module, which can overlay the edge feature map extracted by the second branch, the thermal anomaly feature map extracted by the third branch, and the weight mask of the attention fusion layer in the multi-branch feature extraction and fusion network onto the original optical image in the form of a heat map, to indicate the key image region on which the classification decision is based.

8. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 1, characterized in that: The infrared thermal imager's image acquisition is triggered synchronously with the electrical test signal of the PCB board to ensure that the thermal imaging image captures the heat distribution of the PCB under specific operating conditions.

9. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 1, characterized in that: The lighting scheme of the ring LED light source is determined by a hierarchical decision classification module based on the preliminary analysis results of the first image of the current PCB board, through a pre-trained lighting strategy generator, which determines the brightness and lighting sequence of each channel LED.

10. The PCB intelligent AOI defect classification and detection system based on deep learning according to claim 1, characterized in that: The defect classification and confidence output network adopts a multi-task learning architecture. Its main task is defect multi-classification, and the auxiliary tasks include defect region segmentation mask prediction and defect size regression prediction. The classification confidence score is jointly determined by the entropy value of the output probability of the main task and the consistency of the prediction results of the main and auxiliary tasks.

Images