PCBA surface defect detection method based on abnormal synthesis strategy

Abstract

The invention discloses a PCBA surface defect detection method based on an abnormal synthesis strategy. The method comprises the following steps: firstly, constructing a PCBA target detection data set, and training a target detection model by using the PCBA target detection data set; identifying an electronic component in the PCBA image by using the trained target detection model to obtain an electronic component image; according to the categories, classifying all the electronic component images to obtain defect detection data sets of various electronic components so as to obtain a PCBA defect detection data set; then, constructing a double-branch interconnection diffusion model, generating global abnormal images of various electronic components by using the double-branch interconnection diffusion model, and obtaining an expanded PCBA defect detection data set; and finally, constructing a defect detection model, training the defect detection model by using the expanded PCBA defect detection data set, and using the trained defect detection model for PCBA surface defect detection. When the abnormal image is generated, the structure and appearance information of the background are referred, and the rationality and authenticity of abnormal image generation are improved.

Description

Technical Field

[0001] This invention belongs to the field of PCBA defect detection technology, specifically a PCBA surface defect detection method based on an anomaly synthesis strategy. Background Technology

[0002] In modern electronics manufacturing, printed circuit board assembly (PCBA) has become an essential component of most electronic devices. Its quality directly affects the performance and lifespan of these devices, and surface defect detection is a fundamental aspect of PCBA quality control. As the number of electronic components on printed circuit boards increases and their layout becomes more complex, defect detection becomes increasingly challenging.

[0003] Currently, the main methods employed are manual visual inspection and AOI-based methods. Manual visual inspection offers advantages such as low cost and high flexibility, but it is limited by human subjective judgment and fatigue, often failing to maintain consistency and efficiency. Especially on mass production lines, manual visual inspection struggles to handle large-scale inspection tasks and complex defect types, leading to low inspection efficiency and a high false positive rate. AOI-based inspection methods, leveraging advanced industrial imaging and image processing techniques, can achieve high-precision surface defect detection, offering advantages such as speed and efficiency, enabling the completion of large-scale inspection tasks in a short time. However, printed circuit boards contain numerous electronic components, making it difficult to collect a sufficient number and variety of anomalous samples. Models often focus only on normal samples, and learning solely from normal samples may limit the model's discriminative power. The lack of anomalous data to guide discrimination results in poor model performance in anomaly detection tasks. Due to the lack of prior knowledge of available anomalous images and anomalous categories, existing methods primarily rely on carefully designed data augmentation strategies or external data to generate anomalous images. This results in low-quality anomalous images with unnatural blending of anomalous areas into the background, and an inability to accurately simulate the diversity and complexity of real defects. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to propose a PCBA surface defect detection method based on an anomaly synthesis strategy.

[0005] The present invention solves the aforementioned technical problem by adopting the following technical solution: A method for detecting PCBA surface defects based on anomaly synthesis strategy, characterized by comprising the following steps: Step 1: Construct a PCBA object detection dataset, and use the PCBA object detection dataset to train the object detection model to obtain the trained object detection model; Step 2: Collect a small number of PCBA images and preprocess them. Use the trained target detection model to identify electronic components in the preprocessed PCBA images to obtain electronic component images. Classify all electronic component images according to the electronic component category to obtain defect detection datasets for various electronic components, thereby obtaining PCBA defect detection datasets. Step 3: Construct a bi-branch interconnect diffusion model, and use the bi-branch interconnect diffusion model to generate global anomaly images of various electronic components to obtain an expanded PCBA defect detection dataset. A small number of real global anomaly images and masks for each type of defect in each type of electronic component are obtained. The real global anomaly images and masks are multiplied element by element to obtain real local anomaly images, thus obtaining a real global anomaly image-local anomaly image pair. Based on the SD model, a dual-branch interconnected diffusion model is constructed, including a global branch and a local branch. Both branches include a variational autoencoder and a DA-TransUNet network. A background compensation block is embedded before each dual attention block of the DA-TransUNet network in the global branch. The output features of dual attention blocks at the same position in the DA-TransUNet networks of the two branches interact in the self-attention interaction block. The forward noise addition process of the dual-branch interconnected diffusion model: The real global anomalous image and the local anomalous image are encoded into the latent space by their respective variational autoencoders to obtain the latent representation of the global anomalous image and the latent representation of the local anomalous image; random noise is gradually added to the latent representation of the global anomalous image and the latent representation of the local anomalous image until the maximum diffusion time step is reached to obtain the latent representation of the global anomalous image and the latent representation of the local anomalous image with full noise. The reverse denoising process of the dual-branch interconnected diffusion model: The DA-TransUNet network of the global branch performs progressive denoising on the latent representation of the global anomalous image with full noise to generate the global anomalous image; the DA-TransUNet network of the local branch performs progressive denoising on the latent representation of the local anomalous image with full noise to generate the local anomalous image. In the background compensation block, a pre-trained saliency detection model is used to extract the foreground object mask of the real global anomaly image, and the background image is extracted using the inverse mask of the foreground object mask. The background image is encoded into the latent space by a variational autoencoder to obtain the latent representation of the background image. Random noise is added to the latent representation of the background image at the current diffusion time step to obtain the noisy latent representation of the background image at the current diffusion time step. The noisy latent representation of the background image at the current diffusion time step is used to extract background features by a U-Net encoder. The background features are mapped by a multilayer perceptron to obtain modulated background features. The modulated background features are injected as background information into the dual attention block of the DA-TransUNet network, and the query vector of the dual attention block is then calculated. Key vector Sum value vector The calculation formula is: (4) (5) (6) in, The input features are for the dual attention block. , and This is the weight matrix. The learnable adjustment coefficient, These are the modulated background features; In the self-attention interaction block, the DA-TransUNet network for the global and local branches is processed according to equation (7). Output features of a dual attention block and The process is performed to obtain global-local joint features. ; (7) in, For rearrangement operations, For splicing operations; Using the global-local joint features as a value vector, the DA-TransUNet network of the global and local branches... The output features of each dual attention block are used as query vector and key vector, respectively, and interact through a self-attention mechanism to obtain global-local joint features of information interaction. The global-local joint features of information interaction are added to the global-local joint features, and the added features are successively subjected to rearrangement and splitting operations to obtain two feature sums, which are used as input features of the corresponding decoding layers of the DA-TransUNet network for the global and local branches. Step 4: Build a defect detection model. Train the defect detection model using the expanded PCBA defect detection dataset, and then use the trained defect detection model for PCBA surface defect detection.

[0006] Furthermore, the defect detection model includes a feature extractor and a conditional normalization flow. The test PCBA image is processed by the feature extractor to extract multi-scale features, which are then mapped to the latent space via the conditional normalization flow to obtain a Gaussian distribution of the input features. The log-likelihood value is calculated based on the Gaussian distribution of the input features, and the log-likelihood value of the features at each location is transformed to the interval [0,1] to obtain the anomaly score at each location. Based on the anomaly scores at all locations, the Top-K average aggregation algorithm is used to calculate the anomaly score of the test PCBA image. If the anomaly score of the test PCBA image is greater than the anomaly threshold, then a defect exists on the PCBA surface; otherwise, no defect exists.

[0007] Furthermore, the feature extractor uses a pre-trained EfficientNet-B6 backbone network, and the multi-scale features are the output features of the third, fourth, and fifth feature extraction layers of the EfficientNet-B6 backbone network.

[0008] Furthermore, the noise addition process of the dual-branch interconnect diffusion model is expressed as follows: (2) (3) in, , They are the first Latent representations of global and local anomalies with noise added at each diffusion time step. , These are the latent representations of global anomaly images and the latent representations of local anomaly images, respectively. It is the diffusion coefficient. , It is random noise.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: To address the problem of insufficient anomalous data, this invention proposes a dual-branch interconnected diffusion model to generate anomalous images, overcoming the difficulty in collecting such images. Utilizing a small number of anomalous images for training the detection model significantly improves detection performance, thus enhancing detection accuracy. The global and local branches of the dual-branch interconnected diffusion model are used to generate global and local anomalous images, respectively, avoiding unnatural fusion of anomalous regions and background, as well as unreasonable mask placement.

[0010] A background compensation block is introduced into the global branch. Background prior knowledge is used to constrain the generation of global abnormal images. This allows the model to refer to the structure and appearance information of the background when generating targets and abnormal regions, preventing abnormal regions from being generated into the background region and improving the rationality and realism of the generated abnormal images.

[0011] In the dual-branch interconnected diffusion model, a self-attention interaction block is introduced. The output features of the dual attention blocks of the global branch and the local branch interact with each other in real time through the self-attention interaction block. This allows the global branch to "know" the location and texture of the anomaly, and the local branch to "know" the semantic environment of the surrounding background. This ensures that the position and shape of the anomaly region in the global anomaly image are consistent with the anomaly part in the local anomaly image, making the generated global anomaly image more natural. Attached Figure Description

[0012] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is a schematic diagram of the structure of the dual-branch interconnected diffusion model of the present invention; Figure 3 This is a schematic diagram of the structure of the DA-TransUNet network of the two branches of this invention. Detailed Implementation

[0013] The technical solution of the present invention will be described in detail below with reference to specific drawings and embodiments, but this is not intended to limit the scope of protection of this application.

[0014] This invention relates to a PCBA surface defect detection method based on an anomaly synthesis strategy (hereinafter referred to as the method, see [link]). Figures 1-3 ), including the following steps: Step 1: Acquire several PCBA images using an industrial camera and perform preprocessing including correction and noise reduction to remove background and improve image quality, resulting in preprocessed PCBA images; label the preprocessed PCBA images to obtain a PCBA object detection dataset; train the object detection model using the PCBA object detection dataset to enable it to identify various electronic components in the PCBA images and output images of the electronic components, resulting in a trained object detection model; in this embodiment, the YOLOv8 model is selected as the object detection model. Step 2: Construct a PCBA defect detection dataset, including defect detection datasets for various electronic components; A small number of PCBA images are collected and preprocessed. The trained object detection model is used to identify electronic components in the preprocessed PCBA images to obtain electronic component images. According to the electronic component category, all electronic component images are classified to obtain defect detection datasets for each type of electronic component. Each defect detection dataset for electronic component includes a training set and a test set.

[0015] Step 3: Construct a bi-branch interconnect diffusion model, use the bi-branch interconnect diffusion model to generate global anomaly images of various electronic components, expand the defect detection training data, and obtain the expanded PCBA defect detection dataset. 3.1) Obtain a small number (usually less than 10) of real global anomaly images and their masks for each type of defect (e.g., "capacitor failure") of each type of electronic component, to obtain multiple real global anomaly image-mask pairs. ;in, , Let A represent the true global anomaly image and its mask, respectively. A binary mask is used to identify the anomalous regions in the image. The true global anomaly image and its mask are multiplied element-wise to obtain the true local anomaly image. ;in, This represents element-wise multiplication, thus obtaining the true global anomaly image - local anomaly image pair. For true global anomaly images and real local anomaly images The size is normalized and adjusted to a fixed resolution, such as 512×512 pixels, for subsequent model processing.

[0016] 3.2) Based on the Stable Diffusion (SD) model, construct a two-branch interconnected diffusion model; such as... Figure 2 As shown, the model includes a global branch and a local branch. Both branches include a variational autoencoder (VAE) and a DA-TransUNet network. The VAE is used to map the real anomalous image into the latent space for forward noise addition, while the DA-TransUNet network is used for reverse denoising of the noisy image. The global branch is responsible for learning and generating global anomalous images, and the local branch is responsible for learning and generating local anomalous images. Figure 3 As shown, a background compensation block is embedded before each dual attention block (DA-Block) of the global branch DA-TransUNet network. The output features of dual attention blocks at the same position in the two branches of the DA-TransUNet network interact with each other through a self-attention interaction block, ensuring that the generated global anomaly image and the local anomaly image are perfectly integrated in terms of position, shape and semantics.

[0017] Forward noise addition process of the two-branch interconnected diffusion model: For image pairs composed of real global anomaly images and real local anomaly images First, the variational autoencoder with two branches encodes the image into the latent space to obtain the latent representation of the global anomaly image. and latent representation of local anomalies in images ; (1) in, It is a variational autoencoder; Subsequently, random noise is gradually added to the two latent representations at the same diffusion time step until the maximum diffusion time step is reached, resulting in a global anomalous image latent representation with full noise and a local anomalous image latent representation. (2) (3) in, , They are the first Latent representations of global and local anomalies with noise added at each diffusion time step. It is the diffusion coefficient. , It is random noise; Inverse denoising process: The goal of the diffusion model is to learn the parameters of two DA-TransUNet networks, predict noise using the two DA-TransUNet networks respectively, and progressively denoise the global and local anomaly image latent representations based on the predicted noise, ultimately generating the global and local anomaly images. This is achieved by adding the predicted noise to the overall noise level. and The noise in the data, based on the predicted noise level. and Denoising is performed; the input to the global branch of the DA-TransUNet network is the noisy global anomalous image latent representation. and diffusion time step The input to the local branch of the DA-TransUNet network is a noisy latent representation of a local anomaly image. and diffusion time step .

[0018] To address issues such as object deformation and background contamination that may occur in few-shot learning, this invention introduces a background compensation block. This block utilizes prior background knowledge to constrain the generation of global anomaly images, enabling the model to simultaneously reference the structural and appearance information of the background when generating targets and anomaly regions. This maintains background stability and prevents target shape distortion. For real-world global anomaly images... Using a pre-trained saliency detection model (U 2 -Net) extracts its foreground object mask Using the inverted mask Extract background image ; background image The background image latent representation is obtained by encoding it into the latent space using a variational autoencoder; noise is then added to the latent representation of the background image at the diffusion time step. , obtained the The latent representation of a noisy background image at each diffusion time step. Latent representation of noisy background image Background features are extracted using the U-Net encoder after parameter freezing. Background features After mapping using a learnable multilayer perceptron (MLP), modulated background features are obtained. To align with the current feature distribution space; the modulated background features are injected as background information into the dual attention block of the DA-TransUNet network, thus the query vector of the dual attention block... Key vector Sum value vector The calculation formula is: (4) (5) (6) in, The input features are for the dual attention block. , and This is the weight matrix. The initial value is set to 0.1, which is a learnable adjustment coefficient. The input to the self-attention interaction block is the DA-TransUNet network's global branch and local branch. Output features of a dual attention block and Its shape is , For batch size, , These are the feature height and width, respectively. The number of channels is used to make the information of the global and local branches visible to each other. The two input features are concatenated along the feature dimension, and then rearranged to adjust the feature shape, making global and local features at the same spatial location adjacent. This achieves feature pairing at the same spatial location in the two branches, resulting in the global-local joint feature. Its shape is ; (7) in, For rearrangement operations, For splicing operations; Using the global-local joint features as the value vector of the self-attention mechanism, the DA-TransUNet network with global and local branches... Output features of a dual attention block and Using the query vector and key vector respectively, a self-attention mechanism enables paired features to exchange information, allowing the global branch to "know" the location and texture of the anomaly, and the local branch to "know" the semantic environment of the surrounding background, resulting in a global-local joint feature with information interaction. ; (8) in, This is a self-attention mechanism; The global-local joint features of information interaction are added together, and the added features are then subjected to rearrangement and splitting operations to obtain two features. and The input features of the DA-TransUNet network, which serves as the global and local branches, correspond to the decoding layer.

[0019] Finally, the global and local branches of the two-branch interconnected diffusion model generate global anomaly images, respectively. and local anomaly images .

[0020] The two-branch interconnected diffusion model is trained, and the total training loss is calculated using the following loss function; (9) in, , These are the diffusion losses for the global branch and the local branch, respectively. and The DA-TransUNet network represents the global branch and the local branch, respectively. , For network parameters, It is an L2 norm.

[0021] Based on a small number of real global anomaly image-local anomaly image pairs, global anomaly images and local anomaly images are generated using a trained bi-branch interconnected diffusion model. All generated global anomaly images are added to the training set of the electronic component defect detection dataset to obtain the expanded PCBA defect detection dataset.

[0022] Step 4: Build a defect detection model. Train the defect detection model using the expanded PCBA defect detection dataset, and then use the trained defect detection model for PCBA surface defect detection. The defect detection model includes a feature extractor and a conditional normalized flow (CNFlow). Test PCBA images are processed by the feature extractor to extract multi-scale features. The feature extractor uses an EfficientNet-B6 backbone network pre-trained on the ImageNet dataset. This backbone network can extract rich and discriminative hierarchical features from input images, possessing powerful feature representation capabilities. To capture multi-scale information from local details to global semantics under different receptive fields, multi-scale features... , and These are the output features of the third, fourth, and fifth feature extraction layers of the EfficientNet-B6 backbone network, respectively. The downsampling rates of the three feature extraction layers are 4×, 8×, and 16×, which helps to sensitively identify fine-grained defect regions during PCBA defect detection. Multi-scale features are used as input features for Conditional Normalization Flow (CNFlow). Input features Mapping to the latent space yields a Gaussian distribution of the input features, with high-probability-density regions corresponding to normal images and low-probability-density regions corresponding to abnormal images. Conditional normalization then flows through an invertible neural network that maps the input features to a standard Gaussian latent space via multiple coupled layers. In the middle; calculate the input features based on the Gaussian distribution of the input features. The log-likelihood value; (10) in, Input features The log-likelihood value, Representing input features Gaussian distribution, This represents the presupposed prior distribution of the latent variables. This represents the conditionally normalized flow (CNFlow). Indicates the first Jacobian matrix of each coupling layer Indicates the number of coupling layers. Represents the determinant of a matrix; The log-likelihood values ​​of features at each location in the image are transformed to the interval [0,1] to obtain the anomaly score for each location. The higher the anomaly score, the more abnormal the image appears, and vice versa. The anomaly score is inversely proportional to the log-likelihood value of the features at the location, that is, the higher the log-likelihood value, the lower the anomaly score, and the lower the log-likelihood value, the higher the anomaly score. The Top-K average aggregation algorithm is used to calculate the anomaly score of the test PCBA image for all locations. This strategy takes into account both sensitivity and robustness, avoids interference from noise points, and focuses on the most suspicious areas. The anomaly score of the test PCBA image is compared with the anomaly threshold. If the anomaly score of the test PCBA image is greater than the anomaly threshold, the test PCBA image is considered abnormal, meaning that there is a defect on the PCBA surface. Otherwise, the test PCBA image is considered normal, meaning that there is no defect on the PCBA surface.

[0023] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A PCBA surface defect detection method based on anomaly synthesis strategy, characterized in that, Includes the following steps: Step 1: Construct a PCBA object detection dataset, and use the PCBA object detection dataset to train the object detection model to obtain the trained object detection model; Step 2: Collect a small number of PCBA images and preprocess them. Use the trained target detection model to identify electronic components in the preprocessed PCBA images to obtain electronic component images. Based on the category of electronic components, all electronic component images are classified to obtain defect detection datasets for various electronic components, thereby obtaining PCBA defect detection datasets; Step 3: Construct a bi-branch interconnect diffusion model, and use the bi-branch interconnect diffusion model to generate global anomaly images of various electronic components to obtain an expanded PCBA defect detection dataset. A small number of real global anomaly images and masks for each type of defect in each type of electronic component are obtained. The real global anomaly images and masks are multiplied element by element to obtain real local anomaly images, thus obtaining a real global anomaly image-local anomaly image pair. Based on the SD model, a dual-branch interconnected diffusion model is constructed, including a global branch and a local branch. Both branches include a variational autoencoder and a DA-TransUNet network. A background compensation block is embedded before each dual attention block of the DA-TransUNet network in the global branch. The output features of dual attention blocks at the same position in the DA-TransUNet networks of the two branches interact in the self-attention interaction block. The forward noise addition process of the dual-branch interconnected diffusion model: The real global anomalous image and the local anomalous image are encoded into the latent space by their respective variational autoencoders to obtain the latent representation of the global anomalous image and the latent representation of the local anomalous image; random noise is gradually added to the latent representation of the global anomalous image and the latent representation of the local anomalous image until the maximum diffusion time step is reached to obtain the latent representation of the global anomalous image and the latent representation of the local anomalous image with full noise. The reverse denoising process of the dual-branch interconnected diffusion model: The DA-TransUNet network of the global branch performs progressive denoising on the latent representation of the global anomalous image with full noise to generate the global anomalous image; the DA-TransUNet network of the local branch performs progressive denoising on the latent representation of the local anomalous image with full noise to generate the local anomalous image. In the background compensation block, a pre-trained saliency detection model is used to extract the foreground object mask of the real global anomaly image, and the background image is extracted using the inverse mask of the foreground object mask; the background image is encoded into the latent space by a variational autoencoder to obtain the latent representation of the background image. Random noise is added to the latent representation of the background image at the current diffusion time step to obtain the noisy latent representation of the background image at the current diffusion time step; the noisy latent representation of the background image at the current diffusion time step is used to extract background features through a U-Net encoder; the background features are then mapped through a multilayer perceptron to obtain modulated background features; The modulated background features are injected as background information into the dual attention block of the DA-TransUNet network, then the query vector of the dual attention block... Key vector Sum value vector The calculation formula is: （4） （5） （6） in, The input features are for the dual attention block. , and This is the weight matrix. The learnable adjustment coefficient, These are the modulated background features; In the self-attention interaction block, the DA-TransUNet network for the global and local branches is processed according to equation (7). Output features of a dual attention block and The process is performed to obtain global-local joint features. ; （7） in, For rearrangement operations, For splicing operations; Using the global-local joint features as a value vector, the DA-TransUNet network of the global and local branches... The output features of each dual attention block are used as query vector and key vector, respectively, and interact through a self-attention mechanism to obtain global-local joint features of information interaction. The global-local joint features of information interaction are added to the global-local joint features, and the added features are successively subjected to rearrangement and splitting operations to obtain two feature sums, which are used as input features of the corresponding decoding layers of the DA-TransUNet network for the global and local branches. Step 4: Build a defect detection model. Train the defect detection model using the expanded PCBA defect detection dataset, and then use the trained defect detection model for PCBA surface defect detection.

2. The PCBA surface defect detection method based on anomaly synthesis strategy according to claim 1, characterized in that, The defect detection model includes a feature extractor and a conditional normalization flow. The test PCBA image is processed by the feature extractor to extract multi-scale features, which are then mapped to the latent space by the conditional normalization flow to obtain a Gaussian distribution of the input features. The log-likelihood value is calculated based on the Gaussian distribution of the input features, and the log-likelihood value of the features at each location is transformed to the interval [0,1] to obtain the anomaly score at each location. Based on the anomaly scores at all locations, the Top-K average aggregation algorithm is used to calculate the anomaly score of the test PCBA image. If the anomaly score of the test PCBA image is greater than the anomaly threshold, a defect exists on the PCBA surface; otherwise, no defect exists.

3. The PCBA surface defect detection method based on anomaly synthesis strategy according to claim 2, characterized in that, The feature extractor uses a pre-trained EfficientNet-B6 backbone network, and the multi-scale features are the output features of the third, fourth and fifth feature extraction layers of the EfficientNet-B6 backbone network.

4. The PCBA surface defect detection method based on anomaly synthesis strategy according to any one of claims 1 to 3, characterized in that, The noise addition process of the dual-branch interconnected diffusion model is expressed as follows: （2） （3） in, , They are the first Latent representations of global and local anomalies with noise added at each diffusion time step. , These are the latent representations of global anomaly images and the latent representations of local anomaly images, respectively. It is the diffusion coefficient. , It is random noise.

Images