A small-sample surface defect identification method based on inter-domain common feature transfer.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2023-12-08
- Publication Date
- 2026-06-30
AI Technical Summary
Under small sample conditions, existing technologies struggle to effectively identify surface defects on printed circuit boards (PCBs), especially due to the scarcity of defect samples, significant differences between different batches of products, and obvious common defect characteristics, resulting in insufficient recognition accuracy and generalization ability of deep learning methods.
By using a method based on the transfer of common features between domains, diverse and realistic samples are generated by utilizing the diverse common content features of the source domain and the unique style features of the target domain. The surface defect recognition model is then trained by combining variational autoencoders and generative adversarial networks to extract and separate features.
It improves the generalization ability of the surface defect recognition model, reduces data collection and annotation costs, and enhances recognition accuracy and stability under small sample conditions.
Smart Images
Figure CN117636096B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of industrial surface defect detection, and more specifically, relates to a small sample surface defect identification method based on the transfer of common features between domains. Background Technology
[0002] Surface defect detection is widespread in manufacturing, and surface defect identification is a crucial step in industrial automation. Effective surface defect identification methods can reduce production costs, improve efficiency, and enhance product quality. Currently, deep learning methods, which rely on large amounts of data, are widely used in surface defect identification. However, in actual industrial production, there are often small-sample scenarios, such as scarce defect samples or time-consuming and labor-intensive sample collection and labeling, which significantly reduces the accuracy of deep learning methods in surface defect identification. Therefore, how to achieve effective product surface defect identification under small-sample conditions has become a pressing problem for the manufacturing industry.
[0003] Taking printed circuit boards (PCBs) as an example, PCBs are the foundation of the modern information industry and are widely used in various high-end equipment manufacturing fields such as computers, communication electronic equipment, and military systems. As an important carrier for electrical connection and support, their quality has a great impact on the stability and safety of various high-end equipment products.
[0004] PCB surface defects exhibit the following characteristics: 1) Few defect samples: Collecting a large number of PCB defect samples from the same batch is extremely difficult due to short PCB production line changeover cycles, low PCB manufacturing defect rates, and high costs associated with collecting defect samples. 2) Differences between different batches: PCB products are diverse, and differences exist between different batches. Defect identification algorithms targeting different batches of PCBs suffer from poor stability and insufficient generalization ability. 3) Common defect characteristics in PCB products: Bare PCB products commonly exhibit defects such as residual copper, short circuits, broken lines, missing parts, and protrusions, sharing common defect characteristics.
[0005] For surface defect detection, manual inspection requires a high level of expert knowledge and experience, and inspectors are prone to mental fatigue after long hours, leading to errors in identification. Traditional machine learning methods rely on manually designing feature extraction rules for defect images, which is time-consuming, labor-intensive, and the models are susceptible to interference and noise. Deep learning-based methods often rely on a large number of labeled samples, and the model's generalization ability deteriorates under small sample conditions. In general, among surface defect identification methods with small sample sizes, data augmentation is the most direct and easily understood method to improve the identification effect of surface defects with small sample sizes. Using adversarial generative methods for data augmentation is currently a commonly used approach.
[0006] In small-sample surface defect identification, current image enhancement-based methods only consider the current task sample when performing image enhancement, without utilizing cross-domain common feature knowledge to assist in image generation. The original small sample has limited available information, so the samples generated on this basis also have certain limitations. Summary of the Invention
[0007] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for identifying surface defects in small samples based on the transfer of common features between domains. This method utilizes diverse common content features of the source domain and unique style features of the target domain to generate diverse and realistic samples of the target domain, thus solving the problem of limited available information in small samples.
[0008] To achieve the above objectives, according to one aspect of the present invention, a method for identifying small-sample surface defects based on inter-domain common feature transfer is provided, the method comprising the following steps:
[0009] (1) Collect defect image sample data and construct a surface defect recognition model. The surface defect recognition model includes a feature extraction and separation module, an image generation module and a defect recognition module. The domain in which small samples exist in the current task is taken as the target domain, and the domain in which there are sufficient similar samples is taken as the source domain.
[0010] (2) Based on the feature extraction and separation module, the common content features and unique style features of the source domain are separated, and the common content features of the source domain are extracted; at the same time, the unique style features of the target domain are extracted.
[0011] (3) The image generation module combines the common content features transferred from the source domain with the unique style features of the target domain to generate samples of the target domain;
[0012] (4) Input the generated target domain image and the original target domain image into the defect recognition module to train the defect recognition module, and then use the trained surface defect recognition model to recognize surface defects.
[0013] Furthermore, G Src D is a source domain defect image generator. Src E is a source domain defect image discriminator. Content E Style These are encoders for common content features and unique style features, respectively; the separation of common content features and unique style features of source domain defect images relies on variational autoencoders and generative adversarial networks.
[0014] Furthermore, the source domain image is encoded using a source domain common content feature encoder. Heyuan Domain Unique Style Feature Encoder Obtain common content features of the source domain Unique style characteristics of Heyuan Domain KL divergence loss is used to encourage the learned latent variables to follow the prior distribution;
[0015]
[0016] Common content features of the source domain Unique style characteristics of Heyuan Domain Enter G Src The generated source domain image is obtained. For image reconstruction loss, encourage and This effectively decouples features and enables the generator to produce more realistic images.
[0017]
[0018] Furthermore, in the stages of extracting and separating common content features and unique style features from the source domain, the loss function during training is:
[0019]
[0020] In the formula, λ1, λ2, λ3, and λ4 are the coefficients of the corresponding losses. For KL divergence loss, For image reconstruction loss, For feature reconstruction loss, To combat the losses.
[0021] Furthermore, in the stage of extracting domain-specific style features of the target domain, the loss function during training is:
[0022]
[0023] Furthermore, the loss function during the training phase of the image generation module is:
[0024]
[0025] Furthermore, the loss function of the defect identification module is:
[0026]
[0027] Loss of the defect identification module C for:
[0028] loss C =Loss CE
[0029] Iterative optimization of parameters:
[0030]
[0031] in, Let μ be the parameter of the defect identification module in the i-th iteration. i Let be the learning rate of the model in the i-th iteration. Let $\frac{i+1}{i}$ be the parameters of the defect identification module in the (i+1)th iteration, and $\frac{i}{i}$ be the loss function. C This is the overall loss function for the defect identification module.
[0032] Furthermore, in the target domain generator G Tar During training, Add loss Use it to force the generation of diversity between target domains; The definition is as follows:
[0033]
[0034] R is a pre-trained relational network used to learn to map cross-domain image pairs to values representing their content similarity. It is used to force diversity in content generation between target domains. The loss function of the R network during pre-training is... Add adversarial loss to the generative adversarial network in the target domain. Update G Tar and D Tar To generate more realistic images of defects in the target domain.
[0035] The present invention also provides a small sample surface defect identification system based on the transfer of common features between domains. The system includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it performs the small sample surface defect identification method based on the transfer of common features between domains as described above.
[0036] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the small-sample surface defect identification method based on the transfer of common features between domains as described above.
[0037] In summary, compared with the prior art, the small-sample surface defect identification method based on inter-domain common feature transfer provided by the present invention has the following advantages:
[0038] 1. The method of the present invention decouples common content features and unique style features, and extracts common content features of the source domain, so that the content and style can be controlled when synthesizing samples.
[0039] 2. This invention can maintain the diverse relationships of common content feature vectors during migration. On the basis of inheriting the diversity within the source domain, it makes full use of the transferred cross-domain common knowledge and combines the features of the two domains to effectively generate target domain samples.
[0040] 3. This invention utilizes cross-domain common feature knowledge to expand the original small sample dataset, increasing the number and diversity of data samples, thereby enabling the surface defect recognition model to achieve better generalization ability and improving the performance of deep learning models under small sample conditions. On the other hand, data augmentation can achieve similar effects on effective datasets as on large-scale datasets, thus saving the cost of data collection and labeling. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the target domain and source domain involved in the small sample surface defect identification method based on inter-domain common feature transfer provided by the present invention;
[0042] Figure 2 This is a flowchart of the small sample surface defect identification method based on inter-domain common feature transfer provided by the present invention;
[0043] Figure 3 This is a schematic diagram of the model structure involved in the small sample surface defect identification method based on the transfer of common features between domains;
[0044] Figure 4 This is a diagram illustrating the relationship between data augmentation and defect identification. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0046] The domain containing the current task's defect samples (small sample size) is designated as the target domain, while the domain containing sufficiently similar samples is designated as the source domain. Other similar products, due to their similarity to the current task in materials, appearance, etc., inevitably share certain common characteristics of defects. Thoroughly exploring and transferring knowledge from these similar products will effectively alleviate the information deficiency problem caused by the small sample size.
[0047] This invention provides a small-sample surface defect recognition method based on the transfer of common features between domains. The recognition method uses a designed feature extraction and separation module to extract and separate common content features and unique style features of the domain. Then, the extracted feature vectors are used to generate diverse and realistic samples in the image generation module. Finally, the generated samples and the original samples are input into the surface defect recognition model to improve the generalization ability of the surface defect recognition model and realize surface defect recognition under small sample conditions.
[0048] Please see Figure 1 and Figure 2 In training with small samples, the boundary of the target domain classifier often differs greatly from the boundary of the real target domain classifier. Data augmentation using cross-domain knowledge expands the original dataset so that the boundary of the trained classifier gradually approaches the boundary of the real target domain classifier.
[0049] Please see Figure 3 and Figure 4 The identification method mainly includes the following steps:
[0050] S1, collect defect image samples and complete data preprocessing.
[0051] The domain with few samples in the current task is designated as the target domain, while the domain with sufficient similar samples is designated as the source domain. Assuming the source domain dataset has been pre-collected and organized, for the target domain dataset, industrial cameras are used to photograph surface defects in products during industrial production, completing the collection of small sample data for different defect types. The collected images are resized to 3×128×128 for input into the deep neural network. Simultaneously, the data is preprocessed, adjusting the pixel value range of the defect images to [-1,1], which accelerates training while improving the model's robustness.
[0052]
[0053] In the formula, p i Image pixel values, These are the adjusted pixel values. For each defect type, the collected samples are used as the original dataset for the target domain, and the dataset is divided according to the sample category.
[0054] S2, Construct a surface defect recognition model, which includes a feature extraction and separation module, an image generation module, and a defect recognition module.
[0055] In this embodiment, the feature extraction and separation module, the image generation module, and the defect recognition module are all deep neural network models, according to... Figure 3A small-sample surface defect generation model based on the transfer of inter-domain common feature extraction is constructed, where E represents the variational autoencoder for image encoding, G represents the image generator, and D represents the image discriminator. Then, according to... Figure 4 Construct a complete surface defect recognition model based on the extraction and transfer of common features between domains.
[0056] S3 initializes the parameters of the surface defect recognition model.
[0057] In this embodiment, the parameters in the surface defect recognition model are initialized to θ, wherein the feature extraction and separation module, the image generation module, and the defect recognition module are respectively initialized to θ. F θ G θ C :
[0058] θ={θ F ,θ G ,θ C}
[0059] S4 separates the common content features and unique style features of the source domain, and extracts the common content features of the source domain.
[0060] In this embodiment, source domain samples are input into the feature extraction and separation module to extract and separate common content features and unique style features of the domain. Strictly speaking, a domain consists of two parts: a feature space X and a marginal probability distribution P(X). Here, the domain where small sample sizes exist in the current task is taken as the target domain, while the domain with sufficient similar samples is taken as the source domain. Src D is a source domain defect image generator. Src E is a source domain defect image discriminator. Content E Style These are encoders for common content features and unique style features, respectively. The separation of common content features and unique style features in source domain defect images relies on variational autoencoders (VAEs) and generative adversarial networks (GANs). The source domain image is encoded by the source domain common content feature encoder. Heyuan Domain Unique Style Feature Encoder Obtain common properties within the source domain
[0061] Capacity characteristics Unique style characteristics of Heyuan Domain KL divergence loss is used to encourage the learned latent variables to follow the prior distribution (normal distribution), which helps prevent overfitting and is a regularization method.
[0062]
[0063] Common content features of the source domain Unique style characteristics of Heyuan Domain Enter G Src The generated source domain image is obtained. For image reconstruction loss, encourage and This effectively decouples features and enables the generator to produce more realistic images.
[0064]
[0065] The generated source domain image Re-input source domain common content feature encoder Heyuan Domain Unique Style Feature Encoder New common content features of the source domain are obtained. Unique style characteristics of Heyuan Domain The same KL divergence loss is used to encourage the learned latent variables to follow the prior distribution (normal distribution), which helps prevent overfitting and is a regularization method.
[0066]
[0067] The loss is used to reconstruct common content features of the source domain. To reconstruct loss for common style features common to the source domain, the source domain generator is encouraged to generate more realistic images:
[0068]
[0069]
[0070] In addition, due to the same common content features and different unique style characteristics Generate two images Adding counter-loss Encourage encoders with unique style characteristics It can effectively separate unique style features while preventing the generator from ignoring domain-specific style features.
[0071]
[0072]
[0073] The total training loss function for this stage is:
[0074]
[0075] Where λ1, λ2, λ3, and λ4 are the coefficients of the corresponding losses. For KL divergence loss, For image reconstruction loss, For feature reconstruction loss, To combat the losses.
[0076] S5, the source domain model is used to extract the domain-specific style features of the target domain (small sample) to obtain the domain-specific style features of the target domain.
[0077] In this embodiment, the unique style features of the target domain are first extracted. The difference between the defect image synthesized based on the common content features of the source domain and the unique style features of the target domain and the original defect image in the target domain is defined as the target domain image reconstruction loss.
[0078]
[0079] KL divergence loss is used to encourage the learned target domain-specific style features to follow a prior distribution (normal distribution). Since there is relatively little data in the target domain, this approach helps prevent overfitting and is a regularization method.
[0080]
[0081] Loss of unique style characteristics reconstruction Encourage the separation of unique style features of the target domain.
[0082]
[0083] The total training loss function for this stage is:
[0084]
[0085] Where λ1, λ2, and λ3 are the coefficients of the corresponding losses, For KL divergence loss, For image reconstruction loss, The loss is reconstructed for the style features of the target domain. From this point onward, the unique style features of the target domain are... Common content features of source domain All images have been effectively extracted, laying the groundwork for the next step of surface defect image synthesis.
[0086] Combining S4 and S5, the overall parameters of the feature extraction and separation model are optimized. The calculation process is as follows:
[0087]
[0088]
[0089] in, Let μ be the parameter of the feature extraction and separation module in the i-th iteration.i Let be the learning rate of the model in the i-th iteration. Here are the parameters of the feature extraction and separation module in the (i+1)th iteration, and the loss is... F Let be the overall loss function of the feature extraction and separation module. For each sample, backpropagation of the gradient is performed, and the optimal feature extraction and separation module can be obtained through iterative optimization.
[0090] S6 randomly combines the common content features of the diverse source domains with the unique style features of the target domain to generate diverse and realistic samples for the target domain.
[0091] In this embodiment, the target domain generator G Tar Common content features transferred from the source domain are combined with specific style features unique to the target domain to generate samples for the target domain. In G Tar During training, Add loss Use it to force the generation of diversity between target domains. The definition is as follows:
[0092]
[0093] R is a pre-trained relational network used to learn to map cross-domain image pairs to values representing their content similarity. It is used to force diversity in content generation between target domains. The loss function of the R network during pre-training is... Then, similar to a typical GAN network, an adversarial loss is added to the generative adversarial network in the target domain. Update G Tar and D Tar To generate more realistic target domain defect images, the transferred samples are then mixed with the original target domain samples and input into a pre-trained surface defect recognition network to verify the effectiveness of the transfer generation for small sample size problems.
[0094] The overall loss function during the training phase of the image generation module is:
[0095]
[0096] Optimize the overall parameters of the image generation module, i.e. Figure 3 In stage 3, the calculation process is as follows:
[0097]
[0098] in, μ represents the parameters of the image generation module in the i-th iteration. i Let be the learning rate of the model in the i-th iteration. The parameters of the image generation module in the (i+1)th iteration are loss.G Let be the overall loss function of the image generation module. For each sample, backpropagation of the gradient is performed, and the optimal image generation module can be obtained through iterative optimization.
[0099] S7, the generated defect image and the original image are fused together to train the defect recognition module.
[0100] In this embodiment, the surface defect separation module uses a ResNet18 deep neural network. The generated target domain image and the original target domain image are input into the ResNet18 deep neural network for classification training. The loss function during the training phase is the cross entropy loss.
[0101]
[0102] Loss of the defect identification module C for:
[0103] loss C =Loss CE
[0104] Iterative optimization of parameters:
[0105]
[0106] in, Let μ be the parameter of the defect identification module in the i-th iteration. i Let be the learning rate of the model in the i-th iteration. Let $\frac{i+1}{i}$ be the parameters of the defect identification module in the (i+1)th iteration, and $\frac{i}{i}$ be the loss function. C Let be the overall loss function of the defect identification module. For each sample, backpropagation of the gradient is performed, and the optimal defect identification module can be obtained through iterative optimization.
[0107] S8, Surface defect image recognition of the target domain.
[0108] In this embodiment, the surface defect identification process for samples in the testing phase is as follows:
[0109] S801 performs data preprocessing on the collected raw target domain data samples;
[0110] S802: Input the processed target domain data samples into the trained ResNet18 surface defect classification network. Through the forward propagation of the last fully connected layer, the probability of each sample belonging to each known category is obtained, and the category with the highest probability is taken as the predicted defect category of the sample, so as to realize the surface defect identification of industrial products.
[0111] The present invention also provides a small sample surface defect identification system based on the transfer of common features between domains. The system includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it performs the small sample surface defect identification method based on the transfer of common features between domains as described above.
[0112] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the small-sample surface defect identification method based on the transfer of common features between domains as described above.
[0113] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for identifying surface defects in small samples based on the transfer of common features between domains, characterized in that, The method includes the following steps: (1) Collect defect image sample data and construct a surface defect recognition model. The surface defect recognition model includes a feature extraction and separation module, an image generation module and a defect recognition module. The domain in which small samples exist in the current task is taken as the target domain, and the domain in which there are sufficient similar samples is taken as the source domain. (2) Based on the feature extraction and separation module, the common content features and unique style features of the source domain are separated, and the common content features of the source domain are extracted; at the same time, the unique style features of the target domain are extracted. (3) The image generation module combines the common content features transferred from the source domain with the unique style features of the target domain to generate samples of the target domain; (4) Input the generated target domain image and the original target domain image into the defect recognition module to train the defect recognition module, and then use the trained surface defect recognition model to recognize surface defects. For source domain defect image generator, For source domain defect image discriminator, These are encoders for common content features and unique style features, respectively; the separation of common content features and unique style features in source domain defect images relies on variational autoencoders and generative adversarial networks; Source domain images are encoded using a source domain common content feature encoder. Heyuan Domain Unique Style Feature Encoder To obtain common content features of the source domain Unique style characteristics of Heyuan Domain ; KL divergence loss is used to encourage the learned latent variables to follow the prior distribution; Common content features of the source domain Unique style characteristics of Heyuan Domain enter The generated source domain image is obtained. ; For image reconstruction loss, encourage and This effectively decouples features and enables the generator to produce more realistic images. Among them, the generated source domain image Re-input source domain common content feature encoder Heyuan Domain Unique Style Feature Encoder New common content features of the source domain are obtained. Unique style characteristics of Heyuan Domain .
2. The small-sample surface defect identification method based on inter-domain common feature transfer as described in claim 1, characterized in that: In the extraction and separation stages of common content features and unique style features in the source domain, the loss function during training is: In the formula, , , , These are the coefficients corresponding to the loss. For KL divergence loss, For image reconstruction loss, For feature reconstruction loss, To combat the losses.
3. The small-sample surface defect identification method based on inter-domain common feature transfer as described in claim 2, characterized in that: During the extraction of domain-specific style features in the target domain, the loss function during training is: 。 4. The small-sample surface defect identification method based on inter-domain common feature transfer as described in claim 2, characterized in that: The loss function during the training phase of the image generation module is: 。 5. The small-sample surface defect identification method based on inter-domain common feature transfer as described in claim 2, characterized in that: The loss function for the defect identification module is: Loss of the defect identification module for: Iterative optimization of parameters: in, For the first The parameters of the defect identification module in the next iteration. For the first The learning rate of the model in each iteration. For the first The parameters of the defect identification module in the next iteration. This is the overall loss function for the defect identification module.
6. The small-sample surface defect identification method based on inter-domain common feature transfer as described in claim 4, characterized in that: In the target domain generator During training, Add loss This is used to force the generation of diversity among target domains; The definition is as follows: A pre-trained relational network is used to learn to map cross-domain image pairs to values representing their content similarity, which is used to force diversity in content generated between target domains. The loss function of the network during pre-training is Add adversarial loss to the generative adversarial network in the target domain. renew and To generate more realistic images of defects in the target domain.
7. A small-sample surface defect identification system based on inter-domain common feature transfer, characterized in that: The system includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it performs the small sample surface defect identification method based on the transfer of common features between domains as described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the small-sample surface defect identification method based on the transfer of common features between domains as described in any one of claims 1-6.
Citation Information
Patent Citations
CN115049627A
WO2020186914A1