Method for training a defect data generation model

Abstract

The application discloses a training method of a defect data generation model, relates to the field of deep learning, separates original defect images through a foreground-background separation network to obtain a first defect background image and a first defect foreground image; generates a false normal image based on a generation network; inputs the first defect foreground and background images and the false normal image into a fusion network to generate a fusion defect image; separates the fusion defect image through the foreground-background separation network to obtain a second defect background and foreground image; respectively calculates a first cycle consistency loss between the first and second defect background images and a second cycle consistency loss between the first and second defect foreground images; adjusts a GAN loss based on the numerical values of the first and second cycle consistency losses, and iteratively trains to obtain a defect data generation model. The scheme can generate a large amount of defect data under the condition of a small amount of existing data, and simultaneously has the diversity of data.

Description

Technical Field

[0001] This application relates to the field of deep learning, and in particular to a training method for a defect data generation model based on adversarial learning. Background Technology

[0002] Using deep learning to solve industrial defect detection problems has become mainstream. However, training a good model often requires a large amount of data; the larger the data volume, the stronger the model performance. But in practical applications, it is often difficult to collect a large amount of defect data. To address the problem of small defect sample sizes, using generative models to generate more types of samples to improve model performance is a data-level solution, which can also be considered a more advanced data augmentation scheme.

[0003] In related technologies, defect data generation methods based on adversarial learning include the following:

[0004] There are three generative methods: unconditional GAN-based, conditional GAN-based, and CycleGAN-based. The first method utilizes the characteristics of generative adversarial networks (GANs), using existing defective data as a training set to train the GAN. After training, random noise is input into the model, and it outputs defective data. The second method, which requires generating different types of defective data in practical applications, adds category control to the first method, allowing the model to output different categories of defective data. The third method, unlike the first two, inputs random noise; it inputs normal data and outputs defective data, allowing the generation of more defective data from a large amount of normal data.

[0005] The first two problems stem from the fact that training a good GAN network requires a large amount of data, which contradicts the application of defect data generation. The third method, utilizing a large amount of normal data, addresses some of the problems of the first two, but CycleGAN itself is a network designed for style transfer, and the generated defect data often lacks diversity. Furthermore, CycleGAN requires training multiple models to generate different categories of defect data.

[0006] Furthermore, evaluating GAN models is extremely difficult, and the current mainstream evaluation methods, FID and IS, both have certain limitations. FID is the most widely used and can effectively assess the realism of the generated data, but it cannot assess the diversity of the generated data. The biggest problem with IS is that if the generated data differs from ImageNet's categories, the metric becomes meaningless. Summary of the Invention

[0007] This application provides a training method for a defect data generation model, addressing the problems of insufficient defect data and unrealistic generated defect data during model training. The method includes:

[0008] The original defect image is separated by a foreground / background separation network to obtain a first defect background image and a first defect foreground image;

[0009] Generating fake normal images based on generative networks;

[0010] The first defect foreground image, the first defect background image, and the fake normal image are input into a fusion network for image fusion to generate a fused defect image;

[0011] The fused defect image is separated by the foreground-background separation network to obtain a second defect background image and a second defect foreground image;

[0012] Calculate the first cycle consistency loss of the first defect background image and the second defect background image, and the second cycle consistency loss of the first defect foreground image and the second defect foreground image, respectively.

[0013] The GAN loss is numerically adjusted based on the first and second cyclic consistency losses, and the defect data generation model is obtained through iterative training.

[0014] Specifically, the generation network is connected to an M-Zn mapping network in the front stage, and the foreground / background separation network is connected to an M-Zd mapping network in the front stage.

[0015] The M-Zn mapping network randomly inputs a Zn vector as a parameter, and the M-Zd mapping network randomly inputs a Zd vector as a parameter and a separate class label L. The mapping network includes a normalization layer and several fully connected layers, which are used to transform the input vector from a Gaussian distribution to a learnable spatial distribution.

[0016] Specifically, the foreground / background separation network separates the original defect image based on the separation category label and Zd vector to obtain the first defect background image and the first defect foreground image, wherein the first defect foreground image is the defect data image and the first defect background image is the normal image after the defect data has been removed.

[0017] Specifically, the GAN loss includes at least a first GAN loss, a second GAN loss, a third GAN loss, and a fourth GAN loss;

[0018] The first GAN loss is determined based on matching and difference calculation between the first defective background image and the real normal image;

[0019] The second GAN loss is determined based on matching and difference calculations between the second defective background image and the real normal image;

[0020] The third GAN loss is determined based on the matching and difference calculation between the fake normal image generated by the generative network and the real normal image.

[0021] The fourth GAN loss is determined by matching and calculating the difference between the fused defect image generated by the fusion network and the real normal image.

[0022] Specifically, the network structure of the GAN loss includes a generator and a discriminator, and the optimization objective is expressed as follows:

[0023]

[0024] in, The data is real data, z is Gaussian noise in the input parameter Zd vector, G is the generator, and D is the discriminator. The training process is to first freeze D and train G to ensure that the generated image can fool G; then freeze G and train D to ensure that D can identify more realistic fake normal images. The optimization objective is intuitively represented as a fake normal image generated by the generator that is similar to the real image, and the similarity is the consistency of the distribution in the high-dimensional space.

[0025] Specifically, the GAN loss is represented as follows:

[0026]

[0027] in, and Represent the Zn vector and the Zd vector; Indicates the loss of the third GAN. Represents a generator network The output is a fake normal image; Indicates the first GAN loss. This represents the foreground image of the first defect. This represents the background image of the first defect. Indicates the input foreground / background separation network The original defect image, Indicates converged network The three input images , and ; Indicates the loss of the fourth GAN. This represents the fusion defect image generated by the fusion network; Indicates the second GAN loss. Second defect background image; Cyclic consistency loss is used for loss constraints; This is a hyperparameter.

[0028] Specifically, the first cycle consistency loss and the second cycle consistency loss are represented as follows:

[0029]

[0030] in, Represents the mapping space.

[0031] Specifically, when the first cycle consistency loss and the second cycle consistency loss are 0, or decrease to a set threshold, the iterative training ends and the defect data generation model is obtained.

[0032] Specifically, after training and obtaining the defect data generation model, the method further includes:

[0033] The fused defect image output by the defect data generation model is trained with K folds to obtain the confidence scores of all images;

[0034] Abnormal data is identified based on the confidence level value, and then filtered out to obtain the true defect data;

[0035] The real defect data is classified, trained, and evaluated to obtain the evaluation results.

[0036] The beneficial effects of the technical solution provided in this application include at least the following: Based on a generative adversarial network (GAN), a foreground / background separation network and a fusion network are added. The foreground / background separation network can divide the original defect image into a first defect background image containing the normal portion and a first defect foreground image containing the defect portion. The fusion network takes the first defect background and foreground images, as well as the real defect image output by the generative network, as common inputs, thus fusing and outputting a fused defect image containing more detailed features. During the training phase, the fused defect image is iteratively calculated to further separate the second defect background and foreground images, and the cycle consistency loss is calculated with the first defect background and foreground images respectively, thereby adjusting the GAN loss and improving model accuracy. This not only fully utilizes the existing large amount of normal data and small amount of defect data, but also utilizes the fusion network to generate diverse defect data. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the generation method based on unconditional GANs.

[0038] Figure 2 This is a schematic diagram of the production method based on conditional GANs;

[0039] Figure 3 This is a schematic diagram of the generation method based on CycleGAN;

[0040] Figure 4This is a flowchart of the training method for the defect data generation model provided in the embodiments of this application;

[0041] Figure 5 This is a framework diagram of the training method for the defect data generation model;

[0042] Figure 6 This is a schematic diagram of the mapping network structure;

[0043] Figure 7 This is a schematic diagram of the generator network Gn. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0046] Figure 1 This is a schematic diagram of the generation method based on unconditional GANs; Figure 2 This is a schematic diagram of the production method based on conditional GANs; Figure 3 This is a schematic diagram of the generation method based on CycleGAN. A comparison shows that... Figure 2 exist Figure 1 Based on this, category control is added, allowing the model to output defect data of different categories. Figure 3 Unlike the previous two methods, this method takes normal data as input and outputs defective data, and can generate more defective data using a large amount of normal data.

[0047] Figure 4 This is a flowchart of the training method for the defect data generation model provided in this application embodiment, including the following steps:

[0048] S1, the original defect image is separated by a foreground / background separation network to obtain a first defect background image and a first defect foreground image.

[0049] The original defect data in this application is a small amount of input data necessary for model training. It can be a pre-made original image with defective content. In this solution, "defect" refers to data that introduces missing parts of the real image information when processing image data in industry or other sectors, such as images with noise, distorted images, or images with added encoding. The original background image in this solution is an image formed by artificially intervening in a normal image to introduce "defects." For example, adding noise to a layer of a normal image can be separated into layers using a foreground / background separation network. The first defective background image is the normal image after "exposing the defective data" (restored to the image before noise addition according to the module's parsing capability). The first defective background image is the image with the "exposed" noise (defective data).

[0050] S2 generates fake normal images based on a generative network.

[0051] like Figure 5 The diagram shows the framework of the training method for the defect data generation model. The generator network Gn is a network module specifically designed to generate fake normal images. These fake normal images are images virtually synthesized by the network module based on built-in parameters and input parameters. These images possess the characteristics of real photographs and do not exhibit any "defects." For example, AI-synthesized fake faces do not correspond to real people, yet the naked eye cannot distinguish them from the real thing.

[0052] It should be noted that there is no strict execution order for steps S1 and S2. They can be executed simultaneously, or S2 can be executed first and then S1. This application does not limit this.

[0053] The generator network is connected to an M-Zn mapping network in its pre-stage, and the foreground / background separation network is connected to an M-Zd mapping network in its pre-stage. The M-Zn mapping network receives random input parameters (Zn vector), and the M-Zd mapping network receives random input parameters (Zd vector) and separation class labels (L).

[0054] The structure of the mapping network is as follows Figure 6 As shown, the mapping network consists of a normalization layer and several cascaded fully connected layers. The mapping network maps any random vector z to w. Wherein, , Z is a random spatial distribution, and W is a learnable arbitrary spatial distribution, thus transforming the input vector from a Gaussian distribution to a learnable spatial distribution.

[0055] The generative network can be any multi-layered network structure (such as ResNet). The input *w* to the mapping network is fed into different layers to ensure diversity in the generated images across high, middle, and low levels. Its structure is as follows: Figure 7As shown, the content and number of block modules are set according to the actual situation, and each block module is connected to the input by w.

[0056] The overall structure of the foreground-background separation network is similar to that of the generator network. The difference is that w is obtained by inputting random input z and class label L. In addition, there are two output images: one is the foreground (defect) and the other is the background (normal).

[0057] S3, input the first defect foreground image, the first defect background image, and the fake normal image into the fusion network to perform image fusion and generate a fused defect image.

[0058] To obtain the largest quantity and diversity of defect data (images) with the fewest possible images, this scheme incorporates a fusion network. Instead of inputting the first defect foreground image, the first defect background image, and the generated fake normal image, the network performs image fusion. The fused image is different from any of the previous images, thus effectively ensuring diversity. In contrast, simple Cyclic Gan constraints result in low diversity.

[0059] S4. The fused defect image is separated by a foreground-background separation network to obtain a second defect background image and a second defect foreground image.

[0060] After generating the fused defect image, the foreground and background separation network is used for further separation. Similar to step S1, the result is a normal second defect background image and a second defect foreground image (defect data image).

[0061] S5, calculate the first cycle consistency loss of the first defect background image and the second defect background image, and the second cycle consistency loss of the first defect foreground image and the second defect foreground image, respectively.

[0062] In addition to generating foreground and background, a foreground / background generation network should also be able to separate the generated defect data into foreground and background. Figure 5 The two foreground / background separation networks shown share weights, and the foreground and background they generate should be very similar. Therefore, a cycle consistency loss can be used to constrain this. The formula is expressed as follows:

[0063]

[0064] in, Represents the mapping space.

[0065] S6, based on the numerical adjustment of the GAN loss using the first and second cyclic consistency losses, and iteratively trains to obtain the defect data generation model.

[0066] The first and second cycle consistency losses are training constraints. When the first and second cycle consistency losses are 0 or reduced to a set threshold, the condition for ending the iteration training is met; otherwise, the GAN loss is continuously adjusted based on the difference.

[0067] The GAN loss includes at least a first GAN loss, a second GAN loss, a third GAN loss, and a fourth GAN loss. The first GAN loss is determined by matching and calculating the difference between a first defective background image and a real normal image; the second GAN loss is determined by matching and calculating the difference between a second defective background image and a real normal image; the third GAN loss is determined by matching and calculating the difference between a fake normal image generated by the generator network and a real normal image; and the fourth GAN loss is determined by matching and calculating the difference between a fused defective image generated by the fusion network and a real normal image.

[0068] The GAN loss network structure includes a generator and a discriminator, and the optimization objective is represented as follows:

[0069]

[0070] in, The data is real data, z is Gaussian noise in the input parameter Zd vector, G is the generator, and D is the discriminator. These are generated fake sample images. This is the confidence score output by the discriminator. The intuitive optimization goal is to make the images generated by the generator G similar to real images. This similarity should be a consistency of distribution in a high-dimensional space, but it's difficult to constrain this simply with a loss function. The discriminator is used to distinguish between real and generated data; it can be simply understood as a binary classifier. The training process of a GAN involves first freezing D, training G to make the generated images fool G, then freezing G, training D to enable D to recognize more realistic generated images, and so on, iterating in this way.

[0071] The purpose of the iterative process is to adjust the internal parameters. The GAN loss is represented as follows:

[0072]

[0073] in, This represents the numerical value of GAN loss. Indicates the loss of the third GAN. Represents a generator network The output is a fake normal image; Indicates the first GAN loss. This represents the foreground image of the first defect. This represents the background image of the first defect. Indicates the input foreground / background separation network The original defect image, Indicates converged network The three input images , and ; Indicates the loss of the fourth GAN. This represents the fusion defect image generated by the fusion network; Indicates the second GAN loss. Second defect background image; Cyclic consistency loss is used for loss constraints; These are hyperparameters used for parameter adjustment during iteration.

[0074] Based on the above process, images are continuously input for training until the conditions are met to obtain a defect data generation model.

[0075] Considering the limitations of existing GAN evaluation methods, and since the ultimate goal is to improve model performance, the final model's evaluation metrics (such as accuracy) are directly incorporated into the training process. However, since a small amount of generated data will inevitably contain unexpected results, directly adding defective data to training may not necessarily lead to significant improvement. In fact, it could even have side effects if the GAN model collapses. Therefore, after training the model using defective data, it is necessary to clean the generated defective data, including the following steps:

[0076] The fused defect images output by the defect data generation model are trained using K-fold folding to obtain the confidence scores of all images.

[0077] Abnormal data is identified based on the confidence level, and then filtered out to obtain the true defect data.

[0078] The real defect data is classified, trained, and evaluated to obtain the evaluation results.

[0079] Outlier data can be identified using the confidence learning module. Confidence learning estimates the joint distribution of noisy and true labels to identify and filter out erroneous samples. These filtered samples are then incorporated into real defective data for training and evaluation of the classification model. The best evaluation result is output as the current evaluation result of the GAN model.

[0080] This step differs from the limitations of existing GAN evaluation methods by directly using the evaluation metrics of the final detection model, ensuring effectiveness. Furthermore, it utilizes confidence learning to clean the data, removing a small amount of outlier data generated by GAN, thereby further improving overall performance.

[0081] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above. The devices and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible changes and modifications, or equivalent changes to equivalent embodiments without departing from the technical solution of the present invention. This does not affect the substantive content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A training method for a defect data generation model, characterized in that, The method includes: The original defect image is separated by a foreground / background separation network to obtain a first defect background image and a first defect foreground image; Generating fake normal images based on generative networks; The first defect foreground image, the first defect background image, and the fake normal image are input into a fusion network for image fusion to generate a fused defect image; The fused defect image is separated by the foreground-background separation network to obtain a second defect background image and a second defect foreground image; Calculate the first cycle consistency loss of the first defect background image and the second defect background image, and the second cycle consistency loss of the first defect foreground image and the second defect foreground image, respectively. The GAN loss is numerically adjusted based on the first and second cyclic consistency losses, and the defect data generation model is obtained through iterative training. The GAN loss includes a first GAN loss, a second GAN loss, a third GAN loss, and a fourth GAN loss. The first GAN loss is calculated based on the first defective background image and a real normal image. The second GAN loss is calculated based on the second defective background image and the real normal image. The third GAN loss is calculated based on the fake normal image and the real normal image. The fourth GAN loss is calculated based on the fused defective image and the real normal image. The network structure of the GAN loss includes a generator and a discriminator, with an optimization objective... It is expressed as follows: in, These are real data, and z is Gaussian noise in the input parameter Zd vector. and They are the generator and the discriminator, respectively. These are generated fake sample images. It is to identify the output confidence; the training process first freezes the output. ,train To ensure that the generated fake sample images are fooled Then freeze. ,train ,make sure It can identify fake normal images; Representing real data Expectations This represents the expectation of Gaussian noise.

2. The method according to claim 1, characterized in that, The generating network is connected to an M-Zn mapping network in the front stage, and the front-background separation network is connected to an M-Zd mapping network in the front stage. The M-Zn mapping network randomly inputs a Zn vector as a parameter, and the M-Zd mapping network randomly inputs a Zd vector as a parameter and a separate class label L. The mapping network includes a normalization layer and several fully connected layers, which are used to transform the input vector from a Gaussian distribution to a learnable spatial distribution.

3. The method according to claim 2, characterized in that, The foreground / background separation network separates the original defect image based on the separation category label and Zd vector to obtain the first defect background image and the first defect foreground image. The first defect foreground image is the defect data image, and the first defect background image is the normal image after the defect data has been removed.

4. The method according to claim 3, characterized in that, The GAN loss is represented as follows: in, and Represent the Zn vector and the Zd vector; Indicates the loss of the third GAN. Represents a generator network The output is a fake normal image; Indicates the first GAN loss. This represents the foreground image of the first defect. This represents the background image of the first defect. Indicates the input foreground / background separation network The original defect image, Indicates converged network The three input images , and ; Indicates the loss of the fourth GAN. This represents the fusion defect image generated by the fusion network; Indicates the second GAN loss. Second defect background image; Cyclic consistency loss is used for loss constraints; This is a hyperparameter.

5. The method according to claim 4, characterized in that, The first cycle consistency loss and the second cycle consistency loss are expressed as follows: in, Represents the mapping space.

6. The method according to claim 1, characterized in that, When the first cycle consistency loss and the second cycle consistency loss are 0, or decrease to a set threshold, the iterative training ends and the defect data generation model is obtained.

7. The method according to claim 1, characterized in that, After training and obtaining the defect data generation model, the method further includes: The fused defect image output by the defect data generation model is trained with K folds to obtain the confidence scores of all images; Abnormal data is identified based on the confidence level value, and then filtered out to obtain the true defect data; The real defect data is classified, trained, and evaluated to obtain the evaluation results.

Images