A generative data augmentation method for surface defect images

Abstract

This invention proposes a generative data augmentation method for surface defect images. This method establishes two adversarial models: a generative model and a discriminative model. These two models learn the data distribution of surface defect images and are trained alternately through a competitive mechanism, ultimately achieving generative data augmentation of surface defect images and enhancing the data distribution. By performing generative data augmentation on surface defect images, the sample size of the training dataset is increased, enriching the feature representation of the data. This alleviates the problem of insufficient data for subsequent deep learning tasks and improves the generalization ability of the model. The new data generated by this method retains the inherent attributes of real data, providing more comprehensive and richer training data for subsequent surface defect detection and classification, laying the foundation for improving the performance of subsequent models.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of surface defect images, and particularly relates to a surface defect image generative data enhancement method. BACKGROUND

[0002] Surface defects refer to various unevenness and mutation phenomena on the surface of materials. Surface defects of industrial products not only affect the appearance, but also have a serious impact on their use performance. When using deep learning technology to identify and locate surface defects, a large number of surface images of industrial products are needed to train the deep learning network. However, the number of defects in the surface images collected in the factory is limited, which is difficult to meet the training of the deep learning network model.

[0003] Data enhancement technology can expand the amount of surface defect data. It makes each collected defect image play a greater utilization efficiency in the case of limited surface defect samples. Traditional data enhancement adopts pre-set data transformation rules to expand the data on the existing data, including scaling, cropping and mirroring to a certain extent, and adding salt and pepper noise or Gaussian white noise in the image. However, these methods essentially only transform the existing defects, and the similarity of the obtained defect features and the original features is high, and the generalization of the deep learning model training process is limited.

[0004] In recent years, deep learning generation technology has developed rapidly and has achieved great success in tasks such as character generation and landscape generation. In these task scenarios, the images to be generated have macroscopically understandable semantics, and it is easy to learn the characteristics of the target to be generated from the original samples. However, surface defects have random surface size, uneven shape, irregular distribution, and various types, which makes the existing generation model unable to be directly migrated to the surface image generation task.

[0005] In summary, there is a need to find a new surface defect image data enhancement method. First, a surface real image and simulation image discrimination model is constructed, then a surface simulation image generation model is designed, the difference between the generated simulation image and the real image is compared, the generation model parameters are optimized until the surface defect image close to the real image is generated, and the surface simulation image generation model obtained in this way can be used to generate surface simulation images. SUMMARY

[0006] In order to solve the problem of surface defect image sample scarcity in the prior art, which is difficult to meet the training of the deep learning network model, and the generalization of the existing data enhancement technology for the deep learning model training process is limited, the present application provides a surface defect image generative data enhancement method.

[0007] The technical scheme of the present application is: a surface defect image generation type data enhancement method, comprising the following steps:

[0008] Step 1: obtain the surface defect original image, remove the image with poor data quality, and obtain the original real sample data set X;

[0009] Step 2: set the training parameters of the surface simulation image generation model G(z) and the surface real image and simulation image discrimination model D(x): learning rate l r , attenuation rate λ1, loss function type L D , optimizer type; maximum training round N, early stopping ratio P;

[0010] Step 3: build the surface real image and simulation image discrimination model D(x), and randomly initialize the parameters θ D0 of the surface real image and simulation image discrimination model D(x);

[0011] Step 4: design the surface simulation image generation model G(z), and randomly initialize the surface simulation image generation model parameters θ G ;

[0012] Step 5: assign the label of the image in the original real sample image data set X as 1;

[0013] Step 6: input the image x real in the original real sample data set X into the surface real image and simulation image discrimination model D(x), and the obtained model output is D(x real );

[0014] Step 7: calculate the loss L D (D(x real ), 1) between D(x real ) and the label of the image in the original real sample image data set X, and update the model parameters of D(x) by back propagation;

[0015] Step 8: randomly generate n noises z, input the noise z into the surface simulation image generation model G(z), generate a batch of surface simulation defect images x fake , and assign the label of the surface simulation image x fake as 0;

[0016] Step 9: pass the surface simulation defect image x fake in step 9 through the surface real image and simulation image discrimination model D(x), and the output is D(x fake );

[0017] Step 10: calculate D(x fake ) in step 9 and the surface simulation defect image x fakeLoss L between the tags D (D(x fake ),0), and the model parameters of D(x) are updated by back propagation;

[0018] Step 11: calculate the loss L between D(x fake ) and the tags of the images in the original real sample image data set X D (D(x fake ),1), and the model parameters of G(z) are updated by back propagation;

[0019] Step 12: randomly generate n noise z, input the noise z into the surface simulation image generation model G(z), generate a batch of images x, and use the real image and simulation image discrimination model D(x) to discriminate the batch of images x, and the proportion of the number of real images in the total number of images in the batch is p;

[0020] Step 13: repeat steps 6 and 12, when p>P, the training is completed, and a surface simulation image generation model G(z) capable of generating images close to real surface defect images is obtained; if the number of training rounds reaches N, but still does not meet the requirements, then re-initialize the parameters of the surface simulation image generation model G(z) and the surface real image and simulation image discrimination model D(x), and then repeat steps 6 to 12 until a model that meets the result of p>P is obtained.

[0021] Further, in step 2, the loss function type L D The formula is:

[0022]

[0023] Where y is the real label for discrimination, is the result of the discriminator.

[0024] Further, in step 3, the surface real image and simulation image discrimination model D(x) is composed of N1 convolution layers, M1 batch normalization layers and activation functions.

[0025] Further, in step 4, the surface simulation image generation model G(z) is composed of N2 deconvolutions, M2 batch normalization layers and activation functions.

[0026] Inventive effect

[0027] The technical effect of the present application is as follows: the present application proposes a surface defect image generative data enhancement method, which realizes generative data enhancement of surface defect images by constructing a generative adversarial model. Specifically, the advantages include the following aspects:

[0028] (1) The model in the application is composed of a generative model and a discriminative model two modules, wherein the surface simulation image generation model is responsible for generating new surface defect images, and the surface real image and simulation image discriminative model is responsible for identifying whether the generated image is real. The two models realize the data distribution learning of the surface defect image through the antagonistic mechanism of steps 6 to 12. The generative model adopts an inverse convolutional neural network, and generates a new surface defect image through random noise. The p discriminative model adopts a convolutional neural network, and performs two classification on the generated image and the real image, and optimizes the generative model to improve the authenticity of the generated image. The alternating training of the generative model and the discriminative model makes the finally generated image difficult to be distinguished by the discriminative model, and achieves the effect of enhancing the data distribution. Through the generative data enhancement of the surface defect image, the number and the type of the training data set are increased, the feature expression of the data is enriched, the problem of insufficient data amount for subsequent other deep learning tasks is reduced, and the generalization ability of the model is improved.

[0029] (2) The new data generated by the method retains the inherent properties of the real data, provides more comprehensive and real training data for subsequent surface defect detection and classification, and realizes the improvement of the model performance.

[0030] (3) The method is an effective data-driven method, which realizes the data enhancement of the surface defect image by learning the distribution function of the spatial data.

[0031] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0032] Fig. 1 : Surface defect image generative data enhancement method construction flowchart

[0033] Fig. 2 : Industrial product surface real image

[0034] Fig. 3 : Industrial product surface simulation image DETAILED DESCRIPTION

[0035] Reference Figs. 1-3 , the application proposes a surface defect generative data enhancement method aiming at the problem of surface defect image sample scarcity, realizes the expansion of surface defect image samples, and the steps of the method are as follows.

[0036] Step 1: Obtain the surface defect original image, remove the images with poor data quality such as blurred images, and obtain a high-quality original real sample data set X.

[0037] Step 2: Analyze the original real sample dataset X, classify the surface defects contained therein, and summarize the surface defect types and characteristics.

[0038] Step 3: Set the training parameters of the surface simulation image generation model G(z) and the surface real image and simulation image discrimination model D(x): learning rate l r , decay rate λ1, loss function type L D , optimizer type. Maximum training rounds N, early stopping ratio P.

[0039] Step 4: Build the surface real image and simulation image discrimination model D(x), which consists of N1 convolutional layers, M1 batch normalization layers, and activation functions. Initialize the parameters θ D0 of the surface real image and simulation image discrimination model D(x).

[0040] Step 5: Design the surface simulation image generation model G(z), which consists of N2 deconvolutions, M2 batch normalization layers, and activation functions. Randomly initialize the generation model parameters θ G .

[0041] Step 6: Assign the label of the image in the original real sample image dataset X as 1.

[0042] Step 7: Pass the image in the original real sample dataset X through the surface real image and simulation image discrimination model D(x), and the image x real after the model, the output is D(x real ).

[0043] Step 8: Calculate the loss L real (D(x real ),1) between D(x D ) and the label of the image in the original real sample image dataset X, and update the model parameters of D(x) by backpropagation.

[0044] Step 9: Randomly generate a batch of noise z, input the noise z into the surface simulation image generation model G(z), generate a batch of surface simulation defect images x fake , and assign the label of the surface simulation image x fake as 0.

[0045] Step 10: Pass the surface simulation defect image x fake through the real image and simulation image discrimination model D(x), and the output is D(x fake ).

[0046] Step 11: Calculate the loss L fake between D(x fake ) and the label of the surface simulation defect image x D(D(x fake ),0), and backpropagating to update the model parameters of D(x).

[0047] Step 12: Calculate the loss L fake ) between D(x D ) and the label of the image in the original real sample image dataset X. fake ,1), and backpropagating to update the model parameters of G(z).

[0048] Step 13: Randomly generate a batch of noise z, input the noise z into the surface simulation image generation model G(z) to generate a batch of images x, and use the real image and simulation image discrimination model D(x) to discriminate the batch of images x. The proportion of the number of images judged as real images in x is p.

[0049] Repeat steps 7 and 13. When p>P, the training is completed. The surface simulation image generation model G(z) capable of generating surface defect images close to real surface defect images is obtained. If the number of training rounds reaches N and still does not meet the requirements, reinitialize the parameters of the surface simulation image generation model G(z) and the surface real image and simulation image discrimination model D(x) and repeat steps 7 to 13 until a model that meets the result of p>P is obtained.

[0050] The embodiments of the present application are described in detail below. The embodiments described are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0051] Database introduction of this example: The database of this example is composed of industrial product surface images collected in the factory. The total number of images is 200. Each image has a resolution of 640x640.

[0052] Step 1: Data cleaning of the surface image data collected in the factory, removing low-quality images to obtain a high-quality original real sample dataset X.

[0053] Step 2: Analysis of the original real sample dataset X, classification of the surface defects contained therein, and induction of surface defect types and characteristics.

[0054] Step 3: Design of the training parameters of the surface simulation image generation model G(z) and the surface real image and simulation image discrimination model D(x): set the learning rate to 0.0003, the decay rate to 0.5, and the loss function to the cross-entropy loss function: The optimizer uses the Adam optimizer, the maximum number of training rounds is 500, and the early stopping ratio is 0.8.

[0055] Step 4: Construct the network structure of the discrimination model D(x) for real and simulated images. The structure consists of: first, a convolution operation, followed by the LeakyReLU activation function; then, another convolution operation, batch normalization, and the LeakyReLU activation function, repeating this structure three times; finally, another convolution operation, and finally, classification using a Sigmoid activation function. Initialize the network parameters. The expression for the LeakyReLU activation function is:

[0056]

[0057] Step 5: Construct the network structure of the surface simulation image generation model G(z). The structure consists of one deconvolution, one batch normalization, and one ReLU activation function, repeated three times. Next, perform a deconvolution operation, and finally pass through the Tanh activation function. Initialize the generation model parameters θ by adding random noise. G The expressions for the ReLU activation function and the Tanh activation function mentioned are as follows:

[0058] ReLU(x) = (x) + =max(0,x)

[0059]

[0060] Step 6: Assign a label of 1 to the images in the original real sample image dataset X.

[0061] Step 7: Using the discriminant model D(x) between real and simulated images from the original real sample dataset X, image x... real After the model is applied, the output is D(x) real ).

[0062] Step 8: Calculate D(x) real The cross-entropy loss L between the labels of images in the dataset X and the original real sample image dataset. D (D(x real ),1), and use the Adam optimizer to backpropagate and update the model parameters of D(x).

[0063] Step 9: Randomly generate a batch of noise z, input the noise z into the surface simulation image generation model G(z), and generate a batch of surface simulation defect images x. fake and the surface simulation image x fake The label is assigned a value of 0.

[0064] Step 10: Transfer the surface simulation defect image x fake The discriminant model D(x) based on real and simulated images outputs D(x).fake )。

[0065] Step 11: Calculate the loss L fake ) between D(x fake ) and the label of the image in the original real sample image dataset X. D (D(x fake ), 0), and use the Adam optimizer to update the model parameters of D(x) by backpropagation.

[0066] Step 12: Calculate the loss L fake ) between D(x D ) and the label of the image in the original real sample image dataset X. fake (D(x ), 1), and use the Adam optimizer to update the model parameters of G(z) by backpropagation.

[0067] Step 13: Randomly generate a batch of noise z, input the noise z into the surface simulation image generation model G(z), generate a batch of images x, and use the discriminant model D(x) of real images and simulation images to distinguish the images x, and the proportion of real images in x is p.

[0068] Repeat steps 7 and 13, when p > 0.8, complete the training. Get the surface simulation image generation model G(z) that can generate images close to real surface defect images. If the training round number reaches 500 and still does not meet the requirements, re-initialize the parameters of the surface simulation image generation model G(z) and the discriminant model D(x) and repeat steps 7 to 13. Until the model that meets p > 0.8 is obtained.

[0069] Step 14: Use the trained model G(z) to generate surface simulation images.

[0070] Figure 1 is a real image of an industrial product surface. Fig. 2 Figure 2 is a simulation image of an industrial product surface generated using the method of the present application. It can be seen that the effect of the simulation image is very close to the real surface defect image, and can be further used for deep learning tasks.

[0071] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the principles and purposes of the present application within the scope of the present application.

Claims

1. A surface defect image generative data augmentation method, characterized by, comprising the steps of: Step 1: Obtain the original image of surface defects, remove images with poor data quality, and obtain the original real sample dataset ; Step 2: Set up surface simulation image generation model , surface real image and simulation image discrimination model Training parameters: learning rate , decay rate , loss function type , optimizer type; maximum training rounds , early stop ratio is ; Step 3: building a surface real image and simulation image discrimination model , randomly initializing parameters of the surface real image and simulation image discrimination model ;​ Step 4: Designing a surface simulation image generation model Randomly initializing surface simulation image generation model parameters ; Step 5: The original real sample image dataset The label of the image is assigned as 1. Step 6: Obtain the original real sample dataset Images in Input surface real image and simulation image discrimination model The model output is ; Step 7: Compute and the original real sample image dataset loss between labels of images in and backpropagate updates to the model parameters; Step 8: Randomly generate n noises Model for generating surface simulation images Input noise Generate a batch of surface simulation defect images and surface simulation images The label is assigned a value of 0; Step 9: surface simulation defect image in step 8 discrimination model of surface real image and simulation image , output is ; Step 10: Compute the loss between the surface simulation defect image in Step 9 and the surface simulation defect image in Step 8 and backpropagate the update to the model parameters ; Step 11: Compute and the original real sample image dataset loss between labels of images in and backpropagate updates to the model parameters; Step 12: Randomly generate n noises Model for generating surface simulation images Input noise Generate a batch of images This batch of images Discriminant model using real and simulated images The proportion of images identified as real to the total number of images in the batch is p. Step 13: repeat step 6 and 12, when P, finish training, get surface simulation image generation model which can generate surface simulation image close to real surface defect image ; if the number of training rounds reaches N, still not meet the requirements, then re-initialize the parameters of surface simulation image generation model and surface real image and simulation image discrimination model , and then repeat step 6 to 12 until the model that meets the result can be obtained.

2. The surface defect image generative data augmentation method of claim 1, wherein, In step 2, the loss function type The formula is: wherein y is the true label for which the discrimination is performed, the result of the discrimination by the discriminator.

3. The surface defect image generative data augmentation method of claim 1, wherein, In the step 3, the surface real image and simulation image discrimination model The structure is composed of one convolutional layer, one batch normalization layer and an activation function.

4. The surface defect image generative data augmentation method of claim 1, wherein, In step 4, the surface simulation image generation model By Layer deconvolution, Layer batch normalization layer and activation function.

Images