Defect detection method based on cycle generative adversarial network and structural similarity

Abstract

The present application relates to a kind of defect detection methods based on cycle generative adversarial network and structural similarity, comprising the following steps: step S1: obtaining defect picture and pre-processing, as training dataset;Step S2: CycleGAN model is built, and training is based on training dataset, obtain a model that will be mapped into defect picture without defect picture;Step S3: the picture to be detected is input into the trained CycleGAN model, and the difference between original drawing and repair drawing is compared using structural similarity algorithm, and difference binary image is obtained;Step S4: difference binary image is carried out connected domain noise reduction and morphological processing, and original image exists defect, then the white area in this binary image is the defect shape extracted.This application has the advantages of high detection accuracy, high robustness to complex textured surface, can accurately detect small defects, etc.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision and the field of industrial production, and in particular to a defect detection method based on a cycle generative adversarial network and structural similarity. BACKGROUND

[0002] "Object detection" is one of the core problems in the field of computer vision (CV), and its task is to locate the desired target (object) from a given image or video and identify different targets, outputting the category of the target. The application scenarios of object detection are very wide, among which the most widely studied are face detection, pedestrian detection, vehicle detection, etc. Defect detection is a specific application of object detection, and the target to be detected is the possible defects on the surface of the product. For a long time, industrial defect detection has relied on manual work. However, relying on manual work to detect surface defects is time-consuming, labor-intensive and strongly subjective, and the efficiency and accuracy cannot be well guaranteed. Therefore, using a computer to automatically detect defects has always been a research hotspot in the field of computer vision. The methods of defect detection based on computer vision can be divided into traditional methods and deep learning methods. The traditional methods of defect detection include support vector machine, naive Bayes, Markov random field, etc., while the deep learning methods are mostly based on convolutional neural network (CNN), such as SSD, YOLO, FCN, U-Net, etc. Compared with traditional methods, defect detection algorithms based on convolutional neural network can extract deeper defect features in most scenarios, and thus are often superior to traditional methods. Therefore, defect detection based on convolutional neural network has become the mainstream target detection algorithm and has played an excellent performance. However, the current deep learning methods for defect detection still have some shortcomings: 1. Most of them need labeled (image-level or pixel-level) training data, and labeling a large amount of data is undoubtedly time-consuming and labor-intensive and subjective; 2. The detection accuracy is not high or the detection error rate is too high, which is difficult to meet the requirements of practical applications.

[0003] Since industrial surface defect detection is still mainly completed by manual work, there are problems such as high labor intensity, low efficiency, and strong subjective influence, which cannot meet the high efficiency and high precision requirements of industrial production, and most of the defect detection methods based on machine vision also have the shortcomings of needing manual labeling of a large amount of training data, long training time, and poor detection effect on complex texture surfaces. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a defect detection method based on a cycle generative adversarial network and structural similarity, which has the advantages of high detection accuracy, high robustness to complex texture surfaces, and accurate detection of small defects.

[0005] To achieve the above purpose, the present application adopts the following technical solutions:

[0006] A defect detection method based on cycle generative adversarial network and structural similarity, comprising the following steps:

[0007] Step S1: Obtain a defect picture and pre-process it as a training data set;

[0008] Step S2: Construct a CycleGAN model and train it based on the training data set to obtain a model that maps the defect picture to a defect-free picture;

[0009] Step S3: Input the picture to be detected into the trained CycleGAN model, and use the structural similarity algorithm to compare the differences between the original picture and the repaired picture to obtain a difference binary image;

[0010] Step S4: Perform connected domain noise reduction and morphological processing on the difference binary image. If the original image has defects, the white area in the binary image is the extracted defect shape.

[0011] Further, the CycleGAN model includes a positive-to-negative generator, a negative-to-positive generator, a positive sample discriminator, and a negative sample discriminator.

[0012] Further, the generator adopts a structure of down-sampling + residual block + up-sampling, and two Involution blocks are added after the input image is down-sampled to 256 channels, located at both ends of the residual block.

[0013] Further, the discriminator first down-samples the input image using four convolution blocks, and the kernel size of the four convolution blocks is 4x4. After four convolutions, the channel number increases to 512, and then one convolution is performed to reduce the channel number to 1. Finally, global average pooling is used to compress it into a 1x1 scalar, whose value range is [0, 1], representing the confidence of the discriminator that the input image is a real sample.

[0014] Further, the CycleGAN model loss function includes adversarial L GAN loss cycle consistency loss L cycle and identity loss L identity .

[0015] Further, the L GAN includes L GAN_G and L GAN_D , which represent the optimization objectives of the generator G and the discriminator D, respectively, both using L2 loss measurement, as shown in equations (1) and (2),

[0016] L GAN_G =∑ p∈P (D n (G p2n (p))-1)2 +∑ n∈N (D p (G n2p (n))-1) 2 (1)

[0017] L GAN_D =∑ p∈P [(p-1) 2 +(D n (G p2n (p))-0) 2 ]+∑ n∈N [(n-1) 2 +(D p (G n2p (n))-0) 2 ] (2)

[0018] where 0 and 1 represent all-0 tensor and all-1 tensor respectively.

[0019] Further, a loss function replacement strategy is used, that is, the SSIM loss is used to realize L cycle in the first k epochs, and the loss function is replaced by L1 loss in the subsequent epochs to obtain similar illumination and color details to the original image:

[0020]

[0021] Further, the L identity adopts the same measurement strategy as L cycle , that is, first using SSIM loss and then using L1 loss, so the identity loss is described as:

[0022]

[0023] So the loss functions of the generator G and the discriminator D are respectively:

[0024] L G =L GAN_G +αL cycle +βL identity (5)

[0025] L D =L GAN_D (6)

[0026] where α and β are hyperparameters.

[0027] Further, the structural similarity algorithm is specifically:

[0028] Three comparison functions are defined: brightness comparison function l(x, y), contrast comparison function c(x, y), and structure comparison function s(x, y):

[0029]

[0030]

[0031]

[0032] wherein μ x , σ x , σ xy respectively represent the mean of x, the variance of x, the covariance of x and y;

[0033] Let C3=C2 / 2, then the SSIM index function is expressed as:

[0034]

[0035] In the above formula, the calculation of the mean, variance and covariance is completed in the local area within the sliding window, and the global SSIM score of the whole picture is finally obtained by traversing the whole picture pixel by pixel and averaging the scores of all local areas within the sliding window.

[0036] Compared with the prior art, the present application has the following beneficial effects:

[0037] The present application does not require manual annotation, has fast model convergence speed, high detection accuracy, strong robustness in different backgrounds, and the like, and can greatly improve the efficiency of defect detection BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a schematic diagram of the training stage in an embodiment of the present application;

[0039] Figure 2 is a schematic diagram of the generator G in an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of the discriminator D in an embodiment of the present application;

[0041] Figure 4 is a schematic diagram of the Involution in an embodiment of the present application;

[0042] Figure 5 is a data set in an embodiment of the present application;

[0043] Figure 6 is a folder structure for storing the training and test data sets in an embodiment of the present application;

[0044] Figure 7 is a trend of the test accuracy and error rate with the change of training epoch in an embodiment of the present application;

[0045] Figure 8is the detection effect when the model is trained for 50 epochs in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] The present application provides a defect detection method based on a cycle generative adversarial network and structural similarity, comprising the following steps:

[0048] Step S1: Obtain a defect picture and pre-process it as a training data set;

[0049] Step S2: Construct a CycleGAN model and train it based on the training data set to obtain a model that maps a defect picture to a non-defect picture;

[0050] Step S3: Input the picture to be detected into the trained CycleGAN model, and use a structural similarity algorithm to compare the differences between the original picture and the repaired picture to obtain a difference binary image;

[0051] Step S4: Perform connected domain noise reduction and morphological processing on the difference binary image, and if the original image has defects, the white area in the binary image is the extracted defect shape.

[0052] REFERENCE Figure 1 , which is a schematic diagram of the training phase, in this embodiment, P and N are positive and negative samples in the training set respectively, P and N are false samples obtained by mapping the original samples P and N once, and respectively and are false samples obtained by reverse mapping once. G p2n and are positive-to-negative and negative-to-positive generators, D p and D n are positive sample discriminators and negative sample discriminators respectively. L GAN is an adversarial loss used to measure the difference between the generated false sample and the real sample. L cycle is a cycle consistency loss used to measure the difference between the image obtained by sequentially mapping the real sample through two generators and the original image. L identity is an identity loss used to reduce the probability of incorrectly detecting non-defect samples as defect samples.

[0053] In this embodiment, the generator G adopts a classic U-net structure, i.e. a structure of down-sampling + residual block + up-sampling, as shown in Figure 2The Involution operator (see 5 for details) is a new network operator proposed in CVPR 2021, which has two characteristics of channel independence and spatial specificity (contrary to convolution), can better adapt to different spatial positions to extract different features, and can effectively reduce the network parameter amount when the input channel is large. Because the defect area and non-defect area of the image generally have different local features in the actual application of surface defect detection, the generator network in this method adds two Involution blocks after the input image is down-sampled to 256 channels, which are located at both ends of the residual block. The unique spatial specificity of the Involution operator helps to better extract defect features to improve the performance of the generator model.

[0054] In this embodiment, the discriminator D first uses four convolution blocks to down-sample the input image, and the kernel size of the four convolution blocks is 4x4, as shown in the following formula (3). Figure 3 After four times of convolution, the channel number of the input image increases to 512, and then it is reduced to 1 through one more convolution. Finally, a global average pooling is used to compress it into a scalar of 1x1, and the value range of the scalar is [0, 1], which represents the confidence of the discriminator that the input image is a real sample.

[0055] In this embodiment, the CycleGAN model loss function includes adversarial loss L GAN , cycle consistency loss L cycle and identity loss L identity .

[0056] L GAN is divided into L GAN_G and L GAN_D in code implementation, which represent the optimization objectives of the generator G and the discriminator D respectively, and both are measured by L2 loss, as shown in the following formulas (1) and (2), where 0 and 1 represent all-0 tensor and all-1 tensor respectively. The generator hopes to generate fake samples to deceive the discriminator, that is, the input fake sample makes the discriminator output as close to 1 as possible, while the discriminator hopes to distinguish the real sample and the fake sample generated by the generator as much as possible, so it hopes to output as close to 1 as possible for the real sample, and as close to 0 as possible for the fake sample.

[0057] L GAN_G =∑ p∈P (D n (G p2n (p))-1) 2 +∑ n∈N (D p (G n2p (n))-1) 2 (1)

[0058] L GAN_D=∑ p∈P [(p-1) 2 +(D n (G p2n (p))-0) 2 ]+∑ n∈N [(n-1) 2 +(D p (G n2p (n))-0) 2 ] (2)

[0059] L cycle cycleis the cycle-consistency loss. We want the sample obtained by the real sample after a forward mapping and a backward mapping to be as consistent as possible with the original sample, i.e., G n2p (G p2n (p))≈p、G p2n (G n2p (n))≈n.

[0060] The SSIM loss has the advantage of fast convergence speed, but simply using SSIM as the loss metric is easy to cause changes in the brightness of the generated picture and deviations in the color, which will introduce unnecessary noise and have a negative impact on the subsequent defect detection and extraction. In this embodiment, a loss function replacement strategy is used, i.e., using SSIM loss to realize Lcyclein the first k epochs to obtain faster network convergence speed. And in the subsequent epochs, the loss function is replaced by L1 loss to obtain similar illumination and color details to the original picture. Wherein k is empirically set to 10.

[0061]

[0062] In addition, in order to reduce the error rate of detection (the probability of detecting a defect-free sample as a defect sample), the method also uses an important loss function L identity , i.e., the identity loss. When the defect-free positive sample P is input into the generator G n2p , P does not need to be changed in any way, so we want the reconstructed sample P to be as close to P as possible to avoid introducing unnecessary interference noise and causing false detection. L identity The same measurement strategy as L cycle is used, i.e., first using SSIM loss and then using L1 loss, so the identity loss is described as:

[0063]

[0064] Therefore, the loss functions of the generator G and the discriminator D are respectively:

[0065] L G = L GAN_G + αLcycle +βL identity (5)

[0066] L D =L GAN_D (6)

[0067] Where α and β are hyperparameters. Preferably, α = 10 and β = 5.

[0068] In this embodiment, brightness and contrast, which are related to the structure of an object, are used as the definition of structural information in the image. Therefore, three contrast functions are defined: brightness contrast function l(x, y), contrast contrast function c(x, y), and structure contrast function s(x, y).

[0069]

[0070]

[0071]

[0072] Where μ x σ x σ xy Let C3 represent the mean of x, the variance of x, and the covariance of x and y, respectively. For simplification, let C3 = C2 / 2, then the SSIM exponential function can be expressed as:

[0073]

[0074] In image quality assessment, calculating the SSIM index locally is more effective than globally. First, the statistical features of an image are typically unevenly distributed spatially. Second, image distortion also varies spatially. Third, within normal viewing distances, people can only focus their gaze on a single region of an image, so local processing better aligns with the characteristics of the human visual system. Fourth, local quality detection can obtain a mapping matrix of spatial quality changes in the image, the results of which can be applied to other applications. Therefore, in the above formula, the calculation of the mean, variance, and covariance is completed within the local region of the sliding window, and the entire image is traversed pixel by pixel. Finally, the global SSIM score is the average of the scores of all local regions within the sliding window.

[0075] For the application of SSIM in this defect detection method, the SSIM sliding window size is also one of the hyperparameters. Preferably, the sliding window size is set to 9.

[0076] In this embodiment, the novel operator Involution, based on spatial specificity and channel independence, was proposed at CVPR 2021. Compared to traditional convolution, Involution cleverly divides the network computation into two parts: kemelgeneration and Multiply-Add, significantly improving the parameter and computational cost of traditional CNNs.

[0077] Figure 4 This is a schematic diagram of the Involution kernel. Unlike convolution kernels, the Involution kernel H is based on a single pixel rather than its relationship with neighboring pixels. Its shape depends on the size of the input feature map X. The kernel generation function generates the involution kernel based on a specific pixel, i.e., → K×K×. Therefore, the kernel H is defined as:

[0078]

[0079] Where W1 and W0 represent linear transformations, W0 reduces the dimensionality of a specific pixel representation from 1×1×C to 1×1×C / r (where r represents the reduction ratio), and σ represents Batch Normalization (BN) and non-linear activation. W1 transforms 1×1×C / r into K×K×G, and finally uses the generated involution kernel to perform convolution operations on the specific pixel region.

[0080] For surface defect detection, the Involution operator can meet the different visual capabilities required in different regions of the image (defect region and non-defect region), making the defect contours extracted by the model closer to the real defect contours, and helping to reduce the probability of misdetecting non-defect regions that are similar to defect features as defects.

[0081] Example 1:

[0082] In this embodiment, the computer environment is as follows: CPU: Intel(R) Xeon(R) CPU E5-2620 v4@2.10GHz, GPU: GeForce GTX 1080Ti, Memory: 128G, Python: 3.6.13, PyTorch: 1.7.1.

[0083] 1. Dataset Creation

[0084] The dataset is derived from Class 6 of the public dataset DAGM2007. Figure 5 a, Figure 5 Class b) and Class 7 Figure 5 c, Figure 5 (d) DAGM2007 is a commonly used dataset for industrial surface defect detection. It has 10 different categories, and each category has a training set and a test set. All images are grayscale images in PNG format, 512×512 pixels.

[0085] In this embodiment, all training and test images are resized to 256×256 pixels before being input into the model, and the data is standardized with a mean and standard deviation of 0.5 to accelerate the model's convergence speed. Furthermore, the training images are randomly cropped and flipped to improve the model's robustness.

[0086] The size of the dataset used in this method is shown in Table 1:

[0087] Table 1

[0088]

[0089] The folder structure for storing training and test datasets is as follows: Figure 6 As shown:

[0090] Define training hyperparameters

[0091] In this method, some hyperparameters need to be adjusted to obtain better detection results. After extensive comparative experiments, some of the main hyperparameters used in this method are as follows: Learning rate: 0.0002, Learning rate decay rate: 0.98, BatchSize: 1, epoch: 100, Image size: 256, SSIM window size: 9.

[0092] This method uses the argparse module to define the hyperparameters required for training. argparse is a standard command-line parsing module built into Python, which allows users to directly pass the parameters required for code execution from the command line, making code debugging easier.

[0093] 3. Model Training

[0094] After defining the various hyperparameters, start training the model. The training command is as follows:

[0095] python train.py--class_name Class6--ssimL1--involution--identity_loss--min_area 250

[0096] The output of the final part of the training is as follows:

[0097] ...

[0098] Epoch: 95 finish.Dn loss: 0.16, Dp loss: 0.24, G loss: 1.48, lr: 0.000029

[0099] Epoch: 96 finish.Dn loss: 0.15, Dp loss: 0.24, G loss: 1.65, lr: 0.000028

[0100] Epoch: 97 finish.Dn loss: 0.16, Dp loss: 0.25, G loss: 1.55, lr: 0.000028

[0101] Epoch: 98 finish.Dn loss: 0.17, Dp loss: 0.24, G loss: 1.43, lr: 0.000027

[0102] Epoch: 99 finish.Dn loss: 0.16, Dp loss: 0.24, G loss: 1.47, lr: 0.000027

[0103] Epoch: 100 finish.Dn loss: 0.17, Dp loss: 0.25, G loss: 1.43, lr: 0.000026

[0104] Where Dn loss represents the loss of discriminator Dn, Dp loss represents the loss of discriminator Dp, and G loss represents G. n2p and G p2n The sum of the losses is the sum of the losses of the generator and the discriminator. G loss measures the similarity between the real and generated images, but a lower G loss is not always better, as an excessively low G loss may lead to overfitting of the model to the training set, thus reducing the model's performance on the test set. As can be seen from the output, with the increase of training epochs, the generator loss and discriminator loss do not fluctuate significantly, indicating that the generator and discriminator have reached a balance at this point.

[0105] The model trained using this method underwent 100 iterations, taking approximately 3 hours.

[0106] 4. Model Testing

[0107] Load the saved model for testing, and observe the trends of accuracy and error rate with training epochs as follows: Figure 7 As shown; the detection performance after 50 epochs of model training is as follows. Figure 8 As shown, (a) is the test image, (b) is the repaired image, (c) is the extracted defect, and (d) is the label. It can be seen that this method only requires 50-60 epochs (approximately 1.5 hours) of training to achieve good image repair and defect detection results. The faster training speed facilitates rapid deployment in real-world application scenarios.

[0108] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A defect detection method based on cycle generative adversarial network and structural similarity, characterized in that, The method comprises the following steps: Step S1: obtaining a defect picture and preprocessing as a training data set; Step S2: constructing a CycleGAN model and training based on the training data set to obtain a model for mapping the defect picture to a non-defect picture; Step S3: inputting a picture to be detected into the trained CycleGAN model, and using a structural similarity algorithm to compare the differences between the original picture and the repaired picture to obtain a difference binary picture; Step S4: performing connected domain noise reduction and morphological processing on the difference binary picture, and if the original picture has defects, the white area in the binary picture is the extracted defect shape; The CycleGAN model comprises a positive-to-negative generator, a negative-to-positive generator, a positive sample discriminator and a negative sample discriminator; The generator adopts a structure of down-sampling + residual block + up-sampling, and two Involution blocks are added after the input image is down-sampled to 256 channels, which are located at both ends of the residual block; The structural similarity algorithm is specifically: Three comparison functions are defined: a brightness comparison function l(x,y), a contrast comparison function c(x,y) and a structural comparison function s(x,y): where μ x , σ x , σ xy denote the mean of x, the variance of x, the covariance of x and y, respectively; Let C3=C2 / 2, and the SSIM exponential function is expressed as: In the above formula, the calculation of the mean, variance and covariance is completed in the local area of the sliding window, and the global SSIM score of the whole picture is the average value of all local area scores. 2.The defect detection method based on cycle generative adversarial network and structural similarity according to claim 1, wherein, The discriminator first adopts four convolution blocks to down-sample the input image, the kernel size of the four convolution blocks is 4x4, the channel number is increased to 512 after four times of convolution of the input image, and then the channel number is reduced to 1 through one convolution, and finally a global average pooling is used to compress a scalar of 1x1, the value range of the scalar is [0, 1], which represents the confidence of the discriminator to the input image as a real sample. 3.The defect detection method based on cycle generative adversarial network and structural similarity according to claim 1, wherein, The CycleGAN model loss function includes an adversarial loss L GAN , a cycle consistency loss L cycle , and an identity loss L identity . 4.The defect detection method based on cycle generative adversarial network and structural similarity according to claim 3, wherein, The L GAN Including L GAN_G And L GAN_D , respectively, the optimization objective of the generator G and the discriminator D, both using L2 loss metrics, as shown in equations (1), (2), L GAN_G =∑ p∈ P(D n (G p2n (p))-1) 2 +∑ n∈N (D p (G n2p (n))-1) 2 (1) L GAN_D =∑ p∈P [(p-1) 2 +(D n (G p2n (p))-0) 2 ]+∑ n∈N [(n-1) 2 +(D p (G n2p (n))-0) 2 ] (2) Where 0 and 1 represent all-0 tensor and all-1 tensor respectively. 5.The defect detection method based on cycle generative adversarial network and structural similarity according to claim 4, characterized in that, A loss function replacement strategy is used, i.e. L is implemented using SSIM loss for the first k epochs cycle and the loss function is replaced by L1 loss for the later epochs to obtain similar illumination and color details as the original image: 6.The defect detection method based on cycle generative adversarial network and structural similarity according to claim 5, wherein, The L identity Adopting L cycle Using the same metric strategy, i.e., first using SSIM loss and then L1 loss, the identity loss is described as follows: Therefore, the loss functions of the generator G and the discriminator D are respectively: L G = L GAN_G + αL cycle + βL identity (5) L D = L GAN_D (6) Where alpha and beta are hyperparameters.

Images