Defect Sample Generation System, Method and Storage Medium Based on Generative Adversarial Network

Abstract

The present invention discloses a defective sample generation system, method and storage medium based on a generative adversarial network. The system includes a data acquisition module, a clustering module, a generative adversarial module and a generation module. The clustering module is used for effectively clustering a third-party data set and screening out available third-party sample data with similar characteristics to the target scarce data set. The generative adversarial module is used for training a generative adversarial network with a training set. The generator model of the generative adversarial network includes an encoder, a fusion embedding module and a decoder. The fusion embedding module is used for performing feature fusion on the feature matrix fm of the third-party sample data with Pt to form a new feature map fn' and inputting the new feature map fn' into the decoder to obtain a target generated defective map. The present invention can effectively solve the problems of insufficient data volume and unbalanced data distribution in industrial product defect detection, thereby improving the detection efficiency and having high practicability.

Description

Technical Field

[0001] The present invention belongs to the technical field of expanding scarce data, and particularly relates to a defective sample generation system, method and storage medium based on a generative adversarial network. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technologies such as machine learning and deep learning penetrating into the production field, the industrial manufacturing industry has undergone great changes. However, since the production process is a complex process involving multiple factors, although the product yield can be improved through the automatic control and refined management of the production process, the appearance of defective products cannot be avoided. The circulation and use of defective products will reduce the service life of machines, and in severe cases, it may cause accidents of machine destruction and human casualties. Due to the small number of relevant defective products and the high collection cost, the current neural network defective product detection model relying on a large number of training samples cannot effectively solve the problem of scarce sample detection. Currently, manufacturers still use manual screening of defective products, which is inefficient and extremely costly. Therefore, effectively solving the problem of scarce industrial defective samples and accelerating the advancement of industrial products towards the high end of the global industrial chain is of great significance.

[0003] Scarce samples generally refer to the extremely lack of relevant sample quantities, which has currently become the main bottleneck for the industrial implementation of deep learning technologies. Currently, researchers mainly focus on using generative models to augment data to make up for the insufficient sample quantity to solve the problem of scarce samples. Currently, generative models are mainly divided into traditional generative models and deep generative models based on neural networks. Traditional generative models such as rotation, flipping, and cropping, in the case of scarce quantities, the number of simulated samples obtained by traditional models is limited and still belongs to scarce samples, and the cropping method cannot solve the scenario where the number of defects in the target sample is small and the size is small.

[0004] The deep generative models based on neural networks mainly include: Encoder-Decoder, Diffusion models, and generative adversarial network models. The problems among them are as follows: (1) a large amount of labeled data is required, (2) the quality of the synthesized samples is not high and the defects are blurred, (3) the specific position information of the defects on the target is not considered, resulting in the generated pictures either not containing defects or the generated defects being outside the target boundary or generating low-quality defects. To sum up, the present invention proposes a scarce defective sample generation solution that, in the context of extremely scarce defective samples, effectively fuses the feature learning of third-party data samples with the same defect characteristics and a large number of defect-free samples to generate a large number of high-quality simulated samples and evaluate them. Summary of the Invention

[0005] The purpose of the present invention is to provide a defect sample generation system, method and storage medium based on a generative adversarial network, aiming to solve the above problems. The present invention effectively solves the problem of extremely scarce sample quantity in industrial defect detection and realizes the ability to generate a large number of pictures with similar target defect shapes. The present invention clusters third-party data samples with target defect features and scarce defect samples, and through the generation of high-quality defects at specific positions, alleviates the requirements for the quantity and quality of the data set in industrial defect detection, and has good practicability.

[0006] The present invention is mainly realized through the following technical solutions:

[0007] A defect sample generation system based on a generative adversarial network, including a data acquisition module, a clustering module, a generative adversarial module, and a generation module;

[0008] The data acquisition module is used to collect industrial scarce defect pictures to obtain a target scarce data set and collect third-party industrial scarce defect pictures to obtain a third-party data set, and mark the defects;

[0009] The clustering module is used to effectively cluster the third-party data set and screen out available third-party sample data with similar characteristics to the target scarce data set;

[0010] The generative adversarial module is used to form a training set with the third-party sample data and the target defect-free sample to train the generative adversarial network; the generative adversarial network includes a generator model and a discriminator model. The generator model includes an encoder, a fusion embedding module, and a decoder arranged in sequence from front to back, and the encoder is used to process the target defect-free sample to obtain the feature map f of the target defect-free sample n ; The fusion embedding module is used to divide the feature map f of the target defect-free sample n into k patches, map each patch to the histogram space, select the patch P with the largest histogram value t to fuse the features of P t and the feature matrix f of the third-party sample data m based on a specific position to form a new feature map f n ' and input it into the decoder to obtain the target generated defect map; the discriminator model is used to measure the similarity between the generated target generated defect map and the defect-free map;

[0011] The generation module is used to input the target defect-free image and the third-party defect mask image into the trained generative adversarial network to generate defect samples.

[0012] To better implement the present invention, further, the clustering module includes an encoder-decoder network and a clustering unit. The encoder is used to map the mask graphs of the third-party dataset samples and the mask graphs of the target scarce dataset samples to the same latent space representation, and the decoder is used to reconstruct the original images from the feature vectors output by the encoder to observe the extraction performance; the clustering unit is used to cluster the third-party dataset and the target scarce dataset in the same latent space based on the Euclidean distance.

[0013] To better implement the present invention, further, an evaluation module is further included. The evaluation module is used to verify the effectiveness of the target generated defective graph, and is used to train two detection models with the same parameters respectively using the third-party sample data and the set samples of the third-party sample data and the target generated defective graph, and apply the two trained detection models to detect the samples of the target scarce dataset respectively, and compare the detection results of the two detection models to evaluate the usability of the target generated defective graph.

[0014] The present invention is mainly implemented through the following technical solutions:

[0015] A method for generating defective samples based on a generative adversarial network, which is carried out using the above-mentioned defective sample generation system, includes the following steps:

[0016] Step S100: Collect a third-party dataset and a target scarce dataset containing defects, and label the defects;

[0017] Step S200: Merge the third-party dataset and the target scarce dataset to form a training set for training the encoder-decoder network in the clustering module; use the trained encoder to extract the feature values of the third-party dataset, and then use the clustering unit based on the Euclidean distance to effectively cluster the third-party dataset samples with different formats and colors with the target scarce dataset samples with similar defect characteristics, and screen out the third-party sample data;

[0018] Step S300: Form a training set with the third-party sample data and the target defect-free samples, and train the generative adversarial network. Use the fusion embedding module to fuse the feature matrix f m of the third-party sample data with the feature matrix f n of the target defect-free samples:

[0019] LWC(P t , f m ) = M m × f m + (1 - M m ) × P t

[0020] where: P t and f mThey are represented as flawless features and flawed features respectively;

[0021] M m is the mask matrix, marking the flawless pixels as 0 and retaining the defective pixels;

[0022] Then, all patches are combined to form a new feature map f n ', f n 'Input into the decoder to obtain the target generation defect map;

[0023] Step S400: using the third-party sample data and the set of samples of the third-party sample data and the target generated defect map to train two detection models with the same parameters, and applying the two trained detection models to detect samples of the target scarce dataset, observing the detection results to evaluate the usability of the generated data;

[0024] Step S500: Input the target flawless image and the third-party flaw mask image into the trained generative adversarial network to generate a large number of generated flaw samples with similar target flaws.

[0025] In order to better implement the present invention, further, in step S200, a threshold of the Euclidean distance is set to group the feature maps of the third-party data set samples, and the feature maps greater than the threshold are grouped into the non-target defect set, otherwise they are grouped into the target defect set, and finally a set of third-party sample data is obtained.

[0026] In order to better implement the present invention, further, in step S300, the fusion embedding module first transforms the feature map f of the target flawless sample into n Divide into k patches, and the size of each patch is the same as f m The same; then the eigenvalue of each patch is calculated and mapped into the histogram space. The calculation formula is as follows:

[0027]

[0028] Where m is the number of patches,

[0029] h is the height of the patch,

[0030] w is the width of the patch,

[0031] c is the number of channels of the patch,

[0032] By comparing the histogram values of all patches, the patch P with the highest score is selected t , the P t The area containing the input image object is the largest, P t Determined fn The patch contains an object.

[0033] To better implement the present invention, further, the loss function of the generative adversarial network is as follows:

[0034]

[0035] Among them, L D is the loss function of the discriminator model:

[0036]

[0037] Among them, D() represents the discriminator model, and D(x) is used to calculate the similarity between the generated defective image and the defect-free image;

[0038] L G is the loss function of the generator model, and includes a generative adversarial loss, a generated image loss, and a defect quality loss. The specific formula is expressed as:

[0039]

[0040] Among them, D() represents the discriminator model, and D(G(x)) is used to calculate the similarity between true and false samples.

[0041] L n is the generated image loss, which is calculated as follows:

[0042]

[0043] Among them, x represents the input defect-free sample.

[0044] represents the defective image generated from the decoder Dec n generated defective image,

[0045] t represents relaxing the reconstruction constraint.

[0046] The generated image loss L n is used to ensure that the generated image is approximate to the input defect-free image;

[0047] L m is the defect quality loss, which is calculated as follows:

[0048] L m = |m d - m r |

[0049] Among them, m r represents the third-party defect mask image.

[0050] m d represents the reshaped synthetic mask image patch.

[0051] A computer-readable storage medium stores computer program instructions, and when the program instructions are executed by a processor, the above-mentioned method is implemented.

[0052] The beneficial effects of the present invention are as follows:

[0053] (1) The present invention can effectively solve the problems of insufficient data volume and unbalanced data distribution in industrial product defect detection, thereby improving the detection efficiency and having high practical value and economic benefits;

[0054] (2) The present invention uses a trained encoder to map the third-party sample features and scarce defect image features into the same latent space representation. Then, using a clustering algorithm based on Euclidean distance, the third-party samples with different formats and colors are effectively clustered with the target samples with similar defect features, and more available third-party sample data are screened out;

[0055] (3) The present invention designs a generation model, uses a fusion embedding module to divide patches and map them into histograms to determine the generated defect positions and fuse the third-party defect features and defect-free image features, and uses a discriminator to measure the similarity between the generated defect map and the defect-free map. Based on this, high-quality defects can be generated at specific positions;

[0056] (4) The present invention trains two detection models with the same parameters using the clustered third-party samples and the new samples synthesized from the third-party samples and the generated samples respectively, and applies the two trained detection models to the original scarce defect samples respectively, and observes the detection results of the two detection models to evaluate the effectiveness of the generated data, which has good practicability. Description of the Drawings

[0057] Figure 1 It is a schematic diagram of the operation process of the defect sample generation system of the present invention;

[0058] Figure 2 It is a flowchart of the operation of the clustering module;

[0059] Figure 3 It is a schematic diagram of the structure of the encoder-decoder network of the clustering module;

[0060] Figure 4 It is a schematic diagram of the structure of the generator model;

[0061] Figure 5 It is a schematic diagram of the structure of the discriminator model. Detailed Embodiments

[0062] Embodiment 1:

[0063] A defect sample generation method based on a generative adversarial network, such asFigure 1 As shown, first, the industrial scarce defect image data is collected by the data acquisition module, and a large number of third-party industrial defect sample data are collected and the defects are labeled. Secondly, after the third-party samples with different formats and colors and the target scarce samples with similar defect characteristics are subjected to feature extraction by the clustering module, an effective clustering is performed using the clustering algorithm based on the Euclidean distance, and more available third-party sample data are screened out and the clustering performance is evaluated by the clustering evaluation formula. Then, the selected third-party sample data and the defect-free images of the target samples are used by the generative adversarial module, and the fusion embedding module (Fusion patch-embedding module) is combined with the fusion generation method to generate the target defect samples. Finally, the evaluation module uses the detection model and applies it to the original defect samples, and observes the detection results to evaluate the effectiveness of the generated data.

[0064] This embodiment mainly includes the following steps:

[0065] ①Collect the third-party data set and the target scarce data set containing defects and label the defects;

[0066] ②Form a training set from the third-party data set and the target scarce data set for training the encoder-decoder network. Use the trained encoder structure to extract the feature values of each data sample, and then use the clustering algorithm based on the Euclidean distance to effectively cluster the third-party samples with different formats and colors and the target scarce samples with similar defect characteristics. Divide them into target defect classes and non-target defect classes, screen out more available third-party sample data, and then evaluate the clustering performance using the clustering evaluation formula.

[0067] ③Form a training set from the selected third-party sample data and the target defect-free samples for training the generative adversarial network. First, input the defect-free samples into the encoder-decoder network of the generator model to obtain the feature matrix f n , then introduce the feature matrix f m of the third-party samples, and fuse f m with f n . Through the fusion embedding module, first divide the defect-free sample feature map f n into k patches, then map each patch to the histogram space, and select the patch P t with the largest histogram value. Then, based on a specific position, use the layer-wise composition method to fuse P t with f m to form a new feature map f n ', and then input f n ' into Dec nAmong them, the final generated defective image is obtained. Then, the discriminator model is used to measure the similarity between the generated defective image and the defect-free image.

[0068] ④ Input the trained generative adversarial network with the target defect-free image and the third-party defect mask image to generate a large number of generated samples with similar target defects. Then, train the detection model and apply it to the original scarce defect samples, and observe the detection results to evaluate the usability of the generated data.

[0069] In this embodiment, considering that there is a large number of third-party defect image datasets, some of which do contain defect features similar to the target defects. Therefore, these feature supports are extracted from the third-party defect image dataset through the clustering module to help the generation model learn the target defect features. As Figure 2 shown, in order to better extract the required defect features, first mask and mark the defect-free pixels, and at the same time only retain the defect pixels in each original defect image to form a mask image. This is because there are significant differences in the objects between the third-party images and the target images. Extracting features from the original third-party defect images results in the extracted features mainly containing the information of the objects rather than the defect features. Then, input the third-party mask image and the target mask image into an encoder-decoder model (E m -Dec m ). As Figure 3 shown, E m extracts the features of the mask image to form a feature map in the latent space, and Dec m observes the extraction performance through reconstruction. Since there are many images from different third-party datasets and their sizes may be different. In this way, all images are resized to the same size before being fed into E m , and the output of E m also maintains the same dimension.

[0070] Preferably, as Figure 3 shown, the encoder in the encoder-decoder network consists of several convolutional layers, batch normalization layers, and LeakyRelu activation function layers, and is used to compress different input images into vectors of the same dimension. The decoder consists of several transposed convolutional layers, batch normalization layers, and LeakyRelu activation function layers, and is used to reconstruct the feature vectors obtained by the encoder into the original images to observe the extraction performance.

[0071] In this embodiment, clustering is performed by calculating the distance between the third-party and target mask image features in the latent space. Because when directly measuring the distance between the third-party image and the target mask image, since the positions of the defects in the third-party image may not correspond to the positions of the defects in the target image, the measured distance value cannot reflect the true distance. The distance between images with similar defect features at different positions is greater than the distance between images with similar positions but different defect features, resulting in defect images at different positions being clustered into the target defect set. This embodiment maps the images to the latent space, so that redundant information such as the background in the feature map will be discarded, and only the key features will be retained. This helps to find more similar features from the third-party set to supplement the target features. To measure the distance between feature maps, this embodiment uses the Euclidean distance as the metric, and the distance formula is expressed as:

[0072]

[0073] where x and y represent the target feature map and the third-party feature map respectively, and n represents the dimension of the latent space. Considering the scarcity of target defect images, we directly classify these images into the target defect set. Secondly, the Euclidean distance between a target feature map and a third-party feature map is calculated one by one until all feature maps are processed. Then, we set a threshold, and use this threshold to group the third-party feature maps. Feature maps with a distance greater than the threshold are grouped into the non-target defect set, and the rest are grouped into the target defect set. After that, we continue to process the defect target feature maps until the calculation of the last feature map is completed. After the clustering is completed, the target defect set and the non-target defect set are obtained. The difference between our metric and the traditional clustering metric is that the traditional clustering metric (such as K-means) randomly selects two samples as two clustering points in the space, and then groups the remaining instances into the corresponding two clusters. This results in different feature matrices being grouped into the target defect set, leading to the generation model synthesizing unwanted defect images. Finally, the clustering evaluation formula is used to count the ratio of the number of third-party samples similar to the target defects in the target defect set to the number of remaining non-similar target samples. The larger the ratio, the more samples with similar features to the target samples can be selected from a large number of third-party samples, indicating that the performance of the clustering module is better. The clustering evaluation formula is expressed as:

[0074]

[0075] where N target and N non-target represent the number of samples similar to the target defects and the number of non-similar target defects in the target defect set respectively.

[0076] In this embodiment, since the current generation model can only generate data with the characteristics of training samples and requires a sufficient number of training samples. If the number of training samples is scarce, it is impossible to effectively capture the characteristics of training samples, resulting in poor quality of the generated data. In addition, most generation models focus on distribution transformation and ignore the learning of defect detail information, resulting in the generated data not containing defects or the generated defect positions being in the background rather than on the target surface. Based on this, considering that the defect-free image and the defective image have similar distributions and the same objects, this embodiment uses the defect-free image as the input of the generator. To synthesize an image with the required defects, another input of the generator is the feature matrix f from a third-party image in the target defect set m . Secondly, a new Encoder-Decoder model (E n -Dec n ) is adopted as the backbone in the generator, and its input is the defect-free image. Use E n to extract object features. To retain more object information, E n is designed to have fewer layers. In this way, the size of the extracted feature map f n is larger than that of f m . Since the ratio of the size of the target object to the size of the entire image may be small (i.e., the image contains a large area of blank background). In this way, f n contains a large amount of blank background features. If f m is directly integrated into f n , the synthesized defects may be on the background rather than on the surface of the generated object. To avoid this situation, this embodiment designs a fusion embedding module. The difference between this module and the patch-embedding method in Transformer is that Transformer is used to linearly map the complete feature map into a sequence of feature blocks and gradually pass the local feature map to the Transformer branch, thereby effectively reducing the computational amount and the number of parameters of the overall network. And our fusion embedding module is to determine the patch containing the object in the entire feature map, so as to fuse the defect features with the specific patch, so that the generated defects can be on the target object rather than on the blank background. Specifically, first we divide f n into k patches, and the size of each patch is the same as that of f m . Then, we calculate the eigenvalue of each patch and map each patch into the histogram space. The calculation formula is as follows:

[0077]

[0078] Where k is the number of patches, h is the height of a patch, w is the width of a patch, and c is the number of channels of the patch. As is well known, features including those of an image (e.g., color, shape, position, and angle) are extracted. The segmented patches also contain these features. If a patch contains the features of an object, its histogram value will be relatively large; if a patch contains the features of a background, its value will be relatively small. Therefore, we integrate f m with the patch having the highest score among all patches in terms of the histogram value, and we call such a patch P t， That is to say, P t contains the largest area of the object in the input image, so P t determines the patch in f n that contains the object. After that, we integrate f m with P t using the layer-wise composition fusion generation method, and the specific composition formula is as follows:

[0079] LMC(P t , f m ) = M m × f m + (1 - M m ) × P t

[0080] Where P t and f m represent the flawless feature and the defective feature respectively, and M m is a mask matrix that marks the flawless pixels as 0 and retains the defective pixels. When we regard the patch (i.e., P t ) as a sub-feature map, layer-wise composition outputs the integrated feature map, and we call this integrated feature map P t '. In this case, P t ' includes the features of the defective image and the features of the flawless image. Since most of the area of P t is part of the object, the defect will be on the object. As for the other patches, they contain the background information and other parts of this object, so they remain unchanged and are directly combined with P t '. After that, we combine all the patches to form a new feature map, called f n ', and then input f n ' into Dec n to synthesize an image with the desired defect. The generator model is specifically as shown in Figure 4 .

[0081] The loss function of the generator model includes the generative adversarial loss, the generated image loss, and the defect quality loss function, which is specifically expressed by the formula:

[0082]

[0083] Among them, P r represents the real data distribution, P G represents the generated data distribution, D() represents the discriminator model, x represents the input defect-free sample, represents the defect map generated from Dec n is the generated image loss, which is used to ensure that the generated image is similar to the input defect-free image. L1 loss is used and the reconstruction constraint is relaxed by t, so that the generator will not suppress the generation of defects. The defect quality loss function is expressed as follows:

[0084] L m = |m d - m r |

[0085] Among them, m r represents the third-party defect mask image, and m d represents the reshaped synthetic mask image patch.

[0086] In addition, as Figure 5 shown, the discriminator model is used to measure the similarity between the generated defect map and the defect-free map. Therefore, the loss function of the discriminator model is as follows:

[0087]

[0088] Among them, x represents the defect-free sample or the generated defect map. Therefore, the total loss function of the generation model is as follows:

[0089]

[0090] Among them, L D represents the loss of the discriminator D, and L G represents the loss of the generator G. During the training process, the parameters of the generation model are updated by maximizing L D and minimizing L G to achieve the purpose of Nash equilibrium.

[0091] ​Since the industrial generation defect sample evaluation method often synthesizes generated data and original training samples into new samples and uses them to train detectors, if the recognition accuracy of this detector is higher than that of the detector trained only with original training samples, it can prove that the generated data is effective. Disadvantages of existing methods: 1) The types of industrial defects may be single and cannot be classified, resulting in classification failure; 2) The sample quantity is scarce, unable to fit the data distribution, and unable to divide the training set and the test set. Therefore, in this embodiment, to verify the effectiveness of the generated data of scarce defect samples, two detection models (such as YoloX) with the same parameters are trained with the third-party samples after clustering and the new samples synthesized by the third-party samples and the generated samples respectively, and the two trained detection models are respectively applied to the original scarce defect samples to observe the detection results to evaluate the effectiveness of the generated data.

[0092] Preferably, the present invention uses the O-ring dataset and the MVTEC dataset for experimental verification. The O-ring dataset contains 200 defect-free images and 12 defect images with impurity defects. Since the object of the O-ring only accounts for 40% of the entire image, it is challenging to generate the required defects on the object. The MVTEC dataset is a publicly available defect detection dataset, including 15 categories of images, each category consisting of a large number of defect-free images and a small number of defect images. This dataset contains a total of 70 different types of defects, such as scratches, dents, contamination, etc. In this embodiment, the images of the Tile category are selected for the experiment, and 150 defect-free images and 10 defect images with scratch defects are used. In addition, since our defect images are extremely scarce and cannot be used to train the generation model, we collect a large number of third-party datasets to provide defect information for the generator. These third-party datasets come from the publicly available DAGM, Magnetic Tile, and CrackForestDataset. Among them, the types of defects and the number of defect images are shown in Table 1 below. To ensure that the performance of the experiment is not affected by the image size, we uniformly adjust the size of all images in the experiment to 3×1024×1024.

[0093] Table 1 Statistical situation of publicly available datasets

[0094] Dataset Defect category Number of defect images DAGM Category2 100 MagneticTile Porosity 127 CrackForestDataset Crack 162

[0095] The present invention first uses a clustering module to process the dataset. For the O-ring dataset, the third-party dataset is classified according to 12 defect images of O-rings, and they are divided into a target defect set and a non-target defect set. Through this method, we can screen out third-party features similar to the target features. Since the porosity defects from Magnetic Tile are similar to the impurity defects of O-rings, we hope to select more images with porosity defects to supplement our target features. Therefore, we calculate the ratio of the number of images with porosity defects in the target defect set to the number of the remaining images. The clustering results are shown in Table 2 below. For the MVTEC dataset, according to 10 defect images, the third-party dataset is divided so that crack images similar to the target features can be selected, and the ratio of the number of crack images in the target defect set to the number of the remaining images is calculated. The clustering results are shown in Table 3 below.

[0096] Table 2 Clustering Results of O-ring Dataset

[0097]

[0098] Table 3 Clustering Results of MVTEC Dataset

[0099]

[0100] It is noted from Table 3 that our clustering results depend on the thresholds we set. We conducted a large number of experiments at various thresholds and quantitatively evaluated the performance through ratio. The higher the ratio, the more effectively our clustering method can select the third-party features we need. At the same time, we compared with K-means, and the results show that K-means cannot meet our expectations because it randomly selects two samples as the clustering points instead of clustering based on the target features, resulting in the dominant number of defects dissimilar to the target features in the results.

[0101] The present invention uses the third-party features in the target defect set after clustering and fuses them with the target defect-free images to generate the required defect images, and compares them with SDGAN, StyleAlign, StyleGAN2, StyleGAN, and CutPaste. However, these models such as SDGAN, StyleAlign, StyleGAN2, and StyleGAN3 cannot generate defects because these models focus on distribution matching rather than detail generation. CutPaste first randomly takes a rectangular region patch of any size, then selectively rotates or changes the pixel values of the patch, and then copies the changed patch to any place of the original sample to obtain a supposed defect image, but the supposed defect image is not similar to the real defect image.

[0102] To evaluate the usability of the synthetic images, we use Yolox as the evaluation model and utilize Precision, Recall, and as the metrics for quantitatively measuring the evaluation results. Precision is the proportion of samples predicted as positive examples (e.g., the true class is positive and the model predicts positive) that are actually positive. Recall is the proportion of samples that are actually positive examples and are predicted as positive examples by the model. Although CutPaste synthesizes unwanted defects, we still enhance these synthetic images to compare the performance of the defects. Since other generative models (i.e., SDGAN, StyleAlign, StyleGAN2, and StyleGAN3) cannot synthesize defective images, we do not show their detection performance. Rotation belongs to traditional enhancement methods, but its data augmentation ability is limited, especially when our target dataset is scarce. Excessive rotation is equivalent to copying and pasting the defects rather than new defects not seen in the training set. Therefore, for a fair comparison, we synthesize 45 images by the rotation method and use Cut Paste and this method to synthesize 450 images. Then, we form a new training set with the synthetic images of each model and use the original scarce samples as the test set. The detection results of the O-ring dataset and the MVTEC dataset are shown in Tables 4 and 5.

[0103] Table 4 Detection Results of O-ring Dataset

[0104] Method Precision Recall F1-Score Rotation 44.3％ 65.2％ 52.8％ CutPaste 1.8％ 4.2％ 2.5％ Example 1 75.6％ 74.8％ 75.2％

[0105] Table 5 Detection Results of MVTEC Dataset

[0106] Method Precision Recall F1-Score Rotation 35.8％ 37.4％ 36.6％ CutPaste 0％ 0％ 0％ Example 1 40.2％ 45.7％ 42.8％

[0107] According to the experimental results in Tables 4 and 5, we can see that the present invention is significantly superior to other methods, which illustrates the superiority of the present invention in scarce data augmentation.

[0108] The above is only a preferred embodiment of the present invention and does not impose any formal limitations on the present invention. Any simple modification or equivalent change made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A defective sample generation system based on a generative adversarial network, characterized in that It includes a data acquisition module, a clustering module, a generative adversarial module, and a generation module; The data acquisition module is used to collect industrial scarce defect pictures to obtain a target scarce data set and collect third-party industrial scarce defect pictures to obtain a third-party data set, and label the defects; The clustering module is used to effectively cluster the third-party data set and screen out available third-party sample data with similar characteristics to the target scarce data set; The generative adversarial module is used to form a training set with third-party sample data and target flawless samples to train a generative adversarial network; the generative adversarial network includes a generator model and a discriminator model, and the generator model includes an encoder, a fusion embedding module, and a decoder arranged in sequence from front to back, and the encoder is used to process the target flawless sample to obtain the feature map f of the target flawless sample n ; the fusion embedding module is used to divide the feature map f of the target flawless sample n into k patches, map each patch to the histogram space, and select the patch P with the largest histogram value t , and based on a specific position, fuse P t with the feature matrix f of the third-party sample data m to form a new feature map f n ' and input it into the decoder to obtain the target generated defective map; The discriminator model is used to measure the similarity between the generated target generated defect pictures and defect-free pictures; The generation module is used to input the target defect-free image and the third-party defect mask image into the trained generative adversarial network to generate defect samples; The clustering module includes an encoder-decoder network and a clustering unit. The encoder is used to map the mask pictures of the third-party data set samples and the mask pictures of the target scarce data set samples to the same latent space representation, and the decoder is used to reconstruct the feature vectors output by the encoder into the original image to observe the extraction performance; The clustering unit is used to cluster the third-party data set and the target scarce data set in the same latent space based on the Euclidean distance.

2. The defective sample generation system based on a generative adversarial network according to claim 1, wherein It also includes an evaluation module. The evaluation module is used to verify the effectiveness of the target generated defect pictures. It is used to train two detection models with the same parameters using the third-party sample data and the set samples of the third-party sample data and the target generated defect pictures respectively, and apply the two trained detection models to detect the samples of the target scarce data set, and compare the detection results of the two detection models to evaluate the usability of the target generated defect pictures.

3. A method for generating defective samples based on a generative adversarial network, which is carried out by using the defective sample generation system described in any one of claims 1-2, characterized in that, It includes the following steps: Step S100: Collect a third-party data set and a target scarce data set containing defects, and label the defects; Step S200: Merge the third-party data set and the target scarce data set to form a training set for training the encoder-decoder network in the clustering module; Extract the feature values of the third-party dataset using the trained encoder, and then use the clustering unit based on the Euclidean distance to effectively cluster the third-party dataset samples with different format colors and the target scarce dataset samples with similar defect features, and filter out the third-party sample data; Step S300: Form a training set with the third-party sample data and the target flawless samples, and train to generate an adversarial network. Use the fusion embedding module to generate the feature matrix f of the third-party sample data m and the feature matrix f of the target flawless samples n Fusion: LWC(P t , f m ) = M m × f m + (1 - M m ) × P t Where: P t and f m represent a flawless feature and a defective feature, respectively; M m is a mask matrix that marks the flawless pixels as 0 and retains the defective pixels; Then, all the patches are combined to form a new feature map f n ’, and f n ’ is input into the decoder to obtain the target generated defect map; Step S400: Train two detection models with the same parameters using the third-party sample data and the set samples of the third-party sample data and the target generated defect pictures respectively, and apply the two trained detection models to detect the samples of the target scarce data set, and observe the detection results to evaluate the usability of the generated data; Step S500: Input the target defect-free image and the third-party defect mask image into the trained generative adversarial network to generate a large number of generated defect samples with similar target defects.

4. A method for generating defective samples based on a generative adversarial network according to claim 3, characterized in that In the step S200, set the threshold of the Euclidean distance to group the feature maps of the third-party data set samples, and group the feature maps greater than the threshold into the non-target defect set, otherwise they are grouped into the target defect set, and finally obtain the set of third-party sample data.

5. A method for generating defective samples based on a generative adversarial network according to claim 3, characterized in that, In the step S300, the fusion embedding module first extracts the feature map f of the target perfect sample n and divides it into k patches, where the size of each patch is the same as that of f m ; then, the eigenvalue of each patch is calculated and mapped into the histogram space, and the calculation formula is as follows: Where m is the number of patches, h is the height of the patch, w is the width of the patch, c is the number of channels of the patch, By comparing the histogram values of all patches, select the patch P with the highest score t , where the P t contains the largest area of the input image object, and P t determines the f n patches that contain the object.

6. The method for generating defective samples based on a generative adversarial network according to claim 3, wherein The loss function of the generative adversarial network is as follows: Among them, L D is the loss function of the discriminator model: Among them, P r represents the real data distribution, and P G represents the generated data distribution. D() represents the discriminator model, and D(x) is used to calculate the similarity between the generated defective images and the defect-free images; L G is the loss function of the generator model and includes a generative adversarial loss, a generated image loss, and a defect quality loss, and the specific formula is expressed as: Where D() represents the discriminator model, and D(G(x)) is used to calculate the similarity between real and fake samples, L n To generate the image loss, it is calculated as follows: Where x represents the input defect-free sample, Represents the defective image generated from decoder Dec n ​ t represents relaxing the reconstruction constraint, Generate image loss L n To ensure that the generated image approximates the input flawless image; L m is the defective quality loss and is calculated as follows: L m = |m d - m r | where m r represents a third-party defect mask image m d Represents the reshaped synthetic mask image patch.

7. A computer-readable storage medium storing computer program instructions, characterized in that, When the program instructions are executed by the processor, the method described in any one of claims 4-6 is implemented.

Images