Defect image sample generation method, system, medium and device based on generative adversarial network

Abstract

The present invention discloses a defect image sample generation method based on a generative adversarial network, the method comprising obtaining real defect samples and non-defect samples; constructing and training a first generative adversarial network model; inputting the real defect samples and the non-defect samples into the first generative adversarial network model to generate defect block images; generating coarse synthetic defect samples according to the defect block images and the non-defect samples; constructing and training a second generative adversarial network model; inputting the coarse synthetic defect samples and the real defect samples into the second generative adversarial network model to generate defect image samples. Aiming at the shortcomings of the generated samples for defect detection tasks in industrial scenarios, such as lack of meaningful new distribution and features, deviation from the true distribution, poor image quality, and poor support for supervision information, the present invention adopts a multi-stage generative adversarial network structure to generate a controllable number of defect image samples with defect type and coordinate supervision information, which is applicable to any defect detection model and greatly improves the applicability of the model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, system, medium and device for generating defect image samples based on a generative adversarial network. Background Art

[0002] Product surface defect detection has always been a difficult problem faced by the manufacturing industry. The defect detection link directly affects the quality level and effective production capacity of the product, and is of great significance to improving the efficiency and reducing costs of the factory. The training sample generation method for defect detection can be used to solve the problems of defect detection tasks in the actual operation processes such as production and manufacturing of industrial equipment manufacturing, finished product quality inspection, etc., such as low defect frequency, large intra-class differences, inability to obtain enough defect images covering all forms as effective training samples, and a large amount of manpower and material resources consumed in defect image acquisition and supervision information labeling. Therefore, the training sample problem has received widespread attention, and a large number of sample generation methods have been proposed and applied to generate sufficient training samples as support to guide model training to meet the required detection standards. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a method, system, medium and device for generating defective image samples based on a generative adversarial network, so as to solve at least one shortcoming in the prior art.

[0004] To achieve the above objectives and other related objectives, the present invention provides a defect image sample generation method based on a generative adversarial network, comprising:

[0005] Acquire real defect samples and non-defect samples, wherein the real defect samples are labeled with defect information;

[0006] Build and train the first generative adversarial network model;

[0007] Inputting the real defect sample and the non-defect sample into the first generative adversarial network model to generate a defect block image, wherein the defect block image includes defect type information;

[0008] Generate a coarse synthetic defect sample according to the defect block image and the non-defective sample, wherein the coarse synthetic defect sample carries a defect information label;

[0009] Build and train a second generative adversarial network model;

[0010] Inputting the rough synthetic defect sample and the real defect sample into the second generative adversarial network model to generate defect image samples, wherein the defect image samples have defect information labels,

[0011] The defect information label includes defect type information and defect coordinate information.

[0012] Optionally, the defect image sample generation method based on a generative adversarial network also includes cropping the real defect sample, including: obtaining defect coordinate information of the real defect sample; obtaining an input size to be cropped; using the defect block coordinate information of the real defect sample and the input size to be cropped, cropping the real defect sample to obtain a real defect block of the real defect sample, wherein the real defect block of the real defect sample contains defect type information.

[0013] Optionally, the training of the first generative adversarial network model includes training the first generative adversarial network model using the real defect block.

[0014] Optionally, the first generative adversarial network model includes a generator and a discriminator, and the inputting of the real defect samples and the non-defective samples into the first generative adversarial network model to generate the defect block image includes: obtaining a category label and Gaussian random noise; and inputting the category label and the Gaussian random noise into the generator of the first generative adversarial network model to generate the defect block image.

[0015] Optionally, the generation of a coarse synthetic defect sample using the defect block image and the defect-free sample includes: obtaining the coordinates and scaling factor of a fused ROI area; scaling the defect block image according to the scaling factor; and superimposing the scaled defect block image on the defect-free sample according to the fused ROI area to generate the coarse synthetic defect sample.

[0016] Optionally, the first generative adversarial network model is a CGAN model.

[0017] Optionally, the second generative adversarial network model is a CGAN model.

[0018] To achieve the above objectives and other related objectives, the present invention provides a defect image sample generation system based on a generative adversarial network, comprising:

[0019] A sample acquisition module, used to acquire real defect samples and non-defect samples, wherein the real defect samples are labeled with defect information;

[0020] A first generative adversarial network model determination module, used to construct and train a first generative adversarial network model;

[0021] A defective block image determination module, used to input the real defect sample and the non-defective sample into the first generative adversarial network model to generate a defective block image, wherein the defective block image includes defect type information;

[0022] A coarse synthetic defect sample determination module, used to generate a coarse synthetic defect sample according to the defect block image and the non-defective sample, wherein the coarse synthetic defect sample carries a defect information label;

[0023] A second generative adversarial network model determination module, used to construct and train a second generative adversarial network model;

[0024] The defect image sample determination module is used to input the rough synthetic defect sample and the real defect sample into the second generative adversarial network model to generate a defect image sample, wherein the defect image sample has a defect information label, wherein the defect information label includes defect type information and defect coordinate information.

[0025] To achieve the above-mentioned object and other related objects, the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the defect image sample generating method when executed by a processor.

[0026] To achieve the above-mentioned purpose and other related purposes, the present invention provides a computer device, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the defect image sample generating method when executing the computer program.

[0027] As described above, the defect image sample generation method, system, computer-readable storage medium and computer device based on a generative adversarial network of the present invention have the following beneficial effects:

[0028] In view of the shortcomings of the generated samples in the defect detection task, such as lack of meaningful new distribution and features, deviation from the true distribution, poor image quality, and poor support for supervision information, the present invention adopts a multi-stage generative adversarial network structure to provide a method for generating defect detection training samples, which supports multiple defect types and can generate a controllable number of defect image samples with defect type and coordinate supervision information. The present invention can ensure a large sample coverage rate, greatly improve the model detection performance, and is applicable to any defect detection model with strong applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flowchart of a defect image sample generation method based on a generative adversarial network according to an embodiment of the present invention;

[0030] Figure 2 is a system module diagram for generating defect image samples according to an embodiment of the present invention;

[0031] Figure 3is a schematic diagram of a real defect sample according to an embodiment of the present invention;

[0032] Figure 4A is a schematic diagram of a first generative adversarial network model according to an embodiment of the present invention;

[0033] Figure 4B is a schematic diagram of a second generative adversarial network model according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram of a rough synthetic defect sample according to an embodiment of the present invention;

[0035] Figure 6 It is a schematic diagram of generating defect sample images and labels according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0037] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0038] The purpose of this application is to generate a controllable number of defect image samples with defect type and coordinate supervision information, in order to solve the problems of the generated samples in the existing methods that do not have meaningful new distribution and features, deviate from the true distribution, have poor image quality, and poor support for supervision information, so as to ensure the accuracy of subsequent defect detection. This application proposes a sample generation method based on a generative adversarial network, which can be applied to any defect detection model, has good application prospects and very important practical significance.

[0039] The overall concept of this application is to generate defect image samples in a way that integrates deep learning in all stages. Specifically, in the generation stage, a small number of real defect samples and non-defect samples are first preprocessed, input into the generative adversarial network training with Resnet as the backbone, and embedded into the class conditional training model; after training, defect block images with a controllable number of types are generated, coarse synthetic defect samples are obtained, and supervisory labels are automatically annotated; in the subsequent fusion stage, the coarse synthetic defect samples and real defect samples are input into the generative adversarial network and encoder training with Resnet as the backbone; the coarse synthetic defect samples are embedded in the latent variable space through the encoder, and the foreground and background are reconstructed through generator fusion, and finally a sufficient number of effective generated defect samples and supervisory labels are obtained.

[0040] Figure 1 It is a flowchart of a defect image sample generation method based on a generative adversarial network according to an embodiment of the present application.

[0041] In step 101 , real defect samples and non-defect samples may be collected. Step 101 may be completed by the sample acquisition module 201 .

[0042] In some embodiments, real samples can be obtained by batch acquiring public data images of industrial defect detection. Figure 3 This is a schematic diagram of a real defect sample according to an embodiment of the present application. In the present application, the selected data set is the public data set NEU-DET of surface defects of hot-rolled steel strips, with a total of 1,800 defect sample images, the image pixel size is 200x200, and the image format is .png format. Figure 3 Samples of 6 defect types are shown, with 300 defect samples for each defect type.

[0043] The real samples can be divided into real defect samples and defect-free samples (Defect Free) according to the presence or absence of defects. The real defect samples can be divided into cracks, inclusions, patches, pitted surfaces, rolled-in scales and scratches according to the types of defects. It should be noted that the defect types shown above are only exemplary, and in other embodiments, the defect types may also include other types.

[0044] In an embodiment of the present application, real defect samples and non-defective samples are stored in a storage device in the form of .png image files and are distinguished using a naming format of defect type + number to facilitate subsequent data reading.

[0045] Different types of defect samples and non-defect samples are classified into the folders of Crazing, Inclusion, Patches, Pitted Surface, Rolled-in Scale, Scratches and Defect Free, and the categories are marked. The images in the same directory have the same name but different numbers. For example, crack-type defect samples can be stored in a folder named Crazing, and the images in this folder can be named Crazing1, Crazing2, Crazing3, Crazing4, etc. For another example, inclusion-type defect samples can be stored in a folder named Inclusion, and the images in this folder can be named Inclusion1, Inclusion2, Inclusion3, Inclusion4, etc. The naming method of the images in the folders of Patches, Pitted Surface, Rolled-in Scale, Scratches and Defect Free is the same as that of the images in the folders of Crazing and Inclusion, which will not be repeated here.

[0046] It should be noted that the sample image format, the naming format of the folder, and the naming method of the images in the folder shown above are only exemplary. In some embodiments, the sample image can also be stored in other picture file forms, such as .jpg format, .jpeg format, .gif format, .bmp format, etc. In some embodiments, the folder name can be in Chinese, for example, the folder name can be crack, inclusion, spot, pockmark, mill scale, scratch and defect-free, respectively. In some embodiments, the folder name can also use other names, as long as it can distinguish the presence or absence of defects and the type of defects. In some embodiments, the images in each folder can also be named in other ways. For example, the Chinese name of the defect type of the image + digital number, the Chinese name of the defect type of the image + letter number, the English name of the defect type of the image + letter number, etc. Among them, when using the English name of the defect type of the image + the digital number (or letter number), there may or may not be a space between the type name and the number, and the number of spaces may be set arbitrarily (for example, 1, 2, 3); the English name may use all uppercase letters or partially use uppercase letters (for example, the first letter is capitalized) or all lowercase letters; any connector (for example, "-", "~", etc.) may be added between the type name and the number. In some embodiments, the images in each folder can also be named in the form of number + defect type. The specific naming method is similar to the naming method of defect type + number, which will not be repeated here.

[0047] In step 102 , a first generative adversarial network model may be constructed and trained. Step 102 may be completed by the first generative adversarial network model determination module 202 .

[0048] Industrial defects usually come in multiple categories, so the generative adversarial network in the generation phase uses the CGAN (Conditional Generative Adversarial Network) model to embed conditional information. The generative adversarial network model in the generation phase can also be called the first generative adversarial network model. Figure 4A It is a schematic diagram of the first generative adversarial network model shown in the embodiment of the present application. The first generative adversarial network model 400 may include a generator 401 and a discriminator 402. In some embodiments, the generator 401 may use Resnet as the backbone network, and may increase or decrease the number of residual blocks to improve the network feature reuse capability depending on the texture complexity of the actual defective image. In some embodiments, the discriminator 402 may introduce a Minibatch Discrimination structure to calculate and maintain the differences between the feature maps of a certain layer in the discriminator for all images in the minibatch, so as to alleviate the mode collapse phenomenon in CGAN and enhance the diversity of generated defective blocks. In some embodiments, the discriminator 402 may use a Leaky RELU function to enhance the stability of the network.

[0049] In some embodiments, before training the first generative adversarial network model, it is also necessary to perform cropping preprocessing on the real defect samples, including: obtaining the defect coordinate information of the real defect samples. If the real defect samples have defect information labels, the labels can be used directly as references to read the defect types and coordinates of the real samples in the labels. The NEU-DET dataset contains XML tags with defect types and coordinate information, and the coordinate format is: the leftmost x of the prediction box min , the top of the prediction box y max , the rightmost x of the prediction box max , the bottom of the prediction box y min , thereby obtaining the coordinates of the area to be cropped corresponding to the defective block image.

[0050] Since the cropped real defect blocks will be input into the generative adversarial network in the subsequent generation stage, the size of the input defect blocks needs to be set. Therefore, the cropping preprocessing of the real defect samples also includes obtaining the input size to be cropped.

[0051] The cropping preprocessing of the real defect sample also includes cropping the real defect sample using the defect block coordinate information of the real defect sample and the input size to be cropped to obtain the real defect block of the real defect sample.

[0052] Specifically, for defect sample images whose cropping area is smaller than the input size, a square frame of the input size is cropped with the defect center coordinates as the center to obtain a real defect block of the real defect sample. In the embodiment of the present application, the input size to be cropped is set to 64x64.

[0053] For defect sample images whose cropping area is larger than the input size, they can be directly cropped according to the cropping area to obtain an image block that is exactly the frame column defect block, that is, the real defect block of the real defect sample.

[0054] The real defective blocks can be used as the real input of the generation phase discriminator 402 to train the generation phase generator 401.

[0055] The cropped real defect block is named according to its corresponding defect type and number, such as "crazing-1.png", to retain its defect type information. Preprocessing operations can be performed according to the actual engineering situation and training sample characteristics, such as unifying the image size and contrast. In the embodiment of the present application, the input image size is 64x64, and the output image size is also 64x64.

[0056] During training, the cropped real defect block is used as the image input of the network. At the same time, the defect type name in the image name is read, such as "crazing" in "crazing-1.png". The one-hot encoding corresponding to the defect type is used as the conditional input of the network and input into the network for training.

[0057] Industrial defects usually have multiple types, so the generative adversarial network in the generation stage uses CGAN to introduce conditional information and uses Resnet as the backbone network. Depending on the texture complexity of the actual defect image, the number of residual blocks can be increased or decreased to improve the network feature reuse capability. Therefore, the objective function of the generative adversarial network model in the generation stage is expressed as:

[0058]

[0059] In the above formula, G and D represent the output of the generator and the discriminator respectively, P data and P Z They represent the distribution of real data and Gaussian random noise respectively, x represents the real image data, z represents the randomly sampled Gaussian noise, and c represents the conditional information of the defect type.

[0060] In step 103 , the real defect samples and the non-defect samples may be input into the first generative adversarial network model to generate defective block images. Step 103 may be completed by the defective block image determination module 203 .

[0061] After the training of the generation phase is completed, the type label and Gaussian random noise are input to the generation phase generator 401, and the generated defect block image is saved and named in the form of defect type and number to retain the defect type information.

[0062] In step 104 , a coarse synthetic defect sample is generated according to the defect block image and the non-defective sample. Step 104 may be completed by the coarse synthetic defect sample determination module 204 .

[0063] Generating a rough synthetic defect sample according to the defect block image and the non-defective sample may include:

[0064] Get the coordinates and zoom factor of the fused ROI area;

[0065] Scaling the defective block image according to the scaling factor;

[0066] According to the fused ROI area, the scaled defect block image is superimposed on the non-defective sample to generate the coarse synthetic defect sample.

[0067] In some embodiments, the generated defective block image and non-defective sample image, zoom multiple and fusion ROI area can be selected several times in a random and automatic manner, and the generated defective block image can be superimposed on the non-defective image to obtain a coarse synthetic defect sample. Figure 5 2 is a schematic diagram of a coarsely synthesized defect sample according to an embodiment of the present invention. In the implementation of the present application, the size of the defect-free sample image in the real data is 200x200, so the size of the coarsely synthesized defect sample image is also 200x200.

[0068] The rough synthetic defect sample retains the defect type and defect coordinate information. The defect type information and defect coordinate information are stored in another file directory in the form of tags. Depending on the different ways in which the detection model reads tags, the tag format is also different, such as .xml or .txt format.

[0069] In some implementations, the subsequent detection model is YOLO V5, and the coordinate format of the label of the coarse synthetic defect sample is: center horizontal coordinate relative value xc, center vertical coordinate relative value yc, prediction box relative width w_box, prediction box relative height h_box.

[0070] Assume that the width and height of the defect-free image are w_free and h_free respectively, the width and height of the defective block image after scaling are w_def and h_def respectively, and the horizontal and vertical coordinates of the center of the pre-selected fusion frame are x_fus and y_fus respectively. The coordinates of the coarse synthesis defect sample are automatically marked as follows during coarse synthesis:

[0071]

[0072]

[0073]

[0074]

[0075] In step 105 , a second generative adversarial network model is constructed and trained. Step 105 may be completed by the second generative adversarial network model training module 205 .

[0076] The second generative adversarial network model is also called the fusion stage generative adversarial network model. Figure 4B 4 is a schematic diagram of a second generative adversarial network model according to an embodiment of the present application. The second generative adversarial network model 450 includes a generator 451, a discriminator 452 and an encoder 453. The generator 451 and the discriminator 452 constitute the structure of the second generative adversarial network. The encoder 453 can embed the image code into the latent variable space and input it to the generator 451 in the form of latent variables.

[0077] The second generative adversarial network model can be divided into two stages. In the first stage, the generator 451 and the discriminator 452 can be trained separately. In the second stage, the network weight parameters of the generator 451 and the discriminator 452 are frozen, the encoder 453 is trained independently, and the parameters of the encoder 453 are updated.

[0078] In the original adversarial network, loss adv Based on this, the reconstruction loss loss is calculated using the eigenvalues ​​of the intermediate layer of the discriminator. rec , and introduce SSIM loss function loss SSIM The similarity between the reconstructed image and the original image is comprehensively measured in terms of brightness, contrast, and structure, further improving the authenticity of the generated samples.

[0079] loss fus =loss adv +loss rec +loss SSIM

[0080] in,

[0081] loss rec =D KL (q(z|x)||p(z))+E q(z|x) [log p (x|z)]

[0082]

[0083] In the above formula, μ x and μ yRepresents the mean of the input image x and the reconstructed output image y, σ x and σ y denotes the variance of the input image x and the reconstructed output image y, σ xy represents the covariance of the input image x and the reconstructed output image y, c 1 and c 2 Can be set to a constant value related to the dynamic range of the image.

[0084] In the fusion stage, the coarsely synthesized defect sample image is reconstructed to make the pixel distribution of the superimposed generated defect block close to the background area, so as to obtain a more realistic defect sample. The generator 451, discriminator 452 and encoder 453 in this stage all use the network backbone of Resnet, introduce the image layer and feature layer reconstruction loss of the reconstructed defect sample image and the input real defect image, and under the condition of retaining the defect features of the input image, the fusion edge of the reconstructed defect is close to the background area, so as to generate a more realistic defect sample image.

[0085] In step 106 , the coarse synthetic defect sample and the real defect sample are input into the second generative adversarial network model to generate a defect image sample. Step 106 may be completed by the defect image sample determination module 206 .

[0086] After the second generative adversarial network model is trained, the input rough synthetic defect sample can generate defect samples and labels through the encoder 453 and the generator 452. In the fusion stage, the input image is transformed to 192x192 through Torchvision's transforms, and the output image size is also 192x192.

[0087] The label contains the defect type information and defect coordinate information of the generated defect sample, thereby solving the problem of insufficient training samples and supervised information labeling. In this fusion stage, the uncropped real defect sample is input, that is, the pixel size of the source data is maintained. In some embodiments, according to the actual application situation, enhancement operations such as contrast can also be performed to improve the image quality of the generated sample. Figure 6 It is a schematic diagram of defect sample images and labels generated according to an embodiment of the present application.

[0088] Figure 2 FIG. 1 is a system module diagram for generating defective image samples according to an embodiment of the present application. Figure 2 As shown, the defect image sample generation system 200 includes a sample acquisition module 201, a first generative adversarial network model determination module 202, a defect block image determination module 203, a coarse synthesized defect sample determination module 204, a second generative adversarial network model determination module 205 and a defect image sample determination module 206.

[0089] The sample acquisition module 201 can be used to collect real defect samples and non-defect samples.

[0090] The first generative adversarial network model determination module 202 can be used to construct and train the first generative adversarial network model 400. The first generative adversarial network model 400 can be a CGAN model.

[0091] The defective block image determination module 203 can be used to input the real defect sample and the non-defective sample into the first generative adversarial network model 400 to generate a defective block image. The defective block image determination module 203 can obtain the defect coordinate information of the real defect sample. The defective block image determination module 203 can also obtain the input size to be cropped. The defective block image determination module 203 can also use the defective block coordinate information of the real defect sample and the input size to be cropped to crop the real defect sample to obtain the real defect block of the real defect sample.

[0092] Specifically, for a defect sample image whose cropping area is smaller than the input size, the defect block image determination module 203 can crop a square frame of the input size with the defect center coordinate as the center to obtain a real defect block of the real defect sample. For a defect sample image whose cropping area is larger than the input size, the defect block image determination module 203 can directly crop according to the cropping area to obtain an image block that just frames the defect block, that is, the real defect block of the real defect sample.

[0093] The coarse synthetic defect sample determination module 204 can be used to generate a coarse synthetic defect sample according to the defect block image and the non-defective sample. Specifically, the coarse synthetic defect sample determination module 204 can obtain the fused ROI area coordinates and the scaling factor. The coarse synthetic defect sample determination module 204 can scale the defect block image according to the scaling factor. The coarse synthetic defect sample determination module 204 can superimpose the scaled defect block image on the non-defective sample according to the fused ROI area to generate the coarse synthetic defect sample.

[0094] In some embodiments, the coarse synthetic defect sample determination module 204 can randomly and automatically select the generated defect block image and defect-free sample image, zoom multiple and fusion ROI area several times, and superimpose the generated defect block image on the defect-free image to obtain a coarse synthetic defect sample.

[0095] The second generative adversarial network model determination module 205 can be used to construct and train the second generative adversarial network model 450. The second generative adversarial network model 450 can be a CGAN model. The second generative adversarial network model 450 can first train the generator 451 and the discriminator 452 separately, then freeze the network weight parameters of the generator 451 and the discriminator 452, independently train the encoder 453, and update the parameters of the encoder 453.

[0096] The defect image sample determination module 206 can be used to input the rough synthetic defect samples and the real defect samples into the second generative adversarial network model 450 to generate defect image samples.

[0097] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0098] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] An embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the defect sample generating method when executing the computer program.

[0100] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the defect sample generating method when executed by a processor.

[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0102] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory ((RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0106] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A defect image sample generation method based on a generative adversarial network, characterized in that: include: Acquire real defect samples and non-defect samples, wherein the real defect samples are labeled with defect information; Build and train the first generative adversarial network model; Preprocessing the real defect samples and non-defect samples, then inputting the preprocessed real defect samples and non-defect samples into the first generative adversarial network model with Resnet as the backbone, and embedding the class conditional training model to generate defect block images, wherein the defect block images contain defect type information; The preprocessing of the real defect samples, i.e., cutting the real defect samples, includes: Obtaining defect coordinate information of the real defect sample; Get the input size to be cropped; Using the defect block coordinate information of the real defect sample and the input size to be cropped, the real defect sample is cropped to obtain a real defect block of the real defect sample, The real defect block of the real defect sample contains defect type information; the real defect block can be used as the real input of the discriminator in the generation phase to train the generator in the generation phase; Generate a coarse synthetic defect sample according to the defect block image and the non-defective sample, wherein the coarse synthetic defect sample carries a defect information label; Build and train a second generative adversarial network model; Inputting the rough synthetic defect sample and the real defect sample into the second generative adversarial network model to generate defect image samples, wherein the defect image samples have defect information labels, The defect information label includes defect type information and defect coordinate information.

2. The defect image sample generation method based on generative adversarial network according to claim 1, characterized in that: The training of the first generative adversarial network model comprises: The first generative adversarial network model is trained using the real defect blocks.

3. The defect image sample generation method based on generative adversarial network according to claim 1, characterized in that: The first generative adversarial network model includes a generator and a discriminator, and the inputting the real defect sample and the non-defect sample into the first generative adversarial network model to generate the defect block image includes: Get category labels and Gaussian random noise; The category label and the Gaussian random noise are input into the generator of the first generative adversarial network model to generate the defective block image.

4. The defect image sample generation method based on generative adversarial network according to claim 1, characterized in that: The step of generating a rough synthetic defect sample by using the defect block image and the non-defective sample comprises: Get the coordinates and zoom factor of the fused ROI area; Scaling the defective block image according to the scaling factor; According to the fused ROI area, the scaled defect block image is superimposed on the non-defective sample to generate the coarse synthetic defect sample.

5. The defect image sample generation method based on generative adversarial network according to claim 1, characterized in that: The first generative adversarial network model is a CGAN model.

6. The defect image sample generation method based on generative adversarial network according to claim 1, characterized in that: The second generative adversarial network model is a CGAN model.

7. A defect image sample generation system based on a generative adversarial network, characterized in that: include: A sample acquisition module, used to acquire real defect samples and non-defect samples, wherein the real defect samples are labeled with defect information; A first generative adversarial network model determination module, used to construct and train a first generative adversarial network model; The preprocessing module is used to preprocess real defect samples and non-defect samples, that is, to crop real defect samples, including: Obtaining defect coordinate information of the real defect sample; Get the input size to be cropped; Using the defect block coordinate information of the real defect sample and the input size to be cropped, the real defect sample is cropped to obtain a real defect block of the real defect sample, The real defect block of the real defect sample contains defect type information; the real defect block can be used as the real input of the discriminator in the generation phase to train the generator in the generation phase; A defective block image determination module, used for inputting the preprocessed real defect samples and non-defective samples into a first generative adversarial network model with Resnet as the backbone to generate a defective block image, wherein the defective block image includes defect type information; A coarse synthetic defect sample determination module, used to generate a coarse synthetic defect sample according to the defect block image and the non-defective sample, wherein the coarse synthetic defect sample carries a defect information label; A second generative adversarial network model determination module, used to construct and train a second generative adversarial network model; The defect image sample determination module is used to input the rough synthetic defect sample and the real defect sample into the second generative adversarial network model to generate a defect image sample, wherein the defect image sample has a defect information label, wherein the defect information label includes defect type information and defect coordinate information.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the defect image sample generation method based on a generative adversarial network as described in any one of claims 1 to 6 is implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the defect image sample generation method based on the generative adversarial network described in any one of claims 1 to 7 are implemented.

Images