Defect image generation method and device, equipment and medium

Abstract

The invention provides a defect image generation method and device, equipment and a medium. The method comprises the following steps: acquiring a plurality of normal sample images, and inputting the normal sample images and Gaussian noise into a first generative network; according to the plurality of normal sample images and the Gaussian noise, calculating and obtaining a defect mask corresponding to each normal sample image; and inputting the normal sample images and the defect masks corresponding to the normal sample images into a second generation network, and calculating to obtain a defect sample image corresponding to each normal sample image. According to the method, a large number of defect masks and defect sample images which are high in quality, diversified and reasonable in variety are generated by using a large number of existing normal sample images, and the large number of defect masks and defect sample images are further used for downstream defect detection tasks, so that the application range is wide; and pixel-level labels, namely defect masks, of the defect sample images can be provided while the defect sample images are generated, so that the performance of downstream tasks is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a defect image generation method, device, equipment and medium. Background Art

[0002] Defect detection is critical to ensuring product quality in industrial production processes. Traditional defect detection methods rely on manual inspection or simple image processing techniques, which are not only inefficient but also limited in accuracy by the operator's experience and fatigue.

[0003] In recent years, deep learning-based methods have significantly improved the speed and accuracy of defect detection. However, the performance of these methods depends heavily on large, diverse, and accurately labeled training data. In real industrial scenarios, especially for certain types of defects, we cannot obtain enough defect data, and labeling this data is difficult and expensive.

[0004] Limited by data issues, many unsupervised, semi-supervised, and weakly supervised methods have been proposed and applied to defect detection. Although unsupervised methods save labeling costs and only require normal samples in most cases, they cannot classify defects due to the lack of prior knowledge of defects. Semi-supervised methods use a large amount of unlabeled data and a small amount of labeled data to achieve good detection results while reducing labeling costs. However, if the distribution of unlabeled data is inconsistent with labeled data, it may cause model bias and affect the final performance. Compared with unsupervised methods, weakly supervised methods use coarse-grained labels to improve defect recognition, but their recognition accuracy is weak, and supervised methods still need to be improved. It is not difficult to find from the analysis of the above methods that these methods can be regarded as a compromise relative to supervised defect detection methods when it is not feasible and costly to obtain enough well-labeled defect data in industrial applications.

[0005] In order to solve the data problem, many methods have been proposed. In recent years, in research and practice, generative adversarial networks (GANs) have shown their great potential in many fields, especially in image generation, image restoration, and style transfer. GANs use their adversarial learning characteristics to fit the high-level feature distribution of real data. Many methods try to solve the problem of defective data based on GANs. However, most of these methods face the following problems:

[0006] First, the method itself requires a large amount of defect data.

[0007] Second, it is impossible to provide pixel-level labels of defect images while generating defect data; a few methods can provide pixel-level labels of defect images while generating them, but these methods either still require human intervention or cannot guarantee strict alignment between the generated defect images and pixel-level labels.

[0008] Third, the method is only applicable to texture category defects. Summary of the invention

[0009] The present invention provides a defect image generation method, device, equipment and medium to solve the problem that the defect detection method in the prior art relies on a large amount of defect data, while the defect image generation method cannot provide pixel-level labels of the defect image while generating the defect data, has high cost and a narrow scope of application, and is only applicable to texture category defects.

[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is to provide a defect image generation method, comprising the steps of: obtaining multiple normal sample images, inputting the normal sample images and Gaussian noise into a first generation network; calculating and obtaining a defect mask corresponding to each of the normal sample images based on the multiple normal sample images and Gaussian noise; inputting the normal sample image and the corresponding defect mask into a second generation network, and calculating and obtaining a defect sample image corresponding to each of the normal sample images.

[0011] In some embodiments, the step of calculating and obtaining the defect mask corresponding to each of the normal sample images based on the multiple normal sample images and Gaussian noise specifically includes: performing feature extraction on the normal sample images to obtain the intermediate defect features corresponding to each of the normal sample images; upsampling the intermediate defect features and normalizing the obtained values, and obtaining the multiple defect masks corresponding to each of the normal sample images based on the guidance of the Gaussian noise.

[0012] In some embodiments, the step of calculating and obtaining the defect sample image corresponding to each of the normal sample images specifically includes: extracting the features of the normal sample image and modulating and activating it, and guiding the generation of the defect foreground corresponding to the normal sample image through the defect mask; fusing the normal sample image, the defect mask and the defect foreground to obtain the defect sample image corresponding to each of the normal sample images.

[0013] In some embodiments, the first generation network is obtained by training the first initial generation network, and the training method includes the steps of: obtaining a real defect mask corresponding to a real defect sample image, and generating a training defect mask corresponding to the normal sample image through the first initial generation network; using a first region adversarial loss function to compare the real defect mask with the training defect mask, and modifying and optimizing the parameters of the first initial generation network according to the comparison result; calculating and obtaining a real defect masked image according to the real defect sample image and the real defect mask; calculating and obtaining a training defect masked image according to the normal sample image and the training defect mask; using a first image adversarial loss function to compare the real defect masked image with the training defect masked image, and modifying and optimizing the parameters of the first initial generation network according to the comparison result; when the values ​​of the first region adversarial loss function and the first image adversarial loss function are both minimized, the trained first generation network is obtained.

[0014] In some embodiments, the training method of the first generative network also includes the steps of: performing threshold processing on the normal sample image to obtain a segmentation mask corresponding to the normal sample image; using a target boundary perception loss function to compare the segmentation mask and the training defect mask, and modifying and optimizing the parameters of the first initial generative network accordingly according to the comparison results; when the values ​​of the first region adversarial loss function, the first image adversarial loss function and the target boundary perception loss function are all minimized, the trained first generative network is obtained.

[0015] In some embodiments, the second generation network is obtained by training the second initial generation network, and the training method includes the steps of: obtaining a real defect sample image, and generating a training defect sample image corresponding to the normal sample image through the second initial generation network; using a second image adversarial loss function to compare the real defect sample image and the training defect sample image, and modifying and optimizing the parameters of the second initial generation network accordingly according to the comparison results; obtaining a real defect mask corresponding to the real defect sample image, and using the real defect mask to respectively extract the defect areas of the real defect sample image and the training defect sample image to form corresponding real defect area maps and training defect area maps; using a second area adversarial loss function to compare the real defect area map and the training defect area map, and modifying and optimizing the parameters of the second initial generation network accordingly according to the comparison results; when the values ​​of the second area adversarial loss function and the second image adversarial loss function are both minimized, the trained second generation network is obtained.

[0016] The present invention also provides a defect image generating device, comprising: an acquisition unit, used to acquire multiple normal sample images; a first generation network, used to calculate and acquire a defect mask corresponding to each of the normal sample images based on the multiple normal sample images and Gaussian noise; and a second generation network, used to calculate and acquire a defect sample image corresponding to each of the normal sample images based on the normal sample images and the corresponding defect masks.

[0017] In some embodiments, the first generation network includes a first generator and a first two-level discriminator, and the second generation network includes a second generator and a second two-level discriminator; the first generator is used to calculate and obtain the defect mask corresponding to each normal sample image based on multiple normal sample images and Gaussian noise, and the first two-level discriminator is used to train the first generation network based on the defect mask; the second generator is used to calculate and obtain the defect sample image corresponding to each normal sample image based on the normal sample image and the corresponding defect mask, and the second two-level discriminator is used to train the second generation network based on the defect sample image; the first generator includes a first encoder and a first decoder connected in sequence, and the first two-level discriminator includes a first defect image level discriminator and a first defect area level discriminator; the second generator includes a second encoder, a second decoder and a defect focusing module connected in sequence, and the second two-level discriminator includes a second defect image level discriminator and a second defect area level discriminator; the second decoder includes two feature modulation modules connected in sequence.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method when executing the computer program.

[0019] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0020] The beneficial effects of the present invention are as follows: the present application provides a defect image generation method, device, equipment and medium. The method comprises the steps of: obtaining a plurality of normal sample images, inputting the normal sample images and Gaussian noise into a first generation network; calculating and obtaining a defect mask corresponding to each normal sample image according to the plurality of normal sample images and Gaussian noise; inputting the normal sample image and its corresponding defect mask into a second generation network, and calculating and obtaining a defect sample image corresponding to each normal sample image. The present application utilizes a large number of existing normal sample images to generate a large number of high-quality, diverse and reasonable defect masks and defect sample images, and further uses a large number of defect masks and defect sample images for downstream defect detection tasks (such as defect classification, defect segmentation, etc.). It not only has a wide range of applications, but also can provide pixel-level labels (i.e., defect masks) of defect sample images while generating defect sample images, effectively improving the performance of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of a defect image generation method of the present application;

[0022] Figure 2 It is a data flow diagram of a defect image generation method of the present application;

[0023] Figure 3 is a schematic diagram of a first generation network in a defect image generation method of the present application;

[0024] Figure 4 is a schematic diagram of a first initial generation network in a defect image generation method of the present application;

[0025] Figure 5 is a schematic diagram of a second generation network in a defect image generation method of the present application;

[0026] Figure 6 is a schematic diagram of a second initial generation network in a defect image generation method of the present application;

[0027] Figure 7 is a schematic diagram of a defect detection model in a defect image generation method of the present application;

[0028] Figure 8 It is a schematic diagram of the composition of a defect image generating device of the present application;

[0029] Fig. 9 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0030] Fig.10 It is a schematic block diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0031] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0032] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0033] like Figure 1 As shown, the present invention provides a defect image generation method, comprising the steps of:

[0034] S1: Obtain multiple normal sample images, and input the normal sample images and Gaussian noise into the first generation network.

[0035] S2: According to multiple normal sample images and Gaussian noise, a defect mask corresponding to each normal sample image is calculated and obtained.

[0036] S3: Input the normal sample image and its corresponding defect mask into the second generation network, and calculate and obtain the defect sample image corresponding to each normal sample image.

[0037] This application utilizes a large number of existing normal sample images to generate a large number of high-quality, diverse and reasonable defect masks and defect sample images, and further uses a large number of defect masks and defect sample images for downstream defect detection tasks (such as defect classification, defect segmentation, etc.). It not only has a wide range of applications, but also can provide pixel-level labels (i.e., defect masks) of defect sample images while generating defect sample images, effectively improving the performance of downstream tasks.

[0038] Combine the following Figures 1 to 10 , the present application is further described in detail with specific embodiments.

[0039] Figure 1 The following is a flow chart of a defect image generation method provided by an embodiment of the present application, which is described in detail as follows:

[0040] S1: Obtain multiple normal sample images, and input the normal sample images and Gaussian noise into the first generation network.

[0041] Combination Figure 2As shown, the normal sample image 8 refers to the image of an object without defect problems when the object is subjected to defect detection. There are a large number of normal sample images 8, and all of them can be used. Gaussian noise 81 is randomly generated noise, and the probability density function of Gaussian noise 81 obeys Gaussian distribution (i.e., normal distribution). Common Gaussian noise 81 includes fluctuation noise, cosmic noise, thermal noise, and shot noise. The normal sample image 8 and Gaussian noise 81 are input into the first generation network 2 (i.e., DM-GAN, Defect Mask GAN) for the next step of processing.

[0042] S2: According to multiple normal sample images and Gaussian noise, a defect mask corresponding to each normal sample image is calculated and obtained.

[0043] Specific, combined Figure 2 , Figure 3 As shown, the normal sample image 8 and the Gaussian noise 81 are input into the first encoder 41 in the first generator 4, and the features of the normal sample image 8 are extracted to obtain the intermediate defect features corresponding to each normal sample image 8.

[0044] Furthermore, the intermediate defect features are input into the first decoder 42, and the intermediate defect features are upsampled; and the obtained values ​​are normalized through a normalization layer (i.e., a sigmoid layer), and under the guidance of Gaussian noise 81, multiple defect masks 82 corresponding to each normal sample image 8 are finally obtained, and each defect mask 82 matches the features of the corresponding normal sample image 8, and the value range of the defect mask 82 is (0, 1).

[0045] The process of the first generation network 2 generating the defect mask 82 can be described by the following formula:

[0046] m gen =G(n,z)

[0047] Among them, m gen represents the generated defect mask 82, z represents Gaussian noise 81 conforming to the normal distribution N(0,1), n ​​represents the normal sample image 8, and G represents the first generator 4.

[0048] Furthermore, the normal sample image 8 and the corresponding defect mask 82 are input into the second generation network 3 (i.e., MGDG-GAN, Mask-Guided Defect Generation GAN) for further processing.

[0049] It should be noted that the first generation network 2 is obtained by Figure 4 The first initial generation network is trained in the training method, and the training method includes the following steps:

[0050] First, obtain the real defect mask 84 corresponding to the real defect sample image 86, and generate the training defect mask 85 corresponding to the normal sample image 8 through the first initial generation network. Input the real defect mask 84 and the training defect mask 85 into the first defect region level discriminator 52, use the first region adversarial loss function to compare the real defect mask 84 and the training defect mask 85, and modify and optimize the parameters of the first initial generation network accordingly according to the comparison result.

[0051] The above steps can be expressed by the following formula:

[0052]

[0053] in, represents the first region adversarial loss function, G represents the first generator 4, D mask represents the first defect region level discriminator 52, E represents the expected value, m represents the real defect mask 84, m gen Represents the training defect mask 85.

[0054] At the same time, according to the existing real defect sample image 86 and the real defect mask 84, the real defect masked image 87 is calculated and obtained; according to the normal sample image 8 and the training defect mask 85, the training defect masked image 88 is calculated and obtained.

[0055] The above process can be expressed by the following formula:

[0056] I real =max(d real ,m)=(1-m)⊙d real +m;

[0057] I gen =max(n,m gen )=(1-m gen )⊙n+m gen

[0058] Among them, I real Represents the real defect masked image 87, I gen Representative training defect masked image 88, d real represents the real defect sample image 86, m represents the real defect mask 84, m gen represents the training defect mask 85, n represents the normal sample image 8, and ⊙ represents element-by-element multiplication.

[0059] The real defect masked image 87 and the training defect masked image 88 are input into the first defect image level discriminator 51, and the real defect masked image 87 and the training defect masked image 88 are compared using the first image adversarial loss function, and the parameters of the first initial generation network are modified and optimized accordingly based on the comparison results.

[0060] The above process can be expressed by the following formula:

[0061]

[0062] in, represents the first image adversarial loss function, G represents the first generator 4, and D dmi represents the first defect image level discriminator 51, and E represents expectation.

[0063] Since the defect masked image contains both the defect mask and the real background, by comparing the real defect masked image 87 and the training defect masked image 88 through the first defect image level discriminator 51, the first generator 4 can generate a reasonable defect mask that is more consistent with the background information.

[0064] Further, the normal sample image 8 is thresholded to obtain a segmentation mask 89 corresponding to the normal sample image 8. The segmentation mask 89 is compared with the training defect mask 85 using the target boundary perception loss function 891, and the parameters of the first initial generation network are modified and optimized accordingly according to the comparison result.

[0065] The above process can be expressed by the following formula:

[0066]

[0067] Among them, L oba represents the target boundary perception loss function 891,m b Represents segmentation mask 89, m gen represents the training defect mask 85, H and W represent the height and width of the segmentation mask 89 and the training defect mask 85, respectively, and m b (i, j) represents the pixel value of the segmentation mask 89 at the coordinate (i, j), m gen (i, j) represents the pixel value of the training defect mask 85 at the coordinate (i, j).

[0068] By using the target boundary perception loss function 891 to compare the segmentation mask 89 and the training defect mask 85, the first generator 4 can better perceive the object boundary when the real defect sample image 86 is limited, and has the ability to perceive the object boundary and generate a reasonable defect mask, and penalize the target pixels generated outside the object.

[0069] Further, when the values ​​of the first region adversarial loss function, the first image adversarial loss function and the target boundary perception loss function 891 are all minimized, we get Figure 3 The first generative network 2 trained in .

[0070] S3: Input the normal sample image and its corresponding defect mask into the second generation network, and calculate and obtain the defect sample image corresponding to each normal sample image.

[0071] Specific, combined Figure 2 , Figure 5 As shown, the normal sample image 8 and its corresponding defect mask 82 are input into the second encoder 61 in the second generator 6 to extract the features of the normal sample image 8; and modulated and activated by two feature modulation modules 621 in the second decoder 62, and the defect mask 82 is used to guide the second decoder 62 to generate the defect foreground corresponding to the normal sample image 8. The defect foreground includes the target defect area.

[0072] The generation process of defect prospects can be described by the following formula:

[0073] d f =G(n,m)

[0074] Among them, d f represents defect foreground, G represents the second generator 6, n represents the normal sample image 8, and m represents the defect mask 82.

[0075] Furthermore, the defect foreground is input into the defect focusing module 63, and the normal sample image 8, the defect mask 82 and the defect foreground are fused to finally obtain a defect sample image 83 corresponding to each normal sample image 8. The defect sample image 83 contains specific features, such as the position, shape and size of the defect.

[0076] The process of the second generation network 3 generating the defect sample image 83 can be described by the following formula:

[0077] d gen =DF(d f ,n,m)=d f ⊙m+n⊙(1-m)

[0078] Among them, d gen represents the generated defect sample image 83, DF represents the defect focusing module 63, d f represents defect foreground, n represents normal sample image 8, m represents defect mask 82, and ⊙ represents element-by-element multiplication.

[0079] Through the defect focusing module 63, the second generation network 3 can focus on generating local defect areas rather than the entire image. Therefore, when training the second generation network 3, only a small amount of training data is required.

[0080] It should be noted that the second generation network 3 is Figure 6 The second initial generation network is trained in the training method, and the training method includes the following steps:

[0081] First, a real defect sample image 86 is obtained, and a training defect sample image 91 corresponding to the normal sample image 8 is generated through the second initial generation network. The real defect sample image 86 and the training defect sample image 91 are input into the second defect image level discriminator 71, and the real defect sample image 86 and the training defect sample image 91 are compared using the second image adversarial loss function, and the parameters of the second initial generation network are modified and optimized accordingly according to the comparison results.

[0082] The above steps can be expressed by the following formula:

[0083]

[0084] in, represents the second image adversarial loss function, G represents the second generator 6, and D image represents the second defect image level discriminator 71, E represents the expectation, d real Representative real defect sample image 86, d gen Representative training defect sample image 91.

[0085] By using the second image to counter the loss function, the overall consistency of the image can be guaranteed, making the transition between the defect area and the normal area more natural.

[0086] At the same time, the real defect mask 84 corresponding to the existing real defect sample image 86 is obtained, and the defect areas of the real defect sample image 86 and the training defect sample image 91 are extracted using the real defect mask 84 to form corresponding real defect area map 92 and training defect area map 93.

[0087] The above process can be expressed by the following formula:

[0088] d real_reg =m⊙d real +(1-m);

[0089] d gen_reg =m⊙d gen +(1-m)

[0090] Among them, d real_reg Figure 92,d represents the actual defect areagen_reg Representative training defect area Figure 93,d real represents the real defect sample image 86, m represents the real defect mask 84, d gen Representative training defect sample image 91.

[0091] The real defect area map 92 and the training defect area map 93 are input into the second defect area level discriminator 72, and the real defect area map 92 and the training defect area map 93 are compared using the second area adversarial loss function, and the parameters of the second initial generation network are modified and optimized accordingly according to the comparison results.

[0092] The above process can be expressed by the following formula:

[0093]

[0094] in, represents the second region adversarial loss function, G represents the second generator 6, D defect represents the second defect region level discriminator 72, E represents the expectation, d real_reg Figure 92,d represents the actual defect area gen_reg Figure 93 represents the training defect area.

[0095] Through the above method, the second defect area level discriminator 72 can focus only on defects, and then the second generator 6 can focus on defect details, thereby ensuring the quality of the defect area.

[0096] When the values ​​of the second region adversarial loss function and the second image adversarial loss function are minimized, we get Figure 5 The second generative network 3 trained in .

[0097] Further, such as Figure 7 As shown, a large number of high-quality defect masks 82 and defect sample images 83 generated by the first generation network 2 and the second generation network 3 in the present application can be further used in the downstream defect detection model 9, significantly improving the generalization ability and detection performance of the defect detection model 9, and improving the performance of downstream tasks (such as defect classification and defect segmentation).

[0098] Specifically, it can be used in the following aspects:

[0099] First, data enhancement: The generated defect sample images 83 can be used for data enhancement to expand the diversity of the training set. By increasing the training data containing various defects (i.e., defect sample images 83), the defect detection model 9 can better learn the characteristics of different types of defects and improve the detection ability of unseen defects.

[0100] Second, balance the data set: In practical applications, the normal sample images 8 are usually much more than the defect sample images 83, which will lead to imbalance in the training data and affect the detection performance of the defect detection model 9. The generated defect sample images 83 can be used to balance the data set, so that the defect detection model 9 can pay equal attention to the normal sample images 8 and the defect sample images 83 during the training process, thereby improving the detection accuracy.

[0101] Third, improve segmentation performance: For segmentation tasks, the generated defect masks 82 can be used as pixel-level labels to provide accurate defect location and shape information. These defect masks 82 can be used to train segmentation models, enabling them to more accurately identify and segment defect areas, and improve the accuracy and consistency of segmentation results.

[0102] Corresponding to the defect image generation method of the above embodiment, Figure 8 A schematic diagram of the composition of a defect image generating device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0103] refer to Figure 8 , the defect image generating device comprises:

[0104] The acquisition unit 1 is used to acquire a plurality of normal sample images.

[0105] The first generation network 2 is used to calculate and obtain the defect mask corresponding to each normal sample image according to multiple normal sample images and Gaussian noise.

[0106] The second generation network 3 is used to calculate and obtain the defect sample image corresponding to each normal sample image according to the normal sample image and its corresponding defect mask.

[0107] Specifically, in this embodiment, the first generation network 2 includes a first generator 4 and a first dual-level discriminator 5, and the second generation network 3 includes a second generator 6 and a second dual-level discriminator 7. The first generator 4 includes a first encoder 41 and a first decoder 42 connected in sequence, and the first dual-level discriminator 5 includes a first defect image level discriminator 51 and a first defect area level discriminator 52. The second generator 6 includes a second encoder 61, a second decoder 62 and a defect focus module 63 connected in sequence, and the second dual-level discriminator 7 includes a second defect image level discriminator 71 and a second defect area level discriminator 72. The second decoder 62 includes two feature modulation modules 621 connected in sequence.

[0108] The first generator 4 is used to calculate and obtain the defect mask corresponding to each normal sample image based on multiple normal sample images and Gaussian noise, and the first two-level discriminator 5 is used to train the first generation network 2 based on the defect mask; the second generator 6 is used to calculate and obtain the defect sample image corresponding to each normal sample image based on the normal sample image and its corresponding defect mask, and the second two-level discriminator 7 is used to train the second generation network 3 based on the defect sample image.

[0109] It should be noted that since the above-mentioned defect image generating device is based on the same concept as the embodiment of the method of the present application, its specific functions, the process of each module realizing its respective function and the technical effects brought about can be specifically referred to the description of the embodiment shown in the above-mentioned defect image generating method, and will not be repeated here.

[0110] Based on the same inventive concept, the present application also provides an electronic device, which includes a processor, a memory and a communication circuit, and the processor is connected to the memory and the communication circuit respectively; wherein the communication circuit is used for communication connection, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-mentioned defect image generation method.

[0111] See also Fig. 9 The electronic device described in the embodiment of the present application may specifically include a processor 210 and a memory 220. The memory 220 is coupled to the processor 210.

[0112] The processor 210 is used to control the operation of the electronic device. The processor 210 may also be referred to as a CPU (Central Processing Unit). The processor 210 may be an integrated circuit chip having the ability to process signals. The processor 210 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 210 may also be any conventional processor, etc.

[0113] The memory 220 is used to store computer programs, and may be a RAM, a ROM, or other types of storage terminals. Specifically, the memory 220 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 220 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage terminals, flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 220 is used to store at least one program code.

[0114] The processor 210 is used to execute the computer program stored in the memory 220 to implement the methods described in the various method embodiments of the present application.

[0115] In some embodiments, the electronic device may further include: a peripheral terminal interface 230 and at least one peripheral terminal. The processor 210, the memory 220 and the peripheral terminal interface 230 may be connected via a bus or a signal line. Each peripheral terminal may be connected to the peripheral terminal interface 230 via a bus, a signal line or a circuit board. Specifically, the peripheral terminal includes: at least one of a radio frequency circuit 240, a display screen 250, an audio circuit 260 and a power supply 270.

[0116] The peripheral terminal interface 230 may be used to connect at least one peripheral terminal related to I / O (Input / output) to the processor 210 and the memory 220. In some embodiments, the processor 210, the memory 220, and the peripheral terminal interface 230 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 210, the memory 220, and the peripheral terminal interface 230 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0117] The radio frequency circuit 240 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 240 communicates with the communication network and other IoT devices through electromagnetic signals, and the radio frequency circuit 240 is the communication circuit of the electronic device. The radio frequency circuit 240 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 240 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identity module card, and the like. The radio frequency circuit 240 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 240 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0118] The display screen 250 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 250 is a touch display screen, the display screen 250 also has the ability to collect touch signals on the surface or above the surface of the display screen 250. The touch signal can be input to the processor 210 as a control signal for processing. At this time, the display screen 250 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 250 can be one, set on the front panel of the electronic device; in other embodiments, the display screen 250 can be at least two, respectively set on different surfaces of the electronic device or in a folding design; in other embodiments, the display screen 250 can be a flexible display screen, set on a curved surface or a folding surface of the electronic device. Even, the display screen 250 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 250 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light□EmittingDiode, organic light-emitting diode).

[0119] The audio circuit 260 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 210 for processing, or input them into the radio frequency circuit 240 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 210 or the radio frequency circuit 240 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 260 may also include a headphone jack.

[0120] The power supply 270 is used to power various components in the electronic device. The power supply 270 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 270 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0121] For a detailed description of the functions and execution processes of each functional module or component in the electronic device embodiment of the present application, reference can be made to the description in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0124] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0125] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above-mentioned defect image generating method.

[0126] See also Fig.10 , if the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 300. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions / computer programs to enable an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage media.

[0127] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, and will not be repeated here.

[0128] It can be seen that the present invention discloses a defect image generation method, device, equipment and medium. The method includes the steps of: obtaining multiple normal sample images, inputting the normal sample images and Gaussian noise into a first generation network; calculating and obtaining the defect mask corresponding to each normal sample image according to the multiple normal sample images and Gaussian noise; inputting the normal sample image and its corresponding defect mask into a second generation network, and calculating and obtaining the defect sample image corresponding to each normal sample image. This application generates a large number of high-quality, diverse and reasonable defect masks and defect sample images by utilizing a large number of existing normal sample images, and further uses a large number of defect masks and defect sample images for downstream defect detection tasks (such as defect classification, defect segmentation, etc.). It not only has a wide range of applications, but also can provide pixel-level labels (i.e., defect masks) of defect sample images while generating defect sample images, effectively improving the performance of downstream tasks.

[0129] The above are only embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A defect image generation method, characterized in that: Includes steps: Acquire a plurality of normal sample images, and input the normal sample images and Gaussian noise into a first generation network; According to the plurality of normal sample images and Gaussian noise, a defect mask corresponding to each normal sample image is calculated and obtained; The normal sample images and the corresponding defect masks are input into a second generation network, and a defect sample image corresponding to each normal sample image is obtained by calculation.

2. The defect image generation method according to claim 1, characterized in that: The step of calculating and obtaining a defect mask corresponding to each normal sample image according to the plurality of normal sample images and Gaussian noise specifically includes: Performing feature extraction on the normal sample images to obtain intermediate defect features corresponding to each of the normal sample images; The intermediate defect features are upsampled and the obtained values ​​are normalized, and a plurality of defect masks corresponding to each normal sample image are obtained under the guidance of the Gaussian noise.

3. The defect image generation method according to claim 1, characterized in that: The step of calculating and obtaining the defect sample image corresponding to each of the normal sample images specifically includes: Extracting features of the normal sample image and modulating and activating it, and guiding the generation of a defect foreground corresponding to the normal sample image through the defect mask; The normal sample image, the defect mask and the defect foreground are fused to obtain the defect sample image corresponding to each of the normal sample images.

4. The defect image generation method according to claim 1, characterized in that: The first generating network is obtained by training the first initial generating network, and the training method comprises the steps of: Obtaining a real defect mask corresponding to a real defect sample image, and generating a training defect mask corresponding to the normal sample image through the first initial generation network; using a first regional adversarial loss function, comparing the real defect mask with the training defect mask, and modifying and optimizing the parameters of the first initial generation network accordingly according to the comparison result; Calculating and acquiring a real defect masked image according to the real defect sample image and the real defect mask; Calculate and obtain a training defect masked image according to the normal sample image and the training defect mask; Using a first image adversarial loss function, the real defect masked image and the training defect masked image are compared, and parameters of the first initial generation network are modified and optimized accordingly according to the comparison result; When the values ​​of the first region adversarial loss function and the first image adversarial loss function are both minimized, the trained first generation network is obtained.

5. The defect image generation method according to claim 4, characterized in that: The training method of the first generating network also includes the steps of: Performing threshold processing on the normal sample image to obtain a segmentation mask corresponding to the normal sample image; using a target boundary perception loss function to compare the segmentation mask with a training defect mask, and modifying and optimizing the parameters of the first initial generation network accordingly according to the comparison result; When the values ​​of the first region adversarial loss function, the first image adversarial loss function, and the target boundary perception loss function are all minimized, the trained first generation network is obtained.

6. The defect image generation method according to claim 1, characterized in that: The second generation network is obtained by training the second initial generation network, and the training method comprises the steps of: Acquire a real defect sample image, and generate a training defect sample image corresponding to the normal sample image through the second initial generation network; Using a second image adversarial loss function, the real defect sample image and the training defect sample image are compared, and the parameters of the second initial generation network are modified and optimized accordingly according to the comparison result; Acquire a real defect mask corresponding to the real defect sample image, and use the real defect mask to extract defect areas of the real defect sample image and the training defect sample image respectively to form a corresponding real defect area map and a training defect area map; Using a second region adversarial loss function, the real defect region map and the training defect region map are compared, and the parameters of the second initial generation network are modified and optimized accordingly according to the comparison result; When the values ​​of the second region adversarial loss function and the second image adversarial loss function are both minimized, the trained second generation network is obtained.

7. A defect image generating device, characterized in that: include: An acquisition unit, used for acquiring a plurality of normal sample images; A first generation network is used to calculate and obtain a defect mask corresponding to each normal sample image according to the plurality of normal sample images and Gaussian noise; The second generation network is used to calculate and obtain the defect sample image corresponding to each normal sample image according to the normal sample image and the corresponding defect mask.

8. The defect image generating device according to claim 7, characterized in that: The first generation network includes a first generator and a first two-level discriminator, and the second generation network includes a second generator and a second two-level discriminator; The first generator is used to calculate and obtain a defect mask corresponding to each normal sample image according to the plurality of normal sample images and Gaussian noise, and the first two-level discriminator is used to train the first generation network according to the defect mask; The second generator is used to calculate and obtain the defect sample image corresponding to each normal sample image according to the normal sample image and the corresponding defect mask, and the second two-level discriminator is used to train the second generation network according to the defect sample image; The first generator includes a first encoder and a first decoder connected in sequence, and the first two-level discriminator includes a first defect image level discriminator and a first defect region level discriminator; The second generator includes a second encoder, a second decoder and a defect focusing module connected in sequence, the second two-level discriminator includes a second defect image level discriminator and a second defect area level discriminator; the second decoder includes two feature modulation modules connected in sequence.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images