A method, apparatus, electronic device, and storage medium for generating defect samples

Abstract

The present application discloses a method, apparatus, electronic device, and storage medium for generating defective samples; the method includes: randomly sampling standard Gaussian noise, using the distribution of the sampled noise as the noise feature at the current time step, inputting the noise feature and the semantic label corresponding to the target defect into a defect generation network to predict the distribution of the diffusion noise at the previous time step; randomly sampling based on the distribution of the perturbation noise to obtain the diffusion noise at the previous time step; calculating the noise feature at the previous time step according to the diffusion noise and the noise feature at the current time step; repeating the above operations until the noise feature at the 0th time step is calculated; generating a local defective image corresponding to the target defect based on the noise feature at the 0th time step; inputting the image into a defect fusion network to generate a defective sample corresponding to the target defect. The embodiments of the present application can generate highly realistic local defects and naturally integrate them into normal samples.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a method, device, electronic device, and storage medium for generating defective samples. Background Art

[0002] In recent years, defect detection has been of great significance in many fields. In the manufacturing industry, defect detection can help enterprises promptly discover defects on the production line and prevent defective products from entering the market. This can not only ensure product quality, improve user satisfaction, but also reduce the costs and losses of enterprises. Or in the medical field, defect detection can help doctors promptly discover the lesions and abnormalities of diseases, thereby improving the diagnostic accuracy and treatment effect. For example, in the field of medical imaging, defect detection can help doctors discover lesions such as tumors and vascular lesions, thereby assisting in diagnosis and treatment.

[0003] Currently, using deep learning technology for defect detection is the mainstream technical means. However, the supervised method based on deep learning requires a large amount of labeled data to train the model. In fact, labeled data is often scarce, especially in the case of defective data. First, since real defective samples are often more difficult to obtain than normal samples, the number of samples for defect detection is often limited, which will limit the performance of the defect detection system. Through defective sample generation technology, a large number of defective samples can be synthesized, so that the defect detection system can more comprehensively learn the characteristics and patterns of defects, thereby improving the detection performance.

[0004] Currently, there are already some defective sample generation methods based on unsupervised learning. For example, methods based on generative adversarial networks (GANs), variational autoencoders (VAEs), etc. These methods can increase the number of defective samples by randomly generating data. However, due to the uncertainty of data generation, the generated defective samples may not be accurate and reliable enough, and there are also certain defect biases. For this reason, researchers have proposed diffusion models to alleviate the problems of the above-mentioned generative models. A diffusion model is a probability-based generative model used to generate samples with highly realistic properties. The basic idea of the diffusion model is to regard the sample as an initial state and gradually diffuse the sample from a random state to a real sample space through a series of random and reverse diffusion processes. Compared with traditional generative models, the diffusion model has the following advantages: the diffusion operation is reversible, and the sample generation process can be precisely controlled, thereby generating high-quality samples; the randomness of the walk in the diffusion process enables the model to have a certain adaptability to different data.

[0005] However, it is difficult to directly generate defect samples based on the above-mentioned generation model. The main reason is that the area of the defect part in the sample is small, and it is difficult for the model to focus on the generation of defects. The network may pay more attention to the generation of the sample background. Therefore, it is difficult for the above-mentioned individual generation models to directly generate highly realistic defect samples. Summary of the Invention

[0006] The present application provides a method, apparatus, electronic device and storage medium for generating defect samples, which can generate highly realistic local defects and naturally integrate them into normal samples.

[0007] In a first aspect, an embodiment of the present application provides a method for generating a defect sample, the method comprising:

[0008] Randomly sample standard Gaussian noise, and use the distribution of the randomly sampled noise as the noise feature f of the current time step T , and input the noise feature f of the current time step T and the semantic label y corresponding to the target defect obtained in advance into the pre-trained defect generation network CUNet, and predict the distribution ε of the diffusion noise of the previous time step of the current time step through the pre-trained defect generation network CUNet θ (f T , T, y);

[0009] Based on the distribution ε of the perturbation noise of the previous time step θ (f T , T, y), randomly sample the diffusion noise of the previous time step; and calculate the noise feature f of the previous time step according to the diffusion noise of the previous time step and the noise feature f of the current time step T ; take the previous time step as the current time step, and repeat the above operations until the noise feature f0 of the 0th time step is calculated; T-1

[0010] Generate a local defect image corresponding to the target defect based on the noise feature f0 of the 0th time step;

[0011] Input the local defect image into the pre-trained defect fusion network, and generate a defect sample corresponding to the target defect through the defect fusion network.

[0012] In a second aspect, an embodiment of the present application further provides a device for generating a defect sample, the device comprising: a prediction module, a calculation module, a generation module and a fusion module; wherein,

[0013] The prediction module is used to randomly sample standard Gaussian noise, and use the distribution of the randomly sampled noise as the noise feature f of the current time step​T , and input the noise feature f of the current time step T and the semantic label y corresponding to the pre-acquired target defect into the pre-trained defect generation network CUNet, and predict the distribution ε of the diffusion noise of the previous time step of the current time step through the pre-trained defect generation network CUNet θ (f T , T, f);

[0014] The calculation module is configured to randomly sample the diffusion noise of the previous time step based on the distribution ε of the perturbation noise of the previous time step θ (f T , T, y); and calculate the noise feature f of the previous time step according to the diffusion noise of the previous time step and the noise feature f of the current time step T ; Take the previous time step as the current time step, and repeat the above operations until the noise feature f0 of the 0th time step is calculated; T-1

[0015] The generation module is configured to generate a local defect image corresponding to the target defect based on the noise feature f0 of the 0th time step;

[0016] The fusion module is configured to input the local defect image into a pre-trained defect fusion network, and generate a defect sample corresponding to the target defect through the defect fusion network.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0018] One or more processors;

[0019] A memory for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a defect sample according to any embodiment of the present application.

[0021] In a fourth aspect, an embodiment of the present application provides a storage medium, on which a computer program is stored, and when the program is executed by a processor, the method for generating a defect sample according to any embodiment of the present application is implemented.

[0022] ​An embodiment of the present application provides a method, device, electronic device, and storage medium for generating defective samples. First, random sampling is performed on standard Gaussian noise, and the distribution of the noise obtained by random sampling is used as the noise feature at the current time step. Then, the noise feature at the current time step and the semantic label corresponding to the target defect obtained in advance are input into a pre-trained defect generation network, and the distribution of the diffusion noise at the previous time step of the current time step is predicted by the pre-trained defect generation network. Then, diffusion noise at the previous time step is randomly sampled based on the distribution of the perturbation noise at the previous time step; and based on the diffusion noise at the previous time step and the noise feature at the current time step, the noise feature at the previous time step of the current time step is calculated; the previous time step is taken as the current time step, and the above operations are repeated until the noise feature at the 0th time step is calculated; then, a local defective image corresponding to the target defect is generated based on the noise feature at the 0th time step; finally, the local defective image is input into a pre-trained defect fusion network, and a defective sample corresponding to the target defect is generated by the defect fusion network. That is to say, in the technical solution of the present application, the method for generating defective samples is based on multiple stages. In the first stage, a local defective image is generated through a defect generation network; in the second stage, a highly realistic defective sample is generated through a defect fusion network. And the present application gradually diffuses the sample from a random state to a real sample space through a reverse diffusion process. The present application has the following advantages compared with traditional generation models: the diffusion operation is reversible, and the generation process of the sample can be precisely controlled, so as to generate high-quality samples. In the prior art, since the defective part occupies a small area of the sample, it is difficult for the model to focus on the generation of defects. Therefore, it is difficult to directly generate defective samples based on the generation model, and the network may pay more attention to the generation of the sample background. Therefore, compared with the prior art, the method, device, electronic device, and storage medium for generating defective samples provided by the embodiments of the present application can generate highly realistic local defects and naturally integrate them into normal samples; moreover, the technical solution of the embodiments of the present application is simple and convenient to implement, easy to popularize, and has a wider application range. Description of the Drawings

[0023] Figure 1 It is the first flowchart of the method for generating defective samples provided by the embodiment of the present application;

[0024] Figure 2 It is the first flowchart of the method for training a defect generation network provided by the embodiment of the present application;

[0025] Figure 3 It is the schematic diagram of the principle of the method for training a defect generation network provided by the example of the present application;

[0026] Figure 4 It is the second flowchart of the method for training a defect generation network provided by the embodiment of the present application;

[0027] Figure 5 It is the first process schematic diagram of the training method of the defect fusion network provided by the embodiment of the present application;

[0028] Figure 6 It is the second process schematic diagram of the training method of the defect fusion network provided by the embodiment of the present application;

[0029] Figure 7 It is the principle schematic diagram of the training method of the defect fusion network provided by the example of the present application;

[0030] Figure 8 It is the structure schematic diagram of the generating device of the defect sample provided by the embodiment of the present application;

[0031] Figure 9 It is the structure schematic diagram of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0032] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the convenience of description, only parts related to the present application rather than all structures are shown in the drawings.

[0033] Embodiment 1

[0034] Figure 1 It is the first process schematic diagram of the generating method of the defect sample provided by the embodiment of the present application. This method can be executed by the generating device of the defect sample or an electronic device. The device or the electronic device can be implemented in a software and / or hardware manner, and the device or the electronic device can be integrated in any intelligent device with network communication functions. As Figure 1 shown, the generating method of the defect sample can include the following steps:

[0035] S101. Randomly sample the standard Gaussian noise, use the distribution of the randomly sampled noise as the noise feature of the current time step, and input the noise feature of the current time step and the semantic label corresponding to the target defect obtained in advance into the pre-trained defect generation network, and predict the distribution of the diffusion noise of the previous time step of the current time step through the pre-trained defect generation network.

[0036] In this step, the electronic device can randomly sample the standard Gaussian noise, use the distribution of the randomly sampled noise as the noise feature f T of the current time step, and TThe semantic label y corresponding to the pre-acquired target defect is input into the pre-trained defect generation network CUNet, and the distribution ε of the diffusion noise at the previous time step of the current time step is predicted through the pre-trained defect generation network CUNet θ (f T , T, y). Specifically, the electronic device can randomly sample the standard Gaussian noise, and use the distribution of the randomly sampled noise as the noise feature at the T-th time step, denoted as f T ; then input f T and the semantic label y corresponding to the target defect into the pre-trained defect generation network CUNet, and predict the distribution ε of the diffusion noise at the (T - 1)-th time step through CUNet θ (f T , T, y).

[0037] S102. Randomly sample the diffusion noise at the previous time step based on the distribution of the perturbation noise at the previous time step; and calculate the noise feature at the previous time step according to the diffusion noise at the previous time step and the noise feature at the current time step; take the previous time step as the current time step, and repeat the above operations until the noise feature at the 0-th time step is calculated.

[0038] In this step, the electronic device can randomly sample the diffusion noise at the previous time step based on the distribution ε θ (f T , T, y) of the perturbation noise at the previous time step; and calculate the noise feature f T at the previous time step according to the diffusion noise at the previous time step and the noise feature f T-1 at the current time step; take the previous time step as the current time step, and repeat the above operations until the noise feature f0 at the 0-th time step is calculated. Specifically, the noise feature at the (T - 1)-th time step is: Among them, the Gaussian parameter β t linearly increases within the range of (0.0001, 0.002) during the diffusion of t steps, indicating that the noise amount of each step increases linearly. At the same time, the Gaussian parameter α t = 1 - β t ;

[0039] S103. Generate the local defect image corresponding to the target defect based on the noise feature at the 0-th time step.

[0040] In this step, the electronic device can generate a local defect image corresponding to the target defect based on the noise feature f0 at the 0th time step. Specifically, through the above loop recurrence, the noise feature f0 at the 0th time step is obtained, and f0 is input into the decoder D of the pre-trained hidden layer diffusion model, and the generated local defect image can be obtained, which is represented as 512×512×3 in terms of width, height and number of channels.

[0041] S104. Input the local defect image into the pre-trained defect fusion network, and generate a defect sample corresponding to the target defect through the defect fusion network.

[0042] In this step, the electronic device can input the local defect image into the pre-trained defect fusion network, and generate a defect sample corresponding to the target defect through the defect fusion network. That is, input the imperfect defect sample into the pre-trained defect fusion network, and a highly realistic defect sample can be generated.

[0043] The method for generating defective samples proposed in the embodiments of the present application first randomly samples standard Gaussian noise, uses the distribution of the randomly sampled noise as the noise feature at the current time step, and inputs the noise feature at the current time step and the semantic label corresponding to the target defect obtained in advance into a pre-trained defect generation network. The distribution of the diffusion noise at the previous time step of the current time step is predicted through the pre-trained defect generation network; then, the diffusion noise at the previous time step is randomly sampled based on the distribution of the perturbation noise at the previous time step; and based on the diffusion noise at the previous time step and the noise feature at the current time step, the noise feature at the previous time step of the current time step is calculated; taking the previous time step as the current time step, the above operations are repeatedly executed until the noise feature at the 0th time step is calculated; then, a local defective image corresponding to the target defect is generated based on the noise feature at the 0th time step; finally, the local defective image is input into a pre-trained defect fusion network, and a defective sample corresponding to the target defect is generated through the defect fusion network. That is to say, in the technical solution of the present application, the method for generating defective samples is based on multiple stages. In the first stage, a local defective image is generated through a defect generation network; in the second stage, a highly realistic defective sample is generated through a defect fusion network. And the present application gradually diffuses the sample from a random state to a real sample space through a reverse diffusion process. The present application has the following advantages compared with traditional generation models: the diffusion operation is reversible, and the sample generation process can be precisely controlled, so as to generate high-quality samples. In the prior art, since the defective part occupies a small area of the sample, it is difficult for the model to focus on the generation of defects. Therefore, it is difficult to directly generate defective samples based on the generation model, and the network may pay more attention to the generation of the sample background. Therefore, compared with the prior art, the method for generating defective samples proposed in the embodiments of the present application can generate highly realistic local defects and naturally integrate them into normal samples; moreover, the technical solution of the embodiments of the present application is simple and convenient to implement, easy to popularize, and has a wider scope of application.

[0044] Embodiment 2

[0045] Figure 2 It is the first process schematic diagram of the training method of the defect generation network provided by the embodiments of the present application. Based on the above technical solution, it is further optimized and extended, and can be combined with each of the above optional implementation manners. As Figure 2 shown, the training method of the defect generation network may include the following steps:

[0046] S201. If the defect generation network does not meet the first convergence condition set in advance, extract a predetermined number of defective samples from the pre-constructed local defect set as the training samples for the current batch.

[0047] In this step, if the defect generation network does not meet the pre-set first convergence condition, the electronic device can extract a predetermined number of defect samples from the pre-constructed local defect set I defect as the training samples for the current batch. Specifically, the electronic device can randomly sample 16 training samples in the pre-constructed local defect set I defect i.e., batch size = 16, and the subscript 0 indicates the initial state. To increase the robustness of the model, the diffusion step size t of each sample in each batch is randomly sampled ∈ Uniform({1, 2, …, T}); where T is the predefined maximum time step of 200; the diffusion step size of each sample is not fixed. (b)

[0048] S202. For each training sample in the current batch of training samples, randomly select a step size from the pre-constructed set of diffusion step sizes as the step size corresponding to the training sample.

[0049] S203. Use the current batch of training samples and the step sizes corresponding to each training sample to train the defect generation network, and repeat the above operations until the defect generation network meets the first convergence condition.

[0050] In this step, the electronic device can use the current batch of training samples and the step sizes corresponding to each training sample to train the defect generation network, and repeat the above operations until the defect generation network meets the first convergence condition. The defect generation network in the embodiments of the present application can be a pre-trained hidden layer diffusion model. Figure 3 It is a schematic diagram of the principle of the training method of the defect generation network provided by the example of the present application. As Figure 3 shown, the training method of the defect generation network can include the following steps:

[0051] Step 1: Define the maximum time step of diffusion T = 200, and the Gaussian parameter β t increases linearly during diffusion for t steps within the range of (0.0001, 0.002), indicating that the noise amount of each step increases linearly. At the same time, the Gaussian parameter α t = 1 - β t ;

[0052] Step 2: Randomly sample 16 training samples from the pre-constructed local defect set I defect i.e., batch size = 16, and the subscript 0 indicates the initial state. To increase the robustness of the model, the diffusion step size t of each sample in each batch is randomly sampled (b) ​​∈ Uniform({1, 2, …, T}); where T is the predefined maximum time step of 200; the step size of each sample diffusion is not fixed.

[0053] Step 3: Input the training samples of the current batch into the encoder E of the pre-trained hidden layer expansion model, and obtain the hidden layer features corresponding to each training sample through the encoder E It is represented as having a width, height, and number of channels of 64×64×3. For the standard Gaussian noise ε (b) ~ N(0, 1) is randomly sampled and represented as having a width, height, and number of channels of 64×64×3. Then the noise feature at the t-th step is:

[0054] Step 4: Input f t (b) and the corresponding semantic label y (b) into CUNet, and predict the noise distribution at the t-th step through the CUNet network as: where θ is the parameter of the CUNet network. The loss function adopted by the CUNet network is:

[0055] Step 5: Continuously iterate the operations in Steps 2 to 4 above. During each iteration, optimize the CUNet network based on the minimized loss function L, and update the parameter θ of the CUNet network using the Adam optimization method in each iteration process until the model converges.

[0056] In the training method of the defect generation network proposed in the embodiments of the present application, if the defect generation network does not meet the first convergence condition set in advance, first extract a predetermined number of defect samples from the pre-constructed local defect set as the training samples for the current batch; then for each training sample in the training samples of the current batch, randomly select a step size from the pre-constructed set of diffusion step sizes as the step size corresponding to the training sample; then use the training samples of the current batch and the step sizes corresponding to each training sample to train the defect generation network, and repeat the above operations until the defect generation network meets the first convergence condition. That is to say, the present application can batch-collect a certain number of defect samples in the pre-constructed local defect set, and use the training samples of each batch and the step sizes corresponding to each training sample to train the defect generation network. The present application has the following advantages compared with the traditional generation model: the diffusion operation is reversible, and the generation process of the samples can be precisely controlled, so as to generate high-quality samples. In the prior art, since the defective part accounts for a small area of the sample, it is difficult for the model to focus on the generation of defects. Therefore, it is difficult to directly generate defect samples based on the generation model, and the network may pay more attention to the generation of the sample background. Therefore, compared with the prior art, the method for generating defect samples proposed in the embodiments of the present application can generate highly realistic local defects and naturally integrate them into normal samples; moreover, the technical solution of the embodiments of the present application is simple and convenient to implement, easy to popularize, and has a wider scope of application.

[0057] Embodiment III

[0058] Figure 4 It is the second process schematic diagram of the training method of the defect generation network provided by the embodiments of the present application. Based on the above technical solutions, it is further optimized and extended, and can be combined with the above various optional implementation manners. As Figure 4 shown, the training method of the defect generation network may include the following steps:

[0059] S401. Annotate each defect sample in the pre-constructed real defect sample set, and cut out the real defects in the defect sample set to form the original local defect set.

[0060] In this step, the electronic device can annotate each defect sample in the pre-constructed real defect sample set and cut out the real defects in the defect sample set to form the original local defect set Specifically, the electronic device can annotate each defect sample in the real defect sample set and cut out the real defects to form the original local defect set The cropped defect image is represented as 512×512×3 in terms of width, height, and number of channels.

[0061] S402. For each defect sample in the original local defect set, use it as the first defect sample, randomly select a defect sample from the original defect sample set as the second defect sample corresponding to each first defect sample, and mix each first defect sample with its corresponding second defect sample to obtain each new virtual sample and the label corresponding to each new virtual sample.

[0062] In this step, the electronic device can mix each sample in the original local defect set with another randomly selected sample. The two sampled samples can be represented as: defect A and defect B, denoted as: x A and x B , and the labels are y A and y B . According to the Mixup technique, the new virtual sample and label are respectively: x AB and y AB ; where, x AB = λx A +(1 - λ)x B ; y AB = λy a +(1 - λ)y B ; where, λ is a random number following a Beta distribution. x AB , y AB and the original local defect set are used as the input to the pre-trained hidden layer diffusion model, denoted as I defect . This application uses a data augmentation method based on Mixup for the original local defect set to generate multiple defect and multi-class defect splicing samples, so as to improve the diversity of the matching between samples in the dataset and semantic annotations, thereby increasing the cognitive information provided by the data for different semantic annotations during the training process.

[0063] S403. Construct a local defect set based on each new virtual sample, the label corresponding to each new virtual sample, and the original local defect set.

[0064] S404. If the defect generation network does not meet the preset first convergence condition, extract a predetermined number of defect samples from the pre-constructed local defect set as the training samples for the current batch.

[0065] S405. For each training sample in the training samples of the current batch, randomly select a step size from the pre-constructed diffusion step size set as the step size corresponding to the training sample.

[0066] S406. Use the training samples of the current batch and the step sizes corresponding to each training sample to train the defect generation network, and repeat the above operations until the defect generation network meets the first convergence condition.

[0067] For the training method of the defect generation network proposed in the embodiments of the present application, if the defect generation network does not meet the preset first convergence condition, first extract a predetermined number of defect samples from the pre-constructed local defect set as the training samples of the current batch; then for each training sample in the training samples of the current batch, randomly select a step size from the pre-constructed diffusion step size set as the step size corresponding to the training sample; then use the training samples of the current batch and the step sizes corresponding to each training sample to train the defect generation network, and repeat the above operations until the defect generation network meets the first convergence condition. That is to say, the embodiments of the present application can batch collect a certain number of defect samples in the pre-constructed local defect set, and use the training samples of each batch and the step sizes corresponding to each training sample to train the defect generation network. The embodiments of the present application have the following advantages compared with the traditional generation model: the diffusion operation is reversible, and the generation process of the samples can be precisely controlled, so as to generate high-quality samples. In the prior art, since the defective part accounts for a small area of the sample, it is difficult for the model to focus on the generation of defects. Therefore, it is difficult to directly generate defect samples based on the generation model, and the network may pay more attention to the generation of the sample background. Therefore, compared with the prior art, the method for generating defect samples proposed in the embodiments of the present application can generate highly realistic local defects and naturally integrate them into normal samples; moreover, the technical solution of the embodiments of the present application is simple and convenient to implement, easy to popularize, and has a wider scope of application.

[0068] Embodiment 4

[0069] Figure 5 This is the first process schematic diagram of the training method of the defect fusion network provided by the embodiments of the present application. Based on the above technical solutions, it is further optimized and extended, and can be combined with the above various optional implementation manners. As Figure 5 shown, the training method of the defect fusion network may include the following steps:

[0070] S501. If the defect fusion network does not meet the preset second convergence condition, add the local defect image to the pre-constructed normal sample to obtain an imperfect defect sample set.

[0071] S502. Randomly select an imperfect defect sample from the imperfect defect sample set as the current training sample.

[0072] S503. Use the current training sample to train the defect fusion network, and repeat the above operations until the defect fusion network meets the second convergence condition.

[0073] In this step, when the electronic device trains the defect fusion network, if the discriminator does not meet the third convergence condition preset, the electronic device can first input the current training sample into the defect fusion network, and output the generated defect sample x through the defect fusion network fake ; then use the generated defect sample x fake to train the discriminator until the discriminator meets the third convergence condition; then train the defect fusion network based on the generated defect sample x fake and the trained discriminator.

[0074] Furthermore, when the electronic device trains the discriminator using the generated defect sample x fake , it first inputs the generated defect sample x fake into the discriminator, and calculates the sample perturbation of the generated defect sample x adv through the discriminator based on the pre-determined loss function L fake ; then determines the most real perturbation adversarial sample x in the neighborhood of the generated defect sample x according to the sample perturbation fake ; then randomly samples in the set of real defect samples adv-p to obtain a real defect sample x real ; then inputs the real defect sample x real and the most real perturbation adversarial sample x adv-p into the discriminator, and the discriminator outputs a binary prediction value; and adjusts the parameters in the discriminator according to the binary prediction value; where the target of the prediction value of the real defect sample x real is 1; the target of the prediction value of the most real perturbation adversarial sample x adv-p is 0. fake

[0075] Furthermore, when the electronic device trains the defect fusion network based on the generated defect sample x fake and the trained discriminator, it can first input the generated defect sample x dake into the trained discriminator, and calculates the sample perturbation of the generated defect sample x adv through the trained discriminator based on the pre-determined loss function L fake ; then determines the most unrealistic perturbation adversarial sample x in the neighborhood of the generated defect sample x according to the sample perturbation fake ; then determines the most unrealistic perturbation adversarial sample x adv-F ; then inputs the most unrealistic perturbation adversarial sample x adv-FInput it into the discriminator, and output a binary prediction value through the discriminator; and adjust the parameters in the defect fusion network according to the binary prediction value; wherein, the target of the binary prediction value is 1.

[0076] For the training method of the defect fusion network proposed in the embodiments of the present application, if the defect fusion network does not meet the pre-set second convergence condition, add the local defect image to the pre-constructed normal sample to obtain an imperfect defect sample set; then randomly select an imperfect defect sample from the imperfect defect sample set as the current training sample; then use the current training sample to train the defect fusion network, and repeat the above operations until the defect fusion network meets the second convergence condition. That is to say, the embodiments of the present application can first obtain imperfect defect samples based on local defect images and normal samples, and then use the imperfect defect samples to train the defect fusion network. Compared with the prior art, the method for generating defect samples proposed in the embodiments of the present application can generate highly realistic local defects and naturally integrate them into normal samples; moreover, the technical solution of the embodiments of the present application is simple and convenient to implement, easy to popularize, and has a wider scope of application.

[0077] Embodiment 5

[0078] Figure 6 It is the second process schematic diagram of the training method of the defect fusion network provided by the embodiments of the present application. Based on the above technical solutions, it is further optimized and extended, and can be combined with the above various optional implementation manners. As Figure 6 shown, the training method of the defect fusion network may include the following steps:

[0079] S601. If the defect generation network does not meet the pre-set second convergence condition, add the local defect image to the pre-constructed normal sample to obtain an imperfect defect sample set.

[0080] S602. Randomly select an imperfect defect sample from the imperfect defect sample set as the current training sample.

[0081] S603. If the discriminator does not meet the pre-set third convergence condition, input the current training sample into the defect fusion network, and output the generated defect sample through the defect fusion network.

[0082] S604. Use the generated defect sample to train the discriminator until the discriminator meets the third convergence condition.

[0083] S605. Based on the generated defect sample and the trained discriminator, train the defect fusion network, and repeat the above operations until the defect fusion network meets the second convergence condition.

[0084] Figure 7Schematic diagram of the principle of the training method of the defect fusion network provided for the examples of this application. As Figure 7 shown, the training method of the defect fusion network may include the following steps:

[0085] Step 1: Add local defect images to a pre-constructed normal sample to obtain an imperfect defect sample set Specifically, the electronic device may randomly downsample the defects generated by the defect generation network to an image with a width, height, and number of channels less than 512×512, and then paste it at any position in the normal sample to obtain an imperfect defect sample set, denoted as expressed as having a width, height, and number of channels of 512×512×3.

[0086] Step 2: Randomly sample training samples from and input them into the defect fusion network UNetResNet34, denoted as F, to obtain the generated defect samples

[0087] Step 3: Taking FGSM as an example, according to the discriminator P, for the generated defect sample x fake of the loss function L adv = logP(x fake ), calculate the sample perturbation of the generated defect sample x fake

[0088] Step 4: In the neighborhood of the generated defect sample, find the perturbation adversarial sample x adv-p ; where ∈ is the perturbation factor, and its value can be 0.1; sign(·) represents the sign function.

[0089] Step 5: Randomly sample from the real defect sample set to obtain a real defect sample x real , and input x adv-p and x real into the discriminator FCN, denoted as P, to output a binary prediction value, and their corresponding labels are 0 and 1 respectively. Among them, the loss function of the discriminator is as follows:

[0090] Step 6: Fix the discriminator, recalculate the sample perturbation, and in the neighborhood of the generated defect sample, find the perturbation adversarial sample x adv-F ; Optimize the defect fusion network, and the loss function of the defect fusion network is as follows:

[0091] Step 7: Continuously iterate Steps 2 to 6 to train the defect fusion network and the discriminator. In each iteration process, use the Adam optimization method to update the network parameters until the model converges.

[0092] For the training method of the defect fusion network proposed in the embodiment of the present application, if the defect fusion network does not meet the preset second convergence condition, add the local defect image to the pre-constructed normal samples to obtain an imperfect defect sample set; then randomly select an imperfect defect sample from the imperfect defect sample set as the current training sample; then use the current training sample to train the defect fusion network, and repeat the above operations until the defect fusion network meets the second convergence condition. That is to say, the embodiment of the present application can first obtain imperfect defect samples based on local defect images and normal samples, and then use the imperfect defect samples to train the defect fusion network. Compared with the prior art, the method for generating defect samples proposed in the embodiment of the present application can generate highly realistic local defects and naturally integrate them into normal samples; moreover, the technical solution of the embodiment of the present application is simple and convenient to implement, easy to popularize, and has a wider scope of application.

[0093] Embodiment 6

[0094] Figure 8 It is a schematic structural diagram of the defect sample generation device provided by the embodiment of the present application. As Figure 8 shown, the defect sample generation device includes: a prediction module 801, a calculation module 802, a generation module 803, and a fusion module 804; where

[0095] The prediction module 801 is used to randomly sample the standard Gaussian noise, and use the distribution of the randomly sampled noise as the noise feature f of the current time step T , and input the noise feature f of the current time step T and the semantic label y corresponding to the target defect obtained in advance into the pre-trained defect generation network CUNet, and predict the distribution ε of the diffusion noise of the previous time step of the current time step through the pre-trained defect generation network CUNet θ (f T , T, y);

[0096] The calculation module 802 is used to randomly sample the diffusion noise of the previous time step based on the distribution ε of the perturbation noise of the previous time step θ (f T , T, y); and calculate the noise feature f of the previous time step according to the diffusion noise of the previous time step and the noise feature f of the current time step T , calculate the noise feature f of the previous time stepT-1 ; Take the previous time step as the current time step, and repeat the above operations until the noise feature f0 of the 0th time step is calculated;

[0097] The generating module 803 is configured to generate a local defect image corresponding to the target defect based on the noise feature f0 of the 0th time step;

[0098] The fusion module 804 is configured to input the local defect image into a pre-trained defect fusion network, and generate a defect sample corresponding to the target defect through the defect fusion network.

[0099] The above-described defect sample generating device can execute the method provided in any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference can be made to the defect sample generating method provided in any embodiment of the present application.

[0100] Embodiment Seven

[0101] Figure 9 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. Figure 9 It shows a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present application. Figure 9 The displayed electronic device 12 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0102] As Figure 9 shown, the electronic device 12 is presented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0103] The bus 18 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the multiple bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0104] The electronic device 12 typically includes a variety of computer system-readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0105] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be used for reading and writing on non-removable, non-volatile magnetic media ( Figure 9 not shown, commonly referred to as a "hard disk drive"). Although Figure 9 not shown in, a disk drive for reading and writing on removable non-volatile disks (such as a "floppy disk"), and an optical disk drive for reading and writing on removable non-volatile optical disks (such as CD-ROM, DVD-ROM or other optical media) can be provided. In these cases, each drive can be connected to the bus 18 through one or more data media interfaces. The memory 28 may include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the embodiments of the present application.

[0106] A program / utilities 40 having a set (at least one) of program modules 42 can be stored, for example, in the memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples. The program modules 42 generally perform the functions and / or methods in the embodiments described in the present application.

[0107] The electronic device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the electronic device 12 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through the bus 18. It should be understood that although Figure 9 not shown in, other hardware and / or software modules can be used in combination with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0108] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, implementing the method for generating defective samples provided in the embodiments of the present application.

[0109] Embodiment VIII

[0110] The embodiments of the present application provide a computer storage medium.

[0111] The computer-readable storage medium of the embodiments of the present application may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0112] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0113] The program code contained on the computer-readable medium may be transmitted by any appropriate medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0114] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0115] Note that the above is only a preferred embodiment of this application and the technical principles applied. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of this application. Therefore, although this application has been described in more detail through the above embodiments, this application is not limited to the above embodiments. Without departing from the concept of this application, more other equivalent embodiments can be included, and the scope of this application is determined by the scope of the appended claims.

Claims

1. A method for generating defective samples, characterized in that, The method includes: If the defect generation network CUNet does not meet the pre-set first convergence condition, a predetermined number of defect samples are extracted from the pre-constructed local defect set as the training samples for the current batch; for each training sample in the training samples of the current batch, a step size is randomly selected from the pre-constructed set of diffusion step sizes as the step size corresponding to the training sample; the defect generation network CUNet is trained using the training samples of the current batch and the step sizes corresponding to the respective training samples, and the above operations are repeated until the defect generation network meets the first convergence condition; If the defect fusion network does not meet the pre-set second convergence condition, add the local defect image to the pre-constructed normal samples to obtain an imperfect defect sample set ; In the imperfect defect sample set randomly select an imperfect defect sample as the current training sample; use the current training sample to train the defect fusion network, and repeat the above operations until the defect fusion network meets the second convergence condition; Randomly sample from standard Gaussian noise, and use the distribution of the randomly sampled noise as the noise feature at the current time step , and use the noise feature at the current time step and the semantic label y corresponding to the pre-acquired target defect as inputs to the pre-trained defect generation network CUNet, and predict the distribution of the diffusion noise at the previous time step of the current time step through the pre-trained defect generation network CUNet ; Based on the distribution of the perturbation noise in the previous time step Randomly sample the diffusion noise in the previous time step; and based on the diffusion noise in the previous time step and the noise characteristics of the current time step , calculate the noise characteristics of the previous time step ; Take the previous time step as the current time step, and repeat the above operations until the noise characteristics of the 0th time step are calculated ; Noise features based on the 0th time step Generate a local defect image corresponding to the target defect; Inputting the local defect image into a pre-trained defect fusion network, and generating a defect sample corresponding to the target defect through the defect fusion network.

2. The method according to claim 1, wherein From a pre-built local defect set Before extracting a predetermined number of defect samples as training samples for the current batch, the method further includes: For each defect sample in the pre-constructed real defect sample set perform annotation, and cut out the real defects in the defect sample set to form the original local defect set ; Take each defect sample in the original local defect set as the first defect sample, and randomly select a defect sample from the original defect sample set as the second defect sample corresponding to each first defect sample, and mix each first defect sample with its corresponding second defect sample to obtain each new virtual sample and the label corresponding to each new virtual sample; Based on each new virtual sample, the label corresponding to each new virtual sample, and the original local defect set Construct the local defect set .

3. The method according to claim 1, characterized in that, Training the defect fusion network using the current training sample, including: If the discriminator does not meet the preset third convergence condition, the current training sample is input into the defect fusion network, and a defect sample generated by the defect fusion network is output ; Using the generated defect samples Train the discriminator until the discriminator meets the third convergence condition; Based on the generated defect samples and the trained discriminator are used to train the defect fusion network.

4. The method according to claim 3, characterized in that Using the generated defect samples Training the discriminator, including: Input the generated defect samples into the discriminator, and calculate the sample perturbation of the generated defect samples by the discriminator based on a pre-determined loss function ; ; According to the sample perturbation Determine the most authentic perturbation adversarial sample in the neighborhood of the generated defective sample ; ; Random sampling is carried out in the real defect sample set to obtain a real defect sample ; Input the real defect sample and the most real perturbation adversarial sample into the discriminator, and output a binary prediction value through the discriminator; and adjust the parameters in the discriminator according to the binary prediction value; wherein, the target of the prediction value of the real defect sample is 1; the target of the prediction value of the most real perturbation adversarial sample is 0.

5. The method according to claim 4, characterized in that, Based on the generated defect samples and the trained discriminator to train the defect fusion network, including: Input the generated defect samples into the trained discriminator, and calculate the sample perturbation of the generated defect samples by the trained discriminator based on a pre-determined loss function ; ; According to the sample perturbation In the generated defective sample Determine the least realistic perturbation adversarial sample in the neighborhood of ; Input the least realistic adversarial perturbation sample into the discriminator, and output a binary prediction value through the discriminator; and adjust the parameters in the defect fusion network according to the binary prediction value; wherein, the target of the binary prediction value is 1.

6. A generation device for defective samples, characterized in that The device includes: a prediction module, a calculation module, a generation module, and a fusion module; wherein, The prediction module is used to, if the defect generation network CUNet does not meet the preset first convergence condition, extract a predetermined number of defect samples from the pre-constructed local defect set as the training samples for the current batch; for each training sample in the training samples of the current batch, randomly select a step size from the pre-constructed set of diffusion step sizes as the step size corresponding to the training sample; use the training samples of the current batch and the step sizes corresponding to each training sample to train the defect generation network CUNet, and repeat the above operations until the defect generation network meets the first convergence condition; if the defect fusion network does not meet the preset second convergence condition, add the local defect images to the pre-constructed normal samples to obtain an imperfect defect sample set ; randomly select an imperfect defect sample from the imperfect defect sample set as the current training sample; use the current training sample to train the defect fusion network, and repeat the above operations until the defect fusion network meets the second convergence condition; randomly sample the standard Gaussian noise, and use the distribution of the randomly sampled noise as the noise feature at the current time step , and input the noise feature at the current time step and the semantic label y corresponding to the pre-obtained target defect into the pre-trained defect generation network CUNet, and predict the distribution of the diffusion noise at the previous time step of the current time step through the pre-trained defect generation network CUNet ; The calculation module is used to, based on the distribution of the perturbation noise at the previous time step randomly sample to obtain the diffusion noise at the previous time step; and according to the diffusion noise at the previous time step and the noise characteristics at the current time step , calculate the noise characteristics at the previous time step ; take the previous time step as the current time step, and repeat the above operations until the noise characteristics at the 0th time step are calculated ; The generation module is used to generate a local defect image corresponding to the target defect based on the noise characteristics at the 0th time step ​ The fusion module is configured to input the local defect image into a pre-trained defect fusion network, and generate a defect sample corresponding to the target defect through the defect fusion network.

7. An electronic device, characterized in that, Including: One or more processors; A memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating a defect sample according to any one of claims 1 to 5.

8. A storage medium, on which a computer program is stored, characterized in that, When the program is executed by a processor, it implements the method for generating a defect sample according to any one of claims 1 to 5.

Images