Industrial defect sample generation method and device, electronic equipment and storage medium

Abstract

The invention relates to an industrial defect sample generation method, which comprises the following steps of: processing a defect sample image to obtain condition information; cutting out a to-be-generated defect area from the normal sample, and carrying out image processing to obtain a target generation area; combining the target generation area and the condition information to generate a defect generation graph; and combining the defect generation graph with the cutting area of the normal sample to obtain a final defect image. According to the industrial defect sample generation method provided by the invention, local defect generation is realized, and an image instance enables a diffusion model to pay more attention to the defect instead of background generation. And meanwhile, the diffusion model is generated based on the background mask, so that the generated defect graph is consistent with the background of a normal sample, and the influence of a chromatic aberration problem caused by background inconsistency is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device, electronic equipment and storage medium for generating industrial defect samples. Background Art

[0002] With the development of artificial intelligence, target detection algorithms based on deep neural networks have begun to be widely used in defect detection, but target detection requires a large number of data sets as a basis to improve the accuracy of defect detection. In industrial scenarios, there are often too few defect samples to effectively build defect detection algorithm models such as machine learning and deep learning. In response to this situation, the traditional method is to use artificially created defect samples and use data enhancement methods to increase the number of defect samples, but artificially created defect samples will increase costs and cannot replicate naturally occurring defect features. Even through data enhancement, the diversity of the defect samples themselves cannot be increased, and thus the accuracy of the algorithm cannot be improved.

[0003] Based on this, the existing technology generates defect images through generative adversarial networks (GAN), thereby solving the problem of too few defect samples. However, the defect images generated by generative adversarial networks (GAN) are relatively simple. Generating diverse defect images requires a variety of different defective products as training, which reduces the efficiency of obtaining samples. At the same time, the general generative adversarial network (GAN) generates defects by generating the entire image, which easily leads to unstable defect positions, that is, the defect position and defect type do not match.

[0004] The diffusion model is a generative image algorithm model proposed in recent years. It can generate images of various styles and with text description features through text description. Its basic idea is to systematically perturb the distribution in the data through the forward diffusion step, and then restore the distribution of the data by learning the reverse step process, thereby obtaining a highly flexible and easy-to-calculate generative model.

[0005] However, the existing diffusion models are all generated based on general data training, without fine-tuning on this type of industrial data, resulting in their inherent deficiencies in the ability to generate industrial defects. At the same time, the existing diffusion models are also generated by generating the entire image, while defects often only occupy a small part of the image, so it is easy for the diffusion model to focus on background generation rather than defect generation. Summary of the invention

[0006] Based on this, an object of the present invention is to provide a method for generating industrial defect samples.

[0007] A method for generating industrial defect samples, comprising:

[0008] S10, processing the defect sample image to obtain condition information;

[0009] S20, cutting out the defect area to be generated from the normal sample and performing image processing to obtain the target generation area;

[0010] S30, combining the target generation area and the condition information to generate a defect generation map;

[0011] S40, combining the defect generation image with the cropped area of ​​the normal sample to obtain a final defect image.

[0012] The industrial defect sample generation method described in the present invention realizes local defect generation through a diffusion model based on image examples, and the image examples make the diffusion model focus more on defects rather than background generation. At the same time, by adding a background mask, the generated defect image is consistent with the background of the normal sample, reducing the impact of color difference caused by inconsistent background. In addition, the generation of defects by the diffusion model based on image examples depends on the example image, and there is no need to retrain the model for different scenes, which can be promoted faster on different scenes.

[0013] Furthermore, the step S10 further includes the following steps:

[0014] S11, cropping the defect area of ​​the defect sample to obtain a defect sample;

[0015] S12, extracting features from defect samples to obtain encrypted feature vectors;

[0016] S13, compressing the encrypted feature vector to obtain condition information.

[0017] Furthermore, the step 20 further includes the following steps:

[0018] S21, randomly cropping the background area of ​​the normal sample to obtain the defect area to be generated;

[0019] S22, masking the defective region to be generated to generate a defective region mask;

[0020] S23, inverting the defect area mask to obtain a defect background mask;

[0021] S24, performing a dot multiplication between the defect background mask and the defect area to be generated to obtain a target generation area.

[0022] Furthermore, step S12 extracts features of the defect samples through the CLIP model to obtain an encrypted feature vector.

[0023] The CLIP model can match and encrypt defect samples with random vectors, further strengthening the correlation between the encrypted content and the defect samples.

[0024] Furthermore, step S13 compresses the encrypted feature vector through a multi-layer perceptron to obtain conditional information.

[0025] The multi-layer perceptron quickly compresses the encrypted feature vector through a simple structure, thereby improving the compression efficiency.

[0026] Furthermore, the edge widths of the defect area mask set to zero are 10% of the length and width respectively.

[0027] The optimal size for retaining the background is 10% of the length and width of the mask, that is, the parameter limitation for minimizing the influence of background color difference while ensuring the diversity of defects.

[0028] Furthermore, the diffusion model is an image instance-based diffusion model, which predicts the target generation area at the current time T to obtain the target generation area with pixel distribution at the previous time T-1. The 0-set area in the target generation area, i.e., the black area, is combined with the conditional information to guide the prediction, and the 1-set area, i.e., the background area, predicts the pixel distribution normally; then the target generation area with pixel distribution at the previous time T-2 is repeatedly predicted until the noise map at the 0th time is predicted, i.e., the defect generation map is obtained.

[0029] Furthermore, the diffusion model based on image examples is trained by the original image X of the dataset. s , Target Spawning Area and condition information c for training; the diffusion model training step based on the image instance includes a forward diffusion step and a reverse diffusion step;

[0030] The forward diffusion step comprises:

[0031] A201: Let y0 = X s , that is, the 0th moment is the original image X s , then add variance β to y0 t Gaussian noise Get the noise image y1 at the first moment, whose conditional probability distribution is q(y1|y0), and repeat the Gaussian noise addition on the noise image at the next moment until the pure noise image y at time t t , the noise map y t The conditional probability distribution is q(y t |y t-1 ), and finally get the noise image group {y t |t∈[1,T]};

[0032] The reverse diffusion step comprises:

[0033] A301: noise map y based on condition information c tThrough a U-Net to perform image denoising, we get the noise map t at time t-1 t-1 , and the loss value gradient is obtained by error calculation;

[0034] The error is calculated by the following expression:

[0035]

[0036] in, It is represented as the condition information c and the target generation area Embedding parameter model. The target generation area The parameter model predicts the noise generation area and non-target generation area according to the conditional information c These are areas where the model normally predicts noise and does not combine with the conditional information c to predict noise.

[0037] A302: Feedback the error calculation to the diffusion model based on the image example to adjust and update the parameters.

[0038] Furthermore, the training data set is constructed by the following steps:

[0039] A101: Get any image as the original image X s ;

[0040] A102: Original image X s Crop and get the sample image X R , the example image X R A target to be generated is provided;

[0041] A103: For instance image X R Perform image enhancement to obtain several enhanced sample images X R ;

[0042] A104: Enhanced example image X R Encryption is performed through the CLIP model to obtain an encrypted feature vector;

[0043] A105: The encrypted feature vector is subjected to feature extraction by a multi-layer perceptron to obtain a compressed vector, i.e., conditional information c;

[0044] A106: According to the original image X s Generate an editable area (i.e., a defect area to be generated), represented by a binary mask m∈{0,1}, where 1 is an editable area and 0 is a non-editable area, thereby obtaining an editable mask m;

[0045] A107: construct a Bezier curve fitting for each edge of the editable mask m to obtain a mask m′ of any shape;

[0046] A108: Invert the mask m′ of any shape to obtain the background mask Right now

[0047] A109: The background mask With the original image X s Perform dot multiplication to obtain the target generation area

[0048] A110: X the original image s , Target Spawning Area And some conditional information c constitute the training data.

[0049] Furthermore, the Bezier curve fitting described in step A107 is performed by uniformly sampling 20 points on each side of the editable mask, offsetting the coordinates by 1 to 5 pixels, and finally connecting these points by straight lines, thereby obtaining a mask m′ of any shape.

[0050] A defect sample generating device comprises a defect processing unit, a background processing unit, a defect generating module and a combining module;

[0051] The defect processing unit cuts and compresses the defect sample to obtain condition information;

[0052] The background processing unit cuts out the defect area to be generated from the normal sample and performs image processing to obtain the target generation area;

[0053] The defect generation module combines the target generation area and the condition information through a diffusion model to generate a defect generation map;

[0054] The combining module combines the defect generation image with the cropped area of ​​the normal sample to obtain a final defect image.

[0055] Furthermore, the defect processing unit also includes a defect cropping module, a defect encryption module and a defect compression module;

[0056] The defect cutting module cuts the defect area of ​​the defect sample to obtain a defect sample;

[0057] The defect encryption module inputs the defect sample into the CLIP model for encryption to obtain an encrypted feature vector;

[0058] The defect compression module inputs the encrypted feature vector into a multi-layer perceptron for compression to obtain conditional information.

[0059] The present invention can train a diffusion model based on an image example through any image, thereby reducing the training cost. At the same time, the mask is drawn through the Bezier curve to increase the diversity of the diffusion model based on the image example.

[0060] An industrial defect sample generating device comprises a defect processing unit, a background processing unit, a diffusion model and a combining module;

[0061] The defect processing unit processes the defect sample image to obtain condition information;

[0062] The background processing unit cuts out the defect area to be generated from the normal sample and performs image processing to obtain the target generation area;

[0063] The diffusion model combines the target generation area and the condition information to generate a defect generation map;

[0064] The combining module combines the defect generation image with the cropped area of ​​the normal sample to obtain a final defect image.

[0065] Further, the defect processing unit includes a defect cropping module, a defect encryption module and a defect compression module;

[0066] Cutting the defect area of ​​the defect sample to obtain a defect sample;

[0067] The defect encryption module extracts features from defect samples to obtain encrypted feature vectors;

[0068] The defect compression module compresses the encrypted feature vector to obtain condition information.

[0069] Furthermore, the background processing unit includes a background clipping module, a background mask generating module, a defect background mask acquiring module and a defect background fusion module;

[0070] The background cropping module randomly crops the background area of ​​the normal sample to obtain the defect area to be generated;

[0071] The background mask generation module masks the defect area to be generated to generate a defect area mask;

[0072] The defect background mask acquisition module inverts the defect area mask to obtain a defect background mask;

[0073] The defect background fusion module performs a dot multiplication between the defect background mask and the defect area to be generated to obtain a target generation area.

[0074] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a schematic diagram of the structure of the industrial defect sample generating device of the present invention;

[0076] Figure 2 Generate a method flow chart for industrial defect samples;

[0077] Figure 3 for Figure 2 A specific implementation flow chart of the method shown;

[0078] Figure 4 Flowchart for building a training dataset for a diffusion model based on image instances;

[0079] Figure 5 It is a flow chart of the diffusion model training method based on image instances; DETAILED DESCRIPTION

[0080] In order to enable the diffusion model to focus on defect generation, the present invention refers to the way in which the diffusion model generates images through text feature guidance, and designs a method that uses defect samples as defect examples, and encrypts and compresses the defect examples into a feature text, and uses the feature text as a feature condition (i.e., semantic guidance) to guide the diffusion model to generate the desired defect image of the present invention, thereby reducing the demand for training samples of the diffusion model.

[0081] After the above analysis and design, the present invention finally proposes a method for generating industrial defect samples, and forms an industrial defect sample generating device based on the method.

[0082] Please also see Figure 1 , Figure 2 and Figure 3 ,in, Figure 1 This is a schematic diagram of the structure of the industrial defect sample generating device of the present invention. Figure 2 This is a flow chart of the method for generating industrial defect samples of the present invention. Figure 3 for Figure 2 A flowchart of a specific implementation of the method shown.

[0083] The industrial defect sample generating device comprises a defect processing unit 10, a background processing unit 20, a diffusion model 30 and a combining module 40. The defect processing unit 10 crops and compresses the defect sample to obtain condition information; at the same time, the background processing unit 20 crops the defect area to be generated from the normal sample and performs image processing to obtain the target generation area; the target generation area and the condition information are combined in the diffusion model 30 to generate a defect generation map; the combining module 40 combines the defect generation map with the cropped area of ​​the normal sample to obtain the final defect image.

[0084] Specifically, the defect processing unit 10 includes a defect cropping module 11 , a defect encryption module 12 and a defect compression module 13 .

[0085] The defect cutting module 11 is used to execute step S11: cutting the defect area of ​​the defect sample to obtain a defect sample.

[0086] Specifically, the defect sample is one or more real industrial defect samples. This step cuts out the defect area in the defect sample, and can extract defect features specifically and eliminate the influence of the background.

[0087] The defect encryption module 12 is used to execute step S12: extracting features from defect samples to obtain encrypted feature vectors.

[0088] In the present invention, a CLIP model is used to extract features from defect samples. Specifically, the CLIP model (Contrastive Language-Image Pre-Training, a pre-trained neural network model for matching images and text) extracts encrypted feature vectors from defect samples, and the encrypted feature vectors are more versatile and general than the features extracted by traditional convolutional neural networks. Accordingly, the encrypted feature vectors are more suitable for guiding the diffusion model to perform guided generation.

[0089] The defect compression module 13 is used to execute step S13: compress the encrypted feature vector to obtain condition information.

[0090] In the present invention, the encrypted feature vector is compressed by a multi-layer perceptron, and the compression is to further extract the spatial features of the encrypted feature vector and adjust the encrypted feature vector to a certain depth and width to adapt to subsequent operations. Specifically, the multi-layer perceptron includes multiple fully connected layers and pooling layers.

[0091] The background processing unit 20 includes a background clipping module 21 , a background mask generating module 22 , a defect background mask acquiring module 23 and a defect background fusion module 24 .

[0092] The background cutting module 21 is used to execute step S21: randomly cutting the background area of ​​the normal sample to obtain the defect area to be generated.

[0093] Specifically, the normal sample is one or more real industrial normal samples. The random cutting in this step is to perform random cutting on the defective area of ​​the normal sample, so as to obtain the defective area to be generated.

[0094] The background mask generating module 22 is used to execute step S22: masking the defect area to be generated to generate a defect area mask.

[0095] Specifically, a defect area mask whose values ​​are all 1 in a matrix is ​​generated according to the length and width of the defect area to be generated, and the upper, lower, left and right edges are set to 0.

[0096] Among them, the matrix of 1 and 0 can be dot-multiplied with the pixels of any image. After the dot multiplication, the 0 area in the matrix will set the pixels of the corresponding area of ​​the image to zero, that is, it will become a black area, while the 1 area in the matrix will retain the original pixels of the image after the dot multiplication.

[0097] In one embodiment, the edge width of the defect area mask set to 0 is 10% of the length and width respectively. Among them, the mask length and width are 10% to retain the optimal size of the background, that is, the parameter limitation of minimizing the influence of background color difference while ensuring the diversity of defects.

[0098] The defect background mask acquisition module 23 is used to execute step S23: invert the defect area mask to obtain a defect background mask.

[0099] Specifically, the background is retained by inverting the mask, and the defect area is set to 0, thereby obtaining a defect background mask.

[0100] The defect background mask fusion module 24 is used to execute step S24: performing a dot multiplication between the defect background mask and the defect area to be generated to obtain a target generation area.

[0101] Specifically, the target generation area is obtained by multiplying the defect background mask by the defect area to be generated, and marking the position where the defect will be generated in the defect area to be generated, that is, the target generation area.

[0102] The diffusion model 30 is used to execute step S30: combining the target generation area and the condition information to generate a defect generation map.

[0103] Specifically, the diffusion model is an image instance-based diffusion model, which predicts the target generation area at the current time T to obtain the target generation area with pixel distribution at the previous time T-1. The 0-set area in the target generation area, i.e., the black area, is combined with the conditional information to guide the prediction, and the 1-set area, i.e., the background area, predicts the pixel distribution normally; then the target generation area with pixel distribution at the previous time T-2 is repeatedly predicted until the noise map at the 0th time is predicted, i.e., the defect generation map is obtained.

[0104] The current moment T is a value set by the user, and its value can be dynamically set according to the complexity of the defect.

[0105] The combining module 40 is used to execute step S40: combining the defect generation image with the cropped area of ​​the normal sample to obtain a final defect image.

[0106] The combining operation is a pasting operation. Since the defect generation image is consistent with the cropping area size of the normal sample, the final defect image can be obtained by directly pasting them.

[0107] At the same time, in order to prevent the color difference between the background of the defective image and the background of the normal sample and to make the model adapt to amorphous defects, the present invention combines partial areas of normal samples as background images and combines training with diversified mask shapes to achieve local defect generation of the diffusion model, thereby obtaining a highly versatile diffusion model based on image instances.

[0108] In this embodiment, the diffusion model based on the image instance includes forward diffusion and reverse diffusion, and the forward diffusion is used to add noise to the image to form a noise map group. The reverse diffusion predicts the noise map through the U-Net model, and the U-Net model performs multiple downsampling operations on the image, extracts features to a low dimension, and reconstructs the image through multiple upsampling and dimensionality increase. The upsampling layer and the downsampling layer of the U-Net model are connected by an attention mechanism, and the attention mechanism enhances the effect of conditional information fusion and guides the generation of defect maps. The structure of its diffusion model is a general diffusion model structure, which is not limited here.

[0109] See also Figure 4 , Figure 4 Flowchart for building a training dataset for a diffusion model based on image instances.

[0110] The training data set of the diffusion model based on image instances is constructed by the following steps:

[0111] A101: Get any image as the original image X s .

[0112] Among them, the original image X s There are no restrictions, it can be any image.

[0113] A102: Original image X s Crop and get the sample image X R The example image X R There is a target to be generated (ie, defect).

[0114] Among them, the example image X R It can be any target you want to generate, that is, the subsequent reference to the example image X R The generated target.

[0115] A103: For instance image X R Perform image enhancement, which includes folding, rotation, blurring and deformation, to obtain several enhanced sample images X R .

[0116] The image enhancement may also include conventional image enhancement operations known to those skilled in the art, which are not limited here.

[0117] A104: Enhanced example image X R Encryption is performed through the CLIP model to obtain an encrypted feature vector.

[0118] The CLIP model encryption principle is the same as step S21 and will not be described in detail here.

[0119] A105: The encrypted feature vector is subjected to feature extraction through a multi-layer perceptron to obtain a compressed vector, i.e., the conditional information c.

[0120] Among them, based on several enhanced example images X R , by repeatedly executing steps A104 to A105, a number of condition information c can be obtained. The compression principle of the multi-layer perceptron is the same as that of step S22, and will not be repeated here.

[0121] A106: According to the original image X s Generate an editable area (i.e., a defect area to be generated), represented by a binary mask m∈{0,1}, where 1 is an editable area and 0 is a non-editable area, thereby obtaining an editable mask m.

[0122] Among them, the editable area is a customized area in any image, that is, the area where the image is generated later.

[0123] A107: Construct a Bezier curve fit for each edge of the editable mask m to obtain a mask m′ of arbitrary shape.

[0124] Specifically, the Bezier curve fitting is to uniformly sample 20 points on each side of the editable mask, and make an offset of 1 to 5 pixels on the coordinates, and finally connect these points with straight lines to obtain a mask m′ of any shape.

[0125] Among them, obtaining a mask of any shape can simulate the diversity of defect image shapes. After the model is trained, the authenticity and diversity of defect features generated by the model can be improved.

[0126] A108: Invert the mask m′ of any shape to obtain the background mask Right now

[0127] Among them, the background mask is mainly used to preserve the background, so that the background of the generated image is consistent with the original image X s Background differences.

[0128] A109: The background mask With the original image X s Perform dot multiplication to obtain the target generation area

[0129] A110: X the original image s , Target Spawning Area And some conditional information c constitute the training data.

[0130] Among them, based on a set of training data, multiple trainings can be performed through different conditional information c to learn and generate features.

[0131] It can be understood that, in this embodiment, for a plurality of different types of original images X s , A102 to A110 can be repeatedly executed until sufficient training data is obtained, and finally the training data set is constructed.

[0132] See also Figure 5 , Figure 5 Flowchart of the image instance-based diffusion model training method.

[0133] After the training data set is constructed, the diffusion model based on the image instance is trained. The diffusion model based on the image instance includes a forward diffusion step and a reverse diffusion step. The forward diffusion step includes:

[0134] A201: Let y0 = X s , that is, the 0th moment is the original image X s , then add variance β to y0 t Gaussian noise Get the noise image y1 at the first moment, whose conditional probability distribution is q(y1|y0), and repeat the Gaussian noise addition on the noise image at the next moment until the pure noise image y at time t t , the noise map y t The conditional probability distribution is q(y t |y t-1 ), and finally get the noise image group {y t |t∈[1,T]}.

[0135] Specifically, the diffusion process is expressed as follows:

[0136]

[0137] In which, since the noise addition process is in a multidimensional situation, the I unit matrix is ​​used to indicate that each dimension has the same standard deviation β t . And q(y t |y t-1 ) is a normal distribution with a mean of The variance is ∑ t=β t I.

[0138] Based on this, from y0 to y t Approximate input is performed in an operational way, which is expressed as:

[0139]

[0140] Then, we perform closed-form sampling at any time by reparameterization, letting α t =1-β t ,and as well as We can get:

[0141]

[0142]

[0143] y t =…

[0144]

[0145] Therefore, the noise image set y is generated t The expression is as follows:

[0146]

[0147] Because β t is a hyperparameter based on β t You can calculate α at all times t and Based on this, by sampling the noise at any time t, y can be obtained at one time t .

[0148] The reverse diffusion step comprises:

[0149] A301: noise map y based on condition information c t Use a U-Net to perform image denoising and obtain the noise map y at time t-1 t-1 , and the loss value gradient is obtained by error calculation.

[0150] Specifically, q(y t-1 |y t ), so that the noise graph y t Predict the noise map y t-1 , and then predict to y0 to get the generated graph.

[0151] Among them, the parameter model p approximates q(y t-1 |y t), which can be expressed as:

[0152]

[0153] In order to make the model p(y0) learn the formal data distribution q(y0), optimize the variational lower bound

[0154]

[0155]

[0156]

[0157]

[0158] At the same time, according to the parameter model according to the formula Make a prediction, that is:

[0159]

[0160] According to this, ∈ θ (y t ,t) into μ θ (y t ,t) can be derived:

[0161]

[0162] The parameterization is:

[0163]

[0164] That is, according to the above parameterized expression, the condition information c and the target generation area are introduced to obtain the error calculation expression:

[0165]

[0166] in, It is represented as the condition information c and the target generation area Embedding parameter model. The target generation area The parameter model predicts the noise generation area and non-target generation area according to the conditional information c These are areas where the model normally predicts noise and does not combine with the conditional information c to predict noise.

[0167] A302: Feedback the error calculation to the diffusion model based on the image example to adjust and update the parameters.

[0168] It can be understood that in this embodiment, for noise maps at multiple moments, steps A301 to A302 can be repeatedly executed to train the diffusion model until the prediction to the 0th moment, the generation map of y0 is obtained, or when the loss value obtained by error calculation has reached a preset range. At the same time, for multiple conditional information and target generation areas, the forward diffusion step of step A201 can be repeated, and A302 to A303 can be re-executed multiple times until the loss value is within a preset range, or the number of training times reaches a preset threshold. Ultimately, the image instance-based diffusion model learns how to better combine with the target generation area and generate more realistic images based on a variety of different types of conditional information. The image instance-based diffusion model obtained after the training is completed can generate high-quality images based on the example image.

[0169] Compared with the prior art, the present invention realizes local generation of defects through a diffusion model based on image examples. At the same time, the generation of defects by the diffusion model based on image examples depends on the example image, and there is no need to retrain the model for different scenes, so that it can be promoted to different scenes more quickly. In addition, the present invention adds a background mask to keep the generated defect image consistent with the background of the normal sample, thereby reducing the impact of color difference problems caused by inconsistent background. In addition, in order to make the diffusion model based on image instances adapt to defects of various shapes, the present invention uses Bezier curves to draw masks during the training process, thereby increasing the diversity and authenticity of defects generated by the diffusion model based on image examples. In particular, the present invention can train a diffusion model based on image examples through any image, thereby reducing training costs.

[0170] Based on the same inventive concept, the present application also provides an electronic device, which may be a terminal device such as a server, a desktop computing device or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the industrial defect sample generation method of an embodiment of the present invention; and the memory is used to store a computer program executable by the processor.

[0171] Based on the same inventive concept, the present application also provides a computer-readable storage medium, corresponding to the embodiment of the aforementioned industrial defect sample generation method, wherein the computer-readable storage medium stores a computer program thereon, and when the program is executed by a processor, the steps of the industrial defect sample generation method recorded in any of the aforementioned embodiments are implemented.

[0172] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0173] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A method for generating industrial defect samples, characterized in that: The steps include: S10, processing the defect sample image to obtain condition information; S20, cutting out the defect area to be generated from the normal sample and performing image processing to obtain the target generation area; S30, combining the target generation area and the condition information to generate a defect generation map; S40, combining the defect generation image with the cropped area of ​​the normal sample to obtain a final defect image.

2. The method according to claim 1, characterized in that The step S10 further comprises the following steps: S11, cropping the defect area of ​​the defect sample to obtain a defect sample; S12, extracting features from defect samples to obtain encrypted feature vectors; S13. Compress the encrypted feature vector to obtain conditional information.

3. The method according to claim 1, characterized in that The step 20 further comprises the following steps: S21, randomly cropping the background area of ​​the normal sample to obtain the defect area to be generated; S22, masking the defective region to be generated to generate a defective region mask; S23, inverting the defect area mask to obtain a defect background mask; S24, performing a dot multiplication between the defect background mask and the defect area to be generated to obtain a target generation area.

4. The method according to claim 2, characterized in that: Step S12 extracts features from defect samples using the CLIP model to obtain encrypted feature vectors.

5. The method according to any one of claim 4, characterized in that: Step S13 compresses the encrypted feature vector through a multi-layer perceptron to obtain conditional information.

6. The method according to claim 3, characterized in that: The edge widths of the defect area mask set to zero are 10% of the length and width respectively.

7. A method according to any one of claims 1 to 6, characterized in that: Step S30 generates a defect generation map through a diffusion model based on an image instance; the diffusion model based on an image instance predicts the target generation area at the current time T, and obtains the target generation area with pixel distribution at the previous time T-1. The 0-set area in the target generation area, i.e., the black area, guides the prediction in combination with the conditional information, and the 1-set area, i.e., the background area, predicts the pixel distribution normally; then repeat the prediction of the target generation area with pixel distribution at the previous time T-2, until the noise map at the 0th time is predicted, i.e., the defect generation map is obtained.

8. The method according to claim 7, characterized in that The diffusion model based on image examples is trained by the original image X of the dataset s , Target Spawning Area and condition information c for training; the diffusion model training step based on the image instance includes a forward diffusion step and a reverse diffusion step; The forward diffusion step comprises: A201: Let y0 = X s , that is, the 0th moment is the original image X s , then add variance β to y0 t Gaussian noise Get the noise image y1 at the first moment, whose conditional probability distribution is q(y1|y0), and repeat the Gaussian noise addition on the noise image at the next moment until the pure noise image y at time t t , the noise map y t The conditional probability distribution is q(y t |y t-1 ), and finally get the noise image group {y t |t∈[1,T]}; The reverse diffusion step comprises: A301: noise map y based on condition information c t Use a U-Net to perform image denoising and obtain the noise map y at time t-1 t-1 , and the loss value gradient is obtained by error calculation; The error is calculated by the following expression: A302: Feedback the error calculation to the diffusion model based on the image example to adjust and update the parameters.

9. The method according to claim 8, characterized in that The training dataset is constructed by the following steps: A101: Get any image as the original image X s ; A102: Original image X s Crop and get the sample image X R , the example image X R A target to be generated is provided; A103: For instance image X R Perform image enhancement to obtain several enhanced sample images X R ; A104: Enhanced example image X R Encryption is performed through the CLIP model to obtain an encrypted feature vector; A105: The encrypted feature vector is subjected to feature extraction by a multi-layer perceptron to obtain a compressed vector, i.e., conditional information c; A106: According to the original image X s Generate an editable area (i.e., a defect area to be generated), represented by a binary mask m∈{0,1}, where 1 is an editable area and 0 is a non-editable area, thereby obtaining an editable mask m; A107: construct a Bezier curve fitting for each edge of the editable mask m to obtain a mask m′ of any shape; A108: Invert the mask m′ of any shape to obtain the background mask Right now A109: The background mask With the original image X s Perform dot multiplication to obtain the target generation area A110: X the original image s , Target Spawning Area And some conditional information c constitute the training data.

10. The method according to claim 9, characterized in that The Bezier curve fitting described in step A107 is performed by uniformly sampling 20 points on each side of the editable mask, offsetting the coordinates by 1 to 5 pixels, and finally connecting these points with straight lines to obtain a mask m′ of any shape.

11. An industrial defect sample generating device, characterized in that: It includes a defect processing unit, a background processing unit, a diffusion model and a combination module; The defect processing unit processes the defect sample image to obtain condition information; The background processing unit cuts out the defect area to be generated from the normal sample and performs image processing to obtain the target generation area; The diffusion model combines the target generation area and the condition information to generate a defect generation map; The combining module combines the defect generation image with the cropped area of ​​the normal sample to obtain a final defect image.

12. A device according to claim 11, characterized in that: The defect processing unit includes a defect cropping module, a defect encryption module and a defect compression module; Cutting the defect area of ​​the defect sample to obtain a defect sample; The defect encryption module extracts features from defect samples to obtain encrypted feature vectors; The defect compression module compresses the encrypted feature vector to obtain condition information.

13. A device according to claim 11, characterized in that: The background processing unit includes a background clipping module, a background mask generating module, a defect background mask acquiring module and a defect background fusion module; The background cropping module randomly crops the background area of ​​the normal sample to obtain the defect area to be generated; The background mask generation module masks the defect area to be generated to generate a defect area mask; The defect background mask acquisition module inverts the defect area mask to obtain a defect background mask; The defect background fusion module performs a dot multiplication between the defect background mask and the defect area to be generated to obtain a target generation area.

14. A device according to claim 12, characterized in that: The defect module extracts features of defect samples through the CLIP model to obtain encrypted feature vectors.

15. A device according to claim 14, characterized in that: The defect encryption module compresses the encrypted feature vector through a multi-layer perceptron to obtain conditional information.

16. A device according to claim 13, characterized in that: The edge widths of the defect area mask set to zero are 10% of the length and width respectively.

17. A device according to any one of claims 11 to 16, characterized in that: The diffusion model is a diffusion model based on image instances, which includes forward diffusion and reverse diffusion; the forward diffusion performs Gaussian noise addition on the target generation area to obtain a noise map at the current time T; the reverse diffusion predicts the noise map at the current time T based on the condition information according to the target generation area to obtain a noise map of the noise distribution at the previous time T-1; then the noise map of the noise distribution at the previous time T-2 is repeatedly predicted until the noise map at the 0th time is predicted, that is, the defect generation map is obtained.

18. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an industrial defect sample generating method as claimed in any one of claims 1 to 10 is implemented.

19. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used in the method for generating industrial defect samples according to any one of claims 1 to 10.

Images