Defect image generation method and device, electronic equipment and storage medium

Abstract

The application provides a defect image generation method and device, equipment and a storage medium, comprising: obtaining an initial image, a text condition and a defect mask for a target product; wherein the text condition is used to represent the category of the defect to be generated, and the defect mask is used to control the generation position of the defect to be generated in the initial image; the initial image is encoded and added with noise to obtain an initial latent variable at the current time step; the initial latent variable, the text condition and the defect mask are input into a diffusion model to obtain a first noise prediction value and a second noise prediction value; obtaining a first intensity parameter and a second intensity parameter for the defect to be generated; based on the first noise prediction value, the second noise prediction value, the first intensity parameter and the second intensity parameter, a defect image is obtained. The position and intensity of the defect can be controlled, and a diversified industrial defect image dataset is constructed, thereby providing high-quality training samples for subsequent defect detection and identification.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for generating defective images. Background Technology

[0002] In the field of industrial product manufacturing and quality inspection, acquiring defect sample data is typically costly and limited in quantity. This is especially true for certain low-incidence or novel defects, where only a small number of labeled defect images are often obtained. This limited sample size severely restricts the training performance of deep learning-based defect detection and recognition models, resulting in insufficient model generalization ability. To alleviate the sample shortage problem, existing technologies typically employ data augmentation or generative model-based synthesis methods to expand the defect sample. However, these methods often struggle to simultaneously and precisely control both the location and intensity of defects, leading to insufficient controllability and interpretability of the generated results, thus limiting their application value in industrial scenarios. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and storage medium for generating defective images, in order to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this application, a method for generating a defect image is provided, the method comprising: Obtain an initial image, text conditions, and a defect mask for the target product; wherein the text conditions are used to characterize the category of the defect to be generated, and the defect mask is used to control the generation position of the defect to be generated in the initial image; The initial image is encoded and noise is added to obtain the initial latent variables at the current time step; The initial latent variables, text conditions, and defect mask are input into the diffusion model to obtain the first noise prediction value and the second noise prediction value. Obtain a first intensity parameter and a second intensity parameter for the defect to be generated; wherein, the first intensity parameter is used to control the degree of consistency between the defect to be generated and the text conditions and the defect mask, and the second intensity parameter is used to control the defect intensity of the defect to be generated; Based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter, a defect image is obtained; the defect image is used to construct a defect dataset, and the defect dataset is used for defect detection and identification.

[0005] In one embodiment, the diffusion model is obtained by training a model to be trained, the model to be trained including a backbone network and a first branch, the first branch being embedded in an inner layer of the backbone network; obtaining the diffusion model includes: Obtain defect sample images and their corresponding defect text; The defective sample image is encoded and noise is added to obtain the sample latent variables at the current time step; Input the latent variables of the sample and their corresponding defective text into the model to be trained, freeze all parameters of the backbone network, train the first branch of the model to be trained based on the first loss function, and take the first branch with the minimum first loss function as the trained first branch. Based on the trained first branch and backbone network, a diffusion model is obtained.

[0006] In one embodiment, the model to be trained further includes a second branch, which is attached to the outside of the backbone network; the process of obtaining the diffusion model based on the trained first branch and the backbone network includes: Obtain the defect mask corresponding to the defect sample image; The backbone network and the trained first branch are used as the fused backbone network; all parameters of the fused backbone network are frozen, and the defect mask and sample latent variables are input into the second branch to obtain the branch residual; The branch residuals are injected into the backbone network. Based on the second loss function, the second branch of the model to be trained is trained, and the second branch that minimizes the second loss function is taken as the trained second branch. The fused backbone network and the trained second branch are used as a diffusion model.

[0007] In one possible implementation, the step of inputting the initial latent variables, text conditions, and defect mask into the diffusion model to obtain a first noise prediction value and a second noise prediction value includes: The initial latent variables and text conditions are input into the fusion backbone network of the diffusion model, and the defect mask is input into the second branch of the diffusion model to obtain the first noise prediction value; The text conditions and defect mask are set to null values. The initial latent variables and the null text conditions are input into the fusion backbone network of the diffusion model, and the null defect mask is input into the second branch of the diffusion model to obtain the second noise prediction value.

[0008] In one possible implementation, obtaining the defect image based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter includes: The directional guidance value is obtained by subtracting the first noise prediction value from the second noise prediction value. The defect mask is downsampled to obtain a latent space mask; Based on the second intensity parameter and the latent space mask, the spatial weighted tensor is obtained; Based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter, a defect image is obtained.

[0009] In one possible implementation, obtaining the defect image based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter includes: Based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter, the conditional correction amount is obtained; The second noise prediction value is summed with the conditional correction value to obtain the target noise prediction value at the current time step. Based on the target noise prediction value at the current time step and the initial latent variables at the current time step, the latent variables of the initial image at the previous time step are obtained; The defect image is obtained based on the latent variables of the initial image at the previous time step.

[0010] In one possible implementation, obtaining the defect image based on the latent variables of the initial image at the previous time step includes: The latent variables, text conditions, and defect mask of the initial image at the previous time step are input into the diffusion model to obtain the third and fourth noise prediction values. Based on the third noise prediction value, the fourth noise prediction value, the first intensity parameter, and the second intensity parameter, the latent variables of the initial image at an earlier time step are obtained. The process is repeated in sequence, inputting the latent variables of the initial image at an earlier time step, the text conditions, and the defect mask into the diffusion model until the latent variables of the initial image at a time step of zero are obtained; The defect image is obtained based on the latent variables of the initial image at time step zero.

[0011] In one possible implementation, obtaining the defect image based on the latent variables of the initial image at time step zero includes: The latent variables of the initial image at time step zero are decoded to obtain the defect image.

[0012] According to a second aspect of this application, a defect image generation apparatus is provided, the apparatus comprising: The first acquisition unit is used to acquire an initial image, text conditions, and a defect mask for the target product; wherein, the text conditions are used to characterize the category of the defect to be generated, and the defect mask is used to control the generation position of the defect to be generated in the initial image; The second acquisition unit is used to encode and add noise to the initial image to obtain the initial latent variables at the current time step; The third acquisition unit is used to input the initial latent variables, text conditions and defect mask into the diffusion model to obtain the first noise prediction value and the second noise prediction value. The fourth acquisition unit is used to acquire a first intensity parameter and a second intensity parameter for the defect to be generated; wherein, the first intensity parameter is used to control the consistency between the defect to be generated and the text conditions and the defect mask, and the second intensity parameter is used to control the defect intensity of the defect to be generated. The fifth acquisition unit is used to obtain a defect image based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter; the defect image is used to construct a defect dataset, and the defect dataset is used for defect detection and identification.

[0013] In one embodiment, the diffusion model is obtained by training a model to be trained. The model to be trained includes a backbone network and a first branch, the first branch being embedded in an inner layer of the backbone network. The third acquisition unit is used to acquire defect sample images and their corresponding defect texts; encode and add noise to the defect sample images to obtain sample latent variables at the current time step; input the sample latent variables and their corresponding defect texts into the model to be trained, freeze all parameters of the backbone network, train the first branch of the model to be trained based on a first loss function, and take the first branch with the minimum first loss function as the trained first branch; and obtain the diffusion model based on the trained first branch and the backbone network.

[0014] In one embodiment, the model to be trained further includes a second branch, which is attached to the outside of the backbone network; the third acquisition unit is used to acquire the defect mask corresponding to the defect sample image; the backbone network and the trained first branch are used as a fused backbone network; all parameters of the fused backbone network are frozen, and the defect mask and sample latent variables are input into the second branch to obtain the branch residual; the branch residual is injected into the backbone network, and the second branch of the model to be trained is trained based on the second loss function, and the second branch with the minimum second loss function is used as the trained second branch; the fused backbone network and the trained second branch are used as a diffusion model.

[0015] In one possible implementation, the third acquisition unit is used to input the initial latent variables and text conditions into the fusion backbone network of the diffusion model, and input the defect mask into the second branch of the diffusion model to obtain a first noise prediction value; set the text conditions and defect mask to null values, input the initial latent variables and null text conditions into the fusion backbone network of the diffusion model, and input the null defect mask into the second branch of the diffusion model to obtain a second noise prediction value.

[0016] In one embodiment, the fifth acquisition unit is used to subtract the first noise prediction value from the second noise prediction value to obtain a direction guidance value; perform downsampling processing on the defect mask to obtain a latent space mask; obtain a spatial weighted tensor based on the second intensity parameter and the latent space mask; and obtain a defect image based on the spatial weighted tensor, the direction guidance value, and the first intensity parameter.

[0017] In one possible implementation, the fifth acquisition unit is configured to obtain a conditional correction amount based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter; sum the second noise prediction value and the conditional correction amount to obtain the target noise prediction value at the current time step; obtain the latent variables of the initial image at the previous time step based on the target noise prediction value at the current time step and the initial latent variables at the current time step; and obtain the defect image based on the latent variables of the initial image at the previous time step.

[0018] In one embodiment, the fifth acquisition unit is used to input the latent variables, text conditions, and defect mask of the initial image at the previous time step into the diffusion model to obtain a third noise prediction value and a fourth noise prediction value; based on the third noise prediction value, the fourth noise prediction value, the first intensity parameter, and the second intensity parameter, to obtain the latent variables of the initial image at an even earlier time step; and so on, inputting the latent variables, text conditions, and defect mask of the initial image at an even earlier time step into the diffusion model until the latent variables of the initial image at time step zero are obtained; and based on the latent variables of the initial image at time step zero, to obtain the defect image.

[0019] In one embodiment, the fifth acquisition unit is used to decode the latent variables of the initial image at a time step of zero to obtain a defect image.

[0020] According to a third aspect of this application, an electronic device is provided, comprising: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in this application.

[0021] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this application.

[0022] In this application, an initial image, text conditions, and a defect mask for a target product are obtained; wherein, the text conditions are used to characterize the category of the defect to be generated, and the defect mask is used to control the generation position of the defect to be generated in the initial image; the initial image is encoded and noise is added to obtain an initial latent variable at the current time step; the initial latent variable, text conditions, and defect mask are input into a diffusion model to obtain a first noise prediction value and a second noise prediction value; a first intensity parameter and a second intensity parameter for the defect to be generated are obtained; wherein, the first intensity parameter is used to control the consistency between the defect to be generated and the text conditions and the defect mask, and the second intensity parameter is used to control the defect intensity of the defect to be generated; based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter, a defect image is obtained; the defect image is used to construct a defect dataset, and the defect dataset is used for defect detection and recognition.

[0023] This application introduces a defect mask to achieve precise constraints on the defect generation location, ensuring a high degree of consistency between the defect region and the expected spatial location. By introducing an intensity parameter, the generation intensity of the defect can be continuously adjusted to meet the generation requirements of different defect severity levels. While maintaining the stability of the generation model, it improves the diversity and consistency of defect images, enabling the generation of high-quality industrial defect images even with a small sample size. This significantly reduces the dependence on large-scale real-world defect data and provides data support for subsequent defect detection or identification.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0025] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0026] Figure 1 A schematic diagram illustrating the implementation flow of the defect image generation method according to an embodiment of this application is shown; Figure 2 A flowchart illustrating the training process of the diffusion model according to an embodiment of this application is shown; Figure 3 A schematic diagram of the training process of the diffusion model according to an embodiment of this application is shown; Figure 4 A schematic diagram of the composition structure of the defect image generation apparatus according to an embodiment of this application is shown; Figure 5 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0027] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] This application provides a method for generating defect images, such as... Figure 1 As shown, the method includes: S101: Obtain an initial image, text conditions, and a defect mask for the target product; wherein the text conditions are used to characterize the category of the defect to be generated, and the defect mask is used to control the generation position of the defect to be generated in the initial image.

[0029] In this application, the target product is the product for which defect generation is to be performed. For example, if the goal is to generate defects for a laptop, the target product is a laptop. The initial image can be a normal image taken of the target product, i.e., an image without any defects. The text conditions are the user-specified categories of defects to be generated, such as scratches, dents, etc. The defect mask is a user-specified image corresponding to the initial image size, used to annotate the precise location and shape of defects in the image. The defect mask is a binary image, typically with a background pixel value of 0 (black) and a defect pixel value of 1 (white).

[0030] S102: Encode and add noise to the initial image to obtain the initial latent variables at the current time step.

[0031] This application utilizes a diffusion model in the process of generating defective images. This model requires repeatedly predicting noise at each step (typically 50-1000 steps). Operating in the latent space can significantly shorten training and inference time and reduce memory usage. Furthermore, the initial image space contains numerous details insensitive to humans (texture, noise, subtle lighting changes). Directly learning these details can easily lead the model into irrelevant information, resulting in unstable training or overfitting. Therefore, this application first encodes the initial image to obtain an initial latent space representation. Then, based on the principle of forward diffusion noise scheduling, Gaussian noise is progressively added to the initial latent space representation, which can preserve the semantic structure of the image (object shape, material, defect morphology, etc.) and improve the model's generalization ability. The principle of forward diffusion noise scheduling can be found in related technologies and will not be elaborated here.

[0032] S103: Input the initial latent variables, text conditions, and defect mask into the diffusion model to obtain the first noise prediction value and the second noise prediction value.

[0033] In this application, initial latent variables, text conditions, and a defect mask are input into a diffusion model. The model performs noise prediction based on the input, yielding two predicted values. The first noise prediction is conditional noise, i.e., noise prediction under the conditions of text and mask. The second noise prediction is unconditional noise, i.e., noise prediction without considering text and mask conditions. Two noise prediction values ​​are obtained because if only conditional noise is considered, the final generated defect image can strictly follow the user's text and mask conditions, but it is prone to saturation, artifacts, or reduced diversity (the model over-relies on conditions and ignores prior data). If only unconditional noise is considered, the generated image is natural and diverse, but completely disregards the user's condition control (the defect will not appear). Therefore, this application obtains two noise predictions through the diffusion model, and combining these two noise predictions to generate the defect image ensures high-quality defect image generation. The specific principles and training process of the diffusion model are explained in the relevant sections below, and will not be repeated here.

[0034] S104: Obtain a first intensity parameter and a second intensity parameter for the defect to be generated; wherein, the first intensity parameter is used to control the consistency between the defect to be generated and the text conditions and the defect mask, and the second intensity parameter is used to control the defect intensity of the defect to be generated.

[0035] In this application, the first intensity parameter and the second intensity parameter are adjustable parameters for the user. The first intensity parameter controls the degree to which the generated defect conforms to the text conditions and the defect mask; for example, the larger the first intensity parameter, the closer the defect features and defect location will be to the conditions. The second intensity parameter controls the intensity of the defect; for example, the larger the second intensity parameter, the more obvious the defect; the smaller the second intensity parameter, the less obvious the defect. This embodiment allows for flexible and diverse control of defect generation through the first and second intensity parameters.

[0036] S105: Based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter, a defect image is obtained; the defect image is used to construct a defect dataset, and the defect dataset is used for defect detection and identification.

[0037] In this application, the first noise prediction value and the second noise prediction value can basically determine the generation category and location of defects in the initial image, and the first intensity parameter and the second intensity parameter can basically determine the generation intensity of defects in the initial image. The specific process of obtaining the defect image is detailed in the relevant sections below and will not be repeated here. Because users can flexibly control the location and intensity of defects, the generated defect images are diverse, enabling the construction of rich defect datasets. This overcomes the problem of low model generalization ability caused by too small a sample size in real-world scenarios, and improves the accuracy of defect detection and identification in industrial environments.

[0038] In the schemes shown in steps S101-S105, by introducing a defect-based mask, precise constraints on the defect generation location are achieved, ensuring a high degree of consistency between the defect region and the expected spatial location. The introduction of intensity parameters enables continuous adjustment of the defect generation intensity, meeting the generation requirements for different defect severity levels. While maintaining the stability of the generation model, the diversity and consistency of defect images are improved. High-quality industrial defect image generation can be achieved even with few samples, significantly reducing the dependence on large-scale real-world defect data and providing data support for subsequent defect detection or identification.

[0039] In some alternative schemes, the diffusion model is obtained by training a model to be trained, the model to be trained including a backbone network and a first branch, the first branch being embedded in an inner layer of the backbone network; obtaining the diffusion model includes: Obtain defect sample images and their corresponding defect text; The defective sample image is encoded and noise is added to obtain the sample latent variables at the current time step; Input the latent variables of the sample and their corresponding defective text into the model to be trained, freeze all parameters of the backbone network, train the first branch of the model to be trained based on the first loss function, and take the first branch with the minimum first loss function as the trained first branch. Based on the trained first branch and backbone network, a diffusion model is obtained.

[0040] In this application, the diffusion model is obtained by training a model to be trained. The model to be trained includes a backbone network and a first branch. Combined with... Figure 2 and Figure 3 As shown, in this embodiment, the first branch can be a LoRA branch, which is embedded into an inner layer of the backbone network, specifically an attention layer. The backbone network can be UNet. The LoRA branch is embedded into the key attention layer of UNet, such that the weights of the key attention layer are represented as follows:

[0041] in, This represents the current weights of the key attention layer. Let A and B be the initial weights for the key attention layer. Let A and B represent two trainable matrices of rank r in LoRA.

[0042] During training, defect sample images ( Figure 2 The defect images (in the image) and their corresponding defect text (category) are input into the backbone network. Only the A and B parameters of the LoRA branch are updated, while the remaining parameters are frozen, thus ensuring stable training of the model on small datasets. It can be understood that by adding noise to defect sample images as described above, and then inputting the latent variables of the noisy samples and the defect text into the network for training, the model can associate the appearance, texture structure, and semantic features of defects with the defect type, enabling the model to learn the features corresponding to various defect categories. However, direct training requires updating a large number of model parameters, which is prone to overfitting with few samples and has high training costs. By inserting additional LoRA parameters into the UNet attention layer, and these parameters existing in the form of low-rank decomposition, the low-rank constraint of LoRA forces the model to learn only the core differences, automatically acting as a regularization mechanism. This allows the model to generalize to new backgrounds and poses even with few samples, requiring very few parameter updates. It enables a large model to learn specific defect appearances with minimal parameter cost. Optionally, before inputting the defect sample images into the model, the defect sample images can be preprocessed uniformly, including size normalization, color space standardization, VAE compression encoding, and necessary data cleaning operations, which can save the data computation of the model and ensure the accuracy of model training.

[0043] In this embodiment, when training the first branch, the expression of the first loss function is as follows:

[0044] in, This is the loss value of the first loss function. Represents the mathematical expectation of a random variable. This represents true Gaussian noise. c represents the noise predicted by the model, and c represents the defective text. This represents the mean square error. This is the current time step. These are the latent variables of the sample at the current time step.

[0045] The first loss function characterizes the mean squared error of the difference between the model's predicted noise and the actual noise. When the value of the first loss function is minimized (within the number of prediction iterations), the model is considered to have basically converged, and the first branch training is complete. The trained first branch and backbone network can generate defects of the specified type. The above training objective is not simply to predict the noise itself, but rather a training mechanism that uses noise prediction to recover the image distribution. This allows the model to learn the appearance structure, texture information, and semantic features of the corresponding defects of the target product under limited sample conditions, thus providing a parameter basis for subsequent defect image generation.

[0046] In some alternative schemes, the model to be trained further includes a second branch, which is attached to the outside of the backbone network; the diffusion model obtained based on the trained first branch and the backbone network includes: Obtain the defect mask corresponding to the defect sample image; The backbone network and the trained first branch are used as the fused backbone network; all parameters of the fused backbone network are frozen, and the defect mask and sample latent variables are input into the second branch to obtain the branch residual; The branch residuals are injected into the backbone network. Based on the second loss function, the second branch of the model to be trained is trained, and the second branch that minimizes the second loss function is taken as the trained second branch. The fused backbone network and the trained second branch are used as a diffusion model.

[0047] In this application, since the first branch is embedded inside the backbone network, the trained first branch and the backbone network can be considered as a whole, i.e., the backbone network is fused. The second branch is an external branch of the backbone network, used to control the specific location of the model's learned defects, thereby generating defects at specified locations. Figure 2 , Figure 3 As shown, the second branch can specifically be a pre-trained Segmentation-ControlNet segmentation model. This embodiment updates only the ControlNet branch parameters, enabling the model to learn defect location constraints and achieve precise control over the defect generation location. Specifically, all parameters of the fused backbone network are frozen. The second branch takes the defect mask and sample latent variables as input, and outputs control features at the k-th network layer corresponding to each downsampling block, intermediate block, and upsampling block of the backbone network. These features are then mapped to residual form through zero-initialized convolutions.

[0048] in, This represents the branch residual of the k-th network layer. This is the defect mask corresponding to the defect sample image. This represents the control features extracted by the ControlNet branch from the latent variables at the k-th network layer. ( ) represents the zero-initialized convolution mapping function set after the k-th network layer.

[0049] Branch residuals are injected into k network layers corresponding to the backbone network, thereby guiding defect generation to concentrate in the spatial region indicated by the mask without compromising the original model's generative ability. The branch residuals are injected into the backbone network, and the second branch of the model to be trained is trained based on the second loss function, the expression of which is as follows:

[0050] in, This represents the second loss function. This represents the mathematical expectation after random sampling of the defect sample image and diffusion time step. For defect masks aligned with latent space, This represents the value of the latent space defect mask at the i-th spatial location. This represents the total number of pixels or locations corresponding to the latent space defect region, where i represents the spatial location index in the latent space feature map. This represents the actual noise value at the i-th spatial location. The first branch represents the conditional noise value predicted by the model at the i-th spatial location after fusing mask information. The second loss function characterizes the difference between the predicted noise and the actual noise under the mask condition. When the value of the second loss function is minimized (within the number of prediction iterations), the model is considered to have basically converged, and the training of the second branch is complete. The trained second branch and the fused backbone network constitute the trained diffusion model, which can generate a specified type of defect at a specified location.

[0051] Optionally, the second branch can also be trained using rare color segmentation maps and sample latent variables as input. The rare color segmentation map is obtained by mapping the defect mask to rare colors. It can be understood that during pre-training (e.g., based on an existing dataset), the second branch has already learned to map common colors (red, green, blue, etc.) to specific object categories (people, cars, sky, etc.). If common colors are directly used as the encoding for the defect mask, the model might mistakenly identify the region as an object rather than a defect. Rare colors rarely appear in the pre-training data, therefore they do not activate the original category priors, allowing the second branch to learn new mapping relationships from scratch, improving the model's robustness.

[0052] In some alternative solutions, the step of inputting the initial latent variables, text conditions, and defect mask into the diffusion model to obtain a first noise prediction value and a second noise prediction value includes: The initial latent variables and text conditions are input into the fusion backbone network of the diffusion model, and the defect mask is input into the second branch of the diffusion model to obtain the first noise prediction value; The text conditions and defect mask are set to null values. The initial latent variables and the null text conditions are input into the fusion backbone network of the diffusion model, and the null defect mask is input into the second branch of the diffusion model to obtain the second noise prediction value.

[0053] In this application, the first noise prediction value The expression is as follows:

[0054] Second noise prediction value The expression is as follows:

[0055] in, These are the initial latent variables at the current time step. For text conditions. This represents the noise prediction function for the backbone network. This indicates unconditional input. In other words, the first noise prediction is the result under the condition that both text conditions and defect masks exist simultaneously, while the second noise prediction is the result under the condition that neither text conditions nor defect masks are considered.

[0056] In some alternative solutions, obtaining the defect image based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter includes: The directional guidance value is obtained by subtracting the first noise prediction value from the second noise prediction value. The defect mask is downsampled to obtain a latent space mask; Based on the second intensity parameter and the latent space mask, the spatial weighted tensor is obtained; Based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter, a defect image is obtained.

[0057] In this application, the directional guidance value is the difference between the first noise prediction value and the second noise prediction value, used to characterize the amount of correction required to transition from unconditional to conditional. It informs the model: in the high-dimensional latent space, in order to make the desired defect appear in the image, in which direction should the noise prediction be adjusted during each denoising step? The latent space mask is obtained by downsampling the defect mask to the latent space resolution. The spatial weighted tensor w can be obtained using the following formula:

[0058] in, For latent space masking. λ is the second intensity parameter. As λ increases, the conditional guidance effect within the defect region is enhanced, resulting in a more significant defect structure; as λ decreases, the defect region gradually becomes more consistent with the background. The spatial weighting tensor does not directly act on the noise prediction result itself, but rather on the difference term between the conditional and unconditional prediction results, thus avoiding interference with the basic generation distribution.

[0059] Furthermore, in some alternative schemes, obtaining the defect image based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter includes: Based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter, the conditional correction amount is obtained; The second noise prediction value is summed with the conditional correction value to obtain the target noise prediction value at the current time step. Based on the target noise prediction value at the current time step and the initial latent variables at the current time step, the latent variables of the initial image at the previous time step are obtained; The defect image is obtained based on the latent variables of the initial image at the previous time step.

[0060] In this application, the target noise prediction value at the current time step It can be obtained through the following formula:

[0061] in, is the first strength parameter. e is the directional guidance value. This is a conditional correction amount.

[0062] After obtaining the predicted target noise value and the initial latent variables at the current time step, the latent variables of the initial image at the previous time step can be calculated based on the denoising formulas DDPM or DDIM. The calculation principles of DDPM or DDIM are explained in the relevant technical descriptions and will not be elaborated upon here.

[0063] Furthermore, in some alternative schemes, obtaining the defect image based on the latent variables of the initial image at the previous time step includes: The latent variables, text conditions, and defect mask of the initial image at the previous time step are input into the diffusion model to obtain the third and fourth noise prediction values. Based on the third noise prediction value, the fourth noise prediction value, the first intensity parameter, and the second intensity parameter, the latent variables of the initial image at an earlier time step are obtained. The process is repeated in sequence, inputting the latent variables of the initial image at an earlier time step, the text conditions, and the defect mask into the diffusion model until the latent variables of the initial image at a time step of zero are obtained; The defect image is obtained based on the latent variables of the initial image at time step zero.

[0064] Furthermore, in some alternative schemes, obtaining the defect image based on the latent variables of the initial image at time step zero includes: The latent variables of the initial image at time step zero are decoded to obtain the defect image.

[0065] Assuming the current time step is T, let T = T-1, and then use the latent variables obtained from the previous time step... The text conditions and defect mask are then input into the diffusion model again to obtain the third noise prediction value considering the conditions and the noise prediction value without considering the conditions. Based on a similar calculation process as described above, and based on the third noise prediction value, the fourth noise prediction value, the first intensity parameter, and the second intensity parameter, the latent variables of the initial image at an earlier time step are obtained, i.e. Continue the loop until the latent variable at time step zero is obtained, i.e. Since the diffusion model always operates in the latent space after perceptual compression, its final result is... This is the final output in the latent space, not a defect image visible to the human eye. It needs to be... The input decoder (a fixed decoder identical to the one used during training) maps the image from the latent space back to the pixel space, resulting in a defect image of the specified location, intensity, and category that the end user can see. The decoder can be a VAE decoder; for details on the decoding principle, please refer to the relevant technical documentation, which will not be elaborated here.

[0066] This application also provides a defect image generation apparatus, such as... Figure 4 As shown, the device includes: The first acquisition unit 401 is used to acquire an initial image, text conditions, and a defect mask for the target product; wherein, the text conditions are used to characterize the category of the defect to be generated, and the defect mask is used to control the generation position of the defect to be generated in the initial image. The second acquisition unit 402 is used to encode and add noise to the initial image to obtain the initial latent variables at the current time step; The third acquisition unit 403 is used to input the initial latent variables, text conditions and defect mask into the diffusion model to obtain the first noise prediction value and the second noise prediction value. The fourth acquisition unit 404 is used to acquire a first intensity parameter and a second intensity parameter for the defect to be generated; wherein, the first intensity parameter is used to control the consistency between the defect to be generated and the text conditions and the defect mask, and the second intensity parameter is used to control the defect intensity of the defect to be generated. The fifth acquisition unit 405 is used to obtain a defect image based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter; the defect image is used to construct a defect dataset, and the defect dataset is used for defect detection and identification.

[0067] In some alternative schemes, the diffusion model is obtained by training a model to be trained, the model to be trained including a backbone network and a first branch, the first branch being embedded in the inner layer of the backbone network; the third acquisition unit 403 is used to acquire defect sample images and their corresponding defect texts; encode and add noise to the defect sample images to obtain sample latent variables at the current time step; input the sample latent variables and their corresponding defect texts into the model to be trained, freeze all parameters of the backbone network, train the first branch of the model to be trained based on a first loss function, and take the first branch with the minimum first loss function as the trained first branch; based on the trained first branch and the backbone network, the diffusion model is obtained.

[0068] In some alternative schemes, the model to be trained further includes a second branch, which is attached to the outside of the backbone network; the third acquisition unit 403 is used to acquire the defect mask corresponding to the defect sample image; the backbone network and the trained first branch are used as a fused backbone network; all parameters of the fused backbone network are frozen, and the defect mask and sample latent variables are input into the second branch to obtain the branch residual; the branch residual is injected into the backbone network, and the second branch of the model to be trained is trained based on the second loss function, and the second branch with the minimum second loss function is used as the trained second branch; the fused backbone network and the trained second branch are used as a diffusion model.

[0069] In some alternative schemes, the third acquisition unit 403 is used to input the initial latent variables and text conditions into the fusion backbone network of the diffusion model, and input the defect mask into the second branch of the diffusion model to obtain a first noise prediction value; set the text conditions and defect mask to null values, input the initial latent variables and null text conditions into the fusion backbone network of the diffusion model, and input the null defect mask into the second branch of the diffusion model to obtain a second noise prediction value.

[0070] In some alternative schemes, the fifth acquisition unit 405 is used to subtract the first noise prediction value from the second noise prediction value to obtain a direction guidance value; perform downsampling processing on the defect mask to obtain a latent space mask; obtain a spatial weighted tensor based on the second intensity parameter and the latent space mask; and obtain a defect image based on the spatial weighted tensor, the direction guidance value, and the first intensity parameter.

[0071] In some alternative schemes, the fifth acquisition unit 405 is used to obtain a conditional correction amount based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter; sum the second noise prediction value and the conditional correction amount to obtain the target noise prediction value at the current time step; obtain the latent variables of the initial image at the previous time step based on the target noise prediction value at the current time step and the initial latent variables at the current time step; and obtain the defect image based on the latent variables of the initial image at the previous time step.

[0072] In some alternative schemes, the fifth acquisition unit 405 is used to input the latent variables, text conditions, and defect mask of the initial image at the previous time step into the diffusion model to obtain the third noise prediction value and the fourth noise prediction value; based on the third noise prediction value, the fourth noise prediction value, the first intensity parameter, and the second intensity parameter, to obtain the latent variables of the initial image at an even earlier time step; and so on, inputting the latent variables, text conditions, and defect mask of the initial image at an even earlier time step into the diffusion model until the latent variables of the initial image at time step zero are obtained; and based on the latent variables of the initial image at time step zero, to obtain the defect image.

[0073] In some alternative schemes, the fifth acquisition unit 405 is used to decode the latent variables of the initial image at a time step of zero to obtain a defect image.

[0074] It should be noted that the defect image generation device in this application embodiment solves the problem in a similar way to the aforementioned defect image generation method. Therefore, the implementation process, implementation principle, and beneficial effects of the defect image generation device can be found in the description of the implementation process, implementation principle, and beneficial effects of the aforementioned method. Repeated descriptions will not be repeated.

[0075] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0076] Figure 5 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0077] like Figure 5As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0078] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0079] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the defect image generation method. For example, in some embodiments, the defect image generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the defect image generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the defect image generation method by any other suitable means (e.g., by means of firmware).

[0080] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0082] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0083] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0084] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0085] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0086] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0087] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0088] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating defect images, characterized in that, The method includes: Obtain an initial image, text conditions, and a defect mask for the target product; wherein the text conditions are used to characterize the category of the defect to be generated, and the defect mask is used to control the generation position of the defect to be generated in the initial image; The initial image is encoded and noise is added to obtain the initial latent variables at the current time step; The initial latent variables, text conditions, and defect mask are input into the diffusion model to obtain the first noise prediction value and the second noise prediction value. Obtain a first intensity parameter and a second intensity parameter for the defect to be generated; wherein, the first intensity parameter is used to control the degree of consistency between the defect to be generated and the text conditions and the defect mask, and the second intensity parameter is used to control the defect intensity of the defect to be generated; Based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter, a defect image is obtained; the defect image is used to construct a defect dataset, and the defect dataset is used for defect detection and identification.

2. The method according to claim 1, characterized in that, The diffusion model is obtained by training the model to be trained, which includes a backbone network and a first branch, the first branch being embedded in the inner layer of the backbone network; The acquisition of the diffusion model includes: Obtain defect sample images and their corresponding defect text; The defective sample image is encoded and noise is added to obtain the sample latent variables at the current time step; Input the latent variables of the sample and their corresponding defective text into the model to be trained, freeze all parameters of the backbone network, train the first branch of the model to be trained based on the first loss function, and take the first branch with the minimum first loss function as the trained first branch. Based on the trained first branch and backbone network, a diffusion model is obtained.

3. The method according to claim 2, characterized in that, The model to be trained further includes a second branch, which is attached to the outside of the backbone network; the diffusion model obtained based on the trained first branch and the backbone network includes: Obtain the defect mask corresponding to the defect sample image; The backbone network and the trained first branch are used as the fused backbone network; all parameters of the fused backbone network are frozen, and the defect mask and sample latent variables are input into the second branch to obtain the branch residual. The branch residuals are injected into the backbone network. Based on the second loss function, the second branch of the model to be trained is trained, and the second branch with the minimum second loss function is taken as the trained second branch. The fused backbone network and the trained second branch are used as a diffusion model.

4. The method according to claim 3, characterized in that, The step of inputting the initial latent variables, text conditions, and defect mask into the diffusion model to obtain the first noise prediction value and the second noise prediction value includes: The initial latent variables and text conditions are input into the fusion backbone network of the diffusion model, and the defect mask is input into the second branch of the diffusion model to obtain the first noise prediction value; The text conditions and defect mask are set to null values. The initial latent variables and the null text conditions are input into the fusion backbone network of the diffusion model, and the null defect mask is input into the second branch of the diffusion model to obtain the second noise prediction value.

5. The method according to any one of claims 1 to 4, characterized in that, The process of obtaining a defect image based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter includes: The directional guidance value is obtained by subtracting the first noise prediction value from the second noise prediction value. The defect mask is downsampled to obtain a latent space mask; Based on the second intensity parameter and the latent space mask, the spatial weighted tensor is obtained; Based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter, a defect image is obtained.

6. The method according to claim 5, characterized in that, The process of obtaining the defect image based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter includes: Based on the spatial weighted tensor, the directional guidance value, and the first intensity parameter, the conditional correction amount is obtained; The second noise prediction value is summed with the conditional correction value to obtain the target noise prediction value at the current time step. Based on the target noise prediction value at the current time step and the initial latent variables at the current time step, the latent variables of the initial image at the previous time step are obtained; The defect image is obtained based on the latent variables of the initial image at the previous time step.

7. The method according to claim 6, characterized in that, The process of obtaining the defect image based on the latent variables of the initial image at the previous time step includes: The latent variables, text conditions, and defect mask of the initial image at the previous time step are input into the diffusion model to obtain the third and fourth noise prediction values. Based on the third noise prediction value, the fourth noise prediction value, the first intensity parameter, and the second intensity parameter, the latent variables of the initial image at an earlier time step are obtained. The process is repeated in sequence, inputting the latent variables of the initial image at an earlier time step, the text conditions, and the defect mask into the diffusion model until the latent variables of the initial image at a time step of zero are obtained; The defect image is obtained based on the latent variables of the initial image at time step zero.

8. A defect image generation apparatus, characterized in that, The device includes: The first acquisition unit is used to acquire an initial image, text conditions, and a defect mask for the target product; wherein, the text conditions are used to characterize the category of the defect to be generated, and the defect mask is used to control the generation position of the defect to be generated in the initial image; The second acquisition unit is used to encode and add noise to the initial image to obtain the initial latent variables at the current time step; The third acquisition unit is used to input the initial latent variables, text conditions and defect mask into the diffusion model to obtain the first noise prediction value and the second noise prediction value. The fourth acquisition unit is used to acquire a first intensity parameter and a second intensity parameter for the defect to be generated; wherein, the first intensity parameter is used to control the degree of consistency between the defect to be generated and the text conditions and the defect mask, and the second intensity parameter is used to control the defect intensity of the defect to be generated. The fifth acquisition unit is used to obtain a defect image based on the first noise prediction value, the second noise prediction value, the first intensity parameter, and the second intensity parameter; the defect image is used to construct a defect dataset, and the defect dataset is used for defect detection and identification.

9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images