Defect image generation method and device, equipment and storage medium

Abstract

The invention relates to a defect image generation method and device, equipment and a storage medium, and is applied to the field of artificial intelligence, and the method comprises the steps: obtaining a pure noise image, a defect object image, an original defect image, a defect text description and a template mask image; utilizing a ControlNet model and a diffusion model to generate a defect latent space image expressed by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image; according to the defect submerged space image and the defect real object image, defect target features and background features represented by the submerged space are obtained by using the template mask image, and the defect target features and the background features are fused to generate submerged space fusion features represented by the submerged space; and processing the submerged space fusion features by adopting a diffusion model, and decoding to generate a final defect image. The method has the technical effects that the problem of long tail distribution of the industrial defect image is solved, and the quality of the defect image is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a defect image generation method, device, equipment and storage medium. Background Art

[0002] In the field of semiconductor wafers, the detection of industrial defect images is a key task. However, the unbalanced data sample distribution affects the accuracy of defect detection, especially for rare defects in small samples with a small proportion. This creates a need to generate rare defect images to enhance the accuracy of detection algorithms.

[0003] At present, the method of generating industrial defect images can usually adopt machine vision algorithms or generative adversarial networks (GANs). Traditional machine vision algorithms can generate defect images by randomly cutting a piece of pattern from an image and then pasting the cut pattern to a random position; generative adversarial networks (GANs) are a relatively mature technology for generating images, which generates new images through adversarial training of generators and discriminators.

[0004] Industrial defect images usually have a long-tail distribution problem, that is, some defect types are extremely rare. However, traditional machine vision algorithms or generative adversarial networks (GANs) algorithms require a large number of samples for training. Too few samples may result in poor quality and low diversity of generated defect images, making it difficult to solve the long-tail distribution problem in industrial defect images. Summary of the invention

[0005] In order to help solve the problem of long-tail distribution of industrial defect images, the present application provides a defect image generation method, device, equipment and storage medium.

[0006] In a first aspect, the present application provides a defect image generation method, which adopts the following technical solution: the method comprises:

[0007] Acquire a pure noise image, a defect object image, an original defect image, a defect text description, and a template mask image, wherein the defect text description corresponds to the original defect image, and the template mask image corresponds to the defect in the original defect image;

[0008] Using a ControlNet model and a diffusion model, generating a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image;

[0009] According to the defect latent space image and the defect physical image, the template mask image is used to obtain defect target features and background features represented by the latent space, and the defect target features and background features are fused to generate latent space fusion features represented by the latent space;

[0010] The diffusion model is used to process the latent space fusion features and decode them to generate a final defect image.

[0011] In a specific implementation scheme, the method of using the ControlNet model and the diffusion model to generate a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image comprises:

[0012] Using the template mask image to perform mask processing on the original defect image, and generate a defect mask image;

[0013] Training the ControlNet model according to the defect mask image, the defect text description and the template mask image, and generating a ControlNet optimization model;

[0014] The pure noise image is iteratively denoised using a diffusion model, and a defect latent space image represented by a latent space is generated based on the pure noise image using the ControlNet optimization model.

[0015] In a specific implementation scheme, the iterative denoising calculation method includes:

[0016] x t-1 =μ θ (x t , t)+σ t ∈

[0017] Among them, x t represents the noise image at time t, μ θ represents the denoising network in the diffusion model, σ t represents the noise intensity, and ∈ represents the prediction noise.

[0018] In a specific possible implementation scheme, the method of obtaining defect target features and background features represented by the latent space using the template mask image according to the defect latent space image and the defect physical image, and fusing the defect target features and background features to generate latent space fusion features represented by the latent space includes:

[0019] Performing latent space encoding on the defect object image by using an encoder function, and generating a defect object latent space image;

[0020] Resizing the template mask image according to the size of the latent space image and generating a template mask optimized image;

[0021] Using the template mask optimization image, respectively extract defect target features in the defect latent space image and background features in the defect object latent space image;

[0022] The defect target feature and the background feature are added and fused to generate the latent space fusion feature.

[0023] In a specific implementation scheme, the step of respectively extracting defect target features in the defect latent space image and background features in the defect object latent space image by using the template mask optimization image includes:

[0024] Extracting defect target features in the defect latent space image using the template mask optimization image;

[0025] Performing a negation operation on the template mask optimized image and generating a template mask negated image;

[0026] The template mask inverted image is used to extract background features in the defective object latent space image.

[0027] In a specific implementation scheme, before the diffusion model is used to process the latent space fusion features and decode to generate a final defect image, the method further includes:

[0028] Using a preset visual model to extract full-image features of the defective physical image;

[0029] Using a preset adaptation function to process the full-image feature and generate a full-image adaptation feature;

[0030] Integrating the full-image adaptation features into the diffusion model and generating a diffusion optimization model;

[0031] The step of using the diffusion model to process the latent space fusion features and decoding to generate a final defect image includes:

[0032] The diffusion optimization model is used to process the latent space fusion features and decode them to generate a final defect image.

[0033] In a specific implementation scheme, the step of using the diffusion model to process the latent space fusion features and decoding to generate a final defect image includes:

[0034] Iteratively denoising the latent space fusion features using the diffusion optimization model, and generating a defect latent space optimization image;

[0035] Using a preset decoder function to decode the defect latent space optimization image to generate the final defect image;

[0036] Among them, the calculation method of iterative denoising includes:

[0037] x t-1 =μ θ (x t , t, c)+σ t ∈

[0038] Among them, x t represents the noise image at time t, μ θ represents the denoising network in the diffusion optimization model, σ t represents the noise intensity, ∈ represents the predicted noise, and c represents the conditional information in the ControlNet model.

[0039] In a second aspect, the present application provides a defect image generating device, which adopts the following technical solution: the device comprises:

[0040] An original image acquisition module, used to acquire a pure noise image, a defect real object image, an original defect image, a defect text description, and a template mask image, wherein the defect text description corresponds to the original defect image, and the template mask image corresponds to the defect in the original defect image;

[0041] A latent space image acquisition module is used to generate a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image by using a ControlNet model and a diffusion model;

[0042] A feature fusion module, used to obtain defect target features and background features represented by the latent space according to the defect latent space image and the defect physical image using the template mask image, and fuse the defect target features and background features to generate latent space fusion features represented by the latent space;

[0043] The defect image generation module is used to process the latent space fusion features using the diffusion model and decode to generate a final defect image.

[0044] In a third aspect, the present application provides a computer device, which adopts the following technical solution: it includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute any of the above-mentioned defect image generation methods.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned defect image generation methods.

[0046] In summary, this application has the following beneficial technical effects:

[0047] By fusing the Stable Diffusion model and the ControlNet model, it is possible to quickly adapt to new defect types with very few samples, greatly reducing the reliance on large-scale annotated data, improving the adaptability and efficiency of the model, and thus effectively handling defect types with long tail distributions. Even for extremely rare defects, high-quality and diverse samples can be generated, significantly improving the ability to generate rare defects. In addition, through the technology of latent space feature fusion and secondary image generation, high-quality defect images can be quickly generated, saving time and improving the efficiency of defect image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a defect image generating method in an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of defect target generation in an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of feature acquisition and feature fusion in an embodiment of the present application;

[0051] Figure 4 is a schematic diagram of the style transfer of the prompt image in an embodiment of the present application;

[0052] Figure 5 is a schematic diagram of secondary denoising image generation in an embodiment of the present application;

[0053] Figure 6 It is a complete schematic diagram of the industrial defect image generation process in the embodiment of the present application;

[0054] Figure 7 is a schematic diagram of a defect image generating device in an embodiment of the present application;

[0055] Figure 8 It is a schematic diagram used to embody a computer device in an embodiment of the present application.

[0056] Reference numerals: 701, original image acquisition module; 702, latent space image acquisition module; 703, feature fusion module; 704, defect image generation module. DETAILED DESCRIPTION

[0057] The following combination Figure 1-Figure 8 This application is described in further detail.

[0058] An embodiment of the present application discloses a defect image generation method, which is applied to a defect image generation system. The defect image generation method can generate realistic and detail-rich defect images, while also meeting diverse needs in different scenarios, and provides a powerful data enhancement tool for the training and testing of industrial quality control and defect detection systems.

[0059] In the field of semiconductor wafers, industrial defect images are needed to complete product defect detection. However, industrial defect images have a long-tail distribution problem, that is, some types of defects may be extremely rare, which will lead to insufficient samples of rare types of defects and imbalanced data of industrial defect image samples. Therefore, it is necessary to generate rare defect images to increase data samples.

[0060] At present, the algorithms commonly used for generating industrial defect images include machine vision algorithms, generative adversarial networks (GANs) and standard diffusion models. However, the existing defect image generation methods usually also require a large number of data samples for training and learning to generate high-quality defect images. Therefore, it is difficult to solve the long-tail distribution problem of industrial defect images using the existing defect image generation methods, which may result in the final generated defect images having lower authenticity and lower diversity.

[0061] In addition to the difficulty in solving the long-tail distribution problem, the existing defect image generation method also has some drawbacks in the actual use of the algorithm itself. For example, due to the imbalance of data samples and the small sample size of rare defect images, the traditional computer vision method is difficult for computer vision algorithms to conduct effective learning and training and difficult to adapt to various defect types, which may lead to insufficient diversity and authenticity of the generated defect images; the training process of generative adversarial networks (GANs) is complex and unstable, and it is prone to mode collapse, resulting in a lack of diversity in the generated images; the text description of the standard diffusion model has limitations. For some objects that are difficult to describe in words, such as defect images in the industrial / semiconductor field, the effect of text prompts may be weakened due to the inability to accurately describe them. For images in specific fields, professional domain knowledge is required, otherwise it is difficult for users to accurately describe them in text. In order to help solve the long-tail distribution problem in industrial defect images, the present application provides a defect image generation method.

[0062] Reference Figure 1 , the method comprises the following steps:

[0063] S10, acquiring a pure noise image, a defect object image, an original defect image, a defect text description, and a template mask image, wherein the defect text description corresponds to the original defect image, and the template mask image corresponds to the defect in the original defect image.

[0064] Specifically, a pure noise image, a real defect image, an original defect image, a defect text description, and a template mask image are obtained, wherein the defect text description corresponds to the original defect image, the template mask image is a binary image with the defect area being 1 and the background area being 0, and the defect area in the template mask image corresponds to the defect target in the original defect image.

[0065] S20, using the ControlNet model and the diffusion model, generates a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image.

[0066] Specifically, the ControlNet model is an "auxiliary" neural network model structure that aims to control pre-trained large diffusion models, such as the Stable Diffusion model, to support more input conditions to control the image generation process. The ControlNet model allows the introduction of additional control signals during the generation process. In the embodiment of the present application, it can be used to guide the generation of defect targets. Structurally, the ControlNet model is usually used as a parallel branch of the StableDiffusion model backbone. The feature map generated by the ControlNet model is fused with the feature map of the model's backbone network to affect the final image generation result.

[0067] The Stable Diffusion model is a model that uses artificial intelligence technology to generate high-resolution, high-fidelity images based on text descriptions. In an embodiment of the present application, additional mask image control conditions are input into the Stable Diffusion model by combining the mask image with the ControlNet model and the diffusion model. In the process of the Stable Diffusion model generating defect images based on pure noise images, the ControlNet model is used to guide the generation of defect target features. The application of mask images combined with the ControlNet model and the diffusion model can enable the model to learn defect features more attentively without being disturbed by background information, thereby effectively dealing with small sample problems and long-tail distribution industrial defect image generation problems.

[0068] S30, according to the defect latent space image and the defect physical image, using the template mask image to obtain the defect target features and background features represented by the latent space, and fusing the defect target features and the background features to generate the latent space fusion features represented by the latent space.

[0069] Specifically, the Stable Diffusion model is implemented based on the Latent Diffusion Models, which is a simulation of the diffusion process in a low-dimensional latent space. Therefore, the Stable Diffusion model specifically acts on the low-dimensional latent space and operates in the latent space, which is more computationally efficient and can retain more semantic information. The mask image is a binary image that uses 0 and 1 to distinguish between the defect area and the background area. Therefore, in the low-dimensional latent space, the mask image can shield unnecessary information to obtain the defect target feature information and background information required by the user. After that, the defect target feature information and background information required by the user are fused, and the generated defect target can be easily combined with different backgrounds, increasing the diversity of the generated image.

[0070] S40, using a diffusion model to process the latent space fusion features and decode to generate a final defect image.

[0071] Specifically, the diffusion model (Stable Diffusion model) is used to generate a secondary image for the fusion features of the latent space obtained. Finally, the decoder function built into the Stable Diffusion model is used to decode the secondary generated latent space image to generate the final defect image required by the user. The use of the diffusion model for secondary image generation can apply additional denoising steps at the fusion boundary to eliminate possible seams, further enhance the details and authenticity of the image, and ensure that the generated defects are visually consistent with the background, thereby improving the quality of the defect image.

[0072] In this application, by fusing the Stable Diffusion model and the ControlNet model, it is possible to quickly adapt to new defect types with very few samples, greatly reducing the reliance on large-scale annotated data, improving the adaptability and efficiency of the model, and thus effectively handling defect types with long tail distributions. Even for extremely rare defects, high-quality and diverse samples can be generated, significantly improving the ability to generate rare defects. In addition, through the technology of latent space feature fusion and secondary image generation, high-quality defect images can be quickly generated, saving time and improving the efficiency of defect image generation.

[0073] In one embodiment, using the ControlNet model and the diffusion model, the step of generating a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image can be specifically performed as follows:

[0074] First, the original defect image is masked using the template mask image, that is, the background in the original defect image is shielded using the template mask image, only the information of the defect area is retained, and a defect mask image is generated, in which the defect area is 1 and the background area is 0. After that, the ControlNet model is trained according to the defect mask image, the defect text description and the template mask image, and the ControlNet optimization model is generated; specifically, the defect text description and the generated defect mask image are input into the initial ControlNet model, and the ControlNet model is trained, with the purpose of enabling the ControlNet model to learn how to generate corresponding defect features according to the defect mask image.

[0075] Finally, the diffusion model (Stable Diffusion model) is used to iteratively denoise the pure noise image. In each denoising process, the ControlNet optimization model is used to generate a defect latent space image represented by the latent space based on the pure noise image. The generated defect latent space image contains the defect target, and the defect target includes the location and type of the defect.

[0076] It should be noted that denoising generation is the core mechanism of the Stable Diffusion model. The noise is gradually removed through the Stable Diffusion model, and the characteristics of the defect target image are gradually revealed. The calculation formula of iterative denoising can be expressed as:

[0077] x t-1 =μ θ (x t , t)+σ t ·∈;

[0078] Among them, x t represents the noise image at time t, μ θ represents the denoising network in the diffusion model, σ t represents the noise intensity, and ∈ represents the prediction noise.

[0079] Schematic diagram reference for defect target generation Figure 2 As shown in FIG. 1 , the defect mask image is input into the ControlNet model for model training and learning, and the denoising model (ie, denoising network) in the diffusion model (Stable Diffusion model) is used to iteratively denoise the pure noise image. In each denoising process, the ControlNet model guides the generation of defect features according to the provided defect mask image to generate the defect latent space image Z gen .

[0080] In the present application, the scheme of combining the mask image with the ControlNet model and the Stable Diffusion model provides more accurate control of the defect position and shape during the image generation process, reduces the interference of background information on defect generation, and can enhance the generation capability of rare defect types, thereby more effectively generating high-quality and diversified industrial defect images, especially for defect types with scarce samples. In addition, the scheme can also improve the learning efficiency of the model for defect features.

[0081] In one embodiment, according to the defect latent space image and the defect object image, the steps of obtaining the defect target features and background features represented by the latent space using the template mask image, and fusing the defect target features and the background features to generate the latent space fusion features represented by the latent space can be specifically performed as follows:

[0082] First, perform latent space encoding. Use the encoder function of the Stable Diffusion model to encode the defective object image in latent space and generate the defective object latent space image. Latent space encoding can convert the image from pixel space to latent space, reduce the dimension and capture high-level features. The calculation formula of latent space encoding can be expressed as:

[0083] z t =ε(i);

[0084] Among them, ε represents the encoder function, i represents the defect image, z t represents the latent space representation.

[0085] Afterwards, considering that the size of the latent space is different from that of the original image, the template mask image needs to be adjusted. Specifically, the template mask image is resized according to the size of the latent space image, and a template mask optimized image is generated. The resizing method can use a bilinear interpolation algorithm. In practical applications, users can choose a suitable algorithm according to actual needs, and there is no restriction here.

[0086] Afterwards, the template mask optimization image is used to extract the defect target features in the defect latent space image and the background features in the defect object latent space image. Specifically, the template mask optimization image is used to extract the defect target features in the defect latent space image, and the template mask optimization image is used to shield the background feature information of the image, and only the defect target features are retained. The calculation formula for defect target feature extraction can be expressed as:

[0087] z object =z gen Mask;

[0088] Among them, Mask represents the template mask optimization image, zobject represents the defect target feature in the latent space, z gen represents a defect latent space image;

[0089] The template mask optimization image is inverted to generate a template mask inverted image. The template mask inverted image is used to extract the background features in the defect object latent space image. The calculation formula for background feature extraction can be expressed as:

[0090] z background =z real (1-Mask);

[0091] Among them, z background represents the background features in the latent space, z real Represents the latent space image of the defect object.

[0092] Finally, the defect target features and background features are added and fused to generate the latent space fusion features. The fusion calculation method can be expressed as:

[0093] z final =z background +z object ;

[0094] Among them, z final Represents the latent space fusion feature.

[0095] Schematic diagram of feature acquisition and feature fusion Figure 3 As shown, the defect object image is encoded by an encoder to generate a defect object latent space image, and then feature extraction is performed on the defect latent space image and the defect object latent space image respectively to obtain the defect target feature and background feature in the latent space, and then the features are fused to obtain the latent space fusion feature.

[0096] In the present application scheme, by acquiring the defect target features and background features in the latent space and performing feature fusion, an industrial defect image containing real background information and precisely controlled defect targets can be generated. In this scheme, the position and shape of the defect target in the final image can be precisely controlled through mask operation, and the introduction of the background of the physical image maintains the authenticity of the generated image and the consistency of the context. In addition, the generated defect target can be combined with different backgrounds to increase the diversity of the generated image, which is of great significance for enhancing the training data set of the defect detection model, solving the problem of unbalanced data distribution (long-tail distribution), and improving the accuracy of defect detection.

[0097] In one embodiment, considering that the defect image is generated only by text description in the first image generation process, however, only using text description will have certain limitations. Therefore, before using the diffusion model to process the latent space fusion features and decoding to generate the final defect image, the following steps may be performed:

[0098] First, a preset visual model is used to extract the full-image features of the defective physical image. It should be noted that the role of the defective physical image used in the embodiment of the present application is to prompt the image. Its purpose is to influence the composition, style and color of the generated image through the physical image. By using the physical image to transfer the prompt style, the complex text prompt engineering can be effectively reduced, and the image information can be directly used to more accurately guide the final image generation process; for the selection of prompt images, the defective physical image that has been used before can be reused, or a new physical image can be obtained again. In the embodiment of the present application, reuse is used as an example for explanation. In the actual processing process, the user can make a choice according to the actual situation, and there is no restriction here. The calculation method for full-image feature extraction can be expressed as:

[0099] f i =E(I p );

[0100] Among them, E represents the encoder, I p represents the defect image, f i Represents the extracted full-image features.

[0101] Afterwards, the preset adaptation function is used to process the full-image features and generate full-image adaptation features. Specifically, the extracted full-image features are processed through a series of adaptation layers to make the full-image features compatible with different levels of the diffusion model (Stable Diffusion model). The calculation method of the full-image feature adaptation processing can be expressed as:

[0102] f a =A(f i );

[0103] Where A represents the adaptation function, f a Represents the full-image adaptation feature, that is, the full-image feature after adaptation.

[0104] Finally, the full-image adaptation features are integrated into the diffusion model, and a diffusion optimization model is generated; specifically, in the denoising process of the diffusion model, the adapted features are injected into different levels of the model, and the feature injection can usually be achieved by using a cross-attention mechanism or feature addition. The implementation of style transfer in the model denoising process can include two steps: feature matching and adaptive weighting; feature matching refers to the process of image generation in which the model attempts to generate an image that matches the features of the prompt image, which can usually be achieved by minimizing the distance between the generated image and the prompt image in the feature space; adaptive weighting refers to the dynamic adjustment of the influence of the image prompt at different denoising stages, for example, more attention is paid to the overall composition in the early stage of image generation, and more attention is paid to details in the later stage of image generation. Using a diffusion model to process the latent space fusion features and decode to generate the final defect image includes: using a diffusion optimization model to process the latent space fusion features and decode to generate the final defect image.

[0105] Reference to the style transfer diagram of the prompt image Figure 4 As shown in the figure, the full-image features of the defective real object image are extracted through the visual model. The extracted full-image features are global features of the image, without distinguishing between specific defect features or background features. The full-image features are then adapted and injected into the diffusion model.

[0106] In the present application scheme, the overall style and details of the generated image can be flexibly controlled while maintaining the characteristics of industrial defects, which helps to generate richer and more realistic image details. This is of great significance for generating a diverse, high-quality industrial defect image dataset, and can significantly improve the robustness and generalization ability of the defect detection model. Secondly, for target objects in industrial defect images that are difficult to describe in words, pure text prompts have certain limitations, while image prompts provide a more intuitive and direct object representation method, thereby avoiding the limitations of text descriptions. In addition, the prompt images can be switched freely, so that a variety of image styles can be achieved, which improves the diversity of generated images.

[0107] In one embodiment, the steps of using a diffusion model to process the latent space fusion features and decoding to generate a final defect image may be specifically performed as follows:

[0108] After injecting the adapted full-image features into the diffusion model to generate the diffusion optimization model, the diffusion optimization model is used to iteratively denoise the latent space fusion features. Through multi-step denoising, the image details are gradually refined, and a defect latent space optimized image is generated. Afterwards, the defect latent space optimized image is decoded using a preset decoder function, and the final latent space representation is converted back to the pixel space to generate the final defect image.

[0109] Among them, the calculation method of iterative denoising can be expressed as:

[0110] x t-1 =μ θ (x t , t, c)+σ t ·∈;

[0111] Among them, x t represents the noise image at time t, μ θ represents the denoising network in the diffusion optimization model, σ t represents the noise intensity, ∈ represents the predicted noise, and C represents the conditional information in the ControlNet model.

[0112] The decoding calculation can be expressed as:

[0113] I final =D(z final );

[0114] Where D represents the decoder, I final Represents the final defect image.

[0115] Schematic diagram of secondary denoising image generation Figure 5 As shown, the denoising network (denoising model) in the optimized diffusion model is used to process the latent space fusion features, and then the features are converted into pixel space through decoding by the decoder to generate the defect image required by the end user.

[0116] In the present application scheme, secondary denoising image generation can apply additional denoising steps at the fusion boundary to eliminate possible seams, further enhance the details and realism of the image, and ensure that the generated defects are visually consistent with the background, thereby improving the quality of the generated image.

[0117] Reference Figure 6 , which is a complete schematic diagram of industrial defect image generation in the embodiment of the present application. The ControlNet model is trained through a mask image (Mask image), and the generation of defect features is guided in the process of denoising the initial pure noise image by the diffusion model (Stable Diffusion), and a defect latent space image is generated; the physical image is encoded and converted to a latent space representation, and feature extraction is performed on the defect image and the physical image represented by the latent space to obtain the defect target feature and background feature represented by the latent space, and the two features are fused to generate a latent space fusion feature; the full image features of the defect physical image or other prompt images are extracted, and injected into the diffusion model (Stable Diffusion) after adaptation processing. Finally, the latent space fusion feature is used to generate a secondary denoising image by using the optimized diffusion model, and converted to the pixel space after decoding to generate the final defect image.

[0118] Figure 1FIG. 1 is a flow chart of a defect image generation method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0119] Based on the above method, an embodiment of the present application also discloses a defect image generating device.

[0120] Reference Figure 7 , the device includes the following modules:

[0121] The original image acquisition module 701 is used to acquire a pure noise image, a defect real object image, an original defect image, a defect text description, and a template mask image, wherein the defect text description corresponds to the original defect image, and the template mask image corresponds to the defect in the original defect image;

[0122] The latent space image acquisition module 702 is used to generate a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image by using the ControlNet model and the diffusion model;

[0123] The feature fusion module 703 is used to obtain the defect target features and background features represented by the latent space according to the defect latent space image and the defect physical image using the template mask image, and fuse the defect target features and the background features to generate the latent space fusion features represented by the latent space;

[0124] The defect image generation module 704 is used to process the latent space fusion features using a diffusion model and decode and generate a final defect image.

[0125] In one embodiment, the latent space image acquisition module 702 is specifically used to use the template mask image to mask the original defect image and generate a defect mask image; train the ControlNet model according to the defect mask image, the defect text description and the template mask image, and generate a ControlNet optimization model; use the diffusion model to iteratively denoise the pure noise image, and use the ControlNet optimization model to generate a defect latent space image represented by the latent space based on the pure noise image.

[0126] In one embodiment, in the latent space image acquisition module 702: the calculation method of iterative denoising includes:

[0127] x t-1 =μ θ (x t , t)+σ t ∈

[0128] Among them, x t represents the noise image at time t, μ θ represents the denoising network in the diffusion model, σ t represents the noise intensity, and ∈ represents the prediction noise.

[0129] In one embodiment, the feature fusion module 703 is specifically used to use an encoder function to perform latent space encoding on the defect real object image and generate a defect real object latent space image; adjust the size of the template mask image according to the size of the latent space image and generate a template mask optimized image; use the template mask optimized image to respectively extract the defect target features in the defect latent space image and the background features in the defect real object latent space image; add and fuse the defect target features and the background features to generate a latent space fusion feature.

[0130] In one embodiment, the feature fusion module 703 is specifically used to use the template mask optimization image to extract defect target features in the defect latent space image; perform a negation operation on the template mask optimization image and generate a template mask negation image; and use the template mask negation image to extract background features in the defect real object latent space image.

[0131] In one embodiment, the defect image generation module 704 is also used to extract the full-image features of the defect real object image using a preset visual model; use a preset adaptation function to process the full-image features and generate full-image adaptation features; integrate the full-image adaptation features into the diffusion model and generate a diffusion optimization model; use the diffusion model to process the latent space fusion features and decode to generate the final defect image, including: use the diffusion optimization model to process the latent space fusion features and decode to generate the final defect image.

[0132] In one embodiment, the defect image generation module 704 is further used to iteratively denoise the latent space fusion features using a diffusion optimization model and generate a defect latent space optimized image; and decode the defect latent space optimized image using a preset decoder function to generate a final defect image;

[0133] Among them, the calculation method of iterative denoising includes:

[0134] x t-1 =μ θ (xt , t, c)+σ t ∈

[0135] Among them, x t represents the noise image at time t, μ θ represents the denoising network in the diffusion optimization model, σ t represents the noise intensity, ∈ represents the predicted noise, and c represents the conditional information in the ControlNet model.

[0136] The defect image generating device provided in the embodiment of the present application can be applied to the defect image generating method provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0137] It should be noted that: when the defect image generating device provided in the embodiment of the present application generates defect images, only the division of the above-mentioned functional modules / functional units is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the defect image generating device is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the defect image generating method provided in the above method embodiment and the implementation method of the defect image generating device provided in this embodiment belong to the same concept. The specific implementation process of the defect image generating device provided in this embodiment is detailed in the above method embodiment, which will not be repeated here.

[0138] The embodiment of the present application also discloses a computer device.

[0139] Specifically, if Figure 8As shown, the computer device can be a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or otherwise. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0140] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0141] The embodiment of the present application also discloses a computer-readable storage medium.

[0142] Specifically, a computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art will understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the process of the implementation of the above-mentioned methods. Among them, the storage medium may be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memory.

[0143] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A defect image generation method, characterized in that: The method comprises: Acquire a pure noise image, a defect object image, an original defect image, a defect text description, and a template mask image, wherein the defect text description corresponds to the original defect image, and the template mask image corresponds to the defect in the original defect image; Using a ControlNet model and a diffusion model, generating a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image; According to the defect latent space image and the defect physical image, the template mask image is used to obtain defect target features and background features represented by the latent space, and the defect target features and background features are fused to generate latent space fusion features represented by the latent space; The diffusion model is used to process the latent space fusion features and decode them to generate a final defect image.

2. The method according to claim 1, characterized in that: The method of using the ControlNet model and the diffusion model to generate a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image comprises: Using the template mask image to perform mask processing on the original defect image, and generate a defect mask image; Training the ControlNet model according to the defect mask image, the defect text description and the template mask image, and generating a ControlNet optimization model; The pure noise image is iteratively denoised using a diffusion model, and a defect latent space image represented by a latent space is generated based on the pure noise image using the ControlNet optimization model.

3. The method according to claim 2, characterized in that: The calculation method of the iterative denoising includes: x t-1 =μ θ (x t ,t)+σ t ·∈ Among them, x t represents the noise image at time t, μ θ represents the denoising network in the diffusion model, σ t represents the noise intensity, and ∈ represents the prediction noise.

4. The method according to claim 1, characterized in that: The method of obtaining defect target features and background features represented by the latent space using the template mask image according to the defect latent space image and the defect physical image, and fusing the defect target features and background features to generate latent space fusion features represented by the latent space includes: Performing latent space encoding on the defect object image by using an encoder function, and generating a defect object latent space image; Resizing the template mask image according to the size of the latent space image and generating a template mask optimized image; Using the template mask optimization image, respectively extract defect target features in the defect latent space image and background features in the defect object latent space image; The defect target feature and the background feature are added and fused to generate the latent space fusion feature.

5. The method according to claim 4, characterized in that: The step of respectively extracting defect target features in the defect latent space image and background features in the defect object latent space image by using the template mask optimization image comprises: Extracting defect target features in the defect latent space image using the template mask optimization image; Performing a negation operation on the template mask optimized image and generating a template mask negated image; The template mask inverted image is used to extract background features in the defective object latent space image.

6. The method according to claim 1, characterized in that: Before the latent space fusion feature is processed by the diffusion model and decoded to generate a final defect image, the method further includes: Using a preset visual model to extract full-image features of the defective physical image; Using a preset adaptation function to process the full-image feature and generate a full-image adaptation feature; Integrating the full-image adaptation features into the diffusion model and generating a diffusion optimization model; The step of using the diffusion model to process the latent space fusion features and decoding to generate a final defect image includes: The diffusion optimization model is used to process the latent space fusion features and decode them to generate a final defect image.

7. The method according to claim 6, characterized in that: The step of using the diffusion model to process the latent space fusion features and decoding to generate a final defect image includes: Iteratively denoising the latent space fusion features using the diffusion optimization model, and generating a defect latent space optimization image; Using a preset decoder function to decode the defect latent space optimization image to generate the final defect image; Among them, the calculation method of iterative denoising includes: x t-1 =μ θ (x t ,t,c)+σ t ·∈ Among them, x t represents the noise image at time t, μ θ represents the denoising network in the diffusion optimization model, σ t represents the noise intensity, ∈ represents the predicted noise, and c represents the conditional information in the ControlNet model.

8. A defect image generating device, characterized in that: The device comprises: The original image acquisition module (701) is used to acquire a pure noise image, a defect real object image, an original defect image, a defect text description and a template mask image, wherein the defect text description corresponds to the original defect image, and the template mask image corresponds to the defect in the original defect image; A latent space image acquisition module (702) is used to generate a defect latent space image represented by a latent space based on the pure noise image according to the original defect image, the defect text description and the template mask image using a ControlNet model and a diffusion model; A feature fusion module (703) is used to obtain defect target features and background features represented by the latent space according to the defect latent space image and the defect physical image using the template mask image, and fuse the defect target features and background features to generate latent space fusion features represented by the latent space; The defect image generation module (704) is used to process the latent space fusion features using the diffusion model and decode to generate a final defect image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images