Defective sample generation method and device

Abstract

The invention provides a defective sample generation method and device, and the method comprises the steps: obtaining a pre-configured low-pass filter, and obtaining a diffusion model used for unconditional image generation, a classification model used for image classification, and a segmentation model used for segmenting a defective region in an image; through the sampling process of the diffusion model, an initial image is sampled, a flaw sample is generated, and in the sampling process, within the action time step range of a preset image guiding mode, an image containing preset features is generated based on the guiding of the image guiding mode, the image guiding mode comprises at least one of the following modes: a contour guiding mode, a classification guiding mode and a segmentation guiding mode. According to the method and the device, the defective samples can be effectively controlled, and the quality of the generated defective samples is improved, so that the generalization performance of a downstream defect detection model / algorithm can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating defect samples. Background Art

[0002] Quality inspection is a crucial step in industrial product production, and visual defect detection is a common method. Industrial product inspection often faces the problem of a large number of normal samples and a difficulty in obtaining abnormal samples, which poses challenges for data-driven machine learning algorithms.

[0003] Sample generation techniques are used to generate abnormal samples for training machine learning algorithms. Diffusion models, such as the Denoising Diffusion Probabilistic Model (DDPM), have made significant progress in data generation. However, they have not been used in the field of defect sample generation, primarily due to the differences between industrial defect images and ordinary natural images.

[0004] Chinese patent application publication number CN116664975A ​​discloses a method for generating defect samples based on a generative adversarial network (GAN). This method employs a GAN to train a generative model using collected normal and defect samples. This model can generate defective image samples. However, this method lacks effective control over the generated samples, resulting in the generated defect samples failing to meet the requirements of industrial defect images, leading to poor practicality. Summary of the Invention

[0005] At least one embodiment of the present application provides a method and apparatus for generating defect samples, which can achieve effective control of defect samples and improve the quality of the generated defect samples.

[0006] According to a first aspect of the present application, at least one embodiment provides a method for generating a defect sample, comprising:

[0007] Obtaining a pre-configured low-pass filter, and obtaining a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image;

[0008] The initial image is sampled through a sampling process of the diffusion model to generate defect samples, wherein during the sampling process, an image containing preset features is generated based on a preset image-guided method within an effective time step range of the image-guided method, wherein the image-guided method includes at least one of the following:

[0009] A contour guidance mode, for guiding the generation of a reference background contour using the low-pass filter;

[0010] A classification guidance method for guiding the generation of defects of a target category using the classification model;

[0011] The segmentation guidance method is used to use the segmentation model to guide the generation of defects of a target shape.

[0012] Optionally, within the time step range of the contour guidance method, generating an image containing preset features based on the contour guidance method includes:

[0013] Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter;

[0014] Acquire a second intermediate image obtained by passing a reference image through the low-pass filter and noise addition process, wherein the reference image is an image providing the reference background contour;

[0015] Perform pixel subtraction on the second intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a first image, and replace the current image with the first image.

[0016] Optionally, within the time step range of the contour guidance method, generating an image containing preset features based on the contour guidance method includes:

[0017] Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter;

[0018] Determining an average value of the pixel colors of the defect, and assigning the average value to the pixel colors of pixels at corresponding positions in the reference image based on the target shape of the defect, to obtain a second intermediate image; obtaining a third intermediate image after passing the second intermediate image through the low-pass filter and noise addition process, where the reference image is the image providing the reference background contour;

[0019] Perform pixel subtraction on the third intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a second image, and replace the current image with the second image.

[0020] Optionally, within the effective time step range of the classification guidance method, generating an image containing preset features based on the classification guidance method includes:

[0021] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0022] Using the classification model, the current image x is calculated t The probability p of belonging to the target category y φ (y|x t ); According to the formula The second mean is calculated, where express x t gradient;

[0023] The current image is sampled according to the second mean and the first variance Σ to generate an image of the next time step of the current time step.

[0024] Optionally, within the effective time step range of the segmentation guidance method, generating an image containing preset features based on the segmentation guidance method includes:

[0025] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0026] The current image is segmented using the segmentation model to obtain a first shape m′ of the defect area, and a similarity sim between the target shape of the defect and the first shape m′ is calculated. ψ (m,m′), and according to the formula The third mean is calculated, where m represents the binary mask of the target shape, Indicates sim ψ (m,m′) for x t gradient;

[0027] The current image is sampled according to the third mean and the first variance Σ to generate an image of the next time step of the current time step.

[0028] Optionally, within the action time step range of the classification guidance mode and the segmentation guidance mode, generating an image containing preset features based on the classification guidance mode and the segmentation guidance mode includes:

[0029] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0030] Using the classification model, the current image x is calculatedt Belongs to the category y in the target category i The probability p φ (y i |x t ); Use the segmentation model to segment the current image to obtain a first shape m′ of the defect area, and calculate the similarity sim between the target shape of the defect and the first shape m′ ψ (m,m′);

[0031] According to the formula The fourth mean is calculated, where λ c represents the preset strength parameter of the classification guidance method, λ s represents the preset strength parameter of the segmentation guidance method, n represents the number of categories in the target category, represents logp φ (y i |x t ) for x t The gradient of m represents the binary mask of the target shape of the defect, Indicates sim ψ (m,m′) for x t The gradient of , ⊙ represents the multiplication of the corresponding pixels;

[0032] The current image is sampled according to the fourth mean and the first variance Σ to generate an image of the next time step of the current time step.

[0033] Optionally, obtaining a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image includes:

[0034] The diffusion model, the classification model, and the segmentation model are trained and obtained respectively based on a first training set, wherein the first training set includes pre-collected defect samples and normal samples.

[0035] Optionally, the above method further includes:

[0036] Using the generated multiple defect samples, updating the first training set;

[0037] Using the updated first training set, training a detection model for detecting defect samples;

[0038] After detecting samples using the detection model and updating the first training set using the defect samples detected by the detection model, and updating the reference background contour, target category and target shape of the defect, the method returns to the step of obtaining a pre-configured low-pass filter, and obtaining a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in the image.

[0039] According to a second aspect of the present application, at least one embodiment provides a device for generating a defect sample, comprising:

[0040] A first acquisition module is used to acquire a pre-configured low-pass filter, and acquire a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image;

[0041] A guidance module is configured to sample the initial image through a sampling process of the diffusion model to generate defect samples, wherein during the sampling process, within a time step range of a preset image guidance method, an image containing preset features is guided and generated based on the image guidance method, wherein the image guidance method includes at least one of the following:

[0042] A contour guidance mode, for guiding the generation of a reference background contour using the low-pass filter;

[0043] A classification guidance method for guiding the generation of defects of a target category using the classification model;

[0044] The segmentation guidance method is used to use the segmentation model to guide the generation of defects of a target shape.

[0045] Optionally, the guiding module includes:

[0046] The first contour guidance module is configured to generate an image containing preset features based on the contour guidance method within a time step range of the contour guidance method, specifically comprising:

[0047] Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter;

[0048] Acquire a second intermediate image obtained by passing a reference image through the low-pass filter and noise addition process, wherein the reference image is an image providing the reference background contour;

[0049] Perform pixel subtraction on the second intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a first image, and replace the current image with the first image.

[0050] Optionally, the guiding module includes:

[0051] The second contour guidance module is configured to generate an image containing preset features based on the contour guidance method within the effective time step range of the contour guidance method, specifically comprising:

[0052] Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter;

[0053] Determining an average value of the pixel colors of the defect, and assigning the average value to the pixel colors of pixels at corresponding positions in the reference image based on the target shape of the defect, to obtain a second intermediate image; obtaining a third intermediate image after passing the second intermediate image through the low-pass filter and noise addition process, where the reference image is the image providing the reference background contour;

[0054] Perform pixel subtraction on the third intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a second image, and replace the current image with the second image.

[0055] Optionally, the guiding module includes:

[0056] The classification guidance module is configured to guide the generation of an image containing preset features based on the classification guidance method within the effective time step range of the classification guidance method, specifically comprising:

[0057] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0058] Using the classification model, the current image x is calculated t The probability p of belonging to the target category y φ (y|x t ); According to the formula The second mean is calculated, where represents logp φ (m|x t ) for x t gradient;

[0059] The current image is sampled according to the second mean and the first variance Σ to generate an image of the next time step of the current time step.

[0060] Optionally, the guiding module includes:

[0061] The shape guidance module is configured to generate an image containing preset features based on the segmentation guidance method within the effective time step range of the segmentation guidance method, specifically comprising:

[0062] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0063] The current image is segmented using the segmentation model to obtain a first shape m′ of the defect area, and a similarity sim between the target shape of the defect and the first shape m′ is calculated. ψ (m,m′), and according to the formula The third mean is calculated, where m represents the binary mask of the target shape, Indicates sim ψ (m,m′) for x t gradient;

[0064] The current image is sampled according to the third mean and the first variance Σ to generate an image of the next time step of the current time step.

[0065] Optionally, the guiding module includes:

[0066] The combined guidance module is configured to guide the generation of an image containing preset features based on the classification guidance mode and the segmentation guidance mode within the action time step range of the classification guidance mode and the segmentation guidance mode, specifically comprising:

[0067] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0068] Using the classification model, the current image x is calculated t Belongs to the category y in the target category i The probability p φ (y i |x i ); Use the segmentation model to segment the current image to obtain a first shape m′ of the defect area, and calculate the similarity sim between the target shape of the defect and the first shape m′ ψ (m,m′);

[0069] According to the formula The fourth mean is calculated, where λ c represents the preset strength parameter of the classification guidance method, λ srepresents the preset strength parameter of the segmentation guidance method, n represents the number of categories in the target category, represents logp φ (y i |x t ) for x t The gradient of m represents the binary mask of the target shape of the defect, Indicates sim ψ (m,m′) for x t The gradient of , ⊙ represents the multiplication of the corresponding pixels;

[0070] The current image is sampled according to the fourth mean and the first variance Σ to generate an image of the next time step of the current time step.

[0071] According to the third aspect of the present application, at least one embodiment provides a method and device for generating defect samples, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in any one of the first aspects.

[0072] According to the fourth aspect of the present application, at least one embodiment provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the steps of any method of the first aspect are implemented.

[0073] According to a fifth aspect of the present application, at least one embodiment provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in any one of the first aspects.

[0074] Compared with the existing technology, the defect sample generation method and device provided in the embodiments of the present application realize the control of the background contour, category and shape of the defect, which can increase the diversity of the generated samples. The generated defect samples conform to the abnormal characteristics of actual industrial defective products, thereby improving the quality of the generated defect samples and further improving the generalization performance of downstream defect detection models / algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0076] Figure 1 Schematic diagram of industrial defect image;

[0077] Figure 2 This is a flowchart of a method for generating defect samples according to an embodiment of the present application;

[0078] Figure 3 A comparative schematic diagram of the model training process of an embodiment of the present application;

[0079] Figure 4 An example diagram of a method for generating defect samples provided in an embodiment of the present application;

[0080] Figure 5 A schematic structural diagram of a device for generating defective samples according to an embodiment of the present application;

[0081] Figure 6 Schematic diagram of another structure of the device for generating defective samples according to an embodiment of the present application. DETAILED DESCRIPTION

[0082] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0083] It should be understood that references throughout this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present application. Therefore, the appearance of "in one embodiment" or "in an embodiment" throughout this specification does not necessarily refer to the same embodiment. Furthermore, these particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. The terms "first," "second," and so on, used in the specification and claims of this application are used to distinguish similar items and are not necessarily used to describe a particular order or sequential sequence. It should be understood that such usage is interchangeable where appropriate, such that the embodiments of the present application described herein can, for example, be implemented in an order other than that illustrated or described herein. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus. The term "and / or" used in the specification and claims refers to at least one of the connected items.

[0084] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0085] The following description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the spirit and scope of this disclosure. The various examples may appropriately omit, substitute, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0086] Industrial defect images containing defects (and their corresponding defect-free images) have a clear structure and usually have three elements: similar normal background, defect category and defect shape. Figure 1 The image shows three elements of an industrial defect image, using hazelnuts as a representative example of an industrial product. The normal background is black. Defect categories include holes and prints, and defect shapes range from approximately rectangular to approximately circular. To improve the quality of anomaly detection in industrial products, a large number of defect samples are required to train machine learning models. However, defect samples in actual industrial products are rare, making them difficult to obtain. This presents challenges for data-driven machine learning.

[0087] In order to generate defect samples that meet the requirements of defective industrial products, the embodiment of the present application controls the generation of defect samples based on one or more of the above three factors. The control includes: normal background control, defect category control and defect shape control, etc., thereby improving the quality of the generated defect samples.

[0088] Please refer to Figure 2 , the embodiment of the present application provides a method for generating defect samples, which can achieve effective control of defect samples and improve the quality of the generated defect samples. Figure 2 As shown, the method includes the following steps:

[0089] Step 21: Obtain a pre-configured low-pass filter, and obtain a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image.

[0090] Here, the low-pass filter No training is required, low-pass filter The downsampling coefficient N is adjustable. By adjusting the downsampling coefficient N, the degree of retaining contour details can be adjusted. For example, the smaller the N value, the more contour details are retained.

[0091] The diffusion model may be a DDPM model or other diffusion model. The diffusion model, classification model, and segmentation model may be obtained through training. In the embodiment of the present application, the diffusion model, classification model, and segmentation model are trained based on a first training set, wherein the first training set includes pre-collected defect samples and normal samples.

[0092] As a basis for the following steps, such as Figure 3 As shown, a training set (training data set) can be prepared first. The training method adopted in the embodiment of the present application is: first, various types of defect samples (defective images) and normal samples (normal images) are combined into a whole training set (first training set), and the diffusion model for unconditional image generation is trained using the training set. Classification models for image classification and a segmentation model for segmenting defect areas Taking the defect images of the hazelnut category in the MVTec dataset as an example, it contains a small number of defect images of four categories: hole, print, cut, and crack, and a large number of normal images. All images are merged together and the above three models are trained under the same noise conditions.

[0093] Figure 3 A comparative diagram of the training process is given. Figure 3 The training method shown in the left part is based on various types of defect samples, and the corresponding diffusion model, classification model, and segmentation model are trained respectively. Figure 3 The training method shown in the middle right part is to obtain a combined training set (first training set) by combining defective samples and normal samples, and to train the diffusion model, classification model, and segmentation model based on the training set.

[0094] Step 22: Sampling the initial image through the sampling process of the diffusion model to generate defect samples. During the sampling process, an image containing preset features is generated based on a preset image-guided method within a time step range of the image-guided method. The image-guided method includes at least one of the following:

[0095] A contour guidance mode, for guiding the generation of a reference background contour using the low-pass filter;

[0096] A classification guidance method for guiding the generation of defects of a target category using the classification model;

[0097] The segmentation guidance method is used to use the segmentation model to guide the generation of defects of a target shape.

[0098] Here, the sampling process of the diffusion model (also referred to as the diffusion process) typically includes multiple time steps. Embodiments of the present application can preconfigure the time steps at which various image guidance methods are effective. Thus, within the time step range in which a certain image guidance method is effective, the extended model can be guided to generate an image containing preset features based on the effective image guidance method. One or more image guidance methods can be effective in the same time step. The effectiveness of a certain image guidance method in a certain time step means that the image guidance method is activated in that time step, i.e., the image guidance method guides the generation of an image containing preset features in that time step based on the image guidance method.

[0099] Through the above steps, the embodiment of the present application first collects a small number of defect samples and a certain amount of normal samples to form a basic data set (the first training set), and then uses the above-mentioned controllable diffusion model for defect generation to generate a large number of defect samples. During the above-mentioned generation process, prior knowledge or learned knowledge or reasonable assumptions can be used to set the reference background contour of the defect sample, the target category and target shape of the defect, thereby controlling the background contour, category and shape of the defect to increase the diversity of the generated samples, and the generated defect samples conform to the abnormal characteristics of actual industrial defective products, thereby improving the quality of the generated defect samples, and further improving the generalization performance of the downstream defect detection model / algorithm.

[0100] After step 22 above, the embodiment of the present application may further include the following steps:

[0101] Step 23: Update the first training set using the multiple defect samples generated in step 22.

[0102] Step 24: Use the updated first training set to train a detection model for detecting defective samples.

[0103] Step 25: Detect the sample using the detection model, thereby detecting defect samples of more categories and shapes.

[0104] In step 26, the embodiment of the application may further utilize the defect samples detected by the detection model to update the first training set and update the reference background contour, target category, and target shape of the defect. Then, the process returns to step 21 above.

[0105] By iteratively executing the above steps 21 to 26, the embodiment of the present application can improve the quality of the generated defect samples and improve the detection accuracy and generalization ability of the detection model for detecting defect samples.

[0106] The following combination Figure 4 , the above-mentioned image guidance methods of the embodiments of the present application are explained respectively. Figure 4 In [1], ODG represents the contour-guided method, CDG represents the classification-guided method, and SDG represents the segmentation-guided method. represents the low-pass filter, represents the diffusion model, represents the classification model, represents the segmentation model. Figure 4 The time step range of the contour guidance method ODG is The time step range of CDG is The time step range of the segmentation guidance method SDG is m1 and m2 represent two shapes of defects, and y1 and y2 represent two categories of defects.

[0107] 1. Contour guidance method

[0108] Generating an image containing preset features based on the contour guidance method within a time step range of the contour guidance method includes:

[0109] (1) Obtaining a current image generated by sampling the diffusion model in a current time step, and obtaining a first intermediate image obtained after the current image passes through the low-pass filter.

[0110] (2) Obtaining a second intermediate image after the reference image passes through the low-pass filter and noise adding process, wherein the reference image is the image providing the reference background contour.

[0111] (3) Subtracting the second intermediate image from the first intermediate image pixel by pixel, then adding the pixel by pixel to the current image to obtain a first image, and replacing the current image with the first image. Subsequently, the sampling process of the extended model can be continued based on the first image.

[0112] For example, Figure 4 The time step range in which the contour guidance method works is Image x t In the diffusion model The image x′ is obtained during the sampling process t-1 , the reference image r containing the reference background contour is low-pass filtered Get the low-pass filtered image And the noise adding process obtains Updated x t-1 Obtained by the following formula

[0113]

[0114] By adjusting the downsampling coefficient N of the low-pass filter, the degree of retained contour details can be adjusted. The smaller N is, the more details are retained.

[0115] Some types of defects are closely related to color, such as prints and holes, which appear close to white and black, respectively. For these defects, embodiments of the present application can further introduce color control. In a contour-guided approach, a preset defect shape in the reference image is first assigned the average color value of the defect's pixels. The preset defect shape can be pre-set or determined by a segmentation model. Specifically, it can be expressed by the following formula:

[0116]

[0117] m={m i},c i and m i are the average color value and shape of a defect, (1-m) represents the area other than the defect shape, and ⊙ is pixel multiplication, which means the corresponding pixels are multiplied. n represents the number of defect categories. The above formula 2 indicates that for the pixel point corresponding to the defect shape, the average value of the pixel color of the defect is assigned, that is, The other pixel points except the pixel points corresponding to the defect shape retain the original color, that is, r⊙(1-m).

[0118] In the contour guidance method of the embodiment of the present application, after adding color control (Coloring), the following steps may be specifically included:

[0119] (1) Obtaining a current image generated by sampling the diffusion model in a current time step, and obtaining a first intermediate image obtained after the current image passes through the low-pass filter.

[0120] (2) determining an average value of the pixel color of the defect, assigning the pixel color of the pixel point at the corresponding position in the reference image to the average value according to the target shape of the defect, and obtaining a second intermediate image; obtaining a third intermediate image after the second intermediate image passes through the low-pass filter and the noise addition process, wherein the reference image is the image that provides the reference background contour.

[0121] (3) Subtracting the third intermediate image from the first intermediate image pixel by pixel, then adding the pixel by pixel to the current image to obtain a second image, and replacing the current image with the second image. Subsequently, the sampling process of the extended model can be continued based on the second image.

[0122] 2. Classification guidance method

[0123] Generating an image containing preset features based on guidance of the classification guidance method within an effective time step of the classification guidance method includes:

[0124] (1) Obtaining a current image sampled and generated by the diffusion model in the current time step, and obtaining a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step.

[0125] (2) Using the classification model, calculate the current image x t The probability p of belonging to the target category y φ (y|x t ); According to the formula The second mean is calculated, where express x t gradient.

[0126] (3) Sampling the current image according to the second mean and the first variance ∑ to generate an image of the next time step of the current time step.

[0127] Classification guidance method CDG through classification model x t For classification, its guided category is y. For example, first obtain the first mean μ and first variance ∑ of the sampling through the diffusion model of DDPM, and obtain the guided x through the following formula t-1

[0128]

[0129] p φ (y|x t ) represents x t The probability of belonging to a specified discrete class label y can be calculated by the classification model; represents a normal distribution; express x t The gradient is used to guide the sampling process to generate data of the specified category. The variance ∑ is a constant.

[0130] 3. Split boot method

[0131] Generating an image containing preset features based on the segmentation guidance method within a time step range of the segmentation guidance method includes:

[0132] (1) Obtaining a current image sampled and generated by the diffusion model in the current time step, and obtaining a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step.

[0133] (2) Segmenting the current image using the segmentation model to obtain a first shape m′ of the defect area, and calculating the similarity sim between the target shape of the defect and the first shape m′ ψ (m,m′), and according to the formula The third mean is calculated, where m represents the binary mask of the target shape, Indicates sim ψ (m,m′) for x t gradient.

[0134] (3) Sampling the current image according to the third mean and the first variance ∑ to generate an image of the next time step of the current time step.

[0135] Segmentation-guided SDG uses segmentation models x t Segmentation is performed to obtain the defect foreground segmentation m′, whose guiding shape is m. First, the similarity sim between m and m′ is calculated by the following formula ψ (m,m′):

[0136] sim ψ (m,m′)=MSE(m,m′) (4)

[0137] In the above formula, m is the binary mask of the target shape, and m′ is the image x during the sampling process of the same size. t The inferred defect shape is represented by MSE, which represents the mean squared error of the pixel.

[0138] Then sample the guided x by the following formula t-1 Sampling of:

[0139]

[0140] here, The above bootstrapping process is to make the value of Formula 4 smaller (more similar).

[0141] 4. Combined guidance method of classification guidance method and segmentation guidance method

[0142] Generating an image containing preset features based on the classification guidance mode and the segmentation guidance mode within the action time step range of the classification guidance mode and the segmentation guidance mode, including:

[0143] (1) Obtaining a current image sampled and generated by the diffusion model in the current time step, and obtaining a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step.

[0144] (2) Using the classification model, calculate the current image x t Belongs to the category y in the target category i The probability p φ (y i |x t ); Use the segmentation model to segment the current image to obtain a first shape m′ of the defect area, and calculate the similarity sim between the target shape of the defect and the first shape m′ ψ (m,m′).

[0145] (3) According to the formula:

[0146] The fourth mean is calculated, where λ c represents the preset strength parameter of the classification guidance method, λ s represents the preset strength parameter of the segmentation guidance method, n represents the number of categories in the target category, represents logp φ (y i |x t ) for x t The gradient of m represents the binary mask of the target shape of the defect, Indicates sim ψ (m,m′) for x t The gradient of , ⊙ represents the multiplication of the corresponding pixels.

[0147] (4) Sampling the current image according to the fourth mean and the first variance ∑ to generate an image of the next time step of the current time step.

[0148] When CDG and SDG are jointly guided, the guidance process is summed up by the coefficients:

[0149]

[0150] Among them, λ c and λ s The strength of CDG and SDG can be controlled. n is the number of guided categories, and ⊙ is pixel multiplication, which is used to control the guidance to work only within the range of m. Pixel multiplication means pixel by pixel, that is, the multiplication between corresponding pixels.

[0151] From the above description, it can be seen that the embodiment of the present application can independently control the three factors of defect generation and adjust their scope and intensity. Classification Model and segmentation models They can be trained independently without affecting each other, so this method is easy to implement.

[0152] In order to better understand the above embodiments, the method of the embodiments of the present application is further explained below through a more specific example.

[0153] like Figure 3 As shown in the figure, this example collects normal samples of hazelnut and various typical defect samples such as hole, print, cut, and crack. The typical image size is 256×256. Train an unconditional DDPM diffusion model The diffusion time step T is typically 1000.

[0154] In order to guide the diffusion model, the classification model is trained under the same noise conditions as DDPM. and segmentation models And define a low-pass filter L with a downsampling coefficient N of 32. In addition, define the reference image r, the required guided categories y1 (hole), y2 (print) and the shape masks m1, m2, whose average colors are c1 = (0,0,0) (black RGB value) and c2 = (255,255,255) (white RGB value) respectively.

[0155] The time step range of the selected contour guide is [1000, 500], the time step range of the selected category guide is [500, 0], and the time step range of the selected shape guide is [500, 0]. c is 10 2 , select the shape-guided proportional coefficient λ s is 10 4 Through the controllable diffusion sampling for defect generation, iterative sampling starts from t = T = 1000, and finally the generated image x0 is obtained as follows Figure 4 shown.

[0156] Please refer to Figure 5 , a structure of a device for generating a defect sample provided in an embodiment of the present application includes:

[0157] A first acquisition module 51 is configured to acquire a pre-configured low-pass filter, a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image;

[0158] The guidance module 52 is configured to sample the initial image through a sampling process of the diffusion model to generate defect samples. During the sampling process, an image containing preset features is generated based on a preset image guidance method within a time step range of the image guidance method. The image guidance method includes at least one of the following:

[0159] A contour guidance mode, for guiding the generation of a reference background contour using the low-pass filter;

[0160] A classification guidance method for guiding the generation of defects of a target category using the classification model;

[0161] The segmentation guidance method is used to use the segmentation model to guide the generation of defects of a target shape.

[0162] Optionally, the guiding module includes:

[0163] The first contour guidance module is configured to generate an image containing preset features based on the contour guidance method within a time step range of the contour guidance method, specifically comprising:

[0164] Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter;

[0165] Acquire a second intermediate image obtained by passing a reference image through the low-pass filter and noise addition process, wherein the reference image is an image providing the reference background contour;

[0166] Perform pixel subtraction on the second intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a first image, and replace the current image with the first image.

[0167] Optionally, the guiding module includes:

[0168] The second contour guidance module is configured to generate an image containing preset features based on the contour guidance method within the effective time step range of the contour guidance method, specifically comprising:

[0169] Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter;

[0170] Determining an average value of the pixel colors of the defect, and assigning the average value to the pixel colors of pixels at corresponding positions in the reference image based on the target shape of the defect, to obtain a second intermediate image; obtaining a third intermediate image after passing the second intermediate image through the low-pass filter and noise addition process, where the reference image is the image providing the reference background contour;

[0171] Perform pixel subtraction on the third intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a second image, and replace the current image with the second image.

[0172] Optionally, the guiding module includes:

[0173] The classification guidance module is configured to guide the generation of an image containing preset features based on the classification guidance method within the effective time step range of the classification guidance method, specifically comprising:

[0174] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0175] Using the classification model, the current image x is calculated t The probability p of belonging to the target category y φ (y|x t ); According to the formula The second mean is calculated, where represents logp φ (y|x t ) for x t gradient;

[0176] The current image is sampled according to the second mean and the first variance Σ to generate an image of the next time step of the current time step.

[0177] Optionally, the guiding module includes:

[0178] The shape guidance module is configured to generate an image containing preset features based on the segmentation guidance method within the effective time step range of the segmentation guidance method, specifically comprising:

[0179] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0180] The current image is segmented using the segmentation model to obtain a first shape m′ of the defect area, and a similarity sim between the target shape of the defect and the first shape m′ is calculated.ψ (m,m′), and according to the formula The third mean is calculated, where m represents the binary mask of the target shape, Indicates sim ψ (m,m′) for x t gradient;

[0181] The current image is sampled according to the third mean and the first variance Σ to generate an image of the next time step of the current time step.

[0182] Optionally, the guiding module includes:

[0183] The combined guidance module is configured to guide the generation of an image containing preset features based on the classification guidance mode and the segmentation guidance mode within the action time step range of the classification guidance mode and the segmentation guidance mode, specifically comprising:

[0184] Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step;

[0185] Using the classification model, the current image x is calculated t Belongs to the category y in the target category i The probability p φ (y i |x t ); Use the segmentation model to segment the current image to obtain a first shape m′ of the defect area, and calculate the similarity sim between the target shape of the defect and the first shape m′ ψ (m,m′);

[0186] According to the formula The fourth mean is calculated, where λ c represents the preset strength parameter of the classification guidance method, λ s represents the preset strength parameter of the segmentation guidance method, n represents the number of categories in the target category, represents logp φ (y i |x t ) for x t The gradient of m represents the binary mask of the target shape of the defect, Indicates sim ψ (m,m′) for x t The gradient of , ⊙ represents the multiplication of the corresponding pixels;

[0187] The current image is sampled according to the fourth mean and the first variance Σ to generate an image of the next time step of the current time step.

[0188] Optionally, the first acquisition module is further configured to respectively train the diffusion model, the classification model, and the segmentation model based on a first training set, wherein the first training set includes pre-collected defect samples and normal samples.

[0189] Optionally, the above device further includes:

[0190] An iterative control module is configured to update the first training set using the generated multiple defect samples; train a detection model for detecting defect samples using the updated first training set; detect samples using the detection model, and update the first training set using the defect samples detected by the detection model. After updating the reference background contour, target category, and target shape of the defect, the first acquisition module is triggered to execute the steps of acquiring a preconfigured low-pass filter, a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image.

[0191] It should be noted that the various systems provided in the above embodiments are devices corresponding to the above methods for generating defect samples. The implementation methods in the above embodiments are applicable to the embodiments of the devices and can achieve the same technical effects. The above-mentioned devices provided in the embodiments of the present application can implement all the method steps implemented in the above-mentioned method embodiments and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiments will not be detailed here.

[0192] Please refer to Figure 6 , a structural diagram of another method and device for generating defect samples provided in an embodiment of the present application, the device includes: a processor 601, a transceiver 602, a memory 603, a user interface 604 and a bus interface.

[0193] In the embodiment of the present application, the device further includes: a program stored in the memory 603 and executable on the processor 601 .

[0194] The transceiver 602 is configured to transmit and receive data under the control of the processor;

[0195] The processor 601 is configured to read the computer program in the memory and perform the following operations:

[0196] Obtaining a pre-configured low-pass filter, and obtaining a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image;

[0197] The initial image is sampled through a sampling process of the diffusion model to generate defect samples, wherein during the sampling process, an image containing preset features is generated based on a preset image-guided method within an effective time step range of the image-guided method, wherein the image-guided method includes at least one of the following:

[0198] A contour guidance mode, for guiding the generation of a reference background contour using the low-pass filter;

[0199] A classification guidance method for guiding the generation of defects of a target category using the classification model;

[0200] The segmentation guidance method is used to use the segmentation model to guide the generation of defects of a target shape.

[0201] It is understandable that in the embodiment of the present application, when the computer program is executed by the processor 601, each process of the above-mentioned defect sample generation method embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0202] exist Figure 6 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 601 and memory represented by memory 603. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 602 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 604 may also be an interface capable of connecting external or internal devices as required, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, etc.

[0203] The processor 601 is responsible for managing the bus architecture and general processing, and the memory 603 can store data used by the processor 601 when performing operations.

[0204] It should be noted that the device in this embodiment is a device corresponding to the above-mentioned method for generating defect samples, and the implementation methods in the above-mentioned embodiments are all applicable to the embodiments of the device and can achieve the same technical effects. In the device, the transceiver 602 and the memory 603, as well as the transceiver 602 and the processor 601 can be connected to each other through a bus interface. The functions of the processor 601 can also be implemented by the transceiver 602, and the functions of the transceiver 602 can also be implemented by the processor 601. It should be noted that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effects. The parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0205] In some embodiments of the present application, a computer-readable storage medium is further provided, on which a program is stored. When the program is executed by a processor, the following steps are implemented:

[0206] Obtaining a pre-configured low-pass filter, and obtaining a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image;

[0207] The initial image is sampled through a sampling process of the diffusion model to generate defect samples, wherein during the sampling process, an image containing preset features is generated based on a preset image-guided method within an effective time step range of the image-guided method, wherein the image-guided method includes at least one of the following:

[0208] A contour guidance mode, for guiding the generation of a reference background contour using the low-pass filter;

[0209] A classification guidance method for guiding the generation of defects of a target category using the classification model;

[0210] The segmentation guidance method is used to use the segmentation model to guide the generation of defects of a target shape.

[0211] When the program is executed by the processor, it can implement all the implementation methods of the above-mentioned defect sample generation method and achieve the same technical effect. To avoid repetition, it will not be described here.

[0212] An embodiment of the present application further provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned defect sample generation method embodiment are implemented, and the same technical effects can be achieved. To avoid repetition, they are not described here.

[0213] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0214] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0215] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0216] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0217] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0218] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0219] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for generating a defect sample, characterized in that: include: Obtaining a pre-configured low-pass filter, and obtaining a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image; The initial image is sampled through a sampling process of the diffusion model to generate defect samples, wherein during the sampling process, an image containing preset features is generated based on a preset image-guided method within an effective time step range of the image-guided method, wherein the image-guided method includes at least one of the following: A contour guidance mode, for guiding the generation of a reference background contour using the low-pass filter; A classification guidance method for guiding the generation of defects of a target category using the classification model; The segmentation guidance method is used to use the segmentation model to guide the generation of defects of a target shape.

2. The method according to claim 1, wherein Generating an image containing preset features based on the contour guidance method within a time step range of the contour guidance method includes: Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter; Acquire a second intermediate image obtained by passing a reference image through the low-pass filter and noise addition process, wherein the reference image is an image providing the reference background contour; Perform pixel subtraction on the second intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a first image, and replace the current image with the first image.

3. The method according to claim 1, wherein Generating an image containing preset features based on the contour guidance method within a time step range of the contour guidance method includes: Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter; Determining an average value of the pixel colors of the defect, and assigning the average value to the pixel colors of pixels at corresponding positions in the reference image based on the target shape of the defect, to obtain a second intermediate image; obtaining a third intermediate image after passing the second intermediate image through the low-pass filter and noise addition process, where the reference image is the image providing the reference background contour; Perform pixel subtraction on the third intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a second image, and replace the current image with the second image.

4. The method according to claim 1, wherein Generating an image containing preset features based on guidance of the classification guidance method within an effective time step of the classification guidance method includes: Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step; Using the classification model, the current image x is calculated t The probability p of belonging to the target category y φ (y|x t ); According to the formula The second mean is calculated, where represents logp φ (y|x t ) for x t gradient; The current image is sampled according to the second mean and the first variance Σ to generate an image of the next time step of the current time step.

5. The method according to claim 1, wherein Generating an image containing preset features based on the segmentation guidance method within a time step range of the segmentation guidance method includes: Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step; The current image is segmented using the segmentation model to obtain a first shape m′ of the defect area, and a similarity sim between the target shape of the defect and the first shape m′ is calculated. ψ (m,m′), and according to the formula The third mean is calculated, where m represents the binary mask of the target shape, Indicates sim ψ (m,m′) for x t gradient; The current image is sampled according to the third mean and the first variance Σ to generate an image of the next time step of the current time step.

6. The method according to claim 1, wherein Generating an image containing preset features based on the classification guidance mode and the segmentation guidance mode within the action time step range of the classification guidance mode and the segmentation guidance mode, including: Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step; Using the classification model, the current image x is calculated t Belongs to the category y in the target category i The probability p φ (y i |x t ); Use the segmentation model to segment the current image to obtain a first shape m′ of the defect area, and calculate the similarity sim between the target shape of the defect and the first shape m′ ψ (m,m′); According to the formula The fourth mean is calculated, where λ c represents the preset strength parameter of the classification guidance method, λ s represents the preset strength parameter of the segmentation guidance method, n represents the number of categories in the target category, represents logp φ (y i |x t ) for x t The gradient of m represents the binary mask of the target shape of the defect, Indicates sim ψ (m,m′) for x t The gradient of , ⊙ represents the multiplication of the corresponding pixels; The current image is sampled according to the fourth mean and the first variance Σ to generate an image of the next time step of the current time step.

7. The method according to claim 1, wherein Obtain diffusion models for unconditional image generation, classification models for image classification, and segmentation models for segmenting defect regions in images, including: The diffusion model, the classification model, and the segmentation model are trained and obtained respectively based on a first training set, wherein the first training set includes pre-collected defect samples and normal samples.

8. The method according to claim 7, wherein Also includes: Using the generated multiple defect samples, updating the first training set; Using the updated first training set, training a detection model for detecting defect samples; After detecting samples using the detection model and updating the first training set using the defect samples detected by the detection model, and updating the reference background contour, target category and target shape of the defect, the method returns to the step of obtaining a pre-configured low-pass filter, and obtaining a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in the image.

9. A device for generating defective samples, characterized in that: include: A first acquisition module is used to acquire a pre-configured low-pass filter, and acquire a diffusion model for unconditional image generation, a classification model for image classification, and a segmentation model for segmenting defect areas in an image; A guidance module is configured to sample the initial image through a sampling process of the diffusion model to generate defect samples, wherein during the sampling process, within a time step range of a preset image guidance method, an image containing preset features is guided and generated based on the image guidance method, wherein the image guidance method includes at least one of the following: A contour guidance mode, for guiding the generation of a reference background contour using the low-pass filter; A classification guidance method for guiding the generation of defects of a target category using the classification model; The segmentation guidance method is used to use the segmentation model to guide the generation of defects of a target shape.

10. The device according to claim 9, wherein The guiding module includes: The first contour guidance module is configured to generate an image containing preset features based on the contour guidance method within a time step range of the contour guidance method, specifically comprising: Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter; Acquire a second intermediate image obtained by passing a reference image through the low-pass filter and noise addition process, wherein the reference image is an image providing the reference background contour; Perform pixel subtraction on the second intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a first image, and replace the current image with the first image.

11. The device according to claim 9, wherein The guiding module includes: The second contour guidance module is configured to generate an image containing preset features based on the contour guidance method within the effective time step range of the contour guidance method, specifically comprising: Acquire a current image generated by sampling the diffusion model in a current time step, and acquire a first intermediate image obtained after the current image passes through the low-pass filter; Determining an average value of the pixel colors of the defect, and assigning the average value to the pixel colors of pixels at corresponding positions in the reference image based on the target shape of the defect, to obtain a second intermediate image; obtaining a third intermediate image after passing the second intermediate image through the low-pass filter and noise addition process, where the reference image is the image providing the reference background contour; Perform pixel subtraction on the third intermediate image and the first intermediate image, and then perform pixel addition on the current image to obtain a second image, and replace the current image with the second image.

12. The device according to claim 9, wherein The guiding module includes: The classification guidance module is configured to guide the generation of an image containing preset features based on the classification guidance method within the effective time step range of the classification guidance method, specifically comprising: Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step; Using the classification model, the current image x is calculated t The probability p of belonging to the target category y φ (y|x t ); According to the formula The second mean is calculated, where represents logp φ (y|x t ) for x t gradient; The current image is sampled according to the second mean and the first variance Σ to generate an image of the next time step of the current time step.

13. The device according to claim 9, wherein The guiding module includes: The shape guidance module is configured to generate an image containing preset features based on the segmentation guidance method within the effective time step range of the segmentation guidance method, specifically comprising: Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step; The current image is segmented using the segmentation model to obtain a first shape m′ of the defect area, and a similarity sim between the target shape of the defect and the first shape m′ is calculated. ψ (m,m′), and according to the formula The third mean is calculated, where m represents the binary mask of the target shape, Indicates sim ψ (m,m′) for x t gradient; The current image is sampled according to the third mean and the first variance Σ to generate an image of the next time step of the current time step.

14. The device according to claim 9, wherein The guiding module includes: The combined guidance module is configured to guide the generation of an image containing preset features based on the classification guidance mode and the segmentation guidance mode within the action time step range of the classification guidance mode and the segmentation guidance mode, specifically comprising: Acquire a current image sampled and generated by the diffusion model in a current time step, and acquire a first mean μ and a first variance ∑ generated by the diffusion model for sampling the current image in the current time step; Using the classification model, the current image x is calculated t Belongs to the category y in the target category i The probability p φ (y i |x t ); Use the segmentation model to segment the current image to obtain a first shape m′ of the defect area, and calculate the similarity sim between the target shape of the defect and the first shape m′ ψ (m,m′); According to the formula The fourth mean is calculated, where λ c represents the preset strength parameter of the classification guidance method, λ s represents the preset strength parameter of the segmentation guidance method, n represents the number of categories in the target category, represents logp φ (y i |x t ) for x t The gradient of m represents the binary mask of the target shape of the defect, Indicates sim ψ (m,m′) for x t The gradient of , ⊙ represents the multiplication of the corresponding pixels; The current image is sampled according to the fourth mean and the first variance Σ to generate an image of the next time step of the current time step.

15. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images