Industrial product defect sample controllable generation method based on reference image guidance

Abstract

The invention relates to an industrial product defect sample controllable generation method based on reference image guidance, and the method comprises the steps: extracting the pattern information of a reference defect through a reference defect processing module and a reference defect feature coding module, and taking the pattern information as the guidance information of a de-noising UNet; using a defect position and detail processing module and a defect position and detail coding module to generate the other part of guide information of the de-noised UNet; constructing a defect pattern loss function based on a Gram matrix and training a model; in a reasoning stage, according to a defect intensity parameter input by a user, a trained model is used, an image which is similar to a reference defect and has specified intensity is generated in a given area through step-by-step denoising from random noise, and the image is kept consistent with an input background image in a background area. According to the method, the defect image is used for guiding, and diversified defect samples are generated according to the input defect intensity control parameters, so that data samples which are richer and closer to real defect images are provided for defect detection model training.

Description

Technical Field

[0001] The present invention belongs to the field of industrial automated quality inspection, and in particular to a method for controllably generating defective samples of industrial products based on reference image guidance. Background Art

[0002] Against the backdrop of the rapid development of industrial automation and intelligent manufacturing, defect detection, as a key link in quality control, is crucial to ensuring product reliability and safety. In recent years, deep learning-based visual models applied to surface defect detection / segmentation tasks have become an important choice for many industrial quality inspection systems. However, the excellent performance of deep learning models in the field of industrial quality inspection is inseparable from massive, diverse, and well-annotated training sets. Generally speaking, industrial production lines have a high yield rate, and obtaining a large number of defect samples is very difficult or even impossible; collecting a defect dataset that comprehensively covers all possible features, especially those that are small and inconspicuous, is still challenging. Especially in some specific fields, such as semiconductors and automotive parts, obtaining a sufficient number of defect samples to build and train effective detection models is a huge challenge.

[0003] In real-world applications, users have accumulated some defect sample data from other defect detection scenarios. However, in new defect detection scenarios, only defect-free data and a small amount of defect sample data are available. The defect patterns in the original scene and the defect patterns in the new scene have certain similarities. However, training a defect detection model solely on the original defect sample data cannot effectively detect defects in the new scene. Therefore, it is necessary to migrate the defect data from the original scene to defect-free samples in the new scene. The generated defect samples are used to train the defect detection model. They must be authentic and diverse, and the defect-free areas should be consistent with the defect-free images in the new scene.

[0004] To address the problem of insufficient defect samples, traditional data augmentation methods can be used to increase the sample size. This method typically randomly samples defective regions from existing defect samples, then rotates, crops, and filters these regions before pasting them onto normal samples to create synthetic defect samples. While these methods alleviate the problem of insufficient samples to some extent, the synthetic samples differ significantly from real samples. Directly using these synthetic samples to train defect detection models does not effectively improve their performance.

[0005] To address the issue of unrealistic generated samples, several deep learning-based defect sample generation methods have been developed in recent years. A representative example is the generative adversarial network (GAN)-based approach. However, the training process of GAN-based defect sample generation models (An Anomaly Feature-Editing-Based Adversarial Network for Texture Defect Visual Inspection, IEEE Transactions on Industrial Informatics, vol. 17, no. 3) is highly unstable.

[0006] The denoising diffusion probabilistic model (DDPM) is another type of image generation model that generates images through both denoising and denoising. DDPM training stability is significantly improved compared to GANs, and the generated samples are closer to real-world samples. Some researchers have generated industrial defect images by training with industrial defect data (Leveraging Latent Diffusion Models for Training-Free In-Distribution Data Augmentation for Surface Defect Detection, CBMI2024) or fine-tuning diffusion models trained on natural images (Diffusion-based Image Generation for In-distribution Data Augmentation in Surface Defect Detection, VISAPP2024). Some methods (Controllable Image Synthesis of Industrial Data using Stable Diffusion, WCAV2024) use a staged training approach and introduce ControlNet (Adding Conditional Control to Text-to-Image Diffusion Models, ICCV2023) to generate defects at specific locations in an image. These methods use text to guide defect generation and a defect mask to control the location of generated defects. While these methods can control the location of generated defects, they cannot guarantee consistency between non-defective areas and the input background image. Furthermore, the diversity of defect samples required for defect detection generally refers to defect type and intensity. However, the diversity of images generated by these methods stems from the capabilities of the denoising diffusion model itself, which is inconsistent with the requirements of defect detection models (diverse defect types and a wide distribution of defect intensities). Summary of the Invention

[0007] The purpose of the present invention is to overcome the shortcomings of the existing technology and propose a controllable generation method of industrial product defect samples based on reference image guidance. By combining the generation capability of the diffusion model and the guidance information of the defect image, the intensity of the defect is controlled according to actual needs, and diversified defect samples are generated, thereby providing the defect detection model with richer data samples that are closer to the real defect image.

[0008] The present invention solves the technical problem by adopting the following technical solutions:

[0009] A method for controllably generating defective samples of industrial products based on reference image guidance comprises the following steps:

[0010] Step 1: Perform affine transformation on the input defect image through the reference defect processing module to generate a variety of reference defect images;

[0011] Step 2: Use the reference defect feature encoding module to extract the style information of the reference defect image and inject it into the cross-attention module of the denoising UNet as the guidance information for defect generation;

[0012] Step 3: Extract edge detail information of the defect through the defect location and detail processing module to generate a detail guide image of the defect;

[0013] Step 4: Use the defect location and detail encoding module to encode the defect detail guidance image to generate another part of the guidance information of the denoising UNet;

[0014] Step 5: Construct a defect style loss function based on the Gram matrix and train the model;

[0015] Step 6. In the inference stage, according to the defect intensity parameters input by the user, the trained model is used to gradually denoise the random noise through the denoising UNet, and a defect image with a specified intensity similar to the reference defect is generated in a given area, and the background area is kept consistent with the input background image.

[0016] Moreover, the specific implementation method of step 1 is: input the image with defects and the defect location mask , H I For the input image Height, W I For the input image Width;

[0017] in Representative coordinates The pixels are defective and the rest are non-defective. .according to The size of the reference defect image is , is the height of the reference defect image, The width of the reference defect image is then segmented to obtain the defect image. ,in , Represents matrix element-by-element multiplication, and Perform a random affine transformation, and the randomly generated affine transformation matrix is , the processed image is represented as = .

[0018] Moreover, the specific implementation method of step 2 is: output global features through DINOv2 ( ) and image patch features ( ):

[0019]

[0020] Global Features and image patch features After merging, the complete reference defect image features are obtained , the merging method is to splice on the first dimension of the matrix, referring to the defect image features Input to the linear layer and finally get the defect feature encoding = , the parameters contained in it are expressed as , the linear layer is represented as , which is input into the cross-attention module in the denoising UNet.

[0021] Moreover, the specific implementation method of step 3 is: input image Convert to grayscale , using a gradient-based edge extraction operator and Extract the gradient information in the vertical and horizontal directions respectively, obtain the edge information of the defect based on the gradient information, and finally perform the corrosion operation Remove some noise data and optimize the edge details of the defect:

[0022]

[0023] in represents the convolution operation; therefore, the final input model details guide image Expressed as:

[0024]

[0025] in Represents matrix concatenation in the channel dimension.

[0026] Moreover, the specific implementation method of step 4 is: the defect location and detail encoding module guides the image with details For input information, encode the image to generate another part of the guidance information of the denoising UNet . It is added to the output of the denoising UNet encoder and used as the input of the denoising UNet decoder.

[0027] Moreover, the loss function in step 5 is:

[0028]

[0029] in, The loss function originally used to train the denoising diffusion probability model DDPM is used to measure the difference between the added noise and the predicted noise. is the latent space representation of the image after UNet denoising Latent space representation of reference defects The style differences between Used to balance the order of magnitude difference between the two loss functions, For denoising UNet, is the time step for adding noise, Adding the latent space representation of the input image The characteristics after step noise, is the characteristic of the reference defect.

[0030] The advantages and positive effects of the present invention are:

[0031] 1. The present invention performs an affine transformation on the input defect image through a reference defect processing module to generate a variety of reference defect images; uses a reference defect feature encoding module to extract the style information of the reference defect image and injects it into the cross-attention module of the denoising UNet as guidance information for defect generation; extracts the edge detail information of the defect through the defect position and detail processing module to generate a detail guidance image of the defect; uses the defect position and detail encoding module to encode the detail guidance image of the defect to generate another part of the guidance information of the denoising UNet; in the verification stage, according to the defect intensity parameter input by the user, the trained model is used to gradually denoise the random noise through the denoising UNet, and a defect image of a specified intensity similar to the reference defect is generated in a given area, and the background image is kept consistent with the input background image in the background area. By combining the generation capability of the diffusion model and the guidance information of the defect image, the present invention can control the intensity of the defect according to actual needs and generate a variety of defect samples, thereby providing the defect detection model with richer data samples that are closer to the real defect image.

[0032] 2. By referring to the defect feature encoding module, the present invention can extract the style information of the defect from the reference defect and inject it into the cross-attention module of the denoising UNet of the SD model as guidance information for defect generation. The present invention uses image guidance to overcome the shortcoming of text guidance that the text is difficult to accurately describe the specific style of the defect.

[0033] 3. In order to effectively supervise the training of the SD model to obtain defect style information from the reference defect rather than the complete content information of the reference defect, the present invention proposes a defect style loss function based on the Gram matrix; a reference defect processing module is proposed to perform data enhancement through affine transformation to ensure that the model can focus more on the style information of the reference defect rather than the complete content information.

[0034] 4. In order to control the generated defect samples in the background area to be consistent with the input defect-free image, and to control the intensity of the generated samples in the defect area, the present invention proposes a method of mixing the denoising UNet prediction value and the original background image features in the latent space, and uses a user-input number to specify the intensity of the generated defect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a structural diagram of the model training phase of the present invention;

[0036] Figure 2 This is a structural diagram of the inference stage of the model of the present invention;

[0037] Figure 3 Schematic diagram of defect samples and their annotation information in the MSD_Seg2 dataset according to an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the defect generation effect with controllable intensity according to an embodiment of the present invention;

[0039] Figure 5 Schematic diagram comparing the defects generated by the present invention with the original defects in the data set. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings.

[0041] The Stable Diffusion (SD) model has achieved good results in many image generation and editing tasks. Currently, most SD models use text information to control the target to be generated. However, in the industrial field, the image category is single and the semantics are vague, and it is difficult to accurately describe the defect information in words. Therefore, the present invention uses the defect reference defect image as the guiding information. In the field of defect generation, the location of the defect is very important. The present invention specifies the location of the generated defect through the input mask. The present invention is based on the SD model and uses the defect image and the defect location (mask) to guide and realize the generation of an image similar to the reference defect at the specified location. The present invention is a method based on deep learning. First, the model needs to be trained using a training data set. The structure of the model is as follows Figure 1 As shown in Figure 2, the QKV module is represented as the cross-attention module in the denoising UNet.

[0042] This invention evolves from the SD model. Unlike the SD model, which uses text as a guide to generate the desired image, this invention uses images as the guiding information. To achieve image guidance, this invention constructs a reference defect processing module and a reference defect feature encoding module. To enable the model to generate defects at specified locations based on defect edge detail information, this invention constructs a defect location and detail processing module and a defect location and detail encoding module.

[0043] A method for controllably generating defective samples of industrial products based on reference image guidance comprises the following steps:

[0044] Step 1: Perform affine transformation on the input defect image through the reference defect processing module to generate diversified reference defect images.

[0045] The main goal of the training phase is to enable the model to generate similar defects on the input image based on the style of the reference defect.

[0046] During the training phase, the input is an image with defects and the defect location mask ,in Representative coordinates The pixels are defective and the rest are non-defective. .according to The size of the reference defect image can be obtained as , and then segment the defective part image to get the defective part image ,in , Represents matrix element-by-element multiplication. When generating defects, it is necessary to balance the diversity and fidelity of the generated samples (compared with the reference defects). We do not want the model to generate images that are exactly the same as the reference defects, nor do we want to generate images that are too different from the reference defects. Therefore, during training, Perform a random affine transformation. Assume that the randomly generated affine transformation matrix is , then the processed image can be expressed as = .

[0047] Step 2: Use the reference defect feature encoding module to extract the style information of the reference defect image and inject it into the cross-attention module of the denoising UNet as the guidance information for defect generation.

[0048] To ensure that the style of the generated defect image resembles the reference defect, this paper uses a reference defect feature encoding module to extract defect image features as guidance information for the denoising UNet in the SD model. The SD model (High-Resolution Image Synthesis with Latent Diffusion Models, CVPR2022) uses text to guide the denoising UNet. The text is fed into the denoising UNet via the text encoder of CLIP (Learning Transferable Visual Models from Natural Language Supervision, ICML2021). Other work (Objectstitch: Object Compositing with Diffusion Model, CVPR2023 and Paint by example: Exemplar-based Image Editing with Diffusion Models, CVPR2023) uses the image encoder of CLIP to inject the encoded image features into the denoising UNet as guidance information. Because CLIP is trained using coarse text and image pairs, its image encoder cannot capture the fine-grained style information of the defect image. Furthermore, fine-grained defect style information is difficult to describe in text. Therefore, this paper uses a reference defect feature encoding module to extract this fine-grained style information. This module consists of two parts: DINOv2 and linear mapping layer (Linear).

[0049] DINOv2 (DINOv2: Learning Robust Visual Features without Supervision, TMLR2024) is an image encoder pre-trained on a large image dataset. It can obtain the visual features of images and can be used for downstream visual tasks without fine-tuning. In the model of this invention, the parameters of DINOv2 are fixed. As the backbone network of the reference defect feature encoding module, it can be expressed as DINOv2 is a Transformer-based model whose output consists of two parts: global features ( ) and image patch features ( ).

[0050]

[0051] Global Features and image patch features After merging, the complete reference defect image features are obtained , the merging method is to concatenate on the first dimension of the matrix. The linear layer is the trainable part of the module, and the parameters contained in it are expressed as , the linear layer is represented as Reference defect image features Input to the linear layer and finally get the defect feature encoding = , which is input into the cross-attention module in the denoising UNet.

[0052] Step 3: Extract the edge detail information of the defect through the defect location and detail processing module to generate a detail guidance image of the defect.

[0053] The reference defect only provides the style information required to generate the defect. To generate a high-quality defect image, the location and detail information of the generated defect are also required. The defect location and detail processing module provides input information for the subsequent defect location and detail encoding module.

[0054] The original defect image already provides detailed information about the defect. To prevent the diffusion model from directly copying the original defect image into the final output, the defect image needs to be processed to retain the defect edge information as detailed guidance for input into the denoising UNet. This is done to balance the diversity and fidelity of the generated samples.

[0055] First, the input image Convert to grayscale , and then use the gradient-based edge extraction operator ( and ) extract the vertical and horizontal gradient information respectively, and then obtain the edge information of the defect based on the gradient information, and finally perform the corrosion operation Remove some noise data and optimize the edge detail information of defects.

[0056]

[0057] in Represents a convolution operation.

[0058] The training goal of the model is to Specified) generates defects, while the images generated in other areas are the same as the input images, so the details of the final input model guide image It can be expressed as:

[0059]

[0060] in Represents matrix concatenation in the channel dimension.

[0061] Step 4: Use the defect location and detail encoding module to encode the defect detail guidance image to generate another part of the guidance information of the denoising UNet.

[0062] This module uses the output image of the defect location and detail processing module As input information, the image is encoded to generate another part of the guidance information of the denoising UNet. Similar to ControlNet (Adding Conditional Control to Text-to-Image Diffusion Models, ICCV2023), this invention copies the encoder part of the denoising UNet as the defect location and detail encoding module, which is expressed as During model training, the parameters of the denoising UNet are always kept frozen, while the defect location and detail encoding modules The parameters are trainable. The defect location and detail encoding in the final input denoising UNet can be expressed as . It is added to the output of the denoising UNet encoder and used as the input of the denoising UNet decoder.

[0063] Step 5. Construct a defect style loss function based on the Gram matrix and train the model.

[0064] The trainable part of the model includes the linear layer in the reference defect encoding module , defect location coding module The parameters of the rest of the model come from the pre-trained SD model, specifically https: / / huggingface.co / lllyasviel / ControlNet-v1-1 / blob / main / control_v11p_sd15_inpaint.pth used in the experiment. Given an image , after the VAE (Variational Auto Encoder) image encoder (expressed as ), the latent space representation of the image is obtained , using the forward denoising algorithm (Denoising Diffusion Probabilistic Models, NeurIPS2020) Add time steps of noise, we get , where the added noise is expressed as ( ). If the denoising UNet is expressed as , training the model so that it can and To predict the added noise, the loss function used during training is

[0065]

[0066] For denoising UNet, is the time step for adding noise, Adding the latent space representation of the input image The characteristics after step noise, is the feature of the reference defect. Another goal of model training is to make the generated image as consistent as possible with the style of the reference defect. Therefore, the present invention measures the latent space representation of the image after UNet denoising. Latent space representation of reference defects Here we use the Gram matrix The squared error between the two is used to measure the style difference between the two

[0067]

[0068] in Represents the position of the defect area in the latent space, which can be expressed by Obtained through simple scaling. The above loss function ensures only that the generated defects are consistent with the reference defects, without focusing on other areas. This loss function is used to balance the diversity and fidelity of the generated defects.

[0069] The loss function used in the final model training is ,in Used to balance the order of magnitude difference between the two loss functions.

[0070] Step 6: Use the trained model (denoising UNet) to gradually denoise the random noise according to the defect intensity parameters input by the user, the background image, and the location of the defect to be generated, and keep the background area consistent with the input background image to generate the final defect image. That is, use the trained model to infer the desired result based on the user's input. The structure of the inference stage is as follows: Figure 2 shown.

[0071] In the inference phase, DM passes through random noise Denoising is performed in multiple time steps to obtain the desired image. The noise removed at each step is predicted by the UNet, so the non-defective areas of the image generated by the DM (the areas outside the mask, also known as the background area) may vary. The method of the present invention introduces "defect-free background image" and "defect location" information during the inference phase. This allows defects to be generated in a specified area (specified by the mask) based on a reference defect image input, while maintaining the background area as close to the input background image as possible.

[0072] Assume that the given random noise is , as the first Initial value of step denoising Given a reference defect , after processing by the reference defect feature encoding module, the guiding conditions of the denoising UNet are obtained Unlike the training phase, the reference defect does not need to go through the reference defect processing module in the inference phase. , detail contour image of the defect and background image , get the input of defect location and detail encoding module Among them, the detail contour image of the defect It can be drawn manually by the user or be empty (a matrix of all zeros), or it can be obtained from the defect image through an image edge extraction algorithm. It is directly used as the input of the defect location and detail encoding module without going through the defect location and detail processing module, and finally the guiding conditions of the denoising UNet are obtained. Then the denoising UNet predicts the noise that needs to be removed in this step according to the guidance conditions During the step-by-step iterative denoising, the defect area is determined by the denoising UNet prediction result and the background image corresponding to the area, while other areas are directly determined by the input background image. Here, the present invention introduces the defect intensity control parameter , the information used to specify the defect area comes from the ratio of background and denoising UNet.

[0073]

[0074] in and is the parameter of forward noise addition (Denoising Diffusion Probabilistic Models, NeurIPS2020), From the above formula, we can see that when When the defect area is completely determined by the output of the denoising UNet, the guide condition and Generates severe defects, which essentially degenerates into BLDM (Blended Latent Diffusion, SIGGRAPH2023); when When , the output of the denoising UNet is completely discarded, and the final result is only related to the input background image, that is, the defect area is also a normal background image; when When , the image generated by the defect area is determined by the output of the denoising UNet and the background image. Determine which part of the information is dominant.

[0075] During inference, use formula (6) to iterate until ,get , using the VAE image decoder (denoted as ) to get the final generated image .

[0076] Based on the aforementioned method for controllable generation of industrial product defect samples guided by reference images, some generated defect images were collected to verify their quality. The data generated by the present invention was then used to train a defect detection model to verify the effectiveness of the present invention as a data augmentation method for improving the performance of the defect detection model.

[0077] like Figure 2 As shown in the MSD_Seg2 dataset (Yanqi Bao, Kechen Song, Jie Liu, Yanyan Wang, Yunhui Yan, Han Yu, Xingjie Li. Triplet-Graph Reasoning Network for Few-shot Metal Generic Surface Defect Segmentation [J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70, 5011111), it contains 12 defects for six products. In this experiment, one of these products (steel) was selected, containing four defects: abrasion-mask (Am), liquid (Ld), patches (Pa), and scratches (Sc).

[0078] The dataset contains 198 images of steel surface defects and 200 images without defects. All defects are annotated using pixel-level masks, and the image resolution is 200*200 pixels. The dataset is randomly divided into training and test sets in a ratio of approximately 1:4. The sample distribution of the dataset is shown in Table 1.

[0079] Table 1. Sample distribution in the dataset

[0080]

[0081] First, the method of the present invention is trained using the training set, and no defect-free images are used during training. Then, a defect-free image is randomly selected as the "background image" ( Figure 4 First column), two samples are randomly selected from the test set, and the defective part of one sample is used as the “reference defect” ( Figure 4 The second column), another sample mask is used to specify the location where the defect is generated ( Figure 4 The third column) does not specify "defect details", specifies different intensities, and generates defects such as Figure 3 shown. Figure 4 The fourth column is the original image used to obtain the second column of images, and the fifth to eighth columns are the generated defect images, where the defect intensity increases in sequence.

[0082] from Figure 4 It can be seen that the generated defect is visually similar to the input reference defect, and its position can be aligned with the input defect position, indicating that the method of the present invention can controllably generate defects of different intensities at a specified position according to the style of the input defect, proving the effectiveness of the method of the present invention.

[0083] In order to visually observe the quality of generated defects, the present invention randomly selects a sample on the test set and removes the defective part ( Figure 5 The first column) as the "background image", and the cut-out part as the "reference defect" ( Figure 5 The second column shows the location of the defect and the location of the original defect ( Figure 5 The third column is the same as the previous one. Without specifying "defect details", the generated model is as follows Figure 5 shown. Figure 5 The fourth column is the original image, and the fifth column is the generated defect image, where the intensity is specified as maximum.

[0084] from Figure 5 It can be seen from the fourth and fifth columns that the defects generated by the method of the present invention are very similar to real defects, and the defect edge transition is also very natural, indicating that the defects generated by the method of the present invention are very realistic, proving the effectiveness of the method of the present invention.

[0085] Generating defect samples is primarily intended to improve the performance of defect segmentation models. This experiment primarily verifies whether the samples generated by the proposed method are beneficial for training defect segmentation models. In this experiment, the classic defect segmentation method, DeepLabV3+ (Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, ECCV 2018), was used. First, the proposed method was trained on a training set. Then, defect samples were generated from defect-free samples. The DeepLabLabV3+ model was trained using both the "training set" (first row of Table 2) and the "training set + generated samples" (second row of Table 2), respectively. Finally, evaluation metrics related to defect segmentation were calculated on a test set, as shown in Table 2.

[0086] This paper mainly calculates four evaluation indicators of defect segmentation, PA (pixcal-accuracy, pixel accuracy) and mPA (Mean Pixel Accuracy, average pixel accuracy), among which the first The pixel accuracy of class defects is , represents the number of pixels that are correctly classified, Indicates the total number of pixels, ; IoU (Intersection over Union, intersection ratio) and mIoU (Mean Intersection over Union, average intersection ratio), the intersection ratio represents the ratio of the intersection and union of the two sets of true values and predicted values, where the first The intersection-over-union ratio of class defects is , average intersection-union ratio .

[0087] Table 2 The impact of samples generated by the method of the present invention on the defect segmentation model

[0088] As can be seen from Table 2, the defect segmentation model is trained using the samples generated by the method of the present invention plus the original training set samples. The mPA reaches 97.95% and the mIoU reaches 81.38% on the test set, which are improved compared with the defect segmentation model trained without using the generated samples. This shows that the defect samples generated by the method of the present invention can effectively improve the performance of the defect segmentation model, proving the effectiveness of the method of the present invention.

[0089] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A method for controllable generation of industrial product defect samples based on reference image guidance, characterized by: The following steps are involved: Step 1: Perform affine transformation on the input defect image through the reference defect processing module to generate a variety of reference defect images; Step 2: Use the reference defect feature encoding module to extract the style information of the reference defect image and inject it into the cross-attention module of the denoising UNet as the guidance information for defect generation; Step 3: Extract edge detail information of the defect through the defect location and detail processing module to generate a detail guide image of the defect; Step 4: Use the defect location and detail encoding module to encode the defect detail guidance image to generate another part of the guidance information of the denoising UNet; Step 5: Construct a defect style loss function based on the Gram matrix and train the model; Step 6: Based on the defect intensity parameter input by the user, the trained model is used to gradually denoise the random noise through the denoising UNet, and an image with a specified intensity and a reference defect is generated in a given area, while keeping the background image consistent with the input in the background area.

2. The method for controllably generating defective samples of industrial products based on reference image guidance according to claim 1, characterized in that: The specific implementation method of step 1 is: input the image with defects and the defect location mask , H I For the input image Height, W I For the input image Width; in Representative coordinates The pixels are defective and the rest are non-defective. ,according to The size of the reference defect image is , is the height of the reference defect image, is the width of the reference defect image, and then the defect part image is segmented to obtain the defect part image ,in , Represents matrix element-by-element multiplication, and Perform a random affine transformation, and the randomly generated affine transformation matrix is , the processed image is represented as = .

3. The method for controllably generating defective samples of industrial products based on reference image guidance according to claim 2, characterized in that: The specific implementation method of step 2 is: output global features through DINOv2 and image patch features : ； Global Features and image patch features After merging, the complete reference defect image features are obtained , the merging method is to splice on the first dimension of the matrix, referring to the defect image features Input to the linear layer and finally get the defect feature encoding = , the parameters contained in it are expressed as , the linear layer is represented as , which is input into the cross-attention module in the denoising UNet.

4. The method for controllably generating defective samples of industrial products based on reference image guidance according to claim 3, characterized in that: The specific implementation method of step 3 is: Convert to grayscale , using a gradient-based edge extraction operator and Extract the gradient information in the vertical and horizontal directions respectively, obtain the edge information of the defect based on the gradient information, and finally perform the corrosion operation Remove some noise data and optimize the edge details of the defect: ； in represents the convolution operation; therefore, the final input model details guide image Expressed as: ； in Represents matrix concatenation in the channel dimension.

5. The method for controllably generating defective samples of industrial products based on reference image guidance according to claim 4, characterized in that: The specific implementation method of step 4 is: the defect location and detail encoding module guides the image with details For input information, encode the image to generate another part of the guidance information of the denoising UNet , It is added to the output of the denoising UNet encoder and used as the input of the denoising UNet decoder.

6. The method for controllably generating defective samples of industrial products based on reference image guidance according to claim 5, characterized in that: The loss function in step 5 is: ； in, The loss function originally used to train the denoising diffusion probability model DDPM is used to measure the difference between the added noise and the predicted noise. is the latent space representation of the image after UNet denoising Latent space representation of reference defects The style differences between Used to balance the order of magnitude difference between the two loss functions, For denoising UNet, is the time step for adding noise, Adding the latent space representation of the input image The characteristics after step noise, is the characteristic of the reference defect.

Images