Methods, systems, and related devices for generating simulated defect images

Abstract

The application relates to a method, a system and related equipment for generating simulated defect images. The method comprises the following steps: acquiring a first image set and a second image set; training an image generation model combining a low-rank adaptive model and a stable diffusion model using the first image set and the second image set; obtaining a defect region edge image set and a normal sample edge image set; selecting one image from the defect region edge image set and the normal sample edge image set respectively, pasting the defect region edge image to a random position of the normal sample edge image to obtain a simulated defect sample edge image, and simultaneously generating a mask of the pasting area as a label of a simulated defect position; and generating a corresponding simulated defect image through a pre-trained ControlNet and the image generation model by taking the simulated defect sample edge image as a constraint condition, thereby solving the technical problem of insufficient defect samples in the related art when an industrial defect is detected by using a deep learning method.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular to a method, system and related equipment for generating simulated defect images. Background Technology

[0002] Deep learning-based detection methods typically rely heavily on large-scale, precisely labeled training data. In industrial production, product yields are usually high, defect samples are relatively scarce and exhibit a long-tailed distribution, making the collection of certain low-frequency defect samples extremely difficult. Collecting large amounts of high-quality labeled data can be very challenging and costly. The scarcity of defect data limits the model's generalization ability, while the uneven distribution of defect data leads to low recognition rates on these low-frequency defects. Defect image generation technology can learn from existing data distributions to generate a large number of new samples, significantly alleviating the training difficulties caused by data scarcity.

[0003] Most industry practices employ methods based on Generative Adversarial Networks (GANs) and Diffusion Models to generate defect image samples. For example, GAN-based methods train a generator to produce defect images from random noise, while a discriminator distinguishes between real and generated defect images. The generator and discriminator are trained alternately, with multiple rounds of training continuously improving both the generator's generation ability and the discriminator's discrimination ability. Ultimately, this enables the generator to produce sufficiently realistic defect images. However, using GANs to generate defect samples has limitations. Because there is no explicit mapping between the latent vectors of the GAN and the generated images, the location of the generated defect cannot be simultaneously determined during sample generation. On the other hand, Diffusion Model-based methods gradually add noise to real samples to create a forward diffusion process, training a denoising model. This denoising model then gradually recovers the defect image sample from the noise during the backward diffusion process. While this method can generate defects at specified locations, fine-tuning the stable diffusion model requires manual pixel-level annotation of the defect regions in the samples, incurring additional manual costs.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, system, and related equipment for generating simulated defect images, to at least solve the technical problem of insufficient defect samples when using deep learning methods to detect industrial defects in related technologies.

[0006] According to one aspect of the embodiments of this application, a method for generating simulated defect images is provided, comprising: acquiring a first image set containing defect sample images and a second image set containing normal sample images; training an image generation model combining a low-rank adaptation model and a stable diffusion model using the first image set and the second image set; performing edge extraction on the defect sample images in the first image set and the normal sample images in the second image set respectively to obtain a defect region edge image set and a normal sample edge image set; selecting one image from each of the defect region edge image set and the normal sample edge image set, pasting the defect region edge image to a random position in the normal sample edge image to obtain a simulated defect sample edge image, and simultaneously generating a mask of the pasted area as a label for the simulated defect position; using the simulated defect sample edge image as a constraint condition, generating a corresponding simulated defect image through a pre-trained ControlNet and the image generation model.

[0007] Optionally, edge extraction is performed on the defect sample images in the first image set to obtain a defect region edge image set, including: marking the defect location of each defect sample image in the first image set with a box-level annotation to obtain defect annotation data of the first image set; cropping defect region images from the defect sample images based on the defect annotation data of the first image set to obtain a defect region image set; and performing edge extraction on each defect region image in the defect region image set to obtain the defect region edge image set.

[0008] Optionally, training an image generation model combining a low-rank adaptation model and a stable diffusion model using the first image set and the second image set includes: using the low-rank adaptation model to adjust the parameters of the stable diffusion model to optimize the learning ability of the image generation model for the defective sample images and the normal sample images.

[0009] Optionally, pasting the defect region edge image onto a random position of a normal sample edge image to obtain a simulated defect sample edge image includes: randomly generating a defect region of the same size as the defect region edge image on the normal sample edge image, and filling the defect region with the defect region edge image to obtain the simulated defect sample edge image.

[0010] Optionally, the method further includes: randomly generating a transformation matrix for each defect region edge image in the defect region edge image set; performing an affine transformation on each defect region edge image based on the randomly generated transformation matrix to obtain a new defect region edge image; and placing the new defect region edge image into the defect region edge image set.

[0011] According to another aspect of the embodiments of this application, a system for generating simulated defect images is provided, comprising: an image acquisition module for acquiring a first image set containing defect sample images and a second image set containing normal sample images; a model training module for training an image generation model combining a low-rank adaptation model and a stable diffusion model using the first image set and the second image set; an edge extraction module for performing edge extraction on the defect sample images in the first image set and the normal sample images in the second image set respectively, to obtain a defect region edge image set and a normal sample edge image set; a first image generation module for selecting one image from each of the defect region edge image set and the normal sample edge image set, pasting the defect region edge image to a random position in the normal sample edge image to obtain a simulated defect sample edge image, and simultaneously generating a mask of the pasted area as a label for the simulated defect position; and a second image generation module for using the simulated defect sample edge image as a constraint condition, and generating a corresponding simulated defect image through a pre-trained ControlNet and the image generation model.

[0012] Optionally, the edge extraction module includes: a labeling unit, used to perform box-level labeling on the defect location of each defect sample image in the first image set to obtain defect labeling data of the first image set; a cropping unit, used to crop defect region images from the defect sample images according to the defect labeling data of the first image set to obtain a defect region image set; and an extraction unit, used to perform edge extraction on each defect region image in the defect region image set to obtain the defect region edge image set.

[0013] According to another aspect of the embodiments of this application, an electronic device is provided, including: a processor, and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the method described in any one of the above.

[0014] According to another aspect of the present application, a non-transitory machine-readable medium storing computer instructions for causing the computer to perform any of the methods described above is provided.

[0015] According to another aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a computer's processor, causes the computer to perform any of the methods described above.

[0016] In this embodiment, a first image set containing defect sample images and a second image set containing normal sample images are acquired. An image generation model combining a low-rank adaptation model and a stable diffusion model is trained using the first and second image sets. Edge extraction is performed on the defect sample images in the first image set and the normal sample images in the second image set to obtain a defect region edge image set and a normal sample edge image set. One image is selected from each of the defect region edge image set and the normal sample edge image set, and the defect region edge image is pasted onto a random position in the normal sample edge image to obtain a simulated defect sample edge image. Simultaneously, a mask of the pasted area is generated as a label for the simulated defect location. Using the simulated defect sample edge image as a constraint, a corresponding simulated defect image is generated through a pre-trained ControlNet and image generation model. In other words, this embodiment can generate a large number of simulated defect images and their corresponding labeling information even without a large number of real defect samples. This solves the technical problem of insufficient defect samples when using deep learning methods to detect industrial defects in related technologies, achieving the technical effects of reducing dependence on real defect samples, lowering data preparation costs, improving the generalization ability and robustness of the defect detection model, and enhancing the controllability of defect image generation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a method for generating simulated defect images provided in an embodiment of this application;

[0019] Figure 2 A flowchart of a method for generating simulated defect images provided for an optional embodiment of this application;

[0020] Figure 3 A schematic diagram of a system for generating simulated defect images provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that embodiments of this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the embodiments of this application. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0023] According to one aspect of the embodiments of this application, a method for generating simulated defect images is provided. Figure 1 A flowchart of a method for generating simulated defect images provided in embodiments of this application is shown below. Figure 1 As shown, the method includes the following steps:

[0024] Step S102: Obtain a first image set containing defective sample images and a second image set containing normal sample images.

[0025] Step S104: Train an image generation model that combines a low-rank adaptation model and a stable diffusion model using the first image set and the second image set.

[0026] The pre-trained Stable Diffusion model is fine-tuned using a low-rank adaptation model (LoRA), enabling the model to quickly generate images of specific types of defects while maintaining its original performance. This method reduces the need for large-scale datasets and lowers training costs and time.

[0027] Step S106: Edge extraction is performed on the defect sample images in the first image set and the normal sample images in the second image set to obtain the defect region edge image set and the normal sample edge image set.

[0028] Step S108: Select one image from the defect region edge image set and one image from the normal sample edge image set, paste the defect region edge image onto a random position in the normal sample edge image to obtain a simulated defect sample edge image, and simultaneously generate a mask of the pasted area as a label for the simulated defect position.

[0029] Edge extraction is performed on both the defective sample image and the normal sample image to obtain the edge image of the defective region and the edge image of the normal sample. Then, the edge image of the defective region is randomly pasted onto the edge image of the normal sample to form a new simulated defective sample edge image. Simultaneously, a mask is automatically generated to mark the location of the simulated defect as a box-level annotation.

[0030] Step S110: Using the simulated defect sample edge image as a constraint, generate the corresponding simulated defect image through the pre-trained ControlNet and image generation model.

[0031] A pre-trained ControlNet is used to guide the image generation process of the stable diffusion model. ControlNet can use the provided edge map (i.e., the edge image of the simulated defect sample after pasting) as an additional constraint to control the position and shape of defects in the generated image, ensuring that the generated simulated defect image is more realistic, accurate, and meets actual needs.

[0032] The simulated defect images described above can be used as defect sample images for training image generation models.

[0033] In this embodiment, a first image set containing defect sample images and a second image set containing normal sample images are acquired. An image generation model combining a low-rank adaptation model and a stable diffusion model is trained using the first and second image sets. Edge extraction is performed on the defect sample images in the first image set and the normal sample images in the second image set to obtain a defect region edge image set and a normal sample edge image set. One image is selected from each of the defect region edge image set and the normal sample edge image set, and the defect region edge image is pasted onto a random position in the normal sample edge image to obtain a simulated defect sample edge image. Simultaneously, a mask of the pasted area is generated as a label for the simulated defect location. Using the simulated defect sample edge image as a constraint, a corresponding simulated defect image is generated through a pre-trained ControlNet and image generation model. In other words, this embodiment can generate a large number of simulated defect images and their corresponding labeling information even without a large number of real defect samples. This solves the technical problem of insufficient defect samples when using deep learning methods to detect industrial defects in related technologies, achieving the technical effects of reducing dependence on real defect samples, lowering data preparation costs, improving the generalization ability and robustness of the defect detection model, and enhancing the controllability of defect image generation.

[0034] As an optional embodiment, edge extraction is performed on the defect sample images in the first image set to obtain a defect region edge image set, including: marking the defect location of each defect sample image in the first image set with a box-level annotation to obtain defect annotation data of the first image set; cropping defect region images from the defect sample images based on the defect annotation data of the first image set to obtain a defect region image set; and performing edge extraction on each defect region image in the defect region image set to obtain a defect region edge image set.

[0035] The above box-level annotation is used to indicate the position and size of the target object by drawing a box around it.

[0036] To determine the approximate location of defects in each defect sample image, bounding box annotations are needed for the defect sample images in the first image set. Only the smallest rectangle containing the defect needs to be marked on the image, without pixel-level precision. This not only reduces the workload and cost of manual annotation but also provides sufficient information for subsequent steps.

[0037] Based on the bounding box-level annotation data described above, the region containing the defect is cropped from each defect sample image to form a defect region image set. The cropping process should be as close as possible to the bounding box annotations to ensure that the defect region is completely preserved, while minimizing the retention of non-defect regions to facilitate subsequent edge extraction operations.

[0038] For the obtained set of defect region images, edge detection algorithms (such as Canny, Sobel, etc.) are used to extract edges from each defect region image. The result of edge extraction is a binary image that highlights the contours of objects in the image, i.e., the set of defect region edge images. Edge images are the basis for subsequent affine transformations and pasting operations because they provide shape and boundary information of the defects, which helps to accurately simulate defects on normal sample images.

[0039] In this embodiment, by employing box-level annotation instead of pixel-level annotation, the cost and time of manual annotation are significantly reduced, while the efficiency of data preparation is improved. This method provides an efficient and low-cost solution for generating defect image samples and their corresponding defect location annotations, which is particularly suitable for fields such as industrial product quality inspection, where the number of defect samples may be very limited. This method not only significantly reduces the workload of manual annotation but also ensures that the generated defect samples have sufficient diversity and realism, thereby improving the performance of deep learning-based defect detection models.

[0040] As an optional embodiment, an image generation model combining a low-rank adaptation model and a stable diffusion model is trained using a first image set and a second image set, including: using the low-rank adaptation model to adjust the parameters of the stable diffusion model to optimize the learning ability of the image generation model for defective sample images and normal sample images.

[0041] The first image set is a dataset containing defective sample images; the second image set is a dataset containing normal sample images, which are images of products or environments without defects.

[0042] Choose a pre-trained stable diffusion model as the base model. This model has learned the distribution features of a large number of images and can generate high-quality images. Construct a low-rank adaptive model, which adjusts the original model by introducing a small number of parameters, thereby fine-tuning it for a specific task without affecting most of the parameters of the original model.

[0043] Low-rank adaptation layers are inserted into the key layers of the stable diffusion model. These layers have small matrix weights to capture the differences between defective and normal samples. The model is jointly trained using a first image set and a second image set, with defective and normal samples alternately input for each batch of training data to ensure that the model can learn both types of image features simultaneously. Appropriate loss functions are designed, such as combining adversarial loss, reconstruction loss, and perceptual loss, to guide the model to generate more realistic defective images and improve its ability to distinguish between defective and normal samples. Regularization techniques, such as L2 regularization, can also be used to prevent overfitting, especially when there are few defective samples.

[0044] In this embodiment, the low-rank adaptation model requires only a small number of parameter adjustments, thus enabling it to quickly adapt to new task requirements while maintaining the performance of the original model, reducing training time and computational resource consumption. By adding a low-rank adaptation layer to the stable diffusion model, the model can learn the features of defective and normal samples without changing most parameters, improving its generalization ability to different defect types. Due to the effectiveness of the low-rank adaptation model, good learning results can be achieved even with a small number of defective samples, reducing the dependence on large-scale labeled data.

[0045] As an optional embodiment, pasting the defect region edge image onto a random position of a normal sample edge image to obtain a simulated defect sample edge image includes: randomly generating a defect region of the same size as the defect region edge image on the normal sample edge image, filling the defect region with the defect region edge image, and obtaining a simulated defect sample edge image.

[0046] The defect region edge image set is a collection of defect sample images that have undergone edge extraction processing; each image contains only the edge information of the defect. The normal sample edge image set is a collection of images obtained by edge extraction from normal sample images, used to provide the basis for pasting the defect region.

[0047] One defect region edge image is randomly selected from the defect region edge image set, and one normal sample edge image is randomly selected from the normal sample edge image set. Ensure that the selected defect region edge image and the normal sample edge image have the same resolution; if their sizes are inconsistent, they can be matched by scaling, cropping, or padding. A rectangle is randomly generated on the normal sample edge image to represent the location of the defect region. The size of this rectangle should match the selected defect region edge image. Ensure that the randomly generated rectangle does not exceed the boundary of the normal sample edge image; if the rectangle partially exceeds the boundary, its position needs to be adjusted so that it is completely within the image. The defect region edge image is placed within the randomly generated rectangle on the normal sample edge image. This can be accomplished by simple pixel replacement, i.e., replacing the corresponding pixel values ​​in the normal sample edge image with the pixel values ​​from the defect region edge image. Simultaneously, a binary mask is generated to represent the location of the defect region; the defect region position in the mask is 1, and other positions are 0. This mask can be used as a label for the defect location in subsequent generation processes. The final result is a simulated defect sample edge image and its corresponding defect location mask.

[0048] In addition, depending on the application scenario, some additional processing can be performed on the pasted image, such as smoothing the transition to make the defect area blend more naturally with the surrounding environment, or applying affine transformations (such as rotation, scaling, and translation) to increase the diversity of defects.

[0049] In this embodiment, an automated pasting process can generate a large number of simulated defect samples and their corresponding bounding box annotations without additional manual intervention, significantly reducing the cost and time of manual annotation. Randomly selecting defect region edge images and pasting locations, and performing additional transformations on the pasted images, increases the diversity and randomness of the generated defects, helping to improve the model's generalization ability. Since edge images of the defect regions are used, the shape and structural features of the defects are well preserved during the pasting process, ensuring that the generated defect samples are sufficiently realistic. Each generated simulated defect sample edge image is accompanied by a corresponding defect location annotation, simplifying the data preparation process. The large number of simulated defect samples generated in this way can effectively supplement the deficiencies of actual defect samples, improve the data distribution for model training, and thus enhance the performance of deep learning-based defect detection models. This method can adjust the size, position, and number of defect regions according to different needs, making it suitable for various types of defect detection tasks and offering high flexibility.

[0050] As an optional embodiment, the above method further includes: randomly generating a transformation matrix for each defect region edge image in the defect region edge image set; performing an affine transformation on each defect region edge image based on the randomly generated transformation matrix to obtain a new defect region edge image; and placing the new defect region edge image into the defect region edge image set.

[0051] Optionally, based on the selected transformation type and random parameters, a corresponding affine transformation matrix is ​​randomly generated for each image in the defect region edge image set. The transformation types include, but are not limited to, rotation, scaling, translation, and shearing. A reasonable parameter range is set for each transformation type using random parameters. For example, for rotation, an angle range (e.g., -30 degrees to +30 degrees) can be set; for scaling, a ratio range (e.g., 0.8 to 1.2 times) can be set.

[0052] The process iterates through the image set of defect region edges, applying the corresponding affine transformation matrix to each image. During the affine transformation, it's crucial to ensure the transformed image does not exceed the boundaries of the original image. If any portion of the transformed image exceeds the boundaries, cropping, padding, or other methods can be used to handle the excess portion, ensuring the transformed image size matches the original. Figure 1 The new defect region edge image, after affine transformation, is added to the defect region edge image set for subsequent pasting operations. It should be noted that you can choose to replace the original image or add a new image, depending on whether you want to preserve the original defect features.

[0053] Furthermore, as described previously, images are randomly selected from the updated set of defect region edge images and pasted onto normal sample edge images to generate simulated defect sample edge images. Since affine transformations may alter the shape and location of defects, some additional adjustments to the pasting position may be necessary during the pasting process to ensure the defect region is completely within the normal sample edge image.

[0054] In this embodiment, randomly generated affine transformations can generate more variations from the originally limited defect samples, greatly increasing the diversity of defect morphology and location. Introducing affine transformations into defect samples helps the model learn a wider range of defect features, making the model more robust and better able to adapt to different environments and conditions in practical applications. Defects on industrial products are often not fixed but change due to various factors during the manufacturing process. Affine transformations can more realistically simulate these changes, making the generated defect samples closer to reality and helping to train a better-performing detection model. Since all transformations are performed automatically without human intervention, and corresponding bounding box annotations are automatically generated, low manual costs can be maintained even after performing a large number of affine transformations.

[0055] The optional embodiments of this application are described in detail below.

[0056] Figure 2 A flowchart of a method for generating simulated defect images, provided as an optional embodiment of this application, is shown below. Figure 2 As shown, the method includes the following steps: acquiring data, training an image generation model, cropping defect regions, edge extraction, affine transformation, generating edge images, and generating defect images. The specific implementation process of each step is as follows:

[0057] Data acquisition: Acquire a set of defective sample images (corresponding to the first image set mentioned above) and a set of normal sample images (corresponding to the second image set mentioned above); mark the defect locations in the defective sample images with bounding boxes.

[0058] Training the image generation model: Train an image generation model that combines a low-rank adaptation model (LoRA) and a stable diffusion model using the original defective sample image set and the original normal sample image set.

[0059] Defect region cropping: Based on the defect annotation data of the original defect image sample set, the defect region image is cropped to obtain the defect region image.

[0060] Edge extraction: Edge extraction is performed on the defective region image to obtain a defective edge image set; edge extraction is performed on the normal image sample set to obtain a normal sample edge image set.

[0061] Affine transformation: A transformation matrix is ​​randomly generated for each sample in the defect region edge image sample set; based on the randomly generated transformation matrix, an affine transformation is performed on each sample.

[0062] Generate edge images: Randomly select a defect region edge image from the defect region edge image set; randomly generate a defect region of the same size as the defect region edge image on any normal sample edge image in the normal sample edge image set; fill the defect region with the defect region edge image, and repeat the above process to obtain a large number of simulated defect edge images.

[0063] Generate defect images: Using simulated defect edge images as additional constraints, generate several simulated defect images using a pre-trained ControlNet and a generative model.

[0064] This application obtains the edge map of the defect by performing edge detection on the cropped defect region image and pastes it to a random position on the edge map of the normal sample. Using ControlNet, the edge map is introduced as an additional constraint when generating simulated defect sample images, thus controlling the position and shape of the generated defects and automatically generating bounding box-level annotations for the defect locations. Compared with existing technologies, the method proposed in this application only requires manual bounding box-level annotation of the defect sample images, eliminating the need for pixel-level annotation. Furthermore, annotations can be generated simultaneously when generating defect sample images, significantly reducing manual annotation costs. By fine-tuning Stable Diffusion using LoRA, the prior knowledge gained during model pre-training is fully utilized, and random affine transformations are applied to the defect region edge map, resulting in sufficiently diverse generated defects.

[0065] According to another aspect of the embodiments of this application, a system for generating simulated defect images is provided. Figure 3 This is a schematic diagram of a system for generating simulated defect images provided in an embodiment of this application, such as... Figure 3 As shown, the system for generating simulated defect images includes: an image acquisition module 302, a model training module 304, an edge extraction module 306, a first image generation module 308, and a second image generation module 310. The system will be described in detail below.

[0066] Image acquisition module 302 is used to acquire a first image set containing defective sample images and a second image set containing normal sample images;

[0067] The model training module 304 is connected to the image acquisition module 302 and is used to train an image generation model that combines a low-rank adaptation model and a stable diffusion model using the first image set and the second image set.

[0068] The edge extraction module 306 is connected to the model training module 304 and is used to extract edges from defect sample images in the first image set and normal sample images in the second image set, respectively, to obtain a defect region edge image set and a normal sample edge image set.

[0069] The first image generation module 308 is connected to the edge extraction module 306 mentioned above. It is used to select one image from the defect region edge image set and the normal sample edge image set, paste the defect region edge image onto a random position of the normal sample edge image to obtain a simulated defect sample edge image, and at the same time generate a mask of the pasted area as a label for the simulated defect position.

[0070] The second image generation module 310, connected to the first image generation module 308, is used to generate corresponding simulated defect images by using the simulated defect sample edge image as a constraint condition and through a pre-trained ControlNet and image generation model.

[0071] In the embodiments of this application, the system can generate a large number of simulated defect images and their corresponding annotation information without a large number of real defect samples. This solves the technical problem of insufficient defect samples when using deep learning methods to detect industrial defects in related technologies. It achieves the technical effects of reducing dependence on real defect samples, reducing data preparation costs, improving the generalization ability and robustness of defect detection models, and enhancing the controllability of defect image generation.

[0072] It should be noted that the image acquisition module 302, model training module 304, edge extraction module 306, first image generation module 308 and second image generation module 310 mentioned above correspond to steps S102 to S110 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment.

[0073] Optionally, the edge extraction module includes: a labeling unit, used to label the defect location of each defect sample image in the first image set at the box level to obtain defect labeling data of the first image set; a cropping unit, used to crop defect region images from the defect sample images according to the defect labeling data of the first image set to obtain a defect region image set; and an extraction unit, used to extract edges from each defect region image in the defect region image set to obtain a defect region edge image set.

[0074] As an optional embodiment, the above-mentioned model training module 304 includes: a parameter adjustment unit, used to adjust the parameters of the stable diffusion model using a low-rank adaptation model to optimize the learning ability of the image generation model for defective sample images and normal sample images.

[0075] As an optional embodiment, the first image generation module 308 includes a filling unit, which is used to randomly generate a defect region of the same size as the defect region edge image on the normal sample edge image, and fill the defect region edge image into the defect region to obtain a simulated defect sample edge image.

[0076] As an optional embodiment, the system further includes: an affine transformation module, used to randomly generate a transformation matrix for each defect region edge image in the defect region edge image set, perform an affine transformation on each defect region edge image based on the randomly generated transformation matrix to obtain a new defect region edge image, and put the new defect region edge image into the defect region edge image set.

[0077] According to another aspect of the embodiments of this application, an electronic device is provided, including: a processor, and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the methods of the embodiments of this application.

[0078] According to another aspect of the present application, a non-transitory machine-readable medium storing computer instructions for causing a computer to perform the methods of the present application is provided.

[0079] According to another aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a computer's processor, causes the computer to perform the methods of the embodiments of this application.

[0080] refer to Figure 4 The present invention describes a structural block diagram of an electronic device that can serve as a server or client in embodiments of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0081] like Figure 4 As shown, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0082] Multiple components in the electronic device are connected to I / O interface 405, including: input unit 406, output unit 407, storage unit 408, and communication unit 409. Input unit 406 can be any type of device capable of inputting information into the electronic device. Input unit 406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 408 may include, but is not limited to, disks and optical discs. Communication unit 409 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0083] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of this application may be implemented as a computer program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0084] Computer programs used to implement the methods of the embodiments of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0085] In the context of embodiments of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0086] It should be noted that the term "comprising" and its variations used in the embodiments of this application are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of this application are illustrative and not restrictive. Those skilled in the art should understand that, unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0087] The steps described in the method embodiments provided in this application can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of this application is not limited in this respect.

[0088] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence from or alternative to other embodiments. The various embodiments in this specification are described in a related manner, with reference to each other for similar or identical parts. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0089] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for generating simulated defect images, characterized in that, include: Obtain a first image set containing defective sample images and a second image set containing normal sample images; An image generation model combining a low-rank adaptation model and a stable diffusion model is trained using the first image set and the second image set; Edge extraction is performed on defective sample images in the first image set and normal sample images in the second image set to obtain defective region edge image set and normal sample edge image set; One image is selected from the defect region edge image set and one image is selected from the normal sample edge image set. The defect region edge image is pasted onto a random position in the normal sample edge image to obtain a simulated defect sample edge image. At the same time, a mask of the pasted area is generated as a label for the simulated defect position. Using the simulated defect sample edge image as a constraint, corresponding simulated defect images are generated through a pre-trained ControlNet and the image generation model.

2. The method according to claim 1, characterized in that, Edge extraction is performed on the defect sample images in the first image set to obtain a defect region edge image set, including: The defect locations of each defect sample image in the first image set are annotated with bounding boxes to obtain the defect annotation data of the first image set. Based on the defect annotation data of the first image set, the defect region image is cropped from the defect sample image to obtain the defect region image set; Edge extraction is performed on each defect region image in the defect region image set to obtain the defect region edge image set.

3. The method according to claim 1, characterized in that, An image generation model combining a low-rank adaptation model and a stable diffusion model is trained using the first image set and the second image set, including: The low-rank adaptation model is used to adjust the parameters of the stable diffusion model to optimize the learning ability of the image generation model for the defective sample image and the normal sample image.

4. The method according to claim 1, characterized in that, The defect region edge image is pasted onto a random position in the normal sample edge image to obtain a simulated defect sample edge image, including: A defect region of the same size as the defect region edge image is randomly generated on the normal sample edge image, and the defect region edge image is filled into the defect region to obtain the simulated defect sample edge image.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: A transformation matrix is ​​randomly generated for each defect region edge image in the defect region edge image set. Based on the randomly generated transformation matrix, an affine transformation is performed on each defect region edge image to obtain a new defect region edge image. The new defect region edge image is then placed into the defect region edge image set.

6. A system for generating simulated defect images, characterized in that, include: The image acquisition module is used to acquire a first image set containing defective sample images and a second image set containing normal sample images; The model training module is used to train an image generation model that combines a low-rank adaptation model with a stable diffusion model using the first image set and the second image set; The edge extraction module is used to extract edges from the defect sample images in the first image set and the normal sample images in the second image set, respectively, to obtain a defect region edge image set and a normal sample edge image set; The first image generation module is used to select one image from the defect region edge image set and one image from the normal sample edge image set, paste the defect region edge image onto a random position of the normal sample edge image to obtain a simulated defect sample edge image, and at the same time generate a mask of the pasted area as a label for the simulated defect position. The second image generation module is used to generate corresponding simulated defect images by using the edge images of the simulated defect samples as constraints and through the pre-trained ControlNet and the image generation model.

7. The system according to claim 6, characterized in that, The edge extraction module includes: The annotation unit is used to annotate the defect location of each defect sample image in the first image set at the box level to obtain the defect annotation data of the first image set. The cropping unit is used to crop out the defect region image from the defect sample image based on the defect annotation data of the first image set to obtain the defect region image set. An extraction unit is used to extract edges from each defect region image in the defect region image set to obtain the defect region edge image set.

8. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 5.

9. A non-transitory machine-readable medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When executed by the computer's processor, the computer program is used to cause the computer to perform the method of any one of claims 1 to 5.

Images