Defect image generation method, generation device, generation system and storage medium

Abstract

The invention discloses a defect image generation method, a defect image generation device, a defect image generation system and a computer readable storage medium. The defect image generation method provided by the embodiment of the invention comprises the steps that a plurality of defect training sets with different defect types are acquired, and each defect training set corresponds to one defect type; according to each defect training set, finely adjusting one corresponding local redrawing model; obtaining a background image and a defect mark on the background image; and according to each local redrawing model, generating a defect corresponding to the defect type on the background image based on the defect label to obtain a defect image. Thus, the fusion degree and the visual effect of the generated defect and the background are good, high-quality and diversified defect images can be obtained, production line migration is facilitated in an industrial detection project, and the project period can be effectively shortened.

Description

Technical Field

[0001] The present application relates to the technical field of image generation, and in particular to a defect image generation method, a defect image generation device, a defect image generation system and a computer-readable storage medium. Background Art

[0002] In recent years, industrial defect detection models based on deep learning have been widely used in various industrial inspection projects, greatly improving the inspection efficiency. However, such models need to obtain a large amount of high-quality defect data before they can start model optimization, which results in a long data collection cycle during the production line migration process. Therefore, from the perspective of data processing, how to expand defect images in large quantities in the early stages of industrial inspection projects is an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide a defect image generation method, a defect image generation device, a defect image generation system and a computer-readable storage medium to solve at least one of the above-mentioned technical problems.

[0004] The defect image generation method of the embodiment of the present application includes:

[0005] Acquire a plurality of defect training sets with different defect types, wherein each defect training set corresponds to one defect type;

[0006] Fine-tune a corresponding local redrawing model according to each of the defect training sets;

[0007] Acquire a background image and defect annotations on the background image;

[0008] According to each of the local redrawing models, a defect corresponding to the defect type is generated on the background image based on the defect annotation to obtain a defect image.

[0009] In some embodiments, obtaining a plurality of defect training sets with different defect types includes:

[0010] Acquire a defect sample set, where the defect sample set includes a plurality of defect samples;

[0011] Obtaining the defect type of the defect included in each of the defect samples;

[0012] The plurality of defect samples are classified according to the defect type to obtain a plurality of defect training sets.

[0013] In some embodiments, fine-tuning a corresponding local redrawing model according to each defect training set includes:

[0014] respectively setting text prompt words corresponding to each of the defect training sets;

[0015] Obtaining the segmentation annotation corresponding to each defect sample in each defect training set;

[0016] According to each of the defect training sets, the text prompt words corresponding to each of the defect training sets, and the segmentation annotations corresponding to each of the defect samples, a corresponding local redrawing model is fine-tuned.

[0017] In some embodiments, generating a defect corresponding to the defect type on the background image based on the defect annotation according to each of the local redrawing models to obtain a defect image includes:

[0018] According to each of the local redrawing models, a defect corresponding to the defect type is generated on the background image based on the defect annotation and the corresponding text prompt word to obtain the defect image.

[0019] In some embodiments, the obtaining of the background image and the defect annotation on the background image includes:

[0020] Acquire the background image and a first fine annotation set by a user on the background image;

[0021] The step of generating a defect corresponding to the defect type on the background image based on the defect annotation according to each of the local redrawing models to obtain a defect image includes:

[0022] According to each of the local redrawing models, a defect corresponding to the defect type and having a defect shape and a defect position corresponding to the first fine annotation is generated on the background image to obtain the defect image.

[0023] In some embodiments, the obtaining of the background image and the defect annotation on the background image includes:

[0024] Acquire the background image, a coarse annotation set by a user on the background image, and a second fine annotation randomly generated based on the coarse annotation;

[0025] The step of generating a defect corresponding to the defect type on the background image based on the defect annotation according to each of the local redrawing models to obtain a defect image includes:

[0026] According to each of the local redrawing models, a defect corresponding to the defect type and having a defect shape corresponding to the second fine annotation is generated at a position on the background image corresponding to the coarse annotation to obtain the defect image.

[0027] In some embodiments, after generating a defect corresponding to the defect type based on the defect annotation on the background image according to each of the local redrawing models to obtain a defect image, the defect image generation method further includes:

[0028] Determine and generate a sample set according to the defect image and the corresponding defect annotation;

[0029] Obtain a real sample set, and input the generated sample set and the real sample set into a verification model;

[0030] The verification model is trained by using the generated sample set and the real sample set to verify the quality of the generated sample set.

[0031] The defect image generating device of the embodiment of the present application includes:

[0032] A first acquisition module is used to acquire a plurality of defect training sets with different defect types, wherein each defect training set corresponds to one defect type;

[0033] A model fine-tuning module, used for fine-tuning a corresponding local redrawing model according to each defect training set;

[0034] A second acquisition module is used to acquire a background image and defect annotations on the background image;

[0035] An image generation module is used to generate defects corresponding to the defect type on the background image based on the defect annotation according to each of the local redrawing models, so as to obtain a defect image.

[0036] The defect image generation system of the embodiment of the present application includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the defect image generation method of any of the above embodiments is implemented.

[0037] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the program is executed by a processor, the defect image generating method of any of the above-mentioned embodiments is implemented.

[0038] In the defect image generation method, defect image generation device, defect image generation system and computer-readable storage medium of the embodiment of the present application, a corresponding local redrawing model is fine-tuned according to each defect training set, and a defect of the corresponding defect type is generated on the background image based on the defect annotation according to each local redrawing model to obtain a defect image. In this way, the generated defect has a good degree of fusion with the background and a good visual effect, and high-quality and diversified defect images can be obtained. In industrial inspection projects, it is convenient to migrate production lines and can effectively shorten the project cycle.

[0039] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0041] Figure 1 is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0042] Figure 2 is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0043] Figure 3 is a schematic diagram of the principle of a local redrawing model of certain embodiments of the present application;

[0044] Figure 4 is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0045] Figure 5 is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0046] Figure 6 is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0047] Figure 7 is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0048] Figure 8 is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0049] Fig. 9 is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0050] Fig.10is a flowchart of a defect image generation method according to certain embodiments of the present application;

[0051] Fig.11 is a schematic diagram of a module of a defect image generating device according to some embodiments of the present application;

[0052] Fig.12 is a schematic diagram of a module of a defect image generation system according to some embodiments of the present application;

[0053] Fig.13 It is a schematic diagram of the connection status between a computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0054] The following is a further description of the embodiments of the present application in conjunction with the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements or elements with the same or similar functions from beginning to end. In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the present application.

[0055] In recent years, industrial defect detection models based on deep learning have been widely used in various industrial inspection projects, greatly improving the inspection efficiency. Industrial defect detection models require a large amount of training data and have high quality requirements. Only after obtaining a large amount of high-quality defect data can the model optimization work be started, resulting in a long data collection cycle during the production line migration process.

[0056] At present, in the field of industrial inspection, defect injection is used to generate defects. Through image processing, script processing and other methods, workers inject the collected defects or similar defects in past projects into the normal sample images collected on site, thereby expanding the model training set. However, this method lacks learning ability, and the generated defect samples are relatively simple, and the quality gap between them and real defect samples is large.

[0057] In academia and on some production lines, deep learning-based image generation algorithms are used, among which the diffusion model is the most popular. The diffusion model can be fine-tuned and learned based on a small number of images, and then generated in large quantities, which is better than defect injection in terms of generation diversity. However, this method binds defects to the background and cannot be used for production line migration.

[0058] Therefore, please refer to Figure 1 and Figure 2 , the embodiment of the present application provides a defect image generation method. The defect image generation method comprises:

[0059] 010: Obtain multiple defect training sets with different defect types, where each defect training set corresponds to one defect type;

[0060] 020: Fine-tune a corresponding local redrawing model according to each defect training set;

[0061] 030: Obtain a background image and defect annotations on the background image;

[0062] 040: According to each local redrawing model, a defect of a corresponding defect type is generated on the background image based on the defect annotation to obtain a defect image.

[0063] In the defect image generation method of the embodiment of the present application, a corresponding local redrawing model is fine-tuned according to each defect training set, and a defect of the corresponding defect type is generated on the background image based on the defect annotation according to each local redrawing model to obtain a defect image. In this way, the generated defect has a good degree of fusion with the background and a good visual effect, and high-quality and diversified defect images can be obtained. In industrial inspection projects, it is convenient to migrate production lines and can effectively shorten the project cycle.

[0064] Specifically, multiple defect training sets are obtained, each defect training set corresponds to a defect type, and the defect types corresponding to the multiple defect training sets are different. According to each defect training set, a local redrawing model can be fine-tuned respectively, so that the local redrawing model learns the defect characteristics of a defect type, thereby generating defects of the defect type corresponding to the defect training set.

[0065] Based on multiple defect training sets, multiple local repainting models can be fine-tuned. The local repainting model can use the Stable Diffuion V1.5 Inpainting model. The model principle is as follows: Figure 3 As shown in the figure, compared with the ordinary diffusion model, the Stable Diffuion V1.5 Inpainting model adds the encoding information of the original image in each step of the model denoising, so that the unmasked area remains intact, that is, only the local area in the original image is regenerated.

[0066] After fine-tuning is completed, the background image for defect generation and the defect annotation on the background image are obtained. In industrial inspection projects, the background image may be an image including industrial devices. The defect annotation may characterize the location and / or defect type and / or defect morphology of the defect to be generated. According to different local redrawing models, defects of different defect types may be generated on the background image based on the defect annotation to obtain a defect image.

[0067] It can be understood that a background image can be input into different local redrawing models with different defect annotations to generate multiple defect images with different defect categories. A background image can also be input into the same local redrawing model with different defect annotations to generate multiple defect images with the same defect category.

[0068] In the related technology, defects are generated by defect injection. This method lacks learning ability and has a poor degree of fusion between defects and background. Especially when there are large differences in industrial devices between different projects, colors and textures will not correspond, resulting in extremely poor visual effects after injection. There is a large gap in quality and diversity between the generated defect samples and the real defect samples, which is not conducive to the learning of subsequent industrial defect detection models.

[0069] In the implementation mode of the present application, defect images are generated based on the local redrawing model. The local redrawing model can learn the features of defects, and after fine-tuning, generate defects based on defect annotations on the background image. Each local redrawing model only learns the features of one defect type and is used to generate defects of one defect type. The generated defects have a good degree of fusion with the background and a good visual effect, and high-quality and diversified defect images can be obtained.

[0070] Among other related technologies, a deep learning-based image generation algorithm is used, and the generated defects are bound to the background. When the production line is migrated, it is necessary to collect data again and train the generation model under the new production line, which results in a longer project cycle for industrial inspection projects.

[0071] In the implementation mode of the present application, by inputting different background images, defects can be generated on different background images based on the local redrawing model, so that when the production line is migrated, only the background image under the new production line needs to be collected to generate the defect image corresponding to the new production line. In this way, the project cycle can be effectively shortened.

[0072] See also Figure 4 In some embodiments, a plurality of defect training sets (i.e., 010) with different defect types are obtained, including:

[0073] 011: Obtain a defect sample set, where the defect sample set includes multiple defect samples;

[0074] 012: Obtain the defect types of defects included in each defect sample;

[0075] 013: Classify multiple defect samples according to defect types to obtain multiple defect training sets.

[0076] Specifically, the defect training set is obtained by classifying the defect sample set, and the defect sample set includes multiple defect samples, and the defect samples can be obtained from other production lines. Each defect sample includes at least one defect of at least one defect type. The multiple defect samples are image-classified according to the defect type of the defect included in each defect sample to obtain multiple defect training sets with different defect types.

[0077] Since fine-tuning of a local redrawing model only requires defects of one defect type, when the defect sample includes multiple defects of multiple defect types, it is necessary to eliminate the defects of the redundant defect types in the defect sample and only retain the defects of one defect type.

[0078] For example, if the multiple defect samples in the defect sample set include defects of three defect types in total, then three defect training sets can be obtained according to the defect sample set, namely, a first defect training set corresponding to defect type A, a second defect training set corresponding to defect type B, and a third defect training set corresponding to defect type C. Defect sample X includes defects of defect type A, defects of defect type B, and defects of defect type C. If the defects of defect type B and defect type C in defect sample X are eliminated, defect sample X can be classified into the first defect training set. If the defects of defect type A and defect type C in defect sample X are eliminated, defect sample X can be classified into the second defect training set. If the defects of defect type A and defect type B in defect sample X are eliminated, defect sample X can be classified into the third defect training set.

[0079] It should be noted that since the local redrawing model is a model obtained by pre-training based on a large-scale data set and has good generalization ability, when fine-tuning the local redrawing model, only a small number of defect samples are needed to make the local redrawing model adapt to the overall image style and local defect characteristics of the industrial scene. That is to say, in the implementation mode of the present application, each defect training set can only include a small number of defect samples, for example, each defect training set can only include 3-5 defect samples. This is conducive to shortening the project cycle of industrial inspection projects.

[0080] See also Figure 2 and Figure 5 In some embodiments, a local redrawing model (i.e., 020) is fine-tuned according to each defect training set, including:

[0081] 021: Set the text prompt words corresponding to each defect training set respectively;

[0082] 022: Obtain the segmentation annotation corresponding to each defect sample in each defect training set;

[0083] 023: According to each defect training set, the text prompt word corresponding to each defect training set, and the segmentation annotation corresponding to each defect sample, fine-tune a corresponding local redrawing model.

[0084] Specifically, the Dreambooth algorithm can be used to fine-tune the local redrawing model. The Dreambooth algorithm fine-tunes the model by injecting a custom theme into the local redrawing model. Users can use only a few pictures of a specific object as input to allow the local redrawing model to learn the features of the object and generate new pictures including the object under different text prompts. The generated new pictures can maintain the main features of the object while presenting a variety of scenes and postures.

[0085] The process of fine-tuning the local redrawing model based on the Dreambooth algorithm is as follows: set the text prompt words corresponding to each defect training set. For example, the text prompt word of a defect training set can be set to "a photo of olis", where "olis" is a rare word that cannot be divided, which can avoid being interfered by the prior knowledge in the local redrawing model.

[0086] It should be noted that, since each defect training set fine-tunes a local redrawing model accordingly, multiple local redrawing models subsequently independently generate defects of corresponding defect types without interfering with each other. Therefore, the text prompt words corresponding to multiple defect training sets can be the same or different.

[0087] In addition, it is also necessary to obtain the segmentation annotation of each defect sample in each defect training set. In the defect training set, each defect sample includes a corresponding segmentation annotation. The segmentation annotation can be a manual defect annotation or an annotation during inspection in other production lines. For example, the defect sample can be manually annotated based on the Labelme annotation tool to obtain the segmentation annotation. The segmentation annotation includes the defect outline and defect category. The outline can characterize the defect location and defect morphology.

[0088] Each defect training set, the corresponding text prompt word, and the segmentation annotation corresponding to each defect sample are input into a corresponding local redrawing model. The segmentation annotation forms a mask on the corresponding defect sample. The local redrawing model learns the mask area in the defect sample and binds the text prompt word to the defect type corresponding to the defect training set to complete the fine-tuning of the local redrawing model.

[0089] It should be noted that by inputting the segmentation annotations of the defect samples into the local redrawing model to fine-tune the local redrawing model, the learning of the background and defects in the defect samples can be decoupled, so that the local redrawing model can focus on learning the features of the defects and improve the quality of the subsequently generated defect images.

[0090] See also Figure 6 In some embodiments, according to each local redrawing model, a defect corresponding to the defect type is generated on the background image based on the defect annotation to obtain a defect image (ie, 040), including:

[0091] 041: According to each local redrawing model, a defect of a corresponding defect type is generated on the background image based on the defect annotation and the corresponding text prompt word to obtain a defect image.

[0092] Specifically, after the local redrawing model is fine-tuned, the background image, the defect annotation on the background image, and the corresponding text prompt words are input into the local redrawing model. For example, when fine-tuning the local redrawing model, the text prompt word input is "a photo of olis". When generating the defect image, the text prompt word including "olis" is input into the local redrawing model together with the background image and the defect annotation on the background image. Guided by the defect annotation and the text prompt word, the local redrawing model can generate defects of the corresponding defect type on the background image to obtain a defect image.

[0093] Taking three local redrawing models as an example, the first local redrawing model is fine-tuned based on the first defect training set corresponding to defect type A, the second local redrawing model is fine-tuned based on the second defect training set corresponding to defect type B, and the third local redrawing model is fine-tuned based on the third defect training set corresponding to defect type C. The text prompt words corresponding to the three local redrawing models are all "a photo of olis".

[0094] Input the background image into the first local redrawing model, and generate a defect of defect type A on the background image based on the defect annotation and the text prompt word "olis", and obtain the corresponding defect image. Input the background image into the second local redrawing model, and generate a defect of defect type B on the background image based on the defect annotation and the text prompt word "olis", and obtain the corresponding defect image. Input the background image into the third local redrawing model, and generate a defect of defect type C on the background image based on the defect annotation and the text prompt word "olis", and obtain the corresponding defect image.

[0095] After research, it was found that the defect image generation method of the implementation mode of the present application uses less than 24G of graphics card resources and takes less than one hour in the model fine-tuning stage and the defect generation stage. It has sufficient feasibility for on-site implementation of industrial inspection projects and can effectively solve the problem of small number of defect samples in the early stage of industrial inspection projects and difficulty in model optimization.

[0096] It should be noted that the defect annotation on the background image can be set by the user, and the user can set the defect annotation on the background image in two ways. The following describes in detail the two defect annotation methods and the corresponding defect image generation process.

[0097] See also Figure 2 and Figure 7In some embodiments, obtaining a background image and defect annotations on the background image (ie, 030) includes:

[0098] 031: Obtain a background image and a first fine annotation set by a user on the background image;

[0099] At this time, according to each local redrawing model, a defect of a corresponding defect type is generated on the background image based on the defect annotation to obtain a defect image (ie, 040), including:

[0100] 042: According to each local redrawing model, a defect corresponding to the defect type and having a defect shape and a defect position corresponding to the first fine annotation is generated on the background image to obtain a defect image.

[0101] Specifically, in the first defect annotation mode, the defect annotation includes a first fine annotation. The user can set the first fine annotation on the background image, and the first fine annotation includes the defect category, defect morphology and defect location of the defect to be generated, and the defect morphology and defect location can be represented by the defect outline. The background image and the first fine annotation set by the user on the background image are obtained, and the background image and the first fine annotation are input into the local redrawing model. The local redrawing model can generate a defect corresponding to the defect type and the defect morphology and defect location corresponding to the first fine annotation on the background image to obtain a defect image.

[0102] In related technologies, the defect generation process lacks sufficient interactivity. Once the model starts training or reasoning, the channel for manual guidance is blocked, and it is difficult for staff to limit the location of defect generation by adding some guidance conditions. Defects generated by the model may appear in unconventional locations, such as areas outside the industrial device in the defect image, causing the model to pass inspection in these areas, reducing the yield rate.

[0103] In the implementation mode of the present application, the user can set a first fine annotation on the background image, and the local redrawing model generates defects on the background image based on the first fine annotation set by the user to obtain a defect image. In the process of generating the defect image, the user can guide the position and shape of the defect generation by setting the first fine annotation. In this way, it has good interactivity, can achieve precise control of the defect generation position, and then filter out unreasonable defect images by controlling the position of defect generation.

[0104] See also Figure 2 and Figure 8 In some embodiments, obtaining a background image and defect annotations on the background image (ie, 030) includes:

[0105] 032: Acquire a background image, a coarse annotation set by a user on the background image, and randomly generate a second fine annotation based on the coarse annotation;

[0106] At this time, according to each local redrawing model, a defect of a corresponding defect type is generated on the background image based on the defect annotation to obtain a defect image (ie, 040), including:

[0107] 043: According to each local redrawing model, a defect corresponding to the defect type and the defect morphology corresponding to the second fine annotation is generated at a position corresponding to the coarse annotation on the background image to obtain a defect image.

[0108] Specifically, in the second defect annotation mode, the defect annotation includes a coarse annotation and a second fine annotation. The user can set a coarse annotation on the background image. The coarse annotation can be a polygon or other arbitrary shape. The coarse annotation can limit the location of the generated defect in the background image. After setting the coarse annotation, the local redrawing model can randomly generate a second fine annotation in the area included in the coarse annotation based on the coarse annotation. The second fine annotation has the same format as the first fine annotation, including the desired defect category, defect morphology and defect location. The defect morphology and defect location can both be represented by the defect outline.

[0109] After obtaining the second fine annotation, the local redrawing model can generate a defect of the corresponding defect type and defect morphology corresponding to the second fine annotation at the position corresponding to the coarse annotation on the background image to obtain a defect image. It can be understood that based on a coarse annotation in a background image, multiple second fine annotations can be randomly generated, so that multiple defects with different defect morphologies can be generated to obtain multiple different defect images. In this way, the time for manual annotation can be shortened, which is suitable for scenes that require large-scale generation.

[0110] In the implementation mode of the present application, the user controls the position of defect generation through coarse annotation, and the local redrawing model randomly generates a second fine annotation based on the coarse annotation to control the defect shape of the generated defect. In this way, while achieving control over the defect generation position, the defect shape is made more diverse, and while ensuring a certain degree of interactivity, a large number of defect images can be generated.

[0111] It should be noted that in other implementations, the ControlNet neural network can be used to replace the local redrawing model for defect generation. The ControlNet neural network can also guide the reasoning of the model by adding other information such as background and annotation.

[0112] See also Figure 2 and Fig. 9 In some embodiments, after generating defects of corresponding defect types based on defect annotations on the background image according to each local redrawing model to obtain a defect image (ie, 040), the defect image generation method further includes:

[0113] 050: Determine and generate a sample set according to the defect image and the corresponding defect annotation;

[0114] 060: Get the real sample set, and input the generated sample set and the real sample set into the verification model;

[0115] 070: Train the verification model using the generated sample set and the real sample set to verify the quality of the generated sample set.

[0116] Specifically, after obtaining the defect image, the quality of the generated defect image can be verified. The generated sample set is determined based on the defect image and the corresponding defect annotation. If the defect image is generated based on the first fine annotation, the first fine annotation is used as the annotation of the defect image in the generated sample set; if the defect image is generated based on the coarse annotation and the second fine annotation, the second fine annotation is used as the annotation of the defect image in the generated sample set.

[0117] After that, a real sample set is obtained, which includes real defect samples obtained in the production line and corresponding real defect annotations. The generated sample set and the real sample set are combined and input into the verification model together. The verification model is trained by the generated sample set and the real sample set to verify the quality of the generated sample set. The verification model can be any model that needs to be trained and applied to industrial inspection.

[0118] Taking the segmentation model as the verification model as an example, the annotation accuracy of the trained segmentation model is used as the model detection rate. After research, it is found that in small sample scenarios, for example, when there are less than 30 real defect samples of each defect type, data expansion based on the generated defect image pairs can reduce the demand for real defect samples by 50%. In other words, using 50% real defect samples and a large number of defect images, the model detection rate obtained by training 100% real defect samples can be achieved, so that the data collection cycle of industrial inspection projects can be shortened by 50%.

[0119] See also Fig.10 , the embodiment of the present application also provides a defect image generating device 100. The defect image generating device 100 includes a first acquisition module 10, a model fine-tuning module 20, a second acquisition module 30 and an image generating module 40. The first acquisition module 10 is used to acquire a plurality of defect training sets with different defect types, wherein each defect training set corresponds to a defect type. The model fine-tuning module 20 is used to fine-tune a corresponding local redrawing model according to each defect training set. The second acquisition module 30 is used to acquire a background image and defect annotations on the background image. The image generating module 40 is used to generate defects of corresponding defect types on the background image based on the defect annotations according to each local redrawing model to obtain a defect image.

[0120] In some embodiments, the first acquisition module 10 is specifically used to acquire a defect sample set, which includes multiple defect samples; acquire the defect type of the defect included in each defect sample; and classify the multiple defect samples according to the defect type to obtain multiple defect training sets.

[0121] In certain embodiments, the model fine-tuning module 20 is specifically used to set text prompt words corresponding to each defect training set; obtain the segmentation annotation corresponding to each defect sample in each defect training set; and fine-tune a corresponding local redrawing model according to each defect training set, the text prompt words corresponding to each defect training set, and the segmentation annotation corresponding to each defect sample.

[0122] In some embodiments, the image generation module 40 is specifically configured to generate defects of corresponding defect types on the background image based on each local redrawing model based on the defect annotation and the corresponding text prompt word to obtain a defect image.

[0123] In some embodiments, the second acquisition module 30 is specifically used to acquire the background image and the first fine annotation set by the user on the background image. At this time, the image generation module 40 is specifically used to generate a defect corresponding to the defect type and the defect shape and defect position corresponding to the first fine annotation on the background image according to each local redrawing model, and obtain a defect image.

[0124] In some embodiments, the second acquisition module 30 is specifically used to acquire the background image, the coarse annotation set by the user on the background image, and randomly generate the second fine annotation based on the coarse annotation. At this time, the image generation module 40 is specifically used to generate a defect corresponding to the defect type and the defect morphology corresponding to the second fine annotation at the position corresponding to the coarse annotation on the background image according to each local redrawing model, and obtain a defect image.

[0125] In some embodiments, the defect image generating device 100 further includes a quality verification module 50. After generating defects of corresponding defect types on the background image based on defect annotations according to each local redrawing model and obtaining a defect image, the quality verification module 50 is used to determine a generated sample set according to the defect image and the corresponding defect annotations; obtain a real sample set, and input the generated sample set and the real sample set into a verification model; train the verification model through the generated sample set and the real sample set to verify the quality of the generated sample set.

[0126] It should be pointed out that the explanation of the defect image generating method in the aforementioned embodiment is also applicable to the defect image generating device 100 in the embodiment of the present application, and will not be elaborated here.

[0127] See also Fig.11The embodiment of the present application further provides a defect image generation system 200. The defect image generation system 200 includes one or more processors 210 and a memory 220. The memory 220 stores a computer program. When the computer program is executed by the processor 210, the defect image generation method of any of the above embodiments is implemented.

[0128] For example, when the computer program is executed by the processor 210, the following defect image generation method is implemented:

[0129] 010: Obtain multiple defect training sets with different defect types, where each defect training set corresponds to one defect type;

[0130] 020: Fine-tune a corresponding local redrawing model according to each defect training set;

[0131] 030: Obtain a background image and defect annotations on the background image;

[0132] 040: According to each local redrawing model, a defect of a corresponding defect type is generated on the background image based on the defect annotation to obtain a defect image.

[0133] For another example, when the computer program is executed by the processor 210, the following defect image generation method is implemented:

[0134] 011: Obtain a defect sample set, where the defect sample set includes multiple defect samples;

[0135] 012: Obtain the defect types of defects included in each defect sample;

[0136] 013: Classify multiple defect samples according to defect types to obtain multiple defect training sets.

[0137] It should be pointed out that the explanations and descriptions of the defect image generating method and the defect image generating device 100 in the aforementioned embodiments are also applicable to the defect image generating system 200 in the embodiments of the present application, and will not be elaborated here.

[0138] See also Fig.12 The embodiment of the present application further provides a computer-readable storage medium 300 on which a computer program 310 is stored. When the program is executed by a processor 320, the defect image generation method of any of the above embodiments is implemented.

[0139] For example, when the computer program 310 is executed by the processor 320, the following defect image generation method is implemented:

[0140] 010: Obtain multiple defect training sets with different defect types, where each defect training set corresponds to one defect type;

[0141] 020: Fine-tune a corresponding local redrawing model according to each defect training set;

[0142] 030: Obtain a background image and defect annotations on the background image;

[0143] 040: According to each local redrawing model, a defect of a corresponding defect type is generated on the background image based on the defect annotation to obtain a defect image.

[0144] For another example, when the computer program 310 is executed by the processor 320, the following defect image generation method is implemented:

[0145] 011: Obtain a defect sample set, where the defect sample set includes multiple defect samples;

[0146] 012: Obtain the defect types of defects included in each defect sample;

[0147] 013: Classify multiple defect samples according to defect types to obtain multiple defect training sets.

[0148] It should be pointed out that the explanations and descriptions of the defect image generating method and the defect image generating device 100 in the aforementioned embodiments are also applicable to the computer-readable storage medium 300 in the embodiments of the present application, and will not be elaborated here.

[0149] In summary, in the defect image generation method, defect image generation device 100, defect image generation system 200 and computer-readable storage medium 300 of the embodiment of the present application, a corresponding local redrawing model is fine-tuned according to each defect training set, and a defect of the corresponding defect type is generated on the background image based on the defect annotation according to each local redrawing model to obtain a defect image. In this way, the generated defect has a good degree of fusion with the background and a good visual effect, and high-quality and diversified defect images can be obtained. In industrial inspection projects, it is convenient to migrate production lines and can effectively shorten the project cycle.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0151] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0152] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0153] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0154] A person of ordinary skill in the art will appreciate that all or part of the steps carried by the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0155] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. A defect image generation method, characterized in that: include: Acquire a plurality of defect training sets with different defect types, wherein each defect training set corresponds to one defect type; Fine-tune a corresponding local redrawing model according to each of the defect training sets; Acquire a background image and defect annotations on the background image; According to each of the local redrawing models, a defect corresponding to the defect type is generated on the background image based on the defect annotation to obtain a defect image.

2. The defect image generation method according to claim 1, characterized in that: The step of obtaining a plurality of defect training sets with different defect types includes: Acquire a defect sample set, where the defect sample set includes a plurality of defect samples; Obtaining the defect type of the defect included in each of the defect samples; The plurality of defect samples are classified according to the defect type to obtain a plurality of defect training sets.

3. The defect image generation method according to claim 1, characterized in that: The fine-tuning of a corresponding local redrawing model according to each defect training set comprises: respectively setting text prompt words corresponding to each of the defect training sets; Obtaining the segmentation annotation corresponding to each defect sample in each defect training set; According to each of the defect training sets, the text prompt words corresponding to each of the defect training sets, and the segmentation annotations corresponding to each of the defect samples, a corresponding local redrawing model is fine-tuned.

4. The defect image generation method according to claim 3, characterized in that: The step of generating a defect corresponding to the defect type on the background image based on the defect annotation according to each of the local redrawing models to obtain a defect image includes: According to each of the local redrawing models, a defect corresponding to the defect type is generated on the background image based on the defect annotation and the corresponding text prompt word to obtain the defect image.

5. The defect image generation method according to claim 1, characterized in that: The obtaining of the background image and the defect annotation on the background image comprises: Acquire the background image and a first fine annotation set by a user on the background image; The step of generating a defect corresponding to the defect type on the background image based on the defect annotation according to each of the local redrawing models to obtain a defect image includes: According to each of the local redrawing models, a defect corresponding to the defect type and having a defect shape and a defect position corresponding to the first fine annotation is generated on the background image to obtain the defect image.

6. The defect image generation method according to claim 1, characterized in that: The obtaining of the background image and the defect annotation on the background image comprises: Acquire the background image, a coarse annotation set by a user on the background image, and a second fine annotation randomly generated based on the coarse annotation; The step of generating a defect corresponding to the defect type on the background image based on the defect annotation according to each of the local redrawing models to obtain a defect image includes: According to each of the local redrawing models, a defect corresponding to the defect type and having a defect shape corresponding to the second fine annotation is generated at a position on the background image corresponding to the coarse annotation to obtain the defect image.

7. The defect image generation method according to claim 1, characterized in that: After generating a defect corresponding to the defect type based on the defect annotation on the background image according to each of the local redrawing models to obtain a defect image, the defect image generation method further includes: Determine and generate a sample set according to the defect image and the corresponding defect annotation; Obtain a real sample set, and input the generated sample set and the real sample set into a verification model; The verification model is trained by using the generated sample set and the real sample set to verify the quality of the generated sample set.

8. A defect image generating device, characterized in that: include: A first acquisition module is used to acquire a plurality of defect training sets with different defect types, wherein each defect training set corresponds to one defect type; A model fine-tuning module, used for fine-tuning a corresponding local redrawing model according to each defect training set; A second acquisition module is used to acquire a background image and defect annotations on the background image; An image generation module is used to generate defects corresponding to the defect type on the background image based on the defect annotation according to each of the local redrawing models, so as to obtain a defect image.

9. A defect image generation system, characterized in that: The defect image generation system includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the defect image generation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the defect image generating method described in any one of claims 1 to 7 is implemented.

Images