Defect data generation method, generation device, generation system and storage medium
By acquiring and utilizing the defect training set, fine-tuning and training the diffusion model and labeling synthesizer, generating defect images and labeling, the problem of long time for obtaining and labeling defect data in the early stage of industrial detection projects is solved, and the project cycle is shortened.
Patent Information
- Application Number
- CN202411973205.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-18
AI Technical Summary
In the early stages of industrial inspection projects, the time for obtaining and labeling defective data is long, resulting in a longer project cycle.
By obtaining multiple defect types and consistent number of defect training sets, fine-tuning the diffusion model and training annotation synthesizer respectively, to generate defect images and labels for corresponding defect types.
It realizes the rapid generation of high-quality and diverse defect images and labeling, shortens the time for early data acquisition and labeling of projects, and effectively shortens the project cycle.
Smart Images
Figure CN119850577A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data generation, and in particular to a defect data generation method, a defect data generation device, a defect data generation system and a computer-readable storage medium. Background Art
[0002] In recent years, thanks to the vigorous development of deep learning, industrial inspection models have made greater breakthroughs and can achieve fairly high detection accuracy in many complex scenarios. However, such models need to obtain a large amount of high-quality defect data before they can start iterative verification of the model. Data collection takes a long time, which seriously prolongs the project cycle. Therefore, from the perspective of data processing, how to expand defect data in large quantities at the beginning of industrial inspection projects is an urgent problem to be solved. Summary of the invention
[0003] The embodiments of the present application provide a defect data generating method, a defect data generating device, a defect data generating system and a computer-readable storage medium to solve at least one of the above-mentioned technical problems.
[0004] The defect data generation method of the embodiment of the present application includes:
[0005] Acquire multiple defect types and defect training sets with the same number as the defect types, wherein the defect training sets correspond to the defect types one by one;
[0006] Fine-tune a corresponding diffusion model according to each of the defect training sets;
[0007] According to each of the defect training sets and a corresponding diffusion model, a corresponding annotation synthesizer is trained respectively;
[0008] A defect image corresponding to the defect type is generated based on each of the diffusion models, and a defect annotation corresponding to the defect image is generated according to a corresponding one of the annotation synthesizers.
[0009] In some embodiments, obtaining a plurality of defect types and a defect training set having the same number as the defect types includes:
[0010] Obtain multiple defect samples;
[0011] Obtaining the defect type of the defect included in each of the defect samples;
[0012] The plurality of defect samples are classified according to the defect type to obtain a plurality of defect training sets.
[0013] In some embodiments, fine-tuning a corresponding diffusion model according to each defect training set includes:
[0014] respectively setting text prompt words corresponding to each of the defect training sets;
[0015] According to each defect training set and the corresponding text prompt word, a corresponding diffusion model is fine-tuned.
[0016] In some embodiments, the step of training a corresponding annotation synthesizer according to each defect training set and a corresponding diffusion model comprises:
[0017] Initializing weights of a plurality of the annotation synthesizers;
[0018] connecting each of the annotation synthesizers to one of the diffusion models;
[0019] Based on each of the defect training sets and a corresponding diffusion model, obtaining a rough label for each defect sample in each of the defect training sets;
[0020] Inputting the coarse annotation into the annotation synthesizer to obtain corresponding fine annotation;
[0021] Obtaining true annotations for each of the defect samples;
[0022] The loss between the refined annotation and the true annotation is calculated, and parameters of the annotation synthesizer are updated according to the loss.
[0023] In some embodiments, obtaining a rough label of each defect sample in each defect training set based on each defect training set and a corresponding diffusion model includes:
[0024] Input each of the defect training sets into a corresponding diffusion model;
[0025] Acquire defect location information corresponding to each defect sample in a corresponding defect training set based on each diffusion model;
[0026] The rough label of each of the defect samples is determined according to the defect location information.
[0027] In some embodiments, generating a defect image corresponding to the defect type based on each of the diffusion models, and generating a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer, includes:
[0028] connecting each of the diffusion models to a corresponding one of the annotation synthesizers;
[0029] Obtaining the text prompt word input by the user;
[0030] Based on the diffusion model, generating the defect image corresponding to the defect type according to the text prompt word;
[0031] Based on the annotation synthesizer, the defect annotation corresponding to the defect image is generated according to the defect image.
[0032] In some embodiments, after generating a defect image corresponding to the defect type based on each of the diffusion models, and generating a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer, the defect data generating method further includes:
[0033] Determine and generate a sample set according to the defect image and the corresponding defect annotation;
[0034] Obtain a real sample set, and input the generated sample set and the real sample set into a verification model;
[0035] The verification model is trained by using the generated sample set and the real sample set to verify the quality of the generated sample set.
[0036] The defect data generating device of the embodiment of the present application includes:
[0037] A data acquisition module, used to acquire multiple defect types and defect training sets with the same number as the defect types, wherein the defect training sets correspond to the defect types one by one;
[0038] A first training module, used for fine-tuning a corresponding diffusion model according to each defect training set;
[0039] A second training module is used to train a corresponding annotation synthesizer according to each defect training set and a corresponding diffusion model;
[0040] A data generation module is used to generate a defect image corresponding to the defect type based on each of the diffusion models, and to generate a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer.
[0041] The defect data generation system of the embodiment of the present application includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the defect data generation method of any of the above embodiments is implemented.
[0042] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the program is executed by a processor, the defect data generating method of any of the above-mentioned embodiments is implemented.
[0043] In the defect data generation method, defect data generation device, defect data generation system and computer-readable storage medium of the implementation mode of the present application, multiple diffusion models are fine-tuned according to multiple defect training sets, and then multiple annotation synthesizers are trained respectively, defect images of corresponding defect types are generated based on each diffusion model, and defect annotations corresponding to the defect images are generated according to a corresponding annotation synthesizer. In this way, high-quality and diversified defect images can be generated while high-quality defect annotations corresponding to the defect images can be generated, which not only reduces the time for collecting defect images in the early stage of the project, but also reduces the time for annotating defect images, thereby effectively shortening the project cycle.
[0044] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0046] Figure 1 is a flowchart of a defect data generation method according to certain embodiments of the present application;
[0047] Figure 2 is a flowchart of a defect data generation method according to certain embodiments of the present application;
[0048] Figure 3 is a flowchart of a defect data generation method according to certain embodiments of the present application;
[0049] Figure 4 is a flowchart of a defect data generation method according to certain embodiments of the present application;
[0050] Figure 5 is a flowchart of a defect data generation method according to certain embodiments of the present application;
[0051] Figure 6 is a schematic diagram of the training process of the annotation synthesizer of certain embodiments of the present application;
[0052] Figure 7 is a flowchart of a defect data generation method according to certain embodiments of the present application;
[0053] Figure 8 is a flowchart of a defect data generation method according to certain embodiments of the present application;
[0054] Fig. 9 is a flowchart of a defect data generation method according to certain embodiments of the present application;
[0055] Fig.10 is a module schematic diagram of a defect data generating device in some embodiments of the present application;
[0056] Fig.11 is a module schematic diagram of a defect data generating device in some embodiments of the present application;
[0057] Fig.12 is a module schematic diagram of a defect data generation system of certain embodiments of the present application;
[0058] Fig.13 It is a schematic diagram of the connection status between a computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION
[0059] The following is a further description of the embodiments of the present application in conjunction with the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements or elements with the same or similar functions from beginning to end. In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the present application.
[0060] In recent years, thanks to the vigorous development of deep learning, industrial inspection models have made greater breakthroughs and can achieve fairly high detection accuracy in many complex scenarios. However, there is a common problem with industrial inspection models. Industrial inspection models rely on the quantity and quality of training images. Specifically, for the task of industrial defect segmentation, the quality of annotation is also extremely critical. In the early stages of industrial projects, it is extremely difficult to obtain images with defects, which means that it takes several weeks to collect enough defects before the iterative verification of the segmentation model can be started, which seriously prolongs the overall project cycle.
[0061] At present, in the field of industrial inspection, defect injection is used to generate defects, that is, the staff injects the collected defects or similar defects in past projects into the normal sample images collected on site through image processing, script processing and other methods, so as to expand the model training set. However, in the defect images generated by defect injection, the degree of fit between the defects and the new background is poor, and the color and texture of the defects and the background are often quite different, which makes the defects very abrupt, and the injected defects are fixed in shape and lack diversity.
[0062] See also Figure 1 and Figure 2 , the embodiment of the present application provides a defect data generation method. The defect data generation method includes:
[0063] 010: Obtain multiple defect types and defect training sets with the same number as the defect types, wherein the defect training sets correspond to the defect types one by one;
[0064] 020: Fine-tune a corresponding diffusion model based on each defect training set;
[0065] 030: Train a corresponding annotation synthesizer according to each defect training set and a corresponding diffusion model;
[0066] 040: Generate a defect image of a corresponding defect type based on each diffusion model, and generate a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer.
[0067] In the defect data generation method of the embodiment of the present application, multiple diffusion models are fine-tuned according to multiple defect training sets, and then multiple annotation synthesizers are trained respectively, defect images of corresponding defect types are generated based on each diffusion model, and defect annotations corresponding to the defect images are generated according to a corresponding annotation synthesizer. In this way, high-quality and diversified defect images can be generated while high-quality defect annotations corresponding to the defect images can be generated, which not only reduces the time for collecting defect images in the early stage of the project, but also reduces the time for annotating defect images, thereby effectively shortening the project cycle.
[0068] Specifically, multiple defect types and multiple defect training sets are obtained, and the defect training sets correspond to the defect types one by one, and one defect training set corresponds to one defect type. The number of defect training sets is consistent with the number of defect types. The diffusion model can adopt the Stable Diffusion V1.4 model. According to each defect training set, a diffusion model can be fine-tuned respectively, so that the diffusion model learns the defect characteristics of a defect type, so as to generate a defect image of the defect type corresponding to the defect training set. According to multiple defect training sets, multiple diffusion models can be fine-tuned respectively.
[0069] After fine-tuning is completed, a corresponding annotation synthesizer is trained according to each defect training set and a corresponding diffusion model. The number of annotation synthesizers is equal to the number of diffusion models and corresponds to the number of defect training sets. The structure of the annotation synthesizer is a feature decoder. The annotation synthesizer can generate corresponding defect labels based on the features of the defect image generated by the diffusion model. By training the annotation synthesizer, the quality of the defect labels generated by the annotation synthesizer is improved.
[0070] After the annotation synthesizer is trained, the diffusion model and the corresponding annotation synthesizer can be used to generate defect data, which includes defect images and defect annotations corresponding to the defect images. Based on each diffusion model, a defect image of a corresponding defect type can be generated, and based on an annotation synthesizer corresponding to the diffusion model, a defect annotation corresponding to the defect image of the defect type can be generated.
[0071] For example, if three defect training sets are obtained, namely the first defect training set corresponding to defect type A, the second defect training set corresponding to defect type B, and the third defect training set corresponding to defect type C, the three diffusion models can be fine-tuned according to the three defect training sets, and then the three annotation synthesizers can be trained.
[0072] The first diffusion model can be fine-tuned based on the first defect training set so that the first diffusion model has the ability to generate defect images of defect type A; the first annotation synthesizer can be trained based on the first defect training set and the first diffusion model so that the first annotation synthesizer has the ability to generate high-quality defect labels corresponding to defect type A.
[0073] The second diffusion model can be fine-tuned according to the second defect training set so that the second diffusion model has the ability to generate defect images of defect type B; the second annotation synthesizer can be trained according to the second defect training set and the second diffusion model so that the second annotation synthesizer has the ability to generate high-quality defect labels corresponding to defect type B.
[0074] The third diffusion model can be fine-tuned according to the third defect training set so that the third diffusion model has the ability to generate defect images of defect type C; the third annotation synthesizer can be trained according to the third defect training set and the third diffusion model so that the third annotation synthesizer has the ability to generate high-quality defect labels corresponding to defect type C.
[0075] In the implementation mode of the present application, the generation of defect images is based on the diffusion model, and each diffusion model only learns the characteristics of one defect type, which is used to generate defect images of one defect type. The defect and background in the defect image are well integrated, and the defect morphology is highly diverse, so that high-quality and diversified defect images can be obtained. The generation of defect annotations is based on the annotation synthesizer, and each annotation synthesizer is used to generate defect annotations of one defect type, which can generate high-quality defect annotations.
[0076] In addition, the annotation synthesizer works together with the diffusion model to generate defect annotations corresponding to defect images at the same time. Defect images and defect annotations can be directly applied to segmentation models in industrial inspection projects for training, which not only reduces the time for collecting defect images in the early stage of the project, but also reduces the time for annotating defect images, which can effectively shorten the project cycle.
[0077] See also Figure 3 In some embodiments, obtaining a plurality of defect types and a defect training set (i.e., 010) having the same number as the defect types comprises:
[0078] 011: Get multiple defect samples;
[0079] 012: Obtain the defect types of defects included in each defect sample;
[0080] 013: Classify multiple defect samples according to defect types to obtain multiple defect training sets.
[0081] Specifically, the defect training set is obtained by classifying defect samples. Each defect sample includes at least one defect of at least one defect type. Multiple defect samples are image-classified according to the defect type of the defect included in each defect sample to obtain multiple defect training sets with different defect types.
[0082] Since fine-tuning a diffusion model only requires defects of one defect type, when the defect sample includes multiple defects of multiple defect types, it is necessary to eliminate the defects of the redundant defect types in the defect sample and only retain the defects of one defect type to avoid interference with the subsequent fine-tuning process and training process.
[0083] For example, if a plurality of defect samples include defects of three defect types in total, three defect training sets can be obtained according to the plurality of defect samples, namely, a first defect training set corresponding to defect type A, a second defect training set corresponding to defect type B, and a third defect training set corresponding to defect type C. Defect sample X includes defects of defect type A, defects of defect type B, and defects of defect type C. If the defects of defect type B and defect type C in defect sample X are eliminated, defect sample X can be classified into the first defect training set. If the defects of defect type A and defect type C in defect sample X are eliminated, defect sample X can be classified into the second defect training set. If the defects of defect type A and defect type B in defect sample X are eliminated, defect sample X can be classified into the third defect training set.
[0084] It should be noted that since the diffusion model is a model obtained by pre-training based on a large-scale data set and has good generalization ability, when fine-tuning the diffusion model, only a small number of defect samples are needed to make the diffusion model adapt to the overall image style and defect characteristics of the industrial scene. That is to say, in the implementation mode of the present application, each defect training set may include only a small number of defect samples, for example, each defect training set may include only 1-5 defect samples. In this way, it can be applied to scenes with very few defect samples, which is conducive to shortening the project cycle of industrial inspection projects.
[0085] See also Figure 4 In some embodiments, fine-tuning a corresponding diffusion model (i.e., 020) according to each defect training set includes:
[0086] 021: Set the text prompt words corresponding to each defect training set respectively;
[0087] 022: Based on each defect training set and the corresponding text prompt word, fine-tune a corresponding diffusion model.
[0088] Specifically, the Dreambooth algorithm can be used to fine-tune the diffusion model. The Dreambooth algorithm fine-tunes the model by injecting custom themes into the diffusion model. Users can use only a few pictures of specific objects as input to allow the diffusion model to learn the characteristics of the object and generate new pictures including the object under different text prompts. The generated new pictures can maintain the main features of the object while presenting a variety of scenes and postures.
[0089] The process of fine-tuning the diffusion model based on the Dreambooth algorithm is as follows: set the text prompt words corresponding to each defect training set. For example, the text prompt word of a defect training set can be set as "a photo of sks", where "sks" is a rare word that cannot be separated, which can avoid interference from prior knowledge in the diffusion model.
[0090] It should be noted that, since each defect training set fine-tunes a diffusion model accordingly, multiple subsequent diffusion models independently generate defects of corresponding defect types without interfering with each other. Therefore, the text prompt words corresponding to multiple defect training sets can be the same or different.
[0091] Each defect training set and the corresponding text prompt word are input into a corresponding diffusion model. The diffusion model learns the features of the defect samples in the defect training set, and binds the text prompt word with the defect type corresponding to the defect training set to complete the fine-tuning of the diffusion model. After the fine-tuning is completed, the text prompt word is input into the corresponding diffusion model to guide the diffusion model to generate defect images of the corresponding defect type. In this way, based on the defect training set and text prompt words, the diffusion model can be adapted to the image domain of a specific industrial scene, and the style of the defect images can be kept consistent while generating a variety of defect images.
[0092] See also Figure 5 and Figure 6 In some embodiments, a corresponding annotation synthesizer (i.e., 030) is trained according to each defect training set and a corresponding diffusion model, including:
[0093] 031: Initialize the weights of multiple annotation synthesizers;
[0094] 032: Connect each annotation synthesizer to a diffusion model;
[0095] 033: Based on each defect training set and a corresponding diffusion model, obtain a rough label of each defect sample in each defect training set;
[0096] 034: Input the coarse annotation into the annotation synthesizer to obtain the corresponding fine annotation;
[0097] 035: Obtain the true annotation of each defect sample;
[0098] 036: Calculate the loss between the refined annotation and the true annotation, and update the parameters of the annotation synthesizer based on the loss.
[0099] Specifically, the weights of multiple annotation synthesizers are randomly initialized, and each annotation synthesizer is connected to a diffusion model. It should be noted that when training the annotation synthesizer, the parameters of the diffusion model need to be frozen. Based on each defect training set and a corresponding diffusion model, a rough annotation of each defect sample in each defect training set can be obtained.
[0100] The coarse annotation is output from the diffusion model and input into the annotation synthesizer connected to the diffusion model. The annotation synthesizer decodes the coarse annotation to obtain the corresponding fine annotation. The fine annotation is in the form of a mask, which can characterize the defect category, defect morphology and defect location. The defect morphology and defect location can both be represented by the defect contour.
[0101] Each defect training set includes the true annotation corresponding to each defect sample, and the true annotation of each defect sample is obtained. The true annotation is also in the form of a mask, and the true annotation includes the defect category, defect morphology and defect location of the defect sample. The true annotation can be a manual defect annotation. For example, the defect sample can be manually annotated based on the Labelme annotation tool to obtain the true annotation.
[0102] The true annotation is used to compare with the fine annotation obtained by the annotation synthesizer, and the error between the fine annotation and the true annotation is calculated as the loss. The parameters of the annotation synthesizer are optimized and updated according to the loss back propagation. The above process can be repeated to continuously update the parameters of the annotation synthesizer until the maximum number of repetitions is reached or the fine annotation obtained by the annotation synthesizer meets the preset conditions. After training, the annotation synthesizer can generate corresponding defect annotations for defect images while the diffusion model generates defect images, and the defect annotations are of high accuracy and good quality. In this way, there is no need to manually annotate the defect images, which can reduce the time for annotating defect images.
[0103] The process of obtaining coarse annotations based on the defect training set and the diffusion model is described in detail below.
[0104] See also Figure 7In some embodiments, based on each defect training set and a corresponding diffusion model, a rough annotation (ie, 033) of each defect sample in each defect training set is obtained, including:
[0105] 0331: Input each defect training set into a corresponding diffusion model;
[0106] 0332: Based on each diffusion model, obtain defect location information corresponding to each defect sample in a corresponding defect training set;
[0107] 0333: Determine the rough labeling of each defect sample based on the defect location information.
[0108] Specifically, each defect training set is input into a corresponding diffusion model, and the diffusion model is a diffusion model that has been fine-tuned. For example, the first diffusion model is obtained by fine-tuning the first defect training set, and the second diffusion model is obtained by fine-tuning the second defect training set. When obtaining the rough annotation, the first defect training set is input into the first diffusion model, and the second defect training set is input into the second diffusion model.
[0109] It should be noted that before the defect training set is input into the diffusion model, each defect sample needs to be encoded and noise added. The diffusion model denoises each defect sample after encoding and noise addition to restore each defect sample after encoding and noise addition. In the denoising process of the diffusion model, the defect location information related to each defect sample inside the diffusion model can be extracted.
[0110] The defect location information includes attention maps and multi-scale feature maps. The diffusion model includes a neural network U-Net. The self-attention mechanism is introduced into the neural network U-Net, so that the attention map and multi-scale feature map corresponding to the defect sample can be extracted. The attention map and multi-scale feature map of each defect sample are connected to obtain a weighted feature map, which is the coarse annotation corresponding to each defect sample.
[0111] See also Figure 8 In some embodiments, a defect image corresponding to the defect type is generated based on each diffusion model, and a defect annotation corresponding to the defect image is generated according to a corresponding annotation synthesizer (ie, 040), including:
[0112] 041: Connect each diffusion model to a corresponding annotation synthesizer;
[0113] 042: Get the text prompt word entered by the user;
[0114] 043: Based on the diffusion model, the defect image corresponding to the defect type is generated according to the text prompt word;
[0115] 044: Based on the annotation synthesizer, a defect annotation corresponding to the defect image is generated according to the defect image.
[0116] Specifically, after the annotation synthesizer training is completed, each fine-tuned diffusion model is connected to a corresponding annotation synthesizer in the same way as when the annotation synthesizer is trained. The user can guide the diffusion model to generate defect images of the corresponding defect type by inputting the text prompt words set in the diffusion model fine-tuning stage.
[0117] Taking three diffusion models as an example, the first diffusion model is fine-tuned based on the first defect training set corresponding to defect type A, the second diffusion model is fine-tuned based on the second defect training set corresponding to defect type B, and the third diffusion model is fine-tuned based on the third defect training set corresponding to defect type C. The text prompt words corresponding to the three diffusion models are all "a photo ofsks".
[0118] By inputting the text prompt word "sks" into the first diffusion model, a defect image of defect type A can be generated based on the text prompt word. By inputting the text prompt word "sks" into the second diffusion model, a defect image of defect type B can be generated based on the text prompt word. By inputting the text prompt word "sks" into the third diffusion model, a defect image of defect type C can be generated based on the text prompt word.
[0119] In addition, the text prompt words may include other guide words besides "sks", and the guide words may guide the diffusion model to generate features corresponding to the guide words, so that the diffusion model may generate defect images corresponding to the defect type and including features corresponding to the guide words based on the text prompt words. In this way, diversified defect images may be generated.
[0120] While the diffusion model generates a defect image corresponding to the defect type, the defect location information inside the diffusion model is extracted, and the coarse annotation is determined according to the defect location information. The coarse annotation is input into the annotation synthesizer, and the annotation synthesizer decodes the coarse annotation to obtain the defect annotation corresponding to the defect image.
[0121] In related technologies, the defect injection method injects fixed defects in fixed forms and lacks diversity, which has limited effect on improving the learning ability of the model. Even if a large number of new samples are obtained through injection, the model can easily overfit to these fixed forms during the training process, resulting in weak generalization in field applications and poor model detection effects, which cannot effectively shorten the project cycle.
[0122] In the implementation manner of the present application, defect images are generated based on a diffusion model. By inputting text prompt words including different guide words, the diffusion model can be guided to generate diverse defect images, which is beneficial to improving the learning ability and generalization of the model in the subsequent industrial inspection model training process, and effectively shortening the project cycle.
[0123] See also Fig. 9 In some embodiments, after generating a defect image corresponding to a defect type based on each diffusion model and generating a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer (ie, 040), the defect data generation method further includes:
[0124] 050: Determine and generate a sample set according to the defect image and the corresponding defect annotation;
[0125] 060: Get the real sample set, and input the generated sample set and the real sample set into the verification model;
[0126] 070: Train the verification model using the generated sample set and the real sample set to verify the quality of the generated sample set.
[0127] Specifically, after obtaining the defect image, the quality of the generated defect image can be verified. A generated sample set is determined based on the defect image and the corresponding defect annotation. Then a real sample set is obtained, which includes real defect samples and corresponding real defect annotations. The generated sample set and the real sample set are combined and input into the verification model together. The verification model is trained by the generated sample set and the real sample set to verify the quality of the generated sample set. The verification model can be any model that needs to be trained and applied to industrial inspection.
[0128] Taking the segmentation model as the verification model, the annotation accuracy of the trained segmentation model is used as the model detection rate. It is found that in a small sample scenario, for example, when there are less than 30 real defect samples of each defect type, data expansion based on the generated defect image pairs can reduce the demand for real defect samples by 20%. In other words, using 80% real defect samples and a large number of defect images, the model detection rate obtained by training 100% real defect samples can be achieved, which can shorten the data collection cycle of industrial inspection projects by 20%.
[0129] See also Fig.10, the embodiment of the present application also provides a defect data generating device 100. The defect data generating device 100 includes a data acquisition module 10, a first training module 20, a second training module 30 and a data generating module 40. The data acquisition module 10 is used to obtain multiple defect types and defect training sets consistent with the number of defect types, wherein the defect training sets correspond to the defect types one by one. The first training module 20 is used to fine-tune a corresponding diffusion model according to each defect training set. The second training module 30 is used to train a corresponding annotation synthesizer according to each defect training set and a corresponding diffusion model. The data generating module 40 is used to generate a defect image of the corresponding defect type based on each diffusion model, and generate a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer.
[0130] In some embodiments, the data acquisition module 10 is specifically used to acquire multiple defect samples; acquire the defect type of the defect included in each defect sample; and classify the multiple defect samples according to the defect type to obtain multiple defect training sets.
[0131] In some implementations, the first training module 20 is specifically used to set text prompt words corresponding to each defect training set; and fine-tune a corresponding diffusion model according to each defect training set and the corresponding text prompt words.
[0132] In some embodiments, the second training module 30 is specifically used to initialize the weights of multiple annotation synthesizers; connect each annotation synthesizer to a diffusion model; obtain a coarse annotation for each defect sample in each defect training set based on each defect training set and a corresponding diffusion model; input the coarse annotation into the annotation synthesizer to obtain the corresponding fine annotation; obtain the true annotation for each defect sample; calculate the loss between the fine annotation and the true annotation, and update the parameters of the annotation synthesizer according to the loss.
[0133] In some embodiments, the second training module 30 is specifically used to input each defect training set into a corresponding diffusion model; obtain defect location information corresponding to each defect sample in a corresponding defect training set based on each diffusion model; and determine a rough label for each defect sample based on the defect location information.
[0134] In some embodiments, the data generation module 40 is specifically used to connect each diffusion model with a corresponding annotation synthesizer; obtain a text prompt word input by a user; based on the diffusion model, generate a defect image of the corresponding defect type according to the text prompt word; based on the annotation synthesizer, generate a defect annotation corresponding to the defect image according to the defect image.
[0135] See also Fig.11In some embodiments, the defect data generating device 100 further includes a quality verification module 50. After generating a defect image of a corresponding defect type based on each diffusion model and generating a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer, the quality verification module 50 is specifically used to determine a generated sample set according to the defect image and the corresponding defect annotation; obtain a real sample set, and input the generated sample set and the real sample set into the verification model; train the verification model through the generated sample set and the real sample set to verify the quality of the generated sample set.
[0136] It should be pointed out that the explanation of the defect data generating method in the aforementioned embodiment is also applicable to the defect data generating device 100 in the embodiment of the present application, and will not be elaborated here.
[0137] See also Fig.12 The embodiment of the present application further provides a defect data generation system 200. The defect data generation system 200 includes one or more processors 210 and a memory 220. The memory 220 stores a computer program. When the computer program is executed by the processor 210, the defect data generation method of any of the above embodiments is implemented.
[0138] For example, when the computer program is executed by the processor 210, the following defect data generation method is implemented:
[0139] 010: Obtain multiple defect types and defect training sets with the same number as the defect types, wherein the defect training sets correspond to the defect types one by one;
[0140] 020: Fine-tune a corresponding diffusion model based on each defect training set;
[0141] 030: Train a corresponding annotation synthesizer according to each defect training set and a corresponding diffusion model;
[0142] 040: Generate a defect image of a corresponding defect type based on each diffusion model, and generate a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer.
[0143] For another example, when the computer program is executed by the processor 210, the following defect data generation method is implemented:
[0144] 011: Get multiple defect samples;
[0145] 012: Obtain the defect types of defects included in each defect sample;
[0146] 013: Classify multiple defect samples according to defect types to obtain multiple defect training sets.
[0147] It should be pointed out that the explanations of the defect data generating method and the defect data generating device 100 in the aforementioned embodiments are also applicable to the defect data generating system 200 in the embodiments of the present application, and will not be elaborated here.
[0148] See also Fig.13 The embodiment of the present application further provides a computer-readable storage medium 300 on which a computer program 310 is stored. When the program is executed by a processor 320, the defect data generating method of any of the above embodiments is implemented.
[0149] For example, when the computer program 310 is executed by the processor 320, the following defect data generation method is implemented:
[0150] 010: Obtain multiple defect types and defect training sets with the same number as the defect types, wherein the defect training sets correspond to the defect types one by one;
[0151] 020: Fine-tune a corresponding diffusion model based on each defect training set;
[0152] 030: Train a corresponding annotation synthesizer according to each defect training set and a corresponding diffusion model;
[0153] 040: Generate a defect image of a corresponding defect type based on each diffusion model, and generate a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer.
[0154] For another example, when the computer program 310 is executed by the processor 320, the following defect data generation method is implemented:
[0155] 011: Get multiple defect samples;
[0156] 012: Obtain the defect types of defects included in each defect sample;
[0157] 013: Classify multiple defect samples according to defect types to obtain multiple defect training sets.
[0158] It should be pointed out that the explanations and descriptions of the defect data generating method and the defect data generating device 100 in the aforementioned embodiments are also applicable to the computer-readable storage medium 300 in the embodiments of the present application, and will not be elaborated here.
[0159] In summary, in the defect data generation method, defect data generation device 100, defect data generation system 200 and computer-readable storage medium 300 of the embodiment of the present application, multiple diffusion models are fine-tuned according to multiple defect training sets, and then multiple annotation synthesizers are trained respectively, and defect images of corresponding defect types are generated based on each diffusion model, and defect annotations corresponding to the defect images are generated according to a corresponding annotation synthesizer. In this way, high-quality and diversified defect images can be generated while high-quality defect annotations corresponding to the defect images can be generated, which not only reduces the time for collecting defect images in the early stage of the project, but also reduces the time for annotating defect images, thereby effectively shortening the project cycle.
[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0161] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0162] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0163] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0164] A person of ordinary skill in the art will appreciate that all or part of the steps carried by the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.
[0165] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.
Claims
1. A defect data generating method, characterized in that: include: Acquire multiple defect types and defect training sets with the same number as the defect types, wherein the defect training sets correspond to the defect types one by one; Fine-tune a corresponding diffusion model according to each of the defect training sets; According to each of the defect training sets and a corresponding diffusion model, a corresponding annotation synthesizer is trained respectively; A defect image corresponding to the defect type is generated based on each of the diffusion models, and a defect annotation corresponding to the defect image is generated according to a corresponding one of the annotation synthesizers.
2. The defect data generating method according to claim 1, characterized in that: The obtaining of a plurality of defect types and a defect training set having the same number as the defect types comprises: Obtain multiple defect samples; Obtaining the defect type of the defect included in each of the defect samples; The plurality of defect samples are classified according to the defect type to obtain a plurality of defect training sets.
3. The defect data generating method according to claim 1, characterized in that: The fine-tuning of a corresponding diffusion model according to each defect training set comprises: respectively setting text prompt words corresponding to each of the defect training sets; According to each defect training set and the corresponding text prompt word, a corresponding diffusion model is fine-tuned.
4. The defect data generating method according to claim 1, characterized in that: The step of training a corresponding annotation synthesizer according to each defect training set and a corresponding diffusion model comprises: Initializing weights of a plurality of the annotation synthesizers; connecting each of the annotation synthesizers to one of the diffusion models; Based on each of the defect training sets and a corresponding diffusion model, obtaining a rough label for each defect sample in each of the defect training sets; Inputting the coarse annotation into the annotation synthesizer to obtain corresponding fine annotation; Obtaining true annotations for each of the defect samples; The loss between the refined annotation and the true annotation is calculated, and parameters of the annotation synthesizer are updated according to the loss.
5. The defect data generating method according to claim 4, characterized in that: The obtaining, based on each defect training set and a corresponding diffusion model, a rough label of each defect sample in each defect training set includes: Input each of the defect training sets into a corresponding diffusion model; Acquire defect location information corresponding to each defect sample in a corresponding defect training set based on each diffusion model; The rough label of each of the defect samples is determined according to the defect location information.
6. The defect data generating method according to claim 3, characterized in that: The generating a defect image corresponding to the defect type based on each of the diffusion models, and generating a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer, comprises: connecting each of the diffusion models to a corresponding one of the annotation synthesizers; Obtaining the text prompt word input by the user; Based on the diffusion model, generating the defect image corresponding to the defect type according to the text prompt word; Based on the annotation synthesizer, the defect annotation corresponding to the defect image is generated according to the defect image.
7. The defect data generating method according to claim 1, characterized in that: After generating a defect image corresponding to the defect type based on each of the diffusion models, and generating a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer, the defect data generating method further includes: Determine and generate a sample set according to the defect image and the corresponding defect annotation; Obtain a real sample set, and input the generated sample set and the real sample set into a verification model; The verification model is trained by using the generated sample set and the real sample set to verify the quality of the generated sample set.
8. A defect data generating device, characterized in that: include: A data acquisition module, used to acquire multiple defect types and defect training sets with the same number as the defect types, wherein the defect training sets correspond to the defect types one by one; A first training module, used for fine-tuning a corresponding diffusion model according to each defect training set; A second training module is used to train a corresponding annotation synthesizer according to each defect training set and a corresponding diffusion model; A data generation module is used to generate a defect image corresponding to the defect type based on each of the diffusion models, and to generate a defect annotation corresponding to the defect image according to a corresponding annotation synthesizer.
9. A defect data generation system, characterized in that: The defect data generating system includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the defect data generating method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the defect data generating method according to any one of claims 1 to 7 is implemented.