Abnormal image generation method and device
By randomly generating anomaly mask bounding box on normal images and constructing defect description text information, using diffusion model to generate abnormal images, and automatically generating binarized masks, the limited diversity of abnormal images and mask drift in the prior art are solved. The problem of this paper improves the diversity and detection accuracy of abnormal images and reduces the workload of manual labeling.
Patent Information
- Application Number
- CN202410333045.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-22
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-03-22
AI Technical Summary
Existing anomaly image generation methods have limited diversity and mask drift problems, and cannot accurately simulate the diversity of anomalies in the real world and provide accurate anomaly area annotations, resulting in limited robustness and accuracy of the model in detection and classification.
By randomly generating bounding boxes of abnormal masks on normal images, defect description text information of abnormal images is constructed, abnormal images are generated using diffusion models and variational autoencoder, and binarized masks are automatically generated through pixel-level differences, reducing Manual labeling workload improves labeling efficiency and mask accuracy.
It realizes the generation of diversified, highly authentic anomaly images and binarized masks, overcomes the difficulty of lacking sufficient anomaly samples, improves the diversity of anomaly images and the detection accuracy of the model, and reduces the burden of manual annotation.
Smart Images

Figure CN118154714B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of abnormality detection, and in particular to a method and device for generating an abnormal image. Background Art
[0002] In anomaly detection, it is very important to detect and locate abnormal regions in images. However, existing anomaly datasets usually suffer from serious class imbalance problems, with the number of abnormal samples significantly less than normal samples. Due to this imbalance, existing anomaly detection methods usually adopt unsupervised methods, where the model is only trained on defect-free images and then tested on normal and abnormal data. However, unsupervised methods cannot directly learn the unique features and characteristics of abnormal regions, and it is difficult to distinguish the subtle differences between normal and defective regions during detection. Therefore, existing research needs to explore how to generate more abnormal data based on a limited amount of data.
[0003] There are two main problems with existing anomaly image synthesis methods. First, existing methods have limited diversity when synthesizing anomaly images. Typically, existing methods use fixed anomaly patterns or shapes to generate anomaly images, which cannot fully cover the diversity of anomalies in the real world. In reality, anomalies may have different shapes, sizes, locations, and appearances, and existing methods cannot accurately simulate this diversity, so the synthesized anomaly images may not truly reflect the distribution and characteristics of anomalies in real scenes, which limits the robustness of the model in detecting and classifying various anomalies.
[0004] Secondly, the anomaly masks used in existing methods are directly used as labels for synthetic anomaly images, which may lead to the problem of mask drift. Mask drift refers to the deviation between the synthetic anomaly image and its corresponding mask. Due to the design and use of the anomaly mask, the synthetic image may not accurately match the anomaly position and shape indicated by the mask, resulting in inconsistency between the synthetic image and the label. This mask drift may have a negative impact on the training of downstream segmentation models, reducing the accuracy of the model in detecting and classifying real anomalies. Summary of the invention
[0005] To this end, the technical problem to be solved by the present invention is to overcome the problems of limited diversity of abnormal images generated in the prior art and inability to provide accurate abnormal area annotation.
[0006] In order to solve the above technical problems, the present invention provides a method for generating an abnormal image, comprising:
[0007] A bounding box of the abnormal mask is randomly generated on the normal image until the generated bounding box overlaps with the object on the normal image, and the bounding box is used as the target bounding box;
[0008] Obtain the position description text of the abnormal mask according to the position information of the target bounding box in the normal image; construct the defect description text information of the abnormal image to be generated based on the position description text of the abnormal mask and the defect category description text;
[0009] The normal image is input into the trained diffusion model, and the variational autoencoder is used to convert the normal image into latent variables to generate latent features, and T'-step noise is gradually added to the latent features to obtain noise features;
[0010] The noise feature, the target bounding box, and the mask image after the target bounding box is removed from the normal image are connected to obtain the fused image feature;
[0011] The fused image features are input into the U-Net network in the diffusion model, and the fused image features are denoised step by step and iteratively. The defect description text information of the abnormal image to be generated is used as a condition to guide the sampling direction of the U-Net network until the sampling of step T' is completed;
[0012] The sampled features are decoded using a variational autoencoder to obtain the generated abnormal image;
[0013] The pixel-level difference between the normal image and the corresponding generated abnormal image is calculated and converted into a binary mask as the label of the generated abnormal image.
[0014] Preferably, the step of obtaining the position description text of the abnormal mask according to the position information of the target bounding box in the normal image includes:
[0015] The normal image is divided into four quadrants: upper left, lower left, upper right and lower right; if the target bounding box mainly exists in one quadrant, the quadrant is used as the location description text of the abnormal mask; if the target bounding box spans two quadrants, the connection between the two quadrants is used as the location description text of the abnormal mask; if the target bounding box spans more than two quadrants, the center of the normal image is used as the location description text of the abnormal mask.
[0016] Preferably, the step of gradually adding T'-step noise to the potential features comprises:
[0017] For any time step t satisfying 1≤t≤T', the data distribution of the noise characteristics satisfies The formula is:
[0018]
[0019] in, is the data distribution of the noise feature from step t-1 to step t, is the noise characteristic of the tth step, is the noise feature of the t-1th step, β tis the variance of the Gaussian noise added in the tth step, and I is the variance of the standard normal distribution.
[0020] Preferably, the step of inputting the fused image features into a U-Net network in a diffusion model, performing step-by-step iterative denoising on the fused image features, and using the defect description text information of the abnormal image to be generated as a condition to guide the sampling direction of the U-Net network includes:
[0021] Denoising starts from step T' until step 0, where the noise distribution of step t and step t-1 is expressed as:
[0022]
[0023] Where t is the time step, is the fused image feature of the t-1th step, is the fused image feature of the tth step, with 1≤t≤T', d is the condition, i.e., text information; is the data distribution of image feature fusion from step t to step t-1 under the guidance of condition d, is the variance of the Gaussian distribution of the fused image features predicted in the tth step under the guidance of condition d; is the mean of the Gaussian distribution of the fused image features predicted in the tth step under the guidance of condition d, expressed as:
[0024]
[0025] Among them, α t =1-β t , β t When adding noise step by step, the variance of the Gaussian noise added in the tth step; is the noise of the fused image features predicted in the tth step under the guidance of condition d.
[0026] Preferably, in the process of step-by-step iterative denoising of the fused image features, an additional guiding factor is used to control the degree of influence of the conditions on the generated results, and the formula is:
[0027]
[0028] Among them, λ is the guiding factor, which is used to control the influence of conditional generation and unconditional generation on the generation results. is the data distribution of the fused image features at step t under unconditional control, is the data distribution of the fused image features at step t under condition d.
[0029] Preferably, the calculation of the pixel-level difference between the normal image and the corresponding generated abnormal image is performed using the formula:
[0030]
[0031] Among them, x n represents a normal image, Based on x n The abnormal image generated, Represents the pixel-level difference between pixels of the normal image and the corresponding generated abnormal image.
[0032] Preferably, the pixel-level difference between the normal image and the corresponding generated abnormal image is scaled to the interval [0,1], and the binarization threshold is adaptively determined using Otsu's method, and the pixel-level difference between the scaled normal image and the corresponding generated abnormal image is binarized according to the threshold to obtain a binary mask.
[0033] Preferably, after calculating the pixel-level difference between the normal image and the corresponding generated abnormal image and converting the difference into a binary mask, the method further includes:
[0034] In the binary mask, any connected area with an area less than one tenth of the maximum area is removed, and then each pixel of the binary mask is connected to the remaining pixels using the dense conditional random field technology. It is determined whether any two pixels belong to the same category and marked to obtain the optimized binary mask.
[0035] Preferably, the process of training the diffusion model includes:
[0036] The abnormal category labels of the abnormal images in the dataset are converted into defect category description texts. Based on the position description texts of the abnormal masks in the abnormal images and the defect category description texts, the defect description text information of the abnormal images to be generated is constructed. The large-scale language model is used to generate several new defect category text information for each abnormal image.
[0037] Determine the minimum enclosing matrix of the mask of the abnormal image, and according to the minimum enclosing matrix, expand 5 pixels in the four directions of upward, downward, left and right through the rectangular expander to obtain the bounding box of the abnormal image mask;
[0038] The abnormal image is input into the diffusion model, the abnormal image is converted into a latent variable using a variational autoencoder, and T-step noise is gradually added to the latent variable to obtain a noise feature;
[0039] Performing a dot multiplication of the bounding box of the abnormal image mask and the input abnormal image to obtain a mask image with the defect area removed; inputting the mask image with the defect area removed into a variational autoencoder for encoding to obtain an encoded mask image; adjusting the bounding box of the abnormal image mask to be consistent with the size of the encoded mask image;
[0040] For any time step t, the noise feature of the tth step is connected with the bounding box of the resized abnormal image mask and the encoded mask image on the channel to form a fused image feature;
[0041] The fused image features are input into the U-Net network in the diffusion model, and the fused image features are denoised step by step iteratively. The text information of several defect categories corresponding to the abnormal image is used as a condition to guide the sampling direction of the U-Net network, and the training target is obtained to optimize the trainable parameters of the diffusion model; the trainable parameters are the fused image features in the tth step. To the t-1th step, image features are fused The probability distribution of The trainable parameters θ in ;
[0042] The formula for the training objective is:
[0043]
[0044] in, is the training target, E is the expectation, t is the time step, is the fused image feature of step 0, d is the condition, ∈ t is the noise at step t, is the prediction noise of the U-Net network at step t.
[0045] The present invention also provides an abnormal image generating device, comprising:
[0046] A target bounding box acquisition module is used to randomly generate a bounding box of an abnormal mask on a normal image until the generated bounding box overlaps with an object on the normal image, and the bounding box is used as a target bounding box;
[0047] A text information acquisition module is used to acquire the position description text of the abnormal mask according to the position information of the target bounding box in the normal image; based on the position description text of the abnormal mask and the defect category description text, construct the defect description text information of the abnormal image to be generated;
[0048] The noise feature acquisition module is used to input the normal image into the trained diffusion model, use the variational autoencoder to convert the normal image into a latent variable, generate potential features, and gradually add T' steps of noise to the potential features to obtain noise features;
[0049] A fusion image feature acquisition module is used to connect the noise feature, the target bounding box and the mask image after the target bounding box is removed from the normal image to obtain the fusion image feature;
[0050] The conditional sampling module is used to input the fused image features into the U-Net network in the diffusion model, perform step-by-step iterative denoising on the fused image features, and use the defect description text information of the abnormal image to be generated as a condition to guide the sampling direction of the U-Net network until the sampling of step T' is completed;
[0051] An abnormal image generation module is used to decode the sampled features using a variational autoencoder to obtain a generated abnormal image;
[0052] The mask acquisition module is used to calculate the pixel-level difference between the normal image and the corresponding generated abnormal image, and convert the difference into a binary mask as the label of the generated abnormal image.
[0053] The above technical solution of the present invention has the following beneficial effects compared with the prior art:
[0054] The abnormal image generation method described in the present invention constructs defect description text information of the abnormal image to be generated by randomly generating a bounding box and based on the position description text and defect category description text of the abnormal mask, and uses a trained diffusion model to achieve the use of normal images to generate a series of diversified abnormal images with high authenticity and binary masks. In the process of using abnormal images to train the diffusion model, the present invention uses a large-scale language model to generate a number of new defect description text information for each abnormal image, overcomes the difficulty of lacking sufficient abnormal samples by increasing the number of text information, and improves the diversity of the abnormal images finally generated. The present invention automatically generates a binary mask of the abnormal image by calculating the pixel-level difference between the normal image and the corresponding generated abnormal image, which reduces the workload of manual annotation, improves the efficiency of annotation, and can ensure the accuracy of the mask. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:
[0056] Figure 1 is a flow chart of a method for generating an abnormal image according to the present invention;
[0057] Figure 2 It is a structural diagram of an abnormal image generating method of the present invention. DETAILED DESCRIPTION
[0058] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0059] Reference Figure 1As shown, the present invention provides a method for generating an abnormal image, which uses the abnormal images of an existing data set to train a diffusion model, thereby generating an abnormal image and its corresponding abnormal mask on the normal image of the data set.
[0060] This example uses the public dataset MVTEC as the basis for image synthesis in industrial scenarios. The dataset contains 15 common industrial objects, each of which contains a large number of normal images without defects, a small number of abnormal images and their corresponding abnormal precise masks, as well as labels of the objects and defect types corresponding to each image.
[0061] When training the diffusion model, if only rough category information is provided without detailed defect description, it may make it difficult for the diffusion model to distinguish different types of defects. Therefore, before training the diffusion model, the defect category labels in the dataset need to be converted into defect description text information for the abnormal images to be generated, including:
[0062] The defect description text information of the abnormal image includes a position description text of the abnormal mask and a defect category description text.
[0063] The defect category description text is obtained by rewriting the abnormal category labels provided in the dataset. For example, if the object and defect category of an image are hazelnut and crack respectively, the defect description of the image can be expressed as "a significant damage and display the internal hazelnut".
[0064] The position description text of the abnormal mask is determined by analyzing the position of the abnormal mask in the abnormal image. For an abnormal image with only one abnormal mask, the abnormal image is divided into four quadrants: upper left, lower left, upper right and lower right. By analyzing the distribution of the abnormal masks in these quadrants, the position description text can be determined. If the abnormal mask mainly exists in one quadrant, the quadrant is used as the position description text of the abnormal mask; if the abnormal mask spans two quadrants, the connection between the two quadrants is used as the position description text of the abnormal mask; if the abnormal mask spans more than two quadrants, the center of the abnormal image is used as the position description text of the abnormal mask. For an image with multiple defect areas, the position of the defect area represented by each abnormal mask is judged separately to generate the corresponding position description text.
[0065] After obtaining the defect description text information of the abnormal image, a large-scale language model is used to generate several new defect description text information for each abnormal image, so as to obtain defect description text information that maintains the basic meaning of the original description but is more diverse, so as to enhance the ability of the diffusion model to understand various text information.
[0066] In order to enhance the diversity of generated results, this embodiment converts the anomalies of the abnormal image into bounding boxes when training the diffusion model. The specific process includes: determining the minimum enclosing matrix of the abnormal mask of the abnormal image, and based on the minimum enclosing matrix, expanding 5 pixels in the upward, downward, left and right directions respectively through a rectangular expander, in order to include more anomalies and their neighborhood areas in the bounding box to provide valuable context clues and obtain the bounding box of the abnormal image.
[0067] Reference Figure 2 As shown, the abnormal image is input into the diffusion model, and the defect description text information of the abnormal image is used as a condition to train the diffusion model, specifically including:
[0068] In the forward process, a variational autoencoder is used to convert the abnormal image into a latent variable, denoted as z a , and for the hidden variable z a Gradually add T-step noise to obtain the noise feature. For any time step t satisfying 1≤t≤T, its hidden variable The data distribution satisfies The expression of the data distribution is:
[0069]
[0070] in, is the data distribution of the noise feature from step t-1 to step t, is the noise characteristic of the tth step, is the noise feature of the t-1th step, β t is the variance of the Gaussian noise added in the tth step, and I is the variance of the standard normal distribution.
[0071] The bounding box m of the abnormal image b and the abnormal image x a Perform point multiplication to obtain the mask image x after removing the bounding box of the abnormal image a ·m b The mask image after removing the bounding box from the abnormal image is input into the variational autoencoder for encoding to obtain the encoded mask image z m ; Adjust the bounding box m of the abnormal image b and the encoded mask image z m The sizes are consistent to facilitate subsequent image connection.
[0072] For any time step t, the noise feature of the tth step Bounding box with resized anomaly image The encoded mask image z m Connect on the channel to form fused image features
[0073] In the reverse process, since it is impossible to directly infer the parameters required to remove the noise, a U-Net network is used to parameterize the noise. The goal of the U-Net network is to learn the probability distribution from step t to step t-1 Where θ is a trainable parameter. In order to simplify the training process, the present invention directly uses the U-Net network to predict noise, including:
[0074] Fusion of image features The U-Net network in the input diffusion model is used to fuse image features Perform step-by-step iterative denoising, and use several new defect description text information of abnormal images as conditions d to guide the sampling direction of the U-Net network, obtain the training target, and optimize the trainable parameters of the diffusion model. By incorporating several new defect description text information of abnormal images into the training target, the diffusion model can learn to align the sampling with the specific features of the defects in the abnormal images during the generation process. Therefore, the training target It can be expressed as:
[0075]
[0076] in, is the training target, E is the expectation, t is the time step, is the fused image feature of step 0, d is the condition, ∈ t is the noise at step t, is the prediction noise of the U-Net network at step t.
[0077] After obtaining the pre-trained diffusion model through the above process, the pre-trained diffusion model is used to generate defects on the normal image to obtain the abnormal image and its corresponding abnormal mask. The specific process is as follows:
[0078] A bounding box of the abnormal mask is randomly generated on the normal image until the generated bounding box overlaps with the object on the normal image, and the bounding box is used as the target bounding box.
[0079] The position description text of the abnormal mask is obtained according to the position information of the target bounding box in the normal image, including: dividing the normal image into four quadrants: upper left, lower left, upper right and lower right; if the target bounding box mainly exists in one quadrant, then the quadrant is used as the position description text of the abnormal mask; if the target bounding box spans two quadrants, then the connection between the two quadrants is used as the position description text of the abnormal mask; if the target bounding box spans more than two quadrants, then the center of the normal image is used as the position description text of the abnormal mask.
[0080] Based on the position description text and defect category description text of the abnormal mask, the defect description text information of the abnormal image to be generated is constructed.
[0081] The normal image x n Input the trained diffusion model and use the variational autoencoder to convert the normal image into a latent variable to generate the latent feature z n , and gradually add T'-step noise to the potential features, including:
[0082] For any time step t satisfying 1≤t≤T', the data distribution of the noise characteristics satisfies The formula is:
[0083]
[0084] in, is the data distribution of the noise feature from step t-1 to step t, is the noise characteristic of the tth step, is the noise feature of the t-1th step, β t is the variance of the Gaussian noise added in the tth step, and I is the variance of the standard normal distribution.
[0085] After gradually adding T' steps of noise, the noise characteristics are obtained
[0086] The noise characteristics Target bounding box m b and the mask image x after removing the target bounding box from the normal image n ·m b Connect to get the fused image features
[0087] Fusion of image features The U-Net network in the input diffusion model performs step-by-step iterative denoising on the fused image features, and uses the defect description text information of the abnormal image to be generated as a condition to guide the sampling direction of the U-Net network until the sampling of step T' is completed, that is, denoising is performed step by step from step T' to step 0. The noise distribution of step t and step t-1 is expressed as:
[0088]
[0089] Where t is the time step, is the fused image feature of the t-1th step, is the fusion image feature of the tth step, with 1≤t≤T', d is the condition, that is, the defect description text information of the abnormal image to be generated; is the data distribution of image feature fusion from step t to step t-1 under the guidance of condition d, is the variance of the Gaussian distribution of the fused image features predicted in the tth step under the guidance of condition d; is the mean of the Gaussian distribution of the fused image features predicted in the tth step under the guidance of condition d, expressed as:
[0090]
[0091] Among them, α t =1-β t , β t When adding noise step by step, the variance of the Gaussian noise added in the tth step; is the noise of the fused image features predicted in the tth step under the guidance of condition d.
[0092] The defect description text information of the abnormal image to be generated is used as a condition to guide the sampling direction of the U-Net network, guiding the target bounding box area to generate abnormal content corresponding to the defect category, which can ensure that the generated defects are consistent with the characteristics of the specified defect category.
[0093] In the process of step-by-step iterative denoising of the fused image features, an additional guiding factor λ is used to control the influence of the conditions on the generated results. The formula is:
[0094]
[0095] Among them, λ is the guiding factor, which is used to control the influence of conditional generation and unconditional generation on the generation result, and can enhance the influence of the defect description text information of the abnormal image to be generated on the final synthesized image. is the data distribution of the fused image features at step t under unconditional control, is the data distribution of the fused image features at step t under condition d.
[0096] The guiding factor can flexibly adjust the influence of the defect description text information of the abnormal image to be generated on the generated result, so that the user can control the strength of the abnormal effect as needed, thereby meeting the needs of different application scenarios.
[0097] After T' steps of iterative denoising, the fused image features of step 0 are obtained. The fused image features of step 0 Input to the variational autoencoder for decoding to obtain the generated abnormal image
[0098] In order to obtain the abnormal mask label of the abnormal image, the present invention calculates the normal image x n And the corresponding generated abnormal image The pixel-level difference between the normal image x n And the corresponding generated abnormal image The cosine distance score on each channel is:
[0099]
[0100] Among them, x n represents a normal image, Based on x n The abnormal image generated, Represents the pixel-level difference between the pixels of the normal image and the corresponding generated abnormal image. A higher S value indicates a larger difference between the two regions, indicating a higher probability that the region contains a defect.
[0101] The pixel-level difference between the normal image and the corresponding generated abnormal image is scaled to the interval [0,1], and the binarization threshold is adaptively determined using Otsu's method. The pixel-level difference between the scaled normal image and the corresponding generated abnormal image is binarized according to the threshold to obtain a binary mask.
[0102] In order to further eliminate independent areas in the generated abnormal image and separate the most significant anomalies, this embodiment removes any connected areas with an area less than one-tenth of the area of the largest area in the binary mask, because these connected areas may represent smaller or less significant anomalies. Then, the dense conditional random field technology is used to connect each pixel of the binary mask with the remaining pixels, and it is determined whether any two pixels are of the same category and marked to improve the accuracy and completeness of the defective areas in the generated abnormal image, and an optimized binary mask is obtained. as labels for the generated abnormal images.
[0103] The present invention constructs defect description text information of the abnormal image to be generated by randomly generating a bounding box and based on the position description text and defect category description text of the abnormal mask, thereby realizing the generation of a series of diversified abnormal images with high authenticity and binary masks using normal images. In the process of training the diffusion model, a large-scale language model is used to generate a number of new defect description text information for each abnormal image, and the difficulty of lacking sufficient abnormal samples is overcome by increasing the amount of text information, and the diversity of the generated abnormal images is improved. By calculating the pixel-level difference between the normal image and the corresponding generated abnormal image, the binary mask of the abnormal image is automatically generated, which reduces the workload of manual annotation, improves the efficiency of annotation, and can ensure the accuracy of the abnormal mask.
[0104] This embodiment also provides an abnormal image generating device, including:
[0105] A target bounding box acquisition module is used to randomly generate a bounding box of an abnormal mask on a normal image until the generated bounding box overlaps with an object on the normal image, and the bounding box is used as a target bounding box;
[0106] A text information acquisition module is used to acquire the position description text of the abnormal mask according to the position information of the target bounding box in the normal image; based on the position description text of the abnormal mask and the defect category description text, construct the defect description text information of the abnormal image to be generated;
[0107] The noise feature acquisition module is used to input the normal image into the trained diffusion model, use the variational autoencoder to convert the normal image into a latent variable, generate potential features, and gradually add T' steps of noise to the potential features to obtain noise features;
[0108] A fusion image feature acquisition module is used to connect the noise feature, the target bounding box and the mask image after the target bounding box is removed from the normal image to obtain the fusion image feature;
[0109] The conditional sampling module is used to input the fused image features into the U-Net network in the diffusion model, perform step-by-step iterative denoising on the fused image features, and use the defect description text information of the abnormal image to be generated as a condition to guide the sampling direction of the U-Net network until the sampling of step T' is completed;
[0110] An abnormal image generation module is used to decode the sampled features using a variational autoencoder to obtain a generated abnormal image;
[0111] The mask acquisition module is used to calculate the pixel-level difference between the normal image and the corresponding generated abnormal image, and convert the difference into a binary mask as the label of the generated abnormal image.
[0112] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0113] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.
[0114] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0116] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.
Claims
1. A method for generating an abnormal image, It is characterized in that include: A bounding box of the abnormal mask is randomly generated on the normal image until the generated bounding box overlaps with the object on the normal image, and the bounding box is used as the target bounding box; Obtain the position description text of the abnormal mask according to the position information of the target bounding box in the normal image; construct the defect description text information of the abnormal image to be generated based on the position description text of the abnormal mask and the defect category description text; The normal image is input into the trained diffusion model, and the variational autoencoder is used to convert the normal image into a latent variable, generate latent features, and gradually add Step noise and obtain noise characteristics; The noise feature, the target bounding box, and the mask image after the target bounding box is removed from the normal image are connected to obtain the fused image feature; The fused image features are input into the U-Net network in the diffusion model, and the fused image features are denoised step by step. The defect description text information of the abnormal image to be generated is used as a condition to guide the sampling direction of the U-Net network until completion. Sampling of steps; The sampled features are decoded using a variational autoencoder to obtain the generated abnormal image; The pixel-level difference between the normal image and the corresponding generated abnormal image is calculated and converted into a binary mask as the label of the generated abnormal image.
2. The abnormal image generation method according to claim 1, It is characterized in that The step of obtaining the position description text of the abnormal mask according to the position information of the target bounding box in the normal image includes: The normal image is divided into four quadrants: upper left, lower left, upper right and lower right; if the target bounding box mainly exists in one quadrant, the quadrant is used as the location description text of the abnormal mask; if the target bounding box spans two quadrants, the connection between the two quadrants is used as the location description text of the abnormal mask; if the target bounding box spans more than two quadrants, the center of the normal image is used as the location description text of the abnormal mask.
3. The abnormal image generation method according to claim 1, It is characterized in that The potential features are gradually added Step noise, including: To satisfy Any time step , the data distribution of the noise characteristics satisfies , the formula is: ; in, For Step to The data distribution of the step noise characteristics, For the The noise characteristics of the step, For the The noise characteristics of the step, For the The variance of the Gaussian noise added in the step, is the variance of the standard normal distribution.
4. The abnormal image generation method according to claim 1, It is characterized in that The step of inputting the fused image features into the U-Net network in the diffusion model, performing step-by-step iterative denoising on the fused image features, and using the defect description text information of the abnormal image to be generated as a condition to guide the sampling direction of the U-Net network includes: From Step 1 starts denoising gradually until step 0, where Step 1 The step noise distribution is expressed as: ; in, is the time step, For the The fusion image features of the step, For the The fusion image features of the step are , is the condition, i.e., text information; For the condition Under the guidance of Step to The data distribution of the image features is fused step by step. For the condition Under the guidance of The variance of the Gaussian distribution of the fused image features predicted by the first step; For the condition Under the guidance of The mean of the Gaussian distribution of the fused image features predicted by the first step is expressed as: ; in, , When adding noise gradually, The variance of the Gaussian noise added in the first step; For the condition Under the guidance of The noise of the fused image features predicted by the first step.
5. The abnormal image generation method according to claim 4, It is characterized in that In the process of step-by-step iterative denoising of the fused image features, an additional guiding factor is used to control the influence of the conditions on the generated results. The formula is: ; in, is a guiding factor, which is used to control the influence of conditional generation and unconditional generation on the generation results. For unconditional control The data distribution of the fused image features of the step, For the condition Under control The data distribution of the fused image features in the next step.
6. The abnormal image generation method according to claim 1, It is characterized in that The formula for calculating the pixel-level difference between the normal image and the corresponding generated abnormal image is: ; in, represents a normal image, Based on The abnormal image generated, Represents the pixel-level difference between pixels of the normal image and the corresponding generated abnormal image.
7. The abnormal image generation method according to claim 6, It is characterized in that The pixel-level difference between the normal image and the corresponding generated abnormal image is scaled to the interval [0, 1], and the binarization threshold is adaptively determined using Otsu's method. The pixel-level difference between the scaled normal image and the corresponding generated abnormal image is binarized according to the threshold to obtain a binary mask.
8. The abnormal image generation method according to claim 1, It is characterized in that After calculating the pixel-level difference between the normal image and the corresponding generated abnormal image and converting the difference into a binary mask, the method further includes: In the binary mask, any connected area with an area less than one tenth of the maximum area is removed, and then each pixel of the binary mask is connected to the remaining pixels using the dense conditional random field technology. It is determined whether any two pixels belong to the same category and marked to obtain the optimized binary mask.
9. The abnormal image generation method according to claim 1, It is characterized in that The process of training the diffusion model includes: The abnormal category labels of the abnormal images in the dataset are converted into defect category description texts. Based on the position description texts of the abnormal masks in the abnormal images and the defect category description texts, the defect description text information of the abnormal images to be generated is constructed. The large-scale language model is used to generate several new defect category text information for each abnormal image. Determine the minimum enclosing matrix of the mask of the abnormal image, and according to the minimum enclosing matrix, expand 5 pixels in the four directions of upward, downward, left and right through the rectangular expander to obtain the bounding box of the abnormal image mask; The abnormal image is input into the diffusion model, and the abnormal image is converted into a latent variable using a variational autoencoder, and the latent variable is gradually added Step noise and obtain noise characteristics; Performing a dot multiplication of the bounding box of the abnormal image mask and the input abnormal image to obtain a mask image with the defect area removed; inputting the mask image with the defect area removed into a variational autoencoder for encoding to obtain an encoded mask image; adjusting the bounding box of the abnormal image mask to be consistent with the size of the encoded mask image; For any time step , will The noise features of the first step are connected with the bounding box of the resized abnormal image mask and the encoded mask image on the channel to form a fused image feature; The fused image features are input into the U-Net network in the diffusion model, and the fused image features are denoised step by step iteratively. The text information of several defect categories corresponding to the abnormal image is used as a condition to guide the sampling direction of the U-Net network, and the training target is obtained to optimize the trainable parameters of the diffusion model. The trainable parameters are Step fusion of image features To Step fusion of image features The probability distribution of The trainable parameters in ; The formula for the training objective is: ; in, For training goals, For expectations, is the time step, is the fused image feature of step 0, As a condition, For the The noise of the step, For U-Net network The prediction noise of the step.
10. An abnormal image generating device, It is characterized in that include: A target bounding box acquisition module is used to randomly generate a bounding box of an abnormal mask on a normal image until the generated bounding box overlaps with an object on the normal image, and the bounding box is used as a target bounding box; A text information acquisition module is used to acquire the position description text of the abnormal mask according to the position information of the target bounding box in the normal image; based on the position description text of the abnormal mask and the defect category description text, construct the defect description text information of the abnormal image to be generated; The noise feature acquisition module is used to input the normal image into the trained diffusion model, use the variational autoencoder to convert the normal image into a latent variable, generate latent features, and gradually add the latent features. Step noise and obtain noise characteristics; A fusion image feature acquisition module is used to connect the noise feature, the target bounding box and the mask image after the target bounding box is removed from the normal image to obtain the fusion image feature; The conditional sampling module is used to input the fused image features into the U-Net network in the diffusion model, perform step-by-step iterative denoising on the fused image features, and use the defect description text information of the abnormal image to be generated as a condition to guide the sampling direction of the U-Net network until completion. Sampling of steps; An abnormal image generation module is used to decode the sampled features using a variational autoencoder to obtain a generated abnormal image; The mask acquisition module is used to calculate the pixel-level difference between the normal image and the corresponding generated abnormal image, and convert the difference into a binary mask as the label of the generated abnormal image.
