Abnormal sample detection system based on diffusion model

Abstract

The invention belongs to the technical field of anomaly detection, and particularly relates to an abnormal sample detection system based on a diffusion model. The system comprises a word embedding vector training module which applies a set number of abnormal samples, initializes the abnormal samples into a word embedding vector, and inputs the word embedding vector into a diffusion model for guiding the diffusion model to learn; the abnormal sample generation module is used for generating an abnormal sample by combining the word embedding vector trained by the word embedding vector training module with the abnormal mask, and inputting the abnormal sample into the abnormal reconstruction module; the anomaly reconstruction module is used for carrying out image reconstruction on the input anomaly sample, carrying out loss calculation on the reconstructed image and the corresponding normal sample to train a model, splicing the reconstructed image and the anomaly sample and then inputting the spliced image and anomaly sample into the anomaly judgment module; and the abnormity judgment module is used for generating a model, speculating an abnormal mask, and comparing the abnormal mask with an actual abnormal mask to calculate loss.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of anomaly detection, and particularly relates to an anomaly sample detection system based on a diffusion model. BACKGROUND

[0002] In modern industrial manufacturing, with the continuous improvement of production efficiency, how to ensure the efficient operation of equipment and the stable quality of products has become a key factor of enterprise competitiveness. The traditional quality control method relying on manual inspection and experience judgment has been difficult to meet the needs of high-speed, continuous and complex industrial production. Therefore, industrial anomaly detection, as a key technology for automatically identifying the abnormal state of products using sensor data, image analysis or artificial intelligence algorithms, has attracted increasing attention.

[0003] However, the extreme scarcity of abnormal data in real scenarios makes the anomaly detection task (including image-level classification and pixel-level segmentation) extremely challenging. In the face of the reality of the scarcity of abnormal data, some studies have proposed unsupervised learning methods to eliminate the need for abnormal data, which are mainly divided into two categories: reconstruction-based methods and feature embedding-based methods. However, the absence of abnormal samples makes these methods not perform as expected in the anomaly segmentation task. Therefore, some researchers have begun to explore how to generate abnormal samples and use supervised learning to improve the accuracy of anomaly detection and anomaly segmentation tasks. For example, CutPaste generates abnormalities by randomly cropping and randomly pasting to other areas, and DRAEM synthesizes abnormalities by mixing external datasets with normal images. Although these methods have made some progress, the diversity and authenticity of the generated abnormalities are far from what people expect.

[0004] Therefore, in order to solve the problem of insufficient diversity and authenticity of generated abnormalities, people have begun to explore image generation models that can produce more diverse and realistic images. In recent years, diffusion models (Diffusion Models) as a powerful probabilistic modeling method have made breakthroughs in image generation, restoration, reconstruction and other tasks. Diffusion models simulate the gradual "anti-diffusion" process of images from noise to real images, and can reconstruct complex structures with high fidelity, with strong generation and reconstruction capabilities. Using diffusion models in the field of industrial anomaly detection can generate a large number of more realistic and diverse abnormalities, but existing methods of generating abnormalities using diffusion models have complex mask generation, low generation efficiency, or simple mask generation, which cannot reflect the diversity of abnormal forms in the real world.

[0005] Therefore, it is very important to design an anomaly sample detection system based on a diffusion model that can generate high-quality and efficient industrial abnormal samples and greatly improve the performance of anomaly detection and anomaly segmentation tasks. SUMMARY

[0006] The present application is to overcome the prior art, the traditional anomaly detection field exists in the few abnormal samples, the abnormal segmentation task performance is poor and the existing diffusion model based on the generation of abnormal sample generation efficiency is low, the mask form is not enough real, provide a kind of diffusion model based on abnormal sample detection system for realizing the high quality, high efficiency generation of industrial abnormal sample, and can greatly improve the performance of anomaly detection and abnormal segmentation task.

[0007] In order to achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0008] The abnormal sample detection system based on diffusion model comprises:

[0009] The word embedding vector training module: a set number of abnormal samples are applied to initialize a word embedding vector, and the word embedding vector is input into the diffusion model to guide the diffusion model learning;

[0010] The abnormal sample generation module: the word embedding vector trained by the word embedding vector training module is combined with the abnormal mask to generate the abnormal sample, and the abnormal sample is input into the abnormal reconstruction module;

[0011] The abnormal reconstruction module: the input abnormal sample is reconstructed into an image, and the reconstructed image and the corresponding normal sample are used to calculate the loss of the trained model; the reconstructed image and the abnormal sample are spliced and input into the abnormal discrimination module;

[0012] The abnormal discrimination module: the model is generated and the abnormal mask is speculated, and then compared with the actual abnormal mask to calculate the loss.

[0013] As a preferred, the specific process of the word embedding vector training module is as follows:

[0014] By combining the diffusion model with the cross attention mechanism, the keyword "defect" is initialized as a word embedding vector through the natural language processing model, and then the word embedding vector is input into the diffusion model to guide the diffusion model learning;

[0015] The loss function L of the diffusion model is set as follows: DM As follows:

[0016]

[0017] The loss function is the core training target in the diffusion model, which is used to guide the neural network to learn the ability to restore the original image from the noisy image, specifically:

[0018] Noise is sampled from Gaussian distribution Then the noise is added to the image x, and the noisy image x is obtained after t-step noise addition tWhen the diffusion model receives the image x after adding noise t times t and the corresponding number of times t after adding noise, the prediction function θ Start calculating the added noise and compare it with the actual added noise to calculate the corresponding mean square error MSE. Through this core loss function, the model can continuously narrow the gap between the predicted added noise and the actual added noise, and finally learn how to remove image noise, thereby mastering the ability to generate high-quality images.

[0019] As a preferred, the word embedding vector training module specifically further includes the following processes:

[0020] Set the loss function L of the latent diffusion model LDM As follows:

[0021]

[0022] Specifically, represents encoding the image x, represents encoding the text y through the natural language processing model, and then introducing the cross-attention mechanism to predict the noise related to a certain text y added to the image after t times, and finally optimizing the latent diffusion model by calculating the mean square error MSE between and the actual noise θ (y) represents encoding the text y through the natural language processing model, and then introducing the cross-attention mechanism to predict the noise related to a certain text y added to the image after t times, and finally optimizing the latent diffusion model by calculating the mean square error MSE between and the actual noise

[0023] This loss function is the core training target of the latent diffusion model (LDM), which optimizes the original basic diffusion model. In the original diffusion model, the operation is based on the original high-dimensional image space, resulting in high training overhead and low generation efficiency, and lacking flexibility in tasks that require the introduction of conditional information (such as class labels, masks, or semantic prompts). To solve these problems, researchers proposed the idea of diffusion learning in the latent space, i.e. first compressing the image into a low-dimensional latent representation through an encoder, and then training the diffusion model in this compressed space. This approach not only significantly reduces computational cost, but also facilitates the combination of various conditional information.

[0024] Select a pre-trained diffusion model, and then optimize the word embedding vector v to enhance the representation of v for abnormal features. The specific formula is as follows:

[0025]

[0026] where, represents a normal sample; v is the word embedding vector after encoding the text θ (y); v * is the word embedding vector that minimizes the loss function.

[0027] Optimize the loss function of the latent diffusion model to obtain a new loss function L'LDM The loss of focusing the latent diffusion model on the abnormal area is specifically formulated as follows:

[0028]

[0029] wherein, represents the mask of the image area, and represents the element-wise product of two matrices.

[0030] In the learning process of abnormal features, the model only needs to learn part of the mask, so the basic latent diffusion model loss function needs to be optimized to make the model focus on the loss of the abnormal area and more accurately learn the abnormal features of the picture. In this way, the model can only calculate the loss on the area of interest, thereby more effectively focusing on the important area in the image related to the semantic condition. This mechanism is particularly suitable for tasks such as anomaly detection and image repair, and helps to improve the robustness and accuracy of local recognition and generation. After obtaining the trained abnormal feature representation of multiple categories and multiple abnormal types, the next step of abnormal picture generation can be performed.

[0031] Preferably, in the abnormal sample generation module, an abnormal mask generation process and an abnormal sample generation process are included; the abnormal mask generation process is as follows:

[0032] An abnormal generation template is formulated for each type of abnormality, and then an abnormal mask is generated by framing in the abnormal generation template, and the framed mask is subjected to Gaussian noise processing, so that it is no longer a simple rectangle, but an abnormal mask with diverse contours that can well imitate the shapes of real-world abnormalities, and its generation efficiency is higher than that of using a new generation network;

[0033] Or use the abnormal mask of the used abnormal sample to generate an abnormal mask by rotation, mirroring and cropping. Such an abnormal mask generation method generates an abnormal mask that is more consistent with real semantics.

[0034] Preferably, the abnormal sample generation process is as follows:

[0035] The trained word embedding vector is used in the corresponding mask area M box to generate an abnormal sample, and a normal image is input to be encoded to obtain a noisy image z t ′ is the same noisy image as at the beginning, which is continuously denoised and updated by the diffusion model after being input to the diffusion model, M box is the mask area, the normal area is 0, and the abnormal area is 1; z tIt is an abnormal region z t ′ and normal area A noisy image created by stitching together, z t When the input diffusion model is used for denoising, it only denoises and restores normal regions in normal regions, while focusing only on z-regions in abnormal regions. t Denoising is performed on the z-axis, and the image after each denoising step is updated to z. t Subsequently, the abnormal region z t ′ and normal area The final abnormal sample was obtained by splicing the two parts together.

[0036] Among them, z t The calculation formula is as follows:

[0037]

[0038] Preferably, the anomaly reconstruction module operates as follows:

[0039] abnormal samples The input to the reconstruction network yields a reconstructed image, which is then compared with normal samples. Loss calculations are performed to train the anomaly detection model, enabling the normal reconstruction of anomalous samples. The reconstructed image is then compared with the anomalous sample. The images are stitched together to obtain images containing both abnormal and normal features, and then input into the anomaly detection module. This helps the anomaly detection module distinguish between abnormal and normal features and better calculate which pixels are more likely to be abnormal pixels.

[0040] Preferably, the anomaly detection module operates as follows:

[0041] The result of the anomaly reconstruction module is input, and then the anomaly discrimination module generates a prediction of the anomaly mask for the input anomaly image. This anomaly mask is then compared with the mask region M used when generating the anomaly. box The loss is calculated using the following formula:

[0042]

[0043] When the abnormal confidence level of a certain pixel When the value is greater than or equal to a predetermined threshold τ, the corresponding pixel is a normal pixel, and δ is... (i,j) δ is 1; otherwise δ (i,j) It is set to 0, and is used in the subsequent total segmentation loss function L′. seg The calculation is performed using the following formula;

[0044] L′ seg =M box ⊙(1-δ)⊙L seg+ (1-M box ) * L seg

[0045] wherein, L seg is a conventional segmentation loss function, in the mask area according to the judgment of (1-d) to calculate the loss, and in other normal area directly using the conventional segmentation loss function is added to obtain the final segmentation loss, to train the segmentation network, and get more real abnormal segmentation map.

[0046] Compared with the prior art, the present application has the beneficial effects that: (1) the word embedding vector training module of the present application uses a small amount of abnormal samples to construct a conditional representation of abnormal semantics (such as "defect"), and guides the diffusion model to learn abnormal features through a natural language model (such as BERT, CLIP, etc.) and a cross-attention mechanism; secondly, the abnormal generation module combines the semantic vector and the abnormal mask, and only denoises and generates abnormal images in the mask area through the latent diffusion model, effectively controlling the morphology and position of the abnormality, realizing high-quality and controllable abnormal sample synthesis; finally, the abnormal detection module uses a reconstruction subnetwork to restore normal images from the input image, and inputs the reconstructed image and the original image to a discriminant subnetwork, and trains by comparing the real mask and the predicted mask, thereby realizing accurate abnormal detection and segmentation; the combination of the three modules effectively alleviates the problems of scarcity of abnormal samples, untruthful generation, and low subsequent detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0047] Fig. 1 is a functional architecture diagram of the word embedding vector training module in the present application;

[0048] Fig. 2 is a functional architecture diagram of the abnormal sample generation module in the present application;

[0049] Fig. 3 is a functional architecture diagram of the abnormal detection module in the present application;

[0050] Fig. 4 is a framework diagram of the abnormal sample detection system based on the diffusion model of the present application. DETAILED DESCRIPTION

[0051] In order to more clearly illustrate the embodiments of the present application, the specific embodiments of the present application will be described below with reference to the drawings. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained from these drawings without creative labor, and other embodiments can also be obtained.

[0052] As Figs. 1 to 4As shown, the present application provides an abnormal sample detection system based on a diffusion model, mainly comprising the following modules:

[0053] The word embedding vector training module: a certain number of abnormal samples are applied to initialize a word embedding vector, and the word embedding vector is input into the diffusion model to guide the learning of the diffusion model.

[0054] The abnormal sample generation module: the word embedding vector trained by the word embedding vector training module is combined with the abnormal mask to generate the abnormal sample, and the abnormal sample is input into the abnormal reconstruction module.

[0055] The abnormal reconstruction module: the input abnormal sample is reconstructed, the reconstructed image is calculated with the corresponding normal sample to train the model, and the reconstructed image and the abnormal sample are spliced and input into the abnormal discrimination module.

[0056] The abnormal discrimination module: the model is generated and the abnormal mask is inferred, and then compared with the actual abnormal mask to calculate the loss.

[0057] Further, the specific action process of each module is as follows:

[0058] The word embedding vector training module: responsible for using the cross-attention mechanism and the core loss function of the latent diffusion model: and a small amount of abnormal samples to train a word embedding vector containing several hundred parameters, so that the vector can represent the abnormal features of this part of abnormal samples and be applied to the subsequent abnormal sample generation, and the loss function θ (·) is the predicted noise after inputting the parameters, ∈ is the actual noise added at step t, ∈ θ (∈(x),t,τ θ (y)) transmits the image code ∈(x), the noise step t and the prompt direction τ θ (y) into the diffusion model to significantly reduce the calculation cost of the diffusion model, and facilitate the combination with various conditional information, and the specific implementation process comprises the following steps:

[0059] Select a certain amount of known abnormal samples and their respective corresponding as x input to ∈(x) to encode the image for subsequent use of the cross-attention mechanism to train the word embedding vector, and then input the determined prompt word "defect" as y into the natural language processing model such as BERT or CLIP to obtain τ θ (y), that is, the word embedding vector v to be trained later, and then input the encoded and v into the pre-trained diffusion model, Furthermore, considering the needs of the actual task, we focus on learning the loss in the anomaly mask region, and do not calculate the loss for regions outside the anomaly mask. Finally, we apply the formula... We need to minimize the loss function of the potential diffusion model. * As the final word embedding vector.

[0060] Anomaly Generation Module: Based on the types of industrial products for which anomaly samples need to be generated, two anomaly mask generation methods are proposed. One method involves defining a rectangle within a predetermined anomaly template area to generate an anomaly mask, then applying Gaussian noise to the defined mask to make it no longer a simple rectangle but have various different contours to better mimic the shapes of real-world anomaly masks. The second method utilizes previously used... A certain number of anomaly masks are obtained by mirroring, selecting, and cropping, and then anomaly mask region M is generated. box Then, the normal image enter Obtain the image with added noise And according to Go and keep updating z t ′、 and z t M box This is an abnormal mask region; normal regions are represented by 0, and abnormal regions by 1. When denoising the normal region, only the abnormal regions are denoised. Denoising is achieved in the z-region, while in the outlier region, the focus is on z-sounding. t The final abnormal sample is generated by concatenating the denoising results of the last two parts.

[0061] Finally, in the anomaly detection module, normal samples need to be... Corresponding abnormal samples and the anomaly mask region M box The input is fed into the anomaly detection module. Taking the anomaly detection method DRAEM as an example, its anomaly detection module is divided into two parts, including an anomaly reconstruction module and an anomaly discrimination module.

[0062] Will and M box After the input is processed by these two modules, the model's predicted mask image can be obtained. Then for M box as well as Segmentation loss is calculated, but since outlier samples are generated by a diffusion model, M... box Not all pixels within the range are abnormal, so it is necessary to check each pixel individually and apply the formula. Generate a probability distribution map, and finally combine it with the actual segmentation loss function formula L′. seg =Mbox ⊙(1-δ)⊙L seg +(1-M box )⊙L seg , only the segmentation loss of the pixels located in the abnormal mask area (M box ) and judged to be abnormal ((1-δ) is 1, that is, δ is 0) is calculated, and then the final actual loss is obtained.

[0063] In summary, the present application relates to an abnormal sample detection system based on a diffusion model, which aims to generate a variety of abnormal samples that conform to real semantics through a diffusion model, and use them to improve existing abnormal detection systems, improve the quality and efficiency of abnormal sample generation, and enhance the segmentation ability of abnormal detection systems. The technical scheme of the present application includes: constructing a support abnormal set by applying a small number of existing abnormal samples, and inputting the support abnormal set into a pre-trained latent diffusion model. Through the cross-attention mechanism, the training of the word embedding vector is completed by combining the word embedding vector generated by the natural language model coding, which lays the foundation for subsequent abnormal sample generation. After obtaining the word embedding vector that can represent the abnormal features, an abnormal mask generation module is applied to generate the corresponding abnormal mask to specify the area that needs to generate abnormalities. The abnormal mask, normal image and trained word embedding vector are input into the diffusion model to complete the generation of abnormal samples. Finally, the generated abnormal samples and their corresponding abnormal masks are reformatted to produce a new abnormal detection dataset for subsequent abnormal detection systems. Through the abnormal sample detection system based on the diffusion model proposed in the present application, high-quality and efficient generation of industrial abnormal samples can be realized, which greatly improves the performance of the industrial abnormal detection system for abnormal detection and abnormal segmentation tasks.

[0064] From the above description, it can be clearly seen that the present application proposes an abnormal sample detection system based on a diffusion model, which aims to solve the problems of few abnormal samples in the traditional abnormal detection field, poor performance of abnormal segmentation tasks, and low efficiency and unrealistic form of existing abnormal samples generated based on the diffusion model.

[0065] The above only describes the preferred embodiments and principles of the present application in detail. For ordinary skilled persons in the art, the specific implementation methods will change according to the ideas provided by the present application, and these changes should be considered as the protection scope of the present application.

Claims

1. A system for detecting abnormal samples based on a diffusion model, characterized in that, The application comprises: a word embedding vector training module: a set number of abnormal samples are applied to initialize a word embedding vector, which is input into a diffusion model to guide the learning of the diffusion model; an abnormal sample generation module: the word embedding vector trained by the word embedding vector training module is combined with an abnormal mask to generate abnormal samples, which are input into an abnormal reconstruction module; an abnormal reconstruction module: the input abnormal samples are reconstructed, and the reconstructed images are combined with the corresponding normal samples to calculate the loss of the model, and the reconstructed images and abnormal samples are input into an abnormal discrimination module; an abnormal discrimination module: a model is generated to speculate the abnormal mask, which is compared with the actual abnormal mask to calculate the loss.

2. The diffusion model based abnormal sample detection system of claim 1, wherein, The specific process of the word embedding vector training module is as follows: By combining the diffusion model with the cross-attention mechanism, the keyword "defect" is initialized as a word embedding vector through a natural language processing model, and the word embedding vector is input into the diffusion model to guide the learning of the diffusion model; Loss function L of setting diffusion model DM As follows: The loss function is the core training target in the diffusion model, which is used to guide the neural network to learn the ability to restore the original image from the noisy image, specifically: Sampling noise from a Gaussian distribution The noise is then added to the image x to obtain the noisy image x t When the diffusion model receives the image x after t steps of noise addition t And the corresponding number of noise addition t, the prediction function θ The added noise is calculated and compared with the actual added noise 3. The diffusion model based anomalous sample detection system of claim 2, wherein, The word embedding vector training module further comprises the following process: Setting the loss function L of the potential diffusion model LDM As follows: Specifically, ∈(x) represents encoding the image x, τ θ (y) represents encoding the text y by a natural language processing model, then introducing a cross-attention mechanism to predict the noise about the text y added to the image after t times, and finally optimizing the latent diffusion model step by step by calculating the mean square error MSE between and the actual noise ∈. Select a pre-trained diffusion model, and then enhance the representation of the abnormal feature of v by optimizing the word embedding vector v, specifically as follows: wherein, represents a normal sample; v is τ θ (y) a word embedding vector after encoding the text; v * is the word embedding vector that minimizes the loss function; Loss function L for potential diffusion model LDM Optimization is performed to obtain L' LDM Loss for focusing potential diffusion model on abnormal region, specific formula as follows: wherein a mask representing the image region, and denotes an element-wise multiplication of two matrices.

4. The diffusion model based abnormal sample detection system of claim 3, wherein, In the abnormal sample generation module, the abnormal mask generation process and the abnormal sample generation process are included; the abnormal mask generation process is as follows: An abnormal generation template is prepared for each type of abnormality, and then the abnormal mask is generated by framing in the abnormal generation template, and the framed mask is processed by Gaussian noise to make it no longer a simple rectangle, but an abnormal mask with a variety of contours that can well imitate the shape of real-world abnormalities; Or use the abnormal mask of the used abnormal sample to generate the abnormal mask by rotation, mirroring and cropping.

5. The diffusion model based anomalous sample detection system of claim 4, wherein, The abnormal sample generation process is as follows: using the trained word embedding vectors in the corresponding mask region M box Abnormal sample generation is performed, and normal images Input Encoding is performed to obtain a noisy image z t ′ is the initial image with the same noisy image, which is continuously denoised and updated by the diffusion model after being input to the diffusion model, M box is the mask region, 0 for the normal region and 1 for the abnormal region; z t is the abnormal region z t ′ and the normal region are spliced to form a noisy image, z t When input to the diffusion model for denoising, only the normal image region is denoised and restored in the normal region, while only the denoising of z t ′ is focused on, and the image after each step of denoising is updated to z t , and then the restored z t ′ part is spliced to obtain the final abnormal sample where z t The calculation formula is as follows:

6. The diffusion model based anomalous sample detection system of claim 5, wherein, The specific process of the abnormal reconstruction module is as follows: Abnormal samples are classified The input reconstruction network obtains a reconstructed image, and the reconstructed image is obtained by matching the normal sample The loss calculation trains the abnormal detection model, and realizes the normal restoration of the abnormal sample; and the reconstructed image and the abnormal sample are spliced to obtain an image containing abnormal features and normal features, and the image is input into the abnormal discrimination module.

7. The diffusion model based anomalous sample detection system of claim 6, wherein, The specific process of the abnormal discrimination module is as follows: generating a prediction of an exception mask for the input exception picture, and then comparing the exception mask with the mask region M used when generating the exception box The loss is calculated by comparison, and the specific formula is as follows: When the pixel's abnormal confidence level When the value is greater than or equal to a predetermined threshold τ, the corresponding pixel is a normal pixel, and δ is... (i,j) δ is 1; otherwise δ (i,j) The value is 0, which is used in the subsequent overall segmentation loss function L′. seg The calculation is performed using the following formula; L' seg = M box ⊙(1-δ)⊙L seg +(1-M box )⊙L seg wherein, L seg is a conventional segmentation loss function, in the mask region according to (1-δ) to calculate the loss, and in other normal region directly using the conventional segmentation loss function is added to get the final segmentation loss.

Images