Industrial image anomaly detection method based on pseudo anomaly sample generation

Abstract

The invention relates to an industrial image anomaly detection method based on pseudo anomaly sample generation, and relates to the field of industrial image anomaly detection. Comprising the following steps: constructing an SAD model; inputting the training industrial image data set into the SAD model for training, and extracting image features by using the pre-trained WideResnet50; performing projection adaptation on the image features to obtain adapted features; gaussian noise is added to the adapted features; generating a corresponding pseudo-abnormal image for the training data set image, extracting image features and performing adaptation; inputting the adapted normal image features and abnormal image features into a discriminator for anomaly detection; and segmenting the images detected to be abnormal into patch level features, and generating a thermodynamic diagram for abnormal positioning. According to the embodiment of the invention, the training data set is expanded by performing data enhancement on the training sample; a discriminator is helped to better distinguish the difference between a normal image and an abnormal image by generating a pseudo abnormal image corresponding to the training image.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of industrial image anomaly detection, and in particular, to an industrial image anomaly detection method based on pseudo-abnormal sample generation. Background Art

[0002] In recent years, with the advancement of science and technology and the development of computer vision, artificial intelligence and manufacturing, industrial image anomaly detection has gradually become a hot issue in the field of computer vision. With the help of deep learning methods, intelligent visual systems can be built to identify whether there are abnormalities in industrial products on the production line, assisting or even replacing manual work to complete most of the industrial product quality inspection work. Under the condition that the number of normal samples and abnormal samples used in training is small, industrial image anomaly detection mainly uses semi-supervised methods to detect whether there are anomalies in industrial product images. Most of the previous research methods in the field use reconstruction method processes, but the reconstruction method has a complex network structure, slow training speed and running speed. At the same time, the reconstruction network with good performance can also reconstruct anomalies very well, resulting in the model being unable to distinguish between normal images and abnormal images. In order to make normal images and abnormal images better distinguished in high-dimensional space, the method of adding noise to the training image and then denoising and the method of synthesizing pseudo-anomalies by pasting other images on the training image came into being. Both methods can well pull the distribution of normal images and abnormal images apart, so that the model can better distinguish normal images from abnormal images, but the anomalies generated by these two methods are still not close enough to the anomalies in the real world, and the network structure is still relatively complex. At the same time, the effectiveness of generating anomalies at the feature level for improving deep learning network training is ignored. Therefore, how to improve the network structure, simplify the training model, and improve the speed of model training and actual detection application, as well as how to better improve the method of synthesizing anomalies so that the generated anomalies can be closer to the anomalies in the real world and better allow the model to distinguish normal images from abnormal images has become the key to the subsequent research on industrial anomaly detection using traditional methods. Summary of the invention

[0003] The technical problems to be solved by the present invention are:

[0004] In order to avoid the shortcomings of the prior art, the present invention provides an industrial image anomaly detection method based on pseudo-abnormal sample generation, which is used to solve the problems in the prior art such as complex network structure, slow training and running speed, unrealistic generated anomalies, and poor generation of anomalies at the feature level.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] An industrial image anomaly detection method based on pseudo-abnormal sample generation, characterized by comprising:

[0007] Constructing a SAD model, the SAD model includes an image feature extractor, an image feature adapter, an abnormal image generation module and a discriminator; wherein,

[0008] The image feature extractor is a WideResnet50 backbone network with frozen parameters after pre-training with the ImageNet dataset, the image feature adapter uses a single-layer fully connected layer for projection adaptation, the abnormal image generation module uses the DTD dataset image as the texture abnormality source, and the discriminator uses a two-layer multi-layer perceptron for discrimination;

[0009] Inputting the training image into the SAD model for training;

[0010] Using the abnormal image generation module to generate a texture abnormal image corresponding to the training image;

[0011] Extracting image features of the training image and the texture abnormality image using the frozen image feature extractor;

[0012] Using the image feature adapter to adapt the image features of the training image and the image with abnormal texture, to obtain the training image adaptation features and the image with abnormal texture adaptation features;

[0013] A method of adding Gaussian noise to the training image adaptation feature is used to obtain a noise abnormality image adaptation feature;

[0014] Using the discriminator, the training image adaptation features are compared with the texture abnormal image adaptation features and the noise abnormal image adaptation features on the discriminator to determine whether the training image is an abnormal image;

[0015] The prediction result and abnormal heat map of the training image are obtained according to the discriminator's discriminant result.

[0016] A further technical solution of the present invention is as follows: the process of generating the texture abnormality image using the training image comprises:

[0017] Generate two-dimensional Perlin noise, extract peak values ​​from the two-dimensional Perlin noise to obtain a preliminary noise image, extract image foreground from the training image, perform a first operation on the preliminary noise image and the image foreground of the training image to obtain a positioning noise image;

[0018] Performing a second operation on the located noise image and a randomly selected image in the DTD texture data set to obtain an abnormal noise image;

[0019] A third operation is performed on the abnormal noise image and the training image to obtain a texture abnormality image corresponding to the training image.

[0020] A further technical solution of the present invention is as follows: the first operation is an element-by-element multiplication operation, and the second operation and the third operation are mask operations.

[0021] A further technical solution of the present invention: the step of extracting the image features of the training image and the image with abnormal texture by using the frozen image feature extractor further includes:

[0022] The patch-level feature size is determined according to a predetermined neighborhood size, and the extracted patch-level features are aggregated to obtain the overall image features.

[0023] A further technical solution of the present invention is as follows: the process of using an image feature adapter to perform image feature adaptation on the image features of a training image and an image with abnormal texture corresponding to the training image comprises:

[0024] The feature adaptation operation is performed on the image features of the training image and the texture abnormality image corresponding to the training image at a patch level.

[0025] A further technical solution of the present invention is to generate a noise abnormal image adaptation feature corresponding to the training image by adding Gaussian noise to the training image adaptation feature.

[0026] A further technical solution of the present invention is to use the discriminator to train the training image adaptation features by comparing and learning with the texture abnormality image adaptation features and the noise abnormality image adaptation features on the discriminator, and finally the discriminator gives a discrimination result.

[0027] A further technical solution of the present invention: the discriminator comprises:

[0028] Two-layer multilayer perceptron; the loss function is L1 truncation loss.

[0029] A further technical solution of the present invention: the method further comprises:

[0030] The test image is input into the trained SAD model to obtain a prediction result of the test image.

[0031] A further technical solution of the present invention: the step of inputting the test video into the trained SAD model further includes:

[0032] For the anomaly detection, after the discriminator gives an anomaly judgment, the anomaly score of the patch-level image features is calculated, and the heat map is drawn to show the anomaly location.

[0033] The beneficial effects of the present invention are:

[0034] In the embodiments of the present disclosure, through the above-mentioned industrial image anomaly detection method, image anomaly detection can be completed through a simple image feature extractor, an image feature adapter and a discriminator. The network structure is simple, and the training and running speeds are faster. In addition, artificially synthesized texture abnormal images and Gaussian noise are directly added to the adaptation features to construct abnormal image features. Anomalies that are closer to real life can be generated, and generating anomalies at the feature level can make the model recognize anomalies better. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0036] Figure 1 A step diagram showing an industrial image anomaly detection method based on pseudo abnormal sample generation in an exemplary embodiment of the present disclosure is shown;

[0037] Figure 2 A framework diagram of a SAD model in an exemplary embodiment of the present disclosure is shown;

[0038] Figure 3 A flowchart of a method for detecting anomalies in industrial images in an exemplary embodiment of the present disclosure is shown;

[0039] Figure 4 A schematic diagram showing the generation of abnormal pictures by an abnormal image generation module on an MVTec AD data set in an exemplary embodiment of the present disclosure is shown;

[0040] Figure 5 A schematic diagram of anomaly detection and positioning on the MVTec AD dataset in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0041] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0042] In addition, the accompanying drawings are only schematic illustrations of the embodiments of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0043] This example implementation first provides a method for detecting abnormal behavior in a video. Figure 1 As shown in , the method for detecting abnormal behavior in a video may include: steps S101 to S106.

[0044] Step S101: constructing a SAD model, wherein the SAD model includes an image feature extractor, an image feature adapter, an abnormal image generation module and a discriminator; wherein,

[0045] The image feature extractor is a WideResnet50 backbone network with frozen parameters after pre-training with the ImageNet dataset, the image feature adapter uses a single-layer fully connected layer for projection adaptation, the abnormal image generation module uses the DTD dataset image as the texture abnormality source, and the discriminator uses a two-layer multi-layer perceptron for discrimination;

[0046] Step S102: inputting the training image into the SAD model for training, and using the abnormal image generation module to generate a texture abnormal image corresponding to the training image;

[0047] Specifically, two-dimensional Perlin noise is generated, a peak value of the two-dimensional Perlin noise is extracted to obtain a preliminary noise image, an image foreground is extracted from the training image, and an element-by-element multiplication operation is performed on the preliminary noise image and the image foreground of the training image to obtain a positioning noise image;

[0048] Performing a mask operation on the localized noise image and a randomly selected image in the DTD texture data set to obtain an abnormal noise image;

[0049] A mask operation is performed on the abnormal noise image and the training image to obtain a texture abnormality image corresponding to the training image.

[0050] Step S103: inputting the training image and the corresponding texture abnormality image into the image feature extractor, and using the frozen image feature extractor to extract image features of the above two images;

[0051] Specifically, the patch-level feature size is determined according to a predetermined neighborhood size, and the extracted patch-level features are aggregated to obtain the overall image features.

[0052] Step S104: using the image feature adapter to perform projection adaptation on the two image features to obtain training image adaptation features and texture abnormality image adaptation features;

[0053] Specifically, the feature adaptation operation is performed on the image features of the training image and the texture abnormality image corresponding to the training image at the patch level.

[0054] Step S105: Adding Gaussian noise to the above training image adaptation features to construct noise abnormal image adaptation features;

[0055] Step S106: the training image adaptation features and the texture abnormality image adaptation features and the noise abnormality image adaptation features are respectively put into the discriminator for training to obtain the final detection result.

[0056] Specifically, the discriminator includes: two layers of multi-layer perceptrons; the loss function is L1 truncation loss.

[0057] The discriminator is used to train the training image adaptation features by comparing and learning the training image adaptation features with the texture abnormality image adaptation features and the noise abnormality image adaptation features on the discriminator, and the discriminator finally gives a discrimination result.

[0058] For the anomaly detection, after the discriminator gives an anomaly judgment, the anomaly score of the patch-level image features is calculated, and the heat map is drawn to show the anomaly location.

[0059] Through the above-mentioned method based on industrial image anomaly detection, the synthesized texture anomalies can make the synthesized anomalies and the anomalies generated at the feature level closer to the distribution of real anomalies, helping the discriminator to better identify anomalies in the real world. The discriminator trained by the contrastive learning method can better understand and learn anomalies from the comparison between normal and abnormal images, thereby better identifying anomalies during the detection process.

[0060] This application provides a new method for industrial image anomaly detection based on pseudo-abnormal sample generation, which directly extracts features from training images and corresponding generated abnormal images and then directly uses a discriminator to discriminate abnormalities. This application uses the WideResnet50 backbone network pre-trained with the ImageNet dataset for image feature extraction.

[0061] like Figure 2As shown, the present application makes full use of the help of synthetic anomalies for industrial image anomaly detection, and adopts two methods to generate abnormal images and corresponding abnormal features. One branch uses training images to generate corresponding texture feature anomalies, while the other branch makes full use of the advantage that generating anomalies at the feature level can be closer to the anomalies generated in the real world, and directly adds noise to the training image adaptation features to construct abnormal image features. With the respective advantages of the two anomaly generation methods, the present application effectively transfers the knowledge about anomalies to the task of industrial image anomaly detection, and realizes industrial image anomaly detection and anomaly location at the same time.

[0062] like Figure 3 The figure shows a flow chart of the method for industrial image anomaly detection of the present application.

[0063] like Figure 4 As shown, it is a schematic diagram of synthesizing abnormal images on the MVTec AD dataset.

[0064] like Figure 5 As shown in the figure, it is a schematic diagram of anomaly detection and localization on the MVTec AD dataset. The first, fourth and seventh columns are abnormal test images; the second, fifth and eighth columns are real abnormal area label images; the third, sixth and ninth columns are abnormal heat maps.

[0065] The effect of the present application is further described below in conjunction with simulation experiments.

[0066] 1. Simulation conditions

[0067] The simulation of this application is performed in the Linux operating system environment of the server, using an NVIDIA RTX3090 graphics card as computing resources, and the deep learning framework is PyTorch.

[0068] 2. Simulation content and result analysis

[0069] The present application and eight prior art methods (including reconstruction method, synthetic anomaly method and spatial embedding method) were used to conduct comparative simulation on the commonly used MVTec AD dataset.

[0070] The simulation experiment data used in this application is MVTec AD, which is the most commonly used industrial image anomaly detection dataset. It has 15 categories, including 3466 unlabeled images and 1888 labeled images (pixel-level segmentation and annotation), all of which are 700×700 or 1024×1024 in size; the training set has 3629 images, all of which are defect-free images; the test set has 1725 images, which are defective images and non-defective images; the images can be divided into 5 texture categories and 10 object categories, including 73 defects (scratches, dents, dirt, deformation, missing materials, etc., all of which are artificial), and a total of 1900 abnormal areas are annotated;

[0071] Table 1 Comparison results between the proposed method and existing methods on the MVTec AD dataset

[0072]

[0073] The present invention and the existing industrial image anomaly detection method are simulated under the same experimental settings and experimental environment, and the quantitative evaluation results of the MVTec AD data set are shown in Table 1. As can be seen from Table 1, the present invention method is superior to other methods.

[0074] Through the above-mentioned industrial image anomaly detection method, the discriminator is trained by combining the texture anomaly synthesis method and the feature anomaly synthesis method, which can effectively help the discriminator strengthen its understanding of anomalies and better identify anomalies.

[0075] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0076] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. An industrial image anomaly detection method based on pseudo-abnormal sample generation, characterized in that: include: Constructing a SAD model, the SAD model includes an image feature extractor, an image feature adapter, an abnormal image generation module and a discriminator; wherein, The image feature extractor is a WideResnet50 backbone network with frozen parameters after pre-training with the ImageNet dataset, the image feature adapter uses a single-layer fully connected layer for projection adaptation, the abnormal image generation module uses the DTD dataset image as the texture abnormality source, and the discriminator uses a two-layer multi-layer perceptron for discrimination; Inputting the training image into the SAD model for training; Using the abnormal image generation module to generate a texture abnormal image corresponding to the training image; Extracting image features of the training image and the texture abnormality image using the frozen image feature extractor; Using the image feature adapter to adapt the image features of the training image and the image with abnormal texture, to obtain the training image adaptation features and the image with abnormal texture adaptation features; A method of adding Gaussian noise to the training image adaptation feature is used to obtain a noise abnormality image adaptation feature; Using the discriminator, the training image adaptation features are compared with the texture abnormal image adaptation features and the noise abnormal image adaptation features on the discriminator to determine whether the training image is an abnormal image; The prediction result and abnormal heat map of the training image are obtained according to the discriminator's discriminant result.

2. According to claim 1, the method for detecting anomalies in industrial images based on pseudo-abnormal sample generation is characterized in that: The process of generating the texture abnormality image using the training image includes: Generate two-dimensional Perlin noise, extract peak values ​​from the two-dimensional Perlin noise to obtain a preliminary noise image, extract image foreground from the training image, perform a first operation on the preliminary noise image and the image foreground of the training image to obtain a positioning noise image; Perform a second operation on the located noise image and a randomly selected image in the DTD texture data set to obtain an abnormal noise image; A third operation is performed on the abnormal noise image and the training image to obtain a texture abnormality image corresponding to the training image.

3. According to claim 2, the method for detecting anomalies in industrial images based on pseudo-abnormal sample generation is characterized in that: The first operation is an element-by-element multiplication operation, and the second operation and the third operation are mask operations.

4. The method for detecting anomalies in industrial images based on pseudo-abnormal sample generation according to claim 1, characterized in that: The step of extracting the image features of the training image and the image with abnormal texture by using the frozen image feature extractor further includes: The patch-level feature size is determined according to a predetermined neighborhood size, and the extracted patch-level features are aggregated to obtain the overall image features.

5. The method for detecting anomalies in industrial images based on pseudo-abnormal sample generation according to claim 1, characterized in that: The process of using the image feature adapter to perform image feature adaptation on the image features of the training image and the image with abnormal texture corresponding to the training image comprises: The feature adaptation operation is performed on the image features of the training image and the texture abnormality image corresponding to the training image at a patch level.

6. The method for detecting anomalies in industrial images based on pseudo-abnormal sample generation according to claim 1, characterized in that: A noise abnormal image adaptation feature corresponding to the training image is generated by adding Gaussian noise to the training image adaptation feature.

7. The method for detecting anomalies in industrial images based on pseudo-abnormal sample generation according to claim 1, characterized in that: The discriminator is used to train the training image adaptation features by comparing and learning the training image adaptation features with the texture abnormality image adaptation features and the noise abnormality image adaptation features on the discriminator, and the discriminator finally gives a discrimination result.

8. The method for detecting anomalies in industrial images based on pseudo-abnormal sample generation according to claim 7, characterized in that: The discriminator comprises: Two-layer multilayer perceptron; the loss function is L1 truncation loss.

9. The method for detecting anomalies in industrial images based on pseudo-abnormal sample generation according to claim 1, characterized in that: The method further includes: The test image is input into the trained SAD model to obtain a prediction result of the test image.

10. The method for detecting anomalies in industrial images based on pseudo-abnormal sample generation according to claim 9, characterized in that: The step of inputting the test image into the trained SAD model further includes: For the anomaly detection, after the discriminator gives an anomaly judgment, the anomaly score of the patch-level image features is calculated, and the heat map is drawn to show the anomaly location.

Images