An unsupervised fast image anomaly detection and localization method

Abstract

The application discloses a kind of unsupervised fast image anomaly detection and positioning method, it is applied to image anomaly detection and positioning field, for the problem of low detection efficiency existing in prior art;The application obtains the features of known normal image using image feature extraction network and carries out L2 norm normalization;Then the prototype of normal image feature is obtained by non-parametric clustering to the normalized feature;Finally, an anomaly detection and positioning network is constructed by attaching a feature L2 norm normalization, a 1x1 convolution layer, a channel maximum pooling operation and a subtraction operation on the feature extraction network, and the prototype of normal image feature is used as the convolution kernel parameter of 1x1 convolution layer therein;In the anomaly detection process, the anomaly detection and positioning network is constructed by inputting the image to be detected, and fast, end-to-end image anomaly detection and positioning are realized.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to an image anomaly detection and positioning technology. Background Art

[0002] Anomaly detection aims to detect anomalous samples that deviate from a set of normal samples predefined during training. Image anomaly detection and localization not only performs image-level anomaly classification but also locates abnormal regions at the pixel level. In recent years, the detection and localization of subtle image anomalies has become an important task in computer vision, with various practical applications, such as defect detection in industrial optical inspection and tumor detection in medical images.

[0003] In the task of anomaly detection, abnormal samples are often scarce and difficult to collect. Therefore, in academia, anomaly detection is typically modeled using only normal samples, also known as unsupervised anomaly detection. Image anomaly detection and localization based on unsupervised algorithms is an even more challenging task. Currently, image anomaly detection and localization algorithms can be roughly divided into two categories: reconstruction-based methods and out-of-distribution (OOD) detection-based methods. Reconstruction-based methods assume that a model trained on normal images cannot accurately reconstruct the input abnormal image. These methods reconstruct the image as a whole or in a feature space. Anomaly detection and localization are then performed by detecting the difference between the reconstructed image (or image features) and the original image (or original image features). However, these methods require cumbersome network training. OOD detection-based methods detect and localize anomalies by calculating the deviation between the features of the image to be detected and the features of normal images. Currently, these methods generally achieve higher detection accuracy, but because the number of image features used for training is much larger than the number of images, the detection and localization time and storage requirements are significantly increased. At present, different methods have been proposed to solve this problem, such as clustering, density estimation and sampling of image features, but the efficiency of these methods still needs to be improved. Summary of the Invention

[0004] In order to further improve the efficiency of the OOD detection-based method, the present invention proposes a new unsupervised method for fast image anomaly detection and localization. A convolutional neural network is used to extract the features of normal images and perform L2 norm normalization on the features. The prototype of normal image features is learned through a clustering algorithm, and a prototype-based neural network is constructed for image anomaly detection and localization.

[0005] The technical solution adopted by the present invention is: an unsupervised method for rapid image anomaly detection and location, comprising:

[0006] A convolutional neural network is used to extract the feature map of a normal image and L2-normalize the feature vector at each spatial location in the feature map. The feature extraction network can be trained on normal images or a pre-trained model on a large natural image dataset.

[0007] Nonparametric clustering is performed on the normalized feature vectors of a normal image to obtain several clusters. The mean vector of each cluster is calculated as the prototype (vector) of the normal image features. Nonparametric clustering refers to clustering methods that do not require a specific number of cluster categories to be predefined. Nonparametric clustering algorithms often better reflect the data distribution than parametric clustering algorithms and avoid the need for artificially setting the number of cluster categories.

[0008] An image anomaly detection and localization network is constructed by adding a feature L2 norm normalization, a 1×1 convolutional layer, a channel-wise max pooling operation, and a subtraction operation to the feature extraction network. The prototypes (vectors) of normal image features are used as the convolution kernel parameters of the 1×1 convolutional layer. Therefore, the number of convolution kernels in this 1×1 convolutional layer is equal to the number of feature prototypes.

[0009] The image to be detected is fed into the constructed image anomaly detection and localization network to generate an anomaly score map for the image to be processed. Because the resolution of the feature map extracted by the feature extraction network is smaller than that of the input image, the resolution of the score map is also smaller than that of the input image. Therefore, the anomaly score map is scaled to the size of the input image and then smoothed using a Gaussian filter.

[0010] Finally, the maximum score in the anomaly score map is taken as the anomaly score of the image. If the anomaly score is greater than the anomaly detection threshold, the image to be detected is judged to be an abnormal image. For abnormal images, the score map is binarized using the anomaly localization threshold to achieve anomaly localization on the image.

[0011] The beneficial effects of the present invention are as follows: the present invention uses an image feature extraction network to obtain the depth features of a known normal image and performs L2 norm normalization; then, the prototype of the normal image features is obtained by performing non-parametric clustering on the normalized features; finally, an anomaly detection and localization network is constructed by adding a feature L2 norm normalization, a 1×1 convolution layer, a channel maximum pooling operation, and a subtraction operation to the feature extraction network. Since the feature prototype is used as the convolution kernel parameter of the 1×1 convolution layer, no additional training is required; in the anomaly detection process, by inputting the image to be detected into the constructed anomaly detection and localization network, fast, end-to-end image anomaly detection and localization are achieved; compared with the existing technology, the method of the present invention achieves higher detection efficiency while maintaining considerable accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the solution of the present invention; DETAILED DESCRIPTION

[0013] In order to facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings. Figure 1 As shown, the present invention uses a prototype-based anomaly detection and localization neural network to detect and localize image anomalies. First, a convolutional neural network is used to extract the features of normal images and L2 norm normalization is performed on the feature vectors. Then, a non-parametric clustering algorithm is used to learn the prototypes of normal image features. Finally, a feature L2 norm normalization, a 1×1 convolution layer, a maximum channel pooling layer, and a subtraction operation are added to the feature extraction network to perform anomaly detection and localization. The network can achieve end-to-end high-speed image anomaly detection and localization. At the same time, the network directly uses the image feature prototype as the convolution kernel parameter of the 1×1 convolution layer, so no additional network training is required.

[0014] The following is a specific implementation of the present invention, ProtoAD, and experiments are conducted on MVTec AD, an industrial defect detection dataset widely used in academia. This implementation includes the following steps:

[0015] 1. Image feature extraction

[0016] Using pretrained convolutional neural networks to extract image features has demonstrated its effectiveness in various computer vision applications, including image anomaly detection. This example uses a Wide ResNet pretrained on the ImageNet dataset as the feature extractor, but this is not limited to this network; other pretrained models, such as VGG, ResNet, and DenseNet, can also be used.

[0017] The ImageNet dataset used in this embodiment can be referenced as follows: Deng, J., Dong, W., Socher, R., Li, LJ, Li, K., Fei-Fei, L.: Imagenet: A largescale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, Ieee (2009) 248–255; the Wide ResNet used in this embodiment can be referenced as follows: Zagoruyko, S., Komodakis, N.: Wideresidual networks. In: British Machine Vision Conference 2016, British Machine Vision Association (2016).

[0018] ResNet and its variants (e.g., Wide ResNet) contain multiple convolution stages. The deeper the stage, the more abstract the features extracted, and the lower the resolution of the feature map. These feature maps of different resolutions form a feature hierarchy of the input image. The features of each spatial position in the feature map correspond to a receptive field, and the features of the lower layers of the network correspond to a small local area or image patch in the input image. Therefore, the feature vector of each spatial position in the feature map can be regarded as a local feature or patch feature of the image. The deep features of the ImageNet pre-trained network are biased towards classification tasks, have a low correlation with anomaly detection and localization tasks, and the large receptive field is not conducive to localization. Therefore, this embodiment uses features from the low layer (first convolution stage) and the middle layer (second and third convolution stages) and combines them together. Specifically, the feature maps of the middle layer are enlarged to the same resolution as the lower layer, and then they are spliced ​​together as the final image feature map, and the features therein are normalized by the L2 norm.

[0019] For L2-norm normalized vectors, the Euclidean distance and cosine similarity between them are equivalent when comparing neighbor relationships. This is an important property.

[0020] 2. Prototype Learning

[0021] After obtaining the normalized features of the normal image, the present invention obtains the prototype of the features through clustering learning, thereby reducing the time and storage required for anomaly detection. Currently, this type of method generally uses kmeans for clustering. However, the kmeans algorithm requires the number of clusters K to be set in advance, and kmeans is sensitive to initialization. For image data of different categories, how to choose a suitable K is a relatively difficult problem. In order to efficiently perform the anomaly detection task, the present invention selects a non-parametric clustering method, that is, there is no need to pre-set the number of clusters K, and the algorithm determines the appropriate K value based on the data. Specifically, the present invention adopts the FINCH clustering algorithm (Sarfraz, S., Sharma, V., Stiefelhagen, R.: Efficient parameter-free clustering using first neighbor relations. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. (2019) 8934–8943). The FINCH clustering algorithm is a new hierarchical aggregation clustering method proposed in recent years, which is efficient, scalable and parameter-free. The core idea of ​​this algorithm is to use the nearest neighbor information of each data point for clustering. It does not require any parameters to be specified and has low computational overhead. Given the nearest neighbor index of each data point, the adjacency matrix is ​​defined according to the following rules:

[0022]

[0023] in, Represents the nearest neighbors of data point i. This sparse adjacency matrix defines a graph in which connected data points form a cluster. After completing the first round of data partitioning, FINCH uses the center of each cluster to calculate the nearest neighbors, and then recursively aggregates the clusters. In the present invention, a rule for stopping FINCH clustering is given: when the number of clusters obtained in one round of aggregation is less than a certain threshold, the aggregation process stops, and the last round of aggregation results is used as the final clustering result. Then, the average vector of all clusters is used as the prototype of the feature. In this embodiment, the threshold is set to 10,000, and good performance is obtained in the experiment.

[0024] Non-parametric clustering methods that can be selected in practical applications also include the SCC clustering algorithm and the HDBSCAN clustering algorithm. Among them, the SCC clustering algorithm (Sub-Cluster Component Algorithm) is a hierarchical agglomerative clustering algorithm that can handle large-scale data. For details, please refer to: Nicholas Monath, Kumar Avinava Dubey, GuruGuruganesh, Manzil Zaheer, Amr Ahmed, Andrew McCallum, Gokhan Mergen, Marc Najork, Mert Terzihan, Bryon Tjanaka, et al. Scalable Hierarchical Agglomerative Clustering. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1245-1255 (2021). The HDBSCAN clustering algorithm combines the classic density clustering algorithm DBSCAN with the hierarchical clustering algorithm. For details, please refer to: RJGB Campello, D. Moulavi, and J. Sander. 2013. Density-Based Clustering Based on Hierarchical DensityEstimates. Advances in Knowledge Discovery and Data Mining (2013).

[0025] The SCC clustering algorithm, like the FINCH algorithm, is a hierarchical clustering algorithm and can use the same stopping conditions and thresholds as above. The HDBSCAN clustering algorithm can automatically recommend the best results.

[0026] 3. Neural Networks for Anomaly Detection and Localization

[0027] The present invention proposes an end-to-end neural network that does not require additional training to perform anomaly detection and location.

[0028] The network will first calculate the anomaly score of each feature in the feature map. The test image can obtain a H×W×D feature map through the feature extraction network, that is, H×W features, where H and W correspond to the height and width of the feature map, that is, the resolution, and D represents the dimension of the feature. The anomaly score of each normalized feature is obtained by calculating the deviation between it and the normal image feature prototype. The present invention uses 1 minus the maximum value of the cosine similarity between the normalized feature and each prototype as the anomaly score of the feature. The anomaly score of the feature at the feature map position (i, j) can be calculated by the following formula:

[0029]

[0030] Among them, x ij is the normalized feature of position (i, j), m k is the k-th prototype, and cos represents the cosine similarity.

[0031] Because the features and prototypes of the test image are both L2-norm normalized, the cosine similarity between them can be obtained through 1×1 convolution. Based on this equivalence, the present invention constructs a neural network for anomaly detection and positioning. First, a feature L2-norm normalization and a 1×1 convolution layer are added to the feature extraction network, and the prototype (vector) of the normal image feature is used as the convolution kernel parameter of the 1×1 convolution layer. The number of convolution kernels is also the number of feature prototypes K. When non-parametric clustering is used, this number may vary greatly for images of different categories. This 1×1 convolution layer outputs a feature map of size H×W×K, which represents the cosine similarity between H×W normalized features and K prototypes. Then, a channel maximum pooling operation is performed to obtain an H×W normal score map, which represents the cosine similarity between H×W normalized features and their nearest prototypes. The abnormal score map can be further obtained by subtracting the normal score map from 1. Since the spatial resolution of the feature map is lower than the spatial resolution of the input image, the present invention enlarges the abnormal score map to the resolution of the input image and uses a Gaussian filter for smoothing.

[0032] 4. Image anomaly scoring and anomaly location

[0033] For abnormal image detection, the maximum score in the anomaly score map is taken as the image's anomaly score, and then the anomaly detection threshold is used to determine whether the image is abnormal. For image anomaly localization, the anomaly score map can be binarized using the anomaly localization threshold. To select an anomaly detection threshold or an anomaly localization threshold, several values ​​within the range [0, 1] can be sampled as candidate thresholds. The F1-score of anomaly detection or anomaly localization on the test set or validation set is calculated for each candidate threshold. The candidate threshold with the maximum F1-score is then selected as the final anomaly detection threshold or anomaly localization threshold.

[0034] The present invention uses the prototype of normal image features as the convolution kernel parameters of the 1×1 convolution layer. Therefore, the anomaly detection and localization network proposed in the present invention does not require an additional training stage because the feature extraction network used here is a pre-trained network.

[0035] The following is a comparative explanation of the technical effects of the present invention in combination with the prior art:

[0036] AUROC (Area Under the Receiver Operating Characteristic Curve) is the most commonly used metric for anomaly detection and is independent of threshold selection. ROC, or receiver operating characteristic, is a curve plotted against a series of different binary classification thresholds, with the true positive rate as the ordinate and the false positive rate as the abscissa. AUROC, defined as the area under the ROC curve, is a performance metric for detection at various threshold settings, with higher numbers being better. This paper uses image-level AUROC to evaluate anomaly detection performance and pixel-level AUROC to evaluate anomaly localization. Because pixel-level AUROC favors large anomalies, this paper also uses PRO-score (Per-Region-Overlap) to evaluate anomaly localization, which equally weights true anomaly regions of different sizes. The proposed method is compared with state-of-the-art anomaly detection methods. Reconstruction-based methods include Uninformed Students (U-Student), RIAD, MKD, Glance, DAAD, and DREAM, while OOD-based methods include SPADE, PatchSVDD (P-SVDD), CutPaste, PaDiM, and PatchCore (P-Core).

[0037] Table 1 Comparison of algorithm anomaly detection performance in MVTec AD dataset (pixel-level AUROC)

[0038]

[0039] Table 2 Comparison of algorithm anomaly detection performance in MVTec AD dataset (PRO-score)

[0040]

[0041] Table 1 shows the pixel-level AUROC of each algorithm on the MVTec AD dataset. As can be seen from Table 1, methods based on OOD detection usually achieve higher pixel-level AUROC than methods based on reconstruction. Among the methods based on OOD detection, methods using pre-trained deep features perform better than methods based on self-supervised learning. Among them, PatchCore achieved the best pixel-level AUROC, PaDiM ranked second, and the reconstruction-based method DREAM ranked third. The pixel-level AUROC of the method ProtoAD proposed in the present invention is very close to PaDiM and DREAM. In addition, ProtoAD has a higher AUROC in the texture category. Table 2 shows the results of PRO-score. Among them, the Glance algorithm achieved the best results, and the method ProtoAD proposed in the present invention ranked second, which is better than other methods based on OOD detection. In summary, the anomaly localization performance of ProtoAD of the present invention is competitive compared with the current best anomaly detection algorithms.

[0042] Table 3 Comparison of algorithm anomaly detection performance in MVTec AD dataset (image-level AUROC)

[0043]

[0044] We also compare image-level AUROC results for anomaly detection in Table 3. PatchCore again achieves the best AUROC, followed by DREAM. Our method remains competitive and achieves the third-best AUROC, very close to DREAM.

[0045] Table 4 Comparison of inference speed of different algorithms

[0046]

[0047] Table 4 shows the anomaly detection and localization speed of the method of the present invention and the current OOD-based method using pre-trained deep features, where the scores include image-level AUROC, pixel-level AUROC and PRO-score. In the experiments, all methods used WideResNet50 pre-trained on ImageNet as the feature extractor, and cropped the center of the image to a size of 224×224 as input, and conducted experiments on the same machine equipped with NVIDIA GeForce RTX 2080Ti. For the PatchCore method, PatchCore-x% represents the percentage of normal features used in the algorithm. For the PaDiM method, we optimized its implementation through GPU acceleration operations. Compared with previous methods, ProtoAD of the present invention achieved the fastest speed (1.2, 2.7, and 9.5 times that of PaDiM, PatchCore, and SPADE, respectively).

[0048] In this embodiment, specific reference to Uninformed students (U-Student): Bergmann, P., Fauser, M., Sattlegger, D., Steger, C.: Uninformed students: Student-teacher anomalydetection with discriminative latent embeddings. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. (2020) 4183–4192.

[0049] RIAD specific reference: Zavrtanik, V., Kristan, M., Skoˇcaj, D.: Reconstruction by inpainting for visual anomaly detection. Pattern Recognition 112 (2021) 107706.

[0050] Specific reference for MKD: Salehi, M., Sadjadi, N., Baselizadeh, S., Rohban, M. H., Rabiee, H. R.: Multiresolution knowledge distillation for anomaly detection. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. (2021) 14902–14912.

[0051] Specific reference for Glance: Wang, S., Wu, L., Cui, L., Shen, Y.: Glancing at the patch: Anomaly localization with global and local feature comparison. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. (2021) 254–263.

[0052] Specific reference for DAAD: Hou, J., Zhang, Y., Zhong, Q., Xie, D., Pu, S., Zhou, H.: Divide-and-assemble: Learning block-wise memory for unsupervised anomaly detection. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. (2021) 8791–8800.

[0053] Specific reference for DREAM: Zavrtanik, V., Kristan, M., Skoˇcaj, D.: Draem - a discriminatively trained reconstruction embedding for surface anomaly detection. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. (2021) 8330–8339.

[0054] For SPADE, see specifically: Cohen, N., Hoshen, Y.: Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint arXiv:2005.02357 (2020).

[0055] For PatchSVDD (P-SVDD), see specifically: Yi, J., Yoon, S.: Patch svdd: Patch-level svdd for anomaly detection and segmentation. In: Proceedings of the Asian Conference on Computer Vision. (2020).

[0056] For CutPaste, see specifically: Li, C.L., Sohn, K., Yoon, J., Pfister, T.: Cutpaste: Self-supervised learning for anomaly detection and localization. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. (2021) 9664–9674.

[0057] For PaDiM, see specifically: Defard, T., Setkov, A., Loesch, A., Audigier, R.: Padim: a patch distribution modeling framework for anomaly detection and localization. In: International Conference on Pattern Recognition, Springer (2021) 475–489.

[0058] PatchCore (P-Core) specific reference: Roth, K., Pemula, L., Zepeda, J., Scho. . lkopf,B.,Brox,T.,Gehler,P.:Towards total recall in industrial anomaly detection.In:Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.(2022)14318–14328.

[0059] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A method for unsupervised rapid image anomaly detection and localization, characterized in that: include: The feature extraction network is used to extract the feature map of the normal image, and the feature vector at each position of the feature map is normalized by L2 norm to obtain the normalized feature vector of the normal image; Perform non-parametric clustering on the normalized feature vector of the normal image to obtain several clusters, and calculate the average vector of each cluster as the prototype of the normal image feature; An image anomaly detection and localization network is constructed by adding a feature L2 norm normalization, a 1×1 convolution layer, a channel maximum pooling operation, and a subtraction operation to the feature extraction network. The prototype of the normal image feature is used as the convolution kernel parameter of the 1×1 convolution layer. The number of normal image feature prototypes is the same as the number of convolution kernels in the 1×1 convolution layer. Input the image to be detected into the constructed image anomaly detection and localization network to obtain the anomaly score map of the image to be detected; The abnormality score map is used to determine whether the image to be detected is an abnormal image. If so, the abnormality is located.

2. The unsupervised method for rapid image anomaly detection and localization according to claim 1, characterized in that: The non-parametric clustering specifically adopts the FINCH clustering algorithm.

3. The unsupervised method for rapid image anomaly detection and location according to claim 2, wherein: The condition for stopping FINCH clustering is: when the number of clusters obtained in one round of aggregation is less than the set threshold, the aggregation process stops and the last round of clustering results is taken as the final clustering result.

4. The unsupervised method for rapid image anomaly detection and location according to claim 1, wherein: The non-parametric clustering specifically adopts the SCC clustering algorithm.

5. The method for unsupervised rapid image anomaly detection and location according to claim 4, characterized in that: The condition for stopping SCC clustering is: when the number of clusters obtained in one round of aggregation is less than the set threshold, the aggregation process stops and the last round of clustering results is taken as the final clustering result.

6. The unsupervised method for rapid image anomaly detection and location according to claim 1, characterized in that: The non-parametric clustering specifically adopts the HDBSCAN clustering algorithm.

7. The method for unsupervised rapid image anomaly detection and location according to claim 1, wherein: The anomaly score map is calculated by subtracting the cosine similarity between the normalized feature vector at each spatial position on the feature map and the prototype closest to it from 1. The calculation formula for the anomaly score of the normalized feature vector at a spatial position on the feature map is: Among them, x ij is the normalized feature vector of the feature map spatial position (i, j), m k is the k-th prototype, and cos represents the cosine similarity.

8. The unsupervised rapid image anomaly detection and location method according to claim 7, characterized in that: The specific implementation process of anomaly location based on the anomaly score graph is as follows: The maximum score in the anomaly score map is used as the anomaly score of the image to be detected. If the anomaly score of the image to be detected is greater than the anomaly detection threshold, the image to be detected is judged to be an abnormal image; For abnormal images, the score map is binarized using the abnormality localization threshold to achieve abnormality localization on the image to be detected.

Images