A method for detecting and locating prototype residue anomalies

Abstract

The present invention belongs to the field of computer vision anomaly detection and localization, and is specifically a prototype residual anomaly detection and localization method. The present invention reconstructs a segmentation map of the abnormal area by learning feature residuals of different scales and sizes between the abnormal and normal modes. The method mainly consists of two parts: one part is a multi-scale prototype used to represent the residual features of the abnormal to the normal mode; the other part is a multi-scale self-attention mechanism that supports variable-scale abnormal feature learning. In addition, we propose a variety of anomaly generation strategies, which achieve the expansion and diversification of abnormal samples by considering the appearance differences between visible anomalies and invisible anomalies. The verification results of four commonly used data sets show that compared with the mainstream supervised and unsupervised methods in the industry, the anomaly detection of the present invention is more accurate, and the performance of anomaly localization has a greater breakthrough, which confirms the effectiveness and generalizability of the present invention.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a prototype residual abnormality detection and positioning method. Background Art

[0002] Human cognition and vision systems have an inherent ability to perceive anomalies. Humans can not only distinguish defective images from non-defective images, but can also pinpoint the location of anomalies even if they have never seen anomalies or have only seen a limited number of them. Anomaly detection (binary classification of abnormality or normality at the image level) and anomaly localization (binary classification of abnormality or normality at the pixel level) were introduced for the same purpose and have been widely used in various scenarios due to their high efficiency and accuracy, including industrial defect detection [1][9]

[10]

[11]

[14] , medical image analysis

[13] , and video surveillance

[12] . Given its importance, there are many works devoted to designing a method that can do both anomaly detection and localization well, but few methods have been developed to do both tasks well. We believe that these models perform poorly due to the three main characteristics of real-world anomaly data: I) the number of anomaly samples is limited and significantly less than normal samples, resulting in an unbalanced learning problem; II) anomalies are usually subtle and difficult to identify, so the normal pattern still dominates the abnormal image, and identifying all abnormal regions from the entire image is the key to anomaly detection and localization; III) the appearance of anomalies varies greatly, that is, the size, shape and number of abnormal regions are the same, and this appearance variation makes it challenging to accurately locate all abnormal regions.

[0003] Without enough anomalies for training, unsupervised models become the de facto dominant approach, eliminating the imbalance problem by learning the distribution of normal samples [2][3][5][6][7] or generating sufficient synthetic anomalies [8]

[17] . However, these methods are opaque to the true anomalies, and this implicit decision may lead to many false negatives and false positives. At the same time, unsupervised methods rely heavily on the quality of normal samples, so they are not robust enough and perform poorly on uncalibrated or noisy datasets. In addition, such unsupervised models can only accurately locate the abnormal area and can only give a rough range of the anomaly. We attribute this problem to the weak ability of these methods to distinguish between normal and abnormal.

[0004] Recently, some supervised methods have explored different perspectives. DeepSAD

[15] enlarges the difference between the centers of the anomaly and normal classes in the latent space, and obtains a more compact description of the normal class by analyzing the observed anomalies. DRA[4] and DevNet

[16] define anomaly detection as a multiple instance learning (MIL) problem, where an image is classified as an anomaly if any patch of the image is a defective area. MIL-based methods enhance model learning through fine-grained image patch-level methods, effectively reducing the interference of normal patches on detecting abnormal images. However, these methods usually have difficulty accurately locating all abnormal regions under such image-level supervision. In particular, when the abnormal region occupies only a small part of the image patch, the image-level representation may be dominated by the normal region and ignore minor anomalies, which may lead to inconsistent detection and localization performance.

[0005] In summary, in the actual application of anomaly detection and localization, the number of abnormal samples is often far less than that of normal samples. At the same time, abnormal samples are usually difficult to identify and vary greatly. The use of supervised models is prone to overfitting a small number of visible abnormal samples, resulting in unsatisfactory performance. The implicit decisions of unsupervised models are not discriminative enough and are highly dependent on the calibration and quality of the data set. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to propose a prototype residual anomaly detection and positioning method for reconstructing abnormal areas in an image based on a given input image.

[0007] The present invention proposes a variety of anomaly generation strategies, which can not only effectively alleviate the impact of data imbalance, but also explore potential unknown abnormal samples. In addition, the present invention proposes a multi-scale prototype to represent the normal mode. Compared with the previous method of constructing the normal mode based on the feature connection of normal samples or random sampling of feature maps, the present invention uses the prototype obtained by clustering intermediate feature maps of different scales to construct the normal mode, which retains the spatial information and provides a more accurate and representative normal mode. The present invention obtains the residual by the distance between the input image feature map and the closest prototype at each scale, and adds a multi-scale fusion block to realize information exchange between different scales. Secondly, since the abnormal areas vary greatly, it is necessary to capture the relationship between patches from multiple receptive fields. Therefore, we introduce a multi-scale self-attention mechanism, which acts on different receptive fields of feature maps to detect inconsistencies at the patch level of different sizes. Finally, unlike the previous method of training using image-level supervision, the present invention reconstructs the abnormal segmentation map through pixel-level supervision, so that the model pays more attention to the abnormal area. The technical solution of the present invention is specifically described as follows.

[0008] The present invention provides a prototype residual anomaly detection and positioning method, which realizes anomaly detection and positioning of image samples by learning feature residuals of different scales and sizes between abnormal and normal modes, reconstructing a predicted anomaly annotation map based on an anomaly reconstruction network of prototype residuals; wherein:

[0009] The anomaly reconstruction network based on prototype residual adopts a network similar to U-Net, which includes an image feature encoding module, a multi-scale prototype module, two multi-scale fusion modules, a multi-scale self-attention module and an anomaly decoder module; the image feature encoding module obtains the multi-scale feature map of the input image, and the multi-scale prototype module is used to represent the residual features of the anomaly to the normal mode. The prototype of each scale is the feature representation of the normal mode inferred from all feature maps of the scale, which retains the complete spatial information and represents the normal mode; the residual is the pixel-level Euclidean distance between the feature map of each scale of the input image and the prototype closest to the corresponding scale; the first multi-scale fusion module exchanges information of different scales on the feature map and the residual of the input image and then splices them. The spliced ​​image features capture the inconsistency under different receptive fields through the multi-scale self-attention module, and then exchange information through the second multi-scale fusion module; the output of the second multi-scale fusion module is used as a jump connection, and the anomaly decoder module reconstructs the predicted anomaly annotation map based on the output features of the last layer of the image feature encoding module and the jump connection.

[0010] The method is divided into three stages, as follows:

[0011] Phase 1: Building a multi-scale prototype

[0012] Based on all normal samples in the training set, a multi-scale feature map of the input normal image is extracted, and the prototype of each scale is represented by the features of the normal pattern inferred from all feature maps of the scale to construct a multi-scale prototype;

[0013] The second stage: the anomaly synthesis strategy generates various types of anomalies, and trains the anomaly reconstruction network based on the prototype residual through the consistency loss of the real anomaly annotation and the predicted anomaly annotation;

[0014] In the target area of ​​the normal sample, new and rich abnormal samples are generated by expanding the abnormality and simulating the abnormality. The real abnormality annotation is generated by the abnormality generation process; among them: the extended abnormality is to randomly enhance the color and spatial position of the abnormal area of ​​the seen abnormal sample, and then place it in the target area of ​​the normal sample to generate a new abnormal sample; the simulated abnormality is to multiply the random abnormal area generated by Perlin noise by the image of other data sources or the self-enhancement of the image itself, and then place it in the target area of ​​the normal sample to generate a new abnormal sample; the abnormal reconstruction network based on the prototype residual reconstructs the predicted abnormal annotation map through the abnormal decoder module;

[0015] The prototype residual-based anomaly reconstruction network uses pixel-level supervision to reconstruct anomaly annotations. The network's loss function is the weighted sum of the smooth L1 loss and focal loss of the true anomaly annotation and the predicted anomaly annotation:

[0016] L total =Smooth L1 (M o ,M)+λL focal (M o ,M)

[0017] λ is the weight coefficient of focal loss.

[0018] Phase 3: Anomaly Detection and Location

[0019] The model trained in the second stage is used as the inference model, and the normal or abnormal images to be tested are input into the network, and the abnormal annotation images are directly regressed and predicted in an end-to-end manner.

[0020] In the present invention, in the first stage, the image feature encoding module adopts a pre-trained ResNet model.

[0021] In the present invention, in the first stage, the multi-scale prototype constructed based on the normal sample has three scales, and each scale corresponds to the output of the first layer, the second layer and the third layer of the image feature encoding module.

[0022] In the present invention, in the first stage, the number of prototypes of each scale is K, and the ratio of K to the total number of normal samples is a fixed value. The number of prototypes of different data sets changes with the number of normal samples of the data sets.

[0023] In the present invention, the initial state of the prototype of each scale is the feature map randomly sampled from all normal samples of the scale, and then the cluster center is updated by the k-means clustering algorithm. The cluster center is called the prototype. In the k-means update, the distance between two feature maps is measured by the Euclidean distance, and the termination state is determined by the number of iterations.

[0024] In the present invention, in the second stage, the number of abnormal samples in the training set is much less than the number of normal samples, and the distribution of the data set is unbalanced.

[0025] In the present invention, in the second stage, for object-type datasets, the target area is a partial area of ​​the object foreground, and for texture-type datasets, the target area is a partial area of ​​the entire image.

[0026] In the present invention, in the second stage, the enhancement used to generate the extended anomaly and the simulated anomaly includes both color and spatial enhancement. The other data set used to generate the simulated anomaly is a describable texture data set (DTD). The self-enhancement of the image itself in the simulated anomaly also includes random shuffling of the image at the patch level, specifically, dividing an image into n×n grids and randomly shuffling them and then reorganizing them into a new disordered image.

[0027] In the present invention, in the second stage, the input of the multi-scale fusion module is multi-scale features or residuals, and the output is multi-scale fused features or residuals, and the fusion calculation method performed on the output of each scale is different. Preferably, there are three scales.

[0028] In the present invention, in the second stage, each head of the multi-size self-attention module corresponds to a division of patch size, and the information of multiple receptive fields is combined through convolution as the output of the jump connection.

[0029] In the present invention, the anomaly decoder module predicts the annotation map of the anomaly area through upsampling and convolution.

[0030] In the present invention, in the third stage, the average value of the N pixels with the highest scores in the predicted anomaly annotation is used as the anomaly score of the input image.

[0031] The inventors used the image-level receiver operating characteristic curve (I) predicted by the model, the pixel-level receiver operating characteristic curve (P), the regional overlap rate at different thresholds at the pixel level (O), and the average accuracy at the pixel level (A) to comprehensively measure the evaluation indicators of anomaly detection and positioning, and compared the present invention with the previous unsupervised method and supervised method, verifying that the method of the present invention is better than the two methods mentioned above in all evaluation indicators. In addition, the present invention also visualizes the detection and positioning of anomalies, and conducts visual qualitative evaluation with other methods. The method of the present invention is obviously closer to the true labeling of anomalies. In summary, the beneficial effects of the present invention are:

[0032] 1. The present invention uses a variety of anomaly synthesis strategies to solve the problem of insufficient anomaly samples in anomaly detection and location. This strategy starts from the two perspectives of expanding anomalies and simulating anomalies. While further diversifying the anomalies that have been seen, it also effectively explores the potential manifestations of unseen anomalies.

[0033] 2. The present invention aims to solve the problem that supervised learning at the image level will make the neural network have poor effect on subtle anomalies. It introduces residual prototype learning based on pixel level, and uses multi-scale residual information to make the network pay more attention to abnormal areas. The multi-scale residual is obtained by the distance between the feature map of the input image and the prototype representing the normal mode closest to it. After the multi-scale residual interacts with the multi-scale fusion module, it can provide the neural network with the area where the anomaly is located.

[0034] 3. Aiming at the diversity of the size, shape, and number of abnormal regions, the present invention introduces a multi-scale self-attention mechanism. This mechanism uses receptive fields of different sizes to detect patch-level inconsistencies of different sizes, thereby more accurately locating specific regions of various abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the extended anomaly in the synthesis strategy proposed by the present invention. The extended anomaly generates a new anomaly sample by randomly enhancing the color and spatial position of the abnormal area of ​​the seen abnormal sample and placing it in the target area of ​​the normal sample.

[0036] Figure 2 It is a schematic diagram of the prototype residual anomaly detection and positioning method proposed in the present invention. The model is a network similar to U-Net, and the jump connection is enhanced by multi-scale prototypes and multi-scale self-attention mechanisms. The input image obtains the feature residual of each layer through the multi-scale prototype for the feature map of each scale. The multi-scale fusion module is used to fuse the feature maps and residuals of different scales respectively. The multi-scale self-attention mechanism captures the inconsistent information of different spatial scales through the receptive fields of different sizes in the feature residuals of each scale, and then further enhances it as the output of the jump connection through another multi-scale fusion block. The anomaly decoder module includes upsampling and convolution, and uses the output of the jump connection and the last layer of the encoder to predict the annotation map of the abnormal area.

[0037] Figure 3 It is a multi-scale feature fusion module introduced in the present invention, and the fused feature map is the sum of the transformed representations of the three input feature maps.

[0038] Figure 4 This is a visualization result diagram of the present invention on the MVTec data set. For different types of anomalies reflected in different data sets, the anomaly location result of the method proposed by the present invention is more accurate than that of other methods. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0040] In order to alleviate the imbalance caused by the fact that abnormal samples are far less than normal samples, the present invention first proposes to use the seen anomalies to generate extended anomalies ( Figure 1 The present invention first enhances the color diversity of the abnormalities by randomly enhancing the colors of the abnormalities. Figure 1 , A), and then use random spatial enhancement to improve the diversity of abnormal positions, shapes and angles ( Figure 1 , R). In order to make the generated abnormal samples more reasonable, the present invention proposes a target area, and places the enhanced abnormal part in the normal image ( Figure 1 , N) in the specified area, and obtain the cropped abnormal part ( Figure 1 , C). The present invention binarizes C to generate an abnormal annotation ( Figure 1 , M). Extended exception ( Figure 1 , E) is generated as follows:

[0041]

[0042] in, is the pixel-by-pixel inversion of M, ⊙ is the pixel-by-pixel multiplication operation, and β is the transparency parameter used to better combine abnormal and normal parts. For object datasets, the target area is part of the foreground of the object. For texture datasets, the target area is part of the entire image.

[0043] In order to further explore the potential appearance of unseen anomalies, the present invention proposes a simulated anomaly, where a random anomaly area generated by Perlin noise is multiplied by an image from other data sources or a self-enhancement of a normal image and then placed in a target area of ​​a normal sample to generate a new anomaly sample.

[0044] The abnormal reconstruction network based on prototype residual proposed in this invention ( Figure 2 ), is based on deep learning technology. The model is a U-Net-like network, and the jump connection is enhanced by multi-scale prototypes and multi-scale self-attention mechanisms. The residual between the feature map of each scale of the input image and the prototype of the corresponding scale is calculated. The feature map and the residual will be exchanged at multiple scales through two multi-scale fusion modules. The information after interaction will be captured by the multi-scale self-attention mechanism for anomalies at multiple spatial perception angles. The decoder generates predicted anomaly annotations based on this enhanced feature and residual information. It mainly includes the following five modules: (1) image feature encoding module; (2) multi-scale prototype module; (3) multi-scale fusion module; (4) multi-scale self-attention module; (5) anomaly decoder module.

[0045] Given a single RGB image, first, the initial image feature encoding module extracts features at three scales of the image; then, the closest prototype is found among the image features at each scale and the prototype at the corresponding scale, and their distance residual is calculated. The multi-scale features and residuals are spliced ​​together after exchanging information at different scales through a multi-scale fusion module; then, the spliced ​​features are captured by a multi-scale self-attention module to capture the inconsistency under different receptive fields, and then exchange information through another multi-scale fusion module. Finally, the output of the multi-scale module is used as a skip connection, and the anomaly decoder reconstructs the anomaly annotation map based on the encoder features and skip connections through upsampling and convolution. The loss of the predicted anomaly annotation map and the true anomaly annotation map is obtained by the weighted sum of the focal loss and the smooth L1 loss. Such a loss allows the model to focus more on the performance of the abnormal area.

[0046] The second stage of the present invention mainly learns multi-scale feature maps and residuals for reconstructing abnormal prediction annotations. The specific steps are as follows:

[0047] (1) First, the input of the model is a normal sample or an abnormal sample generated online. Abnormal samples include extended anomalies and simulated anomalies. Normal samples are grayscale images with an abnormal label value of 0, and the abnormal label of abnormal samples is generated by the abnormal generation process ( Figure 1 ).

[0048] (2) The input image is passed through the image feature encoding module to extract features at three scales. The parameters of the first three layers of the image feature encoding module are fixed. The Euclidean distance between the feature at each scale and the prototype at that scale is calculated. The one with the smallest distance is called the nearest prototype. The pixel-level Euclidean distance between the feature map and the nearest prototype is the residual between the feature map and the normal mode. (3) The multi-scale feature maps and residuals are spliced ​​together after exchanging information of different scales through a multi-scale fusion module. Figure 3 For the feature maps of the three scales, when the H and W of the target feature map are larger than the input feature map, the operation performed on the input feature map is linear interpolation upsampling and 1×1 convolution to transform it to the same scale as the target feature map. Figure 1 When the target feature map is the same size as the input feature map, no operation is performed; when the H and W of the target feature map are smaller than the input feature map, the operation performed on the input feature map is a deep separation convolution operation, which transforms it to the same size as the target feature map. Figure 1 The fused feature map is the sum of the outputs of the three operations.

[0049] (4) In the design of the multi-scale self-attention mechanism, the patch size of each head is different. After each head performs self-attention calculation independently, they are spliced ​​together and then fused with information from multiple receptive fields through a residual convolution. The multi-scale self-attention mechanism at each scale has multiple layers, and the output of each scale will be exchanged through a multi-scale fusion module to exchange information at different scales.

[0050] (5) The loss of the training network is the weighted sum of the focal loss and the smooth L1 loss. The focal loss allows the network to pay more attention to abnormal areas, and the smooth L1 loss can reduce the network's abnormal value assignment to normal areas, thereby making the normal and abnormal scores more distinguishable at the pixel level.

[0051] In the testing phase of the present invention, a test image is input into the trained network, and the abnormal area of ​​the image can be generated end-to-end.

[0052] The technical solution of the present invention is further described below in conjunction with embodiments.

[0053] Example 1

[0054] The model accepts 256*256 fixed-size images as input to the visual branch. Images of various sizes are normalized to the above sizes by scaling.

[0055] Figure 1 This is a flowchart of the invention for generating extended anomalies and some generated result diagrams. Extended anomalies take into account the diversity of visible anomaly appearances and can generate various types of abnormal samples that are close to the abnormal samples in the training set. The abnormal samples in the training set are far less than the normal samples. In the invention, each type of data set has only ten abnormal samples. Figure 1 Two different data enhancement methods are mentioned, among which color enhancement includes random exposure, tone separation, random sharpening, automatic contrast, random color inversion, gamma contrast, and equal operations without change; spatial enhancement includes translation, rotation, shearing and other methods.

[0056] In the design of the multi-scale prototype module, the image feature encoding module is ResNet-18, and the shapes of the feature maps of the three scales are 64×64×64, 128×32×32 and 256×16×16 respectively. The number of prototypes accounts for 10% of the number of all normal samples. The initial state of the prototype of each scale is a randomly sampled feature map, and then the k-means clustering method is used to update the cluster center, and the distance metric between the two feature maps is the Euclidean distance. The cluster centers of these scales are collectively referred to as multi-scale prototypes in the present invention. For the input image, the feature maps of the three scales are also first extracted through the image feature encoding module. The feature map of each scale finds the closest prototype in the prototype of the corresponding scale, and calculates their pixel-by-pixel Euclidean distance as the residual.

[0057] In the design of the multi-scale feature fusion module, Figure 3 As shown in the figure, for the feature maps of three scales, when the H and W of the target feature map are larger than the input feature map, the operation performed on the input feature map is linear interpolation upsampling and 1×1 convolution to transform it to the same scale as the target feature map. Figure 1 When the target feature map is the same size as the input feature map, no operation is performed; when the H and W of the target feature map are smaller than the input feature map, the operation performed on the input feature map is a deep separation convolution operation, which transforms it to the same size as the target feature map. Figure 1 The fused feature map is the sum of the outputs of the three operations. After the multi-scale features and residuals pass through a multi-scale feature fusion module, the features and residuals of the same scale are spliced ​​together to form three new feature map shapes: 128×64×64, 256×32×32, and 512×16×16.

[0058] In the design of the multi-scale self-attention mechanism, for the feature map of shape c×h×w, the present invention proposes four scales of patch size, namely h, The present invention divides the feature map into patches of size s, flattens it into a one-dimensional vector, and embeds the features through a fully connected layer to obtain the query of the attention model ( in ), perform the same feature embedding operation on the flattened one-dimensional vector to obtain the Key of the attention model and Value The attention matrix is ​​calculated as follows:

[0059]

[0060] Next, A s Restore to the initial spatial resolution. The same operation is followed for other patch sizes, which are spliced ​​together and then the output of the multi-scale attention module is obtained through the residual convolution block. The present invention uses three layers of such modules at each scale, and uses the multi-scale fusion module to exchange information of the three scales as a jump connection of the anomaly decoder.

[0061] Each layer of the anomaly decoder module consists of linear interpolation upsampling and two convolution layers, a maximum pooling layer, and a ReLu activation function layer. The last layer consists of a convolution layer and a Sigmoid activation function layer, and outputs a predicted anomaly annotation map. The predicted anomaly annotation map is the same size as the network input image. The score of each pixel is between 0 and 1. The higher the score, the higher the probability of anomaly. The average of the 100 pixels with the highest scores is used as the image-level anomaly score.

[0062] Method evaluation

[0063] The present invention uses the image-level receiver operating characteristic curve (I) predicted by the model, the pixel-level receiver operating characteristic curve (P), the area overlap rate (O) at different thresholds at the pixel level, and the average accuracy (A) at the pixel level to comprehensively measure the evaluation indicators of anomaly detection and positioning.

[0064] Figure 4 This is a visualization result diagram of the present invention on the MVTec data set. For different types of anomalies reflected in different data sets, the anomaly location result of the method proposed by the present invention is more accurate than that of other methods.

[0065] The experimental results of the receiver operating characteristic curve (I) at the image level and the receiver operating characteristic curve (P) at the pixel level of the MVTec dataset are shown in Table 1. The experimental results of the regional overlap rate (O) and the average accuracy (A) at the pixel level of the MVTec dataset under different thresholds are shown in Table 2. The experimental results of the four indicators of the method of the present invention on the datasets DAGM, BTAD, and KolektorSDD2 are shown in Table 3.

[0066] Table 1

[0067]

[0068] Table 2

[0069]

[0070] Table 3

[0071]

[0072] According to Table 1, Table 2 and Table 3, the model proposed in the present invention can significantly improve the performance of anomaly detection and localization.

[0073] References

[0074] [1] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad–acomprehensive real-world dataset for unsupervised anomalydetection. In CVPR, 2019.

[0075] [2]Paul Bergmann,Michael Fauser,David Sattlegger,and CarstenSteger.Uninformed students:Student-teacher anomaly detection withdiscriminative latent embeddings.In CVPR,2020.

[0076] [3]Hanqiu Deng and Xingyu Li.Anomaly detection via reversedistillation from one-class embedding.In CVPR,2022.

[0077] [4]Choubo Ding,Guansong Pang,and Chunhua Shen.Catching both gray andblack swans:Open-set supervised anomaly detection.In CVPR,2022.

[0078] [5] Ristea,Neelu Madan,Radu Tudor Ionescu,KamalNasrollahi,Fahad Shahbaz Khan,Thomas B Moeslund,and Mubarak Shah.Self-supervised predictive convolutional attentive block for anomaly detection.InCVPR,2022.

[0079] [6]Karsten Roth,Latha Pemula,Joaquin Zepeda,Bernhard ThomasBrox,and PeterGehler.Towards total recall in industrial anomaly detection.InCVPR,2022.

[0080] [7]Mohammadreza Salehi,Niousha Sadjadi,Soroosh Baselizadeh,Mohammad HRohban,andHamid R Rabiee.Multiresolution knowledge distillation for anomalydetection.In CVPR,2021.[8]Vitjan Zavrtanik,Matej Kristan,and Danijel Skocˇaj.Draem-a discriminatively trainedreconstruction embedding for surfaceanomaly detection.In ICCV,2021.

[0081] [9]Jakob Bozˇicˇ,Domen Tabernik,and Danijel Skocˇaj.Mixed supervisionfor surface-defectdetection:From weakly to fully supervised learning.ComputInd,2021.

[0082]

[10] Pankaj Mishra,Riccardo Verk,Daniele Fornasier,Claudio Piciarelli,and Gian Luca Foresti.Vt-adl:A vision transformer network for image anomalydetection and localization.In ISIE,2021.

[0083]

[11] Matthias Wieler and Tobias Hahn.Weakly supervised learning forindustrial opticalinspection.2007.

[0084]

[12] Wen Liu,Weixin Luo,Dongze Lian,and Shenghua Gao.Future frameprediction foranomaly detection–a new baseline.In CVPR,2018

[0085]

[13] SebastianWaldstein,SophieKlimscha,Bianca SGerendas,Rene Donner,Thomas Schlegl,Ursula Schmidt-Erfurth,and GeorgLangs.Identifying and categorizinganomalies in retinal imaging data.arXivpreprint arXiv:1612.00686,2016.

[0086]

[14] Denis Gudovskiy,Shun Ishizaka,and Kazuki Kozuka.Cflow-ad:Real-time unsupervisedanomaly detection with localization via conditionalnormalizing flows.In WACV,2022.

[0087]

[15] Lukas Ruff,Robert A Vandermeulen,Nico Alexander Binder,Emmanuel Klaus-Robert and Marius Kloft.Deep semi-supervisedanomaly detection.arXivpreprint arXiv:1906.02694,2019.

[0088]

[16] Guansong Pang,Choubo Ding,Chunhua Shen,and An-ton van denHengel.Explainabledeep few-shot anomaly detection with deviationnetworks.arXiv preprint arXiv:2108.00462,2021.

[0089]

[17] Mircea Cimpoi,Subhransu Maji,Iasonas Kokkinos,Sammy Mohamed,and Andrea Vedaldi. Describing textures in the wild. In CVPR,2014.

Claims

1. A prototype residual anomaly detection and location method, characterized in that: It is divided into three stages, as follows: Phase 1: Building a multi-scale prototype Based on all normal samples in the training set, a multi-scale feature map of the input normal image is extracted, and the prototype of each scale is represented by the features of the normal pattern inferred from all feature maps of the scale to construct a multi-scale prototype; The second stage: The anomaly synthesis strategy generates various types of anomalies online, and trains the anomaly reconstruction network based on the prototype residual through the consistency loss of the real anomaly annotation and the predicted anomaly annotation; In the target area of ​​normal samples, abundant new abnormal samples are generated by extending abnormalities and simulating abnormalities, and the real abnormal annotations are generated by the abnormal generation process; wherein: the extended abnormality is to generate new abnormal samples by randomly enhancing the color and spatial position of the abnormal area of ​​the abnormal samples in the training set, and placing them in the target area of ​​the normal samples; the simulated abnormality is to generate new abnormal samples by multiplying the random abnormal area generated by Perlin noise by the image randomly sampled and enhanced from other data sets or the self-enhancement of the image itself, and placing it in the target area of ​​the normal sample; the abnormal reconstruction network based on the prototype residual reconstructs the predicted abnormal annotation map through the abnormal decoder module; The prototype residual-based anomaly reconstruction network uses pixel-level supervision to reconstruct anomaly annotations. The network's loss function is the weighted sum of the smooth L1 loss and focal loss of the true anomaly annotation and the predicted anomaly annotation: ； is the weight coefficient of focal loss; and They correspond to the predicted anomaly labels and the actual anomaly labels respectively; Phase 3: Anomaly Detection and Location The model trained in the second stage is used as the inference model, and the normal or abnormal image to be tested is input into the network, and the abnormal annotation image is directly regressed and predicted in an end-to-end manner; where: The abnormal reconstruction network based on prototype residual adopts a U-Net network, which includes an image feature encoding module, a multi-scale prototype module, a multi-scale fusion module, a multi-scale self-attention module and an abnormal decoder module; the image feature encoding module obtains the multi-scale feature map of the input image, and the multi-scale prototype module is used to represent the residual features of the abnormality to the normal mode. The prototype of each scale is the feature representation of the normal mode inferred from all feature maps of the scale, which retains the complete spatial information and represents the normal mode; the residual is the pixel-level Euclidean distance between the feature map of each scale of the input image and the nearest prototype of the corresponding scale; the two multi-scale fusion modules perform self-attention on the feature map of the input image and the residual respectively. After exchanging information at different scales, the images are spliced ​​together. The spliced ​​image features are captured by the multi-scale self-attention module to capture the inconsistency under different receptive fields, and then the information is exchanged through the third multi-scale fusion module. The output of the third multi-scale fusion module is used as a jump connection. The abnormal decoder module reconstructs the predicted abnormal annotation map based on the output features and jump connections of the last layer of the image feature encoding module. Among them: the initial state of the prototype of each scale is the feature map randomly sampled from all normal samples of the scale, and then the cluster center is updated by the k-means clustering algorithm. The cluster center is called the prototype. In the update of k-means, the distance between two feature maps is measured by the Euclidean distance, and the termination state is determined by the number of iterations. The jump connections in the prototype residual-based anomaly reconstruction network are enhanced by multi-scale prototype residuals and multi-scale self-attention mechanisms. The residual between the feature maps of each scale of the input image and the prototype of the corresponding scale is calculated, representing the distance from the normal mode. The feature map and the residual will be used for multi-scale information interaction through two multi-scale fusion modules respectively. The information after interaction will be used for anomaly capture at multiple spatial perception angles through a multi-scale self-attention mechanism. The decoder generates predicted anomaly annotations through this enhanced feature and residual information.

2. The prototype residual abnormality detection and positioning method according to claim 1, characterized in that: In the first stage, the image feature encoding module uses a pre-trained ResNet model; the multi-scale prototype constructed based on normal samples has three scales, each of which corresponds to the output of the first, second and third layers of the image feature encoding module.

3. The prototype residual abnormality detection and positioning method according to claim 1, characterized in that: In the second stage, the number of abnormal samples in the training set is much less than the number of normal samples, and the distribution of the data set is unbalanced.

4. The prototype residual abnormality detection and positioning method according to claim 1, characterized in that: In the second stage, for object datasets, the target area is part of the foreground of the object, and for texture datasets, the target area is part of the entire image.

5. The prototype residual abnormality detection and positioning method according to claim 1, characterized in that: In the second stage, the enhancements used to generate extended anomalies and simulated anomalies include both color and spatial enhancements; the other dataset used to generate simulated anomalies is the descriptive texture dataset DTD; the self-enhancement of the image itself in generating simulated anomalies also includes random shuffling of the image at the patch level, specifically, splitting an image into The grid is randomly shuffled and then reassembled into a new random image.

6. The prototype residual abnormality detection and positioning method according to claim 1, characterized in that: The input of the multi-scale fusion module is multi-scale features or residuals, and the output is multi-scale fused features or residuals. The fusion calculation method performed on the output of each scale is different.

7. The prototype residual abnormality detection and positioning method according to claim 1, characterized in that: Each head of the multi-size self-attention module corresponds to a patch size division, and the information of multiple receptive fields is combined through convolution as the output of the jump connection.

8. The prototype residual abnormality detection and positioning method according to claim 1, characterized in that: The anomaly decoder module predicts the annotation map of the abnormal area through upsampling and convolution; the last layer of the anomaly decoder module consists of a convolution layer and a sigmoid activation function layer, and predicts the average value of the N pixels with the highest scores in the anomaly annotation as the anomaly score of the input image.

Images