Self-supervised image anomaly detection method based on patch aggregation and discrimination

Abstract

The invention provides a self-supervised image anomaly detection method based on patch aggregation and discrimination, and belongs to the technical field of image anomaly detection. The method mainly comprises the steps of abnormal sample synthesis, depth feature extraction, patch aggregation, rapid attention feature fusion, feature correction and patch discrimination. According to the method, firstly, abnormal sample synthesis is carried out according to normal samples acquired from an image anomaly detection data set, then deep feature extraction is carried out on the normal samples and the abnormal samples, patch aggregation is carried out on extracted feature maps, the feature maps subjected to patch aggregation are fused by using rapid attention feature fusion, and the feature maps subjected to patch aggregation are subjected to feature fusion. And performing feature correction and patch discrimination on the fused feature map, integrating discrimination results of each scale, and performing up-sampling to obtain an input sample size so as to obtain a pixel-level discrimination result. According to the method, the problem that an artificially synthesized abnormal sample is not real enough in a conventional image anomaly detection method based on self-supervised learning is relieved, and meanwhile, the reasoning speed and accuracy are well balanced.

Description

Technical Field

[0001] The present invention belongs to the field of image anomaly detection, and in particular relates to a self-supervised image anomaly detection method based on patch aggregation and discrimination. Background Art

[0002] Image anomaly detection is one of the tasks in the field of computer vision. Its purpose is to identify abnormal samples from image data and locate abnormal areas. In recent years, deep learning has been widely used to solve problems in the field of computer vision, and many experts and scholars have also applied deep learning to solve the problem of image anomaly detection. Image anomaly detection methods based on deep learning can be roughly divided into supervised learning methods and unsupervised learning methods.

[0003] Supervised learning methods use abnormal samples with marked abnormal areas for training, and directly use target detection and semantic segmentation methods in computer vision tasks to locate and segment abnormal areas. The advantages of these methods are that they have excellent accuracy and reasoning speed under conditions of known anomalies, but in actual industrial production, the defective rate is usually very low, so the number of abnormal samples is very small, and it is difficult to obtain all types of anomalies. This limits the development and application of supervised learning methods in the industrial field.

[0004] Unsupervised learning methods only use normal samples for training, which solves the problems of supervised learning methods and is therefore gaining more and more attention. There are two basic ideas for unsupervised learning methods: one is image reconstruction. This method assumes that the reconstruction network trained only with normal samples cannot reconstruct abnormal areas well, so the area with large pixel differences before and after reconstruction is the abnormal area; the other idea is feature embedding. The feature space is different from the image RGB pixel space. The feature space is usually sparse. Constructing a classification surface in the feature space makes it easier to classify normal features from abnormal features. However, since unsupervised learning methods have not learned any abnormal knowledge, the network learns a single pattern, which may result in large errors during detection, resulting in reduced accuracy.

[0005] In order to solve the problems existing in unsupervised learning methods, self-supervised learning methods that use artificially synthesized abnormal samples for training have gradually gained attention. Self-supervised learning is also a type of unsupervised learning. It only needs normal samples during training. Unlike conventional unsupervised learning methods, self-supervised learning methods also use abnormal samples synthesized from normal samples for training. Therefore, self-supervised learning can be regarded as supervised learning with a large number of known abnormal samples or unsupervised learning that can learn both normal knowledge and abnormal knowledge. Although self-supervised learning methods have many advantages, their detection effect is very dependent on the authenticity of the synthesized abnormal samples. If the synthesized abnormal samples are not authentic enough, the network will learn false abnormal knowledge, resulting in reduced accuracy. At present, most of the abnormal samples synthesized by the abnormal sample synthesis methods have the problem of insufficient authenticity. In addition, many unsupervised learning methods and self-supervised learning methods do not consider the inference speed and deployment cost, which are very important requirements in actual industrial production. It is generally believed that an excellent image anomaly detection method should have low time complexity and space complexity. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a self-supervised image anomaly detection method based on patch aggregation and discrimination in response to the deficiencies of the prior art. During training, the method distinguishes normal samples from abnormal samples at the pixel level so that the network has the ability to distinguish normal samples from abnormal samples, and improves the accuracy as much as possible while ensuring the inference speed, thereby solving the problems existing in the existing self-supervised learning methods.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a self-supervised image anomaly detection method based on patch aggregation and discrimination, comprising the following steps:

[0008] Step 1: Obtain normal samples from the existing image anomaly detection dataset;

[0009] The image anomaly detection dataset consists of product image data collected in actual industrial production, including N categories of product image data, and has a variety of different anomaly types. Each category of image anomaly detection dataset is divided into a training set and a test set, where the training set only contains normal samples, and the test set contains both normal samples and abnormal samples, and each abnormal sample is annotated at the pixel level;

[0010] Step 2: Obtain the abnormal region mask and synthesize abnormal samples for training, including the following steps:

[0011] Step 2.1: Obtain an abnormal mask image from an existing mask image dataset as an abnormal area mask;

[0012] The mask image dataset consists of binary mask image data of different shapes, positions and sizes. The white area in the mask image is the abnormal area, the black area is the normal area, and the binary image is used as the abnormal mask image, that is, the value of the abnormal area mask is 1, and the value of the normal area mask is 0;

[0013] Step 2.2: Randomly select M abnormal samples in the test set to simulate a small number of known abnormal samples in actual industrial production, and store the pixel values ​​of the abnormal areas in these abnormal samples in a random order into the pixel value database;

[0014] Step 2.3, correcting the abnormal region mask that does not conform to the abnormal semantics of the corresponding category;

[0015] Step 2.4, randomly selecting the same number of pixel values ​​as the corrected abnormal region mask from the pixel value database constructed in step 2.2 and replacing the original pixel values ​​of the region to color the abnormal region mask, and overlaying the colored abnormal region mask on the normal sample to obtain a synthesized abnormal sample;

[0016] Step 3: extracting multi-level features of normal samples and synthesized abnormal samples and constructing a feature map set, including the following steps;

[0017] Step 3.1: Use the multi-level feature extraction part of the classification network pre-trained on a large natural image dataset as a deep feature extractor to perform deep multi-level feature extraction on normal samples and synthetic abnormal samples respectively;

[0018] Step 3.2: construct the multi-level feature images extracted by the deep feature extractor into a feature map set as a representation of the semantics of different levels of the image;

[0019] Step 4: Patch aggregation is performed on the feature map set obtained in step 3, including the following steps:

[0020] Step 4.1: Using the idea of ​​image patch, use a sliding window to generate patches for each feature map in the extracted feature map set;

[0021] Step 4.2: Using the idea of ​​neighborhood aggregation, perform neighborhood aggregation on each generated patch to retain the overall semantic information of each patch;

[0022] Step 5: Perform fast attention feature fusion on each layer of feature map to obtain a fast attention feature map, including the following steps:

[0023] Step 5.1: Process the feature map after patch aggregation, and convert the feature map of a different layer from the feature map of the current processing layer to the same shape as the feature map of the current layer through the lifting channel and up and down sampling;

[0024] Step 5.2: assign different weights to the feature maps processed in step 5.1, and perform weighted summation of the feature maps processed in step 5.1 to obtain the spatial attention feature map of the current layer after the deep and shallow feature information are complemented;

[0025] Step 5.3: Perform global average pooling on the spatial attention feature map, and perform linear transformation on the obtained vector to obtain the weight of each channel of the spatial attention feature map;

[0026] Step 5.4: Multiply the weight obtained in step 5.3 by the broadcast of the spatial attention feature map to obtain the fast attention feature map after the fast attention feature fusion of the current layer feature map;

[0027] Step 6: Perform feature correction and patch discrimination on the fast attention feature map obtained in step 5.4 to achieve image anomaly detection, including the following steps:

[0028] Step 6.1: Use a fully connected layer to project the fast attention feature map after feature fusion from the source domain to the target domain, and modify the fast attention feature map to reduce the deviation between the extracted multi-level features caused by the difference between the image anomaly detection dataset and the large natural image dataset used for pre-training;

[0029] Step 6.2: Use a classifier composed of a multi-layer perceptron to perform pixel-level discrimination on the modified fast attention feature map to obtain a multi-layer discrimination result;

[0030] Step 6.3: Upsample the discrimination results of each layer obtained in step 6.2 to the maximum size of all layer feature maps and then calculate the mean to integrate the discrimination results at different scales, and then upsample the calculated mean to the same size as the input normal sample to obtain the final discrimination result;

[0031] Step 6.4: The sum of the binary cross entropy between the discrimination result of the synthetic abnormal sample for training in step 6.2 and the abnormal mask image and the binary cross entropy between the final discrimination result of the normal sample in step 6.3 and the mask image with all zeros is used as the training loss;

[0032] The training loss loss in step 6.4 is expressed as:

[0033] loss=-(log(1-z n )+M a ×log(z a )+(1-M a )×log(1-z a )) (1)

[0034] Among them, loss is the training loss, Ma is the abnormal mask image, z a is the network output when the synthetic abnormal sample is input, z n is the network output when normal samples are input;

[0035] Step 6.5: The final discrimination results of the abnormal samples in the test set obtained in step 6.3 are further subjected to Gaussian filtering to obtain an abnormal score map.

[0036] The beneficial effect of adopting the above technical solution is that the self-supervised image anomaly detection method based on patch aggregation and discrimination provided by the present invention has the following advantages compared with the prior art:

[0037] (1) Correct the unreal abnormal area mask to improve the authenticity of the artificially synthesized abnormal samples, reduce the false abnormal knowledge learned by the network, and thus improve the accuracy;

[0038] (2) The extracted feature maps are patched and the semantic information within each patch is aggregated so that each pixel can capture more information, reducing the overall computational complexity of the network and thus improving the inference speed. Fast attention feature fusion is then performed to effectively improve the accuracy of anomaly localization through the complementarity of deep and shallow feature information. The fused fast attention feature map is then corrected. Since the pre-trained deep feature extractor used is trained on a large natural image dataset, it has a different distribution from the industrial image anomaly detection dataset. Therefore, the fast attention feature map is projected from the source domain to the target domain to reduce the feature bias caused by the difference in dataset distribution and thus improve accuracy. Finally, a simple and efficient classifier composed of a multi-layer perceptron is used to perform patch discrimination on the fast attention feature map to obtain the pixel-level binary classification cross entropy sum that considers both normal samples and abnormal samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the overall structure of a self-supervised image anomaly detection method based on patch aggregation and discrimination provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of abnormal region mask correction provided by an embodiment of the present invention, wherein (a) is a texture-type abnormal region mask correction, and (b) is an object-type abnormal region mask correction;

[0041] Figure 3 The overall structure diagram of the deep feature extractor provided by the embodiment of the present invention;

[0042] Figure 4 A schematic diagram of patch aggregation provided by an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of fast attention feature fusion provided by an embodiment of the present invention;

[0044] Figure 6 Schematic diagram of feature correction and patch identification provided by an embodiment of the present invention;

[0045] Figure 7 A comparison chart of the relationship between the inference speed and the average pixel-level AUC-ROC of different methods provided in an embodiment of the present invention;

[0046] Figure 8 Anomaly score graphs for all categories of the MVTecAD dataset provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0048] In this embodiment, a self-supervised image anomaly detection method based on patch aggregation and discrimination is provided. Figure 1 As shown, the following steps are included:

[0049] Step 1: Obtain normal samples from the existing image anomaly detection dataset;

[0050] The image anomaly detection dataset in step 1 is composed of product image data collected in actual industrial production, including N types of product image data, and has a variety of different anomaly types. Each category of image anomaly detection dataset is divided into a training set and a test set, wherein the training set only contains normal samples, and the test set contains both normal samples and abnormal samples, and each abnormal sample is annotated at the pixel level, so it is very suitable for evaluating the image anomaly detection algorithm;

[0051] This embodiment uses the image anomaly detection dataset MVTecAD published in 2019 for training and testing. The image anomaly detection dataset contains 15 different categories of image data, including scratches, cracks, stains, and various structural defects, about 70 different types. Each category of image anomaly detection dataset is divided into a training set and a test set. The training set only contains normal samples, and the test set contains both normal samples and abnormal samples. Each abnormal sample is annotated at the pixel level, so it is very suitable for evaluating image anomaly pattern detection algorithms. During training, normal samples are randomly selected from the corresponding categories of the training set of the image anomaly detection dataset. Among them, B is the number of normal samples in each training batch, H is the length of the normal sample image, and W is the width of the normal sample image. During the test, the abnormal samples are sequentially selected from the corresponding categories of the test set of the image anomaly detection dataset. Wherein, H' is the length of the abnormal sample image, and W' is the width of the abnormal sample image; the sizes of the normal sample and the abnormal sample in this embodiment are H=H'=256, W=W'=256;

[0052] Step 2: Obtain the abnormal region mask and synthesize abnormal samples for training, including the following steps:

[0053] Step 2.1: Obtain an abnormal mask image from an existing mask image dataset as an abnormal area mask;

[0054] The mask image data set in step 2.1 is composed of binary mask image data of different shapes, positions and sizes. In the field of image anomaly detection, the white area in the mask image is the abnormal area, the black area is the normal area, and the binarized image is used as the abnormal mask image, that is, the value of the abnormal area mask is 1, and the value of the normal area mask is 0;

[0055] This embodiment uses the NVIDIA irregular mask image dataset as the abnormal mask image dataset to synthesize abnormal samples for training. The NVIDIA irregular mask image dataset has a total of 12,000 mask images, where the shape, position and size of the mask image are random in the mask image. During training, a mask image is randomly selected from the NVIDIA irregular mask image dataset.

[0056] Step 2.2: Randomly select M abnormal samples in the test set to simulate a small number of known abnormal samples in actual industrial production, and store the pixel values ​​of the abnormal areas in these abnormal samples in a random order into the pixel value database;

[0057] In this embodiment, before the training starts, two abnormal samples are randomly selected from the test set of the current category of the MVTecAD dataset to simulate a small number of known abnormal samples in actual industrial production, and the RGB pixel information of the abnormal area is obtained according to the corresponding benchmark truth map, and the pixel information is stored in the database and processed in a disordered order, and these prior knowledge are used to fill the color of the synthetic abnormal area mask;

[0058] Step 2.3, the abnormal region mask that does not conform to the abnormal semantics of the corresponding category is corrected, thereby increasing the authenticity of the synthesized abnormal sample;

[0059] In this embodiment, the abnormal region mask that does not conform to the abnormal semantics of the corresponding category is corrected, and the samples are divided into texture class and object class and processed using different strategies respectively. The workflow on the texture class and object class is as follows: Figure 2 As shown in Figure 2, for texture samples, abnormal samples are directly synthesized. For object samples, the strategy of synthesizing first and then correcting is used to synthesize abnormal samples. If the normal sample S is input n First, randomly select the image with the same shape as S in the NVIDIA irregular mask dataset. n The same number of mask images M n , M n The area with a median value of 1 is the abnormal area. If the sample type is texture, no correction is required and the abnormal mask image M can be directly obtained. a If the sample type is an object, the correction operation should be continued, and the maximum inter-class variance method (Otsu) is used to select the best threshold for the grayscale S n Perform foreground segmentation to obtain the foreground segmentation mask image M f , the specific formula is:

[0060] M f =Otsu(gray(S n )) (2)

[0061] Among them, S n is the normal sample of input, M f Segment the mask image for the foreground;

[0062] The foreground of this embodiment is the object, and the ultimate goal is to synthesize the abnormal samples onto the object instead of the background, so M n With M f Multiplying them together will give the corrected abnormal mask image M a , the specific formula is:

[0063]

[0064] Among them, M a is the abnormal mask image, M n For S n The same number of mask images;

[0065] Step 2.4, randomly selecting the same number of pixel values ​​as the corrected abnormal region mask from the pixel value database constructed in step 2.2 and replacing the original pixel values ​​of the region to color the abnormal region mask, and overlaying the colored abnormal region mask on the normal sample to obtain a synthesized abnormal sample;

[0066] In this embodiment, according to the abnormal mask image M aThe number of pixels in the abnormal area is randomly selected from the pixel value database to fill the area and obtain the abnormal mask image R after filling the color. a , and then R a The abnormal area in S n The synthesized abnormal sample S is obtained on the corresponding area in a , the specific formula is:

[0067] S a =(S n ×(1-M a ))+R a (4)

[0068] Among them, M a is the abnormal mask image, S a is the synthesized abnormal sample;

[0069] Step 3: extracting multi-level features of normal samples and synthesized abnormal samples and constructing a feature map set, including the following steps;

[0070] Step 3.1: Use the multi-level feature extraction part of the pre-trained classification network as a deep feature extractor to perform deep multi-level feature extraction on normal samples and synthetic abnormal samples respectively;

[0071] In order to better extract the multi-level features of normal samples and synthetic abnormal samples, this embodiment adopts a transfer learning strategy. It is generally believed that a classification network pre-trained on a large natural image dataset can obtain a deep feature extraction network with strong generalization ability after removing the classification module. Therefore, in this embodiment, the classification network uses the VGG-19 network pre-trained on the ImageNet dataset. The deep feature extractor of this network after removing the classification module is as follows: Figure 3 As shown;

[0072] Step 3.2: construct the multi-level feature images extracted by the deep feature extractor into a feature map set as a representation of the semantics of different levels of the image;

[0073] In this embodiment, the deep feature extraction network consists of 5 layers, including 16 convolutional layers (Conv) and 5 downsampling layers (Maxpool), generating 5 feature maps of different sizes. The ratio of the number of convolutional layers in each layer is 2:2:4:4:4. Since a pre-trained model trained on large natural image data is used, shallow features are usually representations of more general texture information, which are too universal to distinguish normal and abnormal features well. Therefore, only the last four layers of features are selected, and the feature maps of each level in each layer of features are added. The advantage of this is that the feature map can contain more information without increasing the number of feature map channels. Assume l iis the output feature map of the i-th convolutional layer, C i is the number of channels of the feature map of the current layer, then the set F consisting of these 4 layers of feature maps has the following specific formula:

[0074]

[0075] Step 4: Patch aggregation is performed on the feature map set obtained in step 3, including the following steps:

[0076] In the actual industrial production, the image abnormal area is usually small in area, and only some features in the extracted feature map are sensitive to abnormalities. This means that most of the spatial pixel information in the shallow feature map is of little help to abnormality detection, and most of the channel information in the deep feature map is also of little help to abnormality detection. Therefore, a method can be considered to integrate image information to filter redundant information. This embodiment introduces the idea of ​​image patch and neighborhood aggregation. Although this idea will lose some feature information, it can make each pixel in the image contain the overall semantic information of each patch, which is equivalent to reducing the overall computational workload of the network and thus improving the inference speed.

[0077] In this embodiment, patch aggregation (PAM) needs to set two parameters, namely, the size of the sliding window k = (h k ,w k ), where h k is the length of the sliding window, w k is the width of the sliding window; the step length of each sliding is t = (h t ,w t ), where h t is the step length of the longitudinal sliding, w t is the step length of lateral sliding, the workflow of PAM is as follows Figure 4 As shown;

[0078] Step 4.1: Using the idea of ​​image patch, use a sliding window to generate patches for each feature map in the extracted feature map set;

[0079] In this embodiment, if the input feature map is And f i ∈F,i∈{2,3,4,5}, use a sliding window to map the feature map f i After patch generation, the feature map is obtained Where D = h k × k , L i is the number of pixels in the aggregated feature map, as shown in the following formula:

[0080]

[0081] Among them, h i is the length of the feature map after aggregation, w i is the width of the feature map after aggregation;

[0082] This embodiment will Transformed into Then swap the third dimension with the fourth dimension, and we get After the above transformation, we get B×C i ×L i patches;

[0083] Step 4.2: Using the idea of ​​neighborhood aggregation, each generated patch is subjected to neighborhood aggregation, which can retain the overall semantic information of each patch while losing a small amount of information;

[0084] This embodiment performs neighborhood aggregation on each patch to obtain a feature map The specific formula is:

[0085]

[0086] The neighborhood aggregation operation in this embodiment is actually equivalent to performing global average pooling on each patch. The advantage of this is that the overall semantic information of each patch can be retained while losing a small amount of information. Transformed into At this time f i "That is f i The feature map generated after PAM processing is denoted as p i , all feature maps in set F are processed using PAM respectively, and the set P = {p2, p3, p4, p5} can be obtained;

[0087] In this embodiment, there are many data deformations and dimensional transformations. In order to more conveniently and quickly operate the feature map generated by PAM, embedding and de-embedding operations are defined. They are a pair of inverse operations. The embedding operation can transform three-dimensional data into one-dimensional data, and the de-embedding operation can transform one-dimensional data into three-dimensional data.

[0088] The embedding operation (embed): If the input feature map First, transform p into Then swap the second and third dimensions, and we get Finally, p is transformed into

[0089] The unembedding operation: If the input vector Given that L = h × w, since the feature map generated by PAM satisfies the condition N × C = (B × L) × C, we first transform p into Then swap the second and third dimensions, and we get Since L = h × w, p can finally be transformed into

[0090] Step 5: Perform fast attention feature fusion on each layer of feature maps after patch aggregation to obtain a fast attention feature map, including the following steps:

[0091] Most existing image anomaly detection methods use simple feature fusion methods such as addition and concatenation, and their fusion effect is relatively general or the number of channels of the fused feature map is high. The reasoning speed of many image anomaly detection methods is slow because the feature fusion process or the feature map after feature fusion is too complicated. Therefore, this embodiment adopts a fast feature fusion based on attention mechanism (FAFFM), and its overall schematic diagram is shown as follows: Figure 5 As shown in the figure, FAFFM converts the feature maps of different layers from the feature map of the current layer through the lifting channel and up and down sampling to the same shape as the feature map of the current layer and assigns different weights to these feature maps. The result of the weighted summation of these feature maps is the feature map after the feature information of the current layer is complementary. This feature information complementary method effectively improves the accuracy of abnormality positioning. At the same time, a simple multi-layer perceptron (MLP) structure is used to replace the convolutional neural network (CNN), making the whole method simple and efficient, and effectively improving the accuracy without greatly affecting the inference speed.

[0092] Step 5.1: Process the feature map after patch aggregation, and convert the feature map of a different layer from the feature map of the current processing layer to the same shape as the feature map of the current layer through the lifting channel and up and down sampling;

[0093] In this embodiment, the input feature map And p i ∈P,i∈{2,3,4,5}, feature map And p j ∈P\{p i},j∈{2,3,4,5}\{i}, where “\” means selecting all elements from the set {2,3,4,5} that do not belong to the set {i};

[0094] In this embodiment, p j Perform feature map size processing. The goal of this step is to convert h j Transformed to h i , w j Transformed to w i There are two cases when transforming. If h j ×j >h i × i , then adaptive average pooling (AAP) is used to convert p j The size becomes and p i Same, if h j × j <h i × i , then use the nearest interpolation method to j The size becomes and p i Same, both cases will end up with The specific formula is:

[0095]

[0096] Then p j Perform feature map channel processing. The goal of this step is to convert C j Change to C i , first Perform embed operation and transform it into Then use MLP to transform p j The number of channels becomes C i The result is p i The reference feature map p j The final result after processing is recorded as The specific formula is:

[0097]

[0098] Let p k ∈P\{p i ,p j},k∈{2,3,4,5}\{i,j},p l ∈P\{p i ,p j ,p k},l∈{2,3,4,5}\{i,j,k}, with p i As the benchmark k ,p l Also perform the above operation to obtain the feature map set U i ={p i ,p j→i ,p k→i ,p l→i};

[0099] Then use MLP to transform the feature map set U i The number of channels of all feature maps in the feature map is reduced to the middle channel number η (hyperparameter) to obtain the feature map The specific formula is:

[0100]

[0101] Then p i ',p j→i ',p k→i ',p l→i 'Connect in the channel dimension to get the feature map It can be expressed as:

[0102] p i =cat(p i ',p j→i ',p k→i ',p l→i ') (11)

[0103] And use the fully connected layer again to transform p i The number of channels of " is reduced to 4, and the weights of the feature maps of each layer are obtained through the softmax function It can be expressed as:

[0104] α i =softmax(Linear 4×η→4 (p i ”)) (12)

[0105] Then the weight α i With the feature map set U i The feature maps in the corresponding broadcast multiplication and summation are then input to the fully connected layer with the same number of channels and the feature map is output Finally, use the unembed operation to Transformed into At this time p i "'It's p i The feature map after feature information complementation can be expressed as:

[0106]

[0107] Among them, α i[:,u,:,:] To select only the u-th channel of this feature map;

[0108] Step 5.2: assign different weights to the feature maps processed in step 5.1, and perform weighted summation of the feature maps processed in step 5.1 to obtain the spatial attention feature map of the current layer after the deep and shallow feature information are complemented;

[0109] In this embodiment, the characteristic map p i ”' Perform global average pooling (GAP) to obtain Then input it into the fully connected layer and pass it through the sigmoid function to get p i ”'The weight corresponding to each channel The specific formula is:

[0110]

[0111] Then λ i With p i ”'Broadcast multiplication obtains the spatial attention feature map q after FAFFM feature fusion i , the specific formula is:

[0112]

[0113] Performing the above operations on all feature maps in set P, we can obtain the spatial attention feature map set Q = {q2, q3, q4, q5};

[0114] Step 5.3: Perform global average pooling on the spatial attention feature map, and perform linear transformation on the obtained vector to obtain the weight of each channel of the spatial attention feature map;

[0115] Step 5.4: Multiply the weight obtained in step 5.3 by the broadcast of the spatial attention feature map to obtain the fast attention feature map after the fast attention feature fusion of the current layer feature map;

[0116] Step 6: Perform feature correction and patch discrimination on the fast attention feature map obtained in step 5.4 to achieve image anomaly detection, including the following steps:

[0117] In this embodiment, the abnormal position detected on the fast attention feature map after feature fusion can already correspond to the abnormal position on the input sample. Therefore, the abnormality location can be directly determined by judging whether each pixel in the fast attention feature map after feature fusion has an abnormality. Since the information of each pixel in the fast attention feature map is the result of aggregation of its corresponding patch, the judgment result of each pixel represents the judgment result of the corresponding patch of the pixel, including the following steps: (1) feature correction, projecting the feature from the source domain to the target domain, (2) patch judgment, calculating the probability of the existence of an abnormality in each pixel in the corrected feature map. The overall schematic diagram of this part is shown in FIG. Figure 6 As shown;

[0118] Step 6.1: Use a fully connected layer to project the fast attention feature map after feature fusion from the source domain to the target domain, and modify the fast attention feature map to reduce the deviation between the extracted multi-level features caused by the difference between the image anomaly detection dataset and the large natural image dataset used for pre-training;

[0119] In this embodiment, since the deep feature extractor used is trained on a large natural image dataset, which is usually different from the image anomaly detection dataset in industrial production, directly using the features extracted by the feature extractor will cause the detection effect to be affected by the feature deviation caused by the difference in the dataset. Therefore, a feature correction (FCM) is introduced to project the fast attention feature map from the source domain to the target domain, reduce the feature deviation caused by the difference in the dataset, and thus improve the accuracy. Only a single fully connected layer is used for correction to achieve good results;

[0120] Set up a fully connected layer Input feature map And q i ∈Q, first Perform embed operation and transform it into Then q i Input to Project and output fast attention feature map It can be expressed as:

[0121]

[0122] Among them, e i is the fast attention feature map, is the fully connected layer,

[0123] Perform the above operation on all feature graphs in set Q to obtain feature graph set E = {e2, e3, e4, e5};

[0124] Step 6.2: Use a classifier composed of a multi-layer perceptron to perform pixel-level discrimination on the modified fast attention feature map to obtain a multi-layer discrimination result;

[0125] In this embodiment, the role of patch discrimination (PDM) is to determine whether each patch has an abnormality. Since the information of each pixel in the input feature map is the result of the aggregation of its corresponding patch, each pixel is essentially classified into two categories. Assuming that the probability of an abnormality in each pixel of a normal sample is 0, the probability that an abnormal sample only has an abnormal area in the benchmark truth map is 1, and the probability of other areas is 0, the binary cross entropy is selected as the loss function to train the network. In order to ensure the inference speed, PDM only uses a network with a 6-layer MLP and a 1-layer fully connected layer structure.

[0126] If the input feature map And e i ∈E,i∈{2,3,4,5}, firstly, e i Input the first layer MLP, C i Downgrade to C i / 2, then input to the MLP with the same number of channels and output Then e i 'Enter the second layer MLP and convert C i / 2 down to C i / 4, then input to the MLP with the same number of channels and output Then e i "Input into the third layer MLP, C i / 4 downgraded to C i / 8, then input to the MLP with the same number of channels and output Then e i ”'Enter the fourth fully connected layer and convert C i / 8 is reduced to 1, and then the probability of an abnormality at each pixel is obtained through the sigmoid function. Last pair Use the unembed operation to transform it into The process can be expressed as:

[0127]

[0128] Perform the above operation on all feature maps in set E to obtain feature map set X = {x2, x3, x4, x5};

[0129] Step 6.3: Upsample the discrimination results of each layer obtained in step 6.2 to the maximum size of all layer feature maps and then calculate the mean to integrate the discrimination results at different scales, and then upsample the calculated mean to the same size as the input normal sample to obtain the final discrimination result;

[0130] This embodiment uses the nearest interpolation method to transform the feature map x i The size becomes and Same, get This operation is performed on all elements of the feature map set X and then the average is calculated in the channel dimension. Its function is to integrate the abnormal information in the feature maps of each layer and obtain the feature map The specific formula is:

[0131]

[0132] Then use the nearest interpolation method to change the size of the feature map y to the same as the input sample Same, get the classification result It can be expressed as:

[0133]

[0134] Step 6.4: The sum of the binary cross entropy between the discrimination result of the synthetic abnormal sample for training in step 6.2 and the abnormal mask image and the binary cross entropy between the final discrimination result of the normal sample in step 6.3 and the mask image with all zeros is used as the training loss;

[0135] The training loss in step 6.4 can be expressed as:

[0136] loss=-(log(1-z n )+M a ×log(z a )+(1-M a )×log(1-z a )) (20)

[0137] Among them, loss is the training loss, M a is the abnormal mask image, z a is the network output when the synthetic abnormal sample is input, z n is the network output when normal samples are input;

[0138] In this embodiment, during training, if the input is a normal sample S n , the network output is z n , then the binary cross entropy bce n , the specific formula is:

[0139] bce n = -log(1-z n ) (twenty one)

[0140] If the input is S n Synthesized abnormal sample S a , and the abnormal mask image is M a , the network output is z a , then the binary cross entropy bce a , the specific formula is:

[0141] bce a =-(M a ×log(z a )+(1-M a )×log(1-z a )) (twenty two)

[0142] The total loss during training is as follows:

[0143] loss=bce n +bce a =-(log(1-z n )+M a ×log(za )+(1-M a )×log(1-z a )) (twenty three)

[0144] Step 6.5: The final discrimination result of the abnormal samples in the test set obtained in step 6.3 is further subjected to Gaussian filtering to obtain an abnormal score map;

[0145] In this embodiment, during the test, the test sample S is input. t , and use Gaussian filtering with variance σ to filter the network output z t Process and obtain A t It is the abnormal score map, the specific formula is:

[0146] A t =gaussian(z t ,σ) (24)

[0147] In this embodiment, in order to prove the detection effect, the accuracy and inference speed of the currently popular image anomaly detection methods are compared on the MVTecAD dataset;

[0148] This embodiment is implemented based on the Pytorch framework and trained on a GeForce RTX 3080ti with 12GB of video memory. The optimizer uses AdamW, which is an algorithm that improves Adam using L2 regularization. Since the deep multi-level feature extraction network is a pre-trained network that uses a transfer learning strategy, the weights of the network need to be frozen during training, and only the weights of FAFFM, FCM, and PDM need to be updated. The initialization settings of various parameters during the training process can be seen in Table 1;

[0149] Table 1 Training initialization parameters

[0150] parameter Numeric The number of samples per training batch 4 Number of training rounds 500 Learning Rate 0.00001 Weight decay coefficient 0.00005 Number of worker threads 15 Sample Image Size (256,256) k (4,4) t (4,4) η 16 σ (4,4)

[0151] Where k is the size of the sliding window of PAM, t is the step size of the sliding window of PAM, η is the number of intermediate channels in FAFFM, and σ is the variance of Gaussian filtering;

[0152] This embodiment uses performance evaluation indicators widely used in the field of anomaly detection: the area under the receiver operating characteristic curve (AUC-ROC) and the area under the area overlap curve (AUC-PRO) as accuracy evaluation indicators; the frames per second (FPS) are used as the inference speed evaluation indicator. Since the method of the present invention is a detection method for image anomalies, pixel-level AUC-ROC (P-AUC-ROC) is used as an evaluation indicator. This is because pixel-level AUC-ROC is easier to evaluate the anomaly localization performance of the method. The comparison of pixel-level AUC-ROC and AUC-PRO of different methods can be seen in Tables 2 and 3;

[0153] Table 2 Pixel-level AUC-ROC of different methods (%)

[0154]

[0155] Table 3 AUC-PRO of different methods (%)

[0156]

[0157] It can be seen from Tables 2 and 3 that, compared with the current image anomaly detection methods, the proposed method has higher pixel-level AUC-ROC and AUC-PRO in most categories, and has the highest average pixel-level AUC-ROC and AUC-PRO. The pixel-level AUC-ROC in the Carpet, Leather, Wood, Pill, Screw, and Zipper categories reached the highest level among all methods, and the AUC-PRO in the Carpet, Wood, Screw, and Zipper categories reached the highest level among all methods. The pixel-level AUC-ROC and AUC-PRO in the categories of complex semantic anomalies such as Cable and Transistor are relatively poor;

[0158] The relationship between the inference speed of different methods and the average pixel-level AUC-ROC is shown in Figure 7 As shown;

[0159] from Figure 7It can be seen that under the RTX3080ti test environment, the inference speed of the method of the present invention reaches 115FPS, which is far higher than other methods. The inference speeds of PatchCore and SimpleNet with the same average pixel-level AUC-ROC as in this embodiment are far lower than the inference speed in this embodiment. The method of the present invention optimizes the inference speed in many aspects, such as using a feature extraction network VGG-19 with a simpler structure (PatchCore and SimpleNet use the WRN50 network, and the feature map extracted by the VGG-19 network has a lower number of channels); using patch aggregation to reduce the overall computational load of the network; adopting a simpler and more efficient feature fusion strategy; the method of the present invention has a good balance between inference speed and accuracy. When the accuracy gap is not large, a higher inference speed is more suitable for current actual industrial production conditions;

[0160] The anomaly scores of the method of the present invention on all categories in the MVTecAD dataset are shown in the figure. Figure 8 As shown;

[0161] In summary, through comparative experiments, the method of the present invention has achieved the expected effect. It has improved the accuracy as much as possible on the basis of having a good reasoning speed. Both the reasoning speed and the accuracy have reached a relatively advanced level.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A self-supervised image anomaly detection method based on patch aggregation and discrimination, characterized by: The following steps are involved: Step 1: Obtain normal samples from the existing image anomaly detection dataset; Step 2: Obtain the abnormal area mask and synthesize abnormal samples for training; Step 3: Extract multi-level features of normal samples and synthesized abnormal samples and construct a feature map set; Step 4: Perform patch aggregation on the feature map set obtained in step 3; Step 5: Perform fast attention feature fusion on each layer of feature maps after patch aggregation to obtain a fast attention feature map; Step 6: Perform feature correction and patch discrimination on the fast attention feature map obtained in step 5 to achieve image anomaly detection.

2. The self-supervised image anomaly detection method based on patch aggregation and discrimination according to claim 1, characterized in that: The image anomaly detection dataset consists of product image data collected in actual industrial production, including N categories of product image data, and has a variety of different anomaly types. Each category of image anomaly detection dataset is divided into a training set and a test set, where the training set only contains normal samples, and the test set contains both normal samples and abnormal samples, and each abnormal sample is annotated at the pixel level.

3. The self-supervised image anomaly detection method based on patch aggregation and discrimination according to claim 1, characterized in that: The specific method of step 2 is: Step 2.1: Obtain an abnormal mask image from an existing mask image dataset as an abnormal area mask; Step 2.2: Randomly select M abnormal samples in the test set to simulate a small number of known abnormal samples in actual industrial production, and store the pixel values ​​of the abnormal areas in these abnormal samples in a random order into the pixel value database; Step 2.3, correcting the abnormal region mask that does not conform to the abnormal semantics of the corresponding category; Step 2.4: Randomly select the same number of pixel values ​​as the corrected abnormal area mask from the pixel value database constructed in step 2.2 and replace the original pixel values ​​of the area to color the abnormal area mask, and cover the colored abnormal area mask on the normal sample to obtain a synthetic abnormal sample.

4. The self-supervised image anomaly detection method based on patch aggregation and discrimination according to claim 3, characterized in that: The mask image dataset consists of binary mask image data of different shapes, positions and sizes. The white area in the mask image is the abnormal area, and the black area is the normal area. The binarized image is used as the abnormal mask image, that is, the value of the abnormal area mask is 1, and the value of the normal area mask is 0.

5. The self-supervised image anomaly detection method based on patch aggregation and discrimination according to claim 1, characterized in that: The specific method of step 3 is: Step 3.1: Use the multi-level feature extraction part of the pre-trained classification network as a deep feature extractor to perform deep multi-level feature extraction on normal samples and synthetic abnormal samples respectively; Step 3.2: The multi-level feature images extracted by the deep feature extractor are constructed into a feature map set as a representation of the semantics of different levels of the image.

6. The self-supervised image anomaly detection method based on patch aggregation and discrimination according to claim 5, characterized in that: The specific method of step 4 is: Step 4.1: Using the idea of ​​image patch, use a sliding window to generate patches for each feature map in the extracted feature map set; Step 4.2: Using the idea of ​​neighborhood aggregation, each generated patch is aggregated in the neighborhood to retain the overall semantic information of each patch.

7. The self-supervised image anomaly detection method based on patch aggregation and discrimination according to claim 6, characterized in that: The specific method of step 5 is: Step 5.1: Process the feature map after patch aggregation, and convert the feature map of a different layer from the feature map of the current processing layer to the same shape as the feature map of the current layer through the lifting channel and up and down sampling; Step 5.2: assign different weights to the feature maps processed in step 5.1, and perform weighted summation of the feature maps processed in step 5.1 to obtain the spatial attention feature map of the current layer after the deep and shallow feature information are complemented; Step 5.3: Perform global average pooling on the spatial attention feature map, and perform linear transformation on the obtained vector to obtain the weight of each channel of the spatial attention feature map; Step 5.4: Multiply the weight obtained in step 5.3 by the broadcast of the spatial attention feature map to obtain the fast attention feature map after the fast attention feature fusion of the current layer feature map.

8. The self-supervised image anomaly detection method based on patch aggregation and discrimination according to claim 7, characterized in that: The specific method of step 6 is: Step 6.1: Use a fully connected layer to project the fast attention feature map after feature fusion from the source domain to the target domain, and modify the fast attention feature map to reduce the deviation between the multi-level features extracted due to the difference between the image anomaly detection dataset and the image dataset used for pre-training; Step 6.2: Use a classifier composed of a multi-layer perceptron to perform pixel-level discrimination on the modified fast attention feature map to obtain a multi-layer discrimination result; Step 6.3: Upsample the discrimination results of each layer obtained in step 6.2 to the maximum size of all layer feature maps and then calculate the mean to integrate the discrimination results at different scales, and then upsample the calculated mean to the same size as the input normal sample to obtain the final discrimination result; Step 6.4: The sum of the binary cross entropy between the discrimination result of the synthetic abnormal sample for training in step 6.2 and the abnormal mask image and the binary cross entropy between the final discrimination result of the normal sample in step 6.3 and the mask image with all zeros is used as the training loss; Step 6.5: The final discrimination results of the abnormal samples in the test set obtained in step 6.3 are further subjected to Gaussian filtering to obtain an abnormal score map.

9. The self-supervised image anomaly detection method based on patch aggregation and discrimination according to claim 8, characterized in that: The training loss loss in step 6.4 is expressed as: loss=-(log(1-z n )+M a ×log(z a )+(1-M a )×log(1-z a )) Among them, loss is the training loss, M a is the abnormal mask image, z a is the network output when the synthetic abnormal sample is input, z n is the network output when normal samples are input.

Images