An image anomaly detection method based on self-supervised learning and deep feature reconstruction

Abstract

The application discloses an image anomaly detection method based on self-supervised learning and deep feature reconstruction, which comprises the steps of generating pseudo anomalies, feature extraction, feature reconstruction, optimizing a convolutional autoencoder and the like. The application adopts feature reconstruction instead of traditional image reconstruction, and uses a pre-trained model to extract deep features of images, thereby avoiding the problem of excessively high reconstruction generalization of existing methods. In addition, the application trains a model in a self-supervised learning manner by simulating abnormal images, so that the model can better adapt to an anomaly detection task. The application is an end-to-end image anomaly detection method, has the advantages of good detection effect, small storage space occupation, fast inference speed and the like, and can meet the needs of practical applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image anomaly detection, and in particular to an image anomaly detection method based on self-supervised learning and deep feature reconstruction. BACKGROUND

[0002] Image anomaly detection technology is an important field of computer vision, and has wide application in industries such as industry and medicine. In practical applications, because abnormal samples are very difficult to obtain and abnormal patterns are very unstable, the current technology usually uses an unsupervised learning method to train an anomaly detection model on normal samples, so that the model learns the patterns of normal samples to distinguish anomalies.

[0003] In recent years, some image anomaly detection methods based on deep learning have become mainstream. They mainly have two categories: feature embedding-based methods and reconstruction-based methods. Among them, the feature embedding-based method uses a pre-trained model to extract the deep features of normal images, and then combines traditional anomaly detection models such as Gaussian distribution model, KNN model, and OC-SVM model to perform anomaly detection. This kind of algorithm performs well overall, but consumes a large amount of storage space to retain feature information, and has low inference speed, which is difficult to use in practical applications. The reconstruction-based method has high inference speed, but often cannot effectively detect anomalies due to the high generalization of reconstruction. The method DFR described in reference “Shi Y, Yang J, Qi Z. Unsupervised anomaly segmentation via deep feature reconstruction [J]. Neurocomputing, 2021, 424: 9-22” combines the advantages of both, uses a pre-trained model to extract image deep features, and then detects anomalies through feature reconstruction, balancing inference speed and detection effect. However, the pre-trained model uses a different task from image anomaly detection, so this method performs poorly on some subtle anomalies. SUMMARY

[0004] TECHNICAL PROBLEM

[0005] The present application aims to provide an image anomaly detection method based on self-supervised learning and deep feature reconstruction to solve the technical problem of poor effect of existing image anomaly detection methods.

[0006] TECHNICAL SCHEME

[0007] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0008] An image anomaly detection method based on self-supervised learning and deep feature reconstruction, comprising the following steps:

[0009] Training phase:

[0010] Step S1: Collect normal images I to construct a training set and do preprocessing;

[0011] Step S2: Randomly generate pseudo-anomalies for the normal images I in the training set to obtain the corresponding pseudo-anomaly images I a ;

[0012] Step S3: Feature extraction is performed on the normal images I and the pseudo-anomaly images I a , respectively, to obtain their respective multi-scale feature maps, the multi-scale feature map of the normal image I is denoted as F I , and the multi-scale feature map of the pseudo-anomaly image I a is denoted as F

[0013] Step S4: The multi-scale feature map F a of the pseudo-anomaly image I is input into a convolutional autoencoder to obtain a reconstructed multi-scale feature map F

[0014] Step S5: Define a loss function, repeat steps S2-S4, and continuously update the parameters of the convolutional autoencoder through backpropagation and optimization algorithms;

[0015] Test phase:

[0016] Step S6: Feature extraction is performed on the test image t, and its multi-scale feature map F t is input into the convolutional autoencoder to obtain a reconstructed multi-scale feature map F The anomaly score map of the test image is obtained by calculation;

[0017] Step S7: Post-processing is performed on the anomaly score map to obtain the anomaly detection and positioning results of the test image.

[0018] Preferably, in the step S1, the preprocessing operation is to scale the picture to a size of HxW, H is the height of the scaled picture, and W is the width of the scaled picture.

[0019] Preferably, the specific process of the step S2 is as follows:

[0020] Step S21: Generate a two-dimensional Perlin noise and set a threshold for binaryzation processing to obtain an anomaly mask M a ;

[0021] Step S22: Randomly select a picture A from the anomaly source dataset D and perform random data augmentation on it;

[0022] Step S23: Sample A using the anomaly mask M a to obtain the generated pseudo-anomalies;

[0023] Step S24: superimpose the pseudo anomaly with the normal image I with a probability p to obtain a final pseudo anomaly image I a ; wherein, is the negation result of the anomaly mask M a ; β is a set parameter between 0 and 1; and represents element-wise multiplication.

[0024] Preferably, the specific process of the step S3 is as follows:

[0025] Step S31: input the image i into a pre-trained model using the pre-trained model as a feature extractor to obtain feature maps output by multiple intermediate layers of the pre-trained model; wherein, the feature map output by the intermediate layer j is c j is the number of channels of the feature map F ij , h j is the height of the feature map F ij , and w j is the width of the feature map F ij ;

[0026] Step S32: for each selected intermediate layer j, upsample the feature map F ij to c j ×h0×w0, h0 is the height of the selected shallowest layer feature map, and w0 is the width of the selected shallowest layer feature map;

[0027] Step S33: concatenate the selected multiple intermediate layer feature maps in the channel dimension to obtain a multi-scale feature map wherein, the number of channels c = ∑ j c j ; the multi-scale feature map of the normal image I is denoted as F I , and the multi-scale feature map of the pseudo anomaly image I a is denoted as

[0028] Preferably, in the step S5, the definition of the loss function is as follows: wherein, is the reconstructed multi-scale feature map obtained in the step S4, and F I is the multi-scale feature map of the normal image I.

[0029] Preferably, the specific process of the step S6 is as follows:

[0030] Step S61: pre-process the to-be-tested image t to have a size of H×W;

[0031] Step S62: perform feature extraction on the pre-processed to-be-tested image t to obtain a multi-scale feature map

[0032] Step S63: input the multi-scale feature map F t into the convolutional autoencoder to obtain the reconstructed multi-scale feature map

[0033] Step S64: calculate the anomaly score map of the image to be tested wherein the anomaly score of each position (h, w) wherein i represents a specific channel on the multi-scale feature map, and h and w represent specific coordinates on the multi-scale feature map.

[0034] Preferably, the specific process of step S7 is as follows:

[0035] Step S71: upsample the anomaly score map obtained in step S64 to the size HxW;

[0036] Step S72: smooth the upsampled anomaly score map using a Gaussian filter to obtain the final anomaly score map

[0037] Step S73: set a threshold image_threshold, if the maximum value S max of the anomaly score map S is greater than image_threshold, the image to be tested t is determined to be abnormal; otherwise, it is determined to be normal; set a threshold pixel_threshold, and perform binaryzation processing on the anomaly score map S according to the threshold to obtain the abnormal positioning result.

[0038] Beneficial effects:

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] 1) The present application generates an anomaly score map end-to-end through a deep learning model, avoiding the consumption of a large amount of storage space, while greatly improving the inference speed of the model, which can meet the needs of practical applications;

[0041] 2) The present application uses a pre-trained model to extract image deep features, suppressing the generalization ability of model reconstruction and improving the image anomaly detection effect;

[0042] 3) The present application generates pseudo anomalies and constructs a feature reconstruction auxiliary task to train the model in a self-supervised learning manner, making it more suitable for image anomaly detection; this method is also very flexible, and can improve the detection effect of the model on specific types of anomalies by adjusting the distribution of pseudo anomalies. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1A schematic diagram of an image anomaly detection method based on self-supervised learning and deep feature reconstruction of the present application;

[0044] Figure 2 A schematic diagram of the anomaly detection result of the present application in the MVTec AD dataset. DETAILED DESCRIPTION

[0045] In order to better understand the purpose, structure and function of the present application, the present application will be further described in detail below in combination with the drawings.

[0046] Example 1

[0047] In order to prove the effectiveness of the present method, the MVTec AD dataset is used as the image data source in this embodiment; the MVTec AD dataset contains 5354 high-resolution color images of different targets and texture types, and provides more than 70 different types of anomalies, simulating the real-world industrial detection scene; the running environment of this embodiment is GPU: RTX 3090 (24GB), CPU: 14vCPU Intel(R) Xeon(R) Gold 6330 CPU@2.00GHz, operating system: Ubuntu 20.04.4LTS.

[0048] According to the image anomaly detection method based on self-supervised learning and deep feature reconstruction provided by the present application, as shown in Figure 1 , the steps are as follows:

[0049] Training phase:

[0050] Step S1: select a normal sample of a category of the MVTec AD dataset to construct a training set, and scale the training set image to a size of 256x256;

[0051] Step S2: randomly generate pseudo-anomalies for the normal images I of the training set to obtain the corresponding pseudo-anomaly images I a , the specific process is as follows:

[0052] Step S21: generate a two-dimensional Perlin noise with a size of 256x256, and set a threshold of 0.5 for binary processing to obtain an anomaly mask M a ;

[0053] Step S22: randomly select a picture A from the texture dataset DTD, and perform random data augmentation on it, the specific process being to randomly select 3 from {adjust contrast, adjust brightness, adjust sharpness, adjust saturation, adjust exposure, hue separation, inverse color, histogram equalization, affine transformation} to perform data augmentation on the picture A;

[0054] Step S23: superimpose the pseudo anomaly on the normal image I with a probability of 0.5 to obtain a final pseudo anomaly image I a Sample A to obtain the generated pseudo anomaly;

[0055] Step S24: superimpose the pseudo anomaly on the normal image I with a probability of 0.5 to obtain a final pseudo anomaly image I a ; wherein, is the negation result of the anomaly mask M a ; β is a set parameter, which is uniformly sampled between 0.2 and 1; and ⊙ represents element-wise multiplication.

[0056] Step S3: perform feature extraction on the normal image I and the pseudo anomaly image I a respectively to obtain respective multi-scale feature maps, and the multi-scale feature map of the normal image I is denoted as F I , and the multi-scale feature map of the pseudo anomaly image I a is denoted as The specific process is as follows:

[0057] Step S31: use the ImageNet pre-trained model wide_resnet50_2 as a feature extractor, and input the two images into wide_resnet50_2 respectively to obtain the feature maps output by layer1, layer2 and layer3; wherein the (channel, height, width) of the layer1 output feature map of each image is (256, 64, 64), the (channel, height, width) of the layer2 output feature map is (512, 32, 32), and the (channel, height, width) of the layer3 output feature map is (1024, 16, 16);

[0058] The pre-trained model here can be replaced by other models, such as resnet18, vgg19, etc.; generally, the features extracted by the shallow layer contain more texture detail information and have a smaller receptive field, while the features extracted by the deep layer contain more semantic information and have a larger receptive field; the first three layers of the pre-trained model wide_resnet50_2 are used in this embodiment, and the last layer is discarded, because the pre-trained model is trained on ImageNet, and the semantic features of the deep layer are biased towards ImageNet, which deviates from the semantic features of the actual data and affects the performance of anomaly detection;

[0059] Step S32: use the nearest up-sampling method to up-sample each image layer2 and layer3 to (512, 64, 64) and (1024, 64, 64) respectively;

[0060] Step S33: For each image, the layer1, layer2, layer3 feature maps are spliced in the channel dimension to obtain a multi-scale feature map with (channel, height, width) of (1792, 64, 64); the multi-scale feature map of the normal image I is denoted as F I , and the multi-scale feature map of the pseudo abnormal image I a is denoted as F

[0061] Step S4: The multi-scale feature map F a of the pseudo abnormal image I is input into the convolutional autoencoder to obtain a reconstructed multi-scale feature map F , wherein the structure of the convolutional autoencoder is shown in Table 1:

[0062] Table 1 Structure of the convolutional autoencoder

[0063] Layer Operation Layer1 Conv2d(in: 1792, out: 1024, kernel_size: 1x1, stride: 1); BatchNorm2d; ReLU Layer2 Conv2d(in: 1024, out: 256, kernel_size: 1x1, stride: 1); BatchNorm2d; ReLU Layer3 Conv2d(in: 256, out: 128, kernel_size: 1x1, stride: 1) Layer4 Conv2d(in: 128, out: 256, kernel_size: 1x1, stride: 1); BatchNorm2d; ReLU Layer5 Conv2d(in: 256, out: 1024, kernel_size: 1x1, stride: 1); BatchNorm2d; ReLU Layer6 Conv2d(in: 1024, out: 1792, kernel_size: 1x1, stride: 1)

[0064] Step S5: A loss function is defined, and steps S2-S4 are repeated to continuously update the parameters of the convolutional autoencoder through back propagation and optimization algorithms; wherein the loss function is batch_size is 8, the optimizer is Adam, learning_rate is 0.0001, and a total of 700 epochs are trained;

[0065] Test phase:

[0066] Step S6: Feature extraction is performed on the to-be-tested image t, and the multi-scale feature map F t of the to-be-tested image t is input into the convolutional autoencoder to obtain a reconstructed multi-scale feature map F The abnormal score map of the to-be-tested image is obtained by calculation, and the specific process is as follows:

[0067] Step S61: The to-be-tested image t is preprocessed, and the size is changed to 256x256;

[0068] Step S62: Feature extraction is performed on the preprocessed to-be-tested image t to obtain a multi-scale feature map F t with (channel, height, width) of (1792, 64, 64);

[0069] Step S63: The multi-scale feature map F t is input into the convolutional autoencoder to obtain a reconstructed multi-scale feature map F

[0070] Step S64: The abnormal score map of the to-be-tested image is obtained by calculation where the anomaly score of each position (h, w)

[0071] Step S7: Post-processing the anomaly score map to obtain the anomaly detection and positioning results of the test image, the specific process is as follows:

[0072] Step S71: Upsampling the anomaly score map S obtained in step S64 to the size of 256x256 by using the bilinear interpolation method;

[0073] Step S72: Smoothing the upsampled anomaly score map by using a Gaussian filter (σ=4) to obtain the final anomaly score map S

[0074] Step S73: Setting a threshold image_threshold, if the maximum value S of the anomaly score map S is greater than image_threshold, the test image t is determined to be abnormal; otherwise, it is determined to be normal; setting a threshold pixel_threshold, and performing binaryzation processing on the anomaly score map S according to the threshold to obtain the abnormal positioning result. max

[0075] The area under the receiver operating characteristic curve (AUROC) is usually used to evaluate the performance of the method in the field of image anomaly detection, the imageAUROC is used for image-level anomaly detection, and the pixelAUROC is used for pixel-level anomaly detection; in addition, the evaluation index AUPRO is also used for pixel-level anomaly detection, which can reduce the preference of AUROC for large area anomalies.

[0076] The present embodiment is tested on the MVTecAD dataset, wherein the image_threshold and the pixel_threshold are obtained according to the maximum F1-score, Figure 2 Fig. 2 is a schematic diagram of the anomaly detection results of the present application on the MVTecAD dataset, and Table 2 is the test results of the present application on the MVTecAD dataset.

[0077] Table 2 Test results of the present application on the MVTecAD dataset

[0078]

[0079]

[0080] As can be seen from the results in Table 2, the present application has excellent performance in image-level and pixel-level anomaly detection tasks, and has high inference speed.

[0081] ​​Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles and advantages of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. An image anomaly detection method based on self-supervised learning and deep feature reconstruction, characterized in that, The method comprises the following steps: Step S1: collecting normal images I to construct a training set and performing preprocessing; Step S2: randomly generating pseudo-abnormalities for normal images I of the training set to obtain corresponding pseudo-abnormal images I a ; Step S3: feature extraction is performed on the normal image I and the pseudo abnormal image I a respectively to obtain respective multi-scale feature maps, the multi-scale feature map of the normal image I is denoted as F I , and the multi-scale feature map of the pseudo abnormal image I a is denoted as F Step S4: feeding the pseudo-anomalous image I a into a convolutional autoencoder to obtain a reconstructed multi-scale feature map ​ Step S5: defining a loss function, repeating steps S2-S4, and continuously updating the parameters of the convolutional autoencoder through back propagation and optimization algorithm; Step S6: feature extraction is performed on the to-be-tested image t to obtain a multi-scale feature map F t The reconstructed multi-scale feature map is obtained by inputting the to-be-tested image into the convolutional autoencoder The anomaly score map of the to-be-tested image is obtained by calculation. Step S7: post-processing the anomaly score map to obtain the anomaly detection and positioning results of the image to be tested; The specific process of step S2 is: Step S21: generate two-dimensional Perlin noise, and set a threshold for binaryzation to obtain an anomaly mask M a ; Step S22: randomly selecting a picture A from the anomaly source data set D and performing random data augmentation on it; Step S23: masking the abnormality with the abnormality mask M a Sampling A to get the generated pseudo abnormality; Step S24: superimpose the pseudo anomaly with the normal image I with a probability p to obtain a final pseudo anomaly image I a ; wherein, is an inverse result of the anomaly mask M a ; β is a set parameter between 0 and 1; and ⊙ represents element-wise multiplication. The specific process of step S6 is: Step S61: preprocessing the image to be tested t to a size of HxW; H is the height of the scaled picture, and W is the width of the scaled picture; Step S62: feature extraction is performed on the preprocessed to-be-detected image t to obtain a multi-scale feature map h0 is the height of the selected shallowest layer feature map, w0 is the width of the selected shallowest layer feature map; c is the channel number; Step S63: input the multi-scale feature map F t into the convolutional autoencoder to obtain the reconstructed multi-scale feature map Step S64: calculate the anomaly score map of the image to be tested wherein the anomaly score of each position (h, w) wherein i represents a specific channel on the multi-scale feature map, and h and w represent specific coordinates on the multi-scale feature map.

2. The image anomaly detection method based on self-supervised learning and deep feature reconstruction according to claim 1, characterized in that, In the step S1, the preprocessing operation is to scale the picture to a size of HxW.

3. The image anomaly detection method based on self-supervised learning and deep feature reconstruction according to claim 2, characterized in that, The specific process of step S3 is: Step S31: inputting the image i into the pre-trained model using the pre-trained model as a feature extractor to obtain feature maps output by multiple intermediate layers of the pre-trained model; wherein the feature map output by the intermediate layer j is c j F ij is a channel number of the feature map F j is a height of the feature map F ij is a width of the feature map F j is a width of the feature map F ij ​ Step S32: For each intermediate layer j selected, the feature map F ij Up-sampling to c j × h0 × w0; Step S33: Splice the selected plurality of intermediate layer feature maps in the channel dimension to obtain a multi-scale feature map wherein the number of channels c = ∑ j c j ; the multi-scale feature map of the normal image I is denoted as F I , and the multi-scale feature map of the pseudo-abnormal image I a is denoted as F 4. The image anomaly detection method based on self-supervised learning and deep feature reconstruction according to claim 3, characterized in that, In the step S5, the definition of the loss function is wherein, is the reconstructed multi-scale feature map obtained in step S4, F I is the multi-scale feature map of the normal image I.

5. The image anomaly detection method based on self-supervised learning and deep feature reconstruction according to claim 4, characterized in that, The specific process of step S7 is: Step S71: Obtain the anomaly rating map obtained in step S64. Upsampled to size H×w; Step S72: Smooth the anomaly score map after upsampling using a Gaussian filter to obtain the final anomaly score map Step S73: Set a threshold image_threshold. If the maximum value S of the anomaly rating image S is exceeded... max If the value is greater than image_threshold, the image t to be tested is judged as abnormal; otherwise, it is judged as normal. A threshold pixel_threshold is set, and the abnormal scoring image S is binarized according to this threshold to obtain the abnormal location result.

Images