An image anomaly detection method based on self-supervised learning and knowledge distillation
Patent Information
- Application Number
- CN202410112546.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-26
- Publication Date
- 2026-09-18
- Estimated Expiration
- 2044-01-26
AI Technical Summary
[0004]本发明目的在于提供一种基于自监督学习与知识蒸馏的图像异常检测方法,以解决现有图像异常检测方法效果不佳的技术问题
[0051] 1) This invention generates anomaly rating maps end-to-end using a deep learning model, eliminating the need for complex post-processing, making training and convergence easier, and meeting the needs of practical applications;
Smart Images

Figure CN117934425B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image anomaly detection technology, and in particular relates to an image anomaly detection method based on self-supervised learning and knowledge distillation. Background Technology
[0002] Image anomaly detection technology is an important field of computer vision, with wide applications in industries such as manufacturing and healthcare. In real-world scenarios, because anomalous samples are very difficult to obtain and anomaly patterns are highly variable, current techniques typically use unsupervised learning methods to train anomaly detection models on normal samples, enabling the models to learn the patterns of normal samples and distinguish anomalies.
[0003] In recent years, deep learning-based image anomaly detection methods have become mainstream. Regarding anomaly map generation methods, they mainly fall into two categories: image segmentation-based methods and post-processing methods based on handcrafted features. Image segmentation-based methods typically use the concatenation of the original image with the output of a reconstruction network as input, followed by a classic segmentation network, such as UNet, to generate the anomaly map. These methods generally perform well, but the joint training of the reconstruction and segmentation networks is generally inefficient, requiring significant computational resources and time. Furthermore, the generalization ability of the reconstruction network results in mediocre scores for subtle anomalies. Post-processing methods based on handcrafted features are simple to implement, but less robust than the former, and require considerable time to fine-tune parameters for different datasets. Currently, unsupervised learning and image reconstruction methods are commonly used for image anomaly detection, with some incorporating UNet to achieve pixel-level anomaly detection, avoiding complex post-processing steps. However, these methods generally suffer from several problems. First, the quality of the reconstruction network is relatively poor, affecting overall performance. Second, the overly strong generalization ability for negative samples leads to poor performance in detecting subtle anomalies. This means that the method may need further improvement in practical applications to enhance its accuracy and robustness in handling anomalies. Summary of the Invention
[0004] The purpose of this invention is to provide an image anomaly detection method based on self-supervised learning and knowledge distillation, so as to solve the technical problem that existing image anomaly detection methods are not effective.
[0005] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:
[0006] An image anomaly detection method based on self-supervised learning and knowledge distillation includes the following steps:
[0007] Training phase:
[0008] Step S1: Collect normal images to construct a training set and perform preprocessing. The preprocessing first involves data augmentation, and then uniform scaling is performed after augmentation.
[0009] Step S2: Randomly generate pseudo-anomalies for the normal images I in the training set to obtain the corresponding pseudo-anomaly images I_a;
[0010] Step S3: Input the normal image I and the pseudo-abnormal image I_a into the pre-trained encoder for feature extraction, obtaining their respective multi-scale feature maps and compressed feature vectors. The encoder consists of an encoding part and a compressed feature vector extraction part. The encoding part is used to load the pre-trained model; its parameters are frozen and do not participate in training. The compressed feature extraction part is trainable. The multi-scale feature map of the normal image I is denoted as E_I. i For i = 1, 2, 3, the compressed feature vector of the normal image I is denoted as v_I; the multi-scale feature map of the pseudo-anomaly image I_a is denoted as E_I_a. i The compressed feature vector of the pseudo-anomaly image I_a is denoted as v_I_a, where i = 1, 2, 3; i represents the level, the smaller i is, the shallower the level, and the larger the corresponding feature map size; feature maps with the same i have the same size.
[0011] Step S4: Input the compressed feature vectors of the normal image I and the pseudo-abnormal image I_a into the decoder to obtain multiple reconstructed or repaired feature maps D_I at different scales. i , D_I_a i i = 1, 2, 3; the reconstructed feature map of normal image I is D_I i The repair feature map of the pseudo-anomaly image I_a is D_I_a. i ;
[0012] Step S5: Combine the feature maps E of the encoder at multiple scales i i = 1, 2, 3, and the multi-scale feature maps D reconstructed by the decoder. i The elements i = 1, 2, 3 are multiplied positionally and then negatively, and bilinear interpolation is performed on each element to make them the same size as the compressed image. The segments are then concatenated along the channel dimension to generate the input_seg of the segmentation network.
[0013] Step S6: Pass the input_seg into the segmentation network and output the anomaly score map mask_out;
[0014] Step S7: Define the loss function:
[0015]
[0016] Among them, E i D represents multiple scale feature maps output by the encoder. iThe output of the decoder consists of multiple scale feature maps, mask_out is the output of the segmentation network, mask_target is the anomaly mask of the pseudo-anomaly image, j represents the j-th element of mask_out and mask_target, and N represents the number of elements in mask_out and mask_target.
[0017] Cosine loss;
[0018]
[0019] Focal_loss(mask_out,mask_target)=-(mask_target·log(mask_out)+(1-mask_target)·log(1-mask_out))*(1-p_t) 4 ;
[0020] in:
[0021] p_t = mask_out * mask_target + (1 - mask_out) * (1 - mask_taregt); Repeat steps S2 to S6 to continuously update the parameters of the decoder, feature vector extraction network, and segmentation network through backpropagation and optimization algorithms;
[0022] Testing phase:
[0023] Step S8: Input the image to be tested t into the network cascaded with the encoder, decoder and segmentation network to obtain the anomaly scoring map, i.e. the pixel-level anomaly localization result;
[0024] Step S9: Post-process the anomaly scoring map to obtain the maximum value among the pixels, and obtain the image-level anomaly detection result of the image to be tested t.
[0025] Preferably, in step S1, the data enhancement method is to randomly select three enhancement methods from contrast, saturation, sharpness, affine transformation, and rotation and execute them in sequence, and then uniformly scale the image after enhancement; the image is scaled to size H×W, where H is the height of the scaled image and W is the width of the scaled image.
[0026] Preferably, the specific process of step S2 is as follows:
[0027] Step S21: Generate two-dimensional Perlin noise and set a threshold for binarization to obtain the anomaly mask M_a;
[0028] Step S22: Randomly sample image A from the anomaly source dataset D and perform random data augmentation on it;
[0029] Step S23: Sample A using the anomaly mask M_a to obtain the generated pseudo-anomaly;
[0030] Step S24: Superimpose the pseudo-anomaly with the normal image I with probability p to obtain the final pseudo-anomaly image I_a; where, is the inverted result of the exception mask M_a; β is a set parameter between 0 and 1; ⊙ indicates element-wise multiplication.
[0031] Preferably, the specific process of step S3 is as follows:
[0032] Step S31: Using a pre-trained model as a feature extractor, input image i into the pre-trained model to obtain feature maps output by multiple intermediate layers of the pre-trained model; wherein, the feature map output by intermediate layer i is... c i For feature map E i The number of channels, h i For feature map E i height, w i For feature map E i The width;
[0033] Step S32: For each selected intermediate layer i, transfer the feature map E i Downsampling to c i ×h0×w0, where h0 is the height of the selected deepest feature map and w0 is the width of the selected deepest feature map;
[0034] Step S33: Concatenate the selected intermediate layer feature maps along the channel dimension to obtain a multi-scale feature map.
[0035] Step S34: Convert the multi-scale feature map As input to the compressed feature vector extraction network, compressed feature vectors of normal and abnormal images are obtained;
[0036] Preferably, the specific process of step S5 is as follows:
[0037] Step S51: Multiply the intermediate layer feature maps corresponding to the encoder and decoder and take the negative number to obtain:
[0038] M i =-E i ×D i i = 1, 2, 3;
[0039] Step S52: Place M i Perform bilinear interpolation and output M. i ', with dimensions of H×W;
[0040] Step S53, Mi 'Concatenate along the channel dimension to obtain input_seg;
[0041] Preferably, the specific process of step S8 is as follows:
[0042] Step S81: Preprocess the image t to be tested, changing its size to H×W;
[0043] Step S82: Perform feature extraction on the preprocessed image t to obtain feature maps E at multiple scales. i ;
[0044] Step S83: Combine feature maps of multiple scales E i The compressed feature vector v is obtained by feeding it into the feature vector extraction network.
[0045] Step S84: Input the compressed feature vector v into the decoder to obtain reconstructed feature maps D at multiple scales. i ;
[0046] Step S85: Perform bilinear interpolation on the multiple scale feature maps of the encoder and the reconstructed multiple scale feature maps of the decoder to make them the same size as the image H×W, and then concatenate them in the channel dimension to generate the input_seg of the segmentation network.
[0047] Step S86: Pass the input_seg into the segmentation network and output the anomaly score map mask_out;
[0048] Preferably, the specific process of step S9 is as follows:
[0049] Step S91: Obtain the anomaly scoring map obtained in step S86. Take the maximum value to obtain the image-level anomaly score, which represents the probability that the image contains an anomaly.
[0050] The image anomaly detection method based on self-supervised learning and knowledge distillation of the present invention has the following advantages:
[0051] 1) This invention generates anomaly rating maps end-to-end using a deep learning model, eliminating the need for complex post-processing, making training and convergence easier, and meeting the needs of practical applications;
[0052] 2) This invention uses multi-layer feature maps between the encoder and decoder to suppress the generalization ability of model reconstruction and improve the image anomaly detection effect;
[0053] 3) This invention generates pseudo-anomalies and constructs an auxiliary task for feature reconstruction, and trains the model in a self-supervised learning manner to make it more adaptable to anomaly detection in images; this method is also very flexible and can improve the model's detection performance for specific types of anomalies by adjusting the distribution of pseudo-anomalies. Attached Figure Description
[0054] Figure 1 This is a schematic diagram of an image anomaly detection method based on self-supervised learning and knowledge distillation according to the present invention;
[0055] Figure 2 This is a schematic diagram of the anomaly detection results of the present invention on the MVTec AD dataset. Detailed Implementation
[0056] To better understand the purpose, structure, and function of this invention, the following detailed description of an image anomaly detection method based on self-supervised learning and knowledge distillation, in conjunction with the accompanying drawings, is provided.
[0057] Example 1:
[0058] To demonstrate the effectiveness of this method, this embodiment uses the MVTec AD dataset as the image data source. The MVTec AD dataset contains 5354 high-resolution color images of different target and texture types, and provides more than 70 different types of anomalies, simulating real-world industrial inspection scenarios. The operating environment of this embodiment is: GPU: Tesla P100-PCIE (16GB), CPU: Intel(R) Xeon(R) Gold 6132 CPU@2.60GHz, operating system: CentOS Linux 7 (Core).
[0059] According to the present invention, an image anomaly detection method based on self-supervised learning and knowledge distillation is provided, such as... Figure 1 As shown, the steps are as follows:
[0060] Training phase:
[0061] Step S1: Select normal samples from one class of the MVTec AD dataset to construct the training set, and scale the training set images to a size of 256×256;
[0062] Step S2: Randomly generate pseudo-anomalies from the normal images in the training set to obtain the corresponding pseudo-anomaly images. The specific process is as follows:
[0063] Step S21: Generate two-dimensional Perlin noise of size 256×256, and set the threshold to 0.5 for binarization to obtain the anomaly mask M_a;
[0064] Step S22: Randomly sample image A from the texture dataset DTD and perform random data augmentation on it. Specifically, select 3 from {adjust contrast, adjust brightness, adjust sharpness, adjust saturation, adjust exposure, tone separation, color inversion, histogram equalization, affine table transformation} to perform data augmentation on image A.
[0065] Step S23: Sample A using the anomaly mask M_a to obtain the generated pseudo-anomaly;
[0066] Step S24: Superimpose the pseudo-anomaly onto the normal image I with a probability of 0.5 to obtain the final pseudo-anomaly image I_a; where, is the inverted result of the anomaly mask M_a; β is a set parameter, uniformly sampled between 0.2 and 1; ⊙ indicates element-wise multiplication.
[0067] Step S3: Input the normal image I and the pseudo-abnormal image I_a into the pre-trained encoder for feature extraction, obtaining multiple scale feature maps and compressed feature vectors for each. The encoder is divided into an encoding part and a compressed feature vector extraction part. The encoding part is used to load the pre-trained model; the parameters of this part are frozen and do not participate in training. The compressed feature extraction part is trainable. As shown in the figure, E1 is the output of two convolutions, E2 is the output of one convolution, and E3 is the output of the three variables. These three variables are concatenated by channel dimension and then subjected to a 1*1 convolution to output the compressed feature vector to the decoder.
[0068] The multi-scale feature map of a normal image I is denoted as E. i For i = 1, 2, 3, the compressed feature vector is denoted as v; the multi-scale feature map of the pseudo-anomaly image I_a is denoted as E. i For i = 1, 2, 3, the compressed feature vector is denoted as v. The specific process is as follows:
[0069] Step S31: Use the ImageNet pre-trained model wide_resnet50_2 as the encoder, inputting normal images and pseudo-abnormal images into wide_resnet50_2 respectively, with batch_size=4, to obtain the feature maps output by its layer1, layer2, and layer3; the feature map output by the intermediate layer i is... c i For feature map E i The number of channels, h i For feature map E i height, w i For feature map E i The width of the feature map is as follows. The (channel, height, width) of the output feature map of layer1 for each image is (256, 64, 64), the (channel, height, width) of the output feature map of layer2 is (512, 32, 32), and the (channel, height, width) of the output feature map of layer3 is (1024, 16, 16).
[0070] The pre-trained model here can be replaced with other models, such as ResNet18, VGG19, etc. Generally, shallow layers extract more texture details and have a smaller receptive field, while deep layers extract more semantic information and have a larger receptive field. In this embodiment, the first three layers of the pre-trained model Wide_ResNet50_2 are used, and the last layer is discarded. This is because the pre-trained model is trained on ImageNet, and the deep semantic features are biased towards ImageNet, which deviates from the semantic features of the actual data and will affect the performance of anomaly detection.
[0071] Step S32: For each selected intermediate layer layer1, layer2, and layer3, downsample the feature map to c. i ×h0×w0, where h0 is the height of the selected deepest feature map and w0 is the width of the selected deepest feature map;
[0072] Step S33: Concatenate the selected intermediate layer feature maps along the channel dimension to obtain a multi-scale feature map.
[0073] Step S34: Convert the multi-scale feature map As input to the compressed feature vector extraction network, compressed feature vectors of normal and abnormal images are obtained;
[0074] The compressed feature vector extraction network consists of a residual block.
[0075] Step S4: Input the compressed feature vectors of the normal image I and the pseudo-abnormal image I_a into the decoder to obtain multiple feature maps D at different scales for reconstruction or repair. i i = 1, 2, 3;
[0076] The de_wide_resnet50_2 model was selected as the decoder, and the structure of the model is mirror-symmetric to the encoder in step S3.
[0077] Step S5: Combine the feature maps E of the encoder at multiple scales i i = 1, 2, 3, and the multi-scale feature maps D reconstructed by the decoder. i The elements i = 1, 2, 3 are multiplied positionally and then negatively, and bilinear interpolation is performed on each element to make them the same size as the compressed image. The segments are then concatenated along the channel dimension to generate the input_seg of the segmentation network.
[0078] Step S6: Pass the input_seg into the segmentation network and output the anomaly score map mask_out;
[0079] The segmentation network consists of four residual modules and an ASPP module, and its specific structure is shown in Table 1.
[0080] Table 1. Segmentation Network Structure
[0081]
[0082]
[0083] In this layer, Layers 9, 10, 11, and 12 are all inputs of Layer 8, and the results are concatenated by channel and then input into Layer 13.
[0084] Step S7: Define the loss function:
[0085]
[0086] Among them, E i D represents multiple scale feature maps output by the encoder. i The output of the decoder consists of multiple scale feature maps, mask_out is the output of the segmentation network, mask_target is the anomaly mask of the pseudo-anomaly image, j represents the j-th element of mask_out and mask_target, and N represents the number of elements in mask_out and mask_target.
[0087] Cosine loss;
[0088]
[0089] Focal_loss(mask_out,mask_target)=-(mask_target·log(mask_out)+(1-mask_target)·log(1-mask_out))*(1-p_t) 4 ;
[0090] in:
[0091] p_t = mask_out * mask_target + (1 - mask_out) * (1 - mask_taregt); Repeat steps S2 to S6 to continuously update the parameters of the decoder, feature vector extraction network, and segmentation network through backpropagation and optimization algorithms;
[0092] Figure 1 In the equation, distill loss is the distillation loss, corresponding to cosine_loss, and segmentation loss is the segmentation loss, corresponding to the sum of the last two terms.
[0093] For each category, 200 epochs were trained. The feature vector extraction network and decoder used the Adam optimizer with a learning rate of 0.005; the segmentation network used the SGD optimizer with a residual module learning rate of 0.1 and an ASPP learning rate of 0.01.
[0094] Testing phase:
[0095] Step S8: Input the image to be tested into the network proposed in this invention to obtain the anomaly scoring map, i.e., the pixel-level anomaly localization result;
[0096] Step S9: Take the maximum value of the anomaly score map to obtain the image-level anomaly detection result of the image to be tested.
[0097] In the field of image anomaly detection, the area under the receiver operating characteristic curve (AUROC) is commonly used to evaluate the performance of methods. Image-level anomaly detection uses image AUROC, while pixel-level anomaly localization uses pixel AUROC. However, considering that many anomalies occupy only a small portion of the pixels in an image, AUROC may not accurately reflect the actual anomaly localization performance. Therefore, an additional pixel-level average precision metric, AP, and the area under the region overlap curve, AUC-PRO, are introduced. Specifically, when using AUC-PRO, this embodiment follows standard settings, where the value is integrated over a false positive rate (FPR) (average false positive rate per pixel) ranging from 0 to 0.3.
[0098] This embodiment was tested on the MVTec AD dataset. Figure 2 This is a schematic diagram of the anomaly detection results of the present invention on the MVTec AD dataset. For each row, from left to right, they are: the image to be tested, the anomaly score map, the anomaly mask label, and the segmentation diagram. Table 2 shows the test results of the present invention on MVTec AD.
[0099] Table 2 Test results of the present invention on MVTec AD
[0100]
[0101]
[0102] As can be seen from the results in Table 2, the present invention performs excellently in both image-level anomaly detection and pixel-level anomaly localization tasks.
[0103] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.
Claims
1. An image anomaly detection method based on self-supervised learning and knowledge distillation, characterized in that, Includes the following steps: Step S1: acquire normal image The training set is constructed and preprocessed. The preprocessing is performed by data enhancement, and then scaling is performed. Step S2: training the normal images of the training set Randomly generate pseudo-abnormalities to obtain corresponding pseudo-abnormal images ; Step S3: normal image and pseudo abnormal image respectively into the pre-trained encoder for feature extraction to obtain a plurality of scale feature maps and compressed feature vectors respectively; the encoder is divided into an encoding part and a compressed feature vector extraction part, the encoding part is used to load the pre-trained model, and the parameters of this part are frozen and do not participate in training; The compressed feature extraction part is trainable; normal image Multiscale feature maps are denoted as Normal image The compressed eigenvector is denoted as ; Pseudo-anomaly images Multiscale feature maps are denoted as False anomaly images The compressed eigenvector is denoted as ;in, Indicates hierarchy, The smaller the value, the shallower the layer, and the larger the corresponding feature map size. The same feature maps have the same size; Step S4: Transfer the normal image and pseudo-anomaly images The compressed feature vectors are fed into the decoder to obtain multiple reconstructed or repaired feature maps at different scales. Normal image The reconstructed feature map is False anomaly images The repair feature map is ; Step S5: Combine the feature maps of the encoder at multiple scales. , and the decoder reconstructs feature maps at multiple scales The elements are multiplied positionally and then negatively divided. Bilinear interpolation is then performed separately to make the result the same size as the compressed image. Finally, the images are concatenated along the channel dimension to generate the input for the segmentation network. ; Step S6: [The sentence is incomplete and requires more context to be translated accurately.] Input to the segmentation network, output an anomaly score map ; Step S7: Define the loss function: ; in, The encoder outputs feature maps at multiple scales. These are multiple scale feature maps output by the decoder. To divide the output of the network, An anomaly mask for a pseudo-anomaly image. express and The j-th element, N represents and The number of elements; This is the cosine loss; ; ; in: Repeat steps S2 to S6, continuously updating the parameters of the decoder, feature vector extraction network, and segmentation network through backpropagation and optimization algorithms; Step S8: Transfer the image to be tested The network, which is cascaded with the encoder, decoder and segmentation network, is used to obtain the anomaly scoring map, i.e., the pixel-level anomaly localization result; Step S9: Post-process the anomaly scoring image to obtain the maximum value among the pixels, thus obtaining the image to be tested. Image-level anomaly detection results; The specific process of step S5 is as follows: Step S51: Multiply the intermediate layer feature maps corresponding to the encoder and decoder and take the negative number to obtain: ; Step S52: ... Perform bilinear interpolation and output The size is × , This represents the height of the scaled image. This represents the width of the scaled image. Step S53, will By splicing along the channel dimension, we obtain .
2. The image anomaly detection method based on self-supervised learning and knowledge distillation according to claim 1, characterized in that, In step S1, the data enhancement method is as follows: three enhancement methods are randomly selected from contrast, saturation, sharpness, affine transformation, and rotation and executed in sequence, and then scaled uniformly after enhancement; Resize the image to the desired size. × .
3. The image anomaly detection method based on self-supervised learning and knowledge distillation according to claim 2, characterized in that, The specific process of step S2 is as follows: Step S21: Generate two-dimensional Perlin noise, set a threshold for binarization, and obtain the anomaly mask. ; Step S22: From the anomaly source dataset Randomly sampled images And perform random data augmentation on it; Step S23: Use an anomaly mask right Sampling is performed to obtain the generated pseudo-anomalies; Step S24: With probability Distinguishing between pseudo-anomalies and normal images The images are then overlaid to obtain the final pseudo-anomaly image. ;in, ; For anomaly mask The inverse result; The parameter is set between 0 and 1; This indicates element-wise multiplication.
4. The image anomaly detection method based on self-supervised learning and knowledge distillation according to claim 3, characterized in that, The specific process of step S3 is as follows: Step S31: Use a pre-trained model as a feature extractor to extract the image. The input is fed into a pre-trained model to obtain feature maps output from multiple intermediate layers of the pre-trained model; where the intermediate layers... The output feature map is , For feature map The number of channels, For feature map height, For feature map The width; Step S32: For each selected intermediate layer , feature map downsampling to , The height of the selected deepest feature map, The width of the selected deepest feature map; Step S33: Concatenate the selected intermediate layer feature maps along the channel dimension to obtain a multi-scale feature map. ; Step S34: Convert the multi-scale feature map As input to the compressed feature vector extraction network, compressed feature vectors of normal and abnormal images are obtained.
5. The image anomaly detection method based on self-supervised learning and knowledge distillation according to claim 4, characterized in that, The specific process of step S8 is as follows: Step S81: Image to be tested Preprocessing is performed, and the dimensions become ; Step S82: Process the preprocessed image to be tested Feature extraction is performed to obtain feature maps at multiple scales. ; Step S83: Combine feature maps at multiple scales The vectors are fed into a feature vector extraction network to obtain compressed feature vectors. ; Step S84: Compress the feature vector The data is fed into the decoder to obtain reconstructed feature maps at multiple scales. ; Step S85: Perform bilinear interpolation on the multiple scale feature maps of the encoder and the reconstructed multiple scale feature maps of the decoder, respectively, so that they are consistent with the image. × The same size, concatenated along the channel dimension, is used to generate the input for the segmentation network. ; Step S86: ... Input to the segmentation network, output an anomaly score map .
6. The image anomaly detection method based on self-supervised learning and knowledge distillation according to claim 5, characterized in that, The specific process of step S9 is as follows: Step S91: Obtain the anomaly scoring map obtained in step S86. Take the maximum value to obtain the image-level anomaly score, which represents the probability that the image contains an anomaly.