Anomaly Detection Method Based on Knowledge Distillation of Asymmetric Autoencoders

Abstract

The present invention provides an anomaly detection method for an asymmetric autoencoder based on knowledge distillation. In order to reduce reconstruction error failure, the decoding network of the autoencoder has fewer convolution kernels than the encoder, forming an asymmetric structural form. The asymmetric structural design can greatly reduce the number of parameters of the autoencoder network, and to a certain extent avoid the problem of reconstruction error failure caused by excessive generalization of the model. The autoencoder is combined with the knowledge distillation algorithm, and the knowledge distillation method is used for network training. While reducing the network parameters, the powerful feature extraction ability of the teacher network is utilized to guide the student network and improve its feature reconstruction ability. In order to enable the encoder as the teacher network to completely distill the knowledge to the decoding network as the student network during the knowledge distillation process, a hierarchical multi-level loss function is proposed, and two types of loss functions are designed to ensure pixel-level similarity while increasing the network receptive field by introducing global directional loss.

Description

Technical Field

[0001] The present invention relates to computer vision and machine learning technologies, and in particular to anomaly detection technology based on deep learning and autoencoders. Background Art

[0002] Anomaly detection technology combines pattern recognition, machine learning, and other disciplines. Data that deviates from the majority of observed data is considered anomaly. The goal of anomaly detection is to build a model using existing normal samples to detect various possible anomalies, without using actual anomaly samples. Early anomaly detection algorithms were mostly applied to data mining. However, with the recent development of related technologies such as computer vision and deep learning, many related works have introduced anomaly detection into the field of image processing to solve the problem of object detection in the absence of samples. This technology has important research significance and application value in fields such as industrial appearance defect detection, medical image analysis, and hyperspectral image processing.

[0003] With the development of deep learning, the main deep learning methods for image anomaly detection can be divided into two categories: reconstruction-based anomaly detection methods and feature embedding-based anomaly detection methods. The core idea of reconstruction-based anomaly detection methods is to build a model to learn the distribution characteristics and representation of normal samples. The model extracts features from the input image and then reconstructs it. Finally, anomaly detection is performed based on the reconstruction error between the input and output images. The current mainstream approach is to implement this method based on autoencoders. First, during the training phase, the autoencoder network is trained using only normal samples. During this process, the encoder learns the characteristics and distribution of normal samples in the latent layer, while the decoder network reconstructs the image based on the latent layer features. Then, during the testing phase, since the autoencoder network has never seen abnormal samples, it cannot reconstruct the anomaly during the reconstruction process. The resulting reconstructed image will partially ignore the anomaly information, appearing to repair the abnormal part. Finally, anomaly identification is performed by calculating the reconstruction error between the original image and the reconstructed image. The reconstruction error can be calculated by calculating the mean squared error between the original and reconstructed images. In addition, generative adversarial networks (GANs) are also often used in anomaly detection based on reconstruction methods. GANomaly, proposed by Samet Akcay et al., is an anomaly detection network based on adversarial generative networks and autoencoder networks. Unlike naive autoencoder networks, GANomaly introduces the adversarial generative network method, using the adversarial process between the generator and the discriminator to ensure that the generator can learn the distribution characteristics of normal samples.

[0004] Another major deep learning approach in anomaly detection is feature embedding-based anomaly detection. This involves feeding input into a model, extracting features, and then distinguishing samples within the feature space. This approach differs from reconstruction methods in that it detects anomalies not in the original space but in a high-dimensional feature space, primarily focusing on differences in feature space. Currently, the mainstream implementation of this approach is based on knowledge distillation. The key idea behind applying knowledge distillation to anomaly detection is this: During the training phase, normal samples are simultaneously fed into a teacher network pre-trained on a large dataset and an untrained, randomly initialized student network. The knowledge of the teacher network is distilled into the student network by minimizing the loss function between the teacher and student networks in the feature space. During the testing phase, abnormal samples are fed into the student network. Since the student network has never seen abnormal samples, while the teacher network, trained on large datasets and possessing strong feature extraction capabilities, ultimately represents them differently in the high-dimensional feature space. Therefore, the difference between the teacher and student networks in this high-dimensional feature space can be used to determine the sample's category.

[0005] Currently, the two mainstream anomaly detection methods can complete some anomaly detection tasks, but we believe that they still have the following problems:

[0006] 1) Reconstruction Error Failure: Reconstruction-based methods are not always effective and may experience reconstruction error failure in practice. Due to the generalization capability of the model, the decoding network may have excellent feature reconstruction capabilities during training. This results in the decoding network being able to reconstruct not only normal samples but also abnormal samples. This results in a smaller reconstruction error between the input and output, and ultimately the model cannot accurately distinguish between normal and abnormal samples.

[0007] 2) Missing RGB spatial information: Compared to reconstruction-based anomaly detection methods, feature embedding-based methods can avoid the problem of reconstruction error failure and improve anomaly detection accuracy to a certain extent. However, this method requires anomaly detection in a high-dimensional space and cannot generate a generated image in the RGB space. This method cannot be used to assist detection tasks that require RGB spatial information and can only be used independently. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a method for enhancing anomaly detection performance by improving its feature reconstruction capability.

[0009] The technical solution adopted by the present invention to solve the above technical problems is an anomaly detection method of an asymmetric autoencoder based on knowledge distillation, which can improve the reconstruction error failure problem of the reconstruction method based on the knowledge distillation method, and at the same time improve the anomaly detection accuracy of the network.

[0010] The steps include:

[0011] 1) During the training phase, only normal samples are input into the asymmetric autoencoder. The encoding network outputs the features of the normal samples to the decoding network, and the decoding network outputs the reconstructed normal samples.

[0012] The asymmetric autoencoder consists of an encoding network and a decoding network with asymmetric structures. A loss function is used to constrain the training process so that the reconstructed normal samples output by the decoding network can restore the normal samples input to the asymmetric autoencoder to the greatest extent possible. During training, the encoding network is used as the teacher network and the decoding network as the student network. A knowledge distillation algorithm is used to transfer the knowledge of the teacher network to the student network.

[0013] The encoding network consists of L feature extraction stages, each of which is composed of a stack of 3×3 convolution kernels. The output of each feature extraction stage is connected to the pooling module. The decoding network consists of L data reconstruction stages, each of which is composed of a stack of fewer 3×3 convolution kernels than the feature extraction stage of the same sequence, forming an asymmetric structure. The output of each data reconstruction stage is connected to the upsampling module.

[0014] 2) In the detection phase, the image to be detected is input into the trained asymmetric autoencoder. The encoding network outputs the features of the image to be detected to the decoding network. The decoding network outputs the reconstructed image to be detected. The difference between the image to be detected and the reconstructed image to be detected is compared. If the difference is greater than a threshold, the image to be detected is judged to be abnormal; otherwise, the image to be detected is judged to be normal.

[0015] Because only normal samples are used during training, during the testing phase, since the network has never seen abnormal data, the abnormal parts will inevitably be missing during the decoding network reconstruction process, resulting in poor reconstruction of abnormal samples. Therefore, abnormality can be judged by comparing the original image with the reconstructed image.

[0016] Loss function L total Through the pixel-level error loss L val And the global direction loss L dir Composition: L total =L val +αL dir , α is the adjustment L val With L dir Hyperparameters of weights.

[0017] Pixel-level error loss L val :

[0018]

[0019] Among them, l is the first stage, N l represents the total number of pixels in the first stage, f T l(·) represents the mapping function of the teacher network at stage l, represents the mapping function of the student network at stage l, represents the pixel-by-pixel input of the teacher network at stage l, Pixel-wise output of the student network at stage l.

[0020] Global directional loss L dir :

[0021]

[0022] Among them, vec(·) is a vectorization function that converts a matrix of any dimension into a one-dimensional vector, ||·|| represents the modulus function of the vector, x l is the input of the teacher network in stage l, y l is the output of the student network in stage l.

[0023] Unlike common symmetric autoencoders, we designed a more lightweight decoder network to mitigate reconstruction error failure. This decoder network has a similar structure to the encoder, but with fewer convolution kernels, resulting in an asymmetric architecture. This asymmetric design significantly reduces the number of parameters in the autoencoder network, to some extent avoiding the reconstruction error failure caused by excessive model generalization. We also incorporate knowledge distillation into network training. Unlike common knowledge distillation algorithms that embed features in a downsampled high-dimensional space, we combine autoencoders with knowledge distillation algorithms. While reducing network parameters, we leverage the powerful feature extraction capabilities of the teacher network to guide the student network and improve its feature reconstruction capabilities. This approach enables rapid network training and produces better generation results. Furthermore, to ensure that the encoder network (the teacher network) fully transfers knowledge to the decoder network (the student network) during the knowledge distillation process, we propose a hierarchical multi-level loss function and design two types of loss functions: a pixel-level distance loss and a global directional loss. This ensures pixel-level similarity while increasing the network's receptive field by introducing a global directional loss.

[0024] The beneficial effect of the present invention is that, by combining knowledge distillation with autoencoders, an anomaly detection method for an asymmetric autoencoder based on knowledge distillation is proposed. The failure of reconstruction error is reduced by designing an asymmetric knot. At the same time, the characteristics of knowledge distillation are utilized to improve the feature extraction ability of the encoding network. In combination with the loss function proposed by us, it is ensured that the decoding network can smoothly learn the knowledge of the encoding network and improve its reconstruction ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the asymmetric autoencoding network structure;

[0026] Figure 2 Schematic diagram of the asymmetric autoencoder network structure based on knowledge distillation;

[0027] Figure 3 Schematic diagram of the training and testing process of the asymmetric autoencoder network based on knowledge distillation. DETAILED DESCRIPTION

[0028] The embodiment steps are as follows:

[0029] (1) Design and construction of asymmetric autoencoding network

[0030] Asymmetric autoencoder network structure such as Figure 1 As shown. The encoding network Encoder Architecture of this network uses VGG16 as the backbone network. The VGG16 network has very good feature extraction capabilities. The present invention removes the fully connected layer in its network and only retains the feature extraction part. The whole is divided into 5 stages. Each stage is stacked by 3×3 convolutions and connected by pooling. In fact, the stacking of 3×3 convolutions can increase the receptive field while reducing the number of network parameters. This is one of the reasons why the VGG16 network has good feature extraction capabilities. The specific convolutions of the 5 stages are 3×3×64, 3×3×64 in the first stage, 3×3×128, 3×3×128 in the second stage, 3×3×256, 3×3×256, 3×3×256 in the third stage, 3×3×512, 3×3×512, 3×3×512 in the fourth stage, and 3×3×512, 3×3×512, 3×3×512 in the fifth stage.

[0031] The right half of the network is the decoding network Decoder Architecture, such as Figure 1 The decoding network shown also uses a stacked convolutional architecture and is formally divided into five stages. Each stage consists of a stack of 3×3 convolutions connected by upsampling. However, the decoding network is thinner, with fewer convolution kernels in each stage, resulting in an asymmetric structure. The specific convolution kernels in the five stages are: 3×3×16, 3×3×16, and 3×3×512 in the fifth stage; 3×3×16, 3×3×16, and 3×3×512 in the fourth stage; 3×3×16, 3×3×16, and 3×3×256 in the third stage; 3×3×16 and 3×3×128 in the second stage; and 3×3×16 and 3×3×16 in the first stage.

[0032] From a theoretical analysis, neural networks have strong nonlinear fitting capabilities, but because of their strong fitting capabilities, the decoding network can also reconstruct anomalies. In the anomaly detection task, this is undoubtedly a manifestation of overfitting. Therefore, in order to avoid overfitting of the decoding network during the reconstruction process, the present invention reduces the number of convolution kernels in the decoding network to ensure that the network has good reconstruction characteristics while avoiding overfitting. The number of trainable parameters in the network before and after reducing the number of convolution kernels is 19710019 and 5836387 respectively. It can be seen that the number of parameters of the decoding network with an asymmetric structure is close to 1 / 4 of that of the symmetric structure. In addition, the embodiment reduces the number of convolutions in each stage, so in order to ensure the improvement of the reconstruction capability, a deconvolution network is selected for upsampling. Compared with the linear interpolation method, deconvolution is a learnable structure with better feature extraction capabilities.

[0033] (1) Design and construct an asymmetric autoencoder network based on knowledge distillation

[0034] Drawing on the anomaly detection method based on knowledge distillation, a reconstruction method based on knowledge distillation is proposed. The structure of the overall network is as follows: Figure 2 As shown in the figure. Since the knowledge distillation algorithm can transfer the knowledge of the teacher network with rich feature extraction capabilities to the student network and has accurate anomaly detection capabilities in high-dimensional space, the present invention sets the encoder as the teacher network and the decoding network as the student network. Unlike other knowledge distillation strategies in anomaly detection that use networks with the same structure as the teacher and student networks, the teacher and student networks are set as two opposite structures. The purpose of this is not only to cleverly transfer the powerful feature extraction capabilities of the teacher network to the student network, but also to enable the network to have the ability to reconstruct features.

[0035] In order to combine the characteristics of knowledge distillation and fully transfer the knowledge of the teacher network to the student network, Figure 2 The present invention not only measures the distance between the final original input and the labeled ground truth and the generated output and heatmap, but also adds multi-scale distance loss in high-dimensional feature space to achieve better knowledge distillation and feature reconstruction. Specifically, it can be summarized as follows:

[0036]

[0037] In the formula, the superscript l represents different stages. represents the mapping function of the teacher network at stage l, x represents the input of the teacher network at stage l, f S l represents the mapping function of the lth stage in the student network, y represents the output of the lth stage in the student network, L is the highest stage number in the network, Figure 2The total number of stages in this embodiment is 5, with l = {0, 1, 2, 3, 4} and L = 4. Compared to traditional autoencoder networks that only measure the difference between the original input and the final generated output, this invention calculates the difference using a multi-scale distance metric. This ensures that the student network has multi-scale reference information during the reconstruction process, improving the accuracy of anomaly detection. Furthermore, during the knowledge distillation process, the knowledge of the teacher network can be more completely and accurately transferred to the student network.

[0038] The implementation of anomaly detection is divided into two stages: training and testing. Figure 3 As shown in the figure, only normal samples are used for training during the training process. Since only normal samples are input during the training process, the encoding network as the teacher network can effectively capture the characteristics and high-dimensional distribution of normal samples. At the same time, the decoding network as the student network can accurately complete the reconstruction of normal samples. In the testing phase, since the network has never seen abnormal data, the abnormal parts will inevitably be missing during the reconstruction process of the decoding network, resulting in poor reconstruction of abnormal samples. Therefore, anomalies can be judged by comparing the difference between the original image and the reconstructed image with a preset threshold. If the difference is greater than the threshold, it is judged as abnormal, otherwise it is normal.

[0039] (2) Design of hierarchical multi-level loss function

[0040] The general autoencoder network only calculates the distance between the original input image and the output reconstructed image as the loss function. This type of loss function only calculates the difference between the input and output results, which ignores the difference between the feature maps and cannot guarantee the quality of the feature maps during the reconstruction process. Therefore, in order to improve the ability to reconstruct features and make full use of the feature extraction capability of the encoder, the present invention designs a hierarchical multi-level loss function, and designs pixel-level loss functions and global direction loss functions from the details and the global perspectives respectively. The pixel-level loss function calculates the mean square error between feature maps of different sizes at the corresponding stages of downsampling on the encoding network and upsampling on the decoding network, as shown in the following formula:

[0041]

[0042] Where l represents the loss function of the current calculation stage l, N l represents the total number of pixels in stage l, represents the mapping function of the teacher network at stage l, represents the mapping function of the student network at stage l, represents the pixel-by-pixel input of the teacher network at stage l, The pixel-by-pixel output of the student network in the first stage. In this network structure, the encoding network is divided into 5 stages and downsampled 5 times, and the encoding inputs of the 5 stages are obtained in sequence. The decoding outputs of the 5 stages are S4, S3, S2, S1, S0. The decoding network is also divided into 5 stages and upsampled 5 times, taking l = {0, 1, 2, 3, 4}, where l = 0 corresponds to the input of the encoder and the output of the decoding network, and the remaining 4 feature maps are the size of the original input image.

[0043] Through the above formula, the embodiment makes full use of the feature information of each scale of the encoding network, which not only ensures that the decoding network has a more accurate reference when upsampling at each stage, but also ensures the quality of the reconstructed features. In terms of knowledge distillation, the encoding network in the network acts as a teacher network, generating feature maps of different scales for the input at each stage, where the high-resolution feature map contains low-level detail information such as color, texture, and edge, while the low-resolution feature map contains more semantic information, and information of different scales jointly expresses complete feature information. It can be seen that the knowledge of the teacher network cannot be fully expressed by using only the loss error between one layer of feature maps, so by using multi-scale loss errors, the teacher network can guide the student network to learn through different feature map information, thereby more completely transferring knowledge to the student network.

[0044] Although the above-mentioned multi-scale pixel-level error loss function can utilize multi-level information to assist knowledge distillation and improve the quality of feature reconstruction at the same time, the calculation of pixel-level error only depends on the one-to-one correspondence between pixels. As an intensive prediction task, anomaly detection not only considers the feature information of the pixel itself, but also needs to consider similar feature information around the pixel. If a pixel in the image is identified as abnormal, the possibility of its surrounding pixels being abnormal is greater. Therefore, based on this feature, the present invention adds a multi-scale global direction loss function while considering the multi-scale pixel-level error loss function. The specific expression is as follows:

[0045]

[0046] Where vec(·) is a vectorization function that converts a matrix of any dimension into a one-dimensional vector, and ||·|| represents the modulus function of the vector. and Represent the mapping functions of the teacher network and the student network in the lth stage, x l is the input of the teacher network in stage l, y l is the output of the student network in stage l.

[0047] Because the vector angle calculation considers all information in the entire feature map, it expands the error range. While calculating pixel-level local errors, it also considers global information between feature maps, which, to a certain extent, expands the network's receptive field. This operation improves the utilization of both global and local information, ensuring that the student network learns more complete knowledge during the knowledge distillation process and improving the quality of the reconstructed feature maps.

[0048] According to the above pixel-level error loss function and the global direction loss function, the total loss error function is as follows (4), where α is the adjustment of L val With L dir Weight hyperparameter. Since both the pixel-level loss function and the global direction loss function are similarity loss functions, this hyperparameter is used to control and adjust the two loss functions to balance their effects. Experiments have found that different values of α can affect network performance, and different values of α should be selected for anomaly detection of different objects.

[0049] L total =L val +αL dir (4)

[0050] To demonstrate the effectiveness of the knowledge distillation-based asymmetric autoencoder anomaly detection algorithm proposed in this example, we tested our model on the publicly available anomaly detection dataset, MVTec AD, using the area under the receiver operating characteristic (ROC) curve (AUC) as the model evaluation metric. This dataset contains 15 different categories of items. We trained our model on each of these 15 categories and calculated the average AUC. Our method's test results surpass those of several cutting-edge methods. Table 1 compares our method with other cutting-edge methods on the MVTec AD dataset, using the area under the ROC curve (AUC) as the evaluation metric.

[0051]

[0052]

[0053] Table 1

[0054] The first three columns in Table 1 are all based on autoencoder network anomaly detection methods, while the fourth and fifth columns are based on GAN anomaly detection methods. These are all reconstruction-based anomaly detection methods. Columns 6 and 7 are feature embedding methods based on knowledge distillation, and the last column is the method proposed in this paper. As can be seen from the table, neither the autoencoder-based nor the GAN-based reconstruction methods ultimately achieved high accuracy for various types of anomalies in the anomaly detection task. This is primarily because the reconstruction-based methods also reconstruct the anomalies when outputting the generated graph, resulting in a small reconstruction error between the original and generated graphs, making it impossible to accurately identify anomalies based on the reconstruction error. The main idea of the feature embedding method based on knowledge distillation is to transfer the knowledge of the more complex teacher network to the lightweight student network. Due to the structural differences between the two, the expression of anomaly features in the hidden high-dimensional space differs, thereby enabling anomaly detection. The feature embedding methods based on knowledge distillation in the above table all achieved an average AUC of over 85%, far exceeding that of the reconstruction-based methods. The present invention combines reconstruction methods with knowledge distillation methods to propose an asymmetric network based on knowledge distillation. The proposed method significantly outperforms other anomaly detection methods based on reconstruction in the table above, both in terms of categories and average AUC. Compared with feature embedding anomaly detection methods based on knowledge distillation, the proposed method achieves higher AUC values in some categories, and its average AUC is approximately 2 points higher than the highest MKD network. This demonstrates the feasibility of applying knowledge distillation strategies to reconstruction methods and proves that the proposed asymmetric structure and multi-scale loss function can also significantly improve performance.

Claims

1. A method for anomaly detection based on an asymmetric autoencoder based on knowledge distillation, characterized in that: The following steps are involved: 1) During the training phase, only normal samples are input into the asymmetric autoencoder. The encoding network outputs the features of the normal samples to the decoding network, and the decoding network outputs the reconstructed normal samples. The asymmetric autoencoder consists of an encoding network and a decoding network with asymmetric structures. A loss function is used to constrain the training process so that the reconstructed normal samples output by the decoding network can restore the normal samples input to the asymmetric autoencoder to the greatest extent possible. During training, the encoding network is used as the teacher network and the decoding network as the student network. A knowledge distillation algorithm is used to transfer the knowledge of the teacher network to the student network. The encoding network consists of L feature extraction stages, each of which is composed of a stack of 3×3 convolution kernels. The output of each feature extraction stage is connected to the pooling module. The decoding network consists of L data reconstruction stages, each of which is composed of a stack of fewer 3×3 convolution kernels than the feature extraction stage of the same sequence, forming an asymmetric structure. The output of each data reconstruction stage is connected to the upsampling module. 2) In the detection phase, the image to be detected is input into the trained asymmetric autoencoder. The encoding network outputs the features of the image to be detected to the decoding network. The decoding network outputs the reconstructed image to be detected. The difference between the image to be detected and the reconstructed image to be detected is compared. If the difference is greater than a threshold, the image to be detected is judged to be abnormal; otherwise, the image to be detected is judged to be normal.

2. The method according to claim 1, wherein: L is 5. According to the data processing order, the five feature extraction stages in the encoding network are as follows: the convolution kernel of the first stage is 3×3×64, 3×3×64, the convolution kernel of the second stage is 3×3×128, 3×3×128, the convolution kernel of the third stage is 3×3×256, 3×3×256, 3×3×256, the convolution kernel of the fourth stage is 3×3×512, 3×3×512, 3×3×512, the convolution kernel of the fifth stage is 3×3×512, 3×3×512, 3×3×512; The five data reconstruction stages in the decoding network are as follows: the convolution kernels of the fifth stage are 3×3×16, 3×3×16, and 3×3×512, the convolution kernels of the fourth stage are 3×3×16, 3×3×16, and 3×3×512, the convolution kernels of the third stage are 3×3×16, 3×3×16, and 3×3×256, the convolution kernels of the second stage are 3×3×16 and 3×3×128, and the convolution kernels of the first stage are 3×3×16 and 3×3×16.

3. The method according to claim 1 or 2, wherein: The upsampling module is implemented using a deconvolutional network.

4. The method according to claim 1, wherein: Loss function L total Through the pixel-level error loss L val And the global direction loss L dir Composition: L total =L val +αL dir , α is the adjustment L val With L dir Hyperparameters of weights.

5. The method according to claim 4, wherein: Pixel-level error loss L val : Among them, l is the first stage, N l represents the total number of pixels in stage l, represents the mapping function of the teacher network at stage l, represents the mapping function of the student network at stage l, represents the pixel-by-pixel input of the teacher network at stage l, Pixel-wise output of the student network at stage l.

6. The method according to claim 4, wherein: Global directional loss L dir : Among them, l is the lth stage, vec(·) is a vectorization function that converts a matrix of any dimension into a one-dimensional vector, and ||·|| represents the modulus function of the vector. and Represent the mapping functions of the teacher network and the student network in the lth stage, x l is the input of the teacher network in stage l, y l is the output of the student network in stage l.

Images