Unsupervised anomaly detection method based on pre-training feature reconstruction

Abstract

The invention provides an unsupervised anomaly detection method based on pre-training feature reconstruction, and belongs to the field of industrial image anomaly detection. According to the method, a multi-level feature extractor is used for extracting multi-level features of an image by using a pre-trained deep neural network; the feature reconstruction network is used for enhancing the reconstruction capability by introducing a global feature extraction module; a self-supervised learning strategy: disrupting feature layout to avoid local overfitting; calculating the similarity between the reconstructed features and the input features to obtain an abnormal score graph; according to the invention, reconstruction is carried out in a feature space, and the model reconstruction capability is enhanced by extracting global features and a self-supervised learning strategy, so that the problem of low precision of a traditional pixel space reconstruction method in anomaly detection application can be overcome.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and industrial inspection, and particularly relates to an unsupervised anomaly detection method based on deep learning, which is applicable to the product defect detection scenario in industrial manufacturing; Background Art

[0002] Image anomaly detection is a technology for identifying objects or regions that do not conform to normal conditions or expectations; in real-world scenarios, abnormal data is often a sign of potential hazards, so it is very important to detect abnormal data in a timely manner; it is widely used in the field of industrial vision inspection and is an important topic in both academia and industry; in industrial production, these abnormal situations may lead to problems such as decreased product quality and low production efficiency; anomaly detection helps to solve these problems; however, abnormal samples are very scarce, and new types of defects may continuously occur in actual production, so it is difficult to meet the training requirements in terms of the quantity and variety of abnormal samples; in these cases, due to the lack of target class samples, traditional object detection and image segmentation methods are no longer applicable; therefore, unsupervised anomaly detection based on unsupervised learning, that is, only using normal samples to train the model, is of great practical significance;

[0003] In recent years, deep learning has been widely applied to image anomaly detection tasks due to the advantages of its algorithms such as convenience and versatility; among them, the method based on image reconstruction has a simple and effective principle and is an important branch in unsupervised anomaly detection; this type of method is based on a core assumption: the reconstruction network only learns the features of normal samples, so it cannot accurately reconstruct abnormal regions; the pixel-by-pixel difference between the original image and the reconstructed image is used to estimate the degree of anomaly; currently, there have been many different studies on the improvement of the reconstruction method; some methods use generative models to enhance the performance of the reconstruction network, including VAE and GAN; there are also some methods that attempt to improve the network structure to enhance the network's reconstruction ability, such as adding skip connections in the network and performing reconstruction at multiple scales; however, previous image reconstruction techniques usually operate in the pixel space, and the pixel-by-pixel mean square error is difficult to fully capture the true difference between the reconstructed image and the original image; moreover, previous reconstruction models often fail to fully model the global features and the overall structure of the samples, resulting in the model relying on local patterns for reconstruction; Summary of the Invention

[0004] In view of the above deficiencies existing in the prior art, the present invention provides an unsupervised anomaly detection method based on pre-trained feature reconstruction;

[0005] In an embodiment according to the present invention, the anomaly detection method includes a multi-level feature extractor, a feature reconstruction network, a self-supervised learning strategy, and an anomaly map acquisition method, and specifically includes the following steps:

[0006] The image is fed into a multi-level feature extractor to extract multi-level features as the fused feature map z i ;

[0007] The fused feature map z i is fed into a reconstruction network for reconstruction;

[0008] Meanwhile, the fused feature map z i is shuffled and reorganized into z i ′ according to the grid partition regions, and z i ′ is also fed into the reconstruction network for reconstruction;

[0009] The input of the multi-level feature extractor is any training sample wherein, x train represents a set of a series of training samples, N represents the total number of training samples; the input image x i is fed into a pre-trained model to obtain multi-level features, φ represents the network pre-trained on a large-scale dataset, and L represents the set of selected feature layers; for each layer l ∈ L, φ l (x i ) represents the output feature of the l-th layer obtained by the sample passing through the network; each selected feature map is adjusted to the same target resolution by bilinear interpolation, which is expressed as:

[0010]

[0011] wherein, represents the l-th layer feature map with adjusted resolution, C l represents the number of channels of the feature map φ l (x i ), H and W are the width and height of the feature map with the largest resolution in the multi-level feature maps {φ l (x i )|l ∈ L}; use to represent the C -dimensional vector at the position (h, w) on the feature map l , and adjust the number of channels of each vector by one-dimensional adaptive average pooling, which is expressed as:

[0012]

[0013] For all selected feature layers L, concatenate the multi-level features in the channel dimension to obtain the concatenated features, which is expressed as:

[0014]

[0015] For the feature map The feature vector at each pixel position is again applied with one-dimensional adaptive average pooling to adjust the channel dimension, obtaining a fused feature map

[0016] The feature reconstruction network includes an encoder network and a decoder network, and there are skip connections between the encoder network and the decoder network;

[0017] The encoder network includes three encoder modules, and each encoder module sequentially includes three bottleneck layers and a global feature extraction layer;

[0018] The bottleneck layer includes three convolutional layers, namely a 1×1 convolution for dimensionality reduction, a 3×3 convolution for feature extraction, and a 1×1 convolution for restoring the dimension; the stride of the 3×3 convolution in the first bottleneck layer among the three bottleneck layers included in the encoder network is 2, and the strides of other convolutions are 1;

[0019] The feature extraction process of the global feature extraction module is as follows:

[0020] The feature map input to the global feature extraction module is denoted as It is evenly divided into K×K grid regions, and the features of each region can be represented as Using to represent the total number of feature vectors included in each region, so the features of each region can be represented as:

[0021] t k ={t k，p |p∈{1, 2,..., P}}

[0022] where, represents each feature vector within each region; for all region features t k calculate the average value of its feature vectors to obtain the region representation feature, denoted as:

[0023]

[0024] For each region representation feature r k , compress it through a fully connected layer to obtain a more compact representation, denoted as:

[0025]

[0026] Flatten the feature map composed of all K×K vectors in the spatial dimension to obtain a global feature vector Let so that the channel dimension of the global feature vector g Finally, the global feature vector g is in H t ×Wt Repeat on the dimension and concatenate the obtained feature map with the feature map t;

[0027] The decoder network includes three decoder modules, and each decoder module sequentially includes one deconvolution bottleneck layer and two bottleneck layers;

[0028] The deconvolution bottleneck layer includes three convolutional layers, namely a 1×1 convolution for dimensionality reduction, a 2×2 deconvolution for upsampling, and a 1×1 convolution for restoring the dimension; the stride of the 2×2 deconvolution in the first bottleneck layer among the three bottleneck layers included in the encoder network is 2, and the strides of other convolutions are 1;

[0029] The self-supervised learning strategy, its process includes:

[0030] Take the fused feature map z i Divide it into K×K grid regions in the plane dimension; for each sub-region after division Arrange them in sequence to obtain an ordered feature list, denoted as: S = [z i,j

[0031] where j ∈ {1, 2,..., K×K}; label the sequence of the list S to obtain an index sequence; randomly shuffle the index sequence to generate a new index sequence; according to the new index, rearrange the feature sub-regions to obtain a list S′ with shuffled order; according to the order of the list S′, reconstruct the shuffled feature map, denoted as:

[0032] Calculate the cosine distance loss function between each feature vector of the input and output of the reconstruction network, denoted as:

[0033]

[0034] Calculate the cosine distance for the feature vectors at all pixel positions on the feature map and take the average to obtain the reconstruction loss, denoted as:

[0035]

[0036] Send the reorganized feature z i ′ and the original feature z i into the reconstruction network at the same time, and for the reconstruction of the reorganized feature z i ′ Calculate its cosine distance loss from the original feature z i to obtain the self-supervised learning loss, denoted as:

[0037]

[0038] ​The overall loss function of this paper is as follows:

[0039]

[0040] Among them, λ is the weight used to balance the influence degree of the reconstruction tasks of the two features z i and z i ' on the model;

[0041] The inference process of the method includes:

[0042] The test sample y i is sent into the multi-level feature extractor to obtain the fused feature z i ; The fused feature z i is input into the reconstruction network to obtain the reconstructed feature Calculate the cosine distance between z i and for each pixel to obtain the original anomaly score map; In order to obtain an accurate localization image, the anomaly score map is upsampled to the resolution of the sample image y i by bilinear interpolation, and then the image is smoothed using a Gaussian filter to obtain the anomaly score map A pix ;

[0043] Compared with the prior art of the present invention, the following beneficial effects are achieved:

[0044] The present invention proposes an anomaly detection method based on pre-trained feature reconstruction, which uses a pre-trained model as a feature extractor to put the image into the feature space and directly performs reconstruction in the feature space; a region of the image will be extracted as a high-dimensional feature vector, and whether there is an anomaly in the corresponding region of the image can be identified by comparing the feature vectors; Secondly, previous reconstruction models often fail to fully model the global features and the overall structure of the samples, resulting in the model relying on local patterns for reconstruction; The present invention adds a global feature extraction module to the reconstruction network design to enhance the network's capture of global features; In addition, the present invention proposes a self-supervised learning strategy of "feature recombination" to promote the model to learn the overall structure of the samples. By shuffling the feature map according to the grid regions and then sending it into the reconstruction network, it avoids the model simply relying on local features for reconstruction, thereby enhancing the model's perception of the overall samples; BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic diagram of the model structure, training and test process of the present invention;

[0046] Figure 2 is Figure 1 a schematic diagram of the feature extractor in the example;

[0047] Figure 3 is Figure 1 a schematic diagram of the feature reconstruction network in the example;

[0048] Figure 4 For Figure 1 Schematic diagram of the self-supervised learning strategy in the example: feature recombination in the instance;

[0049] Figure 5 Visualization schematic diagram of the anomaly detection results of some sample images in the example of the present invention; Detailed implementation manners

[0050] In the following, exemplary embodiments of the present application will be described in conjunction with the accompanying drawings; for the sake of clarity and conciseness, all features of the actual embodiments are not described in the specification; however, it should be understood that many embodiment-specific decisions can be made during the development of any such actual embodiment to achieve the specific goals of the developer, and these decisions may vary with different embodiments;

[0051] Here, it should also be noted that, in order to avoid obscuring the present application with unnecessary details, only the device structures closely related to the solution of the present application are shown in the drawings, and other details less related to the present application are omitted;

[0052] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the drawings; in this document, where feasible, embodiments can be combined with each other, features can be replaced or borrowed between different embodiments, and one or more features can be omitted in one embodiment;

[0053] The embodiments of the present application provide an image anomaly detection and localization method, Figure 1 Shows a flowchart of the image anomaly detection and localization method according to an embodiment of the present application; see Figure 1 , the method flow of the present invention includes:

[0054] Training stage: The training images are input into a multi-level feature extractor to obtain a fused feature map, and the fused feature map is input into a feature reconstruction network for reconstruction; synchronously, the fused feature map is recombined according to the grid region shuffling, and the recombined feature map is input into the feature reconstruction network for reconstruction; the cosine distances are calculated respectively for the reconstruction of the fused feature map and the reconstruction of the recombined feature map to obtain an overall loss function, and the feature reconstruction network is optimized according to the overall loss function;

[0055] Testing stage: The image to be detected is input into a multi-level feature extractor to obtain a fused feature map, and the fused feature map is input into the reconstruction network to obtain a reconstructed feature map; the cosine distance at each pixel position is calculated between the fused feature map and the reconstructed feature map to obtain an anomaly score map, and then the resolution is adjusted to the resolution of the image to be detected by bilinear interpolation; finally, the anomaly score map is smoothed using a Gaussian filter to obtain the final anomaly score map;

[0056] In this embodiment, the training and test images are from the MvTec AD (MVTec Anomaly Detection) dataset; MvTec-AD is a benchmark dataset for industrial vision anomaly detection, aiming to improve and evaluate the performance of anomaly detection algorithms; it contains a total of 15 categories, including 5 texture categories and 10 object categories, with a total of 3,629 pictures, covering a variety of common industrial objects; its training set is normal sample images from the real world, showing the normal state of products in this category; each category in the test set contains normal samples and abnormal samples with specific defects, such as scratches, holes or deformations, etc., and each abnormal sample image has detailed pixel-level annotations; in this embodiment, the training and test images are scaled and cropped to a resolution of 256×256 before being input into the model.

[0057] In this embodiment, WideResnet50-2 pre-trained on ImageNet is used as the feature extraction network φ, and more general intermediate layer features are selected for the feature layer, that is, l∈L = {2, 3}, and the number of channels of the multi-level features output by the feature extractor is c = 512. Figure 2 Shows a schematic diagram of feature extraction in this example. The input image is fused by the second and third layer feature maps extracted by the pre-trained WideResnet50-2 through the steps described in the present invention to obtain a fused feature map.

[0058] Figure 3 Shows the structure of the feature reconstruction network, including an encoder network and a decoder network; the encoder network includes a first encoder module, a second encoder module, and a third encoder module; the decoder network includes a first decoder module, a second decoder module, and a third decoder module; the feature map input to the feature reconstruction network first enters the encoder network, and the first encoded feature is extracted through the first encoder module; the first encoded feature is input to the second encoder module to extract the second encoded feature; the second encoded feature is input to the third encoder to extract the third encoded feature; the three encoded features are respectively input to different layer decoder modules of the decoder network, and the third encoded feature is input to the first decoder module to obtain the first decoded feature; the first decoded feature and the second encoded feature are concatenated in the channel dimension, fused through a 1*1 convolutional layer, and then input to the second decoder module to obtain the second decoded feature; the second decoded feature and the first encoded feature are concatenated in the channel dimension, fused through a 1*1 convolutional layer, and then input to the third decoder module to obtain the reconstructed feature. Figure 3 Also shows the structure of each encoder module and decoder module, as well as the specific structure of the bottleneck layer in each module. Figure 3 The specific process of the global feature extraction module is shown in the lower left corner. The grid division of the global feature extraction module in this example is set to K = 4.

[0059] Figure 4The self-supervised learning strategy in this example is shown. In the self-supervised strategy of this example, the division of the feature map grid is set to K = 4; thus, the entire feature map will be divided into 16 regions of 4×4; each of the divided sub-regions is arranged in sequence to form an ordered feature list, denoted as: S = [z i，j ; where j ∈ {1, 2,..., K×K}; the sequence of list S is marked to obtain an index sequence {1, 2, 3,..., 12, 13, 14, 15}; the order of this index sequence is randomly shuffled to generate a new index sequence; as Figure 4 shown, assume the new index is [4, 8, 2, 13, 5, 10, 1, 16, 3, 9, 7, 15, 12, 6, 14, 11]; according to this rearranged index, the sub-regions are rearranged, thus obtaining a shuffled list S'; according to the order of list S', the shuffled feature map can be reconstructed:

[0060] In this embodiment, the hyperparameter λ of the training loss function is 0.1, and the Adam optimizer with β = (0.5, 0.999) is used to train the feature reconstruction network, and the learning rate is set to 0.001; we set the batch size to 16 and trained for 100 epochs on an RTX4090 GPU; in the test phase, the parameter σ of the Gaussian filter in this embodiment is set to 4;

[0061] In this example, the evaluation metric is the area under the receiver operating characteristic curve (ROC), i.e., AUROC; the calculation of AUROC involves two basic metrics, the false positive rate (FPR) and the true positive rate (TPR). The true positive rate represents the proportion of positive samples that are correctly classified as positive classes, also known as the recall rate:

[0062]

[0063] The false positive rate, also known as the false alarm rate, represents the proportion of negative class samples that are misclassified as positive classes, and the calculation formula is:

[0064]

[0065] The ROC space coordinates define the false positive rate (FPR) as the horizontal axis and the true positive rate (TPR) as the vertical axis. By calculating the TPR and FPR at different classification thresholds of the model, an ROC curve can be drawn, and the area under this curve is denoted as AUROC, whose value ranges from 0 to 1. In this example, the pixel-level AUROC is calculated through the pixel-level anomaly score map of each sample;

[0066] The experimental results of this example are compared with other methods as shown in the following table:

[0067]

[0068] Figure 5 The visualization schematic diagram of the anomaly detection results of some samples in this example is shown, respectively showing large-area defects or structural anomalies and tiny defects; it can be seen that the method of the present invention can produce very good localization effects for various types of anomalies; especially for structural anomalies, the proposed model has particularly good localization performance; such as Figure 5 For the sample with the anomaly of the missing triode in the lower row of the 7th column in, the method of the present invention can completely detect the missing triode.

Claims

1. An unsupervised anomaly detection method based on pre-trained feature reconstruction, characterized in that: include: Multi-level feature extraction; Input the input image into a pre-trained deep neural network to obtain feature maps at multiple levels; Fuse multi-level feature maps to generate a fused feature map; Feature reconstruction network; It includes an encoder network and a decoder network. The encoder network gradually extracts the latent space encoding of the fusion feature map through a multi-layer encoder module, and then gradually reconstructs the features through a multi-layer decoder module. Self-supervised learning strategy; based on the original features output by the multi-level feature extractor, generate features that are shuffled and reorganized by grid area; based on the original feature reconstruction results and the reorganized feature reconstruction results output by the reconstruction network, calculate the original feature cosine distance loss and the reorganized feature cosine distance loss respectively, and jointly optimize the total loss; Abnormal map acquisition method: during inference, the input image is input into a multi-level feature extractor to obtain a fused feature map, the fused feature map is input into a feature reconstruction network to obtain a reconstructed feature map, and the original abnormal map is obtained according to the cosine distance of each pixel of the fused feature map and the reconstructed feature map; the original abnormal map is bilinearly interpolated and upsampled to the input image resolution, and then smoothed by a Gaussian filter to obtain the final abnormal map.

2. The method according to claim 1, characterized in that The multi-level feature extraction step comprises: The input image is sent to the pre-trained network, the intermediate layer output is selected from the pre-trained network, and a multi-level feature map set is extracted, which is expressed as: [f l (x i )], l∈L; For each feature map φ in the feature map set l (x i ) to perform bilinear interpolation and adjust the resolution to be consistent, expressed as: For each feature map Each feature vector in performs one-dimensional adaptive average pooling to adjust the channel dimension, expressed as: For each feature map Stitching in channel dimension Get a feature map, expressed as: For feature maps The feature vector at each pixel position is applied with one-dimensional adaptive average pooling to adjust the channel dimension to obtain a fused feature map, which is expressed as: Among them, φ represents the pre-trained network, φ l represents the lth layer of the pre-trained network, L represents the set of selected pre-trained network layers, x i represents the i-th input image, represents the resolution-adjusted feature map of the lth layer, C l Represents the feature map φ l (x i ), H and W are the feature map sets {φ l (x i )|l∈L}, Representation feature map C at the upper position (h, w) l dimensional vector, and C represents the number of channel dimensions after adjustment.

3. The method according to claim 1, characterized in that The feature reconstruction network includes: an encoder network and a decoder network; The encoder network includes a first encoder module, a second encoder module, and a third encoder module; the decoder network includes a first decoder module, a second decoder module, and a third decoder module; The fused feature map is input into the encoder network, and the first encoding feature is extracted through the first encoder module; the first encoding feature is input into the second encoder module, and the second encoding feature is extracted; The second coding feature is input into a third encoder to extract a third coding feature; The three encoding features are respectively input into the decoder modules of different layers of the decoder network, and the third encoding feature is input into the first decoder module to obtain the first decoding feature; The first decoding feature and the second encoding feature are concatenated in the channel dimension, fused through a 1*1 convolutional layer, and then input into the second decoder module to obtain the second decoding feature; The second decoding feature and the first encoding feature are concatenated in the channel dimension, fused through a 1*1 convolutional layer, and then input into the third decoder module to obtain a reconstructed feature.

4. The encoder module according to claim 3, characterized in that: The encoding process includes: the input passes through three bottleneck layers connected in sequence, then passes through a global feature extraction layer, the bottleneck layer output and the global feature extraction layer output are spliced ​​in the channel dimension, and then fused through a convolution layer; The bottleneck layer includes three convolution activation layers connected in sequence, wherein the convolution kernels are 1, 3 and 1 respectively; The convolution activation layers respectively include sequentially connected convolution layers, normalization and activation functions; The global feature extraction layer comprises the steps of: The input features are divided into K×K grid areas, and the features of each area are expressed as: t k ={t k，p |p∈{1,2,...,P}} For any region feature t k , calculate the average value of its feature vector, and get the regional representation feature, which is expressed as: For each region, the feature r k , which is compressed through the fully connected layer and expressed as: All K×K vectors The constructed feature map is flattened in the plane dimension to obtain the global feature vector g The global eigenvector g is placed in H t ×W t Repeat in the dimension to obtain global features; Where k represents the kth region, P represents the total number of feature vectors contained in each region, p represents the pth feature vector contained in each region, and t k，p Represents each feature vector in each region.

5. The decoder module according to claim 3, characterized in that The decoding process includes: a deconvolution bottleneck layer and two bottleneck layers connected in sequence; The deconvolution bottleneck layer includes a convolution activation layer, a deconvolution activation layer and a convolution activation layer connected in sequence, wherein the convolution kernels are 1, 2 and 1 respectively; The bottleneck layer includes three convolution activation layers connected in sequence, wherein the convolution kernels are 1, 3 and 1 respectively; The deconvolution activation layers respectively include sequentially connected deconvolution layers, normalization and activation functions; The convolution activation layers respectively include sequentially connected convolution layers, normalization and activation functions.

6. The method according to claim 1, characterized in that The self-supervised learning strategy includes the following steps: The fused feature map z i Divided into K×K grid regions, each region is represented by z i，j ; Arrange each of the divided sub-regions in sequence to form an ordered feature list S = [z i,j ], where j∈{1, 2, …, K×K}; Marking the order sequence of the list S to obtain an index sequence, and randomly disrupting the order of the index sequence to generate a new index sequence; Rearranging the feature list according to the new index to obtain a feature list S' with a disrupted order; According to the order of the list S', the reorganized feature map z can be obtained i '; Wherein, i represents the i-th input, and j represents the j-th grid area; The fused feature map z i Send it to the reconstruction network to get the reconstructed feature map The fused feature map z i Reconstruct the feature map with itself The cosine distance is calculated as the reconstruction loss function, which is expressed as: The recombined feature map z i 'Send it into the reconstruction network to get the reconstructed feature map The fused feature map z i Reconstruction of the reorganized feature map The cosine distance is calculated as the self-supervised learning loss function, which is expressed as: The reconstruction loss function and the self-supervised learning loss function are combined as the overall loss function of the reconstruction network, which is expressed as: Among them, i represents the i-th input, h, w represent the pixel (h, w), H and W represent the height and width of the feature map respectively, and λ represents the balance coefficient.

7. The method according to claim 1, characterized in that The abnormal graph acquisition method process includes: Input the test sample image into the pre-trained network to generate a fusion feature map; The reconstruction network generates a reconstructed feature map, and the cosine distance between each pixel is calculated to generate the original anomaly score map; The original anomaly score map is upsampled to the original image size by bilinear interpolation, and is smoothed by Gaussian filtering to output the final anomaly score map.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the industrial image anomaly detection and positioning method according to any one of claims 1 to 7 is implemented.

9. A computer program product, characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, implements the industrial image anomaly detection and positioning method according to any one of claims 1 to 7.

Images