Image anomaly detection method based on self-supervised learning and feature noise reduction

Abstract

The invention discloses an image anomaly detection method based on self-supervised learning and feature noise reduction, and belongs to the technical field of image anomaly detection. Comprising the steps of depth feature extraction, depth feature fusion, depth feature reconstruction and anomaly segmentation. According to the method, the advantages of the multi-scale features of the deep learning model are fully utilized, and the multi-scale features are effectively fused. Meanwhile, after the multi-scale features are fused, the multi-scale features are reconstructed to realize noise reduction operation. And a segmentation head is added at the last of the model to improve the model performance. Compared with an existing method, the model achieves a better anomaly detection effect with the same-level parameter quantity and training cost.

Description

Technical Field

[0001] The present invention relates to the technical field of image anomaly detection, and in particular to an image anomaly detection method based on self-supervised learning and feature denoising. Background Art

[0002] With the rapid development of computer vision and image processing technology, image anomaly detection has been widely used in daily life and industrial applications, such as defect inspection in industrial environments and identification of lesions in medical image analysis. As an important branch of machine learning, self-supervised learning has unique advantages for anomaly detection problems, such as no need for expensive labeling, no need to collect a large number of negative samples, and better generalization ability of the model.

[0003] Previously, anomaly detection was often performed by template matching, statistical methods, frequency domain analysis, etc., but in recent years, anomaly detection methods based on deep learning have become a research hotspot in this field. Feature embedding-based methods and reconstruction-based methods are two important types of anomaly detection methods based on deep learning. Among them, feature embedding-based methods use pre-trained models to extract deep features. Reconstruction-based methods often use structures such as generative adversarial networks or autoencoders. For example, the method DeSTSeg described in the reference "ZHANG X, LI S, LI X, et al. DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detection [J]. 2022." combines the characteristics of feature embedding-based methods and reconstruction-based methods, proposes the idea of ​​denoising model, believes that anomalies are represented as noise at the feature level, and guides the model to learn the denoising process through a teacher-student network. However, its student network directly encodes and decodes at the image level to achieve denoising, and does not fully utilize the powerful representation ability of deep features. Therefore, this method performs poorly on some subtle anomalies. Summary of the invention

[0004] Purpose of the invention: The present invention provides an image anomaly detection method based on self-supervised learning and feature denoising, in order to overcome the deficiencies in the prior art and improve the image anomaly detection effect.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: an image anomaly detection method based on self-supervised learning and feature denoising, comprising the following steps:

[0007] Step 1: After collecting normal images, preprocess them to generate a training set. After collecting normal images and abnormal images in a certain ratio, preprocess them to generate a test set, and generate artificial abnormalities for the images in the training set.

[0008] Step 2: The abnormal image and the normal image are sent to the image anomaly detection model based on self-supervised learning and feature denoising. The feature generation network will generate deep features of the normal image and the abnormal image respectively, and then the deep features of the abnormal image will be fused.

[0009] The structure of the image anomaly detection model based on self-supervised learning and feature denoising is divided into a feature generation network, a deep feature denoising network and a segmentation network.

[0010] The feature generation network adopts the ResNet18 model with pre-trained weights, which is a deep convolutional neural network model that solves the problem of gradient disappearance through residual connections; the ResNet18 model contains 18 layers, which are divided into five residual blocks and the final global average pooling layer. The ReLU activation function is used after each convolution layer in the residual block; the ResNet18 model uses the network weights obtained by training on the ImageNet dataset, but deletes the last residual block of the original ResNet18 model: conv_5x and the final global average pooling.

[0011] Step 3: Send the deep features of the abnormal image after feature fusion to the deep feature denoising network. At this time, start training the deep feature denoising network. After the training is completed, save the network model parameters of the deep feature denoising network.

[0012] The deep feature denoising network includes an encoder structure and a decoder structure. The encoder structure is a ResNet18 model with the first residual block: conv_1x and the last average pooling layer deleted. The decoder structure is an inverse ResNet18 model that replaces all downsampling with bilinear upsampling. The parameters of the encoder and decoder are randomly initialized. During training, the cosine distance between the outputs of the corresponding blocks of the feature extraction network and the deep feature denoising network is minimized.

[0013] Step 4: Start training the segmentation network. At this time, only the abnormal images are sent to the image anomaly detection model based on self-supervised learning and feature denoising. After the training is completed, save the network model parameters of the segmentation network.

[0014] The structure of the segmentation network is as follows: the similarity between the feature layers of the corresponding blocks of the decoders of the feature generation network and the deep feature denoising network is calculated, and the calculation result is upsampled to a quarter of the input image size, and then it is processed in sequence through two residual blocks and a void space pyramid pooling module to obtain the output result of the final segmentation network model; during training, the sum of the focus loss and the absolute error loss L1 between the output of the segmentation network model and the binary mask of the artificial anomaly is minimized;

[0015] Step 5. Load the model parameters of the deep feature denoising network saved in step 3 and the model parameters of the segmentation network saved in step 4 into the deep feature denoising network and the segmentation network respectively. Then put the image to be detected into the image anomaly detection model based on self-supervised learning and feature denoising for detection. Finally, obtain the image-level anomaly detection results and pixel-level anomaly detection results output by the model.

[0016] Furthermore, the specific process of the preprocessing of normal images and the artificial anomaly generation operation in step 1 is as follows:

[0017] Step 1.1, the normal images used for training are uniformly truncated into the same size and put into a training set folder, and the normal images and abnormal images are put into the test set folder, and the normal images and abnormal images are placed separately;

[0018] Step 1.2: Generate artificial anomalies for the images in the training set. Obtain mask M by binarizing with random two-dimensional Perlin noise and a preset threshold. n The abnormal image I is generated by replacing the mask region with a linear combination of any image from the external data source A a :

[0019] I a =β(MΘA)+(1-β)(MΘI n )+(1-M)ΘI n ;

[0020] β represents the opacity factor;

[0021] Furthermore, the specific process in step 2 is as follows:

[0022] Step 2.1. Load the pre-trained ResNet18 model weights of the feature generation network.

[0023] Step 2.2: Normal image I n Send it to the feature generation network to obtain the multi-level deep features of the normal image

[0024] Step 2.3: Abnormal image Ia Send it to the feature generation network to obtain multi-level deep features of abnormal images And set a learnable weight parameter w1, w2, w3, and a bias term ε respectively, and then perform the following feature aggregation operation on the multi-level deep features of the abnormal image to obtain the aggregated deep feature T a .

[0025]

[0026] Furthermore, the specific process in step 3 is as follows:

[0027] Step 3.1. Randomly initialize the encoder and decoder model weights in the deep feature denoising network.

[0028] Step 3.2: The deep features T after the feature aggregation operation on the multi-level deep features of the abnormal image a Send it to the deep feature denoising network to minimize The cosine distance between the output of the corresponding block of the deep feature denoising network and the output of the deep feature denoising network. Set the training round n1 of the deep feature denoising network and the training round n2 of the segmentation network. When the training round is less than or equal to n1, freeze the segmentation network model parameters and only train the deep feature denoising network. When the training round is greater than n1 and less than n2, proceed to step 4.

[0029] Furthermore, the specific process in step 4 is as follows:

[0030] Step 4.1: Randomly initialize the model weights of the segmentation network and freeze the deep feature denoising network model parameters.

[0031] Step 4.2: Input only abnormal image I to the feature generation module a , get multi-level depth anomaly And the aggregated deep features T a , T a The deep feature denoising network is fed into the deep feature denoising network. The similarity between the feature layers of the corresponding blocks of the decoder of the feature generation network and the deep feature denoising network is calculated and fed into the segmentation network for training.

[0032] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image anomaly detection method based on self-supervised learning and feature denoising is implemented.

[0033] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the image anomaly detection method based on self-supervised learning and feature denoising.

[0034] The image anomaly detection method based on self-supervised learning and feature denoising of the present invention has the following advantages:

[0035] The technical means of fusion of multi-level deep features and then denoising of deep features is adopted, and a segmentation network is added to the network, which produces the following technical effects:

[0036] 1. Use the pre-trained ResNet18 model to extract the deep features of the image, which improves the generalization ability of the model and the effect of image anomaly detection.

[0037] 2. The present invention uses deep feature denoising and feature comparison to perform anomaly detection, which avoids the consumption of a large amount of storage space compared to the feature embedding-based method.

[0038] 3. The present invention uses artificial anomalies to generate pseudo anomalies and trains the model in a self-supervised manner. On the one hand, it solves the problem of requiring a large amount of manpower for labeling, and on the other hand, it can also adjust the distribution of generated pseudo anomalies according to different categories, making the model more targeted.

[0039] 4. In the present invention, a segmentation network head is further added after the feature denoising network, thereby further significantly improving the model anomaly detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of the present invention;

[0042] Figure 3 This is the test result of the present invention on the metal_nut category of the MVTecAD dataset;

[0043] Figure 4 It is a cosine error curve diagram of the deep feature denoising network of the present invention on the training set;

[0044] Figure 5 This is a curve diagram of the total error of the segmentation network of the present invention on the training set. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific implementation examples.

[0046] Embodiment 1: An image anomaly detection method based on self-supervised learning and feature denoising, Figure 1 This is a training flowchart for an image anomaly detection method based on self-supervised learning and feature denoising; the details are as follows:

[0047] Step 1: Collect normal images and perform preprocessing to generate a training set. Collect normal images and abnormal images in a certain ratio and perform preprocessing to generate a test set. Generate artificial anomalies for the images in the training set.

[0048] The training set is constructed with positive samples of one category in the MVTecAD dataset, and the training set images are scaled to a size of 224×224.

[0049] Step 2: The abnormal image and the normal image are sent to the image anomaly detection model based on self-supervised learning and feature denoising. The feature generation network will generate deep features of the normal image and the abnormal image respectively, and then the deep features of the abnormal image will be fused.

[0050] The model structure of the image anomaly detection method based on self-supervised learning and feature denoising is divided into a feature generation network, a deep feature denoising network and a segmentation network.

[0051] The feature generation network uses the ResNet18 model with pre-trained weights, which is a deep convolutional neural network model that solves the gradient vanishing problem through residual connections; the ResNet18 model contains 18 layers, which are divided into five residual blocks and the final global average pooling layer. The ReLU activation function is used after each convolution layer in the residual block; the ResNet18 model uses the network weights obtained by training on the ImageNet dataset, but deletes the last residual block of the original ResNet18 model: conv_5x and the final global average pooling. The network structure of ResNet8 is shown in the following table:

[0052] Table 1 is the ResNet18 network structure

[0053]

[0054] Table 2 is an explanation of the terms in Table 1

[0055]

[0056] Step 3: Send the deep features of the abnormal image after feature fusion to the deep feature denoising network. At this time, start training the deep feature denoising network. After the training is completed, save the network model parameters of the deep feature denoising network.

[0057] The deep feature denoising network includes an encoder structure and a decoder structure, wherein the encoder structure is a ResNet18 model with the first residual block: conv_1x and the last average pooling layer deleted, and the decoder structure is an inverse ResNet18 model with all downsampling replaced by bilinear upsampling, wherein the parameters of the encoder and the decoder are randomly initialized; during training, the cosine distance between the outputs of the corresponding blocks of the feature extraction network and the deep feature denoising network is minimized.

[0058] After the training is completed, save the parameters of the trained deep feature denoising network to the specified folder.

[0059] The GPU (graphics processing unit) used for training is Tesla T4.

[0060] When training the deep feature denoising network, the weights of the deep feature denoising network and the segmentation network are randomly initialized, the abnormal image is placed in the model, and the deep feature denoising network training round n1 and the segmentation network training round n2 are set to start training the deep feature denoising network. When the training round is less than or equal to n1, the segmentation network model parameters are frozen and only the deep feature denoising network is trained; when the training round is greater than n1 and less than n2, step 4 is performed. The initialization values ​​of the model parameters are shown in the following table.

[0061] Table 3 Parameter initialization values

[0062]

[0063] The output feature layers of conv2_x, conv3_x, and conv4_x in the ResNet18 model in the feature generation network are represented as T 1 , T 2 , T 3 , where the value at any point is represented by T k (i, j) (k = 1, 2, 3), the output feature layer of the corresponding reverse part in the deep feature denoising network decoder is represented as S 1 , S 2 , S 3 , where the value at any point is represented by S k (i, j) (k = 1, 2, 3), the similar feature maps between the feature layers of the corresponding blocks of the feature generation network and the deep feature denoising network decoder are represented as X1, X2, and X3, respectively, where the value at any point is X k (i,j)(k=1,2,3),C k , H k , W k Indicates the corresponding S kThe number of channels, length and width of the layer feature map, Θ represents the Hadamard product, then the cosine loss value D at any point k (i, j) and the total cosine loss value L cos It can be calculated by the following formula:

[0064]

[0065]

[0066]

[0067] Next, (T 1 , S 1 ), (T 2 , S 2 ), (T 3 , S 3 ) are calculated by the above formula to obtain X1, X2, and X3, and then X2 and X3 are upsampled to the size of X1, and the concatenated feature map is represented as Will Input into the segmentation network model to segment out abnormal areas in the image.

[0068] Step 4: Train the segmentation network model

[0069] The structure of the segmentation network model is two residual blocks followed by a dilated spatial pyramid pooling module. The first residual block has 448 input channels and 256 output channels, and is composed of two 3×3 convolutional layers with a stride of 1. The second residual block has 256 input and output channels, and is also composed of two 3×3 convolutional layers with a stride of 1.

[0070] Next, the abnormal image I a Feed it into the network, freeze the parameters of the deep feature denoising network, and only train the segmentation network. During training, minimize the sum of the focus loss and L1 loss between the network output and the binary mask of the artificial anomaly.

[0071] use Represents the probability map output by the segmentation network model, M ij is the value of any point of the mask M, for The value of any point, p ij The similarity between the probability map output by the segmentation network model and the mask is Let γ be the focusing parameter, then the focusing loss L focal and L1 loss L l1 And the segmentation network model loss L seg It can be calculated by the following formula:

[0072]

[0073]

[0074] L seg =L focal +L l1

[0075] According to the parameter initialization values ​​specified in Table 3, during the training process, if the current training round is less than or equal to 1000, the deep feature denoising network is trained, and if the current training round is greater than 1000 and less than or equal to 5000, the segmentation network is trained. When 5000 rounds of training are completed, the training is stopped.

[0076] After the training is completed, save the trained segmentation network model parameters to the specified folder.

[0077] Step 5: Load the saved model parameters into the corresponding network and perform anomaly detection on the test set.

[0078] Load the deep feature denoising network parameters saved in step 3 and the segmentation network parameters saved in step 4 into the corresponding networks respectively. Then put the image to be tested into the image anomaly detection model based on self-supervised learning and feature denoising for detection. Finally, after obtaining the model output, we calculate the average of the first T maximum values ​​in the anomaly score map to obtain the anomaly score of this image. The segmentation network output obtains the abnormal segmentation image, as shown in Figure 2 The predicted mask (prediction score map) is shown in . The value of T is shown in Table 3.

[0079] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. An image anomaly detection method based on self-supervised learning and feature denoising, characterized in that: The following steps are involved: Step 1: Collect normal images and perform preprocessing to generate a training set, collect normal images and abnormal images and perform preprocessing to generate a test set, and generate artificial anomalies for the images in the training set. Step 2: Send the abnormal image and normal image to the image anomaly detection model based on self-supervised learning and feature denoising. The feature generation network will generate deep features of normal images and abnormal images respectively, and then perform feature fusion on the deep features of abnormal images. The image anomaly detection model structure based on self-supervised learning and feature denoising is divided into a feature generation network, a deep feature denoising network and a segmentation network; The feature generation network uses a ResNet18 model with pre-trained weights, which is a deep convolutional neural network model that solves the problem of gradient vanishing through residual connections; the ResNet18 model contains 18 layers, which are divided into five residual blocks and a final global average pooling layer, and a ReLU activation function is used after each convolutional layer in the residual block; the ResNet18 model uses the network weights obtained by training on the ImageNet dataset: conv_5x and the final global average pooling, Step 3: Send the deep features of the abnormal image after feature fusion to the deep feature denoising network. At this time, start training the deep feature denoising network. After the training is completed, save the network model parameters of the deep feature denoising network. The deep feature denoising network includes an encoder structure and a decoder structure, wherein the encoder structure is a ResNet18 model with the first residual block: conv_1x and the last average pooling layer deleted, and the decoder structure is a reverse ResNet18 model with all downsampling replaced by bilinear upsampling, wherein the parameters of the encoder and the decoder are randomly initialized; during training, the cosine distance between the outputs of the corresponding blocks of the feature extraction network and the deep feature denoising network is minimized, Step 4: Start training the segmentation network. At this time, only the abnormal images are sent to the image anomaly detection model based on self-supervised learning and feature denoising. After the training is completed, save the network model parameters of the segmentation network. The structure of the segmentation network is as follows: the similarity between the feature layers of the decoders of the feature generation network and the deep feature denoising network corresponding blocks is calculated, and the calculation result is upsampled to one-fourth of the input image size, and then it is sequentially processed through two residual blocks and a void space pyramid pooling module to obtain the output result of the final segmentation network model; during training, the sum of the focus loss and the absolute error loss L1 between the output of the segmentation network model and the binary mask of the artificial anomaly is minimized; Step 5. Load the model parameters of the deep feature denoising network saved in step 3 and the model parameters of the segmentation network saved in step 4 into the deep feature denoising network and the segmentation network respectively. Then put the image to be detected into the image anomaly detection model based on self-supervised learning and feature denoising for detection. Finally, obtain the image-level anomaly detection results and pixel-level anomaly detection results output by the model.

2. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: The specific process of preprocessing normal images and artificial anomaly generation in step 1 is as follows: Step 1.1, the normal images used for training are uniformly truncated into the same size and put into a training set folder, and the normal images and abnormal images are put into the test set folder, and the normal images and abnormal images are placed separately; Step 1.2: Generate artificial anomalies for the images in the training set. Obtain mask M by binarizing with random two-dimensional Perlin noise and a preset threshold. n The abnormal image I is generated by replacing the mask region with a linear combination of any image from the external data source A a , where β represents the opacity factor, I a =β(MΘA)+(1-β)(MΘI) n )+(1-M)ΘI n 。 3. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: The specific process in step 2 is as follows: Step 2.1: Load the pre-trained ResNet18 model weights of the feature generation network. Step 2.2: Normal image I n Send it to the feature generation network to obtain the multi-level deep features of the normal image Step 2.3: Abnormal image I a Send it to the feature generation network to obtain multi-level deep features of abnormal images And set a learnable weight parameter w1, w2, w3, and a bias term ε respectively, and then perform the following feature aggregation operation on the multi-level deep features of the abnormal image to obtain the aggregated deep feature T a , 4. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: The specific process in step 3 is as follows: Step 3.1: Randomly initialize the encoder and decoder model weights in the deep feature denoising network. Step 3.2: The deep features T after the feature aggregation operation on the multi-level deep features of the abnormal image a Send it to the deep feature denoising network to minimize The cosine distance between the output of the corresponding block of the deep feature denoising network and the training round n1 of the deep feature denoising network and the training round n2 of the segmentation network are set. When the training round is less than or equal to n1, the segmentation network model parameters are frozen and only the deep feature denoising network is trained. When the training round is greater than n1 and less than n2, step 4 is performed.

5. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: In step 3, the output feature layers of conv2_x, conv3_x, and conv4_x in the ResNet18 model in the feature generation network are represented as T 1 , T 2 , T 3 , where the value at any point is represented by T k (i, j) (k = 1, 2, 3), the output feature layer of the corresponding reverse part in the deep feature denoising network decoder is represented as S 1 , S 2 , S 3 , where the value at any point is represented by S k (i, j) (k = 1, 2, 3), the similar feature maps between the feature layers of the corresponding blocks of the feature generation network and the deep feature denoising network decoder are represented as X1, X2, and X3, respectively, where the value at any point is X k (i,j)(k=1,2,3),C k , H k , W k Indicates the corresponding S k The number of channels, length and width of the layer feature map, Θ represents the Hadamard product, then the cosine loss value D at any point k (i, j) and the total cosine loss value L cos It can be calculated by the following formula: Next, (T 1 , S 1 ), (T 2 , S 2 ), (T 3 , S 3 ) are calculated by the above formula to obtain X1, X2, and X3, and then X2 and X3 are upsampled to the size of X1, and the concatenated feature map is represented as Will Input into the segmentation network model to segment out abnormal areas in the image.

6. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: The specific process in step 4 is as follows: Step 4.1: Randomly initialize the model weights of the segmentation network and freeze the deep feature denoising network model parameters. Step 4.2: Input only abnormal image I to the feature generation module a , get multi-level depth anomaly And the aggregated deep features T a , T a It is sent to the deep feature denoising network, and the similarity between the feature layers of the corresponding blocks of the decoder of the feature generation network and the deep feature denoising network is calculated and then sent to the segmentation network for training.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, an image anomaly detection method based on self-supervised learning and feature denoising is implemented as described in any one of claims 1 to 6 above.

8. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the image anomaly detection method based on self-supervised learning and feature denoising as described in any one of claims 1 to 6 is implemented.

Images