An image anomaly detection method based on self-supervised learning and feature denoising

Abstract

This invention discloses an image anomaly detection method based on self-supervised learning and feature denoising, belonging to the field of image anomaly detection technology. The method includes deep feature extraction, deep feature fusion, deep feature reconstruction, and anomaly segmentation. The method fully leverages the advantages of the multi-scale features of deep learning models to effectively fuse these multi-scale features. Furthermore, after fusing the multi-scale features, they are reconstructed to achieve denoising. A segmentation header is added at the end of the model to improve model performance. Compared with existing methods, this model achieves better anomaly detection results with the same number of parameters and training cost.

Description

Technical Field

[0001] The present invention relates to the technical field of image anomaly detection, and in particular to an image anomaly detection method based on self-supervised learning and feature denoising. Background Art

[0002] With the rapid development of computer vision and image processing technologies, image anomaly detection has gained widespread application in daily life and industrial applications, such as defect inspection in industrial environments and lesion identification in medical image analysis. Self-supervised learning, a key branch of machine learning, offers unique advantages for anomaly detection, such as eliminating the need for expensive labeling and the collection of large numbers of negative samples, and enabling improved model generalization.

[0003] Previously, anomaly detection often relied on template matching, statistical methods, and frequency domain analysis. However, in recent years, deep learning-based anomaly detection methods have become a research hotspot in this field. Feature embedding-based methods and reconstruction-based methods are two important categories of deep learning-based anomaly detection methods. Feature embedding-based methods use pretrained models to extract deep features. Reconstruction-based methods often employ structures such as generative adversarial networks or autoencoders. For example, the method DeSTSeg (DeSTSeg, et al., DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detection [J], 2022) combines the advantages of feature embedding-based and reconstruction-based methods. It proposes a denoising model, assuming that anomalies are represented as noise at the feature level. A teacher-student network guides the model in learning the denoising process. However, its student network performs encoding and decoding directly at the image level to achieve denoising, failing to fully leverage the powerful representational power of deep features. Consequently, this method performs poorly on subtle anomalies. Summary of the Invention

[0004] Purpose of the invention: The present invention provides an image anomaly detection method based on self-supervised learning and feature denoising, in order to overcome the shortcomings of the existing technology and improve the image anomaly detection effect.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: an image anomaly detection method based on self-supervised learning and feature denoising, comprising the following steps:

[0007] Step 1: After collecting normal images, preprocess them to generate a training set. After collecting normal images and abnormal images in a certain ratio, preprocess them to generate a test set, and generate artificial abnormalities for the images in the training set.

[0008] Step 2: The abnormal image and the normal image are fed into an image anomaly detection model based on self-supervised learning and feature denoising. The feature generation network generates deep features for the normal image and the abnormal image respectively, and then performs feature fusion on the deep features of the abnormal image.

[0009] The structure of the image anomaly detection model based on self-supervised learning and feature denoising is divided into a feature generation network, a deep feature denoising network and a segmentation network.

[0010] The feature generation network uses the ResNet18 model with pre-trained weights. It is a deep convolutional neural network model that solves the gradient vanishing problem through residual connections. The ResNet18 model contains 18 layers, which are divided into five residual blocks and a final global average pooling layer. The ReLU activation function is used after each convolutional layer in the residual block. The ResNet18 model uses the network weights obtained by training on the ImageNet dataset, but deletes the last residual block: conv_5x and the final global average pooling of the original ResNet18 model.

[0011] Step 3: The deep features of the abnormal image after feature fusion are fed into the deep feature denoising network. At this point, the deep feature denoising network is trained. After training is complete, the network model parameters of the deep feature denoising network are saved.

[0012] The deep feature denoising network consists of an encoder structure and a decoder structure. The encoder structure is a ResNet18 model with the first residual block: conv_1x and the last average pooling layer deleted. The decoder structure is an inverse ResNet18 model with all downsampling replaced by bilinear upsampling. The parameters of the encoder and decoder are randomly initialized. During training, the cosine distance between the outputs of the corresponding blocks of the feature extraction network and the deep feature denoising network is minimized.

[0013] Step 4: Start training the segmentation network. At this point, only abnormal images are fed into the image anomaly detection model based on self-supervised learning and feature denoising. After training is complete, save the network model parameters of the segmentation network.

[0014] The structure of the segmentation network is as follows: the similarity between the feature layers of the corresponding blocks of the decoder of the feature generation network and the deep feature denoising network is calculated, and the calculated result is upsampled to one-quarter the size of the input image. Then, it is sequentially processed through two residual blocks and a void spatial pyramid pooling module to obtain the output of the final segmentation network model. During training, the sum of the focus loss and the absolute error loss L1 between the segmentation network model output and the binary mask of artificial anomalies is minimized.

[0015] Step 5. Load the model parameters of the deep feature denoising network saved in step 3 and the model parameters of the segmentation network saved in step 4 into the deep feature denoising network and the segmentation network respectively. Then put the image to be tested into the image anomaly detection model based on self-supervised learning and feature denoising for detection. Finally, obtain the image-level anomaly detection results and pixel-level anomaly detection results output by the model.

[0016] Furthermore, the specific process of preprocessing normal images and generating artificial anomalies in step 1 is as follows:

[0017] Step 1.1: Cut all the normal images used for training into the same size and put them into a training set folder. Then put the normal images and abnormal images into the test set folder, and keep them separate.

[0018] Step 1.2: Generate artificial anomalies for the images in the training set. Generate the mask M by binarizing the random two-dimensional Perlin noise and the preset threshold. n The abnormal image I is generated by replacing the mask region with a linear combination of any image from the external data source A a :

[0019] I a =β(MΘA)+(1-β)(MΘI n )+(1-M)ΘI n ;

[0020] β represents the opacity factor;

[0021] Furthermore, the specific process in step 2 is as follows:

[0022] Step 2.1. Load the pre-trained ResNet18 model weights of the feature generation network.

[0023] Step 2.2: Normal image I n Send it into the feature generation network to obtain the multi-level deep features of the normal image

[0024] Step 2.3: Abnormal image Ia Send it to the feature generation network to obtain multi-level deep features of abnormal images And set a learnable weight parameter w1, w2, w3, and a bias term ε for each of them, and then perform the following feature aggregation operation on the multi-level deep features of the abnormal image to obtain the aggregated deep feature T a .

[0025]

[0026] Furthermore, the specific process in step 3 is as follows:

[0027] Step 3.1. Randomly initialize the encoder and decoder model weights in the deep feature denoising network.

[0028] Step 3.2: The depth feature T after the feature aggregation operation on the multi-level depth features of the abnormal image is a Send it to the deep feature denoising network to minimize The cosine distance between the output of the corresponding block of the deep feature denoising network and the training round n1 of the deep feature denoising network and the training round n2 of the segmentation network are set. When the training round is less than or equal to n1, the segmentation network model parameters are frozen and only the deep feature denoising network is trained. When the training round is greater than n1 and less than n2, proceed to step 4.

[0029] Furthermore, the specific process in step 4 is as follows:

[0030] Step 4.1. Randomly initialize the model weights of the segmentation network and freeze the deep feature denoising network model parameters.

[0031] Step 4.2: Input only abnormal image I to the feature generation module a , get multi-level depth anomaly And the aggregated deep features T a , T a The similarity between the feature layers of the corresponding blocks of the decoder of the feature generation network and the deep feature denoising network is calculated and then fed into the segmentation network for training.

[0032] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for detecting anomalies in an image based on self-supervised learning and feature denoising is implemented.

[0033] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the image anomaly detection method based on self-supervised learning and feature denoising.

[0034] The image anomaly detection method based on self-supervised learning and feature denoising of the present invention has the following advantages:

[0035] The technique of fusion of multi-level deep features and then denoising the deep features is adopted, and a segmentation network is added to the network, which produces the following technical effects:

[0036] 1. Use the pre-trained ResNet18 model to extract the deep features of the image, which improves the generalization ability of the model and the effect of image anomaly detection.

[0037] 2. The present invention uses deep feature denoising and feature comparison to perform anomaly detection, which avoids the consumption of a large amount of storage space compared with the feature embedding-based method.

[0038] 3. The present invention uses artificial anomalies to generate pseudo anomalies and trains the model in a self-supervised manner. On the one hand, it solves the problem of requiring a large amount of manpower for labeling. At the same time, it can also adjust the distribution of generated pseudo anomalies for different categories, making the model more targeted.

[0039] 4. In the present invention, a segmentation network head is further added after the feature denoising network, thereby further significantly improving the model anomaly detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the overall process of the method of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of the present invention;

[0042] Figure 3 This is the test result of the present invention on the metal_nut category of the MVTecAD dataset;

[0043] Figure 4 This is a cosine error curve of the deep feature denoising network of the present invention on the training set;

[0044] Figure 5 This is a graph showing the total error of the segmentation network of the present invention on the training set. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific implementation cases.

[0046] Example 1: An image anomaly detection method based on self-supervised learning and feature denoising, Figure 1 This is a training flowchart for an image anomaly detection method based on self-supervised learning and feature denoising; the details are as follows:

[0047] Step 1: Collect normal images and perform preprocessing to generate a training set. Collect normal images and abnormal images in a certain ratio and perform preprocessing to generate a test set. Generate artificial abnormalities for the images in the training set.

[0048] The training set is constructed with positive samples of one category in the MVTecAD dataset, and the training set images are scaled to a size of 224×224.

[0049] Step 2: The abnormal image and the normal image are fed into the image anomaly detection model based on self-supervised learning and feature denoising. The feature generation network generates deep features for the normal image and the abnormal image respectively, and then the deep features of the abnormal image are fused.

[0050] The model structure of the image anomaly detection method based on self-supervised learning and feature denoising is divided into a feature generation network, a deep feature denoising network and a segmentation network.

[0051] The feature generation network uses the ResNet18 model with pre-trained weights. It is a deep convolutional neural network model that solves the gradient vanishing problem through residual connections. The ResNet18 model contains 18 layers, which are divided into five residual blocks and a final global average pooling layer. The ReLU activation function is used after each convolutional layer in the residual block. The ResNet18 model uses the network weights obtained by training on the ImageNet dataset, but deletes the last residual block: conv_5x and the final global average pooling of the original ResNet18 model. The network structure of ResNet8 is shown in the following table:

[0052] Table 1 is the ResNet18 network structure

[0053]

[0054] Table 2 is an explanation of the terms in Table 1

[0055]

[0056] Step 3: The deep features of the abnormal image after feature fusion are fed into the deep feature denoising network. At this point, the deep feature denoising network is trained. After training is complete, the network model parameters of the deep feature denoising network are saved.

[0057] The deep feature denoising network includes an encoder structure and a decoder structure. The encoder structure is a ResNet18 model with the first residual block: conv_1x and the last average pooling layer deleted. The decoder structure is an inverse ResNet18 model with all downsampling replaced by bilinear upsampling. The parameters of the encoder and decoder are randomly initialized. During training, the cosine distance between the outputs of the corresponding blocks of the feature extraction network and the deep feature denoising network is minimized.

[0058] After training, save the parameters of the trained deep feature denoising network to the specified folder.

[0059] The GPU (graphics processing unit) used for training is Tesla T4.

[0060] When training the deep feature denoising network, randomly initialize the weights of the deep feature denoising network and the segmentation network. Insert the abnormal image into the model, set the training rounds n1 for the deep feature denoising network and n2 for the segmentation network, and begin training the deep feature denoising network. When the training rounds are less than or equal to n1, freeze the segmentation network model parameters and train only the deep feature denoising network. When the training rounds are greater than n1 and less than n2, proceed to step 4. The initialization values ​​of the model parameters are shown in the following table.

[0061] Table 3 Parameter initialization values

[0062]

[0063] The output feature layers of conv2_x, conv3_x, and conv4_x in the ResNet18 model in the feature generation network are represented as T 1 , T 2 , T 3 , where the value at any point is represented by T k (i, j) (k = 1, 2, 3), the output feature layer of the corresponding reverse part in the deep feature denoising network decoder is represented as S 1 , S 2 , S 3 , where the value at any point is represented by S k (i, j) (k = 1, 2, 3), the similar feature maps between the feature layers of the corresponding blocks of the feature generation network and the deep feature denoising network decoder are represented as X1, X2, and X3 respectively, where the value at any point is X k (i,j)(k=1,2,3),C k , H k , W k Indicates the corresponding S kThe number of channels, length and width of the layer feature map are represented by Θ, and the cosine loss value D at any point is k (i, j) and the total cosine loss value L cos It can be calculated by the following formula:

[0064]

[0065] Next, (T 1 , S 1 ), (T 2 , S 2 ), (T 3 , S 3 ) Calculate X1, X2, and X3 respectively through the above formula, then upsample X2 and X3 to the size of X1, and express the concatenated feature map as Will Input into the segmentation network model to segment out abnormal areas in the image.

[0066] Step 4: Train the segmentation network model

[0067] The segmentation network model consists of two residual blocks followed by a dilated spatial pyramid pooling module. The first residual block has 448 input channels and 256 output channels, and is composed of two 3×3 convolutional layers with a stride of 1. The second residual block has both 256 input and output channels, and is also composed of two 3×3 convolutional layers with a stride of 1.

[0068] Next, the abnormal image I a Feed it into the network, freeze the parameters of the deep feature denoising network, and train only the segmentation network. During training, minimize the sum of the focus loss and L1 loss between the network output and the binary mask of the artificial anomaly.

[0069] use Represents the probability map output by the segmentation network model, M ij is the value of any point of the mask M, for The value of any point, p ij The similarity between the probability map output by the segmentation network model and the mask is Let γ be the focusing parameter, then the focusing loss L focal and L1 loss L l1 And the segmentation network model loss L seg It can be calculated by the following formula:

[0070]

[0071] L seg =L focal +L l1

[0072] According to the parameter initialization values ​​specified in Table 3, during training, if the current training round is less than or equal to 1000, the deep feature denoising network is trained. If the current training round is greater than 1000 and less than or equal to 5000, the segmentation network is trained. Training is stopped after 5000 rounds.

[0073] After training, save the trained segmentation network model parameters to the specified folder.

[0074] Step 5: Load the saved model parameters into the corresponding network and perform anomaly detection on the test set.

[0075] Load the deep feature denoising network parameters saved in step 3 and the segmentation network parameters saved in step 4 into the corresponding networks respectively. Then put the image to be tested into the image anomaly detection model based on self-supervised learning and feature denoising for detection. Finally, after obtaining the model output, we calculate the average of the first T maximum values ​​in the anomaly score map to obtain the anomaly score of this image. The segmentation network output obtains the abnormal segmentation image, as shown in Figure 3 The predicted mask (prediction score map) is shown in . The value of T is shown in Table 3.

[0076] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. An image anomaly detection method based on self-supervised learning and feature denoising, characterized in that: The following steps are involved: Step 1: Collect normal images and preprocess them to generate a training set. Collect normal and abnormal images and preprocess them to generate a test set. Generate artificial anomalies for the images in the training set. Step 2: The abnormal image and the normal image are fed into the image anomaly detection model based on self-supervised learning and feature denoising. The feature generation network will generate deep features of the normal image and the abnormal image respectively, and then perform feature fusion on the deep features of the abnormal image. The image anomaly detection model based on self-supervised learning and feature denoising is structured into a feature generation network, a deep feature denoising network and a segmentation network; The feature generation network uses the ResNet18 model with pre-trained weights, which is a deep convolutional neural network model that solves the gradient vanishing problem through residual connections. The ResNet18 model contains 18 layers, which are divided into five residual blocks and a final global average pooling layer. The ReLU activation function is used after each convolutional layer in the residual block. The ResNet18 model uses the network weights obtained by training on the ImageNet dataset: conv_5x and the final global average pooling. Step 3: Send the deep features of the abnormal image after feature fusion into the deep feature denoising network. At this time, start training the deep feature denoising network. After the training is completed, save the network model parameters of the deep feature denoising network. The deep feature denoising network includes an encoder structure and a decoder structure. The encoder structure is a ResNet18 model with the first residual block: conv_1x and the last average pooling layer deleted. The decoder structure is an inverse ResNet18 model with all downsampling replaced by bilinear upsampling. The parameters of the encoder and decoder are randomly initialized. During training, the cosine distance between the outputs of the corresponding blocks of the feature extraction network and the deep feature denoising network is minimized. Step 4: Start training the segmentation network. At this time, only the abnormal images are sent to the image anomaly detection model based on self-supervised learning and feature denoising. After the training is completed, save the network model parameters of the segmentation network. The structure of the segmentation network is as follows: the similarity between the feature layers of the corresponding blocks of the decoder of the feature generation network and the deep feature denoising network is calculated, and the calculated result is upsampled to one-quarter of the input image size. Then, it is sequentially processed through two residual blocks and a void spatial pyramid pooling module to obtain the output of the final segmentation network model; during training, the sum of the focus loss and the absolute error loss L1 between the segmentation network model output and the binary mask of the artificial anomaly is minimized; Step 5: Load the model parameters of the deep feature denoising network saved in step 3 and the model parameters of the segmentation network saved in step 4 into the deep feature denoising network and the segmentation network respectively. Then, put the image to be tested into the image anomaly detection model based on self-supervised learning and feature denoising for detection. Finally, obtain the image-level anomaly detection results and pixel-level anomaly detection results output by the model. The specific process in step 2 is as follows: Step 2.1, load the pre-trained ResNet18 model weights of the feature generation network, Step 2.2: Normal image I n Send it into the feature generation network to obtain the multi-level deep features of the normal image Step 2.3: Abnormal image I a Send it to the feature generation network to obtain multi-level deep features of abnormal images And set a learnable weight parameter w1, w2, w3, and a bias term ε for each of them, and then perform the following feature aggregation operation on the multi-level deep features of the abnormal image to obtain the aggregated deep feature T a , 2. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: The specific process of preprocessing normal images and generating artificial anomalies in step 1 is as follows: Step 1.1: Cut all the normal images used for training into the same size and put them into a training set folder. Then put the normal images and abnormal images into the test set folder, and keep them separate. Step 1.2: Generate artificial anomalies for the images in the training set. Generate the mask M by binarizing the random two-dimensional Perlin noise and the preset threshold. n The abnormal image I is generated by replacing the mask region with a linear combination of any image from the external data source A a , where β represents the opacity factor, Θ represents the Hadamard product, I a =β(MΘA)+(1-β)(MΘI) n )+(1-M)ΘI n 。 3. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: The specific process in step 3 is as follows: Step 3.1: Randomly initialize the encoder and decoder model weights in the deep feature denoising network. Step 3.2: The depth feature T after the feature aggregation operation on the multi-level depth features of the abnormal image is a Send it to the deep feature denoising network to minimize The cosine distance between the output of the corresponding block of the deep feature denoising network and the training round n1 of the deep feature denoising network and the training round n2 of the segmentation network are set. When the training round is less than or equal to n1, the segmentation network model parameters are frozen and only the deep feature denoising network is trained. When the training round is greater than n1 and less than n2, step 4 is performed.

4. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: In step 3, the output feature layers of conv2_x, conv3_x, and conv4_x in the ResNet18 model in the feature generation network are represented as T 1 , T 2 , T 3 , where the value at any point is represented by T k (i, j), k = 1, 2, 3, the output feature layer of the corresponding reverse part in the deep feature denoising network decoder is represented as S 1 , S 2 , S 3 , where the value at any point is represented by S k (i, j), k = 1, 2, 3, the similar feature maps between the feature layers of the corresponding blocks of the feature generation network and the deep feature denoising network decoder are represented as X1, X2, and X3 respectively, where the value at any point is X k (i, j), k = 1, 2, 3, C k , H k , W k Indicates the corresponding S k The number of channels, length and width of the layer feature map are represented by Θ, and the cosine loss value D at any point is k (i, j) and the total cosine loss value L cos It can be calculated by the following formula: Next, (T 1 , S 1 ), (T 2 , S 2 ), (T 3 , S 3 ) Calculate X1, X2, and X3 respectively through the above formula, then upsample X2 and X3 to the size of X1, and express the concatenated feature map as Will Input into the segmentation network model to segment out abnormal areas in the image.

5. The image anomaly detection method based on self-supervised learning and feature denoising according to claim 1, characterized in that: The specific process in step 4 is as follows: Step 4.1: Randomly initialize the model weights of the segmentation network and freeze the deep feature denoising network model parameters. Step 4.2: Input only abnormal image I to the feature generation module a , get multi-level depth anomaly And the aggregated deep features T a , T a The deep feature denoising network is fed into the network, and the similarity between the feature layers of the corresponding blocks of the decoder of the feature generation network and the deep feature denoising network is calculated and then fed into the segmentation network for training.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image anomaly detection method based on self-supervised learning and feature denoising as described in any one of claims 1 to 5 above is implemented.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the image anomaly detection method based on self-supervised learning and feature denoising as described in any one of claims 1 to 5 is implemented.

Images