A two-stage image anomaly detection method based on generative adversarial network
By using a feature extraction module combined with a multi-scale convolutional flow in image anomaly detection, the abnormality score of the image is calculated, and the problem of low detection accuracy in the existing technology is solved, and the implementation of the More efficient identification of abnormal data.
Patent Information
- Application Number
- CN202210229057.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-08
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-08
AI Technical Summary
The existing deep learning-based image anomaly detection method has little difference in the data characteristics of normal and abnormal classes or has low accuracy in precision image detection, especially when the amount of abnormal data is small and the type cannot be completely determined. The results are not good.
Using a two-stage image anomaly detection method based on a generative adversarial network, the image feature extraction module combined with a parallel multi-scale convolutional flow is used to replace the operation of directly inputting the image to the generator, and the generator is replaced with Attention UNet with Attention Gate calculates anomaly score through the reconstruction image and the original image difference in the training phase to achieve more accurate anomaly detection.
It effectively improves the accuracy of image abnormality detection, especially when the amount of abnormal data is small and the feature difference is not large, the detection accuracy is significantly improved through the combination of multi-scale feature fusion and Attention Gate generator.
Smart Images

Figure CN114757257B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer application technology and artificial intelligence, and in particular to a two-stage image anomaly detection method based on a generative adversarial network. Background Art
[0002] Image anomaly detection is a technology that identifies abnormal or illogical samples from a large number of image samples. It is an important research part of machine learning and has practical applications in professional fields such as medical services, industrial production, and video processing.
[0003] The characteristics of anomaly detection datasets are usually that normal data occupies the vast majority, and the amount of abnormal data is very small. The two are in a highly unbalanced state, and usually, the type of anomaly cannot be fully determined. In this dataset, supervised learning cannot achieve a high accuracy rate for image detection, while unsupervised learning can effectively overcome this problem. For example, Ian J et al. proposed the Generative Adverserial Network (GAN) as an unsupervised learning network architecture. GAN is an unsupervised deep learning framework that evaluates generative models through an adversarial process. It is mainly based on the competition between two networks in a zero-sum game framework. GAN consists of two networks, namely the generator and the discriminator. Among them, the goal of the generator is to capture the data distribution, learn image features from the original data, and generate a reconstructed image that is as similar to the real image as possible; the goal of the discriminator is to determine whether the sample comes from the dataset or the generator, which can generally be regarded as a binary classifier. Through the optimization of the zero-sum game framework, both networks can enhance their prediction capabilities until a balance is reached. Later, Samet et al. proposed the GANomaly network to detect anomalies in images. It only needs to use normal data during training, and normal and abnormal data during the test phase. Then, it obtains the anomaly score based on the difference between the reconstructed image and the original image, and then distinguishes the two types of data based on the numerical value of the anomaly score. Based on GANomaly, Samet et al. proposed a skip-GANomaly network architecture with skip-connection, which mainly made certain improvements in the generator part, and can more accurately map high-dimensional data to low-dimensional latent codes, and then reversely map to obtain the reconstructed image. In existing methods, the image is generally directly input into the generator, and the image reconstruction is achieved only by relying on the upsampling and downsampling process of the generator. Therefore, the feature representation of the image is not very accurate, resulting in low accuracy of anomaly detection.
[0004] In summary, the existing deep learning-based image anomaly detection methods do not perform well for images with little difference in data features between normal and abnormal classes or for images with relatively precise data features, and need to be further improved. Summary of the invention
[0005] The purpose of the present invention is to provide a two-stage image anomaly detection method based on a generative adversarial network. Based on the generative adversarial network architecture, only normal data is used to train the model. In the test phase, normal data and abnormal data are distinguished according to the numerical value of the anomaly score between the reconstructed image and the original image. An image feature extraction module combining a convolutional neural network and a parallel multi-scale convolutional flow is also designed. The fused image features are input into the generator, replacing the current operation of directly inputting the image into the generator, so as to solve the existing technical defects and unattainable technical requirements.
[0006] To achieve the above object, the present invention provides the following technical solution: a two-stage image anomaly detection method based on a generative adversarial network, comprising the following steps:
[0007] Step 1: obtaining normal images and abnormal images, and dividing the images into a training set and a test set, wherein the training set contains only normal images, and the test set contains normal images and abnormal images;
[0008] Step 2: Construct an image feature extraction module that combines a convolutional neural network with a multi-scale feature extraction module. The image features of the training set and the image features of the test set are obtained through the image feature extraction module. The multi-scale feature extraction module is composed of three parallel multi-scale convolution flows.
[0009] Step 3: Use skip-GANomaly as a generative adversarial network model and completely replace the generator UNet with the generator with Attention Gate in Attention UNet. Compared with the standard UNet network, the generator with AG focuses more on the area of interest of the task rather than the entire image, and the sub-AGs in each layer can extract and fuse complementary information. The reconstructed image obtained by the generator with AG is better.
[0010] Step 4: In the training phase, the image features obtained from the training set in step 2 are input into the generator in step 3 to obtain a reconstructed image. The reconstructed image and the original image are then input into the discriminator together. The generator and the discriminator are trained using the loss function. The training goal is to make the generator generate a reconstructed image that is as similar as possible to the original image, and to make the discriminator distinguish the original image from the reconstructed image as much as possible.
[0011] Step 5: In the test phase, the image features obtained from the test set in step 2 are input into the generator in step 3 to obtain a reconstructed image. The reconstructed image and the original image are then input into the discriminator to obtain an anomaly score vector consisting of the anomaly scores of each test image. They are normalized to the range of [0, 1] for subsequent comparison.
[0012] Step 6: Use the numerical value of the anomaly score obtained in step 5 to distinguish normal images from abnormal images. For normal images, the difference between the reconstructed image and the original image is very small, so the corresponding anomaly score is small; for abnormal images, the difference between the reconstructed image and the original image is large, so the corresponding anomaly score is large, thereby realizing image anomaly detection.
[0013] Preferably, the step 2 is implemented by the following sub-steps:
[0014] (1.1) Using convolutional neural network 1, extract the image features of the input image;
[0015] (1.2) Use a 1x1 convolution kernel to further extract the features output by (1.1);
[0016] (1.3) Use a 3x3 convolution kernel to further extract the features output by (1.1);
[0017] (1.4) Use a 5x5 convolution kernel to further extract the features output by (1.1);
[0018] (1.5) The outputs of steps (1.2), (1.3) and (1.4) are combined and then added to the output features of step (1.1);
[0019] (1.6) Using convolutional neural network 2, the fusion features obtained in (1.5) are extracted again to obtain the final image features;
[0020] Convolutional neural networks 1 and 2 are two convolutional neural networks used to extract image features, and their structures are the same.
[0021] Preferably, the loss function includes three types: reconstruction loss, adversarial loss and potential loss. The formulas of the loss function are shown in Formulas 1, 2 and 3 respectively, where the reconstruction loss in Formula 1 represents the original image x and the reconstructed image The L1 loss between the two, the adversarial loss in formula 2 represents minimizing the goal of the generator while maximizing the goal of the discriminator, and the potential loss in formula 3 represents the x and x extracted by the last convolutional layer of the discriminator. L2loss between the reconstructed latent expressions;
[0022]
[0023]
[0024]
[0025] Among them, x represents the original image, represents the reconstructed image generated by the generator, p x represents the distribution of the input image x, E represents the mathematical expectation, D represents a function for obtaining the distribution of the intermediate layer of the discriminator, and f represents a function for obtaining the potential representation.
[0026] Preferably, the anomaly score calculation formula is as follows:
[0027] A(x)=λR(x)+(1-λ)L(x)
[0028] Wherein R(x) represents the above formula 1, L(x) represents the above formula 3, and λ represents a weight parameter.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] 1) Based on the existing multi-scale feature fusion method, the present invention designs an image feature extraction module that combines a convolutional neural network with a parallel multi-scale convolutional flow. The fused image features are input into the generator, replacing the current operation of directly inputting the image into the generator.
[0031] 2) The present invention is mainly based on the unsupervised GAN network architecture, and only uses normal data to train the model. In the test phase, normal data and abnormal data are distinguished based on the numerical value of the anomaly score between the reconstructed image and the original image. This can effectively solve the problem of low detection accuracy caused by the high imbalance between the number of abnormal samples and the number of normal samples in the current anomaly detection method.
[0032] 3) The generator UNet in skipGANomaly is completely replaced by the generator with AttentionGate in Attention UNet. Compared with the standard UNet network, the generator with AG focuses more on the area of interest of the task rather than the entire image, and the sub-AGs in each layer can extract and fuse complementary information, resulting in better reconstructed images. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A comparison diagram of the reconstruction effects of the method of the present invention and the generator in skipGANomaly on normal liver CT images;
[0034] Figure 2This is a schematic diagram of the image anomaly detection process;
[0035] Figure 3 This is a two-stage model architecture diagram based on generative adversarial networks;
[0036] Figure 4 This is the network architecture diagram of the multi-scale feature extraction module. DETAILED DESCRIPTION
[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0038] See also Figure 1-4 The present invention provides a technical solution: a two-stage image anomaly detection method based on a generative adversarial network, comprising the following steps:
[0039] Step 1: obtaining normal images and abnormal images, and dividing the images into a training set and a test set, wherein the training set contains only normal images, and the test set contains normal images and abnormal images;
[0040] Step 2: Construct an image feature extraction module that combines a convolutional neural network with a multi-scale feature extraction module. The image features of the training set and the image features of the test set are obtained through the image feature extraction module. The multi-scale feature extraction module is composed of three parallel multi-scale convolution flows.
[0041] Step 3: Use skip-GANomaly as a generative adversarial network model and completely replace the generator UNet with the generator with Attention Gate in Attention UNet. Compared with the standard UNet network, the generator with AG focuses more on the area of interest of the task rather than the entire image, and the sub-AGs in each layer can extract and fuse complementary information. The reconstructed image obtained by the generator with AG is better.
[0042] Step 4: In the training phase, the image features obtained from the training set in step 2 are input into the generator in step 3 to obtain a reconstructed image. The reconstructed image and the original image are then input into the discriminator together. The generator and the discriminator are trained using the loss function. The training goal is to make the generator generate a reconstructed image that is as similar as possible to the original image, and to make the discriminator distinguish the original image from the reconstructed image as much as possible.
[0043] Step 5: In the test phase, the image features obtained from the test set in step 2 are input into the generator in step 3 to obtain a reconstructed image. The reconstructed image and the original image are then input into the discriminator to obtain an anomaly score vector consisting of the anomaly scores of each test image. They are normalized to the range of [0, 1] for subsequent comparison.
[0044] Step 6: Use the numerical value of the anomaly score obtained in step 5 to distinguish normal images from abnormal images, such as Figure 2 As shown in the figure, for normal images, the difference between the reconstructed image and the original image is very small, so the corresponding anomaly score value is small; for abnormal images, the difference between the reconstructed image and the original image is very large, so the corresponding anomaly score value is large. Images with anomaly score values greater than 0.5 are identified as abnormal images, and the rest are normal images, thereby realizing image anomaly detection.
[0045] like Figure 3 , 4 As shown, step 2 is implemented by the following sub-steps:
[0046] (1.1) Using convolutional neural network 1, extract the image features of the input image;
[0047] (1.2) Use a 1x1 convolution kernel to further extract the features output by (1.1);
[0048] (1.3) Use a 3x3 convolution kernel to further extract the features output by (1.1);
[0049] (1.4) Use a 5x5 convolution kernel to further extract the features output by (1.1);
[0050] (1.5) The outputs of steps (1.2), (1.3) and (1.4) are combined and then added to the output features of step (1.1);
[0051] (1.6) Using convolutional neural network 2, the fusion features obtained in (1.5) are extracted again to obtain the final image features;
[0052] like Figure 3 As shown in the figure, the first stage is the image feature extraction module, which includes the following Figure 4 The multi-scale feature extraction module shown in the figure has a second stage, which is a generative adversarial network module. The multi-scale feature extraction module extracts image features from the previous layer through 1x1, 3x3, and 5x5 convolution kernels, then fuses them and adjusts them to the quantity structure required by the subsequent network through a 1x1 convolution kernel.
[0053] Convolutional neural networks 1 and 2 are two convolutional neural networks used to extract image features, and their structures are the same.
[0054] The loss function includes three types: reconstruction loss, adversarial loss and potential loss. The formulas of the loss functions are shown in Formulas 1, 2 and 3 respectively. The reconstruction loss in Formula 1 represents the original image x and the reconstructed image The L1 loss between the two, the adversarial loss in formula 2 represents minimizing the goal of the generator while maximizing the goal of the discriminator, and the potential loss in formula 3 represents the x and x extracted by the last convolutional layer of the discriminator. L2loss between the reconstructed latent expressions;
[0055]
[0056]
[0057]
[0058] Among them, x represents the original image, represents the reconstructed image generated by the generator, p x represents the distribution of the input image x, E represents the mathematical expectation, D represents a function for obtaining the distribution of the intermediate layer of the discriminator, and f represents a function for obtaining the potential representation.
[0059] The anomaly score calculation formula is as follows;
[0060] A(x)=λR(x)+(1-λ)L(x)
[0061] Wherein R(x) represents the above formula 1, L(x) represents the above formula 3, and λ represents a weight parameter.
[0062] Table 1 shows the experimental results of image anomaly detection on the public dataset CIFAR10. In each experiment, any one of the ten categories of CIFAR10 data (bird, car, cat, deer, dog, frog, horse, plane, ship, truck) is selected as abnormal data, and the remaining 9 categories are selected as normal data. In the experiment of Table 1, the batch size of the network parameters used by the method of the present invention is 64 and the image size is 32. The evaluation index used is AUC, which means the area under the ROC curve and can be used to measure the accuracy of the classification model. The value range of AUC is generally [0.5,1]. The closer the value is to 1, the higher the classification accuracy of the classifier. It can be seen from the table that the method proposed in the present invention has significantly improved the accuracy compared with the results obtained by the skip-GANomaly model, which proves the effectiveness of the method.
[0063] Table 1 Comparison of AUC results with the skip-GANomaly model on the CIFAR10 dataset
[0064] method abnormal class=frog abnormal class=cat abnormal class=bird skip-GANomaly 0.965 0.613 0.656 The present invention 0.999 0.853 0.922
[0065] We also conducted the same experiment on a medical image dataset, which was composed of normal and abnormal liver CT images of several patients from the Department of Liver Diseases of Zhejiang Second Hospital. We also performed 4-fold cross processing on the dataset, and Table 2 shows the experimental results of image anomaly detection on the 4-fold liver CT image dataset. The evaluation index used was also AUC. In the experiment in Table 2, the network parameters used by the method of the present invention include a batchsize of 195 and an image size of 32. As can be seen from the table, the experimental results of the method proposed in the present invention on the medical image dataset are also improved compared to skip-GANomaly, which proves the effectiveness of the method.
[0066] like Figure 1 As shown in the figure, the left image is the reconstructed image of skipGANomaly, the middle image is the original image, and the right image is the reconstructed image of the method of the present invention. It can be seen from the figure that the reconstruction effect of the method proposed in the present invention for normal images is better than that of skip-GANomaly.
[0067] Table 2 Comparison of AUC results with the skip-GANomaly model on the 4-fold liver CT image dataset
[0068]
[0069]
[0070] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.
[0071] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved.
[0072] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. A two-stage image anomaly detection method based on generative adversarial network, It is characterized in that The following steps are involved: Step 1: obtaining normal images and abnormal images, and dividing the images into a training set and a test set, wherein the training set contains only normal images, and the test set contains normal images and abnormal images; Step 2: Construct an image feature extraction module that combines a convolutional neural network with a multi-scale feature extraction module. The image features of the training set and the image features of the test set are obtained through the image feature extraction module. The multi-scale feature extraction module is composed of three parallel multi-scale convolution flows. Step 3: Use skip-GANomaly as a generative adversarial network model and completely replace the generator UNet with the generator with AttentionGate in AttentionUNet; Step 4: In the training phase, the image features obtained from the training set in step 2 are input into the generator in step 3 to obtain a reconstructed image. The reconstructed image and the original image are then input into the discriminator together, and the loss function is used to train the generator and discriminator. Step 5: In the test phase, the image features obtained from the test set in step 2 are input into the generator in step 3 to obtain a reconstructed image. The reconstructed image and the original image are then input into the discriminator to obtain an anomaly score vector consisting of the anomaly scores of each test image. They are normalized to the range of [0, 1] for subsequent comparison. Step 6: Use the numerical value of the anomaly score obtained in step 5 to distinguish normal images from abnormal images, thereby realizing image anomaly detection.
2. According to the two-stage image anomaly detection method based on generative adversarial network in claim 1, It is characterized in that The step 2 is implemented by the following sub-steps: (1.1) Using convolutional neural network 1, extract the image features of the input image; (1.2) Use a 1x1 convolution kernel to further extract the features output by (1.1); (1.3) Use a 3x3 convolution kernel to further extract the features output by (1.1); (1.4) Use a 5x5 convolution kernel to further extract the features output by (1.1); (1.5) The outputs of steps (1.2), (1.3) and (1.4) are combined and then added to the output features of step (1.1); (1.6) Using convolutional neural network 2, the fusion features obtained in (1.5) are extracted again to obtain the final image features.
3. According to the two-stage image anomaly detection method based on generative adversarial network in claim 1, It is characterized in that The loss function includes three types: reconstruction loss, adversarial loss and potential loss. The formulas of the loss functions are shown in Formulas 1, 2 and 3 respectively. The reconstruction loss in Formula 1 represents the original image x and the reconstructed image L 1 loss, the adversarial loss in formula 2 represents minimizing the goal of the generator while maximizing the goal of the discriminator, and the potential loss in formula 3 represents the x and The L between the reconstructed potential expressions 2 loss; Among them, x represents the original image, represents the reconstructed image generated by the generator, p x represents the distribution of the input image x, E represents the mathematical expectation, D represents a function for obtaining the distribution of the intermediate layer of the discriminator, and f represents a function for obtaining the potential representation.
4. According to the two-stage image anomaly detection method based on generative adversarial network in claim 3, It is characterized in that The anomaly score calculation formula is as follows; A(x)=λR(x)+(1-λ)L(x) Wherein R(x) represents the above formula 1, L(x) represents the above formula 3, and λ represents a weight parameter.
