An image anomaly detection method based on high- and low-frequency reconstruction

Abstract

The present invention discloses an image anomaly detection method based on high- and low-frequency reconstruction, which includes two stages: model training and model inference. In the model training stage, a generative adversarial network is trained on normal images in a training set, and the network model is trained using a server. The network parameters are optimized by reducing the network loss function until the network converges, thereby obtaining a network model based on high- and low-frequency reconstruction. In the model inference stage, the network model obtained in the model training stage is used to determine whether a new test image is an abnormal image based on an anomaly score. Information of different frequency bands of the image is obtained through a frequency domain decoupling module and reconstructed using different generators. In the encoding stage of the network generator, a channel selection module is used to realize interaction and selection of information of different frequency bands, so that the network can fully utilize the information of different frequency bands, improve the reconstruction capability of the generator, and thereby improve the performance of the image anomaly detection algorithm based on reconstruction.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an image anomaly detection method based on high- and low-frequency reconstruction. Background Art

[0002] The task of image anomaly detection is to identify the boundary between normal and abnormal samples, separating them as closely as possible. Currently, a major challenge in practical anomaly detection is the imbalance of real-world data. Data on abnormal images is often difficult to obtain. For example, when automatically monitoring product defects on a production line, the probability of a defective product is generally very low. Simple supervised learning classification tasks are difficult to apply to image anomaly detection. Therefore, the vast majority of currently reliable image anomaly detection methods are unsupervised. Anomaly detection relies on a training set to determine whether the input data is "similar" to the data in the training set.

[0003] Detecting anomalies by exploiting the differences between normal and abnormal images in the feature space extracted by a pretrained network can achieve good results, but the interpretability is poor. To improve the interpretability of these methods, the image can be segmented for detection, but this incurs a higher computational load. Compared to these methods, reconstruction-based image anomaly detection methods are more intuitive and more interpretable. However, the reconstruction capability of the generator in reconstruction-based image anomaly detection methods significantly affects the anomaly detection performance. Blurred or self-reflected reconstructions can lead to poor anomaly detection results. Therefore, reconstruction-based methods are not yet reliable enough for image anomaly detection tasks.

[0004] Most existing reconstruction-based methods directly reconstruct the original image without distinguishing between information in different frequency bands. However, for image anomaly detection tasks, abnormal elements often exist in the high-frequency information of the image, and information in different frequency bands has different importance for image anomaly detection tasks. Summary of the Invention

[0005] To achieve the above objectives, the present invention addresses the shortcomings of current image anomaly detection algorithms and provides an image anomaly detection method based on high- and low-frequency reconstruction. This method is based on a GAN (Generative Adversarial Network) anomaly detection framework with frequency-band reconstruction. It also utilizes a channel selection module within the GAN generator to enable interaction and selection of information from different frequency bands, achieving excellent image anomaly detection results. This solves the problem that reconstruction-based image anomaly detection methods often have unsatisfactory results in scenarios such as defect detection due to the limited generation capabilities of the generator.

[0006] The present invention employs the following technical solution: a method for image anomaly detection based on high- and low-frequency reconstruction, comprising two phases: model training and model inference. In the model training phase, a generative adversarial network is trained on normal images in a training set. The network model of the generative adversarial network is then trained using a server. The network parameters are optimized by reducing the network loss function until the network converges, resulting in a network model based on high- and low-frequency reconstruction. In the model inference phase, the network model obtained in the model training phase is used to determine whether a new test image is an anomaly based on anomaly scores.

[0007] Furthermore, the model training phase specifically includes the following steps:

[0008] S11. Image data preparation: using the frequency domain decoupling module to obtain information of different frequency bands of normal images in the training set;

[0009] S12. Network model training: The information of different frequency bands of the normal image is passed into the generator for reconstruction, the discriminator is used to distinguish the original image from the reconstructed image, and the adversarial learning method is used for training; the loss function is calculated to obtain the gradient, and then backpropagation is performed to obtain the trained network model.

[0010] Furthermore, the frequency domain decoupling module in step S11 specifically uses a Gaussian convolution kernel to obtain the low-frequency component of the normal image, uses the difference between the original normal image and the low-frequency component to obtain the high-frequency component of the image, and finally obtains information of different frequency bands of the image.

[0011] Furthermore, the network model training in step S12 includes forward propagation of image data and gradient back propagation. In the forward propagation of image data, all normal images in the training set are decoupled by the frequency domain decoupling module and then sent to the network including the generator and the discriminator for training; in the gradient back propagation, the game between the generator and the discriminator constitutes AdversarialLoss, the difference between the reconstructed image reconstructed by the generator and the original normal image constitutes ContextualLoss, and the difference between the reconstructed image reconstructed by the generator and the original normal image in the latent space constitutes LatentLoss; the gradient is calculated according to the loss function, the gradient back propagation updates the network parameters, and the GPU is used for acceleration until the network error is reduced to within the set threshold or the number of network iterations meets the requirement, and the training is stopped to obtain a trained network model.

[0012] Furthermore, the generator is used to encode and then decode the input image to achieve a reconstruction process of the input image; or, the discriminator is used to discriminate between the reconstructed image generated by the generator and the original normal image.

[0013] Furthermore, the generator includes a high- and low-frequency reconstruction module and a channel selection module. The high- and low-frequency reconstruction module includes multiple branches with different frequency bands to reconstruct information of different frequency bands of the image respectively; the channel selection module adds the feature maps of different frequency bands of the image, and obtains the channel attention vectors corresponding to the features of different frequency bands through a global average pooling layer, a fully connected layer, and a softmax layer. The channel attention vector is multiplied by the original feature map to obtain a new feature map, which serves as the input of the next convolutional layer.

[0014] Furthermore, the loss function used by AdversarialLoss is:

[0015]

[0016] The loss function used by the ContextualLoss is:

[0017]

[0018] The loss function used by the LatentLoss is:

[0019]

[0020] in:

[0021] x is a normal image,

[0022] G(x) is the reconstructed image,

[0023] D(x) is the discriminator output,

[0024] f(x) is the feature map of the latent space.

[0025] Furthermore, the model inference stage specifically includes the following steps:

[0026] S21, passing the test image through a frequency domain decoupling module to obtain information of different frequency bands of the test image;

[0027] S22. Information from different frequency bands of the test image is passed into the trained network model to obtain reconstruction error and latent space error. The reconstruction error and latent space error are weighted averaged and fused to obtain the anomaly score inferred by the final model. Whether the test image is an abnormal image is determined based on a preset threshold.

[0028] The image anomaly detection method based on high- and low-frequency reconstruction proposed in the present invention obtains information of different frequency bands of the image through a frequency domain decoupling module and reconstructs it using different generators. In the encoding stage of the network generator, a channel selection module is used to realize the interaction and selection of information of different frequency bands, so that the network can make full use of information of different frequency bands, improve the reconstruction ability of the generator, and thus improve the performance of the reconstruction-based image anomaly detection algorithm.

[0029] The technical solution of the present invention has the following beneficial effects: compared with the traditional discriminant algorithm that uses a pre-trained model to extract features, this method is based on a reconstruction method and has stronger interpretability; compared with the algorithm that directly reconstructs the original image, this method decouples the frequency domain information of the image, reconstructs information of different frequency bands separately, and selects and interacts information of different frequency bands in the encoding stage of the network generator, which can improve the reconstruction capability of the generator and achieve better image anomaly detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the image anomaly detection process based on high and low frequency reconstruction of the present invention.

[0031] Figure 2 Schematic diagram of the channel selection module of the present invention. DETAILED DESCRIPTION

[0032] The specific embodiments of the present invention are further described in conjunction with the accompanying drawings to make the present technical solution clearer and more understandable. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the features in the following embodiments and embodiments can be combined with each other without conflict.

[0033] This embodiment relates to an image anomaly detection method based on high- and low-frequency reconstruction, which includes two stages: model training and model inference.

[0034] During the model training phase, the generative adversarial network is trained only on normal data. A server with high computing performance is used to train the generative adversarial network model. The network parameters are optimized by reducing the network loss function until the network converges, obtaining a network model based on high- and low-frequency reconstruction. This process specifically includes the following steps:

[0035] S11. First, prepare the image data. Specifically, use the frequency domain decoupling module to obtain information of different frequency bands of normal images in the training set. The frequency domain decoupling module uses a Gaussian convolution kernel to obtain the low-frequency component of the normal image, and then uses the difference between the original normal image and the low-frequency component of the normal image to obtain the high-frequency component of the normal image. And so on, to obtain information of different frequency bands of the normal image.

[0036] S12. Then, the network model is trained. The information of different frequency bands of the normal image is passed into the generator for reconstruction to obtain a reconstructed image. The discriminator is used to distinguish the original normal image from the reconstructed image, and the adversarial learning method is used for training. The loss function is calculated to obtain the gradient, and then backpropagation is performed to obtain the trained network model.

[0037] The network model training includes image data forward propagation and gradient back propagation; the details are as follows.

[0038] (1) Image data is forward propagated. All normal images in the training set are decoupled by the frequency domain decoupling module and then sent to the network for training; the network contains a generator and a discriminator.

[0039] The generator encodes and decodes the input normal image to reconstruct the input normal image. The discriminator distinguishes between the generated image and the original input normal image. During training, the generator's goal is to generate realistic images as much as possible to deceive the discriminator. The discriminator's goal, on the other hand, is to distinguish the generator's images from real images. The generator and the discriminator form a dynamic "game."

[0040] The generator includes a high- and low-frequency reconstruction module and a channel selection module. In the high- and low-frequency reconstruction module, the generator consists of two branches: a high-frequency branch and a low-frequency branch, which reconstruct the high-frequency and low-frequency components of a normal image, respectively, to achieve the rational use of the image's high- and low-frequency information. In the channel selection module, during the encoding process, the generator adds feature maps from different frequency bands, passes them through a global average pooling layer, a fully connected layer, and a softmax layer, to obtain the channel attention vectors corresponding to the features of different frequency bands. The channel attention vectors are multiplied by the original feature map to obtain a new feature map that has passed through the channel selection module, i.e., the reconstructed image.

[0041] (2) Gradient back propagation: The game between the generator and the discriminator constitutes AdversarialLoss, the difference between the reconstructed image reconstructed by the generator and the original input normal image constitutes ContextualLoss, and the difference between the reconstructed image reconstructed by the generator and the original normal image in the latent space constitutes LatentLoss; the gradient is calculated according to the loss function, the gradient back propagation updates the network parameters, and the GPU is used for acceleration until the network error is reduced to within the set threshold or the number of network iterations meets the requirements, and the training is stopped to obtain a trained network model.

[0042] The network is optimized using the Adam optimizer. We set the baseline learning rate to 0.002 and use a multi-step learning rate scheduler with a momentum of 0.999.

[0043] The loss functions are:

[0044] The loss function used by AdversarialLoss is:

[0045]

[0046] The loss function used by ContextualLoss is:

[0047]

[0048] The loss function used by LatentLoss is:

[0049]

[0050] in:

[0051] x is a normal image,

[0052] G(x) is the reconstructed image,

[0053] D(x) is the discriminator output,

[0054] f(x) is the feature map of the latent space.

[0055] The gradient is a vector summing the partial derivatives of all variables in the loss function. The direction indicated by the gradient is the direction in which the function value at each point decreases the most. Gradient backpropagation updates the weights of each layer in the GAN until the overall network loss (i.e., the sum of the three loss functions) falls below a set threshold (e.g., 0.5) or the number of network iterations meets the required threshold. Training is terminated, resulting in a trained network model.

[0056] In the model inference phase, the network model obtained in the model training phase is used to determine whether a new test image is an abnormal image based on the anomaly score. Specifically, it includes:

[0057] S21, passing the test image through a frequency domain decoupling module to obtain high-frequency components and low-frequency components of the test image;

[0058] S22. The high-frequency and low-frequency components of the test image are fed into the trained network model to obtain the reconstruction error and the latent space error. The two errors are weighted averaged and fused to obtain the anomaly score inferred by the final model:

[0059] Anomaly(x)=λL con +(1-λ)L lat

[0060] Among them L con is the reconstruction error; L lat is the latent space error; λ is the weight parameter, set to 0.9;

[0061] The test image is judged as an abnormal image based on the set threshold. Specifically, the threshold can be set to 0.5. If the anomaly score is greater than or equal to 0.5, it is judged as an abnormal image, and if it is less than 0.5, it is judged as a normal image.

[0062] The following is a detailed explanation based on the above principles. Figure 1 The figure shows a flowchart of an image anomaly detection method based on high- and low-frequency reconstruction. The method uses the MVTecAD and CIFAR-10 datasets. The MVTecAD dataset contains 5,354 high-resolution images, encompassing 10 categories of objects and 5 categories of textures. The training data for each category ranges from 60 to 320 images. The anomaly data in the test data includes over 70 types of defects, such as cracks and holes. The CIFAR-10 dataset contains 60,000 images with a resolution of 32x32. Anomaly detection on this dataset treats images in one category as anomalies and images in other categories as normal.

[0063] According to the attached Figure 1 Build a network model of the image anomaly detection network with high and low frequency reconstruction, and send the information of different frequency bands of the normal image obtained by the frequency domain decoupling module into the network model. Use the channel selection module in the encoder part of the network generator to realize the selection and interaction of information of different frequency bands, and make full use of the information of different frequency bands of the image.

[0064] according to Figure 2 The channel selection of the encoder part in the network generator is implemented in the form of , the loss is calculated according to the three loss functions mentioned above, and the parameters are iteratively updated according to the gradient backpropagation method, and the GPU is used for acceleration until the network error is reduced to within the set threshold or the number of network iterations meets the requirements. Stop training.

[0065] By employing this method for image anomaly detection, compared to anomaly detection algorithms that directly reconstruct images, this method achieves frequency-domain decoupling of the image, obtaining and reconstructing information from different frequency bands, thereby enabling differentiated utilization of information from different frequency bands. Furthermore, using a channel selection module, this method enables interaction and selection of information from different frequency bands during the encoding process of the network generator. This method effectively enhances the reconstruction capabilities of the generator in reconstruction-based anomaly detection methods, improving the anomaly detection effectiveness of such methods.

[0066] This embodiment involves the attention that the above is only a preferred embodiment of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for image anomaly detection based on high- and low-frequency reconstruction, characterized by: It includes two stages: model training and model inference. In the model training phase, a generative adversarial network is trained on normal images in the training set, and a network model of the generative adversarial network is trained using a server. The network parameters are optimized by reducing the network loss function until the network converges, thereby obtaining a network model based on high and low frequency reconstruction. In the model inference phase, the network model obtained in the model training phase is used to determine whether a new test image is an abnormal image based on the anomaly score; The model training phase specifically includes the following steps: S11. Image data preparation: using the frequency domain decoupling module to obtain information of different frequency bands of normal images in the training set; S12. Network model training: The information of different frequency bands of the normal image is fed into the generator for reconstruction. The discriminator is used to distinguish the original image from the reconstructed image, and adversarial learning is used for training. The loss function is calculated to obtain the gradient, and then backpropagation is performed to obtain the trained network model. The generator includes a high- and low-frequency reconstruction module and a channel selection module. The high- and low-frequency reconstruction module includes multiple branches of different frequency bands, which reconstruct information of different frequency bands of the image respectively. The channel selection module adds the feature maps of different frequency bands of the image, obtains the channel attention vectors corresponding to the features of different frequency bands through the global average pooling layer, the fully connected layer and the softmax, multiplies the channel attention vector with the original feature map to obtain a new feature map, and the new feature map serves as the input of the next convolutional layer.

2. The image anomaly detection method based on high- and low-frequency reconstruction according to claim 1, characterized in that: The frequency domain decoupling module in step S11 specifically uses a Gaussian convolution kernel to obtain the low-frequency component of the normal image, uses the difference between the original normal image and the low-frequency component to obtain the high-frequency component of the image, and finally obtains information of different frequency bands of the image.

3. The image anomaly detection method based on high- and low-frequency reconstruction according to claim 1, characterized in that: The network model training in step S12 includes image data forward propagation and gradient back propagation. In the forward propagation of the image data, all normal images in the training set are decoupled by the frequency domain decoupling module and then sent to the network including the generator and the discriminator for training; In the gradient back propagation, the game between the generator and the discriminator constitutes AdversarialLoss, the difference between the reconstructed image reconstructed by the generator and the original normal image constitutes ContextualLoss, and the difference between the reconstructed image reconstructed by the generator and the original normal image in the latent space constitutes LatentLoss; the gradient is calculated according to the loss function, the gradient back propagation updates the network parameters, and the GPU is used for acceleration until the network error is reduced to within the set threshold or the number of network iterations meets the requirements, and the training is stopped to obtain a trained network model.

4. The image anomaly detection method based on high- and low-frequency reconstruction according to claim 3, characterized in that: The generator is used to encode and then decode the input image to achieve a reconstruction process of the input image; Alternatively, the discriminator is used to discriminate between the reconstructed image generated by the generator and the original normal image.

5. The image anomaly detection method based on high- and low-frequency reconstruction according to claim 3, characterized in that: The loss function used by the AdversarialLoss is: The loss function used by the ContextualLoss is: The loss function used by the LatentLoss is: in: x is a normal image, G(x) is the reconstructed image, D(x) is the discriminator output, f(x) is the feature map of the latent space.

6. The image anomaly detection method based on high- and low-frequency reconstruction according to claim 1, characterized in that: The model inference stage specifically includes the following steps: S21, passing the test image through a frequency domain decoupling module to obtain information of different frequency bands of the test image; S22. Information from different frequency bands of the test image is passed into the trained network model to obtain reconstruction error and latent space error. The reconstruction error and latent space error are weighted averaged and fused to obtain the anomaly score inferred by the final model. Whether the test image is an abnormal image is determined based on a preset threshold.

Images