An image anomaly detection method based on a channel attention registration network

Abstract

The application discloses an image anomaly detection method based on a channel attention registration network. A channel attention module for sensing the importance of high-dimensional feature channels is first constructed, and then a weight-shared anomaly detection network is constructed. The channel attention module is embedded in the feature extraction stage of the anomaly detection network, and a multivariate Gaussian distribution probability expression is constructed for normal samples based on the extracted image high-dimensional features. In the test stage, the query image to be detected and the template image are simultaneously input into the anomaly detection network, the Mahalanobis distance between the splicing features of each pixel position of the query image and the multivariate Gaussian distribution of the template image is calculated, and the anomaly is judged according to the position with a larger Mahalanobis distance. The method can avoid the dependence of the network on object categories, can be effectively generalized to the anomaly detection task of unknown object categories, can enhance the difference perception of the network to the feature channels, can improve the extraction capability of the network to the multi-dimensional features of objects, and can effectively suppress the interference of noise on the anomaly detection.

Description

TECHNICAL FIELD

[0001] The application relates to an image detection method in the field of computer vision, in particular to an image anomaly detection method based on a channel attention registration network. BACKGROUND

[0002] The main task of anomaly detection is to identify infrequent situations different from the normal mode, and is widely applied to the fields of video monitoring, product quality control, medical diagnosis and the like. Traditional anomaly detection methods mainly realize anomaly detection of specific objects by designing task-related features. However, the manual features for specific tasks cannot be applied to new anomaly detection scenes, which greatly limits the application range of the detection method. In recent years, the deep learning method which has attracted wide attention can autonomously extract image features from data, avoiding the shortcomings of manually designed features, and opening up a new direction for anomaly detection.

[0003] However, most of the current methods are only designed for specific object categories, and a deep learning model can only be applied to one object category, which is not conducive to the application expansion of the model.

[0004] In addition, most deep learning networks for anomaly detection pay more attention to the reconstruction of features, and do not consider the difference in feature channel extraction for multi-dimensional anomaly features, which is not conducive to the improvement of anomaly detection accuracy. SUMMARY

[0005] The application aims to solve the problems in the background art and provides an image anomaly detection method based on a channel attention registration network. The corresponding registration features of objects are extracted by constructing a weight-sharing registration network, avoiding the dependence of the network on object categories and enhancing the adaptability of the network to different object categories. In addition, the channel attention module is constructed to fully consider the attention of different channels to different anomaly features, describe the object anomaly features from multiple dimensions, eliminate the influence of noise on anomaly feature registration, and improve the extraction ability of the network to registration-related features.

[0006] To achieve the above purpose, the technical scheme adopted by the application is as follows:

[0007] 1) Construct an anomaly detection network with a channel attention module;

[0008] 2) Train the anomaly detection network by using a loss calculation module;

[0009] 3) Use the trained anomaly detection network to detect the input query image.

[0010] The channel attention registration network refers to a network formed by integrating a channel attention module into the framework of an image registration network.

[0011] The method for implementation includes three parts of constructing a channel attention module, constructing a loss calculation module, and constructing an anomaly detection network.

[0012] The constructed anomaly detection network based on channel attention registration is as shown in Figure 2 .

[0013] The anomaly detection network includes three convolution modules and three channel attention modules; the query image and the template image are processed by the respective first convolution modules to obtain the respective first convolution features QF2 and TF2, and the first convolution features QF2 and TF2 of the query image and the template image are processed by the first channel attention modules to obtain the respective first channel attention features QA2 and TA2; the first channel attention features QA2 and TA2 of the query image and the template image are processed by the second convolution modules to obtain the respective second convolution features QF3 and TF3, and the second convolution features QF3 and TF3 of the query image and the template image are processed by the second channel attention modules to obtain the respective second channel attention features QA3 and TA3; the second channel attention features QA3 and TA3 of the query image and the template image are processed by the third convolution modules to obtain the respective third convolution features QF4 and TF4, and the third convolution features QF4 and TF4 of the query image and the template image are processed by the third channel attention modules to obtain the respective third channel attention features QA4 and TA4.

[0014] The respective convolution operations and channel attention modules in the network share the weights.

[0015] The weight sharing means that the weights are the same. The weights refer to the updateable learning parameters in the channel attention module.

[0016] The first convolution module is mainly composed of two convolution operations Conv1 and Conv2 connected in sequence, the second convolution module is mainly composed of one convolution operation Conv3 connected in sequence, and the third convolution module is mainly composed of one convolution operation Conv4 connected in sequence.

[0017] The constructed channel attention module is mainly aimed at the high-dimensional features of the images extracted by the network, as shown in Figure 3 .

[0018] The first channel attention module, the second channel attention module, and the third channel attention module have the same structure and each include a global average pooling layer, a full connection layer, a Sigmoid activation function, a matrix multiplication operation, and a convolution layer; the input convolution features QF2 and TF2 are processed by the first channel attention module to obtain the respective first channel attention features.

[0019] In the training process of step 2), a portion of non-abnormal images are taken as query images and another portion of non-abnormal images are taken as template images to construct image matching pairs. The constructed image matching pairs are input into the anomaly detection network for training. The first network loss L2 is obtained by calculating the loss through the loss calculation module based on the first channel attention features QA2 and TA2 of the query images and template images, the second network loss L3 is obtained by calculating the loss through the loss calculation module based on the second channel attention features QA3 and TA3 of the query images and template images, and the third network loss L4 is obtained by calculating the loss through the loss calculation module based on the third channel attention features QA4 and TA4 of the query images and template images. Finally, the first network loss L2, the second network loss L3, and the third network loss L4 are summed to obtain the total network loss L.

[0020] The training and optimization are carried out with the goal of minimizing the total network loss L, so that the anomaly detection network reaches its optimal state.

[0021] The loss calculation module specifically processes the following steps: Regularization is performed on the channel attention features of both the query image and the template image at each pixel position (i,j) to obtain a regularized vector. and

[0022]

[0023]

[0024] in, This represents the regularized vector of the channel attention features of the query image at pixel position (i,j). This represents the regularized vector of the channel attention features of the template image at pixel position (i,j); QA ij This indicates the value of the channel attention feature (TA) of the query image at pixel position (i,j). ij This represents the value of the channel attention feature of the template image at pixel position (i,j); Represents the L2 norm;

[0025] Then, at each pixel position (i,j), the regularization vector is applied... and Calculate the loss l between the query image and the template image at pixel position (i,j). ij :

[0026]

[0027] The loss at all pixel positions (i,j) is combined. ij Calculate the network loss between the channel attention features of the query image and the template image. Output:

[0028]

[0029] wherein H represents the height of the channel attention feature, and W represents the width of the channel attention feature.

[0030] The step 3) is specifically:

[0031] 3.1) selecting N query images belonging to normal samples (i.e. non-anomalous) and non-anomalous together as input to the trained anomaly detection network model to obtain respective channel attention features, three channel attention features TA2, TA3 and TA4 can be extracted for each query image, the three channel attention features of the N query images are spliced according to the respective channels to obtain the spliced features of each query image, and the multivariate Gaussian distribution at each pixel position of the query image is calculated according to the spliced features of the N query images;

[0032] 3.2) inputting the query image to be detected into the trained anomaly detection network model to obtain three channel attention features QA2, QA3 and QA4 of the query image to be detected, and splicing the three channel attention features of the query image to be detected to obtain the spliced features of the query image to be detected;

[0033] 3.3) calculating the Mahalanobis distance between the spliced features at each pixel position of the query image to be detected and the multivariate Gaussian distribution at the same position, constructing the Mahalanobis distance at each pixel position into an anomaly score map of the query image to be detected, taking the maximum value on the anomaly score map as the anomaly score of the query image to be detected, and then making the following judgment:

[0034] When the anomaly score exceeds the preset anomaly threshold, the query image to be detected is considered as an abnormal image; otherwise, the query image to be detected is not considered as an abnormal image.

[0035] The anomaly of the present application refers to a situation different from the situation under most of the time, which is defined by the specific needs.

[0036] The method of the present application can be applied to images in the fields of video monitoring, product quality control, medical diagnosis, etc.

[0037] When applied to video monitoring images, the abnormal behavior or movement of moving objects in the images can be detected.

[0038] When applied to product quality control images, defects of products in the images can be detected.

[0039] When applied to medical diagnosis images, diseases in the images can be detected.

[0040] The application is firstly to construct a channel attention module which can perceive the importance of high-dimensional feature channels, then to construct a weight-shared deep learning anomaly detection network, embed the channel attention module into the feature extraction stage of the anomaly detection network, and enhance the difference extraction ability of the registration network to the feature channels. Finally, based on the extracted high-dimensional features of the image, a multivariate Gaussian distribution probability expression is constructed for the normal samples. In the test stage, the query image to be detected and the normal sample image used as the reference template are input into the registration anomaly detection network at the same time, the Mahalanobis distance between the stitching features of each pixel position of the query image and the multivariate Gaussian distribution of the normal sample is calculated, and the Mahalanobis distance at each pixel position of the query image constitutes an anomaly score map of the test query image. The maximum value on the anomaly score map is taken as the anomaly score of the test image. When the anomaly score exceeds a specified threshold, the test query image is considered as an abnormal image.

[0041] The beneficial effects of the application are:

[0042] The method of the application can be applied to anomaly detection tasks of various different object categories, improves the anomaly detection ability of the network for new objects, can avoid the dependence of the network on object categories, can be effectively generalized to anomaly detection tasks of unknown object categories, and avoids the problem of constructing a detection model for each object category.

[0043] Meanwhile, the channel attention module designed by the application can improve the extraction ability of the feature channels to different abnormal features, can enhance the difference perception of the network to the feature channels, improve the extraction ability of the network to the multi-dimensional features of the object, effectively suppress the interference of noise on the anomaly detection, avoid the adverse effects of noise, outliers and the like on the key registration features, and is beneficial to improve the registration accuracy and anomaly detection accuracy of the network. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of the method of the application;

[0045] Figure 2 is a whole logic block diagram of the anomaly detection network of the application;

[0046] Figure 3 is a structural schematic diagram of the channel attention module of the application. DETAILED DESCRIPTION

[0047] The embodiments of the application will be described in detail below with reference to the accompanying drawings.

[0048] It should be understood that the embodiments described in the application are exemplary, and the specific parameters used in the embodiment description are only for the purpose of describing the application and do not limit the application.

[0049] As Figure 1As shown, the specific embodiments of the present application and the implementation process thereof include the following steps:

[0050] Step 1: Construct a channel attention module, construct a channel attention module as shown. Figure 3

[0051] The first channel attention module, the second channel attention module and the third channel attention module are the same in structure, and all include a global average pooling layer, a full connection layer, a Sigmoid activation function, a matrix multiplication operation and a convolution layer; the input convolution features QF2 and TF2 respectively pass through the first channel attention module to obtain respective first channel attention features.

[0052] In specific implementation, the input convolution features are image high-dimensional features F, with a dimension of BxCxHxW, B represents the number of images input to the network, C represents the number of feature channels, H represents the height of the feature, and W represents the width of the feature; the image high-dimensional features F pass through the global average pooling layer to obtain the pooling features PF, with a dimension of BxC; the pooling features PF pass through two layers of full connection layers fc(C / 4,C) and Sigmoid activation functions in sequence to obtain the channel weight matrix W with a dimension of BxC, wherein the first layer of the full connection layer fc(C / 4,C) has a node number of C / 4, and the second layer has a node number of C; finally, the channel weight matrix W performs a matrix multiplication operation with the image high-dimensional features F, and then passes through a convolution layer with a convolution kernel of 1x1 to obtain the channel attention features AF, with a dimension of BxCxHxW.

[0053] In specific implementation, the three channel attention modules constructed are respectively embedded after the convolution operations Conv2, Conv3 and Conv4. The channel attention module B after Conv2 has a value of 32, C has a value of 64, and H and W both have a value of 56; the channel attention module B after Conv3 has a value of 32, C has a value of 128, and H and W both have a value of 28; the channel attention module B after Conv4 has a value of 32, C has a value of 256, and H and W both have a value of 14.

[0054] Step 2: Construct a loss calculation module.

[0055] The features obtained in the present application are actually a feature map.

[0056] Given a pair of query images and template images, the channel attention features extracted from the query images are QA, and the channel attention features extracted from the template images are TA, both with a dimension of HxWxC. The query feature vector and the template feature vector at the pixel position (i,j) in the channel attention features TA are QA ij and TA ij . When calculating the loss at the pixel position (i,j), first, the query feature vector QA ij and the template feature vector TA ij ​The regularization obtains respective regularization vectors, and then a loss function is constructed on the basis of L2 distance between the two regularization vectors. The loss function construction step between the channel attention feature QA and TA is:

[0057] The regularization vector at the pixel position (i, j) corresponding to the query image and the template image is constructed and

[0058]

[0059]

[0060] The loss l(I) at the pixel position (i, j) corresponding to the query image and the template image is calculated ij .

[0061]

[0062] The network loss l(I) between the channel attention features QA and TA is calculated.

[0063]

[0064] In the specific implementation, the anomaly detection network losses L2, L3 and L4 are obtained, the three losses obtained are summed to obtain the total loss L of the network.

[0065] Step 3: An anomaly detection network is constructed according to the channel attention module and the loss calculation module, and the anomaly detection network as shown in Figure 2 is constructed.

[0066] The anomaly detection network includes three convolution modules and three channel attention modules; the query image and the template image are respectively input into the anomaly detection network, and after being processed by respective first convolution modules, respective first convolution features QF2 and TF2 are obtained. The first convolution features QF2 and TF2 of the query image and the template image respectively pass through the first channel attention module to obtain respective first channel attention features QA2 and TA2; the first channel attention features QA2 and TA2 of the query image and the template image respectively pass through the second convolution module to obtain respective second convolution features QF3 and TF3, and the second convolution features QF3 and TF3 of the query image and the template image respectively pass through the second channel attention module to obtain respective second channel attention features QA3 and TA3; the second channel attention features QA3 and TA3 of the query image and the template image respectively pass through the third convolution module to obtain respective third convolution features QF4 and TF4, and the third convolution features QF4 and TF4 of the query image and the template image respectively pass through the third channel attention module to obtain respective third channel attention features QA4 and TA4.

[0067] It should be noted that the corresponding respective convolution operations and channel attention modules in the network share weights.

[0068] Specifically, the weights of the first channel attention module of the query image and the template image are shared, the weights of the second channel attention module of the query image and the template image are shared, and the weights of the third channel attention module of the query image and the template image are shared; and the weights of the first convolution module of the query image and the template image are shared, the weights of the second convolution module of the query image and the template image are shared, and the weights of the third convolution module of the query image and the template image are shared.

[0069] In the first convolution module of the query image and the template image, the first convolution operation in the first convolution module of the query image and the template image shares weights with each other, and the second convolution operation in the first convolution module of the query image and the template image shares weights with each other.

[0070] The first convolution module is mainly composed of two convolution operations Conv1 and Conv2 connected in sequence, the second convolution module is mainly composed of one convolution operation Conv3 connected in sequence, and the third convolution module is mainly composed of one convolution operation Conv4 connected in sequence.

[0071] In a specific implementation, the input image size is 224x224, and ResNet18 is used as a benchmark registration network.

[0072]

[0073] For example, the operation corresponding to Conv2 , where the number in the bracket outside the bracket represents that the number of convolution block stacks is 2. The 3x3 in the convolution block represents the size of the convolution kernel, and 64 represents the number of convolution kernels.

[0074] Step 4: Train the constructed anomaly detection network. The constructed anomaly detection network is trained using the MVTec anomaly detection dataset. The dataset contains more than 5000 high-resolution images covering 15 different object types. The initial learning rate is set to 0.0001, the training batch size is 32, and the maximum number of iterations is 50 epochs. Take a part of the normal images as query images, and the other part of the normal images as template images to construct image matching pairs. The constructed image matching pairs are input into the anomaly detection network for training.

[0075] Step 5: Predict the abnormal area of the object.

[0076] In a specific implementation, after the anomaly network is trained, the input query image to be detected can be detected for anomaly:

[0077] Firstly, N normal sample query images are selected and input into the trained abnormality detection network model to obtain channel attention features, each query image can extract three channel attention features TA2, TA3 and TA4, the three channel attention features of the N query images are spliced according to the respective channels to obtain the spliced features of each query image, and the multivariate Gaussian distribution of each pixel position of the query image is calculated according to the spliced features of the N query images;

[0078] Then, the query image to be detected is input into the trained abnormality detection network model to obtain three channel attention features QA2, QA3 and QA4 of the query image to be detected, and the three channel attention features of the query image to be detected are spliced according to the channels to obtain the spliced features of the query image to be detected.

[0079] Finally, the Mahalanobis distance between the spliced features of each pixel position of the query image to be detected and the multivariate Gaussian distribution at the corresponding pixel position is calculated, the Mahalanobis distances at each pixel position are constructed to form an abnormality score map of the query image to be detected, the maximum value on the abnormality score map is taken as the abnormality score of the query image to be detected, and the following judgment is made:

[0080] When the abnormality score exceeds the preset abnormality threshold, the query image to be detected is considered as an abnormal image; otherwise, the query image to be detected is not considered as an abnormal image.

[0081] In a specific implementation, the number N of normal samples is set to 10. The trained abnormality detection network model is used to construct the multivariate Gaussian distribution at each pixel position of the normal sample, and the spliced features of the test query sample at each pixel position are calculated, and then the abnormality score image of the test query sample is calculated, and the pixel positions higher than the threshold in the abnormality score image are aggregated into a region, and the region is the abnormal region.

[0082] Compared with the prior art, the present application proposes an image abnormality detection method based on a channel attention registration network, constructs an abnormality feature extraction method based on a network registration framework, designs a feature channel attention module based on weights, and constructs a network loss function based on multi-level registration features. The method pays more attention to the registration characteristics of the object, weakens the dependence on the object category, greatly expands the application range of the abnormality detection method. The designed feature attention module can enhance the network feature to capture different abnormal information, overcome the adverse effects of noise on key abnormal features, and improve the abnormality detection accuracy of different object categories. On the MVTec abnormality detection dataset, the present application obtains an image abnormality detection accuracy of 78.9% for unknown object categories. Compared with the similar method TDG, the detection accuracy is improved by 7.8%.

[0083] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited to the above-described embodiments, any equivalent replacement or change made by any skilled person in the art according to the technical solutions and inventive concept of the present application within the technical scope described by the present application should be included in the protection scope of the present application.

Claims

1. An image anomaly detection method based on a channel attention registration network, characterized in that: 1) Construct an anomaly detection network with a channel attention module; 2) The anomaly detection network is trained using the loss calculation module; 3) Use the trained anomaly detection network to perform anomaly detection on the input query image; Step 3) specifically refers to: 3.1) Select N query images that are normal samples and not abnormal and input them into the trained anomaly detection network model to obtain their respective channel attention features. Each query image can extract three channel attention features TA2, TA3, and TA4. After concatenating the three channel attention features of these N query images according to their respective channels, the concatenated features of each query image are obtained. Then, the multivariate Gaussian distribution at each pixel position of the query image is calculated based on the concatenated features of the N query images. 3.2) Input the query image to be detected into the trained anomaly detection network model to obtain the three channel attention features QA2, QA3, and QA4 of the query image to be detected. Then, concatenate the three attention features of the query image to be detected by channel to obtain the concatenated features of the query image to be detected. 3.3) Calculate the Mahalanobis distance between the stitched features at each pixel position of the query image to be detected and the multivariate Gaussian distribution at the same position. Construct an anomaly score map of the query image to be detected using the Mahalanobis distances at each pixel position. Take the maximum value on the anomaly score map as the anomaly score of the query image to be detected, and then make the following judgment: If the abnormal score exceeds the preset abnormal threshold, the image to be detected is considered an abnormal image; otherwise, the image to be detected is not considered an abnormal image. The anomaly detection network includes three convolutional modules and three channel attention modules. The query image and template image are first processed by their respective first convolutional modules to obtain their respective first convolutional features QF2 and TF2. The first convolutional features QF2 and TF2 of the query image and template image are then processed by the first channel attention modules to obtain their respective first channel attention features QA2 and TA2. The first channel attention features QA2 and TA2 of the query image and template image are then processed by the second convolutional modules to obtain their respective second convolutional features QF3 and TF3. The second convolutional features QF3 and TF3 of the query image and template image are then processed by the second channel attention modules to obtain their respective second channel attention features QA3 and TA3. The second channel attention features QA3 and TA3 of the query image and template image are then processed by the third convolutional module to obtain their respective third convolutional features QF4 and TF4. The third convolutional features QF4 and TF4 of the query image and template image are then processed by the third channel attention modules to obtain their respective third channel attention features QA4 and TA4.

2. The image anomaly detection method based on a channel attention registration network according to claim 1, characterized in that: The first convolution module is mainly composed of two convolution operations, Conv1 and Conv2, connected in sequence; the second convolution module is mainly composed of one convolution operation, Conv3, connected in sequence; and the third convolution module is mainly composed of one convolution operation, Conv4, connected in sequence.

3. The image anomaly detection method based on a channel attention registration network according to claim 1, characterized in that: The first, second, and third channel attention modules have the same structure, each including a global average pooling layer, a fully connected layer, a sigmoid activation function, a matrix multiplication operation, and a convolutional layer; the input convolutional features QF2 and TF2 are respectively processed by the first channel attention module to obtain their respective first channel attention features.

4. The image anomaly detection method based on a channel attention registration network according to claim 1, characterized in that: In the training process of step 2), a portion of non-abnormal images are taken as query images and another portion of non-abnormal images are taken as template images to construct image matching pairs. The constructed image matching pairs are input into the anomaly detection network for training. The first network loss L2 is obtained by calculating the loss through the loss calculation module based on the first channel attention features QA2 and TA2 of the query images and template images, the second network loss L3 is obtained by calculating the loss through the loss calculation module based on the second channel attention features QA3 and TA3 of the query images and template images, and the third network loss L4 is obtained by calculating the loss through the loss calculation module based on the third channel attention features QA4 and TA4 of the query images and template images. Finally, the first network loss L2, the second network loss L3, and the third network loss L4 are summed to obtain the total network loss L. The training and optimization are carried out with the goal of minimizing the total network loss L, so that the anomaly detection network reaches its optimal state.

5. The image anomaly detection method based on a channel attention registration network according to claim 4, characterized in that: The loss calculation module specifically processes the following steps: Regularization is performed on the channel attention features of both the query image and the template image at each pixel position (i,j) to obtain a regularized vector. and in, This represents the regularized vector of the channel attention features of the query image at pixel position (i,j). This represents the regularized vector of the channel attention features of the template image at pixel position (i,j); QA ij This indicates the value of the channel attention feature (TA) of the query image at pixel position (i,j). ij This represents the value of the channel attention feature of the template image at pixel position (i,j); Represents the L2 norm; Then, at each pixel position (i,j), the regularization vector is applied... and Calculate the loss l between the query image and the template image at pixel position (i,j). ij : The loss at all pixel positions (i,j) is combined. ij Calculate the network loss between the channel attention features of the query image and the template image. Output: Where H represents the height of the channel attention feature and W represents the width of the channel attention feature.

Images