An image anomaly detection and positioning method based on priori guidance multi semantic fusion

Abstract

The present application relates to the technical field of computer vision processing, and particularly relates to an image anomaly detection and positioning method based on prior guide multi semantic fusion, and implementation of the method includes the following contents: abstract global semantics of normal images are extracted through a pre-trained visual-language network, then a learnable semantic encoding library is constructed, and representative feature vectors of normal samples are stored through vector discretization. Finally, after the above multi semantic features are fused, the fused multi semantic features are taken as inputs of a decoder, and an abnormal image is guided to be reconstructed as a normal image, so that the abnormality is detected and positioned. The present application can accurately detect and position the existing abnormality in an image without any abnormal prior, and greatly improves detection efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision processing, and particularly provides an image anomaly detection and positioning method based on priori guided multi-semantics fusion. BACKGROUND

[0002] Unsupervised anomaly detection refers to detecting and positioning anomalies without any priori knowledge of anomalies. In recent years, unsupervised anomaly detection has attracted more and more attention and has been widely applied to many fields of computer vision, such as medical disease diagnosis, industrial defect detection, and road safety monitoring. Since the surface defects of industrial products have adverse effects on product safety and usability, anomaly detection technology is particularly needed in the industrial field. In addition, in the medical field, anomaly detection can help doctors better diagnose the anomalies existing in medical images.

[0003] Unsupervised anomaly detection methods can generally successfully find anomalies by detecting appearance and behavioral deviations of anomalies. These deviations are mainly divided into two categories: structural anomalies and logical anomalies. Structural anomalies refer to differences in new local structures from normal conditions, while logical anomalies refer to contents that violate the basic logic or geometric constraints of normal samples. The method of embedding a pre-trained model into an anomaly detection model and classifying the features extracted by the network is widely used and can achieve good results. However, for logical anomalies, this method is not very effective.

[0004] The encoder-decoder architecture has great potential and can capture global context information by compressing the input image into a low-dimensional bottleneck, thereby capturing logical anomalies. However, there are still some important problems that have not been solved. Although the network is trained on an anomaly-free dataset, due to its generalization ability, it has the innate ability to capture anomalies of test samples, especially when trained on a dataset with complex semantics. Therefore, there may still be abnormal semantics in the bottleneck of the input decoder, thereby reducing the representation difference of the encoder and the decoder on the anomaly. Secondly, the decoder attempts to reconstruct a high-dimensional representation from a low-dimensional bottleneck. However, due to the lack of different hierarchical features in the low-dimensional bottleneck, the reconstruction is always ambiguous and inaccurate. SUMMARY

[0005] The technical task of the present application is to solve the above-mentioned problems, and provide an image anomaly detection and positioning method based on priori guided multi-semantics fusion.

[0006] The present application provides the following technical solutions:

[0007] An image anomaly detection and positioning method based on priori guided multi-semantics fusion, the implementation of the method includes the following contents:

[0008] The abstract global semantic features of the normal image are extracted by the pre-trained visual-linguistic network;

[0009] The features of the normal image are extracted by the teacher encoder network trained by the external dataset;

[0010] The features extracted by each layer of the teacher network are replaced by vector discretization and the construction of a learnable semantic encoding library;

[0011] After the above multi-semantic features are fused, they are used as the input of the student decoder network to guide the reconstruction of the abnormal image to normal, so as to detect and locate the abnormality.

[0012] The method realizes the following steps:

[0013] Step 1, obtaining image data to be detected, and performing pre-processing operation on the image;

[0014] Step 2, inputting the pre-processed image into the pre-trained teacher encoder network to obtain the global image features;

[0015] Step 3, inputting the text label of the image into the pre-trained visual-linguistic network to extract the abstract global semantic features of the normal image;

[0016] Step 4, discretizing the features of each layer in the teacher encoder network, and constructing a learnable semantic encoding library to replace the extracted features;

[0017] Step 5, fusing the above features and sending them into the student decoder for distillation training;

[0018] Step 6, inputting the test data into the pre-trained network and the pre-trained visual-linguistic network for testing;

[0019] Step 7, calculating the reconstruction error of the test data to detect and locate the abnormality.

[0020] The step 1 includes the following steps:

[0021] Step 11, collecting image data, defining the original image as X∈R C×H×W ; wherein C represents the image dimension, H represents the height of the image, and W represents the width of the image;

[0022] Step 12, performing size scaling processing on the image data, and the formula is as follows:

[0023]

[0024]

[0025] where S(x, y) represents the corresponding pixel point in the scaled image, (x, y) is the pixel point coordinate, X represents the original image, X w and X h represent the width and height of the original image, S w and S h represent the width and height of the scaled image.

[0026] Step 13, normalize the image data, the formula is as follows:

[0027]

[0028] where S max , S min represent the maximum and minimum values of all pixel values in the image, respectively.

[0029] Step 14, standardize the image data, the formula is as follows:

[0030]

[0031] where mean(·) represents the mean value of each channel, and std(·) represents the standard deviation of each channel.

[0032] The step 2 includes the following steps:

[0033] Step 21, obtain the global feature of the image, which is specifically represented as:

[0034]

[0035] where x i represents the data in the data set, represents the pre-trained teacher encoder network.

[0036] Step 22, further down-sampling the global feature f ig to obtain the global feature F ig ∈R C×H×W .

[0037] The step 3 includes the following steps:

[0038] Step 31, obtain the global semantic feature of the image, which is specifically represented as:

[0039]

[0040] where t i represents the label category of the image in the data set, represents the pre-trained visual-linguistic network, t i ∈R 1×512 .

[0041] Step 32, temperature expansion and up-sampling are performed on the global semantic features, so that the size is consistent with the image global feature size, and the feature T is obtained i ∈R C×H×W .

[0042] The step 4 includes the following steps:

[0043] Step 41, the features of each layer in the teacher encoder network are obtained, which is specifically represented as:

[0044]

[0045] Wherein, x i represents the data in the data set, represents a pre-trained network, l represents the features of different layers, and l is selected as 1, 2, 3 and 4.

[0046] Step 42, the features of each layer are discretized, and a learnable semantic code library is constructed, which is specifically represented as:

[0047]

[0048] Wherein, V l ∈R k×c , represents k dimensions of c f i,l (p,q) represents the image features at different layer (p,q) positions, represents the features in . Through the above operation, the output characteristics i,l (p,q)' are finally obtained by aggregating these feature vectors f , and the optimization objective of the reparameterization module is:

[0049]

[0050] Wherein, sg[ ] represents the stop gradient operator.

[0051] The step 5 includes the following steps:

[0052] Step 51, the global semantic features T i and the image global features F ig are fused:

[0053] d i =T i ⊕F ig

[0054] Wherein, ⊕ represents splicing.

[0055] Step 52, the fused features D iis sent into the LeWinBlocks attention mechanism module and is up-sampled to obtain the feature D i .

[0056] Step 53, the global semantic feature T i is up-sampled and the feature of the corresponding layer f i,l is fused:

[0057]

[0058] Step 53 is repeated until the image x' with the same size as the input is obtained.

[0059] Step 54, the student decoder network is guided to learn by using the distillation loss, and the optimization objective of the re-parameterization module is:

[0060]

[0061]

[0062] L mse = MSE(x, x'),

[0063] L total = L mse + L cos + L vg ',

[0064] The step 6 comprises the following steps:

[0065] Step 61, the test image and the corresponding text label are sent into the pre-trained teacher encoder network and the pre-trained visual-linguistic network, respectively.

[0066] Step 62, the global semantic feature and the global image feature of the test image are obtained.

[0067] Step 63, the semantic encoding library obtained in the training stage is used to replace the feature of each layer of the test image to obtain the replaced feature.

[0068] Step 64, the three features are sent into the student decoder network according to the training stage step to fuse and obtain the reconstructed test image.

[0069] The step 7 comprises the following steps:

[0070] Step 71, the reconstruction error is calculated and the difference is detected and the abnormality is located according to the pixel reconstruction of the image;

[0071] Step 72, the size of the feature map is adjusted to the resolution of the original image by bilinear interpolation, and the feature map is smoothed by using the Gaussian kernel σ = 4.

[0072] Compared with the prior art, the present application has the following outstanding beneficial effects:

[0073] The present application proposes a new normal prior guided multi semantic fusion network for unsupervised anomaly detection. The present application significantly improves the recognition ability of anomalies by improving the difference between abnormal input and its reconstruction. By fusing abstract global context information and different levels of semantics to construct the multi semantic features of normal images, it helps to reduce the reproduction of abnormal features. The performance of anomaly detection and positioning is effectively improved. The unsupervised anomaly detection method proposed by the present application has wide application prospect and can be applied to medical disease diagnosis, industrial defect detection, road safety monitoring and other fields. DETAILED DESCRIPTION

[0074] The present application will be further described in detail below in combination with embodiments.

[0075] Embodiment 1

[0076] An image anomaly detection and positioning method based on prior guided multi semantic fusion, the implementation steps of the method are as follows:

[0077] Step 1, acquire image data to be detected, and perform pre-processing operation on the image;

[0078] Step 11, collect image data, define the original image as X C×H×W ; wherein C represents the image dimension, H represents the height of the image, and W represents the width of the image;

[0079] Step 12, perform size scaling processing on the image data, the formula is as follows:

[0080]

[0081]

[0082] Wherein, S(x,y) represents the corresponding pixel point in the scaled image, (x,y) is the pixel point coordinate, X represents the original image, X w and X h represent the width and height of the original image, S w and S h represent the width and height of the scaled image;

[0083] Step 13, normalize the image data, the formula is as follows:

[0084]

[0085] Wherein, S max , S minrespectively represent the maximum and minimum values of the specific values of all pixels in the image;

[0086] Step 14, standardize the image data, the formula is as follows:

[0087]

[0088] Wherein mean(·) represents the mean value of each channel, and std(·) represents the standard deviation of each channel.

[0089] Step 2, input the preprocessed image into the pre-trained teacher encoder network to obtain the global image feature;

[0090] Step 21, obtain the global feature of the image, which is specifically represented as:

[0091]

[0092] Wherein, x i represents the data in the data set, represents the pre-trained teacher encoder network.

[0093] Step 22, further down-sampling the global feature f ig to obtain the global feature F ig ∈R C×H×W .

[0094] Step 3, input the text label of the image into the pre-trained visual-linguistic network to extract the abstract global semantic feature of the normal image;

[0095] Step 31, obtain the global semantic feature of the image, which is specifically represented as:

[0096]

[0097] Wherein, t i represents the label category of the image in the data set, represents the pre-trained visual-linguistic network, and t i ∈R 1×512 .

[0098] Step 32, temperature expansion and up-sampling are performed on the global semantic feature, so that the size is consistent with the size of the image global feature, and the feature T i ∈R C×H×W .

[0099] Step 4, discretize the features of each layer in the teacher encoder network, and construct a learnable semantic code library to replace the extracted features;

[0100] Step 41, obtain the features of each layer in the teacher encoder network, which is specifically represented as:

[0101]

[0102] where x i represents the data in the dataset, represents a pre-trained network, l represents different layers of features, and l is selected as 1, 2, 3, and 4.

[0103] Step 42, the features of each layer are discretized, and a learnable semantic code library is constructed, which is specifically represented as:

[0104]

[0105] where V l ∈R k×c , represents k dimensions of c f i,l (p,q) represents the image features at different layer (p,q) positions, represents the features in . By repeating the above operation, the final output characteristics i,l (p,q)' is obtained by aggregating these feature vectors f , and the optimization objective of the reparameterization module is:

[0106]

[0107] where sg[ ] represents the stop gradient operator.

[0108] Step 5, the above features are fused and sent into the student decoder for distillation training.

[0109] Step 51, the global semantic feature T i and the image global feature F ig are fused:

[0110] d i =T i ⊕F ig

[0111] where ⊕ represents concatenation.

[0112] Step 52, the fused feature D i is sent to the LeWinBlocks attention mechanism module, and upsampling operation is performed to obtain the feature D i .

[0113] Step 53, the global semantic feature T i is upsampled and fused with the feature and the corresponding layer feature f i,l (p,q)' is fused:

[0114]

[0115] Repeat step 53 until the image x' of the same size as the input is obtained.

[0116] Step 54, use the distillation loss to guide the student decoder network to learn, and the optimization target of the reparameterization module is:

[0117]

[0118]

[0119] L mse = MSE(x, x'),

[0120] L total = L mse + L cos + L vg′ ,

[0121] Step 6, input the test data into the pre-trained network and the pre-trained visual-linguistic network for testing.

[0122] Step 61, input the test image and the corresponding text label into the pre-trained teacher encoder network and the pre-trained visual-linguistic network respectively.

[0123] Step 62, obtain the global semantic feature and the global image feature of the test image.

[0124] Step 63, replace the features of each layer of the test image with the semantic code library obtained in the training stage to obtain the replaced features.

[0125] Step 64, input the three features into the student decoder network according to the training stage step to fuse and obtain the reconstructed test image.

[0126] Step 7, calculate the reconstruction error of the test data to detect and locate the anomaly.

[0127] Step 71: calculate the reconstruction error and the difference of image pixel reconstruction to detect and locate the anomaly.

[0128] Step 72, adjust the size of the feature map to the resolution of the original image by bilinear interpolation, and smooth it with a Gaussian kernel σ = 4.

[0129] The above-described embodiments are merely preferred embodiments of the present application, and obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

Claims

1. An image anomaly detection and positioning method based on priori guidance multi-semantic fusion, characterized in that, The implementation of the method includes the following contents: extracting the abstract global semantic features of normal images through a pre-trained visual-linguistic network; extracting the features of normal images through a teacher encoder network trained by an external data set; replacing the extracted features by vector discretization and constructing a learnable semantic encoding library for each layer of the teacher network; fuse the above-mentioned multi-semantic features as the input of the student decoder network, guide the abnormal image to be reconstructed as normal, so as to detect and locate the abnormality; The implementation steps of the method are as follows: Step 1, obtaining the image data to be detected, and performing preprocessing operation on the image; Step 2, inputting the preprocessed image into the pre-trained teacher encoder network to obtain the global image features; Step 3, inputting the text label of the image into the pre-trained visual-linguistic network to extract the abstract global semantic features of the normal image; Step 4, discretizing the features of each layer of the teacher encoder network, and constructing a learnable semantic encoding library to replace the extracted features; Step 5, fuse the above-mentioned features into the student decoder, and perform distillation training; Step 6, inputting the test data into the pre-trained network and the pre-trained visual-linguistic network for testing; Step 7, calculating the reconstruction error of the test data to detect and locate the abnormality; The step 1 includes the following steps: Step 11, collect image data, define the original image as X e R C×H×W ; wherein C represents the image dimension, H represents the height of the image, and W represents the width of the image. Step 12, performing size scaling processing on the image data, and the formula is as follows: where S(x, y) represents the corresponding pixel point in the scaled image, (x, y) is the pixel point coordinate, X represents the original image, X w and X h represent the width and height of the original image, S w and S h represent the width and height of the scaled image; Step 13, normalizing the image data, and the formula is as follows: where Smax and Smin represent the maximum and minimum values of the specific values of all pixels in the image, respectively. max , S min represent the maximum and minimum values of the specific values of all pixels in the image, respectively. Step 14, standardizing the image data, and the formula is as follows: Wherein, mean(·) represents the mean of each channel, and std(·) represents the standard deviation of each channel; The step 2 includes the following steps: Step 21, obtaining the global features of the image, which is specifically represented as: wherein x i represents data in the dataset, represents a pre-trained teacher encoder network; Step 22, obtaining global feature f ig Further down-sampling obtains global feature F ig ∈R C×H×W The step 3 includes the following steps: Step 31, obtaining the global semantic features of the image, which is specifically represented as: where t i a label class representing an image in the dataset, a pre-trained vision-language network, t i ∈R 1 ×512 ; Step 32, temperature expansion and up-sampling are performed on the global semantic features, so that the size thereof is consistent with the size of the image global features, to obtain features T i ∈R C×H×W The step 4 includes the following steps: Step 41, obtaining the features of each layer of the teacher encoder network, which is specifically represented as: wherein x i represent data in the dataset, represent a pre-trained network, l represents different layers of features, and l is selected from 1, 2, 3, and 4. Step 42, discretizing the features of each layer, and constructing a learnable semantic encoding library, which is specifically represented as: wherein V l ∈ R k×c represents k-dimensional feature vectors of c f i,l (p, q) represents image features at different layer (p, q) positions, represents f l e features in f i,l (p, q) ′ , and the final output feature f l v The optimization objective of the reparameterization module is: L vg ′ =‖sg[f l e ]-f l v ‖ 2 +‖f l e -sg[f l v ]‖ 2 Wherein, sg[] represents the stop gradient operator.

2. The image anomaly detection and localization method based on prior-guided multi-semantic fusion according to claim 1, characterized in that, The step 5 includes the following steps: Step 51, fuse the global semantic feature T i and the image global feature F ig together: wherein is represented as splicing; Step 52, the fused feature D i is sent into the LeWinBlocks attention mechanism module and is operated on to obtain the feature D i ; Step 53, global semantic feature T i Up-sampling and feature f l d and feature f i,l (p,q) ′ Fusion: Repeat step 53 until an image x of the same size as the input is obtained ′ ; Step 54, using distillation loss to guide the student decoder network to learn, and the optimization target of the reparameterization module is: M l (i,j) = 1 - cosine(f l e (i,j),f l d (i,j)), L mse = MSE(x, x ′ ), L total = L mse + L cos + L vg ′ , 3. The image anomaly detection and localization method based on prior-guided multi-semantic fusion according to claim 1, characterized in that, The step 6 includes the following steps: Step 61, inputting the test image and the corresponding text label into the pre-trained teacher encoder network and the pre-trained visual-linguistic network respectively; Step 62, obtaining the global semantic features and the global image features of the test image; Step 63, replacing the features of each layer of the test image using the semantic encoding library obtained in the training stage to obtain the replaced features; Step 64, fusing the three features into the student decoder network according to the training stage, and obtaining the reconstructed test image.

4. The image anomaly detection and positioning method based on priori guidance multi-semantic fusion according to claim 1, characterized in that, The step 7 includes the following steps: Step 71: calculating the reconstruction error and the difference between the image pixel reconstruction to detect and locate the abnormality; Step 72, the size of the feature map is adjusted to the resolution of the original image by bilinear interpolation, and it is smoothed with a Gaussian kernel σ = 4.

Images