Cross-category image anomaly detection method and device based on parallel mask automatic encoder

Abstract

The invention discloses a cross-category image anomaly detection method based on a parallel mask automatic encoder. The cross-category image anomaly detection method comprises the steps of obtaining and extracting an image visual feature map of a to-be-recognized image; segmenting the extracted feature map into a plurality of visual marks; based on the visual mark and a preset nominal visual feature pool corresponding to the category of the image, determining a reference feature map corresponding to the feature map; generating a plurality of mask patterns of the feature pattern, and generating a group of masked feature pattern and reference feature pattern based on the plurality of mask patterns, the feature pattern and the reference feature pattern; reconstructing the two groups of feature maps by adopting a trained reconstruction model to generate a group of reconstructed first feature maps and a group of reconstructed reference feature maps; a correlation difference between the two groups of reconstructed feature maps and a reconstruction error of the first feature map are determined, and an anomaly score of the image is determined based on the reconstruction error and the correlation difference. The detection precision can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a cross-category image anomaly detection method and device based on a parallel mask autoencoder. Background Art

[0002] Anomaly detection is an important task that is increasingly used in various fields such as video surveillance, industrial inspection, and medical image analysis. Due to the scarcity of abnormal samples, unsupervised anomaly detection methods have received widespread attention. This method only models the distribution of normal data and then identifies samples that deviate from the normal distribution as abnormal samples. Common methods adopt a one-to-one correspondence scheme to train models for different categories of objects separately. This method consumes a lot of time and memory in practical applications and is not beneficial for object categories with large intra-category diversity. In addition, many reconstruction-based models assume that when presented with an abnormal sample, the model tends to generate normal samples, resulting in larger reconstruction errors that can be used as an indicator for anomaly detection, but this assumption does not always hold true because abnormal inputs can sometimes be effectively reconstructed, which may cause both normal and abnormal samples to be reconstructed equally well and difficult to distinguish, the so-called "same shortcut" problem. At present, existing related methods have combined different strategies, such as neighboring mask attention, memory mechanism, and nominal prototypes, to alleviate the impact of the "same shortcut" problem and prevent reconstruction anomalies. For example, Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, Yan-Feng Wang, et al. proposed in their paper "Registration based few-shot anomaly detection" (European Conference on Computer Vision, 2022) to train a general model by using SimSiam and three spatial transformer network blocks to identify anomalies by comparing the registered features of the test image with the corresponding normal image. Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, Xinyi Le, et al. disclosed a reconstruction-based anomaly detection method UniAD in their paper "A Unified Model for Multi-class Anomaly Detection" (2022 Conference and Workshop on Neural Information Processing Systems). This method constructs a Transformer containing a layer-by-layer query decoder and a neighboring mask attention module to capture multi-class distributions, strengthen the use of query embeddings by excluding the relationship between feature points and their neighbors, and prevent information leakage. But they are still susceptible to abnormal information in the conditions, which may lead to the "same shortcut" problem.For another example, Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick, et al., in their paper “Masked autoencoders are scalable vision learners” (2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition), disclosed a method of simply masking random blocks of the input image and reconstructing the missing pixels. However, directly reconstructing the missing pixels for visual pre-training may cause the model to prioritize short-range dependencies and high-frequency details. Hangbo Bao, Li Dong, Songhao Piao, Furu Wei, et al., in their paper “Beit: Bert pre-training of image Transformers” (International Conference on Learning Representations, 2021), proposed a goal that does not rely on the original pixels, but the method still aims to use a block-level reconstruction model based on MAE to circumvent the “same shortcut” problem, and there is an information leakage problem. Obviously, although these methods emphasize limiting the ability to generate anomalies, these models are still susceptible to abnormal information leakage and may face the “same shortcut” problem.

[0003] That is to say, existing technologies adopt a one-to-one correspondence scheme to train models for different categories of objects separately. This method consumes a lot of time and memory in practical applications, and is not conducive to object categories with large intra-category diversity. In addition, many reconstruction-based networks are still susceptible to abnormal information leakage and may face the "same shortcut" problem, which restricts the performance of the model. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a cross-category image anomaly detection method and device based on a parallel mask autoencoder.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a cross-category image anomaly detection method based on a parallel mask autoencoder, comprising:

[0007] Obtain an image to be recognized;

[0008] Extracting visual features of the image to be identified to obtain a first feature map;

[0009] Segmenting the first feature map into a plurality of visual markers;

[0010] Determining a reference feature map corresponding to the first feature map based on the multiple visual markers and a preset nominal visual feature pool corresponding to the category to which the image to be identified belongs;

[0011] Generate multiple mask maps corresponding to the first feature map, and generate a set of masked first feature maps and a set of masked reference feature maps based on the multiple mask maps, the first feature map and the reference feature map, so as to obtain two sets of feature maps;

[0012] The two groups of feature maps are reconstructed respectively using a trained reconstruction model to generate a group of reconstructed first feature maps and a group of reconstructed reference feature maps, thereby obtaining two groups of reconstructed images; the trained reconstruction model is obtained by training with a plurality of different types of training samples;

[0013] Determine the correlation difference between the two groups of reconstructed images and the reconstruction error of the first feature map, and determine the anomaly score of the image to be identified based on the reconstruction error and the correlation difference.

[0014] The present invention also provides a cross-category image anomaly detection device based on a parallel mask autoencoder, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0015] The memory is used to store computer programs;

[0016] The processor is used to implement the steps of the above-mentioned cross-category image anomaly detection method based on parallel mask autoencoder when executing the program stored in the memory.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] The present invention uses a reconstruction model obtained by training with a plurality of different types of training samples to perform image reconstruction, which can avoid training models for different types of objects separately, and is more efficient and resource-saving. According to the present invention, the feature map of the image to be identified and the reference feature map are respectively masked according to the plurality of mask maps corresponding to the feature map of the image to be identified, and then the two sets of feature maps after masking are input into the reconstruction model for image reconstruction, and then the correlation difference between the two sets of reconstructed feature maps and the reconstruction error of the feature map of the image to be identified are determined, and the abnormal score of the image to be identified is determined based on the reconstruction error and the correlation difference, and the inherent recovery ability difference between normal and abnormal modes under different masking conditions (that is, the abnormal area shows unstable reconstruction when the abnormal image block is masked, while the normal area shows consistent reconstruction) can be used to quantify the reconstruction difference under different masking strategies, and the "same shortcut" problem can be used to eliminate the influence of abnormal information leakage; due to the complexity of generating mask abnormalities, the abnormal area shows more significant reconstruction differences, thereby improving the detection performance and ultimately improving the accuracy of the detection results.

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a cross-category image anomaly detection method based on a parallel mask autoencoder provided by an embodiment of the present invention;

[0021] Figure 2 It is a qualitative result display diagram of abnormality positioning on the MVTec-AD data set obtained by using the method of the present invention provided in an embodiment of the present invention;

[0022] Figure 3 It is a quantitative result display diagram of anomaly detection and positioning on the MVTec-AD data set obtained by using the method of the embodiment of the present invention;

[0023] Figure 4 It is a quantitative result display diagram of anomaly detection and positioning on the VisA data set obtained by using the method of the present invention;

[0024] Figure 5 This is a diagram showing reconstruction results under different masks using the method of the present invention provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0026] Figure 1 is a flow chart of a cross-category image anomaly detection method based on a parallel mask autoencoder provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0027] S101: Obtain an image to be recognized.

[0028] Here, the image to be identified may be one or more images. When there are more images, the categories of the images to be identified may be the same or different. The following S102 to S107 are the process of performing abnormality detection on each identification image.

[0029] S102: extracting visual features of the image to be recognized to obtain a first feature map.

[0030] Here, a pre-trained extraction model may be used to extract the image visual features of each image to be identified, and the extracted feature map (visual feature map) is used as the first feature map of the image to be identified.

[0031] For example, the pre-trained extraction model is a pre-trained EfficientNet. For example, if an image to be recognized is represented as The pre-trained EfficientNetφ is used 0 The extracted feature map of the image to be recognized is expressed as

[0032] S103: Segment the first feature map into multiple visual markers.

[0033] For example, if the size of the first feature map is h×w×c, the first feature map can be divided into h×w visual tokens, and the number of channels of each visual token is c. For example, when the size of the first feature map is 14×14×224, the first feature map can be divided into 196 visual tokens, and the number of channels of each visual token is 224.

[0034] S104: Determine a reference feature map (visual reference feature map) corresponding to the first feature map based on multiple visual markers and a preset nominal visual feature pool corresponding to the category to which the image to be identified belongs.

[0035] The nominal visual feature pool corresponding to the category to which the image to be identified belongs is pre-created for the category to which the image to be identified belongs using the coreset subsampling mechanism, and the nominal visual feature pool contains multiple nominal visual tokens. The specific creation method of the nominal visual feature pool is the existing method mentioned in the paper Towards Total Recall in Industrial Anomaly Detection, which will not be repeated here.

[0036] S105. Generate multiple mask maps corresponding to the first feature map, and generate a group of masked first feature maps and a group of masked reference feature maps based on the multiple mask maps, the first feature map and the reference feature map to obtain two groups of feature maps.

[0037] S106. Use the trained reconstruction model to reconstruct the two groups of feature maps respectively, generate a group of reconstructed first feature maps, and a group of reconstructed reference feature maps, and obtain two groups of reconstructed feature maps; the trained reconstruction model is obtained by training with training samples of multiple different categories.

[0038] S107, determining the correlation difference between the two groups of reconstructed feature maps and the reconstruction error of the first feature map, and determining the abnormality score of the image to be identified based on the reconstruction error and the correlation difference.

[0039] In the present invention, the above S104 is specifically implemented by the following steps:

[0040] S1041. Search for the nominal visual marker that is most similar to each visual marker from a preset nominal visual feature pool corresponding to the category to which the image to be identified belongs, and obtain multiple target nominal visual markers.

[0041] When searching for the most similar nominal visual tag for each visual tag, a nearest-neighbor searching method may be used to find a most similar nominal visual tag for each visual tag, and the nominal visual tag may be used as a target nominal visual tag for the visual tag.

[0042] S1042. Replace corresponding visual markers with multiple target nominal visual markers as reference feature maps corresponding to the first feature map.

[0043] Here, the nominal visual marker that is most similar to each visual marker is used to replace the visual marker, so that a reference feature map corresponding to the first feature map can be obtained.

[0044] Exemplarily, if the size of the first feature map is h×w×c, the size of the reference feature map corresponding to the first feature map is also h×w×c.

[0045] In the present invention, the “generating multiple mask images corresponding to the first feature image” in the above S105 is specifically implemented by the following steps:

[0046] S1051, generating an anomaly score map of the image to be identified; the anomaly score map represents an anomaly score of each image block in the image to be identified.

[0047] Here, an existing anomaly score map generation method is used to generate an anomaly score map of the image to be identified.

[0048] S1052. Use the first mask generator to generate n prototype-oriented masks of the feature map; the mask ratios of the n prototype-oriented masks are different; and n is a positive integer greater than or equal to 1.

[0049] Here, the value of n can be set according to actual needs, and the present invention does not limit this. When generating n prototype-guided mask images of the anomaly score map, the n prototype-guided mask images of the anomaly score map can be controlled by n different mask ratios, and the following n random mask images and n block mask images are similar.

[0050] S1053. Use the second mask generator to generate n random mask images, n blockwise mask images and 1 zero mask image of the first feature map respectively; the mask ratios of the n random mask images are different, and the mask ratios of the n blockwise mask images are also different.

[0051] Here, a random mask algorithm may be used to generate n random mask images of the first feature map, and a block mask algorithm may be used to generate n block mask images of the first feature map.

[0052] Here, the zero mask map indicates that no masking is performed, that is, the zero mask map is a tensor filled with all zeros.

[0053] It should be noted that the principle of generating anomaly score maps of images and the principle of using maskers to generate mask maps of feature maps can be referred to the relevant existing content in the paper One-for-All: Proposal Masked Cross-Class Anomaly Detection, which will not be repeated here.

[0054] S1054, using n prototype-guided mask images, n random mask images, n block mask images and 1 zero mask image as multiple mask images corresponding to the first feature image.

[0055] In the present invention, the above-mentioned step of “generating a set of masked first feature maps and a set of masked reference feature maps based on the multiple mask maps, the first feature maps and the reference feature maps to obtain two sets of feature maps” in S105 is specifically implemented by the following steps:

[0056] S1055, respectively copy the first feature map and the reference feature map 3n+1 copies, to obtain 3n+1 first feature maps and 3n+1 reference feature maps; wherein the 3n+1 first feature maps and the 3n+1 reference feature maps are different from the 3n+1 masks Figure 1 One to one correspondence.

[0057] S1056. Multiply each first feature map by a corresponding mask map to obtain 3n+1 masked first feature maps, and use the 3n+1 masked first feature maps as a group of masked first feature maps.

[0058] S1057. Multiply each reference feature map by a corresponding mask map to obtain 3n+1 masked reference feature maps, and use the 3n+1 masked reference feature maps as a group of masked reference feature maps.

[0059] In the present invention, after obtaining 3n+1 masked first feature maps and 3n+1 masked reference feature maps, in the above S106, the 3n+1 masked first feature maps and the 3n+1 masked reference feature maps are input into the trained reconstruction model, and the trained reconstruction model outputs 3n+1 reconstructed 3n+1 first feature maps and 3n+1 reconstructed reference feature maps.

[0060] In the present invention, each group of reconstructed feature maps includes N reconstructed feature maps; based on this, the "determining the correlation difference between the two groups of reconstructed feature maps, and the reconstruction error of the first feature map" in the above S107 is specifically implemented by the following steps:

[0061] S1071. Determine a first empirical distribution according to the first feature map and N reconstructed first feature maps; the first empirical distribution represents N reconstruction errors of each image block in the first feature map.

[0062] The expression of the first empirical distribution is: Among them, P h represents the first empirical distribution, h represents the first feature map, represents the first feature map of the i-th reconstruction, ∥.∥ 2Denotes the L2 norm. If the size of the first feature map is h×w×c, the first feature map and each reconstructed first feature map contain h×w image blocks. When N reconstructed first feature maps are obtained, each image block can obtain N reconstruction results, so each image block has N reconstruction errors. Represents the reconstruction error of each image block under the i-th mask map.

[0063] S1072. Determine a second empirical distribution based on the reference feature map and N reconstructed reference feature maps; the second empirical distribution represents N reconstruction errors of each image block in the reference feature map.

[0064] The expression of the second empirical distribution is: P r represents the second empirical distribution, r represents the reference feature map, represents the jth reconstructed reference feature map. If the size of the reference feature map is h×w×c, then the reference feature map and each reconstructed reference feature map contain h×w image blocks. When N reconstructed reference feature maps are obtained, each image block can obtain N reconstruction results, so each image block has N reconstruction errors. Represents the reconstruction error of each image block under the i-th mask map.

[0065] S1073. Determine a transmission distance between the first empirical distribution and the second empirical distribution, and use the transmission distance as an association difference between the N reconstructed first feature maps and the N reconstructed reference feature maps.

[0066] Here, the optimal transmission (OT) distance algorithm can be used to determine the transmission distance AssDis(h, r; x) between the first empirical distribution and the second empirical distribution. The expression of the optimal transmission (OT) distance algorithm is:

[0067] Here, by minimizing the objective Create from P h To P r The transmission matrix M * ∈R N×N When the transmission matrix M * Constraints must be followed in, and Indicates that P h and P r The cost matrix C∈R N×N Defined as Transmission matrix M *As a probabilistic reweighting of the cost matrix C with different masks, it measures the importance of different distances between two reconstruction error distributions. Specifically, the expression of the associated difference is: AssDis(h,r;x)=M * C.

[0068] S1074. Determine the difference between each reconstructed first feature map and the first feature map, and determine the reconstruction error of the first feature map based on the L2 norm of the difference.

[0069] Here, the expression of the reconstruction error of the first feature map is: h represents the first feature map, represents the first feature map of the i-th reconstruction, ||.|| 2 represents the L2 norm.

[0070] Here, the association difference and reconstruction error are both tensors.

[0071] In the present invention, the step of “determining the abnormality score of the image to be identified based on the reconstruction error and the associated difference” in S107 is specifically implemented by the following steps:

[0072] S1075. Add the reconstruction error and the associated difference to obtain the error sum.

[0073] Here, the association difference and reconstruction error are both tensors, so the error sum is also a tensor. In addition, the anomaly score map is also a tensor.

[0074] S1076. Multiply the error sum and the anomaly score map element by element to obtain an initial anomaly score.

[0075] S1077: Perform upsampling processing on the initial anomaly score to obtain an anomaly score of the image to be identified.

[0076] Exemplarily, the expression of the anomaly score of the image to be identified is as follows:

[0077]

[0078] Among them, M po is the anomaly score map of the image to be identified.

[0079] Here, since the size of the anomaly score is smaller than the image to be identified, the size of the anomaly score is adjusted to be the same as the size of the image to be identified through upsampling.

[0080] In the present invention, before the above S106, the following steps are also included:

[0081] S201. Obtain a training set, which includes training samples of various categories.

[0082] S202. When reconstructing the model each time, randomly select multiple training samples from the training set.

[0083] Exemplarily, the reconstruction model may be a VIT-based encoder-decoder model.

[0084] S203: extracting image visual features of each selected training sample to obtain a second feature map of each selected training sample.

[0085] Here, the principle of S203 is the same as the principle of S102 described above.

[0086] S204: Generate m different mask images of the second feature map, and embed the m different mask images into the second feature map to obtain a mask feature map.

[0087] Here, a masker can be used to generate n of the second feature map through a random mask algorithm. 1 A random mask map is used to generate n second feature maps through the block mask algorithm. 2 random mask feature maps, n 1 +n 2 =m, where n 1 、n 2 The value can be set according to actual needs.

[0088] Here, by sequentially multiplying m different mask maps with the second feature map element by element, the m different mask maps are embedded in the second feature map to obtain a mask feature map.

[0089] S205, inputting the multiple mask feature maps corresponding to the selected multiple training samples into the reconstruction model to be trained, and obtaining multiple mask feature maps. Figure 1 One corresponds to multiple reconstructed mask feature maps.

[0090] For example, when a mask feature map is When , the reconstructed mask feature map can be expressed as Among them, E represents the encoder, D represents the decoder, and h′ represents the second feature map.

[0091] S206. Determine a loss value based on the difference between multiple mask feature maps and multiple one-to-one reconstructed mask feature maps, adjust the weight parameters of the reconstruction model to be trained according to the loss value, and obtain a trained reconstruction model by multiple iterative training of the reconstruction model to be trained.

[0092] For example, the loss value is expressed as:

[0093] Here, the specific training iteration process is the existing training iteration process, which will not be repeated here.

[0094] The present invention also provides a cross-category image anomaly detection device based on a parallel mask autoencoder, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the steps of the above-mentioned cross-category image anomaly detection method based on a parallel mask autoencoder when executing the program stored in the memory.

[0095] The present invention uses a reconstruction model obtained by training with a plurality of different categories of training samples to perform image reconstruction, which can avoid training models for different categories of objects separately, and is more efficient and resource-saving. According to the present invention, the feature map of the image to be identified and the reference feature map are respectively masked according to the plurality of mask maps corresponding to the feature map of the image to be identified, and then the two sets of feature maps after masking are input into the reconstruction model for feature map reconstruction, and then the correlation difference between the reconstructed two sets of feature maps and the reconstruction error of the feature map of the image to be identified are determined, and the abnormal score of the image to be identified is determined based on the reconstruction error and the correlation difference, and the inherent recovery ability difference between normal and abnormal modes under different masking conditions (that is, the abnormal area shows unstable reconstruction when the abnormal image block is masked, while the normal area shows consistent reconstruction) can be used to quantify the reconstruction difference under different masking strategies, and the "same shortcut" problem can be used to eliminate the influence of abnormal information leakage; due to the complexity of generating mask abnormalities, the abnormal area shows more significant reconstruction differences, thereby improving the detection performance of the model.

[0096] The technical effects of the present invention are further illustrated by three simulation experiments below.

[0097] The three simulation experiments used an NVIDIA GeForce RTX 3090 24GB system and a 64-bit Windows operating system, and the simulation language was Python. The experimental data used was a widely used industrial anomaly detection dataset "MVTec-AD", which contains 15 categories and covers more than 5k high-resolution images, including objects and textures; a recently released large-scale dataset VisA, which consists of 9621 normal images and 1200 abnormal high-resolution images, including images with complex structures, scattered objects, and various types of objects, among which the anomalies include scratches, dents, spots, cracks, and structural defects.

[0098] Experiment 1:

[0099] The input image size of the MVTec-AD dataset is 224×224×3. After inputting the pre-trained EfficientNet model, the feature map becomes 14×14×272, that is, the image block size is 16. Then, the channel dimension of each block is mapped to 768, and then input into a 6-layer ViT-based encoder and a 6-layer ViT-based decoder. We use AdamW for optimization with a weight decay of 0.05. The model of the present invention is trained on 1 GPU (NVIDIA GeForce RTX 3090 24GB) for 1000 cycles with a batch size of 36. The initial learning rate is 1×10 -4 , and drops by 0.1 after 800 cycles. Except for the pre-trained EfficientNet, the model of the present invention is trained from scratch. During testing, the present invention randomly generates 4 masks for each masking strategy. The ratios of random masks and block masks are fixed to 0.3 and 0.1 respectively, while the prototype-guided masks are generated adaptively with the ratio range set between 0.01 and 0.04.

[0100] See also Figure 2 , Figure 2 The figure is a qualitative result display of abnormality positioning on the MVTec-AD data set obtained by the method of the present invention. For object abnormalities and texture damage, the method of the present invention can effectively locate abnormal areas of various types of abnormalities. Figure 3 : is a quantitative result display diagram of anomaly detection and positioning on the MVTec-AD data set obtained by using the method of the embodiment of the present invention. Figure 3 As shown, the performance of anomaly detection and localization of our model is generally better than all competitors.

[0101] Experiment 2:

[0102] See also Figure 4 , Figure 4 It is a quantitative result display diagram of anomaly detection and positioning on the VisA data set obtained by using the method of the present invention. Figure 4 The superior performance of our model compared to other reconstruction-based methods in a unified setting is demonstrated. Robust and consistent performance is demonstrated across a variety of categories, including complex structures, single instances, and multiple instances. These results demonstrate the effectiveness of our model in handling different anomalies.

[0103] Experiment 3:

[0104] See also Figure 5 , Figure 5 It is a display diagram of the reconstruction results under different masks using the method of the present invention. Figure 5The model of the present invention successfully masks suspicious abnormal regions through a prototype-guided masking strategy. When the abnormal region is masked, the present invention successfully reconstructs the abnormality into its corresponding normal sample, and when the abnormal region is directly input into the model, the abnormal part remains unchanged. Subsequently, the association difference learning model of the present invention explores the differences between various reconstruction error maps and successfully distinguishes normal samples from abnormal samples.

[0105] The above experiments prove that the present invention has achieved the most advanced performance in anomaly detection and localization.

[0106] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0108] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good effects.

[0109] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A cross-category image anomaly detection method based on parallel mask autoencoder, characterized in that: include: Obtain an image to be recognized; Extracting visual features of the image to be identified to obtain a first feature map; Segmenting the first feature map into a plurality of visual markers; Determining a reference feature map corresponding to the first feature map based on the multiple visual markers and a preset nominal visual feature pool corresponding to the category to which the image to be identified belongs; Generate multiple mask maps corresponding to the first feature map, and generate a set of masked first feature maps and a set of masked reference feature maps based on the multiple mask maps, the first feature map and the reference feature map, so as to obtain two sets of feature maps; The two groups of feature maps are reconstructed respectively using a trained reconstruction model to generate a group of reconstructed first feature maps and a group of reconstructed reference feature maps, thereby obtaining two groups of reconstructed feature maps; the trained reconstruction model is obtained by training with a plurality of different types of training samples; Determine the correlation difference between the two groups of reconstructed feature maps and the reconstruction error of the first feature map, and determine the anomaly score of the image to be identified based on the reconstruction error and the correlation difference.

2. The cross-category image anomaly detection method based on parallel mask autoencoder according to claim 1, characterized in that: Determining a reference feature map corresponding to the first feature map based on the multiple visual markers and a preset nominal visual feature pool corresponding to the category to which the image to be identified belongs includes: From a preset nominal visual feature pool corresponding to the category to which the image to be identified belongs, searching for a nominal visual marker that is most similar to each visual marker, to obtain a plurality of target nominal visual markers; the nominal visual feature pool is created for the category to which the image to be identified belongs using a core set subsampling mechanism, and the nominal visual feature pool contains a plurality of nominal visual markers; The multiple target nominal visual labels replace corresponding visual labels as reference feature maps corresponding to the first feature map.

3. The cross-category image anomaly detection method based on parallel mask autoencoder according to claim 1, characterized in that: The generating a plurality of mask images corresponding to the first feature image comprises: Generate an anomaly score map of the image to be identified; the anomaly score map represents an anomaly score of each image block in the image to be identified; Using a first mask generator, generating n prototype-guided mask images of the first feature map; the mask ratios of the n prototype-guided mask images are different; n is a positive integer greater than or equal to 1; Using a second mask generator, respectively generating n random mask images, n block mask images and one zero mask image of the first feature image; the mask ratios of the n random mask images are different, and the mask ratios of the n block mask images are also different; The n prototype-guided mask maps, the n random mask maps, the n block mask maps and the 1 zero mask map are used as multiple mask maps corresponding to the first feature map.

4. The cross-category image anomaly detection method based on parallel mask autoencoder according to claim 1, characterized in that: The multiple mask maps include 3n+1 different mask maps, where n is a positive integer greater than or equal to 1; based on the multiple mask maps, the first feature map and the reference feature map, a group of masked first feature maps and a group of masked reference feature maps are generated to obtain two groups of feature maps, including: Copying 3n+1 copies of the first feature map and the reference feature map respectively, to obtain 3n+1 copies of the first feature map and 3n+1 copies of the reference feature map; wherein the 3n+1 copies of the first feature map and the 3n+1 copies of the reference feature map all correspond one-to-one to the 3n+1 different mask maps; Multiply each of the first feature maps by a corresponding mask map to obtain 3n+1 masked first feature maps, and use the 3n+1 masked first feature maps as a group of masked first feature maps; Each of the reference feature maps is multiplied by a corresponding mask map to obtain 3n+1 masked reference feature maps, and the 3n+1 masked reference feature maps are used as a group of masked reference feature maps.

5. The cross-category image anomaly detection method based on parallel mask autoencoder according to claim 1, characterized in that: Each group of reconstructed feature maps includes N reconstructed feature maps; determining the correlation difference between the two groups of reconstructed feature maps and the reconstruction error of the first feature map includes: Determine a first empirical distribution according to the first feature map and N reconstructed first feature maps; the first empirical distribution represents N reconstruction errors of each image block in the first feature map; Determining a second empirical distribution according to the reference feature map and N reconstructed reference feature maps; the second empirical distribution represents N reconstruction errors of each image block in the reference feature map; Determine a transmission distance between the first empirical distribution and the second empirical distribution, and use the transmission distance as an association difference between the N reconstructed first feature maps and the N reconstructed reference feature maps; Determine the difference between each reconstructed first feature map and the first feature map, and determine the reconstruction error of the first feature map based on the L2 norm of the difference.

6. The cross-category image anomaly detection method based on parallel mask autoencoder according to claim 5, characterized in that: The expressions of the first empirical distribution and the second empirical distribution are respectively: Among them, P h represents the first empirical distribution, P r represents the second empirical distribution, h represents the first feature map, represents the first feature map of the i-th reconstruction, ∥.∥2 represents the L2 norm, r represents the reference feature map, Represents the j-th reconstructed reference feature map.

7. The cross-category image anomaly detection method based on parallel mask autoencoder according to claim 5, characterized in that: The expression of the reconstruction error of the first feature map is: in, represents the reconstruction error of the first feature map, i=1,...,N, h represents the first feature map, represents the first feature map of the i-th reconstruction, and ||.||2 represents the L2 norm.

8. The cross-category image anomaly detection method based on parallel mask autoencoder according to claim 3, characterized in that: The determining, based on the reconstruction error and the associated difference, an abnormality score of the image to be identified comprises: After adding the reconstruction error and the associated difference, an error sum is obtained; Multiplying the error sum and the anomaly score map element by element to obtain an initial anomaly score; The initial anomaly score is upsampled to obtain an anomaly score of the image to be identified.

9. The cross-category image anomaly detection method based on parallel mask autoencoder according to claim 1, characterized in that: Before respectively reconstructing the two groups of feature maps using the trained reconstruction model to generate a group of reconstructed first feature maps and a group of reconstructed reference feature maps, the method further includes: Obtaining a training set, wherein the training set includes training samples of multiple different categories; When training the reconstruction model each time, a plurality of training samples are randomly selected from the training set; Extracting image visual features of each selected training sample to obtain a second feature map of each selected training sample; Generate m different mask images of the second feature map, and embed the m different mask images into the second feature map to obtain a mask feature map; Inputting a plurality of mask feature maps corresponding to a plurality of selected training samples into a reconstruction model to be trained, and obtaining a plurality of reconstructed mask feature maps corresponding to a plurality of mask feature maps; Based on the difference between the multiple mask feature maps and the multiple corresponding reconstructed mask feature maps, a loss value is determined, and the weight parameters of the reconstruction model to be trained are adjusted according to the loss value. The trained reconstruction model is obtained through multiple iterative training of the reconstruction model to be trained.

10. A cross-category image anomaly detection device based on a parallel mask autoencoder, comprising a processor, a communication interface, a memory and a communication bus, characterized in that: The processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to implement the method steps described in any one of claims 1-9 when executing the program stored in the memory.

Images