A method and device for cross-class image anomaly detection based on parallel mask autoencoders

Abstract

This invention discloses a cross-category image anomaly detection method based on a parallel masked autoencoder, comprising: acquiring and extracting an image visual feature map of the image to be identified; segmenting the extracted feature map into multiple visual labels; determining a reference feature map corresponding to the feature map based on the visual labels and a preset nominal visual feature pool corresponding to the category to which the image belongs; generating multiple mask maps of the feature map; generating a set of masked feature maps and reference feature maps based on the multiple mask maps, the feature map, and the reference feature map; reconstructing the two sets of feature maps using a trained reconstruction model to generate a set of reconstructed first feature maps and a set of reconstructed reference feature maps; determining the correlation difference between the two sets of reconstructed feature maps and the reconstruction error of the first feature map; and determining the anomaly score of the image based on the reconstruction error and the correlation difference. This invention can improve detection accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, specifically relating to a cross-category image anomaly detection method and device based on a parallel mask autoencoder. Background Technology

[0002] Anomaly detection is a crucial task with increasingly widespread applications in fields such as video surveillance, industrial inspection, and medical image analysis. Due to the scarcity of anomalous samples, unsupervised anomaly detection methods have garnered significant attention. These methods model the distribution of normal data and then identify samples deviating from the normal distribution as anomalous. Common methods employ a one-to-one correspondence approach, training models separately for different object categories. This approach is time-consuming and memory-intensive in practical applications and is detrimental to categories with high intra-category diversity. Furthermore, many reconstruction-based models assume that when presented with an anomalous sample, the model tends to generate a normal sample, leading to a larger reconstruction error that can serve as an indicator of anomaly detection. However, this assumption is not always valid, as anomalous inputs can sometimes be effectively reconstructed, potentially resulting in both normal and anomalous samples being reconstructed equally well, making them difficult to distinguish—the so-called "same shortcut" problem. Current methods mitigate the impact of the "same shortcut" problem and prevent reconstruction anomalies by combining different strategies, such as proximity masking attention, memory mechanisms, and nominal prototypes. For example, Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, Yan-Feng Wang, and others proposed in their paper "Registration based few-shot anomaly detection" (European Conference on Computer Vision, 2022) to train a general model using SimSiam and three spatial transformation network blocks, identifying anomalies by comparing the registered features of test images with corresponding normal images. Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, Xinyi Le, and others disclosed a reconstruction-based anomaly detection method, UniAD, in their paper "A Unified Model for Multi-class Anomaly Detection" (2022 Conference and Workshop on Neural Information Processing Systems). This method constructs a Transformer containing a layer-by-layer query decoder and a neighbor mask attention module to capture multi-class distributions, enhancing the use of query embeddings by excluding relationships between feature points and their neighbors while preventing information leakage. However, they are still susceptible to anomalous information in the conditions, potentially leading to the "same shortcut" problem.For example, Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick, and others disclosed a method in their paper "Masked autoencoders are scalable vision learners" (2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition) to reconstruct missing pixels by simply masking random blocks of the input image. However, directly reconstructing missing pixels for visual pre-training may cause the model to prioritize short-range dependencies and high-frequency details. HangboBao, Li Dong, Songhao Piao, Furu Wei, and others proposed an objective that does not depend on the original pixels in their paper "Beit: Bert pre-training of image Transformers" (International Conference on Learning Representations, 2021), but this method still aims to use a block-level reconstruction model based on MAE to avoid the "same shortcut" problem, resulting in information leakage. Clearly, although these methods emphasize limiting the generation of anomalies, these models are still susceptible to anomaly information leakage and may face the "same shortcut" problem.

[0003] In other words, existing technologies use a one-to-one correspondence approach, training models separately for different categories of objects. This method consumes a lot of time and memory in practical applications and is not conducive to object categories with high diversity. Furthermore, many reconstructed networks are still susceptible to the leakage of abnormal information and may face the "same shortcut" problem, thus limiting the performance of the model. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention provides a cross-category image anomaly detection method and device based on a parallel mask autoencoder.

[0005] The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides a cross-category image anomaly detection method based on a parallel mask autoencoder, comprising:

[0007] Acquire the image to be recognized;

[0008] Extract the visual features of the image to be identified to obtain a first feature map;

[0009] The first feature map is segmented into multiple visual markers;

[0010] Based on the multiple visual markers and the preset nominal visual feature pool corresponding to the category to which the image to be identified belongs, a reference feature map corresponding to the first feature map is determined;

[0011] Generate multiple mask images corresponding to the first feature image. Based on the multiple mask images, the first feature image, and the reference feature image, generate a set of masked first feature images and a set of masked reference feature images to obtain two sets of feature images.

[0012] The two sets of feature maps are reconstructed using the trained reconstruction model to generate a first reconstructed feature map and a reference reconstructed feature map, resulting in two sets of reconstructed images. The trained reconstruction model is obtained by training with training samples of various different categories.

[0013] The correlation difference between the two sets of reconstructed images and the reconstruction error of the first feature map are determined. Based on the reconstruction error and the correlation difference, the anomaly score of the image to be identified is determined.

[0014] The present invention also provides a cross-category image anomaly detection device based on a parallel mask autoencoder, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0015] The memory is used to store computer programs;

[0016] When the processor executes the program stored in the memory, it implements the steps of the cross-category image anomaly detection method based on a parallel mask autoencoder.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] This invention uses a reconstruction model trained with various training samples of different categories for image reconstruction, avoiding the need to train models separately for different object categories, thus achieving greater efficiency and saving resources. Based on multiple mask maps corresponding to the feature maps of the image to be identified, this invention performs masking processing on both the feature maps and reference feature maps of the image to be identified. Then, the two sets of masked feature maps are input into the reconstruction model for image reconstruction. The correlation differences between the two sets of reconstructed feature maps, as well as the reconstruction error of the feature maps of the image to be identified, are determined. Based on the reconstruction error and correlation differences, the anomaly score of the image to be identified is determined. The inherent differences in recovery ability between normal and abnormal patterns under different masking conditions can be utilized (i.e., abnormal regions exhibit unstable reconstruction when abnormal image blocks are masked, while normal regions exhibit consistent reconstruction) to quantify the reconstruction differences under different masking strategies. The "same shortcut" problem can be used to eliminate the impact of abnormal information leakage. Due to the complexity of generating mask anomalies, abnormal regions exhibit more significant reconstruction differences, thereby improving detection performance and ultimately improving the accuracy of the detection results.

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a cross-category image anomaly detection method based on a parallel mask autoencoder provided in an embodiment of the present invention;

[0021] Figure 2 This is a qualitative result illustration of anomaly localization on the MVTec-AD dataset obtained using the method of the present invention, provided by an embodiment of the present invention;

[0022] Figure 3 This is a graph showing the quantitative results of anomaly detection and localization on the MVTec-AD dataset obtained using the method of the embodiments of the present invention;

[0023] Figure 4 This is a diagram showing the quantitative results of anomaly detection and localization on the VisA dataset obtained using the method of this invention;

[0024] Figure 5 This is a diagram illustrating the reconstruction results under different masks using the method of the present invention, as provided in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0026] Figure 1 This is a flowchart illustrating a cross-category image anomaly detection method based on a parallel mask autoencoder provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0027] S101. Obtain the image to be recognized.

[0028] Here, there can be one or more images to be identified. When there are multiple images, they can belong to the same or different categories. S102 to S107 below describe the anomaly detection process for each image to be identified.

[0029] S102. Extract the visual features of the image to be identified to obtain the first feature map.

[0030] Here, a pre-trained extraction model can be used to extract the visual features of each image to be identified, and the extracted visual feature map can be used as the first feature map of the image to be identified.

[0031] For example, the pre-trained extraction model is a pre-trained EfficientNet. For instance, if an image to be recognized is represented as... Then the pre-trained EfficientNetφ is used. 0 The extracted feature map of the image to be identified is represented as follows:

[0032] S103. Divide the first feature map into multiple visual labels.

[0033] For example, if the size of the first feature map is h×w×c, it can be divided into h×w visual tokens, each with c channels. For instance, if the size of the first feature map is 14×14×224, it can be divided into 196 visual tokens, each with 224 channels.

[0034] S104. Based on multiple visual markers and the preset nominal visual feature pool corresponding to the category to which the image to be identified belongs, determine the reference feature map corresponding to the first feature map.

[0035] The nominal visual feature pool corresponding to the category to which the image to be identified belongs is pre-created for the category using the coreset subsampling mechanism. Furthermore, this nominal visual feature pool contains multiple nominal visual tokens. The specific method for creating the nominal visual feature pool is the existing method proposed in the paper "TowardsTotal Recall in Industrial Anomaly Detection," and will not be elaborated upon here.

[0036] S105. Generate multiple mask images corresponding to the first feature image. Based on the multiple mask images, the first feature image, and the reference feature image, generate a set of masked first feature images and a set of masked reference feature images to obtain two sets of feature images.

[0037] S106. The two sets of feature maps are reconstructed using the trained reconstruction model to generate a first reconstructed feature map and a reference reconstructed feature map, resulting in two sets of reconstructed feature maps. The trained reconstruction model is obtained by training with training samples of various different categories.

[0038] S107. Determine the correlation difference between the two sets of reconstructed feature maps and the reconstruction error of the first feature map. Based on the reconstruction error and correlation difference, determine the anomaly score of the image to be identified.

[0039] In this invention, S104 is specifically implemented through the following steps:

[0040] S1041. From the preset nominal visual feature pool corresponding to the category to which the image to be identified belongs, find the nominal visual label that is most similar to each visual label to obtain multiple target nominal visual labels.

[0041] When searching for the most similar nominal visual tag for each visual tag, the nearest-neighbor searching method can be used to find the most similar nominal visual tag for each visual tag and use it as a target nominal visual tag for that visual tag.

[0042] S1042. Replace the corresponding visual labels with multiple target nominal visual labels and use them as the reference feature map corresponding to the first feature map.

[0043] Here, the most similar nominal visual label for each visual label is used to replace the visual label, thereby obtaining the reference feature map corresponding to the first feature map.

[0044] For example, if the size of the first feature map is h×w×c, then the size of the reference feature map corresponding to the first feature map is also h×w×c.

[0045] In this invention, the step of "generating multiple mask images corresponding to the first feature map" in S105 above is specifically implemented through the following steps:

[0046] S1051. Generate an anomaly score map of the image to be identified; the anomaly score map represents the anomaly score of each image block in the image to be identified.

[0047] Here, an existing anomaly score map generation method is used to generate an anomaly score map of the image to be identified.

[0048] S1052. Using the first mask generator, generate n prototype-oriented masks for the feature map; the mask ratios of the n prototype-oriented masks are different; n is a positive integer greater than or equal to 1.

[0049] Here, the specific value of n can be set according to actual needs, and this invention does not limit it. When generating n prototype guide masks for the anomaly score map, the generation of n prototype guide masks for the anomaly score map can be controlled by using n different mask ratios. The same applies to the n random mask maps and n block mask maps described below.

[0050] S1053. Using a second mask generator, n random masks, n blockwise masks, and one zero mask are generated for the first feature map, respectively; the mask ratios of the n random masks are different, and the mask ratios of the n blockwise masks are also different.

[0051] Here, a random masking algorithm can be used to generate n random masking images of the first feature map, and a block masking algorithm can be used to generate n block masking images of the first feature map.

[0052] Here, a zero-mask map means that no masking is performed; that is, a zero-mask map is a tensor that is all zeros.

[0053] It should be noted that the principles of generating anomaly score maps of images and the principles of generating mask maps of feature maps using maskers can be found in the relevant content in the paper One-for-All: Proposal Masked Cross-Class Anomaly Detection, and will not be repeated here.

[0054] S1054. Take n prototype-guided mask images, n random mask images, n block mask images, and 1 zero mask image as multiple mask images corresponding to the first feature image.

[0055] In this invention, the step S105 above, "generating a set of masked first feature maps and a set of masked reference feature maps based on multiple mask maps, first feature maps, and reference feature maps, to obtain two sets of feature maps," is specifically implemented through the following steps:

[0056] S1055. Copy the first feature map and the reference feature map 3n+1 times each, to obtain 3n+1 first feature maps and 3n+1 reference feature maps; wherein, each of the 3n+1 first feature maps and 3n+1 reference feature maps has a different mask than the 3n+1 reference feature maps. Figure 1 One-to-one correspondence.

[0057] S1056. Multiply each first feature map with its corresponding mask map to obtain 3n+1 masked first feature maps. Use these 3n+1 masked first feature maps as a set of masked first feature maps.

[0058] S1057. Multiply each reference feature map by its corresponding mask map to obtain 3n+1 masked reference feature maps. Use these 3n+1 masked reference feature maps as a set of masked reference feature maps.

[0059] In this invention, after obtaining the first feature map after 3n+1 masks and the reference feature map after 3n+1 masks, in the above-mentioned S106, both the first feature map after 3n+1 masks and the reference feature map after 3n+1 masks are input into the trained reconstruction model, and the trained reconstruction model outputs 3n+1 reconstructed first feature maps and 3n+1 reconstructed reference feature maps.

[0060] In this invention, each set of reconstructed feature maps contains N reconstructed feature maps; based on this, the step of "determining the correlation difference between the two sets of reconstructed feature maps and the reconstruction error of the first feature map" in S107 above is specifically implemented through the following steps:

[0061] S1071. Based on the first feature map and N reconstructed first feature maps, determine the first empirical distribution; the first empirical distribution represents the N reconstruction errors of each image block in the first feature map.

[0062] The expression for the first empirical distribution is: Among them, P h Represents the first empirical distribution. h represents the first feature map. Let represent the first reconstructed feature map of the i-th image, and ∥.∥2 represent the L2 norm. If the size of the first feature map is h×w×c, then the first feature map and each reconstructed first feature map contain h×w image blocks. When N reconstructed first feature maps are obtained, each image block can obtain N reconstruction results, and thus each image block has N reconstruction errors. This represents the reconstruction error of each image block under the i-th mask image.

[0063] S1072. Based on the reference feature map and N reconstructed reference feature maps, determine the second empirical distribution; the second empirical distribution represents the N reconstruction errors of each image block in the reference feature map.

[0064] The expression for the second empirical distribution is: P r This represents the second empirical distribution. r represents the reference feature map. Let j represent the reference feature map for reconstruction. If the size of the reference feature map is h×w×c, then both the reference feature map and each reconstructed reference feature map contain h×w image blocks. When N reconstructed reference feature maps are obtained, each image block can obtain N reconstruction results, and thus each image block has N reconstruction errors. This represents the reconstruction error of each image block under the i-th mask image.

[0065] S1073. Determine the transmission distance between the first empirical distribution and the second empirical distribution, and use the transmission distance as the correlation difference between the N reconstructed first feature maps and the N reconstructed reference feature maps.

[0066] Here, the optimal transmission (OT) distance algorithm can be used to determine the transmission distance AssDis(h,r;x) between the first empirical distribution and the second empirical distribution. The expression for the optimal transmission (OT) distance algorithm is:

[0067] Here, by minimizing the target Establish from P h To P r The transmission matrix M * ∈R N×N At that time, the transmission matrix M * Constraints must be followed in, and Indicates that at P h and P r Two uniformly distributed priors are defined in the equation. The cost matrix C ∈ R. N×N Defined as Transmission Matrix M *As a probabilistically reweighted cost matrix C with different masks, it measures the importance of different distances between two reconstruction error distributions. Specifically, the expression for the correlation difference is: AssDis(h,r;x)=M * C.

[0068] S1074. Determine the difference between each reconstructed first feature map and the first feature map, and determine the reconstruction error of the first feature map based on the L2 norm of the difference.

[0069] Here, the expression for the reconstruction error of the first feature map is: h represents the first feature map. Let ||.||2 represent the first feature map of the i-th reconstruction, and let ||.||2 represent the L2 norm.

[0070] Here, both the correlation difference and the reconstruction error are tensors.

[0071] In this invention, the step of "determining the anomaly score of the image to be identified based on reconstruction error and correlation difference" in S107 above is specifically implemented through the following steps:

[0072] S1075. After adding the reconstruction error and the correlation difference, we get the sum of errors.

[0073] Here, the correlation difference and reconstruction error are both tensors, so the error sum is also a tensor. In addition, the anomaly score map is also a tensor.

[0074] S1076. Multiply the error sum element-wise with the outlier score graph to obtain the initial outlier score.

[0075] S1077. Upsample the initial anomaly score to obtain the anomaly score of the image to be identified.

[0076] For example, the expression for the anomaly score of the image to be identified is as follows:

[0077]

[0078] Among them, M po This is the anomaly score map of the image to be identified.

[0079] Here, since the size of the outlier score is smaller than that of the image to be identified, the size of the outlier score is adjusted to be the same as that of the image to be identified through upsampling.

[0080] In this invention, prior to S106 above, the following steps are also included:

[0081] S201. Obtain the training set, which includes training samples of various different categories.

[0082] S202. When reconstructing the model each time, randomly select multiple training samples from the training set.

[0083] For example, the reconstruction model can be a VIT-based encoder-decoder model.

[0084] S203. Extract the visual features of each selected training sample to obtain the second feature map of each selected training sample.

[0085] Here, the principle of S203 is the same as that of S102 described above.

[0086] S204. Generate m different mask images of the second feature map, and embed the m different mask images into the second feature map to obtain the mask feature map.

[0087] Here, a masker can be used to generate n1 random mask images of the second feature map through a random masking algorithm, and n2 random mask feature images of the second feature map can be generated through a block masking algorithm, where n1+n2=m, and the values ​​of n1 and n2 can be set according to actual needs.

[0088] Here, the m different mask images are embedded into the second feature image by sequentially multiplying them element-wise with the second feature image to obtain the mask feature image.

[0089] S205. Input the multiple mask feature maps corresponding to the selected training samples one-to-one into the reconstruction model to be trained to obtain multiple mask features. Figure 1 One-to-one correspondence of multiple reconstructed mask feature maps.

[0090] For example, when a mask feature map is At that time, the reconstructed mask feature map can be represented as Where E represents the encoder, D represents the decoder, and h′ represents the second feature map.

[0091] S206. Based on the differences between multiple mask feature maps and multiple corresponding reconstructed mask feature maps, determine the loss value, adjust the weight parameters of the reconstructed model to be trained according to the loss value, and obtain the trained reconstructed model through multiple iterations of training on the reconstructed model to be trained.

[0092] For example, the expression for the loss value is:

[0093] The specific training iteration process described here is the existing training iteration process, and will not be repeated here.

[0094] The present invention also provides a cross-category image anomaly detection device based on a parallel mask autoencoder, comprising a processor, a communication interface, a memory, and a communication bus. The processor, the communication interface, and the memory communicate with each other through the communication bus. The memory is used to store computer programs. When the processor executes the program stored in the memory, it implements the steps of the above-mentioned cross-category image anomaly detection method based on a parallel mask autoencoder.

[0095] This invention uses a reconstruction model trained with various training samples of different categories for image reconstruction, avoiding the need to train models separately for different object categories, thus achieving greater efficiency and saving resources. Based on multiple mask maps corresponding to the feature maps of the image to be identified, this invention performs masking processing on both the feature maps and reference feature maps of the image to be identified. Then, the two sets of masked feature maps are input into the reconstruction model for feature map reconstruction. The correlation differences between the two sets of reconstructed feature maps, as well as the reconstruction error of the feature maps of the image to be identified, are determined. Based on the reconstruction error and correlation differences, the anomaly score of the image to be identified is determined. This invention can utilize the inherent differences in recovery ability between normal and abnormal patterns under different masking conditions (i.e., abnormal regions exhibit unstable reconstruction when abnormal image blocks are masked, while normal regions exhibit consistent reconstruction) to quantify the reconstruction differences under different masking strategies. It can also utilize the "same shortcut" problem to eliminate the impact of abnormal information leakage. Due to the complexity of generating mask anomalies, abnormal regions exhibit more significant reconstruction differences, thereby improving the detection performance of the model.

[0096] The technical effects of the present invention will be further illustrated by the following three simulation experiments.

[0097] The three simulation experiments used an NVIDIA GeForce RTX 3090 24GB 64-bit Windows operating system, and the simulation language was Python. The experimental data used included the widely used industrial anomaly detection dataset "MVTec-AD," which contains 15 categories and over 5,000 high-resolution images, including objects and textures; and a recently released large dataset, VisA, consisting of 9,621 normal images and 1,200 anomalous high-resolution images, including images with complex structures, scattered objects, and various types of objects, with anomalies including scratches, dents, discoloration, cracks, and structural defects.

[0098] Experiment 1:

[0099] The input image size for the MVTec-AD dataset is 224×224×3. After inputting into the pre-trained EfficientNet model, the feature map becomes 14×14×272, i.e., the image patch size is 16. Then, this invention maps the channel dimension of each patch to 768 before inputting it into a 6-layer ViT-based encoder and a 6-layer ViT-based decoder. We use AdamW for optimization with a weight decay of 0.05. The model of this invention is trained for 1000 epochs on a single GPU (NVIDIA GeForce RTX 3090 24GB) with a batch size of 36. The initial learning rate is 1×10⁻⁶. -4 After 800 epochs, the performance decreased by 0.1. In addition to the pre-trained EfficientNet, the model of this invention is trained from scratch. During testing, this invention randomly generates four masks for each masking strategy. The ratio of random masks to block masks is fixed at 0.3 and 0.1, respectively, while prototype-guided masks are adaptively generated with a ratio ranging from 0.01 to 0.04.

[0100] Please see Figure 2 , Figure 2 This image illustrates the qualitative results of anomaly localization on the MVTec-AD dataset obtained using the method of this invention. The method of this invention can effectively locate anomaly regions of various types, including both object anomalies and texture damage. Figure 3 This is a graph illustrating the quantitative results of anomaly detection and localization on the MVTec-AD dataset obtained using the method described in this embodiment of the invention. Figure 3 As shown, the anomaly detection and localization performance of the model of this invention (Ours) is generally superior to all competitors.

[0101] Experiment 2:

[0102] Please see Figure 4 , Figure 4 This is a graph showing the quantitative results of anomaly detection and localization on the VisA dataset obtained using the method of this invention. Figure 4 The superior performance of the model of this invention compared to other reconstruction-based methods is demonstrated under uniform settings. Across various categories, including complex structures, single instances, and multiple instances, the invention exhibits robust and consistent performance. These results confirm the effectiveness of the model of this invention in handling diverse anomalies.

[0103] Experiment 3:

[0104] Please see Figure 5 , Figure 5 These are diagrams illustrating the reconstruction results using the method of this invention under different masks. Figure 5The invention demonstrates that its model successfully masks suspicious anomalous regions using a prototype-guided masking strategy. When an anomalous region is masked, the invention successfully reconstructs the anomalous region into its corresponding normal sample, while when the anomalous region is directly input into the model, the anomalous part remains unchanged. Subsequently, the invention's association difference learning model explores the differences between various reconstruction error maps, successfully distinguishing between normal and anomalous samples.

[0105] The above experiments demonstrate that the present invention achieves state-of-the-art performance in anomaly detection and localization.

[0106] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0107] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0108] In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0109] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A cross-category image anomaly detection method based on a parallel mask autoencoder, characterized in that, include: Acquire the image to be recognized; Extract the visual features of the image to be identified to obtain a first feature map; The first feature map is segmented into multiple visual markers; Based on the multiple visual markers and the preset nominal visual feature pool corresponding to the category to which the image to be identified belongs, a reference feature map corresponding to the first feature map is determined; Generate multiple mask images corresponding to the first feature image. Based on the multiple mask images, the first feature image, and the reference feature image, generate a set of masked first feature images and a set of masked reference feature images to obtain two sets of feature images. The two sets of feature maps are reconstructed using a pre-trained reconstruction model to generate a first reconstructed feature map and a reference reconstructed feature map, resulting in two sets of reconstructed feature maps. The pre-trained reconstruction model is trained using training samples of various different categories. Each set of reconstructed feature maps contains N reconstructed feature maps. Determine the correlation difference between the two sets of reconstructed feature maps and the reconstruction error of the first feature map; based on the reconstruction error and the correlation difference, determine the anomaly score of the image to be identified. Determining the correlation difference between the two sets of reconstructed feature maps, and the reconstruction error of the first feature map, includes: Based on the first feature map and N reconstructed first feature maps, a first empirical distribution is determined; the first empirical distribution represents the N reconstruction errors of each image block in the first feature map. A second empirical distribution is determined based on the reference feature map and N reconstructed reference feature maps; the second empirical distribution represents the N reconstruction errors of each image block in the reference feature map. Determine the transmission distance between the first empirical distribution and the second empirical distribution, and use the transmission distance as the correlation difference between the N reconstructed first feature maps and the N reconstructed reference feature maps; The difference between each reconstructed first feature map and the first feature map is determined, and the reconstruction error of the first feature map is determined based on the L2 norm of the difference.

2. The cross-category image anomaly detection method based on a parallel mask autoencoder according to claim 1, characterized in that, Based on the multiple visual markers and the preset nominal visual feature pool corresponding to the category to which the image to be identified belongs, a reference feature map corresponding to the first feature map is determined, including: From the preset nominal visual feature pool corresponding to the category to which the image to be identified belongs, the nominal visual tag most similar to each visual tag is searched to obtain multiple target nominal visual tags; the nominal visual feature pool is created for the category to which the image to be identified belongs using a core set subset sampling mechanism, and the nominal visual feature pool contains multiple nominal visual tags; The corresponding visual markers are replaced by the multiple target nominal visual markers, which are used as the reference feature map corresponding to the first feature map.

3. The cross-category image anomaly detection method based on a parallel mask autoencoder according to claim 1, characterized in that, The generation of multiple mask images corresponding to the first feature map includes: An anomaly score map of the image to be identified is generated; the anomaly score map represents the anomaly score of each image patch in the image to be identified. A first mask generator is used to generate n prototype guide masks for the first feature map; the mask ratios of the n prototype guide masks are different; n is a positive integer greater than or equal to 1. A second mask generator is used to generate n random mask images, n block mask images, and 1 zero mask image for the first feature image; the mask ratios of the n random mask images are different, and the mask ratios of the n block mask images are also different. The n prototype guide mask images, the n random mask images, the n block mask images, and the 1 zero mask image are used as multiple mask images corresponding to the first feature image.

4. The cross-category image anomaly detection method based on a parallel mask autoencoder according to claim 1, characterized in that, The plurality of mask images includes 3n+1 different mask images, where n is a positive integer greater than or equal to 1; based on the plurality of mask images, the first feature image, and the reference feature image, a set of masked first feature images and a set of masked reference feature images are generated, resulting in two sets of feature images, including: The first feature map and the reference feature map are copied 3n+1 times each to obtain 3n+1 first feature maps and 3n+1 reference feature maps; wherein, each of the 3n+1 first feature maps and 3n+1 reference feature maps corresponds one-to-one with the 3n+1 different mask maps. Multiply each of the first feature maps by a corresponding mask map to obtain 3n+1 masked first feature maps. Use the 3n+1 masked first feature maps as a set of masked first feature maps. Each of the reference feature maps is multiplied by a corresponding mask map to obtain 3n+1 masked reference feature maps. These 3n+1 masked reference feature maps are then used as a set of masked reference feature maps.

5. The cross-category image anomaly detection method based on a parallel mask autoencoder according to claim 1, characterized in that, The expressions for the first empirical distribution and the second empirical distribution are as follows: ； ； in, This represents the first empirical distribution. This represents the second empirical distribution. , This represents the first feature map. Indicates the first Zhang's reconstructed first feature map, Represents the L2 norm. , This refers to the reference feature map. Indicates the first Zhang's reconstructed reference feature map.

6. The cross-category image anomaly detection method based on a parallel mask autoencoder according to claim 1, characterized in that, The expression for the reconstruction error of the first feature map is: ； in, This represents the reconstruction error of the first feature map. , This represents the first feature map. Indicates the first Zhang's reconstructed first feature map, This represents the L2 norm.

7. The cross-category image anomaly detection method based on a parallel mask autoencoder according to claim 3, characterized in that, The step of determining the anomaly score of the image to be identified based on the reconstruction error and correlation difference includes: The sum of the reconstruction error and the correlation difference is obtained by adding the two errors together. The initial anomaly score is obtained by multiplying the sum of the errors element-by-element by the anomaly score graph. The initial anomaly score is upsampled to obtain the anomaly score of the image to be identified.

8. The cross-category image anomaly detection method based on a parallel mask autoencoder according to claim 1, characterized in that, Before reconstructing the two sets of feature maps using the trained reconstruction model to generate a first set of reconstructed feature maps and a set of reconstructed reference feature maps, the method further includes: Obtain a training set, which includes training samples of various different categories; During each training iteration of the reconstructed model, multiple training samples are randomly selected from the training set. Extract the visual features of each selected training sample to obtain the second feature map of each selected training sample; Generate m different mask images of the second feature map, and embed the m different mask images into the second feature map to obtain the mask feature map; Multiple mask feature maps corresponding to multiple selected training samples are input into the reconstruction model to be trained, resulting in multiple reconstructed mask feature maps corresponding to multiple mask feature maps. Based on the differences between the multiple mask feature maps and the corresponding multiple reconstructed mask feature maps, a loss value is determined. The weight parameters of the reconstructed model to be trained are adjusted according to the loss value. Through multiple iterations of training on the reconstructed model to be trained, the trained reconstructed model is obtained.

9. A cross-category image anomaly detection device based on a parallel mask autoencoder, comprising a processor, a communication interface, a memory, and a communication bus, characterized in that, The processor, the communication interface, and the memory communicate with each other via the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the steps of the method described in any one of claims 1-8.

Images