Unsupervised image anomaly detection method based on multi-scale mask causal relationship

Abstract

The invention discloses an unsupervised image anomaly detection method based on a multi-scale mask causal relationship. The unsupervised image anomaly detection method is specifically implemented according to the following steps: step 1, constructing an unsupervised image anomaly detection network based on the multi-scale mask causal relationship; 2, designing a loss function; 3, training the network model constructed in the step 1 by using the loss function constructed in the step 2; and 4, testing the network trained in the step 3 by using the test data set, outputting a defect detection result picture marked with defects after a picture to be detected is input, and judging whether the input picture is abnormal or not. The method solves the problems that in the prior art, defect training samples are lacked, and a model is poor in robustness on anomalies of different sizes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of unsupervised image anomaly detection, and particularly relates to an unsupervised image anomaly detection method based on multi-scale mask causality. BACKGROUND

[0002] Image anomaly detection is an important task in the field of computer vision, aiming to identify areas, features or objects in images that are inconsistent with normal or expected patterns. It analyzes pixels, regions or entire images in the image to determine whether they conform to the normal pattern, and those that do not conform are considered abnormal. Compared with general classification, detection and segmentation, the definition of abnormal categories in the anomaly detection task is usually difficult. In addition, it is difficult to obtain and the amount of data of abnormal samples is limited. For example, in the case of cancer detection, hospitals may have a large number of normal samples (normal CT images), but the images of cancer patients may be very few. If a simple binary classification task is performed, there is an extreme sample imbalance problem, and the model will have a serious overfitting problem. Therefore, in view of the problem of difficult acquisition of abnormal samples, the current research is mainly based on an unsupervised setting, that is, only normal samples can be used for training in the training stage, and normal samples and different types of defect abnormal samples constitute the test set in the inference stage.

[0003] Compared with other anomaly detection tasks, in image anomaly detection, normal samples include multiple types of products, defects can be regarded as "abnormalities" in appearance, and anomalies in images often appear in a small part of the image, with lower significance and ambiguous semantic concepts. Therefore, general anomaly detection often only needs to distinguish between normal and abnormal samples, while image anomaly detection focuses on detecting abnormal pixels in images. As shown in the anomaly detection task, it is required to locate the pixel position of the anomaly, so the image anomaly task can also be regarded as a kind of anomaly segmentation task. Figure 1

[0004] The existing method is affected by the fact that the image block input size of the pre-trained ViT network [1] can only be a fixed size (such as 16x16), and only a fixed single size input is used for reconstruction, without considering different scale division methods (such as 16x16 and 32x32) for the reconstruction task, and focusing on different size anomalies. And the current anomaly detection method does not analyze the problem from the perspective of causal learning, which will seriously affect the robustness of the model when there are non-causal factors (information inconsistency introduced by division method), which will lead to the judgment of non-causal factors as anomalies, so it is necessary to realize accurate positioning of anomalies and robustness of non-causal factors through the constraint of causal consistency. SUMMARY

[0005] ​The purpose of the present invention is to provide an unsupervised image anomaly detection method based on multi-scale mask causality, which solves the problems of lack of defective training samples and poor robustness of the model to anomalies of different sizes in the prior art.

[0006] The technical solution adopted by the present invention is an unsupervised image anomaly detection method based on multi-scale mask causality, which is specifically implemented according to the following steps:

[0007] Step 1: Construct an unsupervised image anomaly detection network based on multi-scale mask causality;

[0008] Step 2: Design the loss function;

[0009] Step 3: Use the loss function constructed in step 2 to train the network model constructed in step 1;

[0010] Step 4: Use the test data set to test the network trained in step 3. When a picture to be detected is input, the network will output a defect detection result picture with defects marked and determine whether there is any abnormality in the input picture.

[0011] The present invention is also characterized in that:

[0012] Step 1 is implemented as follows:

[0013] Step 1.1, construct a multi-scale mask image block input;

[0014] Step 1.2: Construct a multi-scale mask reconstruction module;

[0015] Step 1.3: Construct a semantic causal consistency constraint module;

[0016] Step 1.4: Construct a multi-scale causal consistency constraint module.

[0017] Step 1.1 is implemented as follows:

[0018] image Use two partitioning methods p1×p1 and p2×p2, p1=16, p2=32, to divide the image I into n1 and n2 image blocks and i∈{1,2,3,…,n1},j∈{1,2,3,…,n2},the original resolution of the image is (h,w), the number of channels is c, and the total number of image blocks obtained by segmentation is n i =hw / p a 2 , the resolution of the segmented image is (p a ,p a ), where a∈{1,2}, select n masked_i =[n amasking to get representative mask input, where η is the masking rate, a∈{1,2}, n masked pixels of the image block are replaced by zero vector, respectively getting mask input of 16x16 division mode and mask input of 32x32 division mode i1∈{1,2,3,…,n masked1},j1∈{1,2,3,…,n masked_2}。

[0019] Step 1.2 is implemented according to the following steps:

[0020] The mask input of 16x16 division obtained in step 1.1 and the image block not masked are input into the ViT-b-16 [3] pre-training model for encoding and decoding, wherein, {1,2,3,…,n masked_1},i2∈{1,2,3,…,n1-n masked_1},the mask input of 32x32 division and the image block not masked are input into the ViT-b-32 [1] pre-training model for encoding and decoding, wherein, j1∈{1,2,3,…,n masked_2},j2∈{1,2,3,…,n2-n masked_2}, finally getting the reconstruction output of two divisions: and wherein, i∈{1,2,3,…,n1},j∈{1,2,3,…,n2},

[0021]

[0022] Y i =D 16×16 (E 16×16 (X i )) (3)

[0023] Y j =D 32×32 (E 32×32 (X j )) (4)

[0024] X i , X j represent the joint of the mask image block of 16x16 division and 32x32 division and the unmasked image block as the encoder E 16×16 (·) and E 32×32input of (·), then the E 16×16 (·) and E 32×32 input of (·) into decoder D 16×16 (·) and D 32×32 output of (·) after reconstruction, wherein E 16×16 (·) and E 32×32 (·) and (·) represent the encoder of ViT-b-16, ViT-b-32 pre-training model respectively [3][1] , D 16×16 (·) and D 32×32 (·) and (·) represent the decoder of ViT-b-16, ViT-b-32 pre-training model respectively [3][1] ;

[0025] Introducing FlexiViT [4] The proposed weight transformation matrix The weight matrix ω of ViT-b-16 pre-training is transformed and loaded into the ViT-b-32 model to obtain the weight matrix ω of input ViT-b-32 model where p * and p represent the size of the transformed image block and the original size of the image block during pre-training respectively, and the knowledge learned by the ViT-b-16 pre-training model on the 16x16 size image block is transferred to the ViT-b-32 model according to this method:

[0026]

[0027] is the transformation matrix calculated by the initial weight matrix ω processed by the bilinear interpolation function F(·), (·) + and (·) T represent the pseudo-inverse and transpose operation of matrix respectively, is the optimal transformation matrix for transforming the initial weight matrix ω of ViT-b-16 to adapt to the parameters of ViT-b-32 model.

[0028] Step 1.3 is implemented according to the following steps:

[0029] Considering the semantic causal consistency before and after the reconstruction of the masked image block, the pixel reconstruction task is converted into a robust semantic reconstruction task, and the semantic transformation T [2] The feature embedding reconstructed by the ViT-b-16, ViT-b-32 pre-training model is assigned a word vector, and the semantic transformation T is a network structure containing an encoder and a hierarchical vector quantization layer [2] The feature embedding obtained by each image block is encoded into a discrete visual semantic label, and first passes through the pre-trained publicly available semantic transformation T [2] Each image is labeled as Large and small grids or Visual semantic labeling of large and small grids V i and V j :

[0030]

[0031] is the masked visual semantic tag under the 16×16 partition mode, is the unmasked visual semantic tag under the 16×16 partition mode, is the masked visual semantic tag under the 32×32 partition mode, is the unmasked visual semantic label in the 32×32 partition mode, i1∈{1,,2,3,…,n masked_1}, i2∈{1,2,3,…,n1-n masked_1}, j1∈{1,2,3,…,n masked_2}, j2∈{1,2,3,…,n2-n masked_2}, and divide all the original image blocks into different and It is also fed into the semantic transformation T for discretization into visual semantic tags and where i∈{1,2,3,…,n1} and j∈{1,2,3,…,n2};

[0032]

[0033] is the original visual semantic label of the masked image block in the 16×16 partition mode, is the original visual semantic label of the unmasked image block in the 16×16 partition mode, is the original visual semantic label of the masked image block in the 32×32 partition mode, is the original visual semantic label of the unmasked image block in the 32×32 partition mode; i1∈{1,2,3,…,n masked_1}, i2∈{1,2,3,…,n1-n masked_1}, (j1∈{1,2,3,…,n masked_2}, (j2∈{1,2,3,…,n2-n masked_2}, using the encoded visual tags by computing the semantic consistency loss before and after reconstruction Measuring Visual Semantic Labeling for Mask Patch Reconstruction Compared with the actual original visual semantic label The differences:

[0034]

[0035] The representative vocabulary set is obtained by clustering a large-scale unlabeled image dataset into 8192 class centers, the number of vocabulary in the representative vocabulary set, and The loss values calculated by the two division methods respectively are minimized to ensure that the visual semantic labels of the masked image blocks after reconstruction are close to the visual semantic labels of the original image blocks , thereby ensuring semantic consistency before and after, where n masked_1 and n masked_2 are the number of mask patches under the two divisions, and represent the probability that the m α (α∈{1,2}) mask patch belongs to the kth visual semantic label, where m1∈{1,2,3,…,n masked_1}, m2∈{1,2,3,…,n masked_2}, k∈{1,2,3,…,8192};

[0036] The overall semantic distribution of all reconstructed image blocks is constrained, and the semantic distribution loss is calculated using the encoded visual semantic labels to measure the distribution difference between the overall visual semantic labels of the image blocks and the actual original visual semantic labels:

[0037]

[0038] where n1 and n2 represent the number of 16x16 and 32x32 divided image blocks, respectively, and The loss values calculated by the two division methods respectively are minimized to ensure that the overall semantic of the image blocks after reconstruction is close to the overall semantic of the original image blocks, thereby ensuring semantic distribution consistency before and after, where represents the true value of the mth image block at the kth visual label, i.e., the true value of the original image block at the kth visual label, and the output value of the mth reconstructed image block at the kth visual label.

[0039] Step 1.4 is implemented according to the following steps:

[0040] By constructing a multi-scale causal consistency constraint module, the scale causal consistency of the two scales is constrained. Specifically, first, the feature embeddings Y i and Y j of the decoder outputs of the two partitions obtained in step 1.2 are subjected to multi-scale causal consistency constraint and Since the sizes of the two partitioned input image blocks are different, i∈{1,2,3,…,n1} and j∈{1,2,3,…,n2}, the sizes of the final feature embeddings obtained are also inconsistent, and the output dimensions of the output feature embeddings of the two partitions are and It is understood that a single image block on the ViT-b-32 partition corresponds to four image blocks on the ViT-b-16 partition spliced according to the corresponding positions, so according to the position index, the corresponding position is found, and then Y j k corresponding Y i l First, the average information of the corresponding region features is extracted using the average pooling operation, where k∈{1,2,3,…,196} and l∈{1,2,3,…,49} represent the position indexes of the feature embeddings of the two partitions, then the feature embeddings of the two branches are flattened to obtain the flattened features F1 and F2 of the two partitions, with a dimension of (1,49*768), and the flattened features F1 and F2 are input into the same linear mapping layer for linear transformation, thereby unifying the feature distribution to obtain the features f(F1) and f(F2) in the same feature space, and the scale causal consistency loss measures the difference between multi-scale features:

[0041]

[0042] where f(·) represents the same linear mapping layer, which is used to map features in different spaces into the same feature space for easy comparison, and ‖·‖ represents the two-norm of the feature embedding, The causal consistency of the features of the two partitions of different scales is constrained, avoiding the destruction of the original causal relationship of the features due to scale changes.

[0043] Step 2 is implemented according to the following steps:

[0044] The total loss of the unsupervised image anomaly detection network is composed of three parts, which are the semantic consistency loss that constrains the semantic consistency before and after the reconstruction of a single image block, the distribution consistency loss that constrains the distribution consistency before and after the reconstruction of the entire image block, and the multi-scale causal consistency loss that measures the causal consistency of features of different scales Therefore, the total loss is represented as:

[0045]

[0046] Step 3 is specifically implemented according to the following steps:

[0047] The network constructed in step 2 is trained using the loss function constructed in step 3 to update the network weight parameters, and the network uses AdamW [6] As the optimizer, the learning rate is set to 0.00001, the batch size is set to 4, 500 epochs are set, and the detailed network parameters are shown in Table 1.

[0048] The unsupervised image anomaly detection method based on multi-scale mask causal relationship has the advantages that different division scales and mask rates have a great influence on the performance of the mask autoencoder, and the model can reconstruct and focus on different size anomalies to improve the model performance by considering these factors and adopting the design of multiple mask inputs of different division scales. [9] The causal consistency learning and reconstruction of normal features are realized by using semantic consistency loss, semantic distribution loss and multi-scale causal consistency loss. In the prediction stage, the anomaly detection of different scale branches is fused to obtain more accurate prediction results. The present application performs well in image multi-class anomaly detection and cross-class anomaly detection tasks, and is superior to the performance of the existing advanced image anomaly detection network. The model fully considers the coarse-grained semantic causal consistency and causal emergence in different macro scales, can make the model have better robustness to different size anomalies, and has certain advantages in quantitative evaluation indicators. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is an image anomaly example involved in the background technology of the unsupervised image anomaly detection method based on multi-scale mask causal relationship of the present application;

[0050] Figure 2 is a model structure diagram of the unsupervised image anomaly detection method based on multi-scale mask causal relationship of the present application;

[0051] Figure 3 is a scale causal consistency module diagram of the unsupervised image anomaly detection method based on multi-scale mask causal relationship of the present application;

[0052] Figure 4 is a semantic consistency module of the unsupervised image anomaly detection method based on multi-scale mask causal relationship of the application;

[0053] Figure 5 is an abnormal positioning result example in the unsupervised image anomaly detection method based on multi-scale mask causal relationship of the application. DETAILED DESCRIPTION

[0054] The application will be described in detail below in combination with the drawings and specific embodiments.

[0055] The unsupervised image anomaly detection method based on multi-scale mask causal relationship of the application proposes a new unsupervised image anomaly detection framework based on mask reconstruction and causal relationship mining to enhance the robustness of the model to non-causal factors. On the one hand, the input image is divided into two different sizes and masked, so that the model can pay attention to different size anomalies. On the other hand, the reconstructed image block is embedded to assign word vectors as the semantic label of each block using a marker. The semantic consistency loss and the distribution consistency loss are used to constrain the semantics of a single image block and the semantic distribution of all image blocks to achieve semantic causal consistency. At the same time, the corresponding image block is found according to the image position embedding during encoding, and the multi-scale causal consistency loss is used to measure the similarity of the corresponding blocks under different divisions to achieve scale causal consistency. In the inference stage, the uncertainty of the obtained label is used to obtain the anomaly score of each image block, and the prediction results of each branch are fused and output as the final prediction result.

[0056] The unsupervised image anomaly detection method based on multi-scale mask causal relationship of the application is implemented in accordance with the following steps:

[0057] Step 1, constructing an unsupervised image anomaly detection network based on multi-scale mask causal relationship;

[0058] Step 1 is implemented in accordance with the following steps:

[0059] Step 1.1, constructing a multi-scale mask image block input;

[0060] Step 1.2, constructing a multi-scale mask reconstruction module;

[0061] Step 1.3, constructing a semantic causal consistency constraint module;

[0062] Step 1.4, constructing a multi-scale causal consistency constraint module.

[0063] The network structure and hyperparameter settings are shown in Table 1, and Step 1.1 is implemented in accordance with the following steps:

[0064] Image Use two partitioning methods p1×p1 and p2×p2, p1=16, p2=32, to divide the image I into n1 and n2 image blocks and i∈{1,2,3,…,n1},j∈{1,2,3,…,n2},the original resolution of the image is (h,w), the number of channels is c, and the total number of image blocks obtained by segmentation is n i =hw / p a 2 , the resolution of the segmented image is (p a ,p a ), where a∈{1,2}, then considering the generation of more mask patches in the difficult-to-reconstruct areas, a combined mask strategy is adopted to train a model with stronger reconstruction ability and better generalization ability. Combining region-restricted masking, frequency-based masking, dynamic masking, random masking and block masking [5] These masking strategies are used to select n masked_i =[n a ×η] image blocks are masked to obtain a representative mask input, where η is the masking rate, a∈{1,2}, and n masked The pixels of the image blocks are replaced by zero vectors, and the mask inputs of 16×16 partitioning are obtained respectively. and 32×32 partitioning mask input

[0065] Step 1.2 is implemented as follows:

[0066] Input the 16×16 divided mask obtained in step 1.1 and the unmasked image patches Input to ViT-b-16 [3] Encoding and decoding are performed in the pre-trained model, where i1∈{1,2,3,…,n masked_1}, i2∈{1,2,3,…,n1-n masked_1}, 32×32 divided mask input and the unmasked image patches Input to ViT-b-32 [1] Encoding and decoding are performed in the pre-trained model, where j1∈{1,2,3,…,n masked_2}, j2∈{1,2,3,…,n2-n masked_2}, and finally get the reconstruction output of two partitions: and Among them, i∈{1,2,3,…,n1}, j∈{1,2,3,…,n2},

[0067]

[0068] Y i = D 16×16 (E 16×16 (X i )) (3)

[0069] Y j = D 32×32 (E 32×32 (X j )) (4)

[0070] X i , X j represent the joint of the masked image patches and the unmasked image patches of 16x16 partition and 32x32 partition as the input of the encoder E 16×16 (·) and E 32×32 (·), then the output of E 16×16 (·) and E 32×32 (·) is input into the decoder D 16×16 (·) and D 32×32 (·) to get the reconstructed output, where E 16×16 (·) and E 32×32 (·) represent the encoder of ViT-b-16, ViT-b-32 pre-training model respectively [3][1] , D 16×16 (·) and D 32×32 (·) represent the decoder of ViT-b-16, ViT-b-32 pre-training model respectively [3][1] ;

[0071] In order to avoid the problem of needing a large number of batches of training caused by loading multiple different pre-training models, FlexiViT is introduced [4] The proposed weight transformation matrix transforms the weight matrix ω of the ViT-b-16 pre-training model and loads it into the ViT-b-32 model to get the weight matrix ω of the input ViT-b-32 model, where p * and p represent the size of the transformed image patch and the original size of the image patch during pre-training respectively, and according to this method, the knowledge learned by the ViT-b-16 pre-training model on the 16x16 size image patch is transferred to the ViT-b-32 model:

[0072]

[0073] is the initial weight matrix ω calculated after being processed by the bilinear interpolation function F(·), (·) + and (·) T represent the pseudo-inverse and transpose operation of the matrix respectively, is the optimal transformation matrix for transforming the ViT-b-16 initial weight matrix ω to adapt to the ViT-b-32 model parameters.

[0074] Step 1.3 is specifically implemented according to the following steps:

[0075] Considering the semantic causal consistency before and after the reconstruction of the masked image block, the pixel reconstruction task is converted into a robust semantic reconstruction task, and a semantic transformation T is used [2] The feature embedding reconstructed by the ViT-b-16 and ViT-b-32 pre-training model is subjected to word vector allocation, and the semantic transformation T is a network structure including an encoder and a hierarchical vector quantization layer [2] The feature embedding obtained by each image block is encoded into a discrete visual semantic label, which is first subjected to a pre-trained publicly available semantic transformation T [2] Each image is labeled as a visual semantic label V of a size grid or a visual semantic label V

[0076] of a size grid i and V j :

[0077] For the masked visual semantic label under the 16x16 division mode, For the unmasked visual semantic label under the 16x16 division mode, For the masked visual semantic label under the 32x32 division mode, For the unmasked visual semantic label under the 32x32 division mode, i1∈{1,,2,3,…,n masked_1}, i2∈{1,2,3,…,n1-n masked_1}, j1∈{1,2,3,…,n masked_2}, j2∈{1,2,3,…,n2-n masked_2}, and all the image blocks of different divisions and are also sent to the semantic transformation T for discrete conversion into visual semantic labels and where i∈{1,2,3,…,n1} and j∈{1,2,3,…,n2};

[0078]

[0079] For the original visual semantic label of the masked image block under the 16x16 division mode, Original visual semantic labels for unmasked image patches under 16x16 partitioning, Original visual semantic labels for masked image patches under 32x32 partitioning, Original visual semantic labels for unmasked image patches under 32x32 partitioning; i1 e {1, 2, 3, …, n masked_1}, i2 e {1, 2, 3, …, n1-n masked_1}, (j1 e {1, 2, 3, …, n masked_2}, (j2 e {1, 2, 3, …, n2-n masked_2}, the semantic consistency loss before and after reconstruction is calculated using the encoded visual labels Visual semantic labels of the masked patch reconstruction The difference between the actual original visual semantic labels :

[0080]

[0081] v represents the vocabulary set obtained by clustering a large-scale unlabeled image dataset into 8192 class centers, |v| represents the number of vocabularies in the vocabulary set, and is set to |v| = 8192. This number is the result of balancing the representation ability of the model, computing resources, and task requirements. and represent the loss values calculated by the two partitioning methods respectively, by minimizing to ensure that the visual semantic labels of the masked image patches after reconstruction are close to the visual semantic labels of the original image patches , thereby ensuring semantic consistency before and after, where n masked_1 and n masked_2 are the number of masked patches under the two partitioning methods, and represent the probability that the m α (α e {1, 2})th masked patch belongs to the kth visual semantic label under the two partitioning methods, where m1 e {1, 2, 3, …, n masked_1}, m2 e {1, 2, 3, …, n masked_2}, k e {1, 2, 3, …, 8192};

[0082] In addition to the semantic consistency constraint before and after the single masked image patch, the overall semantic distribution of all reconstructed image patches is also considered. Similarly, the semantic distribution loss is calculated using the encoded visual semantic labels to measure the distribution difference between the overall visual semantic labels of the image patches and the actual original visual semantic labels:

[0083]

[0084] wherein n1 and n2 represent the number of image blocks of 16x16 partition and 32x32 partition respectively, and represent the loss values calculated by two partition methods respectively, by minimizing ensure that the overall semantics of the image blocks after reconstruction are close to the overall semantics of the original image blocks, thereby ensuring the consistency of the semantics before and after, wherein represents the true value of the mth image block at the kth visual marker, i.e., the true value of the original image block at the kth visual marker, is the output value of the mth image block after reconstruction at the kth visual marker.

[0085] Step 1.4 is implemented according to the following steps:

[0086] Note that the abnormality of each picture at the same position under different partition methods is actually the same, so by constructing a multi-scale causal consistency constraint module, the scale causal consistency under two scales is constrained. Specifically, first, the feature embeddings Y i and Y j output by the decoder under two partitions obtained in step 1.2 are subjected to multi-scale causal consistency constraint. Before multi-scale causal consistency constraint, since the input image blocks under two partitions and are of different sizes, i∈{1,2,3,…,n1}, j∈{1,2,3,…,n2}, the final feature embeddings obtained are also of different sizes, and the output dimensions of the output feature embeddings under two partitions are and It is understood that a single image block on the ViT-b-32 partition corresponds to four image blocks on the ViT-b-16 partition by concatenating corresponding positions, therefore, according to the position index (Position Embeddings) to find the corresponding position, this step ensures that the features of different scales establish correct correspondence in spatial position, which is the basis for cross-scale feature alignment, and then Y j k corresponding Y i lFirst, the average information of the corresponding region features is extracted by using the average pooling operation (Average Pooling), where k e {1, 2, 3, …, 196} and l e {1, 2, 3, …, 49} represent the position indexes of the two divided feature embeddings, then the feature embeddings of the two branches are flattened to obtain the flattened features F1 and F2 of the two divisions, with a dimension of (1, 49*768), the flattened features F1 and F2 are input into the same linear mapping layer for linear transformation, so as to unify the feature distribution to obtain the features f(F1) and f(F2) in the same feature space, and prepare for calculating the feature similarity, using the scale causal consistency loss Measure the difference between multi-scale features:

[0087]

[0088] In the formula, f(·) represents the same linear mapping layer, which is used to map features in different spaces into the same feature space for easy comparison, and ‖·‖ represents the two-norm of the feature embedding, Constrain the causal consistency of the two divided features of the model, avoid destroying the original causal relationship of the features due to the change of the scale.

[0089] Step 2, design the loss function;

[0090] Step 2 is implemented according to the following steps:

[0091] The total loss of the unsupervised image anomaly detection network It is composed of three parts, which are the semantic consistency loss (see formula (14)) that constrains the semantic consistency of the single image block before and after reconstruction, the distribution consistency loss (see formula (17)) that constrains the distribution consistency of the overall image block before and after reconstruction, and the multi-scale causal consistency loss (see formula (18)) that measures the causal consistency of different scale features, so the total loss is represented as:

[0092]

[0093] Step 3, use the loss function constructed in step 2 to train the network model constructed in step 1;

[0094] Step 3 is implemented according to the following steps:

[0095] Train the network constructed in step 2, use the loss function constructed in step 3 to train the network and update the network weight parameters, and the network uses AdamW [6]As an optimizer, the learning rate is set to 0.00001, and the batch size is set to 4, and 500 epochs are set.

[0096] Step 4, test the network trained in step 3 with the test data set, and when inputting a picture to be detected, output a defect detection result picture marked with defects and judge whether the input picture is abnormal.

[0097] Table 1 network structure and hyperparameter setting

[0098]

[0099]

[0100] Example 1

[0101] The unsupervised image anomaly detection method based on multi-scale mask causal relationship of the application is implemented according to the following steps:

[0102] Step 1, construct an unsupervised image anomaly detection network based on multi-scale mask causal relationship;

[0103] Step 2, design a loss function;

[0104] Step 3, train the network model constructed in step 1 using the loss function constructed in step 2;

[0105] Step 4, test the network trained in step 3 with the test data set, and when inputting a picture to be detected, output a defect detection result picture marked with defects and judge whether the input picture is abnormal.

[0106] Example 2

[0107] The unsupervised image anomaly detection method based on multi-scale mask causal relationship of the application is implemented according to the following steps:

[0108] Step 1, construct an unsupervised image anomaly detection network based on multi-scale mask causal relationship;

[0109] Step 1 is implemented according to the following steps:

[0110] Step 1.1, construct a multi-scale mask image block input;

[0111] Step 1.2, construct a multi-scale mask reconstruction module;

[0112] Step 1.3, construct a semantic causal consistency constraint module;

[0113] Step 1.4, construct a multi-scale causal consistency constraint module.

[0114] Step 1.1 is implemented according to the following steps:

[0115] Image The image I is divided into image blocks with quantities n1 and n2 by using two division modes p1x p1 and p2x p2, p1 = 16, p2 = 32 And i is in {1, 2, 3, …, n1}, j is in {1, 2, 3, …, n2}, the original resolution of the picture is (h, w), the number of channels is c, and the total number of image blocks obtained by segmentation is n i = hw / p a 2 The resolution of the segmented image is (p a , p a ), wherein a is in {1, 2}, and then a combined mask strategy is used to train a model with stronger reconstruction capability and better generalization capability by considering that more mask patches are generated in a difficult-to-reconstruct area. [5] These several masking strategies are used to select n masked_i = [n a x eta] image blocks for masking to obtain representative mask inputs, wherein eta is a shielding rate, a is in {1, 2}, and in the training process, the pixels of n masked image blocks are replaced with zero vectors, respectively, to obtain mask inputs of 16x 16 division mode And mask inputs of 32x 32 division mode i1 is in {1, 2, 3, …, n masked1}, and j1 is in {1, 2, 3, …, n masked_2}.

[0116] Step 2, designing a loss function;

[0117] Step 3, training the network model constructed in step 1 by using the loss function constructed in step 2;

[0118] Step 4, testing the network trained in step 3 by using a test data set, when a picture to be detected is input, a defect detection result picture in which defects are marked is output, and whether the input picture exists abnormally is judged.

[0119] Embodiment 3

[0120] The unsupervised image anomaly detection method based on the multi-scale mask causal relationship is implemented according to the following steps:

[0121] Step 1, constructing an unsupervised image anomaly detection network based on a multi-scale mask causal relationship;

[0122] Step 1 is implemented according to the following steps:

[0123] Step 1.1, construct a multi-scale mask image block input;

[0124] Step 1.2: Construct a multi-scale mask reconstruction module;

[0125] Step 1.3: Construct a semantic causal consistency constraint module;

[0126] Step 1.4: Construct a multi-scale causal consistency constraint module.

[0127] Step 1.1 is implemented as follows:

[0128] image Use two partitioning methods p1×p1 and p2×p2, p1=16, p2=32, to divide the image I into n1 and n2 image blocks and i∈{1,2,3,…,n1},j∈{1,2,3,…,n1},the original resolution of the image is (h,w), the number of channels is c, and the total number of image blocks obtained by segmentation is n i =hw / p a 1 , the resolution of the segmented image is (p a ,p a ), where a∈{1,2}, then considering the generation of more mask patches in the difficult-to-reconstruct areas, a combined mask strategy is adopted to train a model with stronger reconstruction ability and better generalization ability. Combining region-restricted masking, frequency-based masking, dynamic masking, random masking and block masking [5] These masking strategies are used to select n masked_i =[n a ×η] image blocks are masked to obtain a representative mask input, where η is the masking rate, a∈{1,2}, and n masked The pixels of the image blocks are replaced by zero vectors, and the mask inputs of 16×16 partitioning are obtained respectively. and 32×32 partitioning mask input

[0129] Step 1.2 is implemented as follows:

[0130] Input the 16×16 divided mask obtained in step 1.1 and the unmasked image patches Input to ViT-b-16 [3] Encoding and decoding are performed in the pre-trained model, where i1∈{1,2,3,…,n masked_1}, i2∈{1,2,3,…,n1-n masked_1}, 32x32 partitioned mask input and unmasked image patches input to ViT-b-32 [1] encoding-decoding in the pre-trained model, where j1∈{1, 2, 3, …, n masked_2}, j2∈{1, 2, 3, …, n2-n masked_2}, finally obtaining the reconstruction output of two partitions: and where i∈{1, 2, 3, …, n1}, j∈{1, 2, 3, …, n2},

[0131]

[0132] Y i = D 16×16 (E 16×16 (X i )) (3)

[0133] Y j = D 32×32 (E 32×32 (X j )) (4)

[0134] X i , X j represent the joint of 16x16 partitioned and 32x32 partitioned mask image patches and unmasked image patches as the input of the encoder E 16×16 (·) and E 32×32 (·), and then input the output of E 16×16 (·) and E 32×32 (·) into the decoder D 16×16 (·) and D 32×32 (·) to obtain the output after reconstruction, where E 16×16 (·) and E 32×32 (·) represent the encoder of ViT-b-16 and ViT-b-32 pre-trained model respectively [3][1] , D 16×16 (·) and D 32×32 (·) represent the decoder of ViT-b-16 and ViT-b-32 pre-trained model respectively [3][1] ;

[0135] In order to avoid the problem of needing a large number of batches of training caused by loading multiple different pre-trained models, FlexiViT is introduced [4] The proposed weight transformation matrix The weight matrix ω of ViT-b-16 pre-training is transformed and loaded into ViT-b-32 model to obtain the weight matrix input into ViT-b-32 model where p * and p represent the size of the transformed image block and the original size of the image block at pre-training, respectively, and the knowledge learned by the ViT-b-16 pre-training model on the 16x16 size image block is transferred to the ViT-b-32 model according to this method:

[0136]

[0137] is the initial weight matrix ω of the transformed matrix processed by the bilinear interpolation function F(·), (·) + and (·) T represent the pseudo-inverse and transpose operations of the matrix, respectively, is the optimal transformation matrix for adapting the ViT-b-16 initial weight matrix ω to the ViT-b-32 model parameters after transformation.

[0138] Step 1.3 is implemented according to the following steps:

[0139] Considering the semantic causal consistency before and after the reconstruction of the masked image block, the pixel reconstruction task is converted into a robust semantic reconstruction task, and the semantic transformation T [2] The feature embedding reconstructed by the ViT-b-16 and ViT-b-32 pre-training models is assigned a word vector, and the semantic transformation T is a network structure containing an encoder and a hierarchical vector quantization layer [2] Each feature embedding obtained by each image block is encoded as a discrete visual semantic label, and first passes through the pre-trained public semantic transformation T [2] Each image is labeled as visual semantic labels V of size grid or i and V j :

[0140]

[0141] is the masked visual semantic label under the 16x16 division mode, is the unmasked visual semantic label under the 16x16 division mode, is the masked visual semantic label under the 32x32 division mode, is the unmasked visual semantic label under the 32x32 division mode, i1∈{1,,2,3,…,n masked_1}, i2∈{1,2,3,…,n1-n masked_1}, j1∈{1,2,3,…,n masked_2}, j2∈{1,2,3,…,n2-n masked_2}, while the different divided original ownership image blocks and are also sent to the semantic transformation T for discrete conversion into visual semantic labels and where i ∈ {1, 2, 3, …, n1} and j ∈ {1, 2, 3, …, n2} ;

[0142]

[0143] is the original visual semantic label of the masked image block under the 16 × 16 division mode, is the original visual semantic label of the unmasked image block under the 16 × 16 division mode, is the original visual semantic label of the masked image block under the 32 × 32 division mode, is the original visual semantic label of the unmasked image block under the 32 × 32 division mode; i1 ∈ {1, 2, 3, …, n masked_1}, i2 ∈ {1, 2, 3, …, n1-n masked_1}, (j1 ∈ {1, 2, 3, …, n masked_2}, (j2 ∈ {1, 2, 3, …, n2-n masked_2}, the semantic consistency loss before and after reconstruction is calculated using the encoded visual labels to measure the visual semantic label of the masked patch reconstruction and the actual original visual semantic label :

[0144]

[0145] V represents the vocabulary set obtained by clustering a large-scale unlabeled image dataset into 8192 class centers, and |V| represents the number of vocabularies in the vocabulary set, which is set to |V| = 8192. This number is the result of balancing the representation ability of the model, computing resources, and task requirements, and other aspects. and represent the loss values calculated by the two division modes respectively, and by minimizing to ensure that the visual semantic label of the masked image block after reconstruction is close to the visual semantic label of the original image block , thereby ensuring semantic consistency before and after, where n masked_1 and n masked_2 are the number of masked patches under the two divisions, and represent the m αP (mask patch (a e {1, 2}) belongs to the kth visual semantic label), where m1 e {1, 2, 3, …, n masked_1}, m2 e {1, 2, 3, …, n masked_2}, k e {1, 2, 3, …, 8192} ;

[0146] In addition to the semantic consistency constraint on the single mask image block before and after, the overall semantic distribution of all image blocks after reconstruction is also considered. Similarly, the semantic distribution loss is calculated using the encoded visual semantic label to measure the distribution difference between the overall visual semantic label of the image block and the actual original visual semantic label:

[0147]

[0148] where n1 and n2 represent the number of 16x16 and 32x32 divided image blocks, respectively, and represent the loss values calculated by the two division methods, respectively, and the minimum is used to ensure that the overall semantics of the image block is close to the overall semantics of the original image block after reconstruction, thereby ensuring the semantic distribution consistency before and after, where represents the true value of the mth image block in the kth visual label, i.e., the true value of the original image block in the kth visual label, is the output value of the mth image block in the kth visual label after reconstruction.

[0149] Step 1.4 is implemented according to the following steps:

[0150] Note that the abnormality of each picture at the same position under different division methods is actually the same, so a multi-scale causal consistency constraint module is constructed to constrain the scale causal consistency under the two scales. Specifically, first, the feature embeddings Y i and Y j output by the decoder under the two divisions obtained in step 1.2 are subjected to multi-scale causal consistency constraint. Before the multi-scale causal consistency constraint is performed, the sizes of the input image blocks and under the two divisions are different, i e {1, 2, 3, …, n1}, j e {1, 2, 3, …, n2}, so the sizes of the final feature embeddings obtained are also different. The output dimensions of the output feature embeddings under the two divisions are and ​It is understood that the single image block on the ViT-b-32 division corresponds to the four image blocks on the ViT-b-16 in the corresponding position splicing, therefore, according to the position index (Position Embeddings) to find the corresponding position, this step ensures that the features of different scales establish correct correspondence in the spatial position, which is the basis for cross-scale feature alignment, and then Y j k Corresponding Y i l First, the average information of the corresponding region features is extracted by using the average pooling operation (Average Pooling), where k is in {1, 2, 3, …, 196} and l is in {1, 2, 3, …, 49} represent the position index of the feature embedding of the two divisions, then the feature embeddings of the two branches are flattened to obtain the flattened features F1 and F2 of the two divisions, the dimension is (1, 49*768), the flattened features F1 and F2 are input into the same linear mapping layer for linear transformation, so as to unify the feature distribution to obtain the features f(F1) and f(F2) in the same feature space, and prepare for calculating the feature similarity, using the scale causal consistency loss Measure the difference between multi-scale features:

[0151]

[0152] In the formula, f(·) represents the same linear mapping layer, which is used to map the features in different spaces into the same feature space for easy comparison, and ‖·‖ represents the two-norm of the feature embedding, Constrain the causal consistency of the features of different scales of the two divisions, and avoid destroying the original causal relationship of the features due to the change of the scale.

[0153] Step 2, design a loss function;

[0154] Step 3, use the loss function constructed in step 2 to train the network model constructed in step 1;

[0155] Step 4, test the network trained in step 3 by using a test data set, when a picture to be detected is input, a defect detection result picture with marked defects will be output, and whether the input picture exists abnormally will be judged.

[0156] Embodiment 4

[0157] The unsupervised image anomaly detection method based on multi-scale mask causal relationship of the present application is implemented according to the following steps:

[0158] Step 1, construct an unsupervised image anomaly detection network based on multi-scale mask causal relationship;

[0159] Step 1 is implemented according to the following steps:

[0160] Step 1.1, constructing a multi-scale mask image block input;

[0161] Step 1.2, constructing a multi-scale mask reconstruction module;

[0162] Step 1.3, constructing a semantic causal consistency constraint module;

[0163] Step 1.4, constructing a multi-scale causal consistency constraint module.

[0164] Step 2, designing a loss function;

[0165] Step 2 is implemented according to the following steps:

[0166] The total loss of the unsupervised image anomaly detection network is composed of three parts, which are the semantic consistency loss (see formula (14)) that constrains the semantic consistency of the single image block before and after reconstruction, the distribution consistency loss (see formula (17)) that constrains the distribution consistency of the whole image block before and after reconstruction, and the multi-scale causal consistency loss (see formula (18)) that measures the causal consistency of different scale features, so the total loss is expressed as:

[0167]

[0168] Step 3, using the loss function constructed in step 2 to train the network model constructed in step 1;

[0169] Step 4, using the test data set to test the network trained in step 3, when inputting a picture to be detected, the defect detection result picture marked with defects will be output and it will be judged whether the input picture exists abnormally.

[0170] Embodiment 5

[0171] The unsupervised image anomaly detection method based on multi-scale mask causal relationship according to the following steps is implemented:

[0172] Step 1, constructing an unsupervised image anomaly detection network based on multi-scale mask causal relationship;

[0173] Step 1 is implemented according to the following steps:

[0174] Step 1.1, constructing a multi-scale mask image block input;

[0175] Step 1.2, constructing a multi-scale mask reconstruction module;

[0176] Step 1.3, constructing a semantic causal consistency constraint module;

[0177] Step 1.4, constructing a multi-scale causal consistency constraint module.

[0178] Step 2, designing a loss function;

[0179] Step 2 is implemented according to the following steps:

[0180] Total loss of the unsupervised image anomaly detection network It consists of three parts, namely the semantic consistency loss that constrains the semantic consistency of the single image block before and after reconstruction (see formula (14)) and the distribution consistency loss that constrains the distribution consistency of the whole image block before and after reconstruction (see formula (17)), and the multi-scale causal consistency loss that measures the causal consistency of different scale features (see formula (18)), so the total loss is expressed as:

[0181]

[0182] Step 3, training the network model constructed in step 1 using the loss function constructed in step 2;

[0183] Step 3 is implemented according to the following steps:

[0184] The network constructed in step 2 is trained using the loss function constructed in step 3 to update the network weight parameters. The network uses AdamW [6] as the optimizer, the learning rate is set to 0.00001, the batch size is set to 4, and 500 epochs are set. For detailed network parameters, see Table 1.

[0185] Step 4, test the network trained in step 3 using the test data set. When inputting a picture to be detected, the defect detection result picture with the defects marked will be outputted and it will be judged whether the input picture is abnormal.

[0186] Example 6

[0187] Test the trained network using the test data set:

[0188] Image anomaly detection experiments are conducted on the trained model using the test data set. The experiment is conducted on the MVTec-AD data set [8]Two different task settings more challenging than traditional single-class anomaly detection (Singel-Class Anomaly Detection) are performed on the above, namely multi-class anomaly detection (Multi-Class Anomaly Detection) and cross-class anomaly detection (Cross-Class Anomaly Detection), as shown in Tables 2 and 3, multi-class anomaly detection refers to anomaly detection in the presence of multiple normal class objects. Cross-class anomaly detection is a more challenging anomaly detection task, cross-class anomaly detection refers to training using normal samples from visible classes, learning a unified and generalizable model to detect anomalies from invisible classes.

[0189] The MVTec-AD dataset is the first comprehensive, multi-object, multi-defect, and pixel-level accurate label-provided anomaly detection dataset, containing 5354 high-resolution color images of different object and texture types. It contains normal images for training and abnormal images for testing, and provides pixel-level accurate labels for anomalies.

[0190] Table 2 Multi-class data settings of MVTec-AD dataset

[0191] Dataset Class Sample Number Defective Samples Normal Samples Training Set Test Set MVTec-AD 15 5354 1258 4096 3629 1725

[0192] Table 3 Cross-class data settings of MVTec-AD dataset

[0193]

[0194] The experiment evaluates the detection and segmentation performance of the model through two evaluation indexes I-AUROC and P-AUROC.

[0195] The present application is based on an unsupervised image anomaly detection method based on multi-scale mask causality, considering that existing methods are limited by the input size of the pre-trained ViT network and only consider a single division method for input reconstruction, without considering multiple scale division methods for reconstruction tasks, focusing on different size anomalies. And the current anomaly detection method does not analyze the problem from the perspective of causal learning, which will seriously affect the robustness of the model when there are non-causal factors interference, which will lead to the judgment of non-causal factors as anomalies, therefore the accurate positioning of anomalies and the robustness of non-causal factors need to be realized through the constraint of causal consistency. The method adopts multi-scale mask reconstruction to make the model have good perception ability to different size anomalies; adopt multi-causal constraint method to enhance the robustness of the model to non-causal factors, enhance the reconstruction ability of the model, and obtain better detection and positioning effect in the existing image multi-class anomaly detection and cross-class anomaly detection tasks.

[0196] Experiments are trained and tested under two task settings of multi-class anomaly detection and cross-class anomaly detection on MVTec-AD dataset. The following comparative methods are: PMAD [5] , DualCAM (the method of the present application).

[0197] Table 4 Quantitative evaluation of multi-class anomaly detection on MVTec-AD dataset, · / · represent I-AUROC and P-AUROC respectively

[0198]

[0199]

[0200] Table 5 Quantitative evaluation of cross-class anomaly detection on MVTec-AD dataset, · / · represent I-AUROC and P-AUROC respectively

[0201]

[0202]

[0203] I-AUROC and P-AUROC are the two most commonly used standard metrics for measuring the anomaly detection performance and anomaly localization performance of anomaly detection models respectively. I-AUROC is based on the binary confusion matrix of anomaly detection to evaluate the overall classification performance of the anomaly detection model, which measures the ability of the model to distinguish between defective images and normal images. P-AUROC focuses on the ability of the model to identify anomalies at the pixel level. The calculation method is to use the anomaly heatmap output by the model, which is usually a two-dimensional array with the same size as the input image, where each value represents the likelihood of the pixel being abnormal. By setting different thresholds, the heat Figure Two values are quantized to obtain the predicted mask of abnormal pixels, which is then compared with the real abnormal mask to calculate TPR and FPR, and draw the ROC curve, and finally calculate the area under the curve. That is, it reflects the accuracy of pixel-level anomaly detection. The value is between 0 and 1, the closer to 1, the better the performance of the model in anomaly localization. From Table 4 and Table 5, the anomaly detection and localization performance of the model of the present application under the two task settings is significantly improved.

[0204] From the qualitative anomaly localization result examples of Figure 5 , the present application proves the superiority of the present application in different sizes of anomaly localization.

[0205] [1] Dosovitskiy A. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020.

[0206] [2] Ramesh A, Pavlov M, Goh G, et al. Zero-shot text-to-image generation[C] / / International conference on machine learning. Pmlr, 2021: 8821-8831.

[0207] [3] Bao H, Dong L, Piao S, et al. BEiT: BERT Pre-Training of Image Transformers[C] / / International Conference on Learning Representations.

[0208] [4] Beyer L, Izmailov P, Kolesnikov A, et al. Flexivit: One model for all patch sizes[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 14496-14506.

[0209] [5] Yao X, Zhang C, Li R, et al. One-for-all: Proposal masked cross-class anomaly detection[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(4): 4792-4800.

[0210] [6] Loshchilov I, Hutter F. Decoupled weight decay regularization[J]. arXiv preprint arXiv:1711.05101, 2017.

[0211] [7] Zhou Y, Xu X, Song J, et al. Msflow: Multiscale flow-based framework for unsupervised anomaly detection [J]. IEEE Transactions on Neural Networks and Learning Systems, 2024.

[0212] [8] Bergmann P, Fauser M, Sattlegger D, et al. MVTec AD--A comprehensive real-world dataset for unsupervised anomaly detection [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 9592-9600.

[0213] [9] Zhang Z, Zhao Z, Zhang X, et al. Industrial anomaly detection with domain shift: A real-world dataset and masked multi-scale reconstruction [J]. Computers in Industry, 2023, 151: 103990.

Claims

1. An unsupervised image anomaly detection method based on multi-scale mask causality, characterized by: Please follow the steps below to implement it: Step 1: Construct an unsupervised image anomaly detection network based on multi-scale mask causality; Step 2: Design the loss function; Step 3: Use the loss function constructed in step 2 to train the network model constructed in step 1; Step 4: Use the test data set to test the network trained in step 3. When a picture to be detected is input, the network will output a defect detection result picture with defects marked and determine whether there is any abnormality in the input picture.

2. The unsupervised image anomaly detection method based on multi-scale mask causality according to claim 1, characterized in that: The step 1 is specifically implemented according to the following steps: Step 1.1, construct a multi-scale mask image block input; Step 1.2: Construct a multi-scale mask reconstruction module; Step 1.3: Construct a semantic causal consistency constraint module; Step 1.4: Construct a multi-scale causal consistency constraint module.

3. The unsupervised image anomaly detection method based on multi-scale mask causality according to claim 2, characterized in that: The step 1.1 is specifically implemented according to the following steps: image Use two partitioning methods p1×o1 and o2×o2, o1=16, p2=32, to divide the image I into image blocks of number n1 and n2 and i∈{1,2,3,…,n1},j∈{1,2,3,…,n2},the original resolution of the image is (h,w), the number of channels is c, and the total number of image blocks obtained by segmentation is n i =hw / p a 2 , the resolution of the segmented image is (p a ,p a ), where a∈{1,2}, select n masked_i =[n a ×η] image blocks are masked to obtain a representative mask input, where η is the masking rate, a∈{1,2}, and n masked The pixels of the image blocks are replaced by zero vectors, and the mask inputs of 16×16 partitioning are obtained respectively. and 32×32 partitioning mask input 4. The unsupervised image anomaly detection method based on multi-scale mask causality according to claim 3, characterized in that: The step 1.2 is specifically implemented according to the following steps: Input the 16×16 divided mask obtained in step 1.1 and the unmasked image patches Input into the ViT-b-16 pre-trained model for encoding and decoding, where i1∈{1,2,3,…,n masked_1 }, i2∈{1,2,3,…,n1-n masked_1 }, 32×32 divided mask input and the unmasked image patches Input into the ViT-b-32 pre-trained model for encoding and decoding, where j1∈{1,2,3,…,n masked_2 }, j2∈{1,2,3,…,n2-n masked_2 }, and finally get the reconstruction output of two partitions: and Among them, i∈{1,2,3,…,n1}, j∈{1,2,3,…,n2}, Y i =D 16×16 (E 16×16 (X i )) (3) Y j =D 32×32 (E 32×32 (X j )) (4) X i 、X j It represents the combination of the masked image blocks of 16×16 partition and 32×32 partition and the unmasked image blocks as the encoder E 16×16 (·) and E 32×32 (·) input, and then E 16×16 (·) and E 32×32 (·) Output-input decoder D 16×16 (·) and D 32×32 (·) to obtain the reconstructed output, where E 16×16 (·) and E 32×32 (·) represents the encoder of ViT-b-16 and ViT-b-32 pre-training models, respectively. 16×16 (·) and D 32×32 (·) represents the decoder of ViT-b-16 and ViT-b-32 pre-trained models respectively; Introducing the weight transformation matrix proposed by FlexiViT The weight matrix ω of the ViT-b-16 pre-training is transformed and loaded into the ViT-b-32 model to obtain the weight matrix of the input ViT-b-32 model where p * and p represent the size of the transformed image patch and the original size of the image patch during pre-training, respectively. This method is used to transfer the knowledge learned by the ViT-b-16 pre-trained model on 16×16 size image patches to the ViT-b-32 model: is the transformation matrix after the initial weight matrix ω is processed by the bilinear interpolation function F(·), (·) + and(·) T Represent the pseudo-inverse and transpose operations of the matrix, It is the optimal transformation matrix that transforms the initial weight matrix ω of ViT-b-16 to adapt it to the ViT-b-32 model parameters.

5. The unsupervised image anomaly detection method based on multi-scale mask causality according to claim 4, characterized in that: The step 1.3 is specifically implemented according to the following steps: Considering the semantic causal consistency of the masked image blocks before and after reconstruction, the pixel reconstruction task is transformed into a robust semantic reconstruction task. The semantic transformation T is used to assign word vectors to the feature embeddings reconstructed by the ViT-b-16 and ViT-b-32 pre-trained models. The semantic transformation T is a network structure consisting of an encoder and a hierarchical vector quantization layer. The feature embeddings obtained for each image block are encoded as a discrete visual semantic tag. First, each image is labeled as Large and small grids or Visual semantic labeling of large and small grids V i and V j : is the masked visual semantic tag under the 16×16 partition mode, is the unmasked visual semantic tag under the 16×16 partition mode, is the masked visual semantic tag under the 32×32 partition mode, is the unmasked visual semantic label in the 32×32 partition mode, i1∈{1,2,3,…,n masked_1 }, i2∈{1,2,3,…,n1-n masked_1 }, j1∈{1,2,3,…,n masked_2 },j2∈{1,2,3,…,n2-n masked_2 }, and at the same time divide all the original image blocks into different and It is also fed into the semantic transformation T for discretization into visual semantic tags and where i∈{1,2,3,…,n1} and j∈{1,2,3,…,n2}; is the original visual semantic label of the masked image block in the 16×16 partition mode, is the original visual semantic label of the unmasked image block in the 16×16 partition mode, is the original visual semantic label of the masked image block in the 32×32 partition mode, is the original visual semantic label of the unmasked image block in the 32×32 partition mode; i1∈{1,2,3,…,n masked_1 }, i2∈{1,2,3,…,n1-n masked_1 }, j1∈{1,2,3,…,n masked_2 }, (j2∈{1,2,3,…,n2-n masked_2 }, using the encoded visual tags by computing the semantic consistency loss before and after reconstruction Measuring Visual Semantic Labeling for Mask Patch Reconstruction Compared with the actual original visual semantic label The differences: The representative vocabulary set is obtained by clustering a large-scale unlabeled image dataset into 8192 class centers. represents the number of words in the vocabulary set, and Represents the loss values ​​calculated by the two partitioning methods, by minimizing Ensure that the masked image blocks are visually semantically labeled after reconstruction Visual semantic labeling with original image patches Close, so as to ensure the semantic consistency before and after, where n masked_1 and n masked_2 is the number of mask patches under the two partitions, and Indicates the mth under two partitions α The probability that (α∈{1,2}) mask patches belong to the kth visual semantic marker, where m1∈{1,2,3,…,n masked_1 }, m2∈{1,2,3,…,n masked_2 }, k∈{1,2,3,…,8192}; Constrain the overall semantic distribution of all reconstructed image blocks, and also use the encoded visual semantic tags to calculate the semantic distribution loss To measure the distribution difference between the overall visual semantic label of the image block and the actual original visual semantic label: Where n1 and n2 represent the number of image blocks divided into 16×16 and 32×32, respectively. and Represents the loss values ​​calculated by the two partitioning methods, by minimizing Ensure that the overall semantics of the image block after reconstruction is close to that of the original image block, thereby ensuring the consistency of the semantic distribution before and after. Represents the mth graph The true value of the image block at the kth visual mark, that is, the true value of the original image block at the kth visual mark, It is the output value of the reconstructed m-th image block on the k-th visual mark.

6. The unsupervised image anomaly detection method based on multi-scale mask causality according to claim 5, characterized in that: The step 1.4 is specifically implemented according to the following steps: By constructing a multi-scale causal consistency constraint module, the scale causal consistency at two scales is constrained. Specifically, the feature embedding Y of the decoder output of the two partitions obtained in step 1.2 is first i and Y j Before performing multi-scale causal consistency constraints, due to the two types of divided input image blocks and The sizes are different, i∈{1,2,3,…,n1}, j∈{1,2,3,…,n2}, so the final feature embedding sizes are also inconsistent. The output dimensions of the output feature embeddings of the two partitions are and It is understood that a single image block on the ViT-b-32 partition corresponds to four image blocks on the ViT-b-16 and is spliced ​​at the corresponding positions. Therefore, the corresponding position is found according to the position index, and then the image block is compared with the Y j k The corresponding Y i l First, the average pooling operation is used to extract the average information of the corresponding regional features. Here, k∈{1,2,3,…,196} and l∈{1,2,3,…,49} represent the position index of the feature embedding of the two partitions. Then, the feature embedding of the two branches is flattened to obtain the flattened features F1 and F2 of the two partitions, with a dimension of (1,49*768). The flattened features F1 and F2 are input into the same linear mapping layer for linear transformation, thereby unifying the feature distribution to obtain the features f(F1) and f(F2) in the same feature space. The scale causal consistency loss is used Measuring the difference of multi-scale features: Where f(·) represents the same linear mapping layer, which is used to map features from different spaces into the same feature space for easy comparison, ‖·‖ represents the bi-norm of feature embedding, The constraint model divides the causal consistency of features at different scales into two parts to avoid destroying the original causal relationship of features due to scale changes.

7. The unsupervised image anomaly detection method based on multi-scale mask causality according to claim 6, characterized in that: The step 2 is specifically implemented according to the following steps: Total loss of unsupervised image anomaly detection network It consists of three parts: semantic consistency loss that constrains the semantic consistency of a single image block before and after reconstruction and the distribution consistency loss before and after constraining the overall image block reconstruction and multi-scale causal consistency loss that measures the causal consistency of features at different scales Therefore, the total loss Expressed as:

8. The unsupervised image anomaly detection method based on multi-scale mask causality according to claim 7, characterized in that: The step 3 is specifically implemented according to the following steps: The network constructed in step 2 is trained using the loss function constructed in step 3 to train the network and update the network weight parameters. The network uses AdamW as the optimizer, the learning rate is set to 0.00001, and the batch size is set to 4, with 500 epochs.

Images