Zero-shot industrial anomaly detection method based on knowledge distillation

Abstract

The present invention discloses a zero-sample industrial anomaly detection method based on knowledge distillation, which belongs to the field of image detection technology. The method includes a training phase and a testing phase; in the training phase, normal industrial images and abnormal industrial images from any category of industrial products are selected; a knowledge distillation network architecture is constructed, including a teacher model, a student model, an image encoder, and a text encoder; the teacher model is trained using training set data, and the knowledge in the trained teacher model is transferred to the student model; in the testing phase, an industrial anomaly detection model is constructed, and the images to be detected in the test set are input into the industrial anomaly detection model to obtain the detection results. The present invention can achieve efficient zero-sample anomaly detection in industrial detection tasks with unknown categories and diverse abnormal forms, providing an efficient, highly generalizable, and robust solution for industrial visual detection.

Description

Technical Field

[0001] The present invention relates to an image anomaly detection method, specifically a zero-sample industrial anomaly detection method based on knowledge distillation, which belongs to the technical field of image detection. Background Art

[0002] Zero-shot image anomaly detection (ZSAD) aims to accurately identify anomalous regions in unknown industrial products without relying on any predefined training data. With increasing demands for data privacy, the increasing cost of obtaining labeled data, and the ever-expanding variety of industrial products, the application value of ZSAD in industrial visual inspection has become increasingly prominent and a hot topic of current research. ZSAD methods must possess strong generalization capabilities to cope with the significant variations in product appearance and the diversity of anomalies across diverse industrial inspection tasks. Therefore, achieving efficient anomaly detection on unknown industrial products has become a core challenge in current research.

[0003] Currently, anomaly detection methods based on large text-image models primarily rely on constructing two types of textual cues, either normal or abnormal, and identifying abnormal regions by calculating the cosine similarity between the image and text in a joint space. Some existing methods employ multi-scale windows or image segmentation strategies to extract dense visual features and then combine them with textual cues to match normal and abnormal regions. However, these methods still have limitations in practical industrial applications.

[0004] First, many existing methods assume that the category of the product to be detected is known and construct corresponding text descriptions based on this category, such as "pictures of normal metal parts" or "undamaged glass sheets." However, in real-world industrial scenarios, especially when it comes to data privacy protection, new product testing, or changing production environments, the category of the detection object is often unknown, making this approach that relies on prior knowledge difficult to generalize. In addition, the vocabulary choice of text prompts can significantly affect the model's detection capabilities. For example, in experiments, simply replacing "screw" with "bolt" or "fastener" can cause significant fluctuations in the average precision metric, affecting the stability and accuracy of anomaly detection.

[0005] Secondly, existing methods often independently map images and text during feature modeling, lacking deep cross-modal interaction and multi-scale feature extraction. This leads to over-reliance on specific textual cues, resulting in insufficient generalization when dealing with diverse scales, complex backgrounds, or diverse anomaly categories. In industrial inspection tasks, anomalies can manifest as subtle cracks, surface defects, deformations, and other forms, and the global matching strategies of existing methods struggle to accurately capture these detailed features.

[0006] In summary, the current anomaly detection methods based on large text-image models still have the following problems:

[0007] (1) It relies on predefined text prompts and is difficult to adapt to unknown categories of industrial products;

[0008] (2) The lack of cross-modal interaction and deep feature modeling leads to insufficient ability to distinguish abnormal areas;

[0009] (3) It has poor adaptability to multi-scale abnormal targets and is difficult to stably detect abnormal areas of different scales in actual industrial scenarios. Summary of the Invention

[0010] Purpose of the invention: In response to the above problems, the purpose of the present invention is to provide a zero-shot industrial anomaly detection method based on knowledge distillation, select the DINO self-supervised model as the teacher model, make full use of its high-quality local feature extraction capability, and combine it with the visual-language alignment capability of the text image large model to achieve accurate detection of abnormal areas of industrial parts.

[0011] Technical solution: The zero-sample industrial anomaly detection method based on knowledge distillation of the present invention includes a training phase and a testing phase;

[0012] The training phase consists of the following steps:

[0013] Select normal and abnormal industrial images from any category of industrial products, perform preprocessing, and construct a training set;

[0014] Build a knowledge distillation network architecture, including a teacher model, a student model, an image encoder, and a text encoder. The teacher model uses the pre-trained DINO model, and the student model includes a lightweight feature matching module and a text attention module.

[0015] Use the training set data to train the teacher model and transfer the knowledge in the trained teacher model to the student model;

[0016] The testing phase includes the following steps:

[0017] Build an industrial anomaly detection model, input the images to be detected in the test set into the industrial anomaly detection model, and obtain the detection results; the industrial anomaly detection model includes an image encoder, a student model, and a text encoder.

[0018] Furthermore, the teacher model is trained using the training set data, and the steps of transferring the knowledge in the trained teacher model to the student model include:

[0019] Input the image to be detected in the training set into the image encoder to obtain four layers of image features to be detected ,in is the first layer image feature, is the second layer image feature, is the third layer image feature, is the fourth layer of image features;

[0020] Process the four layers of image features to be detected to obtain the corresponding correlation matrix , the formula is:

[0021] ,

[0022] Where, for The transpose of a matrix;

[0023] The correlation matrix Input to the lightweight feature matching module for feature conversion to obtain the feature map , the formula is:

[0024] ,

[0025] Where, Represents two layers Normalized convolutional layer composition;

[0026] Input the same image to be detected into the pre-trained DINO model to obtain four layers of image features to be detected ,in is the first layer image feature, is the second layer image feature, is the third layer image feature, is the fourth layer of image features;

[0027] Calculate the correlation matrix after the dot product of the four layers of features output by the DINO model , the formula is:

[0028] ,

[0029] Where, for The transpose of a matrix;

[0030] The binary cross entropy loss BCE is used to calculate the difference between the correlation matrix after the image encoder output transformation and the correlation matrix after the DINO model output transformation. The formula is:

[0031] ,

[0032] Where, represents the binary cross entropy loss;

[0033] use Train the lightweight feature matching module and select the model with the smallest loss as the final lightweight feature matching module;

[0034] The fourth layer image features After processing, it is input into the multi-scale feature mapping module of three different parallel branches to obtain the feature map , so that the feature map The dimension is the same as that of the text encoder;

[0035] The normal state and abnormal state of the image to be detected are constructed into a fixed format text prompt template, and the feature map Align to text embedding space;

[0036] Generate a coarse anomaly heatmap of the image to be detected.

[0037] Furthermore, the fourth layer image features After processing, it is input into the multi-scale feature mapping module of three different parallel branches to obtain the feature map The steps include:

[0038] The fourth layer image features Remove the category markers representing global features, retain only the patch-level markers representing local features, and then resize to a preset size to obtain the feature map ;

[0039] The feature map Input into 3 different parallel branches respectively to obtain three different feature maps 、 and , the formulas are:

[0040] ,

[0041] ,

[0042] ,

[0043] Where, Indication Standard convolution, Indication Standard convolution; Indicates expansion Convolution, R represents the dimension space of the tensor;

[0044] Then concatenate the original features in the channel dimension Combined with the multi-scale features extracted by each branch, the feature map is obtained , the formula is:

[0045] ,

[0046] Where, represents the splicing operation on the channel dimension,

[0047] Use a 1×1 convolution Feature Map Perform dimensionality reduction to obtain feature maps , the formula is:

[0048] ,

[0049] Finally, a residual connection is applied to the feature map With feature map Add together to get the feature map , the formula is:

[0050] .

[0051] Furthermore, the feature map The steps for the same dimensions as the text encoder include:

[0052] The feature map Re-flatten to one dimension to obtain the feature map , the formula is:

[0053] ,

[0054] Where, Represents a flattening operation;

[0055] Then pass it through a 1×1 convolution layer as a linear projection layer Perform feature transformation to obtain feature map , the formula is:

[0056] .

[0057] Furthermore, the normal state and abnormal state of the image to be detected are constructed into a fixed format text prompt template, and the feature map The steps to align to the text embedding space include:

[0058] The text prompt template H is constructed into a fixed format and is expressed as:

[0059] ,

[0060] Where, Is the fixed text part, Indicates the state of an object, including good and damaged, corresponding to normal and abnormal; represents r learnable category labels;

[0061] Generate normal With exception text , respectively expressed as:

[0062] ,

[0063] ,

[0064] Map normal text and abnormal text to the embedding space and obtain feature maps respectively , where C represents the embedding dimension of the text encoder, Indicates the length of the text;

[0065] The feature map Use global average pooling to reduce dimensionality to extract global features and obtain feature maps , the formula is:

[0066] ,

[0067] Where, represents the global average pooling operation;

[0068] Will Expressed as , and then update the category label by adding the residuals , expressed as:

[0069] ,

[0070] Where, Represents a randomly generated marker during initialization;

[0071] The updated category tags are re-injected into the normal text and abnormal text to form visually guided text, which are represented as follows:

[0072] ,

[0073] ,

[0074] Where, Indicates the normal text after the visual guide, Indicates the abnormal text after the visual guide;

[0075] Will and Mapped to the embedding space through the text encoder, the final normal text features are obtained and abnormal text features , respectively expressed as:

[0076] ,

[0077] ,

[0078] Where, represents a linear transformation operation, Represents a normalization operation.

[0079] Furthermore, the step of generating a rough abnormal heat map of the image to be detected includes:

[0080] For the four layers of image features to be detected , remove the category mark and keep only the patch mark to get the feature map , expressed as:

[0081] ,

[0082] Where, Indicates the removal of the category label of the image feature to be detected;

[0083] Then the four-layer feature map Perform linear projection to obtain the projection feature map , expressed as:

[0084] ,

[0085] For each layer Projected feature map of , calculate each patch and normal text features and abnormal text features The cosine similarity between and , expressed as:

[0086] ,

[0087] ,

[0088] For each layer of image features, calculate the difference between normal and abnormal matching scores to obtain the abnormal score matrix of each layer , expressed as:

[0089] ,

[0090] Next, generate multiple layers of heat maps and merge them:

[0091] First, the anomaly score matrix of each layer Reshape into The two-dimensional matrix is expressed as:

[0092] ,

[0093] Where, Represents a matrix resize operation;

[0094] Then the four-layer anomaly score matrix is mean-fused and expressed as:

[0095] ,

[0096] Finally, in order to map the patch-level anomaly score to the original image space, bilinear interpolation is required, which is expressed as:

[0097] ,

[0098] Where, represents the upsampling operation, Indicates the original size of the image. Specify bilinear interpolation as the upsampling method;

[0099] Normalize the heat map and limit the value of the heat map to In the range, it is expressed as:

[0100] ,

[0101] In the formula, min means taking the minimum value of the matrix, and max means taking the maximum value of the matrix;

[0102] Setting the threshold T 0 to generate anomaly segmentation mask:

[0103] ,

[0104] Where, Indicates location The abnormal segmentation mask at , represents the anomaly score of the normalized anomaly heat map at the (x,y) position, T 0 is the threshold for anomaly detection;

[0105] Superimpose the anomaly segmentation mask on the original image to obtain the final anomaly detection result:

[0106] ,

[0107] Where, Represents the original input image, symbol Indicates element-by-element multiplication, which is used to highlight abnormal areas;

[0108] Finally, calculate the prediction loss and select the real mask As a supervisory signal for the multi-scale feature mapping module, it represents the true abnormality of each pixel in the image. E Indicates the height of the image,W Indicates the width of the image, Indicates abnormal area, Indicates normal area;

[0109] For the Lth image in a batch, the total loss of anomaly mask is:

[0110] ,

[0111] Where, represents Focal Loss, express ;

[0112] Summing up the losses of all images in a batch, the final total loss function is:

[0113] ,

[0114] In the formula, batchsize represents the number of samples used for model training or inference in one iteration;

[0115] Using loss function Train the multi-scale feature mapping module and select the model with the smallest loss as the final multi-scale feature mapping module.

[0116] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0117] 1. This paper focuses on industrial anomaly detection tasks and proposes a zero-shot detection method based on knowledge distillation to improve the model's anomaly detection capabilities on unknown industrial product categories. This method relies on normal and abnormal images during training, but does not require a dataset specific to the detection target. Instead, it utilizes other datasets, such as the VisA dataset for training and the MVTec dataset for testing. This allows the model to have stronger generalization capabilities and adapt to the anomaly detection needs of different industrial products.

[0118] 2. This method fully utilizes the high-quality local features generated by the DINO model and optimizes lightweight modules through knowledge distillation, giving the model stronger anomaly recognition capabilities. This eliminates the need to load the complete DINO pre-trained model during the inference phase, reducing computational costs while improving detection efficiency.

[0119] 3. This paper introduces a multi-scale feature fusion module (RMF) to enhance the model's adaptability to anomalies at different scales, and uses a visually guided text attention module (VTAM) to perform fine-grained text embedding updates, making the representation of anomaly areas more accurate.

[0120] 4. This invention adopts a strategy that combines coarse and fine anomaly detection. First, the similarity between the image and text is calculated using features obtained from the image encoder to generate a coarse anomaly detection result. Then, a lightweight feature matching module and VTAM are used to further optimize the abnormal region perception capability. Finally, the multi-level detection results are integrated to improve the robustness and generalization ability of anomaly detection.

[0121] 5. Since the training data of this method is not limited to specific industrial products, but the adaptability of the model is improved through cross-dataset training, the present invention can achieve efficient and accurate anomaly detection in industrial backgrounds with unknown categories and diverse abnormal forms, significantly reducing dependence on abnormal samples of target products, and meeting the dual requirements of real-time and accuracy of industrial detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Figure 1 This is a framework diagram of the method of the present invention in the training phase;

[0123] Figure 2 This is a flowchart of the method of the present invention in the training phase;

[0124] Figure 3 Schematic diagram of the multi-scale feature mapping module;

[0125] Figure 4 Schematic diagram of the text attention module;

[0126] Figure 5 This is a framework diagram of the method of the present invention in the testing phase;

[0127] Figure 6 This is a flowchart of the method of the present invention in the testing phase. DETAILED DESCRIPTION

[0128] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0129] The zero-shot industrial anomaly detection method based on knowledge distillation described in this embodiment includes a training phase and a testing phase;

[0130] The training phase consists of the following steps:

[0131] Select normal and abnormal industrial images from any category of industrial products, perform preprocessing, and construct a training set;

[0132] Build a knowledge distillation network architecture, including a teacher model, a student model, an image encoder, and a text encoder. The teacher model uses the pre-trained DINO model, and the student model includes a lightweight feature matching module and a text attention module.

[0133] Use the training set data to train the teacher model and transfer the knowledge in the trained teacher model to the student model;

[0134] The testing phase includes the following steps:

[0135] Build an industrial anomaly detection model, input the images to be detected in the test set into the industrial anomaly detection model, and obtain the detection results; the industrial anomaly detection model includes an image encoder, a student model, and a text encoder.

[0136] Combine Figures 1 to 2 As shown, during the training phase, normal industrial images and abnormal industrial images from any category of industrial products are selected, and after preprocessing, they are constructed into a training set.

[0137] In this example, the dataset used is the publicly available MVTec AD color image set, which covers 15 categories of industrial products and is divided into training and test sets, both containing normal and abnormal images. Zero-shot evaluation is performed on the VISA dataset.

[0138] You can select 16 training images in the training set at a time and change the size of each image to Pixels, 518 for length and height, 3 for the number of channels, this size is the input requirement of the image encoder, and the normalization parameter is the mean and standard deviation , and random rotation , horizontal flipping and color jittering such as brightness ±0.2, contrast ±0.2, and saturation ±0.2 are performed to enhance data diversity, and finally the preprocessed input image is obtained.

[0139] Furthermore, the teacher model is trained using the training set data, and the steps of transferring the knowledge in the trained teacher model to the student model include:

[0140] The image to be detected in the training set is input into the image encoder. The image encoder adopts the transformer structure to obtain four layers of image features to be detected. ,in is the first layer image feature, is the second layer image feature, is the third layer image feature, is the fourth layer of image features;

[0141] Each layer of image features to be detected consists of category labels of global information and patch-level labels of local information, and the feature size is , where 1370 represents the number of patches, 16 is the number of input images, and 1024 represents the embedding dimension of each patch feature;

[0142] Process the four layers of image features to be detected, first remove the Global information, only retain Local information; then, the features are reshaped into Finally, the corresponding correlation matrix is calculated for each feature , the formula is:

[0143] ,

[0144] Where, for The transpose of a matrix;

[0145] The correlation matrix Input to the lightweight feature matching module for feature conversion to obtain the feature map , to learn to map the features extracted by the image encoder to the DINO model style feature space, the formula is:

[0146] ,

[0147] Where, Represents two layers Normalized convolutional layer composition, including Activate, keeping the number of channels unchanged to enhance the expression of local features while matching the output style of the DINO model;

[0148] The features Reshape into And add the category label of global information so that its final output is the correlation matrix after the dot product of the four-layer features of DINO Stay consistent;

[0149] Input the same image to be detected into the pre-trained DINO model to obtain four layers of image features to be detected ,in is the first layer image feature, is the second layer image feature, is the third layer image feature, is the fourth layer of image features;

[0150] Calculate the correlation matrix after the dot product of the four layers of features output by the DINO model , the formula is:

[0151] ,

[0152] Where, for The transpose of a matrix;

[0153] The binary cross entropy loss BCE is used to calculate the difference between the correlation matrix after the image encoder output transformation and the correlation matrix after the DINO model output transformation. The formula is:

[0154] ,

[0155] Where, represents the binary cross entropy loss;

[0156] use Train the lightweight feature matching module and select the model with the smallest loss as the final lightweight feature matching module;

[0157] The fourth layer image features , whose shape is After processing, it is input into the multi-scale feature mapping module of three different parallel branches to obtain the feature map , so that the feature map The dimension is the same as that of the text encoder;

[0158] The normal state and abnormal state of the image to be detected are constructed into a fixed format text prompt template, and the feature map Align to text embedding space;

[0159] Generate a coarse anomaly heatmap of the image to be detected.

[0160] In this example, an image encoder uses the cross-modal model CLIP trained using contrastive learning and serves as the base model for the zero-shot anomaly segmentation task. During the training phase of this method, a publicly available pre-trained image encoder is used, all its parameters are frozen, and feature extraction is performed on the image to be detected.

[0161] The DINO model is also a pre-trained model that achieves efficient image representation learning through self-distillation and contrastive learning. DINO is trained on unlabeled data, enabling it to capture subtle differences in images and generate high-quality local features. During the training phase of this method, the pre-trained DINO model freezes all parameters and is used to extract high-quality features from industrial images. These features are then distilled into a lightweight feature matching module to enhance the lightweight module's ability to learn the DINO model. Notably, the DINO model is no longer used during testing, and its functionality is replaced by the lightweight feature matching module trained during training.

[0162] The lightweight feature matching module consists of two 3 The proposed method consists of three convolutional blocks, which are used to learn to represent the image features output by the image encoder into high-quality patch-level features close to the output of the DINO model. It replaces the DINO model in the inference stage, reducing the hardware and time costs of inference.

[0163] Combine Figure 3 As shown, further, the fourth layer image features After processing, it is input into the multi-scale feature mapping module of three different parallel branches to obtain the feature map The steps include:

[0164] The fourth layer image features Remove the category markers representing global features, retain only the patch-level markers representing local features, and then resize to a preset size , get the feature map ;

[0165] The feature map Input into 3 different parallel branches respectively to obtain three different feature maps 、 and , the formulas are:

[0166] ,

[0167] ,

[0168] ,

[0169] Where, Indication Standard convolution, Indication Standard convolution; Indicates expansion Convolution, R represents the dimension space of the tensor;

[0170] Then concatenate the original features in the channel dimension Combined with the multi-scale features extracted by each branch, the feature map is obtained , the formula is:

[0171] ,

[0172] Where, represents the splicing operation on the channel dimension,

[0173] Use a 1×1 convolution Feature Map Perform dimensionality reduction to obtain feature maps , the formula is:

[0174] ,

[0175] Finally, a residual connection is applied to the feature map With feature map Add together to get the feature map , the formula is:

[0176] .

[0177] Furthermore, the feature map The steps for the same dimensions as the text encoder include:

[0178] In order to embed image features into the semantic space of the text encoder, the feature map Re-flatten to one dimension to obtain the feature map , the formula is:

[0179] ,

[0180] Where, Represents a flattening operation;

[0181] Then pass it through a 1×1 convolution layer as a linear projection layer Perform feature transformation to obtain feature map , the formula is:

[0182] ,

[0183] Among them, 768 is the output dimension of the text encoder, which allows image features to be seamlessly aligned to the text embedding space, facilitating subsequent multimodal information fusion.

[0184] Furthermore, the normal state and abnormal state of the image to be detected are constructed into a fixed format text prompt template, and the feature map The steps to align to the text embedding space include:

[0185] The text prompt template H is constructed into a fixed format and is expressed as:

[0186] ,

[0187] Where, Is the fixed text part, Indicates the state of an object, including good and damaged, corresponding to normal and abnormal; represents r learnable category labels, used to learn information about specific product categories, and is fixed to 4;

[0188] Generate normal With exception text , respectively expressed as:

[0189] ,

[0190] ,

[0191] Map normal text and abnormal text to the embedding space and obtain feature maps respectively , where C represents the embedding dimension of the text encoder, Represents text length, including fixed templates, state words, and learnable category tags;

[0192] The feature map Use global average pooling to reduce dimensionality to extract global features and obtain feature maps , the formula is:

[0193] ,

[0194] Where, represents the global average pooling operation;

[0195] Will Expressed as , and then update the category label by adding the residuals , expressed as:

[0196] ,

[0197] Where, Represents a randomly generated marker during initialization;

[0198] The updated category tags are re-injected into the normal text and abnormal text to form visually guided text, which are represented as follows:

[0199] ,

[0200] ,

[0201] Where, Indicates the normal text after the visual guide, Indicates the abnormal text after the visual guide;

[0202] Will and Mapped to the embedding space through the text encoder, the final normal text features are obtained and abnormal text features , respectively expressed as:

[0203] ,

[0204] ,

[0205] Where, represents a linear transformation operation, Represents a normalization operation.

[0206] Furthermore, the step of generating a rough abnormal heat map of the image to be detected includes:

[0207] For the four layers of image features to be detected , remove the category mark and keep only the patch mark to get the feature map , expressed as:

[0208] ,

[0209] Where, Indicates the removal of the category label of the image feature to be detected;

[0210] Then the four-layer feature map Perform linear projection to obtain the projection feature map , expressed as:

[0211] ,

[0212] For each layer Projected feature map of , calculate each patch and normal text features and abnormal text features The cosine similarity between and , expressed as:

[0213] ,

[0214] ,

[0215] For each layer of image features, calculate the difference between normal and abnormal matching scores to obtain the abnormal score matrix of each layer , expressed as:

[0216] ,

[0217] when When it is smaller, it means that the patch is closer to the abnormal feature, indicating that the area may be an abnormal area, otherwise it may be a normal area;

[0218] Next, generate multiple layers of heat maps and merge them:

[0219] First, the anomaly score matrix of each layer Reshape into The two-dimensional matrix is expressed as:

[0220] ,

[0221] Where, Represents a matrix resize operation;

[0222] Then the four-layer anomaly score matrix is mean-fused and expressed as:

[0223] ,

[0224] Finally, in order to map the patch-level anomaly score to the original image space, bilinear interpolation is required, which is expressed as:

[0225] ,

[0226] Where, represents the upsampling operation, Indicates the original size of the image. Specify bilinear interpolation as the upsampling method;

[0227] Normalize the heat map and limit the value of the heat map to In the range, it is expressed as:

[0228] ,

[0229] In the formula, min means taking the minimum value of the matrix, and max means taking the maximum value of the matrix;

[0230] Setting the threshold T 0 to generate anomaly segmentation mask:

[0231] ,

[0232] Where, Indicates location The abnormal segmentation mask at , Represents the anomaly score of the normalized anomaly heat map at the (x,y) position, which is used to measure the possibility that the (x,y) position belongs to the abnormal area. T 0 is the threshold for anomaly detection;

[0233] Superimpose the anomaly segmentation mask on the original image to obtain the final anomaly detection result:

[0234] ,

[0235] Where, Represents the original input image, symbol Indicates element-by-element multiplication, which is used to highlight abnormal areas;

[0236] Finally, calculate the prediction loss and select the real mask As a supervisory signal for the multi-scale feature mapping module, it represents the true abnormality of each pixel in the image. E Indicates the height of the image, WIndicates the width of the image, Indicates abnormal area, Indicates normal area;

[0237] For the Lth image in a batch, the total loss of anomaly mask is:

[0238] ,

[0239] Where, represents Focal Loss, express ;

[0240] Summing up the losses of all images in a batch, the final total loss function is:

[0241] ,

[0242] In the formula, batchsize represents the number of samples used for model training or inference in one iteration;

[0243] Using loss function Train the multi-scale feature mapping module and select the model with the smallest loss as the final multi-scale feature mapping module.

[0244] Among them, the threshold for anomaly detection It can be adjusted according to the changes in the data set. The specific methods are as follows:

[0245] First, use the precision_recall_curve function provided by the scikit-learn library to calculate the prediction rate (Precision) and recall rate (Recall) under different thresholds, and then calculate the threshold , the specific formula is:

[0246] ,

[0247] The formula for Focal Loss is:

[0248] ,

[0249] Where, yes The focus parameter is set to 2.0; Expressed as the number of all pixels in the mask.

[0250] The formula is:

[0251] ,

[0252] Where, is a smoothing term, set to , to prevent the denominator from being zero.

[0253] In summary, using Train a lightweight feature matching module using The multi-scale feature mapping module is trained for 10 rounds, the batch size is set to 16, the learning rate is set to 0.0001, and the model with the smallest loss is selected as the final lightweight feature matching module and the multi-scale feature mapping module.

[0254] The testing phase includes the following steps:

[0255] Build an industrial anomaly detection model, input the images to be detected in the test set into the industrial anomaly detection model, and obtain the detection results; the industrial anomaly detection model includes an image encoder, a student model, and a text encoder.

[0256] Combine Figures 5 and 6 As shown in the figure, the specific implementation process of the testing phase is as follows:

[0257] S1, read in the image for testing and resize it to Pixels, as the input image for the testing phase .

[0258] S2, the preprocessed image is fed into the pre-trained image encoder and the student model, i.e., the lightweight feature matching module, to extract four layers of features and correlation matrix .

[0259] S3, Input to the multi-scale feature mapping module to obtain the feature that can be embedded in the text .

[0260] S4, the unified text template is sent to the text encoder, and after a series of operations, the fusion is obtained. Normal and abnormal text embedding .

[0261] S5, combined with text embedding and image features , and obtain the rough anomaly heat map .

[0262] S6, four-layer features extracted using image encoder and correlation matrix , the image fusion feature is obtained through element-by-element and operation , and the concatenated text embedding They are fed together into the VTAM visually guided text attention module.

[0263] S7, in the text attention module, combined with Figure 4 , first record the concatenated text features as Projected to the Query space through the 1D convolution layer to obtain the query features , the formula is:

[0264] ,

[0265] Then reshape the dimensions to Dimensions, where represents a 1D convolutional layer, is the number of attention heads; is the feature dimension after projection; For each head dimension, ; B represents the batch size;

[0266] Then use the 1D convolution layer to fuse the image features Projected into the Key and Value spaces respectively, the formula is:

[0267] ,

[0268] ,

[0269] Where, represents the key after projection of image features, Represents the 1D convolution weight matrix that maps image features to the Key space, Represents the bias item of the Key space mapping, Represents the value after projection by image features, Represents the 1D convolution weight matrix that maps image features to the Value space, Represents the bias item of the Value space mapping;

[0270] Reshape into a multi-headed dimension: .

[0271] Then the similarity between Query and Key is calculated by dot product, the formula is:

[0272] ,

[0273] Where, is a learnable scaling factor, set to 1 at the beginning of training; This is used to scale the dimension to prevent the dot product result from being too large.

[0274] Then use Normalize the attention weights, and the normalized attention matrix It reflects the matching degree between text embedding and image tag.

[0275] Then use the attention weight The weighted sum of Value is:

[0276] ,

[0277] Then reshape into the original feature dimension .

[0278] Then perform L2 normalization on the updated features to keep the feature length consistent. The normalized features As the final optimized text feature.

[0279] Finally, after multimodal interaction, the optimized image features Will be embedded with the optimized text Perform cosine similarity calculation to generate a similarity matrix, remove the category labels to obtain the patch-level label information similarity, and reshape it into an anomaly heat map The heatmap is upsampled to the original image size through bilinear interpolation to obtain a fine prediction result, which is then combined with the coarse prediction heatmap in a 1:1 ratio and normalized to obtain the final prediction result.

[0280] S8, finally get the predicted abnormal heat map By setting the threshold Converted into a binary mask to identify abnormal areas. Black areas are normal areas, and vice versa. If there is no white area in the entire image, then the image is normal.

[0281] This method was trained on the MvTecAD and VisA datasets and tested on each other. The pixel-level area under the curve (AUROC) and pixel-level region overlap (PRO) metrics were calculated. The model trained on the VisA dataset achieved an AUROC of 92.2 and a PRO of 88.1 when tested on the MvTecAD dataset; the model trained on the MvTecAD dataset achieved an AUROC of 96.0 and a PRO of 91.3 when tested on the VisA dataset. These results demonstrate that this method outperforms zero-shot anomaly detection methods that rely solely on textual cues.

Claims

1. A zero-shot industrial anomaly detection method based on knowledge distillation, characterized by: It includes training phase and testing phase; The training phase consists of the following steps: Select normal and abnormal industrial images from any category of industrial products, perform preprocessing, and construct a training set; Build a knowledge distillation network architecture, including a teacher model, a student model, an image encoder, and a text encoder. The teacher model uses the pre-trained DINO model, and the student model includes a lightweight feature matching module and a text attention module. Use the training set data to train the teacher model and transfer the knowledge in the trained teacher model to the student model; The testing phase includes the following steps: Build an industrial anomaly detection model and input the images to be detected in the test set into the industrial anomaly detection model to obtain the detection results. The industrial anomaly detection model includes an image encoder, a student model, and a text encoder. The steps of training the teacher model using the training set data and transferring the knowledge from the trained teacher model to the student model include: Input the image to be detected in the training set into the image encoder to obtain four layers of image features to be detected ,in is the first layer image feature, is the second layer image feature, is the third layer image feature, is the fourth layer of image features; Process the four layers of image features to be detected to obtain the corresponding correlation matrix , the formula is: ， Where, for The transpose of a matrix; The correlation matrix Input to the lightweight feature matching module for feature conversion to obtain the feature map , the formula is: ， Where, Represents two layers Normalized convolutional layer composition; Input the same image to be detected into the pre-trained DINO model to obtain four layers of image features to be detected ,in is the first layer image feature, is the second layer image feature, is the third layer image feature, is the fourth layer of image features; Calculate the correlation matrix after the dot product of the four layers of features output by the DINO model , the formula is: ， Where, for The transpose of a matrix; The binary cross entropy loss BCE is used to calculate the difference between the correlation matrix after the image encoder output transformation and the correlation matrix after the DINO model output transformation. The formula is: ， Where, represents the binary cross entropy loss; use Train the lightweight feature matching module and select the model with the smallest loss as the final lightweight feature matching module; The fourth layer image features After processing, it is input into the multi-scale feature mapping module of three different parallel branches to obtain the feature map , so that the feature map The dimension is the same as that of the text encoder; The normal state and abnormal state of the image to be detected are constructed into a fixed format text prompt template, and the feature map Align to text embedding space; Generate a rough anomaly heat map of the image to be detected; The normal state and abnormal state of the image to be detected are constructed into a fixed format text prompt template, and the feature map The steps to align to the text embedding space include: The text prompt template H is constructed into a fixed format and is expressed as: ， Where, Is the fixed text part, Indicates the state of an object, including good and damaged, corresponding to normal and abnormal; represents r learnable category labels; Generate normal With exception text , respectively expressed as: ， ， Map normal text and abnormal text to the embedding space and obtain feature maps respectively , where C represents the embedding dimension of the text encoder, Indicates the length of the text; The feature map Use global average pooling to reduce dimensionality to extract global features and obtain feature maps , the formula is: ， Where, represents the global average pooling operation; Will Expressed as , and then update the category label by adding the residuals , expressed as: ， Where, Represents a randomly generated marker during initialization; The updated category tags are re-injected into the normal text and abnormal text to form visually guided text, which are represented as follows: ， ， Where, Indicates the normal text after the visual guide, Indicates the abnormal text after the visual guide; Will and Mapped to the embedding space through the text encoder, the final normal text features are obtained and abnormal text features , respectively expressed as: ， ， Where, represents a linear transformation operation, Represents a normalization operation.

2. The zero-shot industrial anomaly detection method based on knowledge distillation according to claim 1 is characterized in that: The fourth layer image features After processing, it is input into the multi-scale feature mapping module of three different parallel branches to obtain the feature map The steps include: The fourth layer image features Remove the category markers representing global features, retain only the patch-level markers representing local features, and then resize to a preset size to obtain the feature map ; The feature map Input into 3 different parallel branches respectively to obtain three different feature maps 、 and , the formulas are: ， ， ， Where, Indication Standard convolution, Indication Standard convolution; Indicates expansion Convolution, R represents the dimension space of the tensor; Then concatenate the original features in the channel dimension Combined with the multi-scale features extracted by each branch, the feature map is obtained , the formula is: ， Where, represents the splicing operation on the channel dimension, Use a 1×1 convolution Feature Map Perform dimensionality reduction to obtain feature maps , the formula is: ， Finally, a residual connection is applied to the feature map With feature map Add together to get the feature map , the formula is: 。 3. The zero-shot industrial anomaly detection method based on knowledge distillation according to claim 2 is characterized in that: Make the feature map The steps for the same dimensions as the text encoder include: The feature map Re-flatten to one dimension to obtain the feature map , the formula is: ， Where, Represents a flattening operation; Then pass it through a 1×1 convolution layer as a linear projection layer Perform feature transformation to obtain feature map , the formula is: 。 4. The zero-shot industrial anomaly detection method based on knowledge distillation according to claim 3 is characterized in that: The steps of generating a rough anomaly heat map of the image to be detected include: For the four layers of image features to be detected , remove the category mark and keep only the patch mark to get the feature map , expressed as: ， Where, Indicates the removal of the category label of the image feature to be detected; Then the four-layer feature map Perform linear projection to obtain the projection feature map , expressed as: ， For each layer Projected feature map of , calculate each patch and normal text features and abnormal text features The cosine similarity between and , expressed as: ， ， For each layer of image features, calculate the difference between normal and abnormal matching scores to obtain the abnormal score matrix of each layer , expressed as: ， Next, generate multiple layers of heat maps and merge them: First, the anomaly score matrix of each layer Reshape into The two-dimensional matrix is expressed as: ， Where, Represents a matrix resize operation; Then the four-layer anomaly score matrix is mean-fused and expressed as: ， Finally, in order to map the patch-level anomaly score to the original image space, bilinear interpolation is required, which is expressed as: ， Where, represents the upsampling operation, Indicates the original size of the image, Specify bilinear interpolation as the upsampling method; Normalize the heat map and limit the value of the heat map to In the range, it is expressed as: ， In the formula, min means taking the minimum value of the matrix, and max means taking the maximum value of the matrix; Setting the threshold T 0 to generate anomaly segmentation mask: ， Where, Indicates location The abnormal segmentation mask at , represents the anomaly score of the normalized anomaly heat map at the (x,y) position, T 0 is the threshold for anomaly detection; Superimpose the anomaly segmentation mask on the original image to obtain the final anomaly detection result: ， Where, Represents the original input image, symbol Indicates element-by-element multiplication, which is used to highlight abnormal areas; Finally, calculate the prediction loss and select the real mask As a supervisory signal for the multi-scale feature mapping module, it represents the true abnormality of each pixel in the image. E Indicates the height of the image, W Indicates the width of the image, Indicates abnormal area, Indicates normal area; For the Lth image in a batch, the total loss of anomaly mask is: ， Where, represents Focal Loss, express ; Summing up the losses of all images in a batch, the final total loss function is: ， In the formula, batchsize represents the number of samples used for model training or inference in one iteration; Using loss function Train the multi-scale feature mapping module and select the model with the smallest loss as the final multi-scale feature mapping module.

Images