A framework for few-shot anomaly detection and classification based on reconstruction-guided cross-modal alignment

Abstract

A small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment includes the following steps: Step 1: Construct a reconstruction network, which reconstructs the dataset image into a completely normal image. The output of the reconstruction network is compared with the input to determine the anomaly information in the input image; Step 2: The cross-modal alignment network includes two branches: a visual encoder and a text encoder. The text encoder and the visual encoder are used to extract text features and visual features respectively; Step 3: The anomaly information and the visual features are subjected to attention fusion, and the visual features with enhanced anomaly details are output; Step 4: By calculating the similarity between the text features described in Step 2 and the visual features described in Step 3, it is determined whether the input image is abnormal and the pixel-level anomaly localization result is obtained. The present invention realizes universal anomaly detection, localization and classification with high precision, strong generalization ability and low resource consumption.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial anomaly detection, and in particular relates to a small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment. Background Art

[0002] In recent years, industrial anomaly detection technology has made significant progress, with detection performance continuously improving. However, in practical applications, detection, segmentation, and anomaly classification remain significant challenges due to the wide variety of defect types, complex morphologies, and limited available training and test data. Deep learning in visual anomaly detection typically involves two main tasks: determining the presence of an anomaly in an image (detection task) and determining the pixel-level location of anomalies (segmentation task). Furthermore, anomaly classification requires further identification of the specific anomaly type, which remains a technical challenge for industrial defect detection and other fields.

[0003] Most current methods focus on anomaly detection and segmentation, primarily because anomalies are often diverse and uncertain, making it difficult to capture all potential anomaly types with a limited sample set. Popular anomaly detection (AD) methods mostly follow an unsupervised learning paradigm, training solely on a large number of normal images to construct a distribution model of normal samples to detect anomalies. During the testing phase, these methods compare normal and abnormal samples, calculating the difference between input and output to obtain an anomaly score. Because these models are often based on the assumption that normal samples are nearly identical, such single models are often only applicable to image data of a single category.

[0004] However, with the growing demand for detecting diverse defect types in the industrial sector, single-category models are unable to meet the needs of multi-category anomaly detection. Therefore, it is particularly important to design general models for multi-category data that can not only handle different types of feature distributions but also maintain high performance in detecting various anomalies. Furthermore, addressing generalization issues in data-scarce scenarios and further improving the accuracy of anomaly classification and localization remain pressing technical challenges in this field.

[0005] The task of small-shot anomaly detection (FSAD) has rapidly developed in recent years. Its goal is to accurately locate and segment anomalous regions in input samples with only a small amount of predefined training data. Furthermore, the task of zero-shot anomaly detection (ZSAD) relies entirely on no pre-defined training data and requires models with strong generalization capabilities to cope with the significant variations in visual appearance, anomalous objects, and background features across diverse industrial inspection tasks. In recent years, the contrastive language-image pre-training (CLIP) model has become a key visual language foundation for solving ZSAD.

[0006] Currently, reconstruction-based methods are usually only applicable to images of a single category, which limits the generalizability of the models to other domains.

[0007] The diffusion model-based method has a long training time and high cost due to its complex denoising and denoising process.

[0008] Feature-based methods require the design of specialized memory to store features, and therefore have higher complexity and storage requirements.

[0009] When the number of target objects is large, both reconstruction-based and feature-based methods require a large number of training samples and deploy specific models for each object category, which makes these methods difficult to apply in real industrial scenarios. In addition, these methods cannot effectively perform the task when the target data is unavailable due to privacy issues or scarcity.

[0010] Improved methods based on CLIP still have not fully addressed their lack of sensitivity to texture anomalies, which makes it difficult to accurately locate subtle anomalies. As a result, the generated anomaly maps often contain large areas of highlight, demonstrating the limitations of CLIP in anomaly localization. Summary of the Invention

[0011] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment, which realizes general anomaly detection, positioning and classification with high precision, strong generalization ability and low resource consumption. This method guides cross-modal alignment by reconstructing the network, can adapt to different categories of data sets without a lot of training, and has good generalization ability. Whether facing new data distributions or unknown anomaly types, the model can respond quickly and maintain high detection accuracy. In addition, this method can maintain low resource consumption in different application scenarios and is suitable for environments with low computing power requirements.

[0012] In order to achieve the above object, the technical solution adopted by the present invention is:

[0013] A framework for small-shot anomaly detection and classification based on reconstruction-guided cross-modal alignment, including a cross-modal alignment network (CLIP), a reconstruction network, and a dual attention fusion module;

[0014] The reconstruction network takes the normal or abnormal RGB image in the dataset as input, reconstructs the image to obtain a completely normal image of the same size as the input, and calculates the difference between the image before and after reconstruction to obtain predicted abnormality information;

[0015] The cross-modal alignment network includes a visual encoder and a text encoder. The text encoder extracts text features based on text prompts, the visual encoder extracts visual features from normal or abnormal RGB images in the dataset, and the text encoder adds additional learnable parameters for fine-tuning.

[0016] The dual attention fusion module performs attention fusion on the abnormal information and the visual features, and outputs visual features with enhanced abnormal detail information;

[0017] The similarity between the text features and the visual features after the abnormal detail information enhancement is used to determine whether the input image is abnormal and obtain a pixel-level abnormality positioning result.

[0018] The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment includes the following steps:

[0019] Step 1: Use the reconstruction network to reconstruct the dataset image into a completely normal image. The reconstruction network outputs consistent results for normal images and reconstructs abnormal areas into normal content for abnormal images through neighborhood features. The output of the reconstruction network is compared with the input to determine the abnormal information in the input image.

[0020] Among them, a completely normal image is an image that does not contain abnormal information. If the reconstruction network receives a normal image, its output will be highly close to the input image. For abnormal images, the reconstruction network will use the neighborhood features around the abnormal position to reconstruct the abnormal area into content consistent with the normal image, thereby generating a normal image with no abnormalities at all.

[0021] Step 2: Using the visual encoder and text encoder branches, the text encoder extracts text features and the visual encoder extracts visual features respectively. The text encoder adds additional learnable parameters for fine-tuning.

[0022] Step 3: Perform attention fusion on the abnormal information and the visual features, and output the visual features with enhanced abnormal detail information;

[0023] Step 4: By calculating the similarity between the text features described in step 2 and the visual features after the abnormal detail information enhancement described in step 3, it is determined whether the input image is abnormal and the pixel-level abnormality positioning result is obtained.

[0024] In step 1:

[0025] For the input dataset image x i First, the features are extracted through the pre-trained backbone network efficientnet-b4, and then f x Indicates that f x ∈R C×H×W, C, H, and W respectively represent the number of channels, height, and width of the feature;

[0026] A memory-based method is used to set up the reconstruction network. The reconstruction network represents the feature from both local and global perspectives. By combining the unique features from different perspectives, it provides a representation for the input, capturing fine-grained details and a broader overall structure;

[0027] By converting the feature into a low-dimensional coordinate and mapping it to a continuous grid, the input feature can be compared with the normal features stored in the grid, and then the difference position can be predicted; The way of storing in the grid enables the reconstruction network to effectively retain normal information and eliminate the anomalies existing between the feature data.

[0028] Furthermore, specifically:

[0029] Step (1): Local feature representation:

[0030] When converting the input feature f x into a local feature represented by low-dimensional coordinates, first, without changing the number of channels, the information of different channels is superimposed through 1D convolution so that the feature mapped to the coordinate representation can take into account the information between different channels. Then, through a convolutional layer with a kernel of 1 the number of channels is changed, and the original channels are mapped to n (n < c) dimensions to achieve coordinate conversion, that is, each pixel is converted into an n-dimensional coordinate value;

[0031]

[0032] where is the representation after the input feature f x is converted into n-dimensional coordinates. Here, i, j (i ∈ h, j ∈ w) represent the position of each pixel point in the feature f x , and T1, T represent the hyperbolic tangent activation by the tanh function;

[0033] Adopting the method of grid design, the grid G is trained as a function with infinitely resolvable coordinates, and the feature corresponding to the input coordinates in the output grid G is output;

[0034] The feature representation corresponding to each n-dimensional coordinate is stored in the grid; Through the grid mapping function S(·), the feature corresponding to the n-dimensional coordinate in the grid G is extracted;

[0035] In this method, the n-dimensional coordinates represent the low-dimensional feature representation of each pixel, that is, the pixel point is mapped to n dimensions and understood as n-dimensional coordinates. The local grid G ​​stores the feature representation corresponding to each position in the grid (that is, each coordinate point). These features are automatically learned by the network during the training process, that is, each position has a vector representation, which represents the local feature of that position. In summary, the feature of a pixel point, such as c-dimension, is converted into an n-dimensional coordinate representation after coordinate transformation. When the n-dimensional coordinate is input into the grid mapping function S, the feature corresponding to the position will be found from the grid G ​​(the stored feature, such as C-dimension).

[0036] Step (2): Global feature representation:

[0037] For the input feature f x , through linear mapping, the number of channels is mapped to n, that is, an n-dimensional coordinate v representing the global feature n :

[0038]

[0039] Among them, Linear is a linear layer network, T a 、T b is the tanh activation function;

[0040] For a learnable global grid By global coordinate v n Capture input features f x The global representation of x The information of each dimension in the grid is set, so the number of grid channels C is set g =C*H*W, so that all information can be taken into account during sampling. The sampling process is as follows:

[0041]

[0042] f g =Re(f′ g )f g ∈(R C×H×W )

[0043]

[0044] After sampling, the obtained feature is represented as f′ g , the shape is Through the Re function, the global feature is represented as f′ g Reshape to R C×H×W The global feature f g ;

[0045] The global feature representation of the final output not only effectively replaces the global anomaly, but also distinguishes the inter-class distribution under the same background;

[0046] Step (3): Fusion feature representation:

[0047] By combining the global and local feature representations, the normal features of the input image Image are effectively learned. The global and local feature representations are connected through the concat(·) operation, and then the normal features are reconstructed through the continuous residual network Rec(·):

[0048] f(x)=Rec(concat(f l , f g ))

[0049] Among them, f l is the local feature of the input image, f g is the global feature of the input image.

[0050] During training, by comparing f(x) with the input feature f x The similarity of the two learnable grids is used to continuously learn, and the mean square error and cosine similarity are combined to evaluate the similarity between the two, and the final feature representation is refined so that it can accurately reconstruct normal features;

[0051] The normal feature representation is obtained by local and global grid sampling, and the reconstructed normal features are obtained through the convolutional network. The reconstructed normal features are compared with the input features to obtain the abnormal feature map pre, which is used as the abnormal feature predicted by the network:

[0052]

[0053] In step (1), when the input coordinates do not have an exact match in the grid G, the method of adjacent feature aggregation is used to interpolate based on the distance between the input feature coordinates and the coordinates of the adjacent features, and finally output the feature representation of the corresponding coordinates:

[0054] S(v, G)=|vn|G[m]+|vm|G[n]

[0055]

[0056] Where S(·) is the grid mapping function, v represents the coordinate, G represents the mapped grid, and m and n are the adjacent coordinates of v;

[0057] Setting up a learnable local grid pass In G l Find the value of the corresponding position in the grid and use it as the representative of the pixel. Finally, output the feature representation of the pixel in the grid. Right now:

[0058]

[0059] Where S(.,G) represents the sampling process from the grid G, and the number of channels of the output feature representation is the number of channels of the features stored in the grid, which means that each channel in the grid is sampled. The local feature representation of the final output is defined as f l .

[0060] In step 2, in the CLIP model, the text prompt template, the text encoder, and the visual encoder work together to achieve cross-modal learning. The text prompt template is a natural language description input to the text encoder to express the semantic content of the image. The text encoder captures the semantic information of the text by converting these text prompts into vector representations. At the same time, the image is converted into a vector representation by the visual encoder to capture the visual features of the image. Finally, CLIP optimizes the outputs of the two encoders through contrastive learning, so that images and texts with the same semantics are closer in the same embedding space, achieving effective alignment between the image and text.

[0061] Specifically:

[0062] Step (1): Design of text prompt template:

[0063] The designed text prompt template is as follows:

[0064] t n =[P1][P2]...[P E ][normal][object].

[0065] t a =[Q1][Q2]...[Q E ][c k ][object].

[0066] C=[c1,c2...c K ](c1:broken, c2:contamination...)

[0067] Among them, t n Represents a normal type text template, t a A text template representing an exception type, C represents the exception type, [P i ],[Q i ](i∈1,...,E) represents the learnable word embeddings of normal and abnormal text templates;

[0068] Step (2): The text encoder extracts text features:

[0069] Learnable parameters are added to the text encoder of CLIP, enabling the model to learn distinguishable abnormal semantic information. The learnable parameters are set and added to the front side of the text prompt embedding and input into the encoder together with the text prompt embedding. After each layer of the encoder is processed, the learnable text embedding is optimized. The details are as follows:

[0070] L m =[l1,l2,...,l M ]

[0071] [′ m+1 ,t m+1 ]=T m (cat((l m , t m ),dim=chanel))

[0072] [l′ m+2 , t m+2 ]=T m+1 (cat((l m+1 , t m+1 ),dim=chanel))

[0073] …

[0074] t M+1 =T M (t M )

[0075] Among them, L m represents the learnable parameters of the m-th layer text encoder, t m represents the learnable text embedding at the mth layer, l′ m+1 T represents the output of the m-th layer's learnable parameters after the m-th layer's text encoder. M represents the frozen M-th layer text encoder, and the cat operation aggregates the learnable parameters and text embedding along the channel dimension;

[0076] Finally, the text features t representing normal and abnormal information are obtained through the text encoder of CLIP n , t a , which is then used to calculate the similarity with CLIP visual features to obtain the loss;

[0077] Step (3): Visual encoder extracts features:

[0078] For the input dataset image x i , after the encoder, we finally get a token-like feature representing global semantic information and the output of multiple intermediate layers:

[0079] f i =F(xi ) f i ∈R H×W×D

[0080]

[0081] Where F(·) represents the visual encoder, f i represents the global image features, Represents the image features at the intermediate layer patch level, |S| represents the set of extracted intermediate layers of the visual encoder;

[0082] The step 3 is specifically as follows:

[0083] The abnormal feature map pre output by the reconstruction module is rich in fine-grained information. According to the visual encoder characteristics of the cross-modal alignment network, a dual attention mechanism is designed to integrate the image features at the |S| intermediate layer patch level They are divided into two groups: high-level patch features and low-level patch features;

[0084] Step (1): Channel Attention

[0085] For high-level patch features The different channels of the abnormal information are focused on, and the shape of the abnormal feature map pre is combined with the high-level patch features Align the shape of the abnormal feature map to simulate the high-level patch features The embedding operation passes through the convolution layer of the same size and adds the high-level patch features. The same position encoding is used to generate the embedding pre′ of the abnormal feature map pre, achieving

[0086]

[0087] pos is the position encoding of the high-level patch feature, conv is the convolution operation, and upsample represents the upsampling operation;

[0088] Define MultiHead(Q, K, V) as the following formula, where the weight matrix is ​​represented by W:

[0089] MultiHead(Q,K,V)=Concat(head1, head2,..., head h )W O

[0090]

[0091] W OIndicates the weight parameter when each head is merged into Concat in multi-head attention, head i is the result of attention calculation, W Q 、W K 、W V They represent the attention weight parameters of the query Q, key K, and value V input during attention calculation respectively;

[0092] Then the generated pre′ and Perform multi-head attention calculations and compare Perform residual connection to obtain features that focus on abnormal information Replace patch-level embedding Then perform similarity calculation with text features;

[0093]

[0094] Step (2): Spatial Attention:

[0095] For low-level patch features Perform spatial attention to focus on pixel information in anomalies;

[0096] Take the spatial attention module in the CBAM attention module and define M C (F) is the spatial attention module function, where for the feature F input to the module, the output is:

[0097] M C (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0098] F is the input feature of the spatial attention module, MaxPool(·) is the maximum pooling operation, AgvPool(·) is the average pooling operation, MLP(·) is a multi-layer perceptron, i.e., a feedforward neural network, and σ represents the sigmoid activation function;

[0099] Then, a two-dimensional spatial attention map is obtained, that is, the weight value of each feature point in the input feature map is obtained;

[0100] Setting low-level patch features To get closer to the shape of the abnormal feature map pre, by Perform deconvolution and then restore the resolution of the feature map;

[0101] Make the low-level patch features Focus on the spatial information of pre and restore it to low-level patch features The shape of , and finally the output feature Output as a new low-level patch feature;

[0102]

[0103] Among them, upsample(·) is the upsampling operation, M C Get the attention weight value for the spatial attention operation, deconv(·) is the deconvolution operation, represents the matrix multiplication operation, and conv2(·) is the convolution operation;

[0104] All visual features of CLIP When attention is performed separately, the comprehensive detection is the best when the number of channel attention and the number of spatial attention are the same, that is,

[0105]

[0106] Combining these two attention mechanisms, the network can better capture complex image features and improve the expressiveness and performance of the model.

[0107] The step 4 is specifically as follows:

[0108] For loss calculation, we align visual embeddings and text embeddings from both global and local perspectives, that is, we optimize the learnable text cues from two perspectives, and design global and local loss functions to adjust the trainable parameters of the model.

[0109]

[0110] Among them, L global is the cross entropy loss, label is the true abnormal type label of the image, M gt ∈R H×W represents the true mask, I represents the full 1 matrix, t [n，a] Represents text features, n and a represent normal and abnormal tags respectively, represents the patch image feature after attention fusion with the abnormal feature map pre; Focal and Dice are two loss functions, P(·) represents the calculation of the abnormal probability score, and λ is a hyperparameter that balances global and local losses;

[0111] During the training process, after the loss is calculated, the backpropagation operation is performed, that is, the gradient of each model parameter with respect to the loss function is calculated by the chain rule, the parameters are updated according to these gradients, and the gradient descent optimization algorithm is used to minimize the loss function.

[0112] Beneficial effects of the present invention:

[0113] First, the proposed framework significantly improves its ability to capture fine-grained anomaly information by integrating a reconstruction guidance strategy with a cross-modal alignment network, thereby overcoming the shortcomings of existing cross-modal alignment network-based methods in anomaly localization. The cross-modal alignment network and reconstruction module in the framework are highly scalable, allowing any pre-trained cross-modal alignment network and reconstruction network to be directly ported to the framework without retraining.

[0114] Specifically, the reconstruction module in the present invention can predict the abnormal information of the input sample, which is then used to guide the cross-modal alignment network, enabling it to focus more on identifying subtle abnormal features. This method not only takes advantage of the generalization advantages of the cross-modal alignment network, but also realizes a unified model that can be widely used in anomaly detection, localization and classification tasks in the industrial field. The model does not need to be trained separately for each anomaly category, which greatly simplifies the training process and improves the applicability of the model. The model proposed in the present invention only needs to be trained on an auxiliary training set, without fine-tuning, and can adapt to any industrial data set for effective anomaly detection and localization, with good generalization.

[0115] Second, this paper employs a dual attention mechanism to achieve reconstruction guidance. This approach combines spatial attention and channel attention strategies. By carefully considering the different characteristics of the extracted patch-level features and using a corresponding attention allocation strategy, the model's attention to various unusual details is enhanced. This means that the model can not only focus on basic features such as simple texture and color, but also conduct in-depth analysis of higher-level features.

[0116] Third: The present invention has conducted extensive reasoning tests on datasets of multiple real industrial scenarios, including MVTecAD, VisA, and MPDD. Compared with existing models, the model of the present invention has demonstrated superior performance. In supervised and small-sample learning tasks, the model of the present invention can not only effectively complete anomaly detection, but also identify anomaly types with an accuracy rate of over 85%. In zero-shot learning tasks, especially on the MPDD dataset, the model's performance indicators for anomaly detection reached an average I-AUROC and P-AUROC of 78.6% and 92.6% respectively, surpassing other advanced anomaly detection models. These results demonstrate the effectiveness and superiority of the model in the field of industrial anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1 It is a schematic diagram of the overall structure of the model of the present invention.

[0118] Figure 2 It is a schematic diagram of the reconstruction module structure of the present invention.

[0119] Figure 3Schematic diagram of the cross-modal alignment network of the present invention.

[0120] Figure 4 2 is a schematic diagram of the dual attention module structure of the present invention.

[0121] Figure 5 It is a schematic diagram of the anomaly detection results of the present invention with different amounts of channel attention and spatial attention allocated.

[0122] Figure 6 Schematic diagram comparing the performance of different attention settings of the model.

[0123] Figure 7 This is a schematic diagram of the VisA and MPDD classification effects in the 0-shot experiment.

[0124] Figure 8 This is a schematic diagram of the abnormal classification effect of the VisA dataset in the 2-shot experiment. DETAILED DESCRIPTION

[0125] The present invention will be described in further detail below with reference to the accompanying drawings.

[0126] like Figure 1 As shown, the present invention discloses a small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment, which can achieve accurate anomaly detection and localization, realize anomaly detection and classification based on the CLIP model, and introduce a reconstruction network, which is used to guide the model to focus on anomaly details.

[0127] First, we designed multiple classes of learnable, object-agnostic text prompt templates within CLIP's text encoder. Furthermore, we tried to include as many exception types as possible within the exception prompt templates as possible across all industry domains. We also added additional learnable parameters to the CLIP text encoder to enable fine-tuning. We also extracted outputs from multiple intermediate layers of the visual encoder to align visual and text features globally and locally.

[0128] Secondly, in order to make the cross-modal alignment network pay more attention to abnormal details, a powerful reconstruction network is trained to calculate the difference between the image before and after reconstruction to obtain the predicted global abnormality feature, which is input into the cross-modal alignment network to guide it to focus on abnormal information.

[0129] Finally, a dual attention fusion module fuses anomaly features with local visual features through spatial attention and channel attention. The fused features are compared with the contextual cue embedding for learning, ultimately achieving accurate prediction, classification, and localization.

[0130] Step 1: Reconstruction module extracts preliminary anomaly information: Aiming at the problem that the cross-modal alignment network (CLIP) is insensitive to detailed texture information, this application proposes to use a reconstruction module to guide CLIP to better perceive anomaly details. The reconstruction module extracts preliminary anomaly detail information by calculating the difference between the input image and the reconstruction output. This anomaly detail information is used as the query key Q for attention calculation with the CLIP visual features, highlighting the abnormal parts in the visual features. Introducing the reconstruction module can effectively guide CLIP to focus on detailed anomalies, thereby improving its accuracy in the anomaly localization task.

[0131] The reconstruction network can perceive normal information but cannot reconstruct abnormal information. By this feature, the input features and the reconstructed normal features of the output can be compared, and anomalies can be detected based on the differences before and after reconstruction. Figure 2 This is the structure of the reconstruction module in the technical framework of the present invention.

[0132] To achieve a multi-class unified model, a memory-based method is used. By converting normal features into low-dimensional coordinates and mapping them to a continuous grid, the input features can be compared with the normal features stored in the grid, and then the difference positions can be predicted. The grid storage method enables this reconstruction module to effectively retain normal information and eliminate any anomalies existing between feature data.

[0133] For the input dataset image x i First, it passes through the pre-trained backbone network efficientnet-b4 to extract features, and the features are represented by f x where f x ∈R C×H×W , and C, H, and W respectively represent the number of channels, height, and width of the features.

[0134] From both local and global perspectives, by combining unique features from different perspectives, fine-grained details and a broader overall structure are captured.

[0135] (1) Local feature representation:

[0136] When converting the input feature f x to a local feature represented by low-dimensional coordinates, first, without changing the number of channels, the information of different channels is superimposed through a 1D convolution so that the feature mapped to the coordinate representation can take into account the information between different channels. Then, through a convolutional layer with a kernel of 1 the number of channels is changed, and the original channels are mapped to n (n < c) dimensions to achieve coordinate conversion, that is, each pixel is converted into an n-dimensional coordinate value. <000049 s>

[0137] <,

[0138] Among them is the input feature f x The representation after converting the n-dimensional coordinates, where i, j (i∈h, j∈w) represents the feature f x At each pixel position in , T1 and T represent the hyperbolic tangent activation of the tanh function;

[0139] To ensure that the features output by the reconstructed network retain all normal information and replace the input abnormal information with similar contextual normal features in the local environment, a grid design method is adopted. The grid is trained as a function with infinite resolution coordinates. When the input coordinates do not have an exact match in the grid, the method of adjacent feature aggregation is used to interpolate based on the distance between the input feature coordinates and the coordinates of the adjacent features, and finally output the feature representation of the corresponding coordinates:

[0140] S(v,G)=|vn|G[m]+|vm|G[n]

[0141]

[0142] Where S(·) is the grid mapping function, v represents the coordinate, G represents the mapped grid, and m and n are the adjacent coordinates of v.

[0143] Setting up a learnable local grid pass In G l Find the value of the corresponding position in the grid and use it as the representative of the pixel. From another perspective, this operation is to store the pixel features in a continuous grid. Using pixel-level coordinates, normal features can be sampled from the local grid, and finally the feature representation that can represent the pixel in the output grid is Right now:

[0144]

[0145] Where S(.,G) represents the sampling process from the grid G, and the number of channels of the output feature representation is the number of channels of the grid, which means that each channel in the grid is sampled, resulting in a more accurate representation. The local feature representation of the final output is defined as f l .

[0146] (2) Global feature representation:

[0147] For the input feature f x , through linear mapping, the number of channels is mapped to n, that is, an n-dimensional coordinate v representing the global feature n :

[0148]

[0149] Among them, Linear is a linear layer network, Ta 、T b is the tanh activation function.

[0150] For a learnable global grid By global coordinate v n Capture input features f x The global representation of x The information of each dimension in the grid is set, so the number of grid channels C is set g =C*H*W, so that all information can be taken into account during sampling. The sampling process is as follows:

[0151]

[0152] f g =Re(f′ g )f g ∈(R C×H×W )

[0153]

[0154] After sampling, the obtained feature is represented as f′ g , the shape is Through the Re function, the global feature is represented as f′ g Reshape to R C×H×W The global feature f g .

[0155] The final output global feature representation not only effectively replaces global anomalies but also distinguishes the inter-class distribution under a unified background.

[0156] (3) Fusion feature representation:

[0157] By combining the global and local feature representations, the normal features of the input image Image are effectively learned. The global and local feature representations are connected through the concat(·) operation, and then the normal features are reconstructed through the continuous residual network Rec(·):

[0158] f(x)=Rec(concat(f l , f g ))

[0159] Among them, f l is the local feature of the input image, f g is the global feature of the input image.

[0160] During training, by comparing f(x) with the input feature f xThe similarity between the two learnable grids allows them to continuously learn. The mean square error and cosine similarity are combined to evaluate the similarity between the two, refining the final feature representation so that it can accurately reconstruct normal features. In actual use, the pre-trained reconstruction module parameters are used, eliminating the need for retraining.

[0161] Therefore, in this application, normal feature representation is obtained by local and global grid sampling, and reconstructed normal features are obtained through convolutional network. The reconstructed normal features are compared with the input features to obtain the abnormal feature map pre, which is used as the abnormal feature predicted by the network:

[0162]

[0163] Through the setting of this module, considering both global and local levels, and combining the characteristics of continuous feature storage, fine-grained abnormal information can be predicted.

[0164] Step 2: Fine-tune the cross-modal alignment network to extract textual and visual features: CLIP has demonstrated excellent performance in classification tasks through a contrastive cue learning strategy. This strategy is applied to the anomaly detection task. By adjusting CLIP's textual cues, it can identify anomaly types and align these cues with image features.

[0165] The cross-modal alignment network structure diagram in the technical framework of the present invention is as follows: Figure 3 As shown. The left figure (a) is the visual encoder of the CLIP model, which is implemented by the ViT network, and the right figure (b) is the text encoder of the CLIP model, which consists of a 12-layer transformer encoder network. In order to ensure the model's zero-shot anomaly detection capability, this application avoids using specific image category information as prompts, but instead uses multiple general and image category-independent texts as prompts. In order to learn these general text prompt templates, global and local context optimization techniques are introduced, which helps to integrate abnormal semantics of different granularities into object-agnostic text embeddings. In addition, by aggregating multi-layer visual intermediate layer features, richer local detail information can be provided, thereby enhancing the model's ability to recognize anomalies. In the training phase, all modules are jointly optimized by combining global and local context optimization methods. In the inference phase, by quantifying the misalignment between text and global and local visual embeddings, the predicted anomaly type is determined, the anomaly score is calculated, and an anomaly map is generated. In the loss calculation part, the anomaly type predicted by the model is compared with the true label to calculate the cross entropy loss, and the anomaly map predicted by the model is compared with the true mask to calculate the Focal and Dice losses, and the model parameters are optimized by Adam.

[0166] Step (1): Text hint template design: CLIP's text hint template is used to provide semantic information in the cross-modal alignment task, enabling the model to understand and match the association between image and text. In this paper, for anomaly detection and classification tasks, a specific anomaly classification text template is designed to describe different anomaly types so that the image can be matched with the template to obtain anomaly classification results. For the pixel-level anomaly localization task, the hint provided by the text template is encoded and similarity is calculated with the pixel-level features to predict whether each pixel is abnormal, and finally generate an anomaly segmentation mask.

[0167] For anomaly detection and classification, the goal is to perceive anomalies, so it's necessary to incorporate prior semantic information about anomalies into the hint template. To facilitate classification, the anomaly text hint template lists as many anomaly categories as possible, such as scratches and holes. Furthermore, because the CLIP model focuses on aligning visual features with object semantics rather than distinguishing between abnormal and normal image semantics, designing a hint template for anomaly types, such as scratches, simply using [Aphotoofa[cls]withscratches] is difficult to distinguish. Instead, a learnable text hint template is employed, such as [{cls}withscratches], as the hint for the scratch class. A learnable token is added to the front of the hint template to ensure that the text embedding encompasses a wider range of semantic information during training. Considering the zero-shot detection task, we hope to maintain good performance on unseen categories. We adopt the same approach as existing CLIP-based anomaly detection methods, using object-independent text hints, replacing {cls} in the hint template with {object}. This approach focuses on normal and abnormal patterns in the image without considering object semantics. The text embeddings generated by this template are more general and can identify anomalies across different objects and domains.

[0168] The designed text prompt template is as follows:

[0169] t n =[P1][P2]...[P E ][normal][object].

[0170] t a =[Q1][Q2]...[Q E ][c k ][object].

[0171] C=[c1,c2...c K ](c1: broken, c2: contamination...)

[0172] Among them, tn Represents a normal type text template, t a A text template representing an exception type, C represents the exception type, [P i ],[Q i ](i∈1,...,E) denotes the learnable word embeddings of normal and abnormal text templates.

[0173] Step (2): The text encoder extracts text features. The main function of the CLIP text encoder is to convert text prompt information into a vector representation that can be aligned with visual features, thereby enabling effective cross-modal learning between images and text. During the training process, the text features are continuously aligned with the CLIP visual features to optimize the trainable parameters. Through this cross-modal comparative learning, the network model can further learn the knowledge between text semantics while understanding the visual level.

[0174] Learnable parameters are added to the text encoder of CLIP, enabling the model to learn distinguishable abnormal semantic information. The learnable parameters are set and added in front of the text prompt embedding and input into the encoder together with the text prompt embedding. After being processed by each layer of the encoder, the learnable text embedding is optimized. The details are as follows:

[0175] L m =[l1, l2, ..., l M ]

[0176] [l′ m+1 ,t m+1 ]=T m (cat((l m , t m ),dim=chanel))

[0177] [l′ m+2 ,t m+2 ]=T m+1 (cat((l m+1 , t m+1 ), dim=chanel)

[0178] …

[0179] t M+1 =T M (t M )

[0180] Among them, L m represents the learnable parameters of the m-th layer text encoder, t m represents the learnable text embedding at the mth layer, l′ m+1 T represents the output of the m-th layer's learnable parameters after the m-th layer's text encoder. MRepresents the frozen M-th layer text encoder, and the cat operation aggregates the learnable parameters and text embedding along the channel dimension.

[0181] Finally, the text features t representing normal and abnormal information are obtained through the CLIP text encoder n , t a , which is subsequently used to calculate the similarity with CLIP visual features to obtain the loss;

[0182] Step (3): Visual encoder extracts features:

[0183] CLIP's visual encoder extracts key visual features from images and converts them into a unified feature space vector, working with the text encoder to achieve comparative learning between images and text. The visual encoder enhances the model's cross-modal alignment and visual understanding capabilities, enabling CLIP to excel in complex visual tasks.

[0184] Since it is difficult to identify some minor anomalies based on global image information, we extract the output of multiple intermediate layers of the visual encoder, namely patch-level image features, to supplement the image detail information.

[0185] For the input dataset image x i , after the encoder, we finally get a token-like feature representing global semantic information and the output of multiple intermediate layers:

[0186] f i =F(x i ) f i ∈R H×W×D

[0187]

[0188] Where F(·) represents the visual encoder, f i represents the global image features, represents the image features at the intermediate layer patch level, and |S| represents the set of extracted intermediate layers of the visual encoder.

[0189] Step 3: Dual attention fusion of visual features and preliminary abnormal information:

[0190] The abnormal feature map pre output by the reconstruction module is rich in fine-grained information. To address the problem of insufficient detail recognition in CLIP, a reconstruction module is proposed to adjust the patch-level image features of the cross-modal alignment network. By performing dual attention calculation on the patch-level image features and the reconstruction features, the attention of the patch-level image features to abnormal details is enhanced. The dual attention module structure in the technical model of the present invention is as follows: Figure 4 shown.

[0191] According to the visual encoder characteristics of the cross-modal alignment network, a dual attention mechanism is designed to integrate the |S| intermediate layer patch-level image features They are divided into two groups: high-level patch features and low-level patch features;

[0192] In the design of this invention, |S| is 4, and spatial attention and channel attention mechanisms are implemented for the outputs of low-level and high-level encoders respectively, which significantly improves the sensitivity of the CLIP model to detail anomalies.

[0193] Step (1): Channel Attention

[0194] For channel attention, as the input image features of the CLIP visual encoder pass through multiple Transformer encoder layers of the CLIP visual encoder (ViT), the spatial dimension information is gradually abstracted. ViT gradually captures higher-level feature representations. These features gradually transform from simple texture and color patterns to more complex shapes, object parts, and scene information.

[0195] Therefore, high-level patch features need to be Pay attention to abnormal information through different channels.

[0196] High-level patch features Contains a large number of channels and has more complex semantic information, so it is necessary to combine the shape of the abnormal feature map pre with the high-level patch features The shape of the patch is aligned to ensure that the high-level patch features are not changed. In the case of information, we increase the attention to abnormal content, so the abnormal feature map pre simulates the high-level patch feature The embedding operation passes through the convolution layer of the same size and adds the high-level patch features. The same position encoding is used to generate the embedding pre′ of the abnormal feature map pre, achieving

[0197]

[0198] pos is the position encoding of the high-level patch feature, conv is the convolution operation, and upsample represents the upsampling operation.

[0199] Define MultiHead(Q, K, V) as the following formula, where the weight matrix is ​​represented by W:

[0200] MultiHead(Q,K,V)=Concat(head1,head2,...,head h)W O

[0201]

[0202] W O Indicates the weight parameter when each head is merged into Concat in multi-head attention, head i is the result of attention calculation, W Q 、W K 、W V They represent the attention weight parameters of the query Q, key K, and value V input during attention calculation.

[0203] Then the generated pre′ and Perform multi-head attention calculations and compare Perform residual connection to obtain features that focus on abnormal information Replace patch-level embedding Then perform similarity calculation with text features;

[0204]

[0205] Step (2): Spatial Attention

[0206] For spatial attention, in anomaly detection and classification, it is necessary to capture the spatial region in the image, that is, pixel information. The Transformer encoder layer of the ViT network divides the original image into multiple fixed-size patches, each patch containing all the color channel information of the corresponding region in the original image. As more and more encoder layers are passed, the spatial position information of each patch may be lost, while the lower layers of the network tend to capture low-level features and relatively contain more spatial information. Therefore, it is suitable for low-level patch features. Perform spatial attention to focus on the pixel information in the anomaly.

[0207] The spatial attention module in the CBAM attention module is adopted. This module focuses on the spatial information of the input image and is a supplement to the channel attention.

[0208] Define M C (F) is the spatial attention module function, where for the feature F input to the module, the output is:

[0209] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0210] F is the input feature of the spatial attention module, MaxPool(·) is the maximum pooling operation, AgvPool(·) is the average pooling operation, MLP(·) is a multi-layer perceptron, i.e., a feedforward neural network, and σ represents the sigmoid activation function.

[0211] Then, a two-dimensional spatial attention map is obtained, that is, the weight value of each feature point in the input feature map is obtained;

[0212] Because the abnormal feature map pre obtained from the reconstruction module contains richer and more accurate pixel information, and the low-level patch features It is the result of segmentation by the ViT network, so the low-level patch features are set To get closer to the shape of the abnormal feature map pre, by Deconvolution is performed, and then the resolution of the feature map is restored. Deconvolution helps to reconstruct the lost high-frequency details and can be regarded as a sparse connection operation. Compared with the fully connected layer, it reduces the number of parameters and the amount of calculation of the model. At the same time, the abnormal feature map pre is upsampled and compared with the low-level patch feature Perform dimension matching.

[0213] In the design of the present invention, it is necessary to make the low-level patch features Focus on the spatial information of pre and restore it to low-level patch features The shape of the final output feature Output as a new low-level patch feature;

[0214]

[0215] Among them, upsample(·) is the upsampling operation, M C Get the attention weight value for the spatial attention operation, deconv(·) is the deconvolution operation, represents the matrix multiplication operation, and conv2(·) is the convolution operation.

[0216] For the allocation of channel attention and spatial attention, there are a total of |S| patch-level image features As reference Figure 5 As shown, in this design, when |S|=4, the ratio of channel attention and spatial attention is ca:sa. Figure 5 It can be seen from the figure that when there is too much channel attention, it is easy to ignore pixel factors, resulting in false detection; when there is too much spatial attention, too many pixel values ​​are paid attention to, making the perception of abnormal pixel values ​​more acute, without considering other features, resulting in false detection.

[0217] Therefore, all visual features of CLIP When attention is performed separately, the comprehensive detection is the best when the number of channel attention and the number of spatial attention are the same, that is,

[0218]

[0219] Channel attention enhances the relationship between features across channels, improving the discriminative power of feature representations. Spatial attention, on the other hand, focuses on key information at different locations in space, effectively addressing the shortcomings of traditional convolutional neural networks in processing multi-scale, shape-varying, and directional information. By combining these two attention mechanisms, the network can better capture complex image features, improving the model's expressiveness and performance.

[0220] Step 4: Loss calculation:

[0221] For loss calculation, we align visual embeddings and text embeddings from both global and local perspectives, that is, we optimize the learnable text cues from two perspectives, and design global and local loss functions to adjust the trainable parameters of the model.

[0222]

[0223] Among them, L global is the cross entropy loss, label is the true abnormal type label of the image, M gt ∈R H×W represents the true mask, I represents the full 1 matrix, t [n，a] Represents text features, n and a represent normal and abnormal tags respectively, represents the patch image feature after attention fusion with the abnormal feature map pre; Focal and Dice are two loss functions, P(·) represents the calculation of the abnormal probability score, and λ is a hyperparameter that balances global and local losses.

[0224] During training, after the loss is calculated, backpropagation is performed. This involves calculating the gradient of each model parameter with respect to the loss function using the chain rule. The parameters are then updated based on these gradients, and the gradient descent optimization algorithm is used to minimize the loss function. This process allows the model's predictions to gradually approach the true values, thereby improving model performance.

[0225] By comparing the classification results and positioning information predicted by the model with the actual labels and masks, calculating the loss function, and using the Adam optimizer to adjust the model parameters, the model performance is gradually optimized, ultimately achieving accurate detection, precise positioning, and classification of anomalies.

[0226] experiment:

[0227] Dataset and evaluation metrics:

[0228] (1) Datasets: Three datasets are used in all experiments of the reconstruction-guided cross-modal alignment anomaly detection framework: MVTec-AD, VisA, and MPDD. During the training and testing process of each experiment, the corresponding dataset is divided into a training set and a test set with a ratio of 70% and 30%.

[0229] In our experiment, we selected 1725 images from the MVTec-AD test dataset, which contains both normal and abnormal samples. We augmented the abnormal samples in these 1725 images by rotating, noisy, and shifting them, reducing the total to approximately 5200 abnormal images. We then combined 30% of these abnormal samples with the 3624 normal samples in the MVTec training set to form a total of 5420 training and test samples, with a normal to abnormal sample ratio of approximately 2.5 / 1. To divide the training and test sets, we divided 80% of the 5420 images containing both normal and abnormal samples into a training set (4336) and 20% into a test set (1024).

[0230] In addition, we conducted anomaly detection and anomaly type classification tests on the 1024-image MVTec-AD test set, and performed zero-shot inference on the MPDD and VisA datasets to evaluate the generalization of our model. Furthermore, we conducted additional experiments on the MPDD and VisA datasets, using the 4336 images from the MVTec-AD training set and a small amount of MPDD and VisA data as the model training set.

[0231] (2) Evaluation metrics: Based on previous work, we used the metrics I-AUROC and P-AUROC. To further validate the superiority of the proposed method, we also added the average precision (AP) in anomaly detection and anomaly localization as an evaluation metric. For anomaly classification, we used the accuracy rate, a direct performance metric that represents the ratio of the number of anomalies correctly classified by the model to the total number of anomalies.

[0232] Implementation details:

[0233] Using the pre-trained CLIP network, we added learnable parameters to its text encoder for fine-tuning. For the reconstruction module, we adopted a pre-trained reconstruction module and obtained a trained model by pre-training on the 3629 training sets in MVTec-AD to meet the experimental requirements of this article.

[0234] The input image is resized to a resolution of 518×518. For the CLIP network, the outputs of the 6th, 12th, 18th, and 24th layers of its visual encoder are extracted as patch-level image embeddings. For the reconstruction module, features are first extracted from the input image. The pre-trained EfficientNet-B4 network is used as the backbone to extract features. The features extracted from the 3rd and 4th layers are used as output features for feature reconstruction.

[0235] The experiment was conducted based on a single-GPU NVIDIA GeForce RTX 3090 server with 24 GB of video memory, a batch size of 2, 18 training rounds, and two learning rates of 9×e(-4) and 1×e(-4) for the text and vision modules, respectively.

[0236] Performance Analysis:

[0237] The model of this application can simultaneously take into account the three functions of anomaly detection, positioning, and classification. For the classification of anomaly types, this application sets 43 types of anomalies of different industrial products in common real industrial scenarios. For samples known or unknown to the model, it has good performance in anomaly detection, positioning, and classification. That is, only one model can simultaneously realize anomaly detection, anomaly positioning, and anomaly classification, and can intercept up to 43 types of anomalies. The performance of the model is explained below through specific indicators.

[0238] (1) 0-shot anomaly detection

[0239] The divided MVTec dataset is used for auxiliary training, and MPDD and VisA are used as target datasets for detection. At the same time, the proposed method is compared with CLIP-based methods (AnomalyCLIP, CLIP, CLIP-AC, WinCLIP, etc.). Under the same settings, the text is changed to a consistent prompt template describing 44 abnormal categories for abnormality classification. The comparison is performed by calculating the mean index of all categories in the dataset. The comparison results of the VisA dataset are shown in Table 1, and the comparison results of the MPDD dataset are shown in Table 2.

[0240] Table 1: Comparison of detection results on the VisA dataset on the ZSAD task

[0241]

[0242] Table 2: Comparison of detection results on the MPDD dataset on the ZSAD task

[0243]

[0244] The comparison results show that for the VisA dataset, the performance of the model proposed in this application on the ZSAD task has reached the detection effect of the most advanced model algorithm in the current same task type while taking into account the classification of anomaly types. At the same time, on the more challenging MPDD dataset, the average I-AUROC and AP indicators of all categories in the dataset reached 78.6% and 84.1%, surpassing other algorithms.

[0245] (2) 2-shot

[0246] This application selects two challenging datasets, MPDD and VisA, for 2-shot small sample testing. The detection results of the VisA dataset are shown in Table 3, and the detection results of the MPDD dataset are shown in Table 4.

[0247] Table 3: 2-shot detection results of VisA dataset

[0248]

[0249] Table 4: 2-shot detection results on the MPDD dataset

[0250]

[0251] Experimental results demonstrate excellent overall performance in anomaly detection and localization on the VisA dataset. Over three categories achieved performance exceeding 90% for anomaly detection, and all categories achieved performance exceeding 90% for anomaly localization, demonstrating outstanding performance in anomaly localization. For the MPDD dataset, over four categories achieved performance exceeding 90% in the two-shot anomaly detection task, with metalplate and tubes achieving an I-AUROC close to 100%, a rarity in two-shot tasks. Furthermore, our model also demonstrated superior performance in the P-AUROC metric for anomaly localization on the MPDD dataset.

[0252] (3) 4-shot

[0253] This application selects two challenging datasets, MPDD and VisA, for 4-shot small sample testing. The detection results of the VisA dataset are shown in Table 5, and the detection results of the MPDD dataset are shown in Table 6.

[0254] Table 5: 4-shot detection results on the VisA dataset

[0255]

[0256] Table 6: 4-shot detection results on the MPDD dataset

[0257]

[0258] Experimental results show that after increasing the sample size, the overall performance in anomaly detection and localization is good in the VisA dataset. Among them, more than 4 categories have anomaly detection indicators higher than 90%, and the indicators of all categories in anomaly localization are higher than 90%, showing outstanding performance in anomaly localization. For the MPDD dataset, more than four categories have performance indicators higher than 90% in the 2-shot anomaly detection task, with obvious advantages in both detection and localization indicators.

[0259] This application uses the image anomaly information calculated by the reconstruction network to guide CLIP for anomaly detection. Therefore, the ablation analysis is mainly conducted on the attention fusion module in the guiding ideology to verify the effectiveness of the guiding ideology of this article.

[0260] The number of channel and spatial attention modules is set, and whether the image-level embedding adds attention modules is analyzed. The training is based on the divided MVTec-AD dataset, and the test is based on the divided MVTec-AD dataset. When other settings are exactly the same, the comparison is as follows: Figure 6 shown.

[0261] Figure 6 The figures represent different ratios of spatial attention and channel attention applied to the patch-level image features output by the four visual encoders, with all other settings identical. By comparing the average values ​​of the two most representative metrics, I-AUROC and P-AUROC, in the test set, we examine the overall performance of anomaly detection and localization on the same dataset. The final experimental results show that the model achieves the best overall results when it only applies two channel attention and two spatial attention operations to the patch-level image features output by the image encoder.

[0262] Visualization of model detection effect:

[0263] For the model whose training set is 4336 images of MVTec-AD, we use 1024 image samples divided by MVTec-AD for testing, detect and segment the anomalies in the samples, and output the anomaly category and corresponding score value. In the 0-shot task, the test results of this application model are as follows: Figure 7 shown.

[0264] To more comprehensively evaluate the performance of the model, we used 4336 images from the MVTec-AD dataset as the training set, and added a small number of samples from the VisA and MPDD datasets to assist in the training of the classification task, thus conducting a small sample task test. During the detection process, only the VisA and MPDD datasets were used for detection. In the 2-shot task, the proposed model can still classify different types of anomalies. The specific test results are shown in the figure below. Figure 8 shown.

[0265] from Figure 8 The data show that the model can still accurately judge and locate anomalies in small sample tasks, and has good performance in classification tasks.

[0266] This paper proposes a zero-shot / few-shot anomaly detection framework using reconstruction-guided cross-modal alignment. The reconstruction module and cross-modal alignment network in this framework are scalable and can be replaced by pre-trained networks of the same type. Furthermore, by introducing anomaly type hints into the text encoder of the cross-modal alignment network, the model can classify anomalies.

[0267] To improve the cross-modal alignment network's ability to detect anomaly details, this paper introduces a reconstruction network to provide guidance. The reconstruction network captures the difference between its output and input, which is used as anomaly information. This anomaly information removes background and guides the patch-level image features extracted by the cross-modal alignment network, focusing more on anomalies.

[0268] By designing spatial attention and channel attention in the attention module, this paper enables the patch-level image features of the cross-modal alignment network to focus on abnormal information from two perspectives. For low-level patch-level image features in the cross-modal alignment network, spatial attention can perceive anomalies from aspects such as color and texture; for high-level patch-level image features in the cross-modal alignment network, channel attention can perceive anomalies from aspects such as semantics. This dual attention mechanism enhances the cross-modal alignment network's ability to perceive fine-grained anomalies and can accurately locate anomalies.

[0269] Furthermore, the present invention incorporates textual cues representing 43 common anomaly types (e.g., scratches, holes, etc.) in the text encoder branch of the cross-modal alignment network. Ultimately, the proposed model is able to identify and classify up to 43 different anomaly categories, which is extremely valuable in industrial applications because it enables more accurate fault diagnosis and preventive maintenance. In this way, the present invention brings a powerful and flexible solution to the fields of industrial automation and quality control.

Claims

1. A small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment, characterized by: Including cross-modal alignment network CLIP, reconstruction network and dual attention fusion module; The reconstruction network takes the normal or abnormal RGB image in the dataset as input, reconstructs the image to obtain a completely normal image of the same size as the input, and calculates the difference between the image before and after reconstruction to obtain the predicted global abnormality features; For the input dataset image x i First, the features are extracted through the pre-trained backbone network efficientnet-b4, and then f x Indicates that f x ∈R C×H×W , C, H, and W represent the number of channels, height, and width of the feature respectively; The memory-based method sets up a reconstruction network, which takes both local and global perspectives to represent features. By combining features from different perspectives, it provides a representation for the input that captures both fine-grained details and broader overall structures. By converting features into low-dimensional coordinates and mapping them to a continuous grid, the input features can be compared with the normal features stored in the grid to predict the difference location. Grid storage enables the reconstruction network to effectively retain normal information and eliminate anomalies between feature data. The cross-modal alignment network includes a visual encoder and a text encoder. The text encoder extracts text features based on text prompts, the visual encoder extracts visual features from normal or abnormal RGB images in the dataset, and the text encoder adds additional learnable parameters for fine-tuning. The dual attention fusion module performs attention fusion on the abnormal feature and the visual feature, and outputs a visual feature with enhanced abnormal detail information; The similarity between the text features and the visual features after the abnormal detail information enhancement is used to determine whether the input image is abnormal and obtain a pixel-level abnormality positioning result.

2. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to the framework of claim 1 is characterized in that: The following steps are included: Step 1: Use the reconstruction network to reconstruct the dataset image into a completely normal image. The reconstruction network outputs consistent results for normal images and reconstructs abnormal areas into normal content for abnormal images through neighborhood features. The output of the reconstruction network is compared with the input to determine the abnormal information in the input image. Step 2: Using the visual encoder and text encoder branches, the text encoder extracts text features and the visual encoder extracts visual features respectively. The text encoder adds additional learnable parameters for fine-tuning. Step 3: Perform attention fusion on the abnormal information and the visual features, and output the visual features with enhanced abnormal detail information; Step 4: By calculating the similarity between the text features described in step 2 and the visual features after the abnormal detail information enhancement described in step 3, it is determined whether the input image is abnormal and the pixel-level abnormality positioning result is obtained.

3. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 2 is characterized in that Specifically: Step (1): Local feature representation: When converting the input feature f x into a local feature in low-dimensional coordinate representation, first, without changing the number of channels, the information of different channels is superimposed through 1D convolution so that the feature mapped to the coordinate representation can take into account the information between different channels. Then, through a convolutional layer with a kernel of 1 the number of channels is changed, and the original channels are mapped to n (n < c) dimensions to achieve coordinate conversion, that is, each pixel is converted into an n-dimensional coordinate value; in is the input feature f x The representation after converting the n-dimensional coordinates, where i, j (i∈h, j∈w) represents the feature f x At each pixel position in , T1 and T represent the hyperbolic tangent activation of the tanh function; Using the grid design method, the grid G ​​is trained as a function with infinite resolution coordinates, and the features corresponding to the input coordinates in the grid G ​​are output; The grid stores the feature representation corresponding to each n-dimensional coordinate; the features corresponding to the n-dimensional coordinates in the grid G ​​are extracted through the grid mapping function S(·); Step (2): Global feature representation: For the input feature f x , through linear mapping, the number of channels is mapped to n, that is, an n-dimensional coordinate v representing the global feature n : Among them, Linear is a linear layer network, T a 、T b is the tanh activation function; For a learnable global grid By global coordinate v n Capture input features f x The global representation of x The information of each dimension in the grid is set, so the number of grid channels C is set g =C*H*W, so that all information can be taken into account during sampling. The sampling process is as follows: f g =Re(f′ g ) f g ∈(R C×H×W After sampling, the obtained feature is expressed as f′ g , the shape is Through the Re function, the global feature is represented as f′ g Reshape to R C×H×W The global feature f g ; The global feature representation of the final output not only effectively replaces the global anomaly, but also distinguishes the inter-class distribution under the same background; Step (3): Fusion feature representation: By combining the global and local feature representations, the normal features of the input image Image are effectively learned. The global and local feature representations are connected through the concat(·) operation, and then the normal features are reconstructed through the continuous residual network Rec(·): f(x)=Rec(concat(f l ,f g )) Among them, f l is the local feature of the input image, f g is the global feature of the input image.

4. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 3 is characterized in that When the input coordinates do not have an exact match in the grid, the neighboring feature aggregation method is used to interpolate based on the distance between the input feature coordinates and the coordinates of the neighboring features, and finally output the feature representation of the corresponding coordinates: S(v,G)=|vn|G[m]+|vm|G[n] Where S(·) is the grid mapping function, v represents the coordinate, G represents the mapped grid, and m and n are the adjacent coordinates of v; Setting up a learnable local grid pass In G l Find the value of the corresponding position in the grid and use it as the representative of the pixel. Finally, output the feature representation of the pixel in the grid. Right now: Where S(.,G) represents the sampling process from the grid G, and the number of channels of the output feature representation is the number of channels of the grid, which means that each channel in the grid is sampled. The local feature representation of the final output is defined as f l .

5. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 4 is characterized in that During training, by comparing f(x) with the input feature f x The similarity of the two learnable grids is used to continuously learn, and the mean square error and cosine similarity are combined to evaluate the similarity between the two, and the final feature representation is refined so that it can accurately reconstruct normal features; The normal feature representation is obtained by local and global grid sampling, and the reconstructed normal features are obtained through the convolutional network. The reconstructed normal features are compared with the input features to obtain the abnormal feature map pre, which is used as the abnormal feature predicted by the network:

6. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 5 is characterized in that In step 2, in the CLIP model, the text prompt template, the text encoder, and the visual encoder work together to achieve cross-modal learning. The text prompt template is a natural language description input to the text encoder to express the semantic content of the image. The text encoder captures the semantic information of the text by converting these text prompts into vector representations. At the same time, the image is converted into a vector representation by the visual encoder to capture the visual features of the image. Finally, CLIP optimizes the outputs of the two encoders through contrastive learning, so that images and texts with the same semantics are closer in the same embedding space, achieving effective alignment between the image and text.

7. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 6 is characterized in that Step (1): Design of text prompt template: The designed text prompt template is as follows: t n =[P1][P2]...[P E ][normal][object]. t a =[Q1][Q2]...[Q E ][c k ][object]. C=[c1,c2…c K ](c1:broken,c2:contamination...) Among them, t n Represents a normal type text template, t a A text template representing an exception type, C represents the exception type, [P i ],[Q i ](i∈1,...,E) represents the learnable word embeddings of normal and abnormal text templates; Step (2): Text encoder extracts text features: Learnable parameters are added to the text encoder of CLIP, enabling the model to learn distinguishable abnormal semantic information. The learnable parameters are set and added to the front side of the text prompt embedding and input into the encoder together with the text prompt embedding. After each layer of the encoder is processed, the learnable text embedding is optimized. The details are as follows: L m ＝[l1，l2,...,l M ] [l′ m+1 ,t m+1 ]=T m (cat((l m ,t m ),dim=chanel)) [l′ m+2 ,t m+2 ]=T m+1 (cat((l m+1 ,t m+1 ),dim=chanel)) … t M+1 =T M (t M ) Among them, L m represents the learnable parameters of the m-th layer text encoder, t m represents the learnable text embedding at the mth layer, l′ m+1 T represents the output of the m-th layer's learnable parameters after the m-th layer's text encoder. M represents the frozen M-th layer text encoder, and the cat operation aggregates the learnable parameters and text embedding along the channel dimension; Finally, the text features t representing normal and abnormal information are obtained through the text encoder of CLIP n , t a , which is subsequently used to calculate the similarity with CLIP visual features to obtain the loss; Step (3): Visual encoder extracts features: For the input dataset image x i , after the encoder, we finally get a token-like feature representing global semantic information and the output of multiple intermediate layers: f i =F(x i ) f i ∈R H×W×D Where F(·) represents the visual encoder, f i represents the global image features, represents the image features at the intermediate layer patch level, and |S| represents the set of extracted intermediate layers of the visual encoder.

8. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 7 is characterized in that The step 3 is specifically as follows: The abnormal feature map pre output by the reconstruction module is rich in fine-grained information. According to the visual encoder characteristics of the cross-modal alignment network, a dual attention mechanism is designed to integrate the image features at the |S| intermediate layer patch level They are divided into two groups: high-level patch features and low-level patch features; Step (1): Channel Attention For high-level patch features The different channels of the abnormal information are focused on, and the shape of the abnormal feature map pre is combined with the high-level patch features Align the shape of the abnormal feature map to simulate the high-level patch features The embedding operation passes through the convolution layer of the same size and adds the high-level patch features. The same position encoding is used to generate the embedding pre′ of the abnormal feature map pre, achieving pos is the position encoding of the high-level patch feature, conv is the convolution operation, and upsample represents the upsampling operation; Define MultiHead(Q, K, V) as the following formula, where the weight matrix is ​​represented by W: MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O W O Indicates the weight parameter when each head is merged into Concat in multi-head attention, head i is the result of attention calculation, W Q 、W K 、W V They represent the attention weight parameters of the query Q, key K, and value V input during attention calculation respectively; Then the generated pre′ and Perform multi-head attention calculations and compare Perform residual connection to obtain features that focus on abnormal information Replace patch-level embedding Then perform similarity calculation with text features; Step (2): Spatial Attention: For low-level patch features Perform spatial attention to focus on pixel information in anomalies; Take the spatial attention module in the CBAM attention module and define M C (F) is the spatial attention module function, where for the feature F input to the module, the output is: M C (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) F is the input feature of the spatial attention module, MaxPool(·) is the maximum pooling operation, AgvPool(·) is the average pooling operation, MLP(·) is a multi-layer perceptron, i.e., a feedforward neural network, and σ represents the sigmoid activation function; Then, a two-dimensional spatial attention map is obtained, that is, the weight value of each feature point in the input feature map is obtained; Setting low-level patch features To get closer to the shape of the abnormal feature map pre, by Perform deconvolution and then restore the resolution of the feature map; Make the low-level patch features Focus on the spatial information of pre and restore it to low-level patch features The shape of the final output feature Output as a new low-level patch feature; Among them, upsample(·) is the upsampling operation, M C Get the attention weight value for the spatial attention operation, deconv(·) is the deconvolution operation, represents the matrix multiplication operation, and conv2(·) is the convolution operation; All visual features of CLIP When attention is performed separately, the comprehensive detection is the best when the number of channel attention and the number of spatial attention are the same, that is, 9. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 7, characterized in that The step 4 is specifically as follows: For loss calculation, we align visual embeddings and text embeddings from both global and local perspectives, that is, we optimize the learnable text cues from two perspectives and design global and local loss functions to adjust the trainable parameters of the model. Among them, L global is the cross entropy loss, label is the true abnormal type label of the image, M gt ∈R H×W represents the true mask, I represents the full 1 matrix, t [n，a] Represents text features, n and a represent normal and abnormal tags respectively, represents the patch image feature after attention fusion with the abnormal feature map pre; Focal and Dice are two loss functions, P(·) represents the calculation of the abnormal probability score, and λ is a hyperparameter that balances global and local losses; During the training process, after the loss is calculated, the backpropagation operation is performed, that is, the gradient of each model parameter with respect to the loss function is calculated by the chain rule, the parameters are updated according to these gradients, and the gradient descent optimization algorithm is used to minimize the loss function.

Images