Small sample anomaly detection and classification framework based on reconstruction guide cross-modal alignment

Abstract

The method is based on a small sample anomaly detection and classification framework for guiding cross-modal alignment through reconstruction, and comprises the following steps: 1, constructing a reconstruction network, reconstructing a data set image into a completely normal image through the reconstruction network, and comparing the output and input of the reconstruction network to determine anomaly information in an input image; 2, the cross-modal alignment network comprises two branches of a visual encoder and a text encoder, and text features and visual features are extracted through the text encoder and the visual encoder respectively; 3, attention fusion is carried out on the abnormal information and the visual features, and the visual features after abnormal detail information enhancement are output; and 4, by calculating the similarity between the text features in the step 2 and the visual features in the step 3, judging whether the input image is abnormal or not and obtaining a pixel-level abnormality positioning result. According to the method, general anomaly detection, positioning and classification with high precision, strong generalization capability and low resource consumption are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial anomaly detection, and in particular relates to a small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment. Background Art

[0002] In recent years, industrial anomaly detection technology has made significant progress and detection performance has been continuously improved. However, in practical applications, due to the wide variety of defects, complex morphology, and limited available training and test data, detection segmentation and anomaly classification tasks still face huge challenges. Deep learning in visual anomaly detection usually includes two main tasks: determining whether there are anomalies in the image (detection task) and determining the location of anomalies at the pixel level (segmentation task). In addition, the anomaly classification task requires further identification of the specific anomaly type, which is still a technical challenge for industrial defect detection and other fields.

[0003] Most current methods focus on anomaly detection and segmentation, mainly because anomalies are usually diverse and uncertain, and it is difficult to cover all potential anomaly types with a limited sample set. Most popular anomaly detection (AD) methods follow the unsupervised learning paradigm, only training on a large number of normal images and building a distribution model of normal samples to detect anomalies. In the testing phase, these methods compare normal and abnormal samples and calculate the difference between input and output to obtain anomaly scores. Since these models are usually based on the assumption that "normal samples are almost the same", such single models can often only be applied to image data of a single category.

[0004] However, with the growing demand for detection of diverse defect types in the industrial field, a single-category model is difficult to meet the task of multi-category anomaly detection. Therefore, it is particularly important to design a general model suitable for multi-category data, which can not only handle different types of feature distributions, but also maintain high performance when detecting various types of anomalies. In addition, solving the generalization problem in scarce data scenarios and further improving the accuracy of anomaly classification and location are still technical difficulties that need to be overcome in this field.

[0005] The task of small-shot anomaly detection (FSAD) has developed rapidly in recent years. Its goal is to accurately locate and segment abnormal regions in input samples with only a small amount of predefined training data. The further zero-shot anomaly detection (ZSAD) task does not rely on any preset training data at all, and requires the model to have strong generalization ability to cope with the significant changes in visual appearance, abnormal objects and background features in different industrial inspection tasks. In recent years, the contrastive language-image pre-training model (CLIP) has become a key visual language basic model for solving the ZSAD task.

[0006] Currently, reconstruction-based methods are usually only applicable to images of a single category, which limits the generalizability of the models to other domains.

[0007] The diffusion model-based method has a long training time and high cost due to its complex denoising and denoising process.

[0008] Feature-based methods require the design of dedicated memory to store features, and thus have higher complexity and storage requirements.

[0009] When the number of target objects is large, both reconstruction-based and feature-based methods require a large number of training samples and deploy specific models for each object category, which makes these methods difficult to apply in actual industrial scenarios. In addition, these methods cannot effectively perform tasks when the target data is not available due to privacy issues or scarce scenarios.

[0010] The improved method based on CLIP still has not completely solved the problem of insufficient sensitivity to texture anomaly information, which makes it difficult to accurately locate subtle anomalies. Therefore, the generated anomaly map usually contains large areas of highlight areas, showing the limitations of CLIP in anomaly localization. Summary of the invention

[0011] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment, which realizes general anomaly detection, positioning and classification with high precision, strong generalization ability and low resource consumption. This method guides cross-modal alignment by reconstructing the network, can adapt to different categories of data sets without a lot of training, and has good generalization ability. Whether facing new data distributions or unknown anomaly types, the model can respond quickly and maintain high detection accuracy. In addition, this method can maintain low resource consumption in different application scenarios and is suitable for environments with low computing power requirements.

[0012] In order to achieve the above object, the technical solution adopted by the present invention is:

[0013] A small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment, including a cross-modal alignment network (CLIP), a reconstruction network, and a dual attention fusion module;

[0014] The reconstruction network takes the normal or abnormal RGB image in the data set as input, reconstructs the image to obtain a completely normal image of the same size as the input, and is used to calculate the difference between the image before and after reconstruction to obtain the predicted abnormal information;

[0015] The cross-modal alignment network includes a visual encoder and a text encoder, the text encoder extracts text features according to text prompts, the visual encoder extracts visual features from normal or abnormal RGB images in the data set, and the text encoder adds additional learnable parameters for fine-tuning;

[0016] The dual attention fusion module performs attention fusion on the abnormal information and the visual features, and outputs the visual features with enhanced abnormal detail information;

[0017] Through the similarity between the text features and the visual features enhanced by the abnormal detail information, it is determined whether the input image is abnormal and a pixel-level abnormality positioning result is obtained.

[0018] The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment includes the following steps:

[0019] Step 1: Use the reconstruction network to reconstruct the dataset image into a completely normal image. The reconstruction network outputs consistent results for normal images, and reconstructs abnormal areas into normal content for abnormal images through neighborhood features. The output and input of the reconstruction network are compared to determine the abnormal information in the input image;

[0020] Among them, a completely normal image is an image that does not contain abnormal information. If the reconstruction network receives a normal image, its output will be highly close to the input image. For abnormal images, the reconstruction network will use the neighborhood features around the abnormal position to reconstruct the abnormal area into content consistent with the normal image, thereby generating a normal image with no abnormalities at all.

[0021] Step 2: Using the visual encoder and text encoder branches, the text encoder and visual encoder are used to extract text features and visual features respectively. The text encoder adds additional learnable parameters for fine-tuning.

[0022] Step 3: Attentionally fuse the abnormal information and the visual features, and output visual features with enhanced abnormal detail information;

[0023] Step 4: By calculating the similarity between the text features described in step 2 and the visual features after the abnormal detail information enhancement described in step 3, it is determined whether the input image is abnormal and the pixel-level abnormality positioning result is obtained.

[0024] In step 1:

[0025] For the input dataset image x i First, the features are extracted through the pre-trained backbone network efficientnet-b4, and then f x It means that f x ∈R C×H×W, C, H, and W respectively represent the number of channels, height, and width of the feature;

[0026] The memory-based method sets up a reconstruction network. The reconstruction network represents features from both local and global perspectives. By combining the unique features from different perspectives, it provides a representation for the input, capturing fine-grained details and a broader overall structure;

[0027] By converting the feature into a low-dimensional coordinate and mapping it to a continuous grid, the input feature can be compared with the normal features stored in the grid, and then the difference position can be predicted; The way of storing in the grid enables the reconstruction network to effectively retain normal information and eliminate the anomalies existing between feature data.

[0028] Furthermore, specifically as follows:

[0029] Step (1): Local feature representation:

[0030] When converting the input feature f x into a local feature represented by low-dimensional coordinates, first, without changing the number of channels, the information of different channels is superimposed through 1D convolution so that the feature mapped to the coordinate representation can take into account the information between different channels. Then, through a convolutional layer with a kernel of 1 the number of channels is changed, and the original channels are mapped to n (n < c) dimensions to achieve coordinate conversion, that is, each pixel is converted into an n-dimensional coordinate value;

[0031]

[0032] where is the representation after converting the input feature f x into n-dimensional coordinates. Among them, i, j (i ∈ h, j ∈ w) represent the position of each pixel point in the feature f x , and T1, T represent the hyperbolic tangent activation by the tanh function;

[0033] Adopt the method of grid design. The grid G is trained as a function with infinitely resolvable coordinates, and the feature corresponding to the input coordinates in the output grid G is output;

[0034] The feature representation corresponding to each n-dimensional coordinate is stored in the grid; through the grid mapping function S(·), the feature corresponding to the n-dimensional coordinate in the grid G is extracted;

[0035] In this method, the n-dimensional coordinates represent the low-dimensional feature representation of each pixel, that is, the pixel point is mapped to n dimensions and understood as n-dimensional coordinates. The local grid G ​​stores the feature representation corresponding to each position in the grid (that is, each coordinate point). These features are automatically learned by the network during the training process, that is, each position has a vector representation, which represents the local features of the position. In summary, the feature of a pixel point, such as c-dimension, is transformed into an n-dimensional coordinate representation after coordinate transformation. When the n-dimensional coordinate is input into the grid mapping function S, the feature corresponding to the position will be found from the grid G ​​(the stored feature, such as C-dimension).

[0036] Step (2): Global feature representation:

[0037] For the input feature f x , through linear mapping, the number of channels is mapped to n, that is, an n-dimensional coordinate v representing the global feature n :

[0038]

[0039] Linear is a linear layer network, T a , T b is the tanh activation function;

[0040] For a learnable global grid Through the global coordinates v n Capture input features f x The global representation of x The information of each dimension in the grid is set to C. g =C*H*W, so that all information can be taken into account in sampling. The sampling process is as follows:

[0041]

[0042] f g =Re(f′ g )f g ∈(R C×H×W )

[0043]

[0044] After sampling, the obtained feature is expressed as f′ g , the shape is Through the Re function, the global feature representation f′ g Reshape to R C×H×W The global feature f g ;

[0045] The global feature representation of the final output not only effectively replaces the global anomaly, but also distinguishes the inter-class distribution under the same background;

[0046] Step (3): Fusion feature representation:

[0047] By combining the global and local feature representations, the normal features of the input image Image can be effectively learned. The global and local feature representations are connected through the concat(·) operation, and then the normal features are reconstructed through the continuous residual network Rec(·):

[0048] f(x)=Rec(concat(f l , f g ))

[0049] Among them, f l is the local feature of the input image, f g is the global feature of the input image.

[0050] During training, by comparing f(x) with the input feature f x The similarity of the two learnable grids is used to continuously learn, and the mean square error and cosine similarity are combined to evaluate the similarity between the two, and the final feature representation is refined so that it can accurately reconstruct normal features;

[0051] The normal feature representation is obtained through local and global grid sampling, and the reconstructed normal features are obtained through the convolutional network. The reconstructed normal features are compared with the input features to obtain the abnormal feature map pre, which is used as the abnormal feature predicted by the network:

[0052]

[0053] In step (1), when the input coordinates have no exact match in the grid G, the method of adjacent feature aggregation is used to interpolate based on the distance between the input feature coordinates and the coordinates of the adjacent features, and finally output the feature representation of the corresponding coordinates:

[0054] S(v, G) = |vn|G[m]+|vm|G[n]

[0055]

[0056] Where S(·) is the grid mapping function, v represents the coordinate, G represents the mapped grid, and m and n are the adjacent coordinates of v;

[0057] Setting up a learnable local grid pass In G l Find the value of the corresponding position in the grid, and use it as the representative of the pixel point. Finally, the feature representation that can represent the pixel point in the output grid is Right now:

[0058]

[0059] Where S(.,G) represents the sampling process from the grid G. The number of channels of the output feature representation is the number of channels of the features stored in the grid, which means that each channel in the grid is sampled. The local feature representation of the final output is defined as f l .

[0060] In the step 2, in the CLIP model, the text prompt template, the text encoder and the visual encoder work together to achieve cross-modal learning. The text prompt template is a natural language description input to the text encoder to express the semantic content of the image. The text encoder captures the semantic information of the text by converting these text prompts into vector representations; at the same time, the image is converted into a vector representation by the visual encoder to capture the visual features of the image. Finally, CLIP optimizes the outputs of the two encoders through contrastive learning, so that images and texts with the same semantics are closer in the same embedding space, achieving effective alignment between the image and the text.

[0061] Specifically:

[0062] Step (1): Design of text prompt template:

[0063] The designed text prompt template is as follows:

[0064] t n =[P1][P2]...[P E ][normal][object].

[0065] t a =[Q1][Q2]...[Q E ][c k ][object].

[0066] C=[c1,c2...c K ](c1:broken, c2:contamination...)

[0067] Among them, t n Represents a normal type of text template, t a A text template representing an exception type. C represents the exception type. [P i ],[Q i ](i∈1,...,E) represents the learnable word embeddings of normal and abnormal text templates;

[0068] Step (2): The text encoder extracts text features:

[0069] Learnable parameters are added to the text encoder of CLIP so that the model can learn distinguishable abnormal semantic information. The learnable parameters are set and added to the front side of the text prompt embedding and input into the encoder together with the text prompt embedding. After being processed by each layer of the encoder, the learnable text embedding is optimized. The specific description is as follows:

[0070] L m =[l1,l2,...,l M ]

[0071] [′ m+1 ,t m+1 ]=T m (cat((l m , t m ),dim=chanel))

[0072] [l′ m+2 , t m+2 ]=T m+1 (cat((l m+1 , t m+1 ),dim=chanel))

[0073] …

[0074] t M+1 =T M (t M )

[0075] Among them, L m represents the learnable parameters of the m-th layer text encoder, t m represents the learnable text embedding at the mth layer, l′ m+1 represents the output of the m-th layer of learnable parameters after the m-th layer of text encoder, T M represents the frozen M-th layer text encoder, and the cat operation aggregates the learnable parameters and text embedding along the channel dimension;

[0076] Finally, the text features representing normal and abnormal information are obtained through the text encoder of CLIP n ,t a , which is then used to calculate the similarity with CLIP visual features to obtain the loss;

[0077] Step (3): Visual encoder extracts features:

[0078] For the input dataset image x i , after the encoder, we finally get a token-like feature representing global semantic information and the output of multiple intermediate layers:

[0079] f i =F(xi ) i ∈R H×W×D

[0080]

[0081] Where F(·) represents the visual encoder, f i represents the global image features, represents the image features at the patch level of the intermediate layer, and |S| represents the set of intermediate layers of the extracted visual encoder;

[0082] The step 3 is specifically as follows:

[0083] The abnormal feature map pre output by the reconstruction module is rich in fine-grained information. According to the visual encoder characteristics of the cross-modal alignment network, a dual attention mechanism is designed to integrate the image features at the |S| intermediate layer patch level They are divided into two groups: high-level patch features and low-level patch features;

[0084] Step (1): Channel Attention

[0085] For high-level patch features The shape of the abnormal feature map pre is combined with the high-level patch features Align the shape of the abnormal feature map to simulate the high-level patch features The embedding operation passes through a convolutional layer of the same size and adds high-level patch features. The same position encoding is used to generate the embedding pre′ of the abnormal feature map pre, achieving

[0086]

[0087] pos is the position encoding of the high-level patch feature, conv is the convolution operation, and upsample represents the upsampling operation;

[0088] Define MultiHead(Q, K, V) as the following formula, where the weight matrix is ​​represented by W:

[0089] MultiHead(Q,K,V)=Concat(head1, head2,..., head h )W O

[0090]

[0091] W OIndicates the weight parameter when each head is merged into Concat in multi-head attention. i is the result of attention calculation, W Q , W K , W V They represent the attention weight parameters of the query Q, key K and value V input during attention calculation respectively;

[0092] Then the generated pre′ and Perform multi-head attention calculations and compare Perform residual connection to obtain features that focus on abnormal information Replace patch-level embedding Then similarity calculation is performed with text features;

[0093]

[0094] Step (2): Spatial Attention:

[0095] For low-level patch features Perform spatial attention to focus on the pixel information in the anomaly;

[0096] Take the spatial attention module in the CBAM attention module and define M C (F) is the spatial attention module function, where for the feature F input into the module, the output is:

[0097] M C (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0098] F is the input feature of the spatial attention module, MaxPool(·) is the maximum pooling operation, AgvPool(·) is the average pooling operation, MLP(·) is a multilayer perceptron, i.e., a feedforward neural network, and σ represents the sigmoid activation function;

[0099] Then, a two-dimensional spatial attention map is obtained, that is, the weight value of each feature point in the input feature map is obtained;

[0100] Setting low-level patch features To get close to the shape of the abnormal feature map pre, by Perform deconvolution and then restore the resolution of the feature map;

[0101] Make the low-level patch features Focus on the spatial information of pre and restore it to low-level patch features The shape of the output feature is finally obtained Output as a new low-level patch feature;

[0102]

[0103] Among them, upsample(·) is the upsampling operation, M C Get the attention weight value for the spatial attention operation, deconv(·) is the deconvolution operation, represents the matrix multiplication operation, conv2(·) is the convolution operation;

[0104] All visual features in CLIP When attention is performed separately, when the number of channel attention and the number of spatial attention are the same, the comprehensive detection is the best, that is,

[0105]

[0106] Combining these two attention mechanisms, the network can better capture complex image features and improve the expressiveness and performance of the model.

[0107] The step 4 is specifically as follows:

[0108] For loss calculation, visual embedding and text embedding are aligned from global and local perspectives, that is, learnable text cues are optimized from two perspectives, and global and local loss functions are designed to adjust the trainable parameters of the model.

[0109]

[0110] Among them, L global is the cross entropy loss, label is the true abnormal type label of the image, M gt ∈R H×W represents the true mask, I represents the all-1 matrix, t [n，a] represents text features, n and a represent normal and abnormal tags respectively, represents the patch image feature after attention fusion with the abnormal feature map pre; Focal and Dice are two loss functions, P(·) represents the calculation of abnormal probability score, and λ is a hyperparameter that balances global and local losses;

[0111] During the training process, after the loss is calculated, the backpropagation operation is performed, that is, the gradient of each model parameter with respect to the loss function is calculated by the chain rule, the parameters are updated according to these gradients, and the gradient descent optimization algorithm is used to minimize the loss function.

[0112] Beneficial effects of the present invention:

[0113] First: The framework proposed in this paper significantly improves the ability to capture fine-grained anomaly information by integrating reconstruction guidance strategy and cross-modal alignment network, thus overcoming the shortcomings of existing cross-modal alignment network-based methods in anomaly location. The cross-modal alignment network and reconstruction module in the framework have good scalability, and any pre-trained cross-modal alignment network and reconstruction network can be directly transplanted into the framework without retraining.

[0114] Specifically, the reconstruction module in the present invention can predict the abnormal information of the input sample, which is then used to guide the cross-modal alignment network so that it can focus more on identifying subtle abnormal features. This method not only takes advantage of the generalization advantage of the cross-modal alignment network, but also realizes a unified model that can be widely used in anomaly detection, localization and classification tasks in the industrial field. The model does not need to be trained separately for each anomaly category, which greatly simplifies the training process and improves the applicability of the model. The model proposed in the present invention only needs to be trained on the auxiliary training set, without fine-tuning, and can adapt to any industrial data set for effective anomaly detection and localization, with good generalization.

[0115] Second: This paper adopts a dual attention mechanism to achieve reconstruction guidance. This method combines the two strategies of spatial attention and channel attention. By carefully considering the different characteristics of the extracted patch-level features and using the corresponding attention allocation strategy, the model's attention to various abnormal details is enhanced. This means that the model can not only pay attention to basic features such as simple texture and color, but also conduct in-depth analysis of higher-level features.

[0116] Third: The present invention has been extensively tested on data sets of multiple real industrial scenarios, including MVTecAD, VisA, and MPDD. Compared with existing models, the model of the present invention has demonstrated excellent performance. In supervised and small sample learning tasks, the model of the present invention can not only effectively complete anomaly detection, but also identify anomaly types with an accuracy rate of over 85%. In zero-sample learning tasks, especially on the MPDD dataset, the model's performance indicators for anomaly detection, the average I-AUROC and P-AUROC reached 78.6% and 92.6% respectively, surpassing other advanced anomaly detection models. These results demonstrate the effectiveness and superiority of the model in the field of industrial anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1 It is a schematic diagram of the overall structure of the model of the present invention.

[0118] Figure 2 It is a schematic diagram of the reconstruction module structure of the present invention.

[0119] Figure 3Schematic diagram of the cross-modal alignment network of the present invention.

[0120] Figure 4 It is a schematic diagram of the structure of the dual attention module of the present invention.

[0121] Figure 5 It is a schematic diagram of the anomaly detection results of the present invention with different amounts of channel attention and spatial attention allocated.

[0122] Figure 6 Schematic diagram comparing the performance of different attention settings of the model.

[0123] Figure 7 This is a schematic diagram of the classification effects of VisA and MPDD in the 0-shot experiment.

[0124] Figure 8 This is a schematic diagram of the anomaly classification effect of the VisA dataset in the 2-shot experiment. DETAILED DESCRIPTION

[0125] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0126] like Figure 1 As shown, the present invention discloses a small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment, which can achieve accurate anomaly detection and positioning, realize anomaly detection and classification based on the CLIP model, and introduce a reconstruction network, which is used to guide the model to focus on abnormal details.

[0127] First, we designed multiple types of learnable object-agnostic text prompt templates in the text encoder of CLIP, and listed all the abnormal types covering the industrial field in the template of abnormal prompts as much as possible. We added additional learnable parameters to the text encoder of CLIP to achieve fine-tuning. At the same time, we extracted multiple intermediate layer outputs of the visual encoder to align the visual features with the text features from a global and local perspective.

[0128] Secondly, in order to make the cross-modal alignment network pay more attention to abnormal details, a powerful reconstruction network is trained to calculate the difference between the image before and after reconstruction to obtain the predicted global abnormal features, which are input into the cross-modal alignment network to guide it to focus on abnormal information.

[0129] Finally, a dual attention fusion module fuses abnormal features with local visual features through spatial attention and channel attention. The fused features are compared with the text hint embedding for learning, ultimately achieving accurate prediction, classification and positioning.

[0130] Step 1: The reconstruction module extracts preliminary anomaly information: Aiming at the problem that the cross-modal alignment network (CLIP) is not sensitive to detailed texture information, this application proposes to use a reconstruction module to guide CLIP to better perceive anomaly details. The reconstruction module extracts preliminary anomaly detail information by calculating the difference between the input image and the reconstruction output. This anomaly detail information serves as the query key Q and performs attention calculation with the CLIP visual features to highlight the abnormal parts in the visual features. Introducing the reconstruction module can effectively guide CLIP to focus on detailed anomalies, thereby improving its accuracy in the anomaly localization task.

[0131] The reconstruction network can perceive normal information but cannot reconstruct abnormal information. Based on this characteristic, the input features can be compared with the reconstructed normal features of the output, and anomalies can be detected according to the differences before and after reconstruction. Figure 2 This is the structure of the reconstruction module in the technical framework of the present invention.

[0132] To implement a multi-class unified model, a memory-based method is used. By converting normal features into low-dimensional coordinates and mapping them to a continuous grid, the input features can be compared with the normal features stored in the grid, and then the difference positions can be predicted. The grid storage method enables this reconstruction module to effectively retain normal information and eliminate any anomalies existing between feature data.

[0133] For the input dataset image x i First, it passes through the pre-trained backbone network efficientnet-b4 to extract features, and the features are represented by f x where f x ∈R C×H×W , and C, H, and W represent the number of channels, height, and width of the features respectively.

[0134] From both local and global perspectives, by combining unique features from different perspectives, fine-grained details and a broader overall structure are captured.

[0135] (1) Local feature representation:

[0136] When converting the input feature f x into a local feature represented by low-dimensional coordinates, first, without changing the number of channels, the information of different channels is superimposed through a 1D convolution so that the feature mapped to the coordinate representation can take into account the information between different channels. Then, through a convolutional layer with a kernel of 1 the number of channels is changed, and the original channels are mapped to n (n < c) dimensions to achieve coordinate conversion, that is, each pixel is converted into an n-dimensional coordinate value.

[0137]

[0138] Among them is the input feature f x The representation after conversion of n-dimensional coordinates, where i, j (i∈h, j∈w) represents feature f x At each pixel position in , T1, T represents the hyperbolic tangent activation of the tanh function;

[0139] In order to ensure that the features output by the reconstruction network can retain all normal information and replace the input abnormal information with similar contextual normal features in the local environment, a grid design method is adopted. The grid is trained as a function with infinite resolution coordinates. When the input coordinates do not have an exact match in the grid, the neighboring feature aggregation method is used to interpolate based on the distance between the input feature coordinates and the coordinates of the neighboring features, and finally output the feature representation of the corresponding coordinates:

[0140] S(v,G)=|vn|G[m]+|vm|G[n]

[0141]

[0142] Where S(·) is the grid mapping function, v represents the coordinate, G represents the mapped grid, and m and n are the adjacent coordinates of v.

[0143] Setting up a learnable local grid pass In G l Find the value of the corresponding position in the grid and use it as the representative of the pixel. From another perspective, this operation is to store the pixel features in a continuous grid. Using pixel-level coordinates, normal features can be sampled from the local grid, and finally the feature representation that can represent the pixel in the output grid is Right now:

[0144]

[0145] Where S(.,G) represents the sampling process from the grid G, and the number of channels of the output feature representation is the number of channels of the grid, which means that each channel in the grid is sampled, so there is a more accurate representation. The local feature representation of the final output is defined as f l .

[0146] (2) Global feature representation:

[0147] For the input feature f x , through linear mapping, the number of channels is mapped to n, that is, an n-dimensional coordinate v representing the global feature n :

[0148]

[0149] Linear is a linear layer network, Ta 、T b is the tanh activation function.

[0150] For a learnable global grid Through the global coordinates v n Capture input features f x The global representation of x The information of each dimension in the grid is set to C. g =C*H*W, so that all information can be taken into account in sampling. The sampling process is as follows:

[0151]

[0152] f g =Re(f′ g )f g ∈(R C×H×W )

[0153]

[0154] After sampling, the obtained feature is expressed as f′ g , the shape is Through the Re function, the global feature representation f′ g Reshape to R C×H×W The global feature f g .

[0155] The final output global feature representation not only effectively replaces the global anomaly, but also distinguishes the inter-class distribution under the same background.

[0156] (3) Fusion feature representation:

[0157] By combining the global and local feature representations, the normal features of the input image Image can be effectively learned. The global and local feature representations are connected through the concat(·) operation, and then the normal features are reconstructed through the continuous residual network Rec(·):

[0158] f(x)=Rec(concat(f l , f g ))

[0159] Among them, f l is the local feature of the input image, f g is the global feature of the input image.

[0160] During training, by comparing f(x) with the input feature f xThe similarity of the two learnable grids is used to continuously learn, and the mean square error and cosine similarity are combined to evaluate the similarity between the two, and the final feature representation is refined so that it can accurately reconstruct normal features. When this method is actually used, the pre-trained reconstruction module parameters are used, and no retraining is required.

[0161] Therefore, in this application, normal feature representation is obtained by local and global grid sampling, and reconstructed normal features are obtained through convolutional network. The reconstructed normal features are compared with the input features to obtain the abnormal feature map pre as the abnormal features predicted by the network:

[0162]

[0163] Through the setting of this module, both global and local levels are taken into consideration, and with the characteristics of continuous feature storage, fine-grained abnormal information can be predicted.

[0164] Step 2: Fine-tune the cross-modal alignment network to extract text and visual features: CLIP has shown excellent performance in classification tasks through contrastive cue learning strategies. This strategy is applied to the anomaly detection task by adjusting CLIP's text cues to enable it to identify anomaly types and align these cues with image features.

[0165] The cross-modal alignment network structure diagram in the technical framework of the present invention is as follows: Figure 3 As shown. The left figure (a) is the visual encoder of the CLIP model, which is implemented by the ViT network, and the right figure (b) is the text encoder of the CLIP model, which is composed of a 12-layer transformer encoder network. In order to ensure the model's ability to detect zero-shot anomalies, this application avoids using the specific category information of the image as a prompt, but instead uses multiple general and image-independent texts as prompts. In order to learn these general text prompt templates, global and local context optimization techniques are introduced, which helps to integrate abnormal semantics of different granularities into object-agnostic text embeddings. In addition, by aggregating multi-layer visual intermediate layer features, richer local detail information can be provided, thereby enhancing the model's ability to recognize anomalies. In the training phase, all modules are jointly optimized by combining global and local context optimization methods. In the inference phase, the misalignment between the text and the global and local visual embeddings is quantified to determine the predicted anomaly type, calculate the anomaly score, and generate an anomaly map. In the loss calculation part, the anomaly type predicted by the model is compared with the true label to calculate the cross entropy loss, and the anomaly map predicted by the model is compared with the true mask to calculate the Focal and Dice losses, and the model parameters are optimized by Adam.

[0166] Step (1): Text hint template design: CLIP's text hint template is used to provide semantic information in the cross-modal alignment task, so that the model can understand and match the association between the image and the text. In the present invention, for the anomaly detection and classification tasks, a specific anomaly classification text template is designed to describe different anomaly types so that the image can be matched with the template to obtain the anomaly classification result. For the pixel-level anomaly localization task, the hint provided by the text template is encoded, and the similarity is calculated with the pixel-level features to predict whether each pixel is abnormal, and finally generate an anomaly segmentation mask.

[0167] For anomaly detection and classification, the goal is to perceive abnormal information, so it is necessary to incorporate prior abnormal semantic information into the prompt template. In order to do classification, all abnormal category information in the industry, such as scratches, holes, etc., is listed as much as possible in the abnormal text prompt template. At the same time, since the CLIP model focuses on the alignment between visual features and object semantic features, rather than abnormal and normal semantics in the image, it is difficult to distinguish abnormalities by simply using [A photo of a [cls] with scratches] for the design of abnormal type prompt templates, such as scratches. A learnable text prompt template is used, such as [{cls} with scratches] as a scratch class prompt, and a learnable token is added before the template to ensure that the text embedding can cover a wider range of semantic information during the training process. Considering the zero-shot detection task, it is hoped that the model can maintain good performance on unseen categories. The same approach as the existing CLIP-based anomaly detection method is adopted, using object-independent text prompts, replacing {cls} in the prompt template with {object}, and focusing on normal and abnormal patterns in the image without considering object semantics. The text embeddings generated by such templates are more general and capable of identifying anomalies across different objects and domains.

[0168] The designed text prompt template is as follows:

[0169] t n =[P1][P2]...[P E ][normal][object].

[0170] t a =[Q1][Q2]...[Q E ][c k ][object].

[0171] C=[c1,c2...c K ](c1: broken, c2: contamination...)

[0172] Among them, tn Represents a normal type of text template, t a A text template representing an exception type. C represents the exception type. [P i ],[Q i ](i∈1,...,E) represents the learnable word embeddings of normal and abnormal text templates.

[0173] Step (2): The text encoder extracts text features. The main function of the CLIP text encoder is to convert text prompt information into a vector representation that can be aligned with visual features, so that effective cross-modal learning can be performed between images and text. During the training process, the text features are continuously aligned with the CLIP visual features to optimize the trainable parameters. Through this cross-modal comparative learning, the network model can further learn the knowledge between text semantics while understanding the visual level.

[0174] Learnable parameters are added to the text encoder of CLIP, so that the model can learn distinguishable abnormal semantic information. The learnable parameters are set and added to the front side of the text prompt embedding, and are input into the encoder together with the text prompt embedding. After being processed by each layer of the encoder, the learnable text embedding is optimized. The specific description is as follows:

[0175] L m =[l1,l2,...,l M ]

[0176] [l′ m+1 ,t m+1 ]=T m (cat((l m , t m ),dim=chanel))

[0177] [l′ m+2 ,t m+2 ]=T m+1 (cat((l m+1 , t m+1 ), dim = chanel)

[0178] …

[0179] t M+1 =T M (t M )

[0180] Among them, L m represents the learnable parameters of the m-th layer text encoder, t m represents the learnable text embedding at the mth layer, l′ m+1 represents the output of the m-th layer of learnable parameters after the m-th layer of text encoder, T MRepresents the frozen M-th layer text encoder, and the cat operation aggregates the learnable parameters with the text embedding along the channel dimension.

[0181] Finally, the text features representing normal and abnormal information are obtained through the CLIP text encoder n ,t a , which is then used to calculate the similarity with CLIP visual features to obtain the loss;

[0182] Step (3): Visual encoder extracts features:

[0183] CLIP's visual encoder mainly extracts key visual features from images and converts them into a unified feature space vector, and cooperates with the text encoder to achieve comparative learning between images and text. The visual encoder can enhance the model's cross-modal alignment and visual understanding capabilities, enabling CLIP to perform well in complex visual tasks.

[0184] Since it is difficult to identify some minor anomalies with global image information, the output of multiple intermediate layers of the visual encoder, namely patch-level image features, are extracted to supplement the image detail information;

[0185] For the input dataset image x i , after the encoder, we finally get a token-like feature representing global semantic information and the output of multiple intermediate layers:

[0186] f i =F(x i ) i ∈R H×W×D

[0187]

[0188] Where F(·) represents the visual encoder, f i represents the global image features, represents the image features at the intermediate layer patch level, and |S| represents the set of extracted intermediate layers of the visual encoder.

[0189] Step 3: Dual attention fusion of visual features and preliminary anomaly information:

[0190] The abnormal feature map pre output by the reconstruction module is rich in fine-grained information. To address the problem of insufficient detail recognition of CLIP, a reconstruction module is proposed to adjust the patch-level image features of the cross-modal alignment network. By performing dual attention calculations on the patch-level image features and the reconstruction features, the attention of the patch-level image features to abnormal details is enhanced. The dual attention module structure in the technical model of the present invention is as follows: Figure 4 shown.

[0191] According to the visual encoder characteristics of the cross-modal alignment network, a dual attention mechanism is designed to integrate the image features at the |S| intermediate layer patch level They are divided into two groups: high-level patch features and low-level patch features;

[0192] In the design of the present invention, |S| is 4, and the spatial attention and channel attention mechanisms are implemented for the outputs of the low-level and high-level encoders respectively, which significantly improves the sensitivity of the CLIP model to abnormal details.

[0193] Step (1): Channel Attention

[0194] For channel attention, when the input image features of the CLIP visual encoder pass through multiple Transformer encoder layers of the CLIP visual encoder (ViT), the information of the spatial dimension is gradually abstracted. ViT gradually captures higher-level feature representations. These features gradually transform from simple textures and color patterns to more complex shapes, object parts, and scene information.

[0195] Therefore, high-level patch features are needed Pay attention to abnormal information through different channels.

[0196] High-level patch features Contains a large number of channels and has more complex semantic information, so it is necessary to combine the shape of the abnormal feature map pre with the high-level patch features The shape of the patch is aligned to ensure that the high-level patch features are not changed. In the case of information, we increase the attention to abnormal content, so the abnormal feature map is pre-simulated with high-level patch features The embedding operation passes through a convolutional layer of the same size and adds high-level patch features. The same position encoding is used to generate the embedding pre′ of the abnormal feature map pre, achieving

[0197]

[0198] pos is the position encoding of the high-level patch feature, conv is the convolution operation, and upsample represents the upsampling operation.

[0199] Define MultiHead(Q, K, V) as the following formula, where the weight matrix is ​​represented by W:

[0200] MultiHead(Q,K,V)=Concat(head1,head2,...,head h)W O

[0201]

[0202] W O Indicates the weight parameter when each head is merged into Concat in multi-head attention. i is the result of attention calculation, W Q , W K , W V They respectively represent the attention weight parameters of the query Q, key K and value V input during attention calculation.

[0203] Then the generated pre′ and Perform multi-head attention calculations and compare Perform residual connection to obtain features that focus on abnormal information Replace patch-level embedding Then similarity calculation is performed with text features;

[0204]

[0205] Step (2): Spatial Attention

[0206] For spatial attention, in anomaly detection and classification, it is necessary to capture the spatial region in the image, that is, pixel information. The Transformer encoder layer of the ViT network divides the original image into multiple fixed-size patches, each of which contains all the color channel information of the corresponding region in the original image. As more and more encoder layers are passed, the spatial position information of each patch may be lost, while the lower layers of the network tend to capture low-level features and contain relatively more spatial information. Therefore, it is suitable for low-level patch features. Perform spatial attention to focus on the pixel information in the anomaly.

[0207] The spatial attention module in the CBAM attention module focuses on the spatial information of the input image and is a supplement to the channel attention.

[0208] Definition M C (F) is the spatial attention module function, where for the feature F input into the module, the output is:

[0209] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0210] F is the input feature of the spatial attention module, MaxPool(·) is the maximum pooling operation, AgvPool(·) is the average pooling operation, MLP(·) is a multilayer perceptron, i.e., a feedforward neural network, and σ represents the sigmoid activation function.

[0211] Then, a two-dimensional spatial attention map is obtained, that is, the weight value of each feature point in the input feature map is obtained;

[0212] Because the anomaly feature map pre obtained from the reconstruction module contains richer and more accurate pixel information, while the low-level patch features is the result of segmentation by the ViT network, so the low-level patch features are set To get close to the shape of the abnormal feature map pre, by Deconvolution is performed to restore the resolution of the feature map. Deconvolution helps to reconstruct the lost high-frequency details and can be regarded as a sparse connection operation. Compared with the fully connected layer, it reduces the number of parameters and the amount of calculation of the model. At the same time, the abnormal feature map pre is upsampled and compared with the low-level patch feature Perform dimension matching.

[0213] In the design of the present invention, it is necessary to make the low-level patch features Focus on the spatial information of pre and restore it to low-level patch features The shape of the output feature is finally obtained Output as a new low-level patch feature;

[0214]

[0215] Among them, upsample(·) is the upsampling operation, M C Get the attention weight value for the spatial attention operation, deconv(·) is the deconvolution operation, represents the matrix multiplication operation, and conv2(·) is the convolution operation.

[0216] For the allocation of channel attention and spatial attention, there are |S| patch-level image features As reference Figure 5 As shown, in this design, when |S|=4, the ratio of channel attention to spatial attention is ca:sa. Experiments have shown that Figure 5 It can be seen that when there is too much channel attention, it is easy to ignore pixel factors and thus misdetection; when there is too much spatial attention, too many pixel values ​​are paid attention to, making the perception of abnormal pixel values ​​more acute, without considering other features, resulting in misdetection.

[0217] Therefore, in CLIP all visual features When attention is performed separately, when the number of channel attention and the number of spatial attention are the same, the comprehensive detection is the best, that is,

[0218]

[0219] Channel attention can enhance the feature relationship between different channels and improve the distinguishing ability of feature representation, while spatial attention focuses on key information at different positions in space, effectively making up for the shortcomings of traditional convolutional neural networks in processing multi-scale, shape changes and direction information. By combining these two attention mechanisms, the network can better capture complex image features and improve the expression ability and performance of the model.

[0220] Step 4: Loss calculation:

[0221] For loss calculation, visual embedding and text embedding are aligned from global and local perspectives, that is, learnable text cues are optimized from two perspectives, and global and local loss functions are designed to adjust the trainable parameters of the model.

[0222]

[0223] Among them, L global is the cross entropy loss, label is the true abnormal type label of the image, M gt ∈R H×W represents the true mask, I represents the all-1 matrix, t [n，a] represents text features, n and a represent normal and abnormal tags respectively, represents the patch image feature after attention fusion with the abnormal feature map pre; Focal and Dice are two loss functions, P(·) represents the calculation of abnormal probability score, and λ is a hyperparameter that balances global and local losses.

[0224] During the training process, after the loss is calculated, the back-propagation operation is performed, that is, the gradient of each model parameter to the loss function is calculated by the chain rule, the parameters are updated according to these gradients, and the gradient descent optimization algorithm is used to minimize the loss function. This process makes the model's predictions gradually approach the true value, thereby improving the performance of the model.

[0225] By comparing the classification results and positioning information predicted by the model with the actual labels and masks, calculating the loss function, and using the Adam optimizer to adjust the model parameters, the model performance is gradually optimized, ultimately achieving accurate detection, precise positioning, and classification of anomalies.

[0226] experiment:

[0227] Dataset and evaluation metrics:

[0228] (1) Datasets: In all experiments based on the reconstruction-guided cross-modal alignment anomaly detection framework, three datasets are used: MVTec-AD, VisA, and MPDD. In the training and testing process of each experiment, the corresponding dataset used is divided into a training set and a test set in a ratio of 70% and 30%.

[0229] In the experiment, we selected 1725 images from the MVTec-AD test dataset, which contains normal samples and abnormal samples. We expanded the abnormal samples in these 1725 images to about 5200 abnormal images by rotation, noise addition, movement, etc. We took 30% of the abnormal samples and combined them with 3624 normal samples in the MVTec training set to form 5420 training and test samples, where the ratio of normal to abnormal samples is about 2.5 / 1. For the division of training and test sets, we divided 80% of the 5420 images containing normal and abnormal samples into training sets (4336) and 20% as test sets (1024).

[0230] In addition, we conducted anomaly detection and anomaly type classification tests on the 1024 MVTec-AD test set, and performed zero-shot reasoning on the MPDD and VisA datasets to evaluate the generalization of our model. At the same time, we added experiments, using 4336 samples from the MVTec-AD training set and a small amount of MPDD and VisA data as model training sets, and conducted small-sample experiments on the MPDD and VisA datasets.

[0231] (2) Evaluation indicators: Based on previous work, the indicators I-AUROC and P-AUROC are used. In order to further verify the superiority of the proposed method, the average precision (AP) in anomaly detection and anomaly localization is added as an evaluation indicator. For anomaly classification, the accuracy rate is used as a direct performance indicator, which indicates the proportion of anomalies correctly classified by the model to the total number of anomalies.

[0232] Implementation details:

[0233] Using the pre-trained CLIP network, we added learnable parameters to its text encoder for fine-tuning. For the reconstruction module, we adopted the pre-trained reconstruction module and obtained a trained model by pre-training on the 3629 training sets in MVTec-AD to meet the experimental requirements of this article.

[0234] The input image is resized to a resolution of 518×518. For the CLIP network, the outputs of the 6th, 12th, 18th, and 24th layers of its visual encoder are extracted as patch-level image embeddings. For the reconstruction module, the input image is firstly subjected to feature extraction, and the pre-trained EfficientNet-B4 network is used as the backbone to extract features. The features extracted from the 3rd and 4th layers are used as output features for feature reconstruction.

[0235] The experiment was conducted based on a single-card NVIDIA GeForce RTX 3090 server with 24 GB of video memory, 2 batches, 18 training rounds, and two learning rates of 9×e(-4) and 1×e(-4) for the text and vision modules, respectively.

[0236] Performance Analysis:

[0237] The model of this application can take into account the three functions of anomaly detection, positioning and classification at the same time. For the classification of anomaly types, this application sets 43 anomaly types of different industrial products in common real industrial scenarios. For samples known or unknown to the model, it has good performance in anomaly detection, positioning and classification. That is, only one model can realize anomaly detection, anomaly positioning and anomaly classification at the same time, and up to 43 types of anomalies can be intercepted. The performance of the model is explained below through specific indicators.

[0238] (1) 0-shot anomaly detection

[0239] The divided MVTec dataset is used for auxiliary training, and MPDD and VisA are used as target datasets for detection. At the same time, the method in this paper is compared with CLIP-based methods (AnomalyCLIP, CLIP, CLIP-AC, WinCLIP, etc.). Under the same settings, the text is changed to a consistent prompt template describing 44 abnormal categories for abnormal classification. The comparison is performed by calculating the mean index of all categories in the dataset. The comparison results of the VisA dataset are shown in Table 1, and the comparison results of the MPDD dataset are shown in Table 2.

[0240] Table 1: Comparison of detection results on the VisA dataset on the ZSAD task

[0241]

[0242] Table 2: Comparison of detection results on the MPDD dataset on the ZSAD task

[0243]

[0244] The comparison results show that for the VisA dataset, the performance of the model proposed in this application on the ZSAD task has reached the detection effect of the most advanced model algorithm in the current same task type while taking into account the classification of anomaly types. At the same time, on the more challenging MPDD dataset, the average I-AUROC and AP indicators of all categories in the dataset reached 78.6% and 84.1%, surpassing other algorithms.

[0245] (2) 2-shot

[0246] This application selects two challenging datasets, MPDD and VisA, for 2-shot small sample testing. The detection results of the VisA dataset are shown in Table 3, and the detection results of the MPDD dataset are shown in Table 4.

[0247] Table 3: 2-shot detection results of VisA dataset

[0248]

[0249] Table 4: 2-shot detection results on the MPDD dataset

[0250]

[0251] The experimental results show that in the VisA dataset, the overall performance in anomaly detection and localization is good, with more than 3 categories having anomaly detection indicators higher than 90%, and all categories having anomaly localization indicators higher than 90%, showing outstanding performance in anomaly localization; for the MPDD dataset, more than four categories have performance indicators higher than 90% in the 2-shot anomaly detection task, and the I-AUROC of the metalplate and tubes categories is close to 100%, which is rare in 2-shot tasks. In addition, for the P-AUROC indicator of anomaly localization, the model in this paper also shows superior performance on the MPDD dataset.

[0252] (3) 4-shot

[0253] This application selects two challenging datasets, MPDD and VisA, for 4-shot small sample testing. The detection results of the VisA dataset are shown in Table 5, and the detection results of the MPDD dataset are shown in Table 6.

[0254] Table 5: 4-shot detection results of VisA dataset

[0255]

[0256] Table 6: 4-shot detection results on the MPDD dataset

[0257]

[0258] The experimental results show that after increasing the sample size, in the VisA dataset, the overall anomaly detection and localization effects are good, among which more than 4 categories have anomaly detection indicators higher than 90%, and the indicators of all categories in anomaly localization are higher than 90%, showing outstanding performance in anomaly localization; for the MPDD dataset, more than four categories have performance indicators higher than 90% in the 2-shot anomaly detection task, and have obvious advantages in both detection and localization indicators.

[0259] This application uses the image anomaly information calculated by the reconstruction network to guide CLIP to perform anomaly detection. Therefore, ablation analysis is mainly performed on the attention fusion module in the guiding ideology to verify the effectiveness of the guiding ideology of this article.

[0260] The number of channel and spatial attention modules is set, and whether the image-level embedding adds attention modules is analyzed. The training is based on the divided MVTec-AD dataset, and the test is based on the divided MVTec-AD dataset. When other settings are exactly the same, the comparison is as follows Figure 6 shown.

[0261] Figure 6 In the figure, different ratios of spatial attention and channel attention are calculated for the patch-level image features output by the four visual encoders under the same other settings. By comparing the means of the two most representative indicators, I-AUROC and P-AUROC, in the test set, the overall performance test results of anomaly detection and localization on the same dataset are observed. The final experimental results show that when the model only performs two channel attentions and two spatial attentions on the patch-level image features output by the image encoder, the overall effect is the most outstanding.

[0262] Visualization of model detection effect:

[0263] For the model whose training set is 4336 images of MVTec-AD, we use 1024 image samples divided by MVTec-AD for testing, detect and segment the anomalies in the samples, and output the anomaly category and the corresponding score value. In the 0-shot task, the test results of the model of this application are as follows: Figure 7 shown.

[0264] In order to more comprehensively evaluate the performance of the model, we use 4336 images in the MVTec-AD dataset as the training set, and add a small number of samples from the VisA and MPDD datasets to assist in the training of the classification task, so as to conduct a small sample task test. In the detection process, only the VisA and MPDD datasets are used for detection. In the 2-shot task, the model in this paper can still classify different types of anomalies. The specific test results are as follows: Figure 8 shown.

[0265] from Figure 8 The data in the paper show that the model can still accurately judge and locate anomalies in small sample tasks, and has good performance in classification tasks.

[0266] This paper proposes a zero-shot / small-shot anomaly detection framework with reconstruction-guided cross-modal alignment. The reconstruction module and cross-modal alignment network in the framework can be extended and replaced by a pre-trained network of the same type. At the same time, since the prompt text of the anomaly type is introduced into the text encoder of the cross-modal alignment network, the model can classify the anomaly type.

[0267] In order to improve the perception of abnormal details by the cross-modal alignment network, the present invention introduces a reconstruction network to guide it. The difference between its output and input is obtained by the reconstruction network and used as abnormal information. The abnormal information removes the background and is used to guide the patch-level image features extracted by the cross-modal alignment network to pay more attention to the abnormalities.

[0268] The present invention designs spatial attention and channel attention in the attention module, so that the patch-level image features of the cross-modal alignment network can pay attention to abnormal information from two perspectives. For low-level patch-level image features in the cross-modal alignment network, spatial attention can perceive abnormalities from aspects such as color and texture; for high-level patch-level image features in the cross-modal alignment network, channel attention can perceive abnormalities from aspects such as semantics. Through the dual attention mechanism, the cross-modal alignment network's perception of fine-grained abnormalities is enhanced, and abnormalities can be accurately located.

[0269] In addition, the present invention designs 43 types of abnormal text prompts (such as scratches, holes, etc.) commonly seen in the industrial field in the text encoder branch of the cross-modal alignment network. Finally, the model of the present invention can recognize and classify up to 43 different abnormal categories, which is very valuable in industrial applications because it can provide more accurate fault diagnosis and preventive maintenance. In this way, the present invention brings a powerful and flexible solution to the field of industrial automation and quality control.

Claims

1. A small sample anomaly detection and classification framework based on reconstruction-guided cross-modal alignment, characterized by: Including cross-modal alignment network (CLIP), reconstruction network and dual attention fusion module; The reconstruction network takes the normal or abnormal RGB image in the data set as input, reconstructs the image to obtain a completely normal image with the same size as the input, and is used to calculate the difference between the image before and after reconstruction to obtain the predicted global abnormal features; The cross-modal alignment network includes a visual encoder and a text encoder, the text encoder extracts text features according to text prompts, the visual encoder extracts visual features from normal or abnormal RGB images in the data set, and the text encoder adds additional learnable parameters for fine-tuning; The dual attention fusion module performs attention fusion on the abnormal information and the visual features, and outputs the visual features with enhanced abnormal detail information; Through the similarity between the text features and the visual features enhanced by the abnormal detail information, it is determined whether the input image is abnormal and a pixel-level abnormality positioning result is obtained.

2. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to the framework of claim 1 is characterized in that: The steps include: Step 1: Use the reconstruction network to reconstruct the dataset image into a completely normal image. The reconstruction network outputs consistent results for normal images, and reconstructs abnormal areas into normal content for abnormal images through neighborhood features. The output and input of the reconstruction network are compared to determine the abnormal information in the input image; Step 2: Using the visual encoder and text encoder branches, the text encoder and visual encoder are used to extract text features and visual features respectively. The text encoder adds additional learnable parameters for fine-tuning. Step 3: Attentionally fuse the abnormal information and the visual features, and output visual features with enhanced abnormal detail information; Step 4: By calculating the similarity between the text features described in step 2 and the visual features after the abnormal detail information enhancement described in step 3, it is determined whether the input image is abnormal and the pixel-level abnormality positioning result is obtained.

3. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 2 is characterized in that: In step 1, for the input dataset image x i First, the features are extracted through the pre-trained backbone network efficientnet-b4, and then f x It means that f x ∈R C×H×W , C, H, and W represent the number of channels, height, and width of the feature, respectively; The memory-based method sets up a reconstruction network, which takes local and global perspectives to represent features. By combining features from different perspectives, it provides a representation for the input, capturing fine-grained details and broader overall structures. By converting features into low-dimensional coordinates and mapping them to a continuous grid, the input features can be compared with the normal features stored in the grid to predict the difference location; grid storage enables the reconstruction network to effectively retain normal information and eliminate anomalies between feature data.

4. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 3 is characterized in that: Specifically: Step (1): Local feature representation: When converting the input feature f x into a local feature in a low-dimensional coordinate representation, first, without changing the number of channels, the information of different channels is superimposed through 1D convolution so that the feature mapped to the coordinate representation can take into account the information between different channels. Then, through a convolutional layer with a kernel of 1 the number of channels is changed, and the original channels are mapped to n (n < c) dimensions to achieve coordinate conversion, that is, each pixel is converted into an n-dimensional coordinate value; in is the input feature f x The representation after conversion of n-dimensional coordinates, where i, j (i∈h, j∈w) represents feature f x At each pixel position in , T1, T represents the hyperbolic tangent activation of the tanh function; Using the grid design method, the grid G ​​is trained as a function of infinite resolution coordinates, and the features in the grid G ​​corresponding to the input coordinates are output; The grid stores the feature representation corresponding to each n-dimensional coordinate. The feature corresponding to the n-dimensional coordinate in the grid G ​​is extracted through the grid mapping function S(·). Step (2): Global feature representation: For the input feature f x , through linear mapping, the number of channels is mapped to n, that is, an n-dimensional coordinate v representing the global feature n : Among them, Linear is a linear layer network, T a , T b is the tanh activation function; For a learnable global grid Through the global coordinates v n Capture input features f x The global representation of x The information of each dimension in the grid is set to C. g =C*H*W, so that all information can be taken into account in sampling. The sampling process is as follows: f g =Re(f′ g )f g ∈(R C×H×W ) After sampling, the obtained feature is expressed as f′ g , the shape is Through the Re function, the global feature representation f′ g Reshape to R C×H×W The global feature f g ; The global feature representation of the final output not only effectively replaces the global anomaly, but also distinguishes the inter-class distribution under the same background; Step (3): Fusion feature representation: By combining the global and local feature representations, the normal features of the input image Image can be effectively learned. The global and local feature representations are connected through the concat(·) operation, and then the normal features are reconstructed through the continuous residual network Rec(·): f(x)=Rec(concat(f l ,f g )) Among them, f l is the local feature of the input image, f g is the global feature of the input image.

5. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 4 is characterized in that: When the input coordinates do not have an exact match in the grid, the neighboring feature aggregation method is used to interpolate based on the distance between the input feature coordinates and the coordinates of the neighboring features, and finally output the feature representation of the corresponding coordinates: S(v, G) = |vn|G[m]+|vm|G[n] Where S(·) is the grid mapping function, v represents the coordinate, G represents the mapped grid, and m and n are the adjacent coordinates of v; Setting up a learnable local grid pass In G l Find the value of the corresponding position in the grid, and use it as the representative of the pixel point. Finally, the feature representation that can represent the pixel point in the output grid is Right now: Where S(., G) represents the sampling process from the grid G, and the number of channels of the output feature representation is the number of channels of the grid, indicating that each channel in the grid is sampled. The local feature representation of the final output is defined as f l .

6. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 5 is characterized in that: During training, by comparing f(x) with the input feature f x The similarity of the two learnable grids is used to continuously learn, and the mean square error and cosine similarity are combined to evaluate the similarity between the two, and the final feature representation is refined so that it can accurately reconstruct normal features; The normal feature representation is obtained through local and global grid sampling, and the reconstructed normal features are obtained through the convolutional network. The reconstructed normal features are compared with the input features to obtain the abnormal feature map pre, which is used as the abnormal feature predicted by the network:

7. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 6 is characterized in that: In the step 2, in the CLIP model, the text prompt template, the text encoder and the visual encoder work together to achieve cross-modal learning. The text prompt template is a natural language description input to the text encoder to express the semantic content of the image. The text encoder captures the semantic information of the text by converting these text prompts into vector representations; at the same time, the image is converted into a vector representation by the visual encoder to capture the visual features of the image. Finally, CLIP optimizes the outputs of the two encoders through contrastive learning, so that images and texts with the same semantics are closer in the same embedding space, achieving effective alignment between the image and the text.

8. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 7 is characterized in that: Step (1): Design of text prompt template: The designed text prompt template is as follows: t n =[P1][P2]...[P E ][normal][object]. t a =[Q1][Q2]...[Q E ][c k ][object]. C=[c1,c2...c K ](c1:broken,c2:contamination...) Among them, t n Represents a normal type of text template, t a A text template representing an exception type. C represents the exception type. [P i ],[Q i ](i∈1,...,E) represents the learnable word embeddings of normal and abnormal text templates; Step (2): The text encoder extracts text features: Learnable parameters are added to the text encoder of CLIP so that the model can learn distinguishable abnormal semantic information. The learnable parameters are set and added to the front side of the text prompt embedding and input into the encoder together with the text prompt embedding. After being processed by each layer of the encoder, the learnable text embedding is optimized. The specific description is as follows: L m ＝[l1，l2，...，l M ] [l′ m+1 ,t m+1 ]=T m (cat((l m ,t m ),dim=chanel)) [l′ m+ 2,t m+ 2]=T m+1 (cat((l m+1 ,t m+1 ),dim=chanel)) … t M+1 =T M (t M ) Among them, L m represents the learnable parameters of the m-th layer text encoder, t m represents the learnable text embedding at the mth layer, l′ m+1 represents the output of the m-th layer of learnable parameters after the m-th layer of text encoder, T M represents the frozen M-th layer text encoder, and the cat operation aggregates the learnable parameters and text embedding along the channel dimension; Finally, the text features representing normal and abnormal information are obtained through the text encoder of CLIP n ,t a , which is then used to calculate the similarity with CLIP visual features to obtain the loss; Step (3): Visual encoder extracts features: For the input dataset image x i , after the encoder, we finally get a token-like feature representing global semantic information and the output of multiple intermediate layers: f i =F(x i ) f i ∈R H×W×D Where F(·) represents the visual encoder, f i represents the global image features, represents the image features at the intermediate layer patch level, and |S| represents the set of extracted intermediate layers of the visual encoder.

9. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 8, characterized in that: The step 3 is specifically as follows: The abnormal feature map pre output by the reconstruction module is rich in fine-grained information. According to the visual encoder characteristics of the cross-modal alignment network, a dual attention mechanism is designed to integrate the image features at the |S| intermediate layer patch level They are divided into two groups: high-level patch features and low-level patch features; Step (1): Channel Attention For high-level patch features The shape of the abnormal feature map pre is combined with the high-level patch features Align the shape of the abnormal feature map to simulate the high-level patch features The embedding operation passes through a convolutional layer of the same size and adds high-level patch features. The same position encoding is used to generate the embedding pre' of the abnormal feature map pre, achieving pos is the position encoding of the high-level patch feature, conv is the convolution operation, and upsample represents the upsampling operation; Define MultiHead(Q, K, V) as the following formula, where the weight matrix is ​​represented by W: MultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O W O Indicates the weight parameter when each head is merged into Concat in multi-head attention. i is the result of attention calculation, W Q , W K , W v They represent the attention weight parameters of the query Q, key K and value V input during attention calculation respectively; Then the generated pre′ and Perform multi-head attention calculations and compare Perform residual connection to obtain features that focus on abnormal information Replace patch-level embedding Then similarity calculation is performed with text features; Step (2): Spatial Attention: For low-level patch features Perform spatial attention to focus on the pixel information in the anomaly; Take the spatial attention module in the CBAM attention module and define M C (F) is the spatial attention module function, where for the feature F input into the module, the output is: M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) F is the input feature of the spatial attention module. MaxPool(·) is the maximum pooling operation, AgvPool(·) is the average pooling operation, MLP(·) is a multilayer perceptron, i.e., a feedforward neural network, and σ represents the sigmoid activation function. Then, a two-dimensional spatial attention map is obtained, that is, the weight value of each feature point in the input feature map is obtained; Setting low-level patch features To get close to the shape of the abnormal feature map pre, by Perform deconvolution and then restore the resolution of the feature map; Make the low-level patch features Focus on the spatial information of pre and restore it to low-level patch features The shape of the output feature is finally obtained Output as a new low-level patch feature; Among them, upsample(·) is the upsampling operation, M C Get the attention weight value for the spatial attention operation, deconv(·) is the deconvolution operation, represents the matrix multiplication operation, conv2(·) is the convolution operation; All visual features in CLIP When attention is performed separately, when the number of channel attention and the number of spatial attention are the same, the comprehensive detection is the best, that is, 10. The small sample anomaly detection and classification method based on reconstruction-guided cross-modal alignment according to claim 8, characterized in that: The step 4 is specifically as follows: For loss calculation, visual embedding and text embedding are aligned from global and local perspectives, that is, learnable text cues are optimized from two perspectives, and global and local loss functions are designed to adjust the trainable parameters of the model. Among them, L global is the cross entropy loss, label is the true abnormal type label of the image, M gt ∈R H×W represents the true mask, I represents the all-1 matrix, t [n，a] represents text features, n and a represent normal and abnormal tags respectively, represents the patch image feature after attention fusion with the abnormal feature map pre; Focal and Dice are two loss functions, P(·) represents the calculation of abnormal probability score, and λ is a hyperparameter that balances global and local losses; During the training process, after the loss is calculated, the backpropagation operation is performed, that is, the gradient of each model parameter with respect to the loss function is calculated by the chain rule, the parameters are updated according to these gradients, and the gradient descent optimization algorithm is used to minimize the loss function.

Images