Industrial product anomaly detection method based on zero sample data

Abstract

The invention discloses an industrial product anomaly detection method based on zero sample data, and the method comprises the steps: inputting a to-be-detected image and a product type thereof, constructing a class condition comparison description text prompt based on the product type, and coding the text prompt into a text feature through a text encoder; extracting multi-layer visual features of the to-be-detected image, and processing the multi-layer visual features through a multi-layer semantic alignment module to generate visual features which are semantically aligned with the text features; calculating the similarity between the aligned visual features and text features to generate an image-level anomaly score for anomaly classification and a pixel-level anomaly segmentation map for anomaly location; constructing an optimization target composed of image-level classification loss and pixel-level segmentation loss, and carrying out model training; and inputting a to-be-detected industrial product image into the trained model, and carrying out anomaly detection and positioning on the industrial product. The method does not need a target industrial product sample, and can effectively improve the precision of zero-sample industrial anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision anomaly detection technology, specifically to an industrial product anomaly detection method based on class conditional cueing and multi-layer semantic alignment using zero-sample data from a multimodal model. Background Technology

[0002] In modern industrial manufacturing, product anomaly detection is a crucial step in ensuring production safety and product quality. However, traditional detection methods rely on large amounts of labeled data, which becomes costly and time-consuming when acquiring data for rapid new product iterations or rare anomaly scenarios. Therefore, zero-sample anomaly detection technology that does not require data on the target industrial product is of great significance.

[0003] Existing zero-shot industrial anomaly detection methods, such as WinCLIP, typically rely on pre-trained multimodal models (CLIP) to construct basic textual cues (e.g., "normal product" and "abnormal product"), encode them as textual features, and then compare their similarity with the visual features of the image to be detected to determine the presence of anomalies. However, existing zero-shot industrial anomaly detection methods suffer from insufficient category specificity and inadequate utilization of shallow visual information, resulting in limited accuracy in identifying and locating diverse anomalies in practical industrial applications. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a zero-shot industrial product anomaly detection method based on class conditional hints and multi-layer semantic alignment using a multimodal model (CLIP). This method utilizes class conditional hints and multi-layer semantic alignment based on the multimodal model CLIP to achieve high-precision anomaly detection and localization of industrial products without requiring labeled data of the target industrial products. By constructing category-specific comparative text hints and multi-layer features of fused images, the performance of zero-shot industrial anomaly detection is improved.

[0005] This invention discloses an industrial product anomaly detection method based on zero-sample data, comprising: Step 1: Obtain the auxiliary training dataset and test dataset; Step 2: Input the images to be detected and their product categories from the auxiliary training dataset, construct class-conditional contrastive descriptive text prompts based on product categories, and use a multimodal model text encoder to encode the text prompts into normal text features and abnormal text features; Step 3: Extract multi-layer visual features from the image to be detected, and process the multi-layer visual features through a multi-layer semantic alignment module to generate visual features that are semantically aligned with the text features. Step 4: Calculate the similarity between the aligned visual features and the text features to generate an image-level anomaly score for anomaly classification and a pixel-level anomaly segmentation map for anomaly localization. Step 5: Construct an optimization objective consisting of image-level classification loss and pixel-level segmentation loss, and train the model. Step 6: Input the images of industrial products to be detected in the test dataset into the trained model. By calculating the image-level anomaly score and the pixel-level anomaly segmentation map, the anomaly detection and localization of industrial products can be achieved.

[0006] As a further improvement of the present invention, in step 1, the auxiliary training dataset is MVTec AD and the test dataset is VisA; or, the auxiliary training dataset is VisA and the test dataset is MVTec AD. Specifically, the datasets MVTec AD and VisA cover two major product categories: objects and textures, with diverse product categories and anomaly types. There is no class overlap between the training dataset and the test dataset to ensure strict adherence to the zero-sample setting. By default, this invention uses VisA as the auxiliary training dataset and tests it on the MVTec AD dataset. When evaluating the VisA dataset, MVTec AD is selected as the auxiliary training dataset.

[0007] As a further improvement of the present invention, step 2 specifically includes: Step 21: Use Large Language Model (LLM) to generate detailed comparative descriptions of normal and abnormal states for a specific product category; Step 22: Embed the detailed comparison descriptions into preset prompt word templates containing learnable text vectors, state descriptors, and category names to form information-enhanced normal text prompts and abnormal text prompts; Step 23: Input the enhanced normal text prompts and abnormal text prompts into the text encoder of the pre-trained multimodal CLIP model to generate the normal text features and abnormal text features.

[0008] As a further improvement of the present invention, step 3 specifically includes: Step 31: Input the image to be detected into the image encoder of the multimodal model, extract shallow visual features from at least one intermediate layer, and extract deep visual features from its final layer. Step 32: Use the cross-attention mechanism guided by the deep visual features to perform semantic enhancement and alignment on each of the shallow visual features to obtain preliminary enhanced features that are aligned with the deep semantics and rich in local details. Step 33: Process the preliminary enhanced features using a self-attention mechanism to strengthen the connection between global and local information within them, thereby obtaining further enhanced features; adaptively weightedly fuse the preliminary enhanced features and the further enhanced features using learnable weights to obtain the final visual features aligned with the previous ones. As a further improvement of the present invention, the cross-attention mechanism projects the deep visual features into a query and the shallow visual features into a key and a value, and then calculates the cross-attention weights based on the semantic relevance between the query and the key to weight the value.

[0009] As a further improvement of the present invention, step 4 specifically includes: Step 41: Extract the global visual features of the image to be detected, calculate the cosine similarity between the global visual features and the normal and abnormal text features generated in Step 2, and obtain the image-level anomaly score to determine whether there is anomaly in the whole image. Step 42: Calculate the cosine similarity between the aligned visual features obtained in Step 3 and the normal and abnormal text features to obtain the normal localization map and abnormal localization map for each layer; sum and normalize the localization maps of all levels to fuse them into the final normal segmentation probability map and abnormal segmentation probability map; fuse the normal segmentation probability map and abnormal segmentation probability map to obtain the final abnormal segmentation result map.

[0010] As a further improvement of the present invention, step 5 specifically includes: Step 51: For image-level classification loss, Focal Loss is used to optimize the global anomaly score; Step 52: For pixel-level segmentation loss, Focal Loss and Dice Loss are used for optimization; Step 53: During model training, keep the image and text encoder parameters in the pre-trained multimodal model frozen, and optimize only the parameters of the newly introduced learnable text prompts and multi-layer semantic alignment modules.

[0011] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs class-conditional comparative description prompting learning, utilizing a large language model to generate informative comparative descriptions of normal and abnormal states for each industrial product category, thereby more accurately retrieving visual knowledge from the CLIP model. Furthermore, it utilizes multi-layer semantic alignment technology, guiding shallow features with deep semantic features to achieve semantic consistency across multiple visual features, fully leveraging key detailed information. Ultimately, this improves the accuracy of detecting and locating anomalies in industrial products without requiring target product samples. Attached Figure Description

[0012] Figure 1 This is a flowchart of the industrial product anomaly detection method based on zero-sample data disclosed in this invention; Figure 2 This is a diagram showing the overall network model architecture design disclosed in this invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 , 2 As shown, this invention provides a zero-shot industrial product anomaly detection method based on class-conditional hints and multi-layer semantic alignment using multimodal models. To train the learnable module introduced in this invention, a publicly available industrial anomaly dataset (VisA or MVTec AD) can be used as an auxiliary training set, ensuring no class overlap between the training set and the final target dataset to be tested, thus adhering to the zero-shot setting. During processing, all input images are normalized to a uniform size of 518x518 pixels with 3 channels. Image-level anomaly detection can be formally defined as a binary classification problem; the model needs to classify samples... Classified as normal ( ) or abnormal ( Anomaly localization extends this concept to the pixel level, using masks. Highlight pixels associated with anomalies to identify anomalous areas.

[0015] Specifically, it includes: Step 1: Obtain the auxiliary training dataset: Obtaining industrial product auxiliary training dataset This includes classification labels for normal and abnormal data, as well as segmentation labels, used for model training. Depend on The images consist of normal and abnormal images. middle, for This indicates that it is normal. This indicates anomalies. The test set contains images from various application domains and different anomaly types to enable the representation of anomalies across a range of different test datasets. The above performs anomaly detection tasks, where each All with The difference lies in the fact that this invention uses publicly available industrial anomaly datasets (such as VisA or MVTec AD) as auxiliary training sets, and ensures that there is no class overlap between the training set and the target dataset to be tested, in order to follow the zero-sample setting.

[0016] Step 2: Construct class-specific comparison description text hints and encode text features. This step aims to generate personalized and informative text descriptions for each product category and encode them as text features.

[0017] Specifically, it includes: Step 21: First, use the Large Language Model (LLM) to generate detailed comparative descriptions of normal and abnormal states for each target category. For a specific product category name (such as "screw"), use the LLM to generate detailed comparative descriptions of its "normal" and "abnormal" states. For example, "normal state" can be described as "threads are clear and intact," and "abnormal state" can be described as "threads are damaged or the top is deformed."

[0018] Step 22: Embed these generated descriptions into a preset prompt template. This template contains learnable text vectors, a generic state descriptor ("normal" or "abnormal"), and a category name. The preset prompt template is represented as follows: (1) (2) in, and surface The learnable text prompt vector represents a predefined generic state descriptor, either "normal" or "abnormal," and indicates the object category name. and These represent fine-grained descriptions of normal and abnormal states generated by a large language model, respectively.

[0019] Step 23: Input the enhanced complete text prompts from the above steps into the CLIP multimodal text encoder to obtain text features. and .

[0020] (3) (4) in, For CLIP text encoder, To describe text features normally, To describe the text features of anomalies, These are normal text description prompts. These are error message descriptions or warning words.

[0021] Step 3: Extract and align multi-layer visual semantic features. This step effectively integrates multi-layer visual features from the CLIP image encoder using a multi-layer semantic alignment (MSA) technique.

[0022] Specifically, it includes: Step 31: Feed the input image to be detected into the pre-trained CLIP model's image encoder. Extract from the CLIP image encoder... The visual features of the layers, among which Indicates the first Layered visual features. Considering the deep visual features of the last layer of CLIP. As a guide, from the corresponding shallow visual features ( Dynamically extract and enhance semantically relevant details in the process.

[0023] Step 32: Cross-attention to achieve semantic alignment. For each shallow layer... ,Will Through projection matrix Get as a query. Shallow features Through projection matrix and Get the keys respectively Sum . use and The semantic relevance between features is used to calculate cross-attention weights, and the information in the shallow features is weighted and fused based on these weights to obtain the preliminarily enhanced shallow features. The resulting features incorporate shallow details relevant to deep semantics and tend to maintain semantic consistency with deep features. The enhancement process can be represented as: (5) (6) (7) (8) in, Indicates cross attention. Representing deep features Guide the first Shallow layer features Obtain features consistent with deep semantics. Representation layer normalization, , and Represents the projection matrix. express Projection dimension, This represents the softmax function.

[0024] Step 33: Self-attention feature enhancement and adaptive weighted fusion of enhanced features. Enhanced shallow features are obtained through cross-attention. Then, a self-attention mechanism is introduced. This mechanism calculates... The correlation between internal elements strengthens the connection between global and local information, resulting in a further enhanced feature map. The process is represented as: (9) (10) (11) (12) in, Indicates self-attention, Representation layer normalization. , and Represents the projection matrix. express Projection dimension, , and express Passing through , and The values ​​obtained from matrix projection are used as the query, key, and value for self-attention, respectively. This represents the softmax function.

[0025] Effective feature fusion. Through a learnable weight... ,Will and Feature adaptive weighted fusion yields the final feature that is semantically aligned with the text features. Layer visual feature map This ensures the semantic consistency of multi-layered visual features. It is represented as: (13) in, Represents learnable weights, used for adjustment Compared to . contributions.

[0026] Step 4: Perform anomaly classification and localization. Image-level and pixel-level anomalies are identified by calculating the similarity between text features and image features. Specifically: Step 41: Implement image-level anomaly detection. For the input image, extract its global visual features using the CLIP image encoder. Text features and Image-level anomaly classification is achieved through cosine similarity calculation. The global anomaly score is calculated using the following expression: (14) in, This indicates the calculation of cosine similarity. This indicates temperature hyperparameters. This represents an exponential function.

[0027] Step 42: Achieve pixel-level anomaly localization. Calculate the visual features of each layer. Features of normal and abnormal text and The cosine similarity is calculated, and the original image resolution is restored using bilinear interpolation to obtain the first... Normal positioning diagram of the layer and anomaly location map : (15) (16) in, Indicates the first Enhanced visual features after multi-layer semantic alignment and These represent normal and abnormal text features for text description enhancement, respectively. This represents the bilinear interpolation operation. This indicates temperature hyperparameters. This represents a regularization operation. Represents the calculation of cosine similarity. This represents an exponential function.

[0028] To integrate information from different levels, the localization maps from all levels are summed and normalized to obtain the final normal segmentation probability map. and anomaly segmentation probability map : (17) (18) in, This represents a regularization operation. and They represent the first Location diagrams of normal and abnormal layers.

[0029] Final anomaly segmentation result image It is obtained by fusing the normal and abnormal segmentation probability maps, and is represented as: (19) in, and These represent the normal segmentation probability map and the abnormal segmentation probability map, respectively. This represents the Gaussian filtering function. This indicates the control smoothing parameter.

[0030] Step 5: Construct the loss function and train the model. The optimization objective of the training consists of two parts: image-level classification loss and pixel-level segmentation loss.

[0031] Specifically, it includes: Step 51: Construct image-level loss. Image-level loss uses Focal loss to optimize the global anomaly score. This is to alleviate the class imbalance problem commonly encountered in anomaly detection tasks. The image-level classification loss can be expressed as: (20) in, This represents the model's predicted value for classifying the image as an anomaly. This represents a regulation factor used to adjust the model's focus on samples of varying difficulty.

[0032] Step 52: Construct pixel-level loss. Use Focal loss. To optimize pixel-level anomaly prediction accuracy, we focus on hard-to-distinguish pixels. The expression is: (twenty one) in, Indicates the total number of pixels. The model represents the first The predicted value of each pixel. This indicates a regulatory factor.

[0033] Using Dice loss To further improve the accuracy of anomaly region segmentation, the overlap between the predicted region and the ground truth labeled region is directly optimized, and its expression is: (twenty two) in, Indicates the total number of pixels. Indicates the first The model prediction value for each pixel. Indicates the first The real label is a pixel.

[0034] Total training loss The sum of the image-level classification loss and the pixel-level segmentation loss is expressed as: (twenty three) Step 53: During training, keep the image and text encoder parameters of the pre-trained multimodal model CLIP frozen, and optimize only the parameters of the learnable text prompts and multi-layer semantic alignment modules introduced in this invention.

[0035] Step 6: Test the multimodal model-based industrial anomaly detection network trained on the auxiliary training set with class-conditional prompting and multi-layer semantic alignment on the test set. Input the image of the industrial product to be detected into the trained model, and realize the detection and localization of anomalies in the industrial product by calculating the image-level anomaly score and pixel-level anomaly segmentation map.

[0036] To verify the effectiveness of the present invention, tests were conducted on publicly available datasets. The image-level and pixel-level detection results (%) of the model with and without class conditional prompts and multi-layer semantic alignment on the VisA dataset are compared, as shown in Table 1. The image-level and pixel-level detection results (%) of the model with and without class conditional prompts and multi-layer semantic alignment on the MVTec AD dataset are also compared, as shown in Table 2.

[0037] Table 1 Table 2 On the VisA dataset, the method of this invention achieved an image-level AUROC of 84.3% and a max-F1 score of 81.7%; and a pixel-level AUROC of 96.2% and a max-F1 score of 38.4%. On the MVTec AD dataset, the method of this invention achieved an image-level AUROC of 92.4% and a max-F1 score of 92.4%; and a pixel-level AUROC of 90.3% and a max-F1 score of 47.2%. These experimental results clearly demonstrate that this invention can effectively improve the accuracy of zero-shot industrial anomaly detection.

[0038] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting anomalies in industrial products based on zero-sample data, characterized in that, include: Step 1: Obtain the auxiliary training dataset and test dataset; Step 2: Input the images to be detected and their product categories from the auxiliary training dataset, construct class-conditional contrastive descriptive text prompts based on product categories, and use a multimodal model text encoder to encode the text prompts into normal text features and abnormal text features; Step 3: Extract multi-layer visual features from the image to be detected, and process the multi-layer visual features through a multi-layer semantic alignment module to generate visual features that are semantically aligned with the text features. Step 4: Calculate the similarity between the aligned visual features and the text features to generate an image-level anomaly score for anomaly classification and a pixel-level anomaly segmentation map for anomaly localization. Step 5: Construct an optimization objective consisting of image-level classification loss and pixel-level segmentation loss, and train the model. Step 6: Input the images of industrial products to be detected in the test dataset into the trained model. By calculating the image-level anomaly score and the pixel-level anomaly segmentation map, the anomaly detection and localization of industrial products can be achieved.

2. The industrial product anomaly detection method as described in claim 1, characterized in that, In step 1, the auxiliary training dataset is MVTec AD and the test dataset is VisA; or, the auxiliary training dataset is VisA and the test dataset is MVTec AD.

3. The industrial product anomaly detection method as described in claim 1, characterized in that, Step 2 specifically includes: Step 21: Use Large Language Model (LLM) to generate detailed comparative descriptions of normal and abnormal states for a specific product category; Step 22: Embed the detailed comparison descriptions into preset prompt word templates containing learnable text vectors, state descriptors, and category names to form information-enhanced normal text prompts and abnormal text prompts; Step 23: Input the enhanced normal text prompts and abnormal text prompts into the text encoder of the pre-trained multimodal CLIP model to generate the normal text features and abnormal text features.

4. The industrial product anomaly detection method as described in claim 1, characterized in that, Step 3 specifically includes: Step 31: Input the image to be detected into the image encoder of the multimodal model, extract shallow visual features from at least one intermediate layer, and extract deep visual features from its final layer. Step 32: Use the cross-attention mechanism guided by the deep visual features to perform semantic enhancement and alignment on each of the shallow visual features to obtain preliminary enhanced features that are aligned with the deep semantics and rich in local details. Step 33: Process the preliminary enhanced features using a self-attention mechanism to strengthen the connection between global and local information within them, thereby obtaining further enhanced features; adaptively weightedly fuse the preliminary enhanced features and the further enhanced features using learnable weights to obtain the final visual features aligned with the initial enhanced features.

5. The industrial product anomaly detection method as described in claim 4, characterized in that, The cross-attention mechanism projects the deep visual features into a query and the shallow visual features into a key and a value. Then, it calculates the cross-attention weights based on the semantic relevance between the query and the key to weight the values.

6. The industrial product anomaly detection method as described in claim 1, characterized in that, Step 4 specifically includes: Step 41: Extract the global visual features of the image to be detected, calculate the cosine similarity between the global visual features and the normal and abnormal text features generated in Step 2, and obtain the image-level anomaly score to determine whether there is anomaly in the whole image. Step 42: Calculate the cosine similarity between the aligned visual features obtained in Step 3 and the normal and abnormal text features to obtain the normal localization map and abnormal localization map for each layer; sum and normalize the localization maps of all levels to fuse them into the final normal segmentation probability map and abnormal segmentation probability map; fuse the normal segmentation probability map and abnormal segmentation probability map to obtain the final abnormal segmentation result map.

7. The industrial product anomaly detection method as described in claim 1, characterized in that, Step 5 specifically includes: Step 51: For image-level classification loss, Focal Loss is used to optimize the global anomaly score; Step 52: For pixel-level segmentation loss, Focal Loss and Dice Loss are used for optimization; Step 53: During model training, keep the image and text encoder parameters in the pre-trained multimodal model frozen, and optimize only the parameters of the newly introduced learnable text prompts and multi-layer semantic alignment modules.

Images