Zero-shot anomaly detection method and device based on abnormal superclass semantic guidance

By constructing an anomaly superclass semantic structure and visual feature mapping, the problem of insufficient utilization of hierarchical semantic relationships of anomaly categories in zero-shot anomaly detection is solved, achieving efficient detection and localization of unknown anomalies, reducing data collection and annotation costs, and improving the model's generalization ability and adaptability to complex scenarios.

CN123069192APending Publication Date: 2026-11-10SUN YAT SEN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611223202.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-13
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing zero-shot anomaly detection methods fail to fully utilize the hierarchical semantic relationships between anomaly categories, resulting in weak model recognition capabilities for unknown anomaly categories. Furthermore, they require extensive data collection and annotation, making it difficult to adapt to new anomaly types.

Method used

An anomaly superclass semantic structure is constructed. Anomaly categories are analyzed using a large language model to form a superclass set. A correspondence between visual features and anomaly superclasses is established. Superclass cue vectors are introduced during model training. A loss function is used to optimize the mapping between visual features and superclass cue vectors, thereby achieving deep fusion of anomaly semantics and visual features.

Benefits of technology

It improves the model's ability to detect unknown anomaly categories and generalize across datasets, reduces data collection and labeling costs, enhances the ability to express complex anomaly patterns, and has good scalability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069192A_ABST
    Figure CN123069192A_ABST
Patent Text Reader

Abstract

The application discloses a zero-shot anomaly detection method and device based on abnormal super-class semantic guidance. The method comprises the following steps: forming a super-class anomaly set by counting abnormal categories; constructing a super-class prompt vector pool composed of normal category prompt words and a plurality of abnormal super-class prompt vectors; setting a matching key vector for each super-class prompt vector to establish a corresponding relationship between visual features and abnormal super-classes; in the model training stage, determining the abnormal super-class and the super-class prompt vector to which the training sample belongs according to the abnormal category label of the training sample, inserting the super-class prompt vector into the visual deep learning layer as an additional prompt word element, and performing visual feature learning together with the current visual feature to fuse abnormal semantics and visual representation, calculating a loss function according to the training output result, and updating the abnormal detection model according to the loss function. The model can improve the detection capability of unknown abnormal categories and the cross-dataset generalization capability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of communications, and in particular to a zero-sample anomaly detection method and apparatus based on anomaly superclass semantic guidance. Background Technology

[0002] With the development of industrial manufacturing, intelligent inspection, medical imaging, and other fields, an increasing number of scenarios require the use of computers to automatically detect anomalies in products or equipment. For example, in industrial inspection, it is necessary to automatically identify whether products have defects such as cracks, gaps, contamination, and deformation; in medical image analysis, it is necessary to automatically detect lesion areas. Because anomalies are usually numerous, have a low probability of occurrence, and new types of anomalies are constantly emerging, enabling computers to identify unknown anomalies is an important research direction in the field of anomaly detection.

[0003] Traditional anomaly detection methods typically rely on training with large amounts of manually labeled data. This requires collecting various anomaly samples in advance and using these samples to train the model. When a new type of anomaly appears, because the model has not learned about that type of anomaly, it is usually necessary to collect data again and retrain the model. This approach not only requires a lot of data collection and labeling work, but also results in a weak ability to identify unknown anomalies, which has significant limitations in practical applications.

[0004] With the development of visual language models, zero-shot anomaly detection methods have emerged. These methods leverage the image and text understanding capabilities of large-scale pre-trained models (such as CLIP) to perform anomaly detection without needing to collect a large number of new anomaly samples in the target task. Therefore, zero-shot anomaly detection has gradually become an important research direction in the field. Currently, most zero-shot anomaly detection methods use text prompts to describe normal and abnormal states, and then calculate the similarity between image features and text features to achieve anomaly recognition. For example, the WinCLIP method uses manually designed text prompts to complete anomaly detection; AnomalyCLIP further employs learnable text prompts, enabling the model to automatically learn semantic representations more suitable for anomaly detection tasks; subsequently, FAPrompt proposed learning multiple anomaly text prompts and dynamically adjusting them using image information, thereby improving the model's ability to express different anomaly patterns. Although the above methods can improve the performance of zero-shot anomaly detection, their main improvements are still focused on the design of text prompts and do not fully utilize the semantic relationships between different anomaly categories. In fact, in actual industrial inspection processes, different anomalies usually have obvious hierarchical relationships. For example, cracks, fractures, and notches, although belonging to different anomaly categories, are essentially all "structural damage"; stains, color changes, and surface contamination, although manifesting differently, all belong to "appearance anomalies"; missing parts and assembly errors can be categorized as "composition anomalies." In other words, different anomaly categories are not independent of each other, but rather share a higher level of common semantics.

[0005] However, existing methods typically learn multiple anomaly cues directly, with each cue representing only one anomaly pattern, without establishing a hierarchical relationship between these anomaly categories. Furthermore, during model inference, they cannot automatically determine which anomaly cue is more suitable for the current image based on the input image; instead, they use fixed cuees or multiple cuees working together for detection, which can easily introduce irrelevant anomaly semantics and negatively impact detection performance. Summary of the Invention

[0006] The following is an overview of the topics described in detail in this article.

[0007] The purpose of this application is to at least partially solve one of the technical problems existing in the related technologies. The embodiments of this application provide a zero-shot anomaly detection method and apparatus based on anomaly superclass semantic guidance, which can improve the model's ability to detect unknown anomaly categories and its cross-dataset generalization ability.

[0008] An embodiment of the first aspect of this application provides a zero-shot anomaly detection method based on anomaly superclass semantic guidance, comprising:

[0009] Obtain an auxiliary anomaly dataset, which includes normal samples and anomaly samples, wherein the anomaly samples have anomaly category labels;

[0010] Based on the auxiliary anomaly dataset, anomaly categories are statistically analyzed, and a superclass anomaly set is formed according to the anomaly categories using a large language model.

[0011] Based on the superclass anomaly set, a superclass cue vector pool is constructed, consisting of normal category cue words and multiple anomaly superclass cue vectors;

[0012] Set a matching key vector for each superclass cue vector to establish a correspondence between visual features and anomalous superclasses;

[0013] During the model training phase, training samples are input into the anomaly detection model. Based on the anomaly category label of the training samples, the anomaly superclass and superclass cue vector are determined. The superclass cue vector is inserted as an additional cue word into the visual deep learning layer to learn visual features together with the current visual features to fuse anomaly semantics and visual representation. The loss function is calculated based on the training output results. The anomaly detection model is updated based on the loss function to obtain the target anomaly detection model.

[0014] The loss function includes cue word separation loss and feature key matching loss. The cue word separation loss constrains the different superclass cue vectors to maintain a large semantic distance, so that each superclass cue vector learns the semantic representation of the corresponding abnormal superclass. The feature key matching loss is used to stabilize the mapping relationship between visual features and superclass cue vectors.

[0015] According to certain embodiments of the first aspect of this application, the step of statistically analyzing anomaly categories based on the auxiliary anomaly dataset and forming a superclass anomaly set based on the anomaly categories using a large language model includes:

[0016] The large language model is used to perform semantic analysis on the anomaly categories. Based on the semantic association, formation cause and visual performance characteristics between different anomaly categories, multiple anomaly categories with similar semantics are divided into several anomaly superclasses.

[0017] According to certain embodiments of the first aspect of this application, a superclass cue vector pool is constructed based on the superclass anomaly set, consisting of normal category cue words and multiple anomaly superclass cue vectors, including:

[0018] Based on the superclass anomaly set, a normal category prompt word corresponding to the normal category and multiple abnormal superclass prompt vectors corresponding to the abnormal superclasses are constructed. The superclass prompt vector pool is composed of the normal category prompt word and the multiple abnormal superclass prompt vectors.

[0019] According to certain embodiments of the first aspect of this application, during the model training phase, a correspondence between category features, matching keys, and superclass cue vectors is established through supervised learning between visual features and keys.

[0020] According to certain embodiments of the first aspect of this application, the loss function is the sum of anomaly detection task loss, cue word separation loss, and feature key matching loss.

[0021] According to certain embodiments of the first aspect of this application, the anomaly category label includes one or more of the following: crack, fracture, notch, scratch, contamination, color change, missing part, assembly error, and deformation.

[0022] According to certain embodiments of the first aspect of this application, cracks, fractures and notches are classified as structural damage superclasses, stains, color changes and surface contamination are classified as appearance abnormalities superclasses, and missing parts and assembly errors are classified as composition abnormalities superclasses.

[0023] According to certain embodiments of the first aspect of this application, during the model testing phase, a visual deep learning network is used to extract category features corresponding to the visual deep learning layer, the similarity between the category features and each matching key is calculated, the superclass cue vector corresponding to the key with the highest similarity is selected, the selected superclass cue vector is inserted into the visual deep learning layer corresponding to the visual deep learning network, and visual features are extracted together with visual features. Visual features are encoded under the guidance of abnormal semantics to generate image-level anomaly detection results and pixel-level anomaly localization results.

[0024] According to a second aspect of this application, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the zero-sample anomaly detection method based on anomaly superclass semantic guidance as described in the first aspect of this application.

[0025] According to a third aspect of this application, a computer storage medium stores computer-executable instructions for performing a zero-sample anomaly detection method based on anomaly superclass semantic guidance as described in an embodiment of the first aspect of this application.

[0026] The above-mentioned solution has at least the following beneficial effects:

[0027] By constructing a hierarchical semantic structure of anomaly superclasses, a dynamic correspondence between visual features and anomaly superclasses is established, and the semantics of anomaly superclasses are introduced into the visual Transformer feature extraction process, which has significant technical effects in zero-shot anomaly detection tasks.

[0028] Using a large language model, semantic analysis and aggregation are performed on specific anomaly categories in the auxiliary anomaly dataset. Multiple specific anomaly categories with similar semantic features, visual appearance, or causes are classified into the same anomaly superclass, thereby establishing a hierarchical semantic structure of anomaly categories, anomaly superclasses, and superclass cue vectors.

[0029] By fully utilizing the hierarchical semantic relationships between anomaly categories, multiple specific anomaly categories can share the semantic knowledge of their corresponding superclasses. When faced with new anomaly categories that did not appear during the training phase, knowledge transfer can still be accomplished by leveraging the semantics of superclasses, thereby improving the model's generalization ability to unknown anomaly categories.

[0030] A Prompt pool is constructed, consisting of a normal Superclass Prompt and multiple abnormal Superclass Prompts. This allows different abnormal superclasses to have independent semantic representations. The Prompt separation loss constraint maintains a large semantic distance between different Superclass Prompts, preventing multiple Prompts from learning repetitive or similar semantic information. This improves the discriminative and expressive power of different abnormal superclasses, enabling the model to cover richer abnormal patterns and improve detection performance in complex abnormal scenarios.

[0031] A corresponding Superclass Key is assigned to each Superclass Prompt, and a mapping relationship between visual features and Superclass Keys is established. During the testing phase, the corresponding Superclass Prompt is automatically determined based on the similarity between the input image CLS Feature and each Superclass Key, achieving adaptive selection of anomalous semantics.

[0032] It eliminates the need for manual specification of anomaly categories and the need to use all anomaly prompts. Instead, it can automatically select the most suitable superclass semantics for the current sample based on the visual features of the input image, thereby reducing the impact of irrelevant semantic information on anomaly detection and improving the accuracy of anomaly classification and localization.

[0033] The dynamically selected Superclass Prompt is directly introduced into multiple Transformer layers of the visual Transformer, enabling the superclass semantics of anomalies to continuously participate in the visual feature extraction process, achieving deep fusion of anomaly semantics and visual features, further enhancing the model's ability to express features of complex anomaly patterns and unknown anomaly categories, and improving zero-shot anomaly detection performance.

[0034] Based on the zero-shot anomaly detection framework, it is only necessary to learn the semantic relationship of anomaly superclasses using auxiliary datasets, which can be transferred to new detection tasks. There is no need to retrain the model for each new anomaly category, which can effectively reduce the cost of anomaly data collection and manual annotation in the target domain. It has high application value in fields such as industrial defect detection, medical image analysis and intelligent manufacturing.

[0035] It exhibits good scalability, with the number of superclasses, Superclass Prompts, and Prompt injection points all adjustable according to specific tasks. When a new anomaly category emerges, model expansion can be achieved simply by mapping its semantic features to existing superclasses or by adding new superclasses, demonstrating promising prospects for engineering applications. Attached Figure Description

[0036] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0037] Figure 1 This is a flowchart illustrating the steps of a zero-shot anomaly detection method guided by anomaly superclass semantics.

[0038] Figure 2 This is a step diagram of the testing phase of a zero-shot anomaly detection method based on anomaly superclass semantic guidance;

[0039] Figure 3 This is a schematic diagram of the principle framework of a zero-shot anomaly detection method based on anomaly superclass semantic guidance. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0042] The terms used in this application are explained below.

[0043] (1) Zero-Shot Anomaly Detection (ZSAD):

[0044] This refers to a method in object detection scenarios where the model can detect and locate unknown anomaly categories without training on anomaly samples of the target category, relying solely on knowledge learned from auxiliary datasets.

[0045] (2) Visual Language Model (VLM):

[0046] A multimodal pre-trained model is a model that can process both image and text information simultaneously and establish a correspondence between image features and text semantic features.

[0047] (3) CLIP (Contrastive Language–Image Pre-training):

[0048] A typical visual language pre-training model learns by comparing large amounts of image-text pairing data, mapping image features and text features to a unified semantic space, thereby achieving cross-modal semantic matching. CLIP mainly consists of a text encoder and a visual encoder (ViT), where the text encoder is used to extract text semantic features, and the visual encoder is used to extract image visual features.

[0049] (4) Vision Transformer (ViT) and Token:

[0050] The Visual Transformer is a visual coding network based on the Transformer architecture. It first divides the input image into multiple fixed-size image patches and maps each image patch to a corresponding image patch token as network input to represent local image region information. At the same time, a class token (CLS token) is added before all patch tokens to represent the global semantic information of the entire image.

[0051] (5) Large Language Model (LLM):

[0052] This refers to an artificial intelligence model pre-trained on a large-scale corpus, possessing natural language understanding and reasoning capabilities, and applicable to tasks such as text understanding, knowledge reasoning, and semantic analysis. This method utilizes a large language model to perform semantic analysis between specific anomaly categories and anomaly superclass classification.

[0053] (6) Prompt Vector:

[0054] The learnable prompt token in the input visual language model is used to guide the model to focus on specific semantic information.

[0055] (7) Superclass:

[0056] A high-level category refers to a group of specific anomaly categories that have similar semantic features, visual appearance, or causes.

[0057] (8) Superclass Prompt:

[0058] This method proposes a learnable Prompt that corresponds one-to-one with anomaly superclasses to represent the semantic information of the corresponding anomaly superclass. Each Superclass Prompt corresponds to an anomaly superclass or a normal category and participates in the visual feature extraction process.

[0059] (9) Matching Key (Key / Superclass Key):

[0060] This refers to the learnable feature vector corresponding to each Superclass Prompt, used to establish the correspondence between visual features and Superclass Prompts. During the testing phase, the corresponding Superclass Prompt is dynamically selected based on the matching results between the visual features of the input image and each Key.

[0061] (10) CLS Feature:

[0062] The output feature corresponding to the Class Token in the Visual Transformer is used to represent the global semantic information of the entire image.

[0063] (11) Image-level:

[0064] Image-level anomaly detection refers to determining whether an entire image contains abnormal targets. It is commonly used for anomaly classification tasks, and its performance is typically evaluated using metrics such as the area under the receiver operating characteristic curve (AUROC) and image-level average precision (AP). AUROC evaluates the model's ability to distinguish between normal and abnormal images, while AP comprehensively evaluates the model's anomaly classification performance at different classification thresholds.

[0065] (12) Pixel-level:

[0066] Pixel-level anomaly detection refers to predicting anomalies for each pixel in an input image to achieve precise localization of anomalous regions. It is commonly used for anomaly localization / segmentation tasks, and its performance is typically evaluated using metrics such as the area under the receiver operating characteristic curve (AUROC) and pixel-level region overlap rate (PRO). Pixel-level AUROC evaluates the model's ability to distinguish anomalies at the pixel level, while pixel-level PRO evaluates the degree of overlap between predicted and actual anomaly regions.

[0067] (13) Losses from anomaly detection tasks:

[0068] This refers to the loss function used to optimize the model's anomaly detection performance, which consists of image-level anomaly detection loss (Global Loss) and pixel-level anomaly detection loss (Local Loss). Global Loss optimizes image-level anomaly classification capabilities, while Local Loss optimizes pixel-level anomaly localization capabilities.

[0069] (14) Prompt separation loss:

[0070] This refers to a loss function designed in this method, which is used to constrain different Superclass Prompts to maintain a large semantic distance, so that different Superclass Prompts learn the semantic representation of the corresponding anomaly superclass, avoid multiple Prompts learning similar semantics, and improve the ability of the Prompt pool to express different anomaly patterns.

[0071] (15) Visual feature-key matching loss:

[0072] This refers to a loss function designed in this method, which is used to establish the mapping relationship between the CLS Feature output by the visual Transformer and the corresponding Superclass Key, so that the input image can accurately match the Key of the corresponding abnormal superclass based on visual features, and further select the corresponding Superclass Prompt.

[0073] To address the problems of insufficient utilization of hierarchical semantic relationships between anomaly categories, lack of stable correspondence between visual features and anomaly semantics, and difficulty in effectively incorporating anomaly semantics into visual feature learning in existing zero-shot anomaly detection methods, embodiments of this application provide a zero-shot anomaly detection method and apparatus based on anomaly superclass semantic guidance. By constructing a hierarchical semantic structure of anomaly categories, establishing a dynamic correspondence between visual features and anomaly superclasses, and introducing the dynamically selected superclass prompt vector into the feature extraction process of the visual deep learning network Transformer, a deep fusion of anomaly semantics and visual features is achieved, which can improve the model's ability to detect unknown anomaly categories and its cross-dataset generalization ability.

[0074] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0075] Reference Figure 1 A zero-shot anomaly detection method based on anomaly superclass semantic guidance includes the following steps:

[0076] In the preparation phase, an auxiliary anomaly dataset is acquired; anomaly categories are statistically analyzed based on the auxiliary anomaly dataset, and a superclass anomaly set is formed according to the anomaly categories using a large language model; a superclass cue vector pool consisting of normal category cue words and multiple anomaly superclass cue vectors is constructed based on the superclass anomaly set; a matching key vector is set for each superclass cue vector to establish the correspondence between visual features and anomaly superclasses;

[0077] During the model training phase, training samples are input into the anomaly detection model. Based on the anomaly category label of the training samples, the anomaly superclass and superclass cue vector are determined. The superclass cue vector is inserted as an additional cue word into the visual deep learning layer to learn visual features together with the current visual features to fuse anomaly semantics and visual representation. The loss function is calculated based on the training output results, and the anomaly detection model is updated based on the loss function to obtain the target anomaly detection model.

[0078] The loss function includes cue word separation loss and feature key matching loss. The cue word separation loss constrains the different superclass cue vectors to maintain a large semantic distance, so that each superclass cue vector learns the semantic representation of the corresponding abnormal superclass. The feature key matching loss is used to stabilize the mapping relationship between visual features and superclass cue vectors.

[0079] Reference Figure 3 The following section provides a detailed description of the zero-shot anomaly detection method guided by anomaly superclass semantics.

[0080] First, obtain the auxiliary anomaly dataset for model training. The auxiliary anomaly dataset includes normal samples and multiple anomaly samples. The anomaly samples have corresponding anomaly category labels, such as cracks, fractures, notches, scratches, contamination, color changes, missing parts, assembly errors, deformation, etc.

[0081] Then, based on the auxiliary anomaly dataset, anomaly categories are statistically analyzed, and a superclass anomaly set is formed according to the anomaly categories using a large language model. This includes the following steps: using a large language model to perform semantic analysis on the anomaly categories, and dividing multiple anomaly categories with similar semantics into several anomaly superclasses based on the semantic association, formation cause, and visual performance characteristics between different anomaly categories.

[0082] For example, cracks, fractures, and notches can be classified as structural damage superclasses, stains, color changes, and surface contamination as appearance anomalies superclasses, and missing parts and assembly errors as composition anomalies superclasses.

[0083] Regarding the construction of anomaly superclasses, this method preferably employs a large language model to perform semantic analysis and clustering of specific anomaly categories, such as using ChatGPT to complete the anomaly superclass division. In other implementations, other large language models with semantic understanding capabilities can also be used to construct anomaly superclasses. Furthermore, domain experts can manually define anomaly superclasses according to actual application needs and establish a mapping relationship between specific anomaly categories and their superclasses; alternatively, methods such as K-means clustering can be used to automatically cluster anomaly categories to obtain the anomaly superclass division results.

[0084] By establishing a hierarchical semantic structure of "specific anomaly category - anomaly superclass" in the above manner, multiple specific anomaly categories can share the semantic knowledge of the corresponding anomaly superclass, providing semantic transfer capability for model recognition of anomaly categories that did not appear during training.

[0085] Then, based on the super-anomaly set, a super-class cue vector pool is constructed, consisting of normal category cue words and multiple anomaly super-class cue vectors, including the following steps:

[0086] Based on the superclass anomaly set, a normal category prompt word corresponding to the normal category is constructed, and multiple abnormal superclass prompt vectors corresponding to the abnormal superclasses are constructed. The superclass prompt vector pool is composed of the normal category prompt word and multiple abnormal superclass prompt vectors.

[0087] For example, if the number of anomalous superclasses is m, then a total of (m+1) learnable anomalous superclass prompt vectors (Superclass Prompts) are constructed. One Prompt corresponds to the normal class, and the remaining m Prompts correspond to different anomalous superclasses. All Superclass Prompts are used as learnable parameters of the model and are continuously updated during training, enabling the model to learn the semantic representation of the corresponding anomalous superclass, thereby achieving independent modeling of the semantics of different anomalous superclasses.

[0088] Then, a matching key vector is set for each superclass cue vector to establish a correspondence between visual features and anomalous superclasses.

[0089] Specifically, to establish the correspondence between visual features and anomalous superclasses, this method further assigns a corresponding Key vector to each Superclass Prompt, and all Keys together form a Key set. Each Key is a learnable parameter used to represent the semantic center of the corresponding anomalous superclass in the visual feature space. During training, the correspondence between the CLS Feature, the Superclass Key, and the Superclass Prompt is established through supervised learning between visual features and Keys.

[0090] During the model training phase, training samples are input into the anomaly detection model. Based on the anomaly category label of the training samples, the corresponding superclass and superclass cue vector are determined. The superclass cue vector is then inserted as an additional cue term into the visual deep learning layer to learn visual features together with the current visual features, thereby fusing anomaly semantics and visual representation.

[0091] Specifically, during model training, for each input training sample, its superclass is first determined based on the anomaly category label, and the corresponding Superclass Prompt is selected. Subsequently, this Superclass Prompt is inserted as an additional Prompt Token into multiple preset Transformer layers of the visual Transformer, participating in subsequent Transformer computations along with the visual features of the current layer. In a preferred embodiment of this method, Prompt injection is performed on the 6th, 12th, 18th, and 24th Transformer layers of the visual Transformer, enabling the anomaly superclass semantics to continuously participate in visual feature learning at different levels of the visual Transformer. This achieves a layer-by-layer fusion of anomaly semantics and visual representation, further enhancing the model's ability to express anomaly features.

[0092] The loss function is calculated based on the training output, and the anomaly detection model is updated based on the loss function to obtain the target anomaly detection model.

[0093] The loss function is the sum of the anomaly detection task loss, the cue word separation loss, and the feature key matching loss.

[0094] The anomaly detection task loss refers to the loss function used to optimize the model's anomaly detection performance, and it consists of image-level anomaly detection loss (Global Loss) and pixel-level anomaly detection loss (Local Loss). Global Loss optimizes image-level anomaly classification capabilities, while Local Loss optimizes pixel-level anomaly localization capabilities.

[0095] The prompt word separation loss improves the discriminative ability between different Superclass Prompts by constraining a large semantic distance between different Superclass prompt vectors, enabling each Superclass prompt vector to learn the semantic representation of the corresponding anomalous Superclass, thus avoiding multiple Prompts from converging to similar semantic spaces and improving the expressive ability of the Prompt pool for different anomalous patterns.

[0096] Feature-key matching loss is used to stabilize the mapping relationship between visual features and superclass cue vectors. During training, the CLS Feature corresponding to the insertion position of the visual Transformer is extracted, and its similarity with all Superclass Keys is calculated. Through supervised learning, the Key corresponding to the abnormal superclass to which the sample belongs obtains the maximum matching score, thereby establishing a one-to-one correspondence between CLS Feature, Superclass Key, and Superclass Prompt.

[0097] After training, the visual features corresponding to different anomaly patterns can cluster around the corresponding Superclass Key, providing a reliable basis for automatically selecting the Superclass Prompt based on visual features during the testing phase. Overall, this method uses a joint optimization approach for training. The final loss function consists of three parts: anomaly detection task loss, Prompt separation loss, and visual feature-key matching loss. By setting appropriate weights to weight each loss term, the method achieves synergistic optimization of anomaly detection capability, anomaly superclass semantic representation capability, and the correspondence between visual features and anomaly semantics.

[0098] Reference Figure 2After the model training is completed, it enters the testing phase. The visual deep learning network is used to extract the category features of the corresponding visual deep learning layers. The similarity between the category features and each matching key is calculated. The superclass cue vector corresponding to the key with the highest similarity is selected and inserted into the corresponding visual deep learning layer of the visual deep learning network. Visual feature extraction is performed together with the visual features. Guided by the anomaly semantics, visual feature encoding is performed to generate image-level anomaly detection results and pixel-level anomaly localization results.

[0099] Since the selection of Superclass Prompt is automatically completed based entirely on the visual features of the input image, even if the test sample belongs to a specific anomaly category that has not appeared during the training phase, the model can still automatically match the closest anomaly superclass based on its visual features and use the semantic knowledge learned by the corresponding anomaly superclass to complete the anomaly detection, thereby effectively improving the model's generalization ability to unknown anomaly categories.

[0100] As shown in Tables 1 and 2, compared with existing methods, this method achieves better detection results in terms of evaluation metrics such as image-level anomaly detection and pixel-level anomaly detection. It achieves optimal or competitive performance on most datasets, verifying the effectiveness of the strategy proposed in this invention.

[0101] Table 1 Image-level results

[0102]

[0103] Table 2 Pixel-level results

[0104]

[0105] The following is a specific implementation method using the zero-sample anomaly detection task for industrial products as an example. First, data preparation is performed by selecting an industrial anomaly auxiliary dataset as training data. This dataset is assumed to contain a normal category and six specific anomaly categories: cracks, scratches, fractures, stains, missing parts, and deformation. A large language model is used to analyze the semantics of these anomaly categories, resulting in three superclasses: structural damage, surface contamination, and missing components. Therefore, m is set to 3, and four Superclass Prompts are constructed: Normal Prompt P0, Structural Damage Prompt P1, Surface Contamination Prompt P2, and Missing Component Prompt P3. A corresponding matching key is assigned to each Prompt. During training, when a crack anomaly image is input, its category label "crack" corresponds to the structural damage superclass. The system automatically selects P1, the structural damage Prompt, and inserts it into the corresponding layer of the visual Transformer. Simultaneously, the anomaly detection loss, Prompt separation loss, and visual feature-key matching loss are calculated to jointly optimize the model parameters. During the testing phase, when faced with a novel crack image not encountered during training, although the specific crack category had not been seen by the model, its visual features, after calculation, showed the highest similarity to Key k1 corresponding to the structural damage superclass. Therefore, the model automatically selected the structural damage Prompt P1 and injected it into the network, completing anomaly detection based on the enhanced features, correctly identifying it as an anomaly and locating the abnormal region. This process fully demonstrates the transfer and adaptation capabilities of this method when facing unknown anomalies through hierarchical semantic modeling of anomaly superclasses and a dynamic Prompt selection mechanism.

[0106] The above methods achieve the following beneficial effects:

[0107] By constructing a hierarchical semantic structure of anomaly superclasses, a dynamic correspondence between visual features and anomaly superclasses is established, and the semantics of anomaly superclasses are introduced into the visual Transformer feature extraction process, which has significant technical effects in zero-shot anomaly detection tasks.

[0108] Using a large language model, semantic analysis and aggregation are performed on specific anomaly categories in the auxiliary anomaly dataset. Multiple specific anomaly categories with similar semantic features, visual appearance, or causes are classified into the same anomaly superclass, thereby establishing a hierarchical semantic structure of anomaly categories, anomaly superclasses, and superclass cue vectors.

[0109] By fully utilizing the hierarchical semantic relationships between anomaly categories, multiple specific anomaly categories can share the semantic knowledge of their corresponding superclasses. When faced with new anomaly categories that did not appear during the training phase, knowledge transfer can still be accomplished by leveraging the semantics of superclasses, thereby improving the model's generalization ability to unknown anomaly categories.

[0110] A Prompt pool is constructed, consisting of a normal Superclass Prompt and multiple abnormal Superclass Prompts. This allows different abnormal superclasses to have independent semantic representations. The Prompt separation loss constrains the different Superclass Prompts to maintain a large semantic distance, preventing multiple Prompts from learning repetitive or similar semantic information. This improves the discriminative and expressive power of different abnormal superclasses, enabling the model to cover richer abnormal patterns and improve detection performance in complex abnormal scenarios.

[0111] A corresponding Superclass Key is assigned to each Superclass Prompt, and a mapping relationship between visual features and Superclass Keys is established. During the testing phase, the corresponding Superclass Prompt is automatically determined based on the similarity between the input image CLS Feature and each Superclass Key, achieving adaptive selection of anomalous semantics.

[0112] It eliminates the need for manual specification of anomaly categories and the need to use all anomaly prompts. Instead, it can automatically select the most suitable superclass semantics for the current sample based on the visual features of the input image, thereby reducing the impact of irrelevant semantic information on anomaly detection and improving the accuracy of anomaly classification and localization.

[0113] The dynamically selected Superclass Prompt is directly introduced into multiple Transformer layers of the visual Transformer, enabling the superclass semantics of anomalies to continuously participate in the visual feature extraction process, achieving deep fusion of anomaly semantics and visual features, further enhancing the model's ability to express features of complex anomaly patterns and unknown anomaly categories, and improving zero-shot anomaly detection performance.

[0114] Based on the zero-shot anomaly detection framework, it is only necessary to learn the semantic relationship of anomaly superclasses using auxiliary datasets, which can be transferred to new detection tasks. There is no need to retrain the model for each new anomaly category, which can effectively reduce the cost of anomaly data collection and manual annotation in the target domain. It has high application value in fields such as industrial defect detection, medical image analysis and intelligent manufacturing.

[0115] It exhibits good scalability, with the number of superclasses, Superclass Prompts, and Prompt injection points all adjustable according to specific tasks. When a new anomaly category emerges, model expansion can be achieved simply by mapping its semantic features to existing superclasses or by adding new superclasses, demonstrating promising prospects for engineering applications.

[0116] An embodiment of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the zero-sample anomaly detection method based on anomaly superclass semantic guidance as described above.

[0117] Embodiments of this application provide a computer storage medium storing computer-executable instructions for executing the zero-sample anomaly detection method based on anomaly superclass semantic guidance as described above.

[0118] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A zero-shot anomaly detection method based on anomaly superclass semantic guidance, characterized in that, include: Obtain an auxiliary anomaly dataset, which includes normal samples and anomaly samples, wherein the anomaly samples have anomaly category labels; Based on the auxiliary anomaly dataset, anomaly categories are statistically analyzed, and a superclass anomaly set is formed according to the anomaly categories using a large language model. Based on the superclass anomaly set, a superclass cue vector pool is constructed, consisting of normal category cue words and multiple anomaly superclass cue vectors; Set a matching key vector for each superclass cue vector to establish a correspondence between visual features and anomalous superclasses; During the model training phase, training samples are input into the anomaly detection model. Based on the anomaly category label of the training samples, the anomaly superclass and superclass cue vector are determined. The superclass cue vector is inserted as an additional cue word into the visual deep learning layer to learn visual features together with the current visual features to fuse anomaly semantics and visual representation. The loss function is calculated based on the training output results. The anomaly detection model is updated based on the loss function to obtain the target anomaly detection model. The loss function includes cue word separation loss and feature key matching loss. The cue word separation loss constrains the different superclass cue vectors to maintain a large semantic distance, so that each superclass cue vector learns the semantic representation of the corresponding abnormal superclass. The feature key matching loss is used to stabilize the mapping relationship between visual features and superclass cue vectors.

2. The zero-shot anomaly detection method based on anomaly superclass semantic guidance according to claim 1, characterized in that, The step of statistically analyzing anomaly categories based on the auxiliary anomaly dataset and forming a superclass anomaly set based on these categories using a large language model includes: The large language model is used to perform semantic analysis on the anomaly categories. Based on the semantic association, formation cause and visual performance characteristics between different anomaly categories, multiple anomaly categories with similar semantics are divided into several anomaly superclasses.

3. The zero-shot anomaly detection method based on anomaly superclass semantic guidance according to claim 1, characterized in that, Based on the aforementioned set of super-anomalies, a super-class cue vector pool is constructed, consisting of normal category cue words and multiple anomaly super-class cue vectors, including: Based on the superclass anomaly set, a normal category prompt word corresponding to the normal category and multiple abnormal superclass prompt vectors corresponding to the abnormal superclasses are constructed. The superclass prompt vector pool is composed of the normal category prompt word and the multiple abnormal superclass prompt vectors.

4. The zero-shot anomaly detection method based on anomaly superclass semantic guidance according to claim 1, characterized in that, During the model training phase, a correspondence between category features, matching keys, and superclass cue vectors is established through supervised learning between visual features and keys.

5. The zero-shot anomaly detection method based on anomaly superclass semantic guidance according to claim 1, characterized in that, The loss function is the sum of the anomaly detection task loss, the cue word separation loss, and the feature key matching loss.

6. The zero-shot anomaly detection method based on anomaly superclass semantic guidance according to claim 1, characterized in that, The anomaly category labels include one or more of the following: cracks, fractures, gaps, scratches, contamination, color changes, missing parts, assembly errors, and deformation.

7. The zero-shot anomaly detection method based on anomaly superclass semantic guidance according to claim 6, characterized in that, Cracks, fractures, and notches are classified as structural damage superclasses; stains, color changes, and surface contamination are classified as appearance anomalies superclasses; and missing parts and assembly errors are classified as composition anomalies superclasses.

8. The zero-shot anomaly detection method based on anomaly superclass semantic guidance according to claim 1, characterized in that, During the model testing phase, the category features of the corresponding visual deep learning layer are extracted using a visual deep learning network. The similarity between the category features and each matching key is calculated. The superclass cue vector corresponding to the key with the highest similarity is selected and inserted into the visual deep learning layer corresponding to the visual deep learning network. Visual features are extracted together with the visual features. Visual features are encoded under the guidance of anomaly semantics to generate image-level anomaly detection results and pixel-level anomaly localization results.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the zero-sample anomaly detection method based on anomaly superclass semantic guidance as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The device stores computer-executable instructions for performing the zero-sample anomaly detection method based on anomaly superclass semantic guidance as described in any one of claims 1 to 8.