Long tail anomaly detection in images

Abstract

Embodiments of the present disclosure provide a method for anomaly detection in a block of an image. The method comprises: collecting first text encodings of first text prompts in a latent space; collecting second text encodings of second text prompts in the latent space; encoding an image to produce features of the image; segmenting the features of the image into feature blocks; projecting each of the feature blocks into the latent space using a projection operator; and comparing the projection of each of the feature blocks to the first text encodings and the second text encodings to detect an anomaly when the projection of a feature block of the feature blocks is closer to the second text encodings than the first text encodings.

Description

Technical Field

[0001] This disclosure generally relates to the training and use of machine learning systems for image processing, and more specifically to systems and methods for detecting long-tailed anomalies in images with learnable class names. Background Technology

[0002] Anomaly detection techniques are used to identify defective images and locate defects (if any) within the images. These defective images can then be used to identify and remove defective products and / or implement corrective measures to repair them. In the fields of image processing and anomaly detection, traditional methods primarily rely on specialized models tailored to individual object classes to perform anomaly detection. Anomaly detection is a critical problem in many manufacturing settings, such as printed circuit board (PCB) manufacturing, semiconductor manufacturing, automotive parts manufacturing, and other product manufacturing.

[0003] Anomaly detection techniques require models to distinguish anomalous patterns by understanding normal behavior or appearance. To reflect real-world manufacturing constraints, most datasets are curated for unsupervised anomaly detection, where defective images are unusable for training. When models are specifically trained on datasets characterized by defect-free images, their ability to identify and characterize anomalies in real-world scenes is compromised.

[0004] In some cases, model accuracy can be improved by training the model on specialized datasets, such as the MVTEC dataset, which has real-world normal images for training and a combination of normal and anomalous images for testing. However, techniques for training on specialized datasets may still require different models for each image category—that is, different models for each class of objects in the image. This limits the scalability of the model.

[0005] Some anomaly detection methods often lack generalization when faced with novel or anomalous patterns not encountered during training. The consequences of ignoring these defects can be critical in applications such as medical diagnostics, industrial quality control, or safety monitoring, where such defects become particularly apparent.

[0006] Therefore, a generalizable and robust anomaly detection model is needed to overcome the above challenges in order to detect anomalies in various object classes in an efficient and accurate manner. Summary of the Invention

[0007] Some implementations aim to employ anomaly detectors to detect and locate anomalies within images. Another objective of some implementations is to perform long-tailed anomaly detection (LTAD) to detect multiple long-tailed types of defects, independent of dataset class names.

[0008] Some implementations are based on the understanding that anomaly detection models must be able to detect defects across many image classes without relying on hard-coded class names that may be uninformative or inconsistent across datasets. In particular, anomaly detection models may need to be able to learn without supervision and be robust to long-tailed distributions for real-world applications.

[0009] Some implementations are based on the understanding that traditional techniques for using a single model to detect anomalies across different object classes can be grouped into two groups based on the level of the image semantics of the model operations.

[0010] In one example, a model can be used to perform anomaly detection by using a reconstruction technique to project the input image onto a manifold of a normal image. Furthermore, the difference between the input image and its projection onto the manifold is used to detect potential anomalies or defects in the input image.

[0011] In another example, anomaly detection can be performed using a model employing semantic techniques. Semantic techniques can be used to build explicit models for classifying normal and anomalous regions. Since anomalous images are absent from the training data, a model for anomaly classification is trained by leveraging knowledge from a visual language foundational model. The model can then use predefined text prompts to detect normal and anomalous regions. These text prompts can correspond to a normal or anomalous classification and an image class name associated with an image in the training dataset. Prompts can be given as, for example, "[CLASS] normal photo" and "[CLASS] anomalous photo," where [CLASS] is a class name in the dataset (such as the training dataset).

[0012] While reconstruction and semantic techniques enable some generalization to object classes, they have several limitations. For example, models based on reconstruction techniques are required to model the manifolds of complex normal images, especially for a wide variety of types. Furthermore, even when training models on large datasets, the distance between the projection of the input image and the manifold of the normal image may be smaller for certain regions with anomalies. In other words, a model trained using reconstruction techniques on a large dataset of normal images may be inaccurate in detecting anomalies within the input image.

[0013] Furthermore, visual language base models used by semantic-based models can provide clarity by structuring anomaly detection as a binary classification problem. However, solving anomaly detection problems becomes difficult when class names in the dataset are fuzzy or unknown to the visual language base model. Therefore, semantic-based models can rely on the accuracy and reliability of class names learned by the visual language base model for anomaly detection. Consequently, the output of semantic-based models may also be inaccurate and unreliable when learning class names can be difficult.

[0014] Furthermore, models based on reconstruction or semantic techniques cannot generalize to long-tailed settings with skewed sample distributions. Long-tailed distributions of different types of images naturally occur in real-world scenarios such as manufacturing. In long-tailed distributions, different objects or object classes may have very different popularity. In other words, long-tailed distributions can occur when the number of images corresponding to different object classes can vary significantly.

[0015] Some implementations are based on the understanding that existing systems for anomaly detection assume that different image classes are filled equally.

[0016] Some implementations are based on the understanding that in most industrial applications, different objects have different costs, production schedules, etc. This results in a long-tailed distribution, where some types have a higher sample size than others.

[0017] Some implementations are based on the understanding that anomaly detection systems not trained to account for such class imbalances tend to overfit popular types and ignore less popular types.

[0018] Therefore, the purpose of some embodiments of this disclosure is to provide a model for anomaly detection that combines reconstruction techniques and semantic techniques for performing anomaly detection.

[0019] Another objective of some embodiments of this disclosure is to formulate the long-tail anomaly detection (LTAD) problem by introducing several long-tail datasets. Such long-tail datasets can be obtained by resampling current anomaly detection benchmarks with different levels of class imbalance, different imbalance factors, and different types of imbalance. Some embodiments may provide a set of performance metrics for evaluating the long-tail setup.

[0020] Some implementations can provide LTAD methods to detect defects from multiple long-tail classes without relying on dataset class names. In the example, the LTAD method combines anomaly detection with reconstruction and semantic techniques. The LTAD method can implement the reconstruction technique using a transformer-based reconstruction model. On the other hand, the LTAD method can implement the semantic technique using a binary classifier that relies on learned pseudo-class names and a pre-trained visual-language base model.

[0021] In some implementations, the LTAD method involves training the model in two phases. In the first phase, or Phase 1, the model learns pseudo-class names and a variational autoencoder (VAE) for feature synthesis. In this way, the training data is augmented to combat long-tailed datasets. Furthermore, in the second phase, or Phase 2, the model learns the parameters of the LTAD's reconstruction and classification modules.

[0022] Therefore, embodiments of this disclosure provide a computer-implemented method for detecting anomalies in blocks of an image. The method uses a processor coupled with stored instructions implementing the steps of the method. The method includes: collecting a first text encoding of a first text cue in a latent space; collecting a second text encoding of a second text cue in the latent space; encoding an image to generate features of the image; and segmenting the features of the image into feature blocks. The method further includes projecting each feature block in the feature blocks into the latent space using a projection operator, and comparing the projection of each feature block in the feature blocks with the first text encoding and the second text encoding to detect an anomaly when the projection of a feature block in a plurality of feature blocks is closer to the second text encoding than the first text encoding. The projection operator is trained to project normal feature blocks of a normal image closer to the first text encoding than the second text encoding, while simultaneously projecting noisy feature blocks of a normal image closer to the second text encoding than the first text encoding.

[0023] According to some implementations, the image encoder is trained to encode the global features of an image into a latent space shared by the text encoder and the image encoder of the visual language base model.

[0024] According to some implementations, the method further includes collecting multiple normal images associated with a class, encoding the multiple normal images using an image encoder to generate features of the multiple normal images, processing the features of the multiple normal images conditionally with a pseudo-class name associated with the class using an image decoder, and training the image decoder to learn the pseudo-class name as a first text encoding. In the example, the pseudo-class name is a first text hint.

[0025] According to some implementations, the method further includes: obtaining the encoding of a pair of contradictory class names in a latent space using a text encoder; obtaining features of multiple normal images using an image encoder; segmenting the features of the multiple normal images; introducing noise into at least some of the segmentation features of the multiple normal images to generate anomalous features; and training a projection operator to project the segmentation features and anomalous features of the multiple normal images into the latent space. In one example, the pair of contradictory class names includes a first text cue and a second text cue. Furthermore, the segmentation features of the multiple normal images are closer to the first text cue, and the anomalous features are closer to the second text cue.

[0026] According to some implementations, the method also includes using a reconstruction model to reconstruct the projection segmentation features of multiple normal images and the features of abnormal images, generating reconstruction loss and semantic loss based on the reconstruction and the encoding of the contradictory class names, and retraining the projection operator and the visual language base model to minimize the semantic loss and reconstruction loss.

[0027] According to some implementations, the reconstructed model is a transformer.

[0028] According to some implementations, the image decoder is trained to learn the encoding of multiple pseudo-class names of multiple classes in the latent space.

[0029] According to some implementations, the training datasets used for image encoders, text encoders, and image decoders include multiple images of multiple classes in a long-tailed distribution.

[0030] According to some implementations, an image encoder is a deep neural network comprising a sequence of layers, wherein each layer in the sequence of layers generates image features, and wherein the features of the image are formed by combining image features from different layers.

[0031] According to some implementations, the method further includes determining the dot product between the projection of the feature block and the first text code to generate a first score, determining the dot product between the projection of the feature block and the second text code to generate a second score, and detecting anomalies in the feature block based on the first score and the second score.

[0032] According to some implementations, the first text prompt is the semantic name of the image's class, and the second text prompt is a modification of the first text prompt.

[0033] According to some implementations, the first text hint is the semantic name of the image's class, and the second text hint is a concatenation of modifier words and the semantic name of the image's class.

[0034] According to some implementations, the first text prompt is the semantic name of the image class of the image being learned to generate images using a visual language base model.

[0035] According to some implementations, the method further includes segmenting the image into blocks corresponding to feature blocks, reconstructing each feature block of the image into an image block using a reconstruction model, comparing the reconstructed image blocks with corresponding partitions of the feature blocks to generate a reconstruction score, and detecting anomalies based on the reconstruction score.

[0036] According to some implementations, the method further includes capturing the result of comparing the projection of each feature block with a first text encoding and a second text encoding as a semantic score, combining the semantic score with the corresponding reconstruction score to produce a combined score, and detecting anomalies based on the combined score.

[0037] In another embodiment, this disclosure provides a system for detecting anomalies in blocks of an image. The system includes a processor and a memory having instructions stored thereon that cause the processor to: collect a first text encoding of a first text cue in a latent space; collect a second text encoding of a second text cue in the latent space; encode the image to generate features of the image; segment the features of the image into feature blocks; and project each feature block in the feature blocks into the latent space using a projection operator. In an example, the projection operator is trained to project normal feature blocks of a normal image closer to the first text encoding than the second text encoding, while simultaneously projecting noisy feature blocks of a normal image closer to the second text encoding than the first text encoding. The instructions also cause the processor to compare the projection of each feature block in the feature blocks with the first and second text encodings to detect an anomaly when the projection of a feature block in a plurality of feature blocks is closer to the second text encoding than the first text encoding.

[0038] In another embodiment, this disclosure provides a non-transitory computer-readable storage medium containing a program executable by a processor to perform a method. The method includes: collecting a first text encoding of a first text cue in a latent space; collecting a second text encoding of a second text cue in the latent space; encoding an image to generate features of the image; segmenting the features of the image into feature blocks; and projecting each feature block in the feature blocks into the latent space using a projection operator. The projection operator is trained to project normal feature blocks of a normal image closer to the first text encoding than the second text encoding, while simultaneously projecting noisy feature blocks of a normal image closer to the second text encoding than the first text encoding. The method further includes comparing the projection of each feature block with the first and second text encodings to detect an anomaly when the projection of a feature block in a plurality of feature blocks is closer to the second text encoding than the first text encoding.

[0039] Other features and advantages will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description.

[0040] The currently disclosed embodiments will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, but generally focus on illustrating the principles of the currently disclosed embodiments. Attached Figure Description

[0041] [ Figure 1 ]

[0042] Figure 1 A schematic diagram of an anomaly detection in an image depicting a system for anomaly detection according to some embodiments of the present disclosure is shown.

[0043] [ Figure 2 ]

[0044] Figure 2 A block diagram of a system for anomaly detection according to some embodiments of the present disclosure is shown.

[0045] [ Figure 3 ]

[0046] Figure 3 A graphical representation of the encoding of a pair of contradictory classifiers in a latent space is shown according to some exemplary embodiments of the present disclosure.

[0047] [ Figure 4 ]

[0048] Figure 4 A schematic diagram depicting the first training phase of a system for anomaly detection according to some example embodiments of the present disclosure is shown.

[0049] [ Figure 5 ]

[0050] Figure 5 A schematic diagram depicting the second training phase of a system for anomaly detection according to some example embodiments of the present disclosure is shown.

[0051] [ Figure 6 ]

[0052] Figure 6 A schematic diagram depicting the second training phase of a system for loss-based anomaly detection according to some example embodiments of the present disclosure is shown.

[0053] [ Figure 7 ]

[0054] Figure 7 A flowchart is shown of a method for training a system for anomaly detection according to some other exemplary embodiments of this disclosure.

[0055] [ Figure 8 ]

[0056] Figure 8 A block diagram is shown depicting an anomaly detected in an image by a system according to some example implementations.

[0057] [ Figure 9 ]

[0058] Figure 9 A flowchart is shown of a method for detecting anomalies in an image according to some example embodiments of the present disclosure.

[0059] [ Figure 10 ]

[0060] Figure 10Use case implementations of the system according to some example embodiments of this disclosure are shown.

[0061] [ Figure 11 ]

[0062] Figure 11 Use case implementations of the system according to some other example embodiments of this disclosure are shown.

[0063] [ Figure 12 ]

[0064] Figure 12 An overall block diagram of a system according to some example embodiments of the present disclosure is shown. Detailed Implementation

[0065] While the accompanying drawings illustrate the embodiments currently disclosed, other embodiments are contemplated, as pointed out in the discussion. This disclosure presents illustrative embodiments by way of representation and not limitation. Those skilled in the art can devise many other modifications and embodiments falling within the scope and spirit of the principles of the embodiments currently disclosed.

[0066] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, systems and methods are shown only in block diagram form to avoid obscuring this disclosure.

[0067] As used in this specification and claims, the terms “for example,” “like,” and “such as,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, when used in conjunction with a list of one or more parts or other items, are each interpreted as open-ended, meaning that the list is not considered to exclude other additional parts or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0068] Overview

[0069] Anomaly detection refers to the identification of patterns or instances that deviate from the norms or expected behavior of a given dataset. The primary goal of anomaly detection is to identify data points exhibiting unusual characteristics that potentially indicate errors, defects, or anomalous conditions. Anomaly detection plays a crucial role in various industries and applications, helping to identify outliers, irregularities, or unexpected events that may have significant impact.

[0070] The embodiments of this disclosure are based on the understanding that anomaly detection is complex and challenging in manufacturing environments where a large number of object classes are produced and the distribution of objects across different classes is highly skewed. Some embodiments are also based on the understanding that anomaly detection in manufacturing environments can help detect, prevent, or analyze mechanical failures. Some applications of anomaly detection in manufacturing environments may include, but are not limited to, detecting defects in products, identifying equipment malfunctions, and identifying processing deviations that may lead to problems.

[0071] Unsupervised anomaly detection (AD) methods aim to identify and locate defects in images without observing any defective images during training. These AD methods can be categorized into three types.

[0072] The first category includes methods that use a different model for each image class or category. In these methods, Algorithm Difference (AD) is performed based on the prediction difference between a pre-trained encoder and a target encoder trained to match the predictions of the pre-trained encoder. In some cases, the method may fit a Gaussian distribution to the feature vectors of normal images and use out-of-distribution criteria to perform AD. In some other cases, reconstruction techniques can be used to train a model to reconstruct normal samples and use the reconstruction error of AD. However, using a model for each category is expensive, has scalability issues and model overhead, and provides limited anomaly generalization and failure or difficulty when identifying global anomalies.

[0073] Furthermore, the second category includes methods that improve reconstruction-based models by using neighborhood attention masks to avoid information leakage by providing interpretations of defective regions using large visual language base models. These methods do not use any class names to detect anomalies when leveraging visual language base models. Reconstruction-based models (especially those operating in an unsupervised manner without class names) face several challenges. Reconstruction-based models (especially simpler architectures such as autoencoders) may fail to capture highly complex patterns present in the training data. In real-world scenarios, anomalies can include a mixture of rare events and variations within normal data. Reconstruction-based models without class information may fail to distinguish these situations, leading to false positives or ignoring subtle anomalies. Therefore, they may fail to accurately reconstruct anomalies that deviate significantly from normal patterns or have patterns significantly different from the training dataset, resulting in detection failures. Furthermore, since reconstruction-based models are trained on normal instances, class-agnostic reconstruction-based models may lack the ability to generalize well to novel anomalies. Additionally, class imbalance in the training data can pose challenges during training, as reconstruction-based models may prioritize learning the majority class, resulting in suboptimal anomaly detection performance for the minority class.

[0074] The third category includes methods that use a single model for all classes but require class names to compute anomaly scores. These methods compute anomaly scores by measuring the similarity between the text feature vectors of several predefined normal and anomalous text prompts and the input image. In some cases, auxiliary or specialized training data can be used to train and test the model. While incorporating class information during model training can provide valuable context, it introduces complexity due to factors such as class imbalance, subjectivity in class definitions, labeling challenges, failure to adapt to dynamic and evolving anomalies, overlap between classes, limited generalization to unknown classes, and dependence on class definitions.

[0075] The embodiments of this disclosure are based on the understanding that previously discussed anomaly detection methods assume a balanced dataset for training, where the number of samples is relatively balanced across different classes. However, this is an unlikely setting for real-world applications, where different objects tend to have different popularity.

[0076] Therefore, this disclosure provides an LTAD method for anomaly detection across multiple object classes using a single model. The LTAD method also addresses the challenges of imbalanced training sets and missing class names. For example, the LTAD method combines AD with reconstruction and semantic techniques to perform multi-class AD. The LTAD method overcomes the challenges caused by ambiguous class names by learning class names consistent with the semantic space of the visual language underlying model. Furthermore, this disclosure provides a training strategy for a single model (hereinafter also referred to as an LTAD-based model) used to train the LTAD method, which uses data augmentation processes to address the data scarcity of long-tail data and learn class names. Subsequently, the LTAD-based model performs accurate anomaly detection, especially for long-tail AD. Moreover, the LTAD method generalizes the LTAD-based model across various datasets and imbalanced configurations to generate accurate and reliable output for anomaly detection.

[0077] Figure 1 A schematic diagram 100 depicting an anomaly detection in image 102 using a system 106 for anomaly detection according to some embodiments of the present disclosure is shown. In an example embodiment, image 102 may correspond to an object depicted as a bottle. However, such depiction of an object should not be construed as limiting. For example, image 102 may relate to other objects such as machinery, food, packaged articles, clothing, chemical agents, etc. In another example, image 102 may relate to a person, such as an instruction for an activity performed by a person. Image 102 can be captured from an image capturing device (e.g., a camera). Figure 1 (Not shown in the image)

[0078] In the example, system 106 can receive multiple images, which may include different objects of the same object class, different objects of different object classes, or different human poses of the same or different people. Furthermore, in manufacturing, the sample distribution (i.e., multiple images across different object classes) is skewed. In other words, different objects can have very different popularity. This leads to the problem of long-tailed adversarial behavior (AD) due to long-tailed datasets.

[0079] Typically, reconstruction-based models may have to perform modeling of complex manifolds, especially for problems requiring many classes, and may fail to generate accurate output for anomaly detection. Furthermore, semantic-based models can use a visual language base model to identify class names, but these identified class names may be ambiguous or unknown to the base model. For example, ambiguity can arise because the class name "bottle" refers to visually different concepts in the first dataset (where it means "bottom of the bottle") and the second dataset (where it means "side of the bottle"). Therefore, the "bottle" label in the first dataset may not accurately provide information to the base model, which might associate the image with alternative labels (e.g., "black sphere"). Sometimes, the class name may simply be unknown to the base model.

[0080] Some implementations of this disclosure are based on the understanding that the visual language base model should learn class names that are best aligned with the images in the training dataset.

[0081] The Visual Language Foundation Model (hereinafter referred to as the Foundation Model) is an artificial intelligence model capable of understanding and processing visual and textual information. It integrates computer vision capabilities for interpreting images with natural language processing (NLP) capabilities for understanding textual information. The Foundation Model is designed to bridge the gap between visual and linguistic information, enabling a more comprehensive understanding of multimodal data.

[0082] In the example, visual language base models can be pre-trained on large datasets containing pairs of images and text, allowing them to learn representations that capture the interactions between visual and linguistic elements. The pre-trained visual language base models can then be fine-tuned for specific downstream tasks. In the context of anomaly detection, visual language base models can be used to enhance the capabilities of anomaly detection methods. For example, the pre-training process of visual language base models enables them to learn semantic representations. These semantic representations capture high-level features and relationships within the data, allowing them to effectively identify anomalies that may exhibit subtle patterns. To this end, visual language base models can capture contextual information to understand the relationship between objects in an image and their corresponding textual descriptions.

[0083] To address the aforementioned challenges, this disclosure provides a system 106 for performing LTAD. System 106 combines AD with reconstruction and semantic techniques. LTAD system 106 is configured to detect anomalies 108 in blocks of image 102.

[0084] In operation, system 106 is configured to collect a first text encoding 110 of a first text prompt and a second text encoding 112 of a second text prompt in latent space 116. In the example, the first and second text prompts are class names learned by the text encoder of the visual language base model during the training phase. Specifically, the text encoder of the visual language base model is configured to process and encode textual information into numerical representations projected into latent space 116. In particular, the first and second text prompts are contradictory class names. In the example, if the first text prompt is "bottle," then the second text prompt is either "damaged bottle" or "deformed bottle." Figure 4 , Figure 5 and Figure 6 The method by which System 106 learns these class names (i.e., the text encoding of the text prompts) is described.

[0085] Furthermore, system 106 is configured to encode image 102 to produce features of image 102. In the example, in the multimodal visual language base model, the text encoder works in conjunction with a visual or image encoder to process textual and visual inputs. For this purpose, the image encoder is configured to encode image 102 to extract features. In the example, the image encoder takes image data of image 102 as input. This can be in the form of raw pixel values ​​or feature maps generated by a pre-trained convolutional neural network (CNN). For example, the image encoder can perform hierarchical feature extraction from the raw pixel values ​​of image 102. For this purpose, the image encoder of the visual language base model can extract relevant features from image 102.

[0086] Furthermore, system 106 is configured to segment the features of image 102 into feature blocks (described as feature blocks 104A, 104B, 104C, 104D, and 104E, and collectively referred to as feature blocks 104). In this example, feature blocks 104 are generated by dividing the extracted features of image 102 into meaningful segments or regions. The stripes that segment the features into feature blocks 104A, 104B, 104C, 104D, and 104E based on regions are merely exemplary and should not be construed as limiting. In other examples, for instance, grid-based segmentation, object-based segmentation, semantic segmentation, pixel-based segmentation, interest-point-based segmentation, region-based segmentation, and texture-based segmentation can be used to segment the extracted features of image 102 to generate feature blocks.

[0087] Subsequently, system 106 is configured to project each feature block 104 into latent space 116 using projection operator 114. In this example, projection operator 114 is configured to create an AI model with a joint multimodal representation in latent space 116 using a text encoder. The joint multimodal representation captures the relationship between visual elements (i.e., feature block 104 of image 102) and text elements (i.e., first text encoding 110 and second text encoding 112). In this way, anomaly detection is performed in a multimodal context.

[0088] In particular, the projection operator 114 is trained to project normal feature blocks of a normal image closer to the first text code 110 than the second text code 112. It can be noted that the first text code 110 is a representation of a first text hint for the "normal" features of an object class. Subsequently, the first text hint indicates the class name corresponding to the normal class. Conversely, the second text code 112 is a representation of a second text hint for the "abnormal" or "anomaly" features of an object class. Subsequently, the second text hint indicates the class name corresponding to the anomalous or anomalous class.

[0089] For example, feature blocks 104A and 104B may correspond to normal features of image 102. In other words, the features of the object (i.e., the bottle) within the region corresponding to feature blocks 104A and 104B do not have any abnormalities or defects. Therefore, feature blocks 104A and 104B are projected to be closer to the first text code 110 of the first text prompt than the second text code 112. Furthermore, feature blocks 104C, 104D, and 104E may correspond to abnormal or noisy features of image 102. The features of the object (i.e., the bottle) within the region corresponding to feature blocks 104C, 104D, and 104E have one or more abnormalities or defects. Therefore, the noisy feature blocks 104C, 104D, and 104E of image 102 are projected to be closer to the second text code 112 of the second text prompt than the first text code 110.

[0090] It can be noted that the first text hint in the Normal class represents the category of instances or data points that are considered standard, typical, or expected. The Normal class represents the majority of instances that conform to standard behavior or patterns. Alternatively, the second text hint, or the Anomalous class, includes instances that deviate from the standard behavior or patterns observed in the first text hint or the Normal class. Anomalous instances are typically rare events, outliers, or instances exhibiting anomalous characteristics. Therefore, the first and second text hints are referred to as contradictory class names.

[0091] Subsequently, system 106 is configured to compare the projection of each feature block 104 with a first text code 110 and a second text code 112 to detect anomalies 108. Specifically, feature blocks (such as feature blocks 104C, 104D, and 104E) are closer to the second text code 112 than to the first text code 110 from the projection of feature block 104. This is because the second text code 112 corresponds to a second text cue, namely, the “anomaly class name.” Anomaly detection is then performed by identifying one or more feature blocks that are closer to the second text code 112 than the first text code 110. According to this example, feature blocks 104C, 104D, and 104E are close to the second text code 112 in the latent space 116, therefore feature blocks 104C, 104D, and 104E are identified as anomalies.

[0092] In the example, the identified anomaly feature blocks 104C, 104D, and 104E can be used to initiate the execution of some downstream tasks. Examples of downstream tasks may include, but are not limited to, actions to stop product production, triggering machine shutdown, evaluating the cause of the anomaly, evaluating the type of anomaly, recommending corrective actions, and generating warnings.

[0093] In some cases, all feature blocks of an image of an object (such as a bottle) may be closer to the first text code 110 or normal class name used to represent the normal features of the object or bottle. In this case, image 102 can be identified as normal or anomalous. In this case, system 106 can continue to perform anomaly detection on another image corresponding to another object belonging to the same class, another view of the same object, or another object belonging to a different class.

[0094] Figure 2 A block diagram 200 of a system 106 for anomaly detection according to some embodiments of the present disclosure is shown. System 106 includes an input interface 202, a memory 204, a processor 206, and an output interface 208. The input interface 202 is configured to accept input data. The input data may include images, videos, or image sequences, for example... Figure 1 Image 102.

[0095] Memory 204 is configured to store a visual language foundation model 210 and a projection operator 114. The visual language foundation model 210 (hereinafter referred to as Model 210) includes an image encoder 212 and a text encoder 214. The image encoder 212 and text encoder 214 are configured to extract meaningful representations from visual data (such as image 102) and text data (such as a first text cue and a second text cue), respectively. The use of both the image encoder 212 and the text encoder 214 enables Model 210 to understand multimodal information, thereby combining insights from both images and text. The image encoder 212 typically processes visual information in the form of images and extracts relevant features. Furthermore, the text encoder 214 processes textual information, i.e., text cuees corresponding to different class names, and converts them into digital representations that capture semantic content.

[0096] In the example, image encoder 212 is trained to encode global features of image 102 into latent space 116. Latent space 116 is shared by image encoder 212 and text encoder 214 of model 210. To this end, model 210 combines representations from image encoder 212 and text encoder 214 to create a joint or multimodal representation referred to as latent space 116. In this way, model 210 captures interactions between modalities. The joint representation, or latent space 116, captures the interactions and relationships between visual and textual elements, allowing model 210 to understand how images and text complement each other. By combining image and textual information, model 210 gains a more comprehensive understanding of the input data or image 102. In some cases, image encoder 212 is trained to encode feature blocks 104 of global features of image 102 into latent space 116. For example, feature block 104 could be segmentation global features of image 102. Image encoder 212 is configured to encode and project feature block 104 in latent space 116, wherein text encoder 214 has encoded and projected first text code 110 and second text code 112.

[0097] Processor 206 is configured to collect a first text code 110 of the first text prompt and a second text code 112 of the second text prompt in the latent space 116. Processor 206 can collect the first text code 110 and the second text code 112 from text encoder 214. Furthermore, processor 206 is configured to encode image 102 using image encoder 212 to generate features of image 102. Additionally, processor 206 is configured to segment the features of image 102 into feature blocks 104.

[0098] The latent space 116 corresponds to a joint representation of visual and textual features generated by the image encoder 212 and the text encoder 214. This enables semantic anomaly detection. In this regard, the first text cue and the second text cue are encoded in the latent space 116 as a first text code 110 and a second text code 112, respectively. The first text code 110 and the second text code 112 can be classifier weight vectors that form the representations of normal and anomalous text cuees in the latent space 116.

[0099] Furthermore, using the image encoder 212 and projection operator 114 of the visual language base model 210, the processor 206 is configured to project each feature block 104 into the latent space 116. For example, the projection operator 114 is configured to project anomalous or abnormal feature blocks, such as feature blocks 104C, 104D, and 104E that are closer to the second text encoding 112 than the first text encoding 110. On the other hand, the projection operator 114 is configured to project normal feature blocks, such as feature blocks 104A and 104B that are closer to the first text encoding 110 than the second text encoding 112.

[0100] Furthermore, processor 206 is configured to detect anomalies in feature block 104 based on the projection of feature block 104 into latent space 116. Specifically, processor 206 can compare whether the feature block is closer to the second text code 112 or the first text code 110 in the latent space. In this regard, the processor is configured to determine the dot product between the projection of feature block (e.g., feature block 104A) and the first text code 110 to generate a first score. Subsequently, processor 206 is configured to determine the dot product between the projection of feature block 104A and the second text code 112 to generate a second score. Based on the comparison between the first score and the second score of feature block 104A, it is determined whether feature block 104A is an anomaly or normal. For example, if the first score (i.e., the distance of feature block 104A from the first text code 110) is greater than the second score (i.e., the distance of feature block 104A from the second text code 112), then feature block 104A is determined to be an anomaly. For example, a first score greater than a second score can indicate that feature block 104A tends toward a second text prompt or an abnormal text prompt, thereby identifying feature block 104A as abnormal. Alternatively, if the first score is less than the second score, it can indicate that feature block 104A tends toward a first text prompt or a normal text prompt, thereby identifying feature block 104A as normal.

[0101] In the example, processor 206 is configured to generate an anomaly score for image 102 based on the difference between the input image 102 and the projection results of each feature block 104 of image 102. The anomaly score can indicate or quantify the level of anomalous behavior in image 102. In this way, the degree of anomaly in the object or bottle associated with image 102 is determined.

[0102] Furthermore, processor 206 is configured to generate an anomaly score for image 102 based on identified anomalous feature blocks from feature block 104. The anomaly score and / or the identified anomalous feature blocks can be provided to output interface 208 to present anomaly 108 or anomaly detection results. Anomaly detection results can correspond to the detection and localization of anomalies in image 102. In some example embodiments, processor 206 is also configured to output a notification based on the detected anomaly 108. The notification can be provided to a human via output interface 208.

[0103] In some implementations, to address the long-tail setting, an initial training phase is performed for data augmentation. The pre-elimination training phase includes learning an autoencoder or variational autoencoder (VAE) and then using it to synthesize features for class names. To make class name sensitive, the VAE is tuned by the textual encoding of class names according to model 210. Furthermore, to address class name ambiguity, a set of learnable class cues is learned via backpropagation during VAE training. In the second training phase, the projection operator 114 is trained using a mixture of real and synthetic examples. Figure 4 , Figure 5 , Figure 6 and Figure 7 Describe the details of training the projection operator 114.

[0104] According to some implementations, system 106 uses reconstruction methods and semantic methods to perform anomaly detection. Anomaly detection via reconstruction methods is achieved by combining image encoder 212 and projection operator 114, which is trained to project feature blocks 104 into a latent space 116. The latent space 116 includes a trained normal image manifold.

[0105] Figure 3 A graphical representation 300 depicting the encoding of a pair of contradictory classifiers in a latent space 116 according to some example embodiments of the present disclosure is shown.

[0106] The latent space 116 includes projections or encodings of text cues or class names. For example, the latent space 116 may include projections of multiple pairs of contradictory classes. For the object class "bottle", a pair of contradictory classes may include, for example, a first text cue as a normal classifier and a second text cue as an anomalous classifier. In a similar manner, different pairs of contradictory classes may correspond to normal and anomalous classifiers for different object classes. During training, projections of multiple pairs of contradictory types into the latent space 116 are learned. Furthermore, during training, normal image manifolds for different object classes can also be created in the latent space 116.

[0107] Furthermore, during inference, feature blocks 104 of image 102 are projected into latent space 116. For example, projection operator 114 is trained to compute the projection of each feature block 104 into latent space 116 of model 210, where classifier parameters for text prompts (such as first text prompts and second text prompts) are defined.

[0108] According to this example, the projection of the first text prompt in the latent space 116 is depicted as first text prompt projection 302A, and the projection of the second text prompt in the latent space 116 is depicted as second text prompt projection 302B. Furthermore, the projections of feature block 104 are depicted as feature projections 304A, 304B, 304C, 304D, and 304E. In this example, feature projection 304A corresponds to feature block 104A, feature projection 304B corresponds to feature block 104B, feature projection 304C corresponds to feature block 104C, feature projection 304D corresponds to feature block 104D, and feature projection 304E corresponds to feature block 104E.

[0109] The projection operator 114 is trained to project feature blocks 104 as feature projections 304A, 304B, 304C, 304D, and 304E into the latent space 116 for projected text prompts. For example, since feature blocks 104A and 104B correspond to regions of image 102 that do not have any anomalies, feature projections 304A and 304B are projected to be closer to the first text prompt projection 302A than the second text prompt projection 302B. On the other hand, since feature blocks 104C, 104D, and 104E correspond to regions of image 102 that include anomalies or defects, feature projections 304C, 304D, and 304E are projected to be closer to the second text prompt projection 302B than the first text prompt projection 302A.

[0110] To determine whether a feature block (e.g., feature block 104A) is abnormal, a first score and a second score of feature block 104A are compared. Specifically, the first score is determined based on the dot product of projection 304A of feature block 104A and the first text prompt projection 302A; and the second score is determined based on the dot product of projection 304A and the second text prompt projection 302B. Based on the first and second scores, feature block 104A is determined to be either abnormal or normal.

[0111] Training Overview

[0112] Figure 4 A schematic diagram 400 is shown depicting the first training phase of a system 106 for anomaly detection according to some other exemplary embodiments of the present disclosure. According to the present disclosure, the training of system 106 is performed in two phases.

[0113] In particular, the first training phase corresponds to class-sensitive data augmentation. The goal of the first training phase is to overcome the data scarcity in the long-tailed training dataset 402 by augmenting the training dataset 402 with normal examples from the minority classes and anomalous examples from all classes. Another goal of the first training phase is to learn the class-sensitive textual cues required by the semantic methods used for anomaly detection. .

[0114] In the example, processor 206 is configured to collect multiple normal images associated with class c and store them as training dataset 402. In some cases, training dataset 402 may include multiple normal images corresponding to multiple different classes. Processor 206 is configured to encode the multiple normal images using image encoder 212 to generate features of the multiple normal images. In the example, for class c from training dataset 402... Normal image The pre-trained image encoder 212 E extracts features as a feature tensor. The features are extracted as a feature tensor. And potential code z.

[0115] In the example, the image encoder 212 is a deep neural network comprising a sequence of layers. Furthermore, each layer in the layer sequence generates image features. Additionally, the image features 404 from different layers are combined to form features of the considered normal image. In the example, the image encoder 212 may have L layers. The feature tensor and latent code can be defined as: and

[0116] In this respect, the latent code z is a feature vector from the last layer (L) of the image encoder 212. The image encoder 212 uses feature tensors and the latent code to generate image encoding 406 or features of a normal image from the training dataset 402 based on features generated by each layer in the layer sequence. Image encoding 406 can indicate features of a normal image. In particular, the image encoder 212 is pre-trained and its weights are frozen. During the training phase, the image encoder 212 can generate image encoding 406 of a normal image by combining image features 404 from different layers.

[0117] Furthermore, the first training phase includes training image decoder 408, D. In this example, image decoder 408 is a VAE-style decoder. Image decoder 408 is learned or trained for feature enhancement of image features conditioned on different pseudo-class names (such as pseudo-class name 414). For example, the architecture of image decoder 408 is the same as that of image encoder 212.

[0118] Processor 206 is configured to process features of multiple normal images conditioned on pseudo-class name 414 using image decoder 408. Image decoder 408 is trained to sample feature vectors of normal images based on VAE-based operations. In the example, from parameters... and normal distribution Latent features of sampled normal images ,in and This is the learned linear transformation. For example, pseudo-class name 414 is the first text prompt. Pseudo-class name 414 is provided to text encoder 214. In this case, text encoder 214 generates the first text code 110 in latent space 116 based on the pre-training of base model 210. In this case, the features of the normal image are processed by adjusting the features on the first text prompt (i.e., the normal classifier). In this way, a normal image manifold corresponding to the first text prompt is generated in latent space 116.

[0119] Processor 206 is also configured to train image decoder 408 to learn pseudo-class name 414 as first text encoding 110. In this respect, image decoder 408 is configured to use latent features. To synthesize feature tensors. Specifically, the layer sequence of the image decoder 408 is configured to generate image features 410, which are then combined to generate synthetic features of the normal image. This is represented as... The first text encoding 110 and the ground truth features generated by the image encoder 212 are fed into the image decoder 408 to generate synthetic features. The synthetic features are provided to the image encoder 212 as feedback 412. In particular, the synthetic features generated by the image decoder 408 and the ground truth features generated by the image encoder 212 are compared to generate the MSE loss and the image decoder 408 is retrained.

[0120] In the long-tailed dataset, the performance of image decoder 408 degrades for classes in training dataset 402 that have few training images. Specifically, the training dataset 402 for image encoder 212, text encoder 214, and image decoder 408 includes multiple images of multiple classes in a long-tailed distribution. In other words, multiple images can include different numbers of images belonging to different classes, such that some classes may be majority due to having a large number of images associated with them, while other classes may be minority due to having a very small number of images associated with them.

[0121] To improve this problem, the image decoder 408 is trained to have prior knowledge about class names, in the form of prototype features derived from the text, i.e., the first text encoding. 110. The first text code 110 represents the class c used for feature synthesis of the class. The first text code 110 can be represented by a pseudo-class name. (Right now, The text encoder 214 of the prompt model 210 is used to obtain the first text encoding 110. Then, the first text encoding 110 is combined with image-related latent features. Cascaded to create or recreate the input to image decoder 408. Based on the input to image decoder 408, a synthesized image has a dimension equal to... Feature tensor Following standard VAE training practices, the image decoder 408 and pseudo-class name 414 are learned by optimizing the loss function. The loss function is defined as:

[0122] For example, the loss function is a combination of mean squared error (MSE) loss and KL-divergence loss to implement a normal distribution in the latent space 116. The MSE loss is applied to the features extracted by the image encoder 212. and features synthesized by image decoder 408 Between. The text encoder 214 and image encoder 212 are associated with the pre-trained base model 210 and remain frozen throughout training. This is intended to enable the learning of an image decoder 408, which can be used to synthesize features from tail classes (i.e., classes with a very small number of images in the training dataset 402). Furthermore, by keeping the text encoder 214 and image encoder 212 frozen, the features of the training images are linked to the semantic representations generated by the text encoder 214 of model 210. Alignment. Furthermore, by utilizing this alignment, the quality of feature synthesis for the tail class is improved. In this way, the image decoder 408 is trained to learn the first text encoding 110 corresponding to the pseudo-class names in the latent space 116. Subsequently, the image decoder 408 is trained to learn encodings of multiple pseudo-class names for multiple classes within the latent space 116. After training, the learned cues, such as the first text cues, are used in the second training phase and the inference phase. .

[0123] In the example, the first stage of training is performed over 100 epochs using the Adam optimizer with a learning rate of 1e-4. Furthermore, a pre-trained visual language base model 210 is used to train the projection operator 114 in the second training stage. For this purpose, each input image from the training dataset 402 is scaled to 224. 224 pixels, and extract image features 404 from the layer sequence of the image encoder 212 of model 210. In the example, the pseudo-class name 414 has a default length of 2 and is initialized with the text "object-object".

[0124] Figure 5 A schematic diagram 500 depicts a second training phase of a system 106 for anomaly detection according to some other exemplary embodiments of the present disclosure. When the first training phase is complete, the image decoder 408 operates as a data augmentation device to generate synthetic feature tensors. or in the characteristic tensor Image features in the semantic neighborhood of the training dataset 402, or image features 404 extracted from real images from the training dataset 402. This is used to augment the training dataset 402 online during the second training phase.

[0125] According to embodiments of this disclosure, two types of data augmentation are performed, such as long-tail class augmentation and anomaly augmentation. To counteract the imbalanced nature of the long-tail training dataset, data augmentation is performed by selecting datasets with probabilistic... or Data augmentation is achieved using real image features 404 or synthetic image features 410. For example, image features 404 and 410 are layer-wise features. These layer-wise features are aggregated or concatenated 502 to produce a single block feature vector, such as normal block features 504 for a normal image. To compensate for the lack of anomalies during training, random noise 506 (sampled from a normal distribution) is added to normal block features 504 to produce pseudo-anomaly block features 508. This process is repeated for all normal blocks during training. Random noise is not added during inference.

[0126] In the example, the probability of selecting the true image feature 404. It is 0.5. For different training datasets, such as MVtec, VisA, and DAGM, the hyperparameter... The values ​​are set to 500, 400, and 300 respectively. It should be noted that using images from the MVtec, VisA, and DAGM datasets to form training dataset 402 is merely exemplary. In some cases, other datasets can be used. In some other cases, training dataset 402 can be generated.

[0127] In the example, the feature vector f corresponding to the normal images from training dataset 402 is segmented into Each block feature vector, i.e., width and height The blocks. For example, the global features of a normal image are segmented into blocks, such that the sum of the features of the blocks or the feature vectors of the normal blocks is defined as the normal block feature 504. In this paper, the superscript 'n' indicates that these are normal features or normal feature vectors.

[0128] Furthermore, anomalous data augmentation is performed to compensate for the lack of anomalous-related images during training, specifically the lack of anomalous or abnormal images in the training dataset 402. In this regard, random noise 506 is added to the normal block features. 504 generates pseudo-abnormal block feature 508.

[0129] Furthermore, data augmentation is used in the second training phase to learn the parameters of the reconstruction model (RM) 510. In the example, the reconstruction model 510 is implemented as a transformer, such as an RM transformer. Furthermore, the RM transformer is configured to reconstruct image features based on the image's projection into the latent space 116. Additionally, data augmentation is used in the second training phase to learn or train the projection operator. 114, to project image features from the block into the latent space 116 of model 210. Combined Figure 6 The training of the data augmentation-based projection operator 114 is described in detail.

[0130] In the example, RM 510 is an anomaly detection model that operates based on the principle of reconstructing input data, such as image features. During training, RM 510 is trained to learn a compressed representation (encoding) of normal image features 404, which is then used to reconstruct the input data. Subsequently, anomalies in the image that differ from the learned normal pattern result in higher reconstruction errors, making them detectable. Examples of transformers used as the reconstruction model 510 may include, but are not limited to, bidirectional encoder representations from a transformer (BERT), Generate Predictive Text (GPT), and a text-to-text transfer transformer (T5).

[0131] According to embodiments of this disclosure, model 210 is trained to incorporate pseudo-anomaly block features. 508 projected onto the reconstructed block features generated by RM510 In 512. In the example, the pseudo-abnormal block feature 508 is projected onto the normal block feature. In a manifold of 504, projection is implemented using an RM transformer or RM 510 to minimize the reconstruction loss 514. The reconstruction loss 514 is defined as:

[0132] Figure 6 Another schematic diagram 600 depicting a second training phase of a system 106 for loss-based anomaly detection, according to some other exemplary embodiments of this disclosure, is shown. In particular, the second phase of training the system 106 for LTAD training includes learning the parameters of RM 510 (such as...). Figure 5(as described in the previous section) and the parameters of the projection operator 114. The parameters of the projection operator 114 are used to map the visual features of the block to the latent space 116 of the semantic AD (SAD) module or model 210. For example, data augmentation is used to train the projection operator. 114 is used for semantic block projection in the latent space 116 of model 210.

[0133] During the second training phase, processor 206 is configured to use text encoder 214 to obtain the encoding of a pair of contradictory class names in latent space 116. This pair of contradictory class names includes a first text cue 602 and a second text cue 604. It can be noted that classifier parameters or dimensions of the class names, such as the first text cue 602 and the second text cue 604, are defined in latent space 116. In the example, the first text cue 602 is a class name with classifier parameters of... The normal classifier. Furthermore, the second text prompt 604 is a classifier with classifier parameters of... An anomaly classifier. In the example, and "A is damaged." Furthermore... These are semantic class names, such as "bottle," learned during the first training phase. First text cue 602 and second text cue 604 are fed into text encoder 214. Text encoder 214 is configured to generate first text encoding 110 and second text encoding 112 in latent space 116.

[0134] Furthermore, processor 206 is configured to obtain features from multiple normal images. Specifically, it obtains ground image features 404 and synthetic image features 410 from the image encoder 212 and image decoder 408, respectively. Subsequently, processor 206 is configured to segment the features of the normal images. Based on the segmentation, normal block features 504 are generated for the normal images.

[0135] Furthermore, the processor 206 is configured to introduce random noise 506 into at least some of the segmentation features or normal block features 504 of a plurality of normal images to generate anomalous features (referred to as pseudo-anomalous block features 508).

[0136] The processor 206 is configured to train the projection operator 114 to project segmentation features (i.e., normal block features 504) and anomalous features (i.e., pseudo-anomalous block features 508) of multiple normal images into the latent space 116. The projection operator 114 is trained to project the normal block features 504 of the multiple normal images closer to the first text prompt or first text code 110, and to project the pseudo-anomalous block features 508 closer to the second text prompt or second text code 112.

[0137] In the example, the projection operator 114 was trained as a computation block The projection into the latent space 116 of model 210 is called semantic encoding 608. In this respect, the projection operator... 114 is trained to encourage alignment between text features and visual or image features by minimizing the binary semantic loss 606. Text features may include the text features of the first text encoding 110. Text features of the second text encoding 112 In addition, visual or image features may include projected block features of normal images, such as normal block feature 504 and pseudo-abnormal block feature 508.

[0138] In the example, semantic loss 606 is the cross-entropy loss. Semantic loss 606 is defined as: (3) Here, c is the image class. It is the semantic score of the image of class c. The text encoder 214 is shared in both the first and second training phases. Furthermore, the text encoder 214 is fixed in both phases. Additionally, the combined loss function for the second phase is... .

[0139] In the example, using the Adam optimizer with a learning rate of 1e-4, the second-stage training was performed over 500 epochs.

[0140] Once the first and second training phases are completed, the average performance of the projection operator 114 is computed for the majority (high) class, minority (low) class, and all (all) classes in the training dataset 402. The performance of the projection operator 114 is evaluated accordingly. Furthermore, during testing, at least two training examples for each class are used as a support set to estimate the normal distribution.

[0141] Early methods for anomaly detection were unsuitable for detecting and locating defects across classes using a single model, resulting in poor performance. Furthermore, newer models are capable of detecting defects across classes using a single model. However, they fail to perform well on imbalanced datasets. To address this, the LTAD-based system

[106] is less affected by skewed distributions and outperforms conventional anomaly detection methods. Moreover, it can be noted that anomaly detection using LTAD is more localized and selective in its application.

[0142] Figure 7 A flowchart 700 is shown illustrating a method for training a system 106 for anomaly detection according to some other exemplary embodiments of this disclosure. (In conjunction with...) Figure 4 , Figure 5 and Figure 6The components are used to describe the steps of the method. As described above, the training of system 106 is performed in two phases.

[0143] In particular, the task or problem of long-tail AD is introduced by proposing a dataset and performance metric tailored to the long-tail setting, along with a novel AD method, LTAD. System 106 is based on the LTAD method and is configured to detect defects from multiple long-tail classes without relying on dataset class names. System 106 combines AD through reconstruction (i.e., by using RM 510) and semantic methods (i.e., by using a base model 210). RM 510 is implemented as a transformer-based reconstruction model, while semantic AD is implemented using a binary classifier that depends on the learned pseudo-class names and the pre-trained model 210. RM 510 and model 210 are learned in two stages.

[0144] At 702, the image decoder 408 processes features of multiple normal images conditioned on a pseudo-class name 414 associated with class c. Multiple normal images associated with class c are stored in the training dataset 402. In the example, the normal images are first processed by a layer sequence of the image encoder 212 to generate image features 404. These image features 404 are combined to generate the true image features of the normal images. The first training phase includes learning pseudo-class names, such as pseudo-class name 414 corresponding to the first text prompt or the normal classifier.

[0145] Real image features Together with the first text encoding of the first text prompt learned by the text encoder 214 110 is provided together to image decoder 408. Image decoder 408 processes real image features conditioned on pseudo-class name 414 or first text encoding 110. To generate synthetic image features Based on synthetic image features, the image decoder 408 is trained to perform augmentation training dataset 402 to counteract long-tail feature synthesis. Combined with, for example... Figure 4 Describe the details of the first training phase.

[0146] At position 704, the projection operator 114 is trained to project segmentation features and noise features from multiple normal images into the latent space 116 of model 210. Specifically, a layer-by-layer cascading mechanism 502 is used to aggregate real image features. and synthetic image features This generates normal block features 504. Furthermore, noise 506 is added to the normal block features 504 to generate anomalous or noisy block features, i.e., pseudo-anomalous block features 508. Subsequently, the normal block features 504 and the pseudo-anomalous block features 508 are fed into the projection operator 114 for training.

[0147] Specifically, the second training phase includes learning the parameters of the RM 510 and projection operator 114 of the LTAD. In this regard, the parameters of the projection operator 114 are learned by having it project normal block features 504 and pseudo-abnormal block features 508 into the latent space 116. The latent space 116 includes a first text code 110 and a second text code 112 based on the encoding generated by the text encoder 214 of the base model 210. The projection operator 114 is trained to project the segmentation features of the normal image (i.e., normal block features 504) closer to the first text code 110 than the second text code 112. Similarly, the projection operator 114 is trained to project noise or anomalous features (i.e., pseudo-abnormal block features 508) closer to the second text code 112 than the first text code 110. The projections of normal block features 504 and pseudo-abnormal block features 508 into the latent space 116 are semantic codes 608. Combined with, for example... Figure 6 Describe the details of training the projection operator 114.

[0148] A semantic loss 606 is generated based on the semantic encoding 608 of the visual features of normal images in latent space 116 and the encoding of the contradictory class names. In one example, the contradictory class names are a first text cue 602 and a second text cue 604. It can be noted that the first text cue 602 is the semantic name of the image class of an image learned to generate image classes using the visual language base model 210. In this example, the first text cue 602 is the semantic name of the class of a normal image, and the second text cue 604 is a modification of the first text cue 602. In another example, the first text cue 602 is the semantic name of the class of a normal image, and the second text cue 604 is a concatenation of modifier words and the semantic name of the image class. For example, the semantic name of the image class is defined as "bottle". Subsequently, the modifier words could be, for example, "damaged", "defective", "crushed", etc.

[0149] In the example, processor 206 is configured to reconstruct projected segmentation features of multiple normal images using RM 510. To learn the parameters of RM 510 in the second stage, the projected segmentation features (i.e., normal block features 504 of the normal images among the multiple normal images) are fed into RM 510. RM 510 reconstructs the images into reconstructed block features 512 based on normal block features 504.

[0150] Furthermore, the processor is configured to generate a reconstruction loss 514 based on the reconstruction. In this regard, the reconstructed block features 512 are compared with normal block features 504 to generate the reconstruction loss 514. Combined with, for example... Figure 5 To describe the details of training the RM 510.

[0151] At point 706, projection operators 114 and RM 510 are retrained to minimize the semantic loss 606 and the reconstruction loss 514. In this respect, the parameters of projection operators 114 and RM 510 are updated to reduce the difference between their predicted outputs and actual values. These differences are measured by the loss functions as the semantic loss 606 and the reconstruction loss 514.

[0152] Implementation Overview

[0153] Figure 8 A block diagram 800 is shown depicting the use of system 106 to detect anomalies 108 in image 102 according to some example implementations. System 106 uses an LTAD architecture for anomaly detection. The LTAD architecture is implemented using a projection operation 114 for performing semantic-based anomaly detection and an RM 510 for performing reconstruction-based anomaly detection.

[0154] In operation, the pre-trained image encoder 212 of model 210 receives image 102. Anomaly detection is performed on image 102. For example, image 102 may depict an object, such as a thing, a person, etc. In particular, image 102 may depict a specific pose of the object. To detect anomalies in the object, image sequences or videos of different poses of the object can be processed to detect and locate any anomalies in any image sequence, thereby detecting and locating any anomalies in the object.

[0155] Image encoder 212 includes a layer sequence. Each layer in the layer sequence generates image features 802 corresponding to image 102. Furthermore, features 806 of image 102 are formed by combining image features 802 from different layers. Image features 802 are combined using a layer-by-layer cascading 804.

[0156] Furthermore, features 806 of image 102 are provided to RM 510 to perform reconstruction-based AD. RM 510 It is a transformer trained to reconstruct features 806 extracted from image 102 I to generate reconstructed image patch 808. A pre-trained image encoder 212 based on L layers generates the reconstructed image patch 808. Given features from class... Image 102, Image encoder 212 from layer Extracting feature tensors Due to the feature tensor from the last layer Representing the global features of image 102, the last layer of feature tensors tends to degrade AD performance. Since reconstruction-based anomaly detection requires local semantics or local features, the last layer of feature tensors is discarded, and bilinear interpolation along the spatial dimension is used to interpolate the first L-1 feature tensors. Remapping to Dimensions ,Right now, .

[0157] In the example, processor 206 is configured to generate feature 806 of the image using image encoder 212. Processor 206 is configured to segment image 102 or feature 806 into blocks corresponding to feature block 106. For example, symbols Used to represent the interpolated version, and Defined as the feature tensor extracted across L-1 layers. In the example, the extracted feature tensor is then split or partitioned into... 104 feature blocks are defined as follows: .

[0158] Processor 206 is configured to use RM 510 as image block 808 to reconstruct each feature block 104 of image 102. Feature block 104 may be fed to RM 510 as a token. The vector. Furthermore, processor 206 is configured to compare the reconstructed image patch 808 with the corresponding partition of feature block 806 to generate a reconstruction score. Given a feature block generated by RM 510. and its corresponding reconstructed image patch This generates a reconstructed score of 810. The reconstructed score of 810 is the squared error and is defined as follows:

[0159] Processor 206 is configured to detect anomalies 108 based on a reconstruction score 810. As can be noted from equation (4), the reconstruction score is proportional to the square of the difference between the reconstructed image patch and the generated feature patch. Subsequently, when the reconstructed image patch is very similar to the feature patch, the difference is small, resulting in a small magnitude of the reconstruction score. When the reconstruction score is small, image 102 may tend to behave normally. Alternatively, when the reconstructed image patch is different from the feature patch, the difference is large, resulting in a large magnitude of the reconstruction score. When the reconstruction score is large, image 102 may tend to behave abnormally.

[0160] In reconstruction-based AD, an RM 510 is trained to reconstruct a normal image. During the inference phase, the RM 510 projects the anomalous image onto the normal image manifold. Anomaly detection can be performed by thresholding the magnitude of the reconstruction error to generate a reconstruction score 810.

[0161] Furthermore, system 106 is configured to perform semantic-based AD. In this respect, the goal of semantic AD is twofold: to give the anomaly detector sensitivity to normal / abnormal classes and to utilize prior knowledge about normal / abnormal classes available in model 210. This allows AD to distinguish between two conditional or contradictory classes without requiring the anomalous images to be trained.

[0162] In the example, a trained projection operator 114 is used to perform semantic-based AD. Projection operator 114 is a feature block. projection A binary classifier. In this respect, the projection operator 114 is configured to receive features 806 of the image 102 generated by the image encoder 212. In particular, the projection operator 114 can receive features 806 as feature blocks 104.

[0163] In the example, feature blocks Layer-by-layer components First, they are projected onto vectors. These vectors are represented as: The layer-by-layer components can be projected into vectors with a text embedding dimension d, i.e., the dimensions of the first text encoding 110 and the second text encoding 112 in the latent space 116 of model 210. In the example, the projection of the layer-by-layer components into the vector is performed by the projection module of the projection operator 114 as follows: , Each projection module is implemented using each layer of the image encoder 212. The layer-by-layer vectors are then aggregated into a single feature block vector. The feature block vector for a feature block is represented as:

[0164] In the example, the layer-by-layer vectors are aggregated into the feature block vector through max pooling at layer L-1. The resulting vector... The vectors are fed into projection operator 114 to project them into a latent space 116 containing classifier parameters with class names. This projection operator 114 is advantageous because typically the image encoder 212 and text encoder 214 of the base model 210 are only globally aligned in the latent space 116 (i.e., only the image features of the entire image 102 extracted by the image encoder 212 are aligned with the text features extracted by the text encoder 214, not with the features of the image patch 104). Therefore, projection operator 114 is used to project block-level features onto the latent space 116 (i.e., the output space of the image encoder 212). The classifier parameters may include classifier parameters for the first text encoder 110 or a normal classifier. And classifier parameters used for the second text encoding 112 or the anomaly classifier. Here, c is the class of image 102. Specifically, text encoder 214 learns classifier parameters with different class names (such as first text cue 602 and second text cue 604) during training. Furthermore, text encoder 214 encodes the classifier parameters and projects them into latent space 116 to generate first text encoding 110 and second text encoding 112. Projection operator 114 computes vectors corresponding to feature blocks 104 of image 102 and projects these vectors into the same latent space 116 with first text encoding 110 and second text encoding 112. During training, projection operator 114 learns encodings that project vectors of anomalous feature blocks closer to the anomalous classifier, which is second text encoding 112 in this context. Similarly, projection operator 114 learns encodings that project vectors of normal feature blocks closer to the normal classifier, which is first text encoding 110 in this context.

[0165] Once the vector of each feature block 104 of the image is projected into the latent space 116, the presence or absence of anomaly 108 is detected. In the example, a softmax layer with temperature scaling is used. To calculate the posterior probability of anomaly 108.

[0166] In the example, processor 206 is configured to capture the result of comparing the projection of each feature block 104 with a first text encoding 110 and a second text encoding 112 as a semantic score. In this regard, a semantic score 812 is determined for the image 102 of type c. The semantic score 812 is defined as:

[0167] here," "" represents the dot product. Therefore, the projection of the feature block is calculated. With the first text encoding The dot product between them is used to produce the first score. Additionally, the projection of the feature block is calculated. With the second text encoding The dot product between them produces the second score. From (6), it can be noted that the semantic score 812 is proportional to the dot product corresponding to the second score. Subsequently, when the second score (i.e., the similarity to the projection or the second text encoding) is high, the semantic score is also high, that is, the feature block under consideration tends to behave abnormally.

[0168] One challenge of semantic-based AD is learning classifier parameters without explicit supervision. and Because there are no abnormal training images. To overcome this problem, the prior knowledge of normal / abnormal classification provided by model 210 is utilized. This is done by representing it as Normal text prompts and representations are The exception text prompts are fed into the text encoder 214 of model 210 for implementation. Normal text prompts and exception text prompts can be applied to all classes. For example, It can be set to "a", and It can be set to "a damaged".

[0169] Furthermore, to enhance the semantic score 812's sensitivity to image semantics, normal text prompts and abnormal text prompts are derived from image-based prompts. Further, based on normal text prompts, abnormal text prompts, and supplementary image class prompts, the text encoder 214 can learn the encoding of a pair of contradictory class names for different object classes during the training phase. In this way, the classifier parameters for the first and second text prompts are learned. Because normal text prompts and abnormal text prompts are defined, in certain situations, a pair of contradictory class names can be learned on the fly for new object classes.

[0170] Based on the example, it is assumed that the class name of the object class is unknown. This is important for supporting classes that are unknown in model 210 or do not exist in the training dataset 402. In particular, instead of assuming the class name of the object class, pseudo-class names 414 are learned according to class c. This is achieved during the training phase (such as...). Figure 4 , Figure 6 and Figure 7 During the process described in the text encoder 214, prompts are given for each type of prompt and learning prompt.

[0171] In the example, the resulting set of semantically sensitive ad prompts is represented as follows: This set of semantically sensitive AD prompts can be mapped to a set of classifier parameters generated by the text encoder 214. (represented as T in equation (7) of model 210). The classifier parameters can be generated based on the following equation: and

[0172] Returning to this example, processor 206 is configured to combine semantic scores (such as semantic score 812) with corresponding reconstruction scores (such as reconstruction score 810) to produce a combined score (described as combined score 814). In the example, combined score 814 can be generated by performing linear operations (such as addition, multiplication, etc.) on semantic score 812 and reconstruction score 810.

[0173] Furthermore, processor 206 is configured to detect anomalies 108 based on combined scores. For example, anomalies in a feature block can be determined based on combined scores generated for a specific feature block. In this way, anomalies are also located in the corresponding feature blocks.

[0174] Figure 9 A flowchart 900 illustrates a method for detecting anomalies 108 in image 102 according to some other exemplary embodiments of this disclosure. (In conjunction with...) Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 To explain the components Figure 9 Components.

[0175] At 902, the first text code 110 of the first text prompt 602 is collected in the latent space 116. Furthermore, at 904, the second text code 112 of the second text prompt 604 is collected in the latent space 116. In this example, the processor 206 is configured to collect the first text code 110 and the second text code 112 from the text encoder 214. The text encoder 214 of the model 210 is used to extract classifier parameters for the first text prompt 602 and the second text prompt 604, and generates the first text code 110 and the second text code 112 in the latent space 116. For example, the first text prompt 602 is a normal classifier, while the second text prompt 604 contradicts the first text classifier and is an anomalous classifier.

[0176] At 904, image 102 is encoded to produce feature 806 of image 102. In this example, processor 206 is configured to encode image 102 using image encoder 212 of model 210.

[0177] At position 906, feature 806 of image 102 is segmented into feature block 104. In this example, processor 206 is configured to use the layers of image encoder 212. Feature 806 is segmented into feature blocks 104. In the example, feature blocks 104 are fed to RM 510 so that RM 510 reconstructs feature blocks 104 into noise-free feature blocks. Based on the reconstructed feature blocks and the corresponding feature blocks 104, a reconstruction score 810 is determined for each block.

[0178] At 908, projection operator 114 is used to project each feature block 104 into the latent space 116. In this example, processor 206 is configured to perform semantic AD by using projection operator 114 to project the vector of each feature block into the latent space 116 in a semantic manner. For example, semantic projection is performed such that the vectors of feature blocks with anomalous or noisy features are projected closer to the second text code 112, while the vectors of feature blocks with normal behavior are projected closer to the first text code 110.

[0179] Furthermore, at 910, the projection of each feature block 104 is compared with the first text code 110 and the second text code 112 to detect anomalies 108. In the example, the processor 206 is configured to generate a dot product of the distance between the projection of the feature block and the first text code 110 and another dot product of the distance between the projection and the second text code 112. Furthermore, the dot products are compared to evaluate whether the projection of the feature block is closer to the first text code 110 or the second text code 112. Based on this comparison, the presence or absence of an anomaly is determined. For example, when the projection is closer to the first text code 110, the feature block corresponding to the projection is considered normal, and vice versa. In the example, a semantic score 812 of the projection is determined based on the dot product of the projections. Similarly, a semantic score is generated for the projection of each feature block 104.

[0180] In the example, a combined score 814 for each feature block 104 is generated based on a combination of the corresponding reconstruction score 810 and the corresponding semantic score 812. Anomalies in the corresponding feature block 104 are detected based on the combined score 814 for each feature block 104. After detecting anomalous feature blocks from feature block 104, downstream tasks can be executed.

[0181] Use Case Overview

[0182] Figure 10 A use case 1000 implementation of system 106 according to some example embodiments of this disclosure is shown. Use case 1000 corresponds to a manufacturing or industrial environment. In this respect, one or more machines may operate within the environment. Subsequently, a large number of products or objects are generated. In the example, system 106 is configured to use LTAD by employing model 210 and RM 510 to detect and locate anomalies in the environment.

[0183] In one example, system 106 can detect anomaly 1006, which may correspond to defects or anomalies in the manufactured product, such as identifying scratches, dents, or other defects in items on the production line. In another example, system 106 can perform operations to replace or supplement manual inspection processes with an automated system to detect anomaly 1106 and improve efficiency and accuracy.

[0184] In the illustrative example scenario, the industrial environment may include a first environment 1002 where machines are performing packaging-related operations. Furthermore, the industrial environment may include a second environment 1004 where machines are performing PCB manufacturing operations. Additionally, different poses of objects on the production line are captured by cameras as one or more images. These images may be provided to system 106 via a communication module. System 106 may analyze the images to detect anomalies 1006. Once anomaly 1006 is detected, the processor 206 of system 106 may generate instructions to trigger one or more actions. These actions may include, for example, a stop action to halt machine operation, generating an alarm notification to inform of anomaly 1006, providing detection for downstream processing, etc. For example, downstream processing may be performed to identify the type, extent, and / or location of anomaly 1006 regarding the actual object or product.

[0185] In the example, an alarm notification may include a combination of audio, visual, or audiovisual notifications, such as “An alarm has been detected!” Additionally or alternatively, an alarm notification may be followed by an alert with the message “Off”.

[0186] Figure 11 An implementation of use case 1100 of system 106 according to some other example embodiments of this disclosure is shown. Use case 1100 corresponds to a vehicle driver assistance system. System 106 can detect abnormal posture of one or more occupants of vehicle 1106 (such as occupant 1108 A and occupant 1108 B (also referred to as occupant 1108) in vehicle 1106).

[0187] In an illustrative example scenario, an occupant 1108 A driving vehicle 1106 may turn their head away from the road ahead. This head-turning gesture can be captured by camera 1110. Camera 1110 can provide these gestures to system 106 via vehicle driver assistance system 1102. System 106 can detect these gestures as anomalous and send an alert notification 1104 to occupant 1108 A based on the detected anomalous gesture. Alert notification 1104 may include a combination of audio, visual, or audiovisual notifications, such as “Alert detected!” Additionally or alternatively, alert notification 1104 may be followed by a reminder with the message “Stay alert, drive safely.”

[0188] In some situations, both occupants 1108 A and 1108 B may move and exhibit potentially abnormal postures. In such cases, system 106 can detect the abnormal posture of each of occupants 1108 A and 1108 B and generate an alarm notification 1104 for each of occupants 1108 A and 1108 B. In some example implementations, system 106 can identify the postures of occupants 1108 A and 1108 B based on human motion recognition technology, human activity recognition technology, etc.

[0189] to this end, Figure 10 and Figure 11 An implementation of system 106 for anomaly detection in industrial environments and vehicle driver assistance systems is described. However, this should not be construed as limiting. In other cases, system 106 can be implemented to detect anomalies in applications such as surveillance and security, medical imaging, quality control, satellite imagery analysis, automated visual inspection, network intrusion detection, agriculture, autonomous vehicles, environmental monitoring, facility management, document verification, etc. Furthermore, several downstream tasks can be performed based on anomaly detection. Examples of downstream tasks may include, but are not limited to, root cause analysis, automated decision-making, classification, alarm and warning generation, process optimization, quality control feedback loops, predictive maintenance, supply chain adjustments, continuous improvement, feedback on system design, regulatory compliance, and customer communication.

[0190] Figure 12 A general block diagram 1200 of a system 106 according to some example embodiments of the present disclosure is shown. System 106 includes a processor 206 configured to execute stored instructions, and a memory 204 storing instructions executable by the processor 206. In some embodiments, the memory 204 is also configured to store a projection operator 114, a visual language base model 210, and a latent space 116. The projection operator 114 corresponds to a binary classifier. Furthermore, the latent space 116 includes encodings of classifier parameters and different text prompts or class names. For example, the latent space 116 includes at least two encodings, namely a first text encoding 110 and a second text encoding 112, such that these two encodings are contradictory.

[0191] In some example implementations, processor 206 uses model 210 to access latent space 116 to project feature blocks into latent space 116. Processor 206 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Memory 204 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Processor 206 is connected to input interface 202 via bus 1204. These instructions implement method 900 for detecting anomalies 108 in image 102, such as in Figure 10 Use case 1000 Figure 11 The anomaly detection described in use case 1100, etc.

[0192] In some implementations, system 106 may have different types and combinations of input interfaces to receive input data 1212. In one implementation, input interface 202 may include an audio-video receiver (AVR), a keyboard, and / or a pointing device such as a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen.

[0193] Alternatively or additionally, the network interface controller (NIC) 1202 may be adapted to connect the system 106 to the network 1210 via the bus 1204. Through the network 1210, input data 1212 can be downloaded and stored in the memory 204 for storage and / or further processing.

[0194] Additionally or alternatively, system 106 may include storage device 1206 for storing training parameters of a pair of contradictory classifiers, annotation pose bags for anomaly detection in input data 1212, etc.

[0195] In addition to input interface 202, system 106 may also include one or more output interfaces 208 to output classification results presented from anomaly detection. For example, system 106 may be linked via bus 1204 to output interface 208 suitable for connecting system 106 to output device 1208. Output device 1208 may include computer monitor, projector, display device, screen, mobile device, audio device, etc.

[0196] Benefiting from the teachings presented in the foregoing description and related drawings, those skilled in the art to which this disclosure pertains will conceive of numerous modifications and other embodiments of the disclosure set forth herein. Therefore, it should be understood that this disclosure is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Furthermore, although the foregoing description and related drawings describe exemplary embodiments in the context of certain example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, combinations of elements and / or functions different from those explicitly described above are also contemplated as being set forth in some of the appended claims. Although specific terminology is used herein, it is used only in a general and descriptive sense and not for limiting purposes.

Claims

1. A computer-implemented method for detecting anomalies in blocks of an image, wherein, The method uses a processor coupled to the stored instructions to implement the steps of the method, the method comprising: Collect the first text encoding of the first text prompt in the latent space; Collect the second text encoding of the second text prompt in the potential space; The image is encoded to generate the features of the image; The features of the image are segmented into feature blocks; Each feature block in the feature blocks is projected into the latent space using a projection operator, wherein the projection operator is trained to project normal feature blocks of the normal image closer to the first text code than the second text code, while simultaneously projecting noisy feature blocks of the normal image closer to the second text code than the first text code; and The projection of each feature block in the feature block is compared with the first text code and the second text code to detect the anomaly when the projection of the feature block in the feature block is closer to the second text code than the first text code.

2. The method according to claim 1, wherein, The image encoder is trained to encode the global features of the image into the latent space shared by the text encoder of the visual language base model and the image encoder.

3. The method according to claim 2, wherein, The method further includes: Collect multiple normal images associated with the class; The image encoder is used to encode the plurality of normal images to generate features of the plurality of normal images; The features of the plurality of normal images are processed using an image decoder with a pseudo-class name associated with the class, wherein the pseudo-class name is the first text prompt; and The image decoder is trained to learn the pseudo-class name as the first text encoding.

4. The method according to claim 3, wherein, The method further includes: The text encoder is used to obtain the encoding of a pair of contradictory class names in the latent space, the pair of contradictory class names including the first text hint and the second text hint; The features of the plurality of normal images are obtained using the image encoder. Segmenting features of the multiple normal images; Noise is introduced into at least some of the segmentation features of the plurality of normal images to generate anomalous features; and The projection operator is trained to project the segmentation features and anomalous features of the plurality of normal images into the latent space, wherein the segmentation features of the plurality of normal images are closer to the first text prompt, and the anomalous features are closer to the second text prompt.

5. The method according to claim 4, wherein, The method further includes: The projection segmentation features of the plurality of normal images and the features of the abnormal images are reconstructed using a reconstruction model; Based on the reconstruction and encoding of the aforementioned pair of contradictory class names, a reconstruction loss and a semantic loss are generated; and The projection operator and the reconstruction model are retrained to minimize the semantic loss and the reconstruction loss.

6. The method according to claim 5, wherein, The reconstruction model is a transformer.

7. The method according to claim 3, wherein, The image decoder is trained to learn the encoding of multiple pseudo-class names of multiple classes within the latent space.

8. The method according to claim 2, wherein, The training datasets used for the image encoder, the text encoder, and the image decoder include multiple images of the multiple classes in a long-tailed distribution.

9. The method according to claim 1, wherein, The image encoder is a deep neural network comprising a sequence of layers, wherein each layer in the sequence of layers generates image features, and wherein the features of the image are formed by combining image features from different layers.

10. The method according to claim 1, wherein, The method further includes: Determine the dot product between the projection of the feature block and the first text encoding to generate a first score; Determine the dot product between the projection of the feature block and the second text encoding to generate a second score; and The anomaly in the feature block is detected based on the first score and the second score.

11. The method according to claim 1, wherein, The first text prompt is the semantic name of the image's class, and the second text prompt is a modification of the first text prompt.

12. The method according to claim 1, wherein, The first text prompt is the semantic name of the class of the image, and the second text prompt is a concatenation of modifier words with the semantic name of the class of the image.

13. The method according to claim 1, wherein, The first text prompt is the semantic name of the class of the image, which is learned to generate an image of the class of the image using a visual language base model.

14. The method according to claim 1, wherein, The method further includes: The image is segmented into blocks corresponding to the feature blocks; Each feature block in the feature blocks of the image is reconstructed using the reconstruction model; The reconstructed feature block is compared with its corresponding partition to generate a reconstruction score; and The anomaly is detected based on the reconstructed score.

15. The method according to claim 14, wherein, The method further includes: The result of comparing the projection of each feature block with the first text encoding and the second text encoding is captured as a semantic score; The semantic score is combined with the corresponding reconstruction score to generate a combined score; and The anomaly is detected based on the combined score.

16. A system for detecting anomalies in blocks of an image, wherein, The system includes a processor and a memory storing instructions thereon, the instructions causing the processor to: Collect the first text encoding of the first text prompt in the latent space; Collect the second text encoding of the second text prompt in the potential space; The image is encoded to generate the features of the image; The features of the image are segmented into feature blocks; Each feature block in the feature blocks is projected into the latent space using a projection operator, wherein the projection operator is trained to project normal feature blocks of the normal image closer to the first text code than the second text code, while simultaneously projecting noisy feature blocks of the normal image closer to the second text code than the first text code; and The projection of each feature block in the feature block is compared with the first text code and the second text code to detect the anomaly when the projection of the feature block in the feature block is closer to the second text code than the first text code.

17. The system according to claim 16, wherein, The system also includes: A text encoder, trained to encode the first text prompt as the first text code and the second text prompt as the second text code in the latent space of the visual language foundation model; and An image encoder, trained to encode global features of the image into a latent space shared by the text encoder and the image encoder of the visual language base model.

18. The system according to claim 17, wherein, The image encoder is a deep neural network comprising a sequence of layers, wherein each layer in the sequence of layers generates image features, and wherein the features of the image are formed by combining image features from different layers.

19. The system according to claim 16, wherein, The instruction causes the processor to: The image is segmented into blocks corresponding to the feature blocks; Each feature block in the feature blocks of the image is reconstructed using the reconstruction model; as well as The reconstructed feature block is compared with the corresponding feature block of the image to generate a reconstruction score; and The anomaly is detected based on the reconstruction score.

20. A non-transitory computer-readable storage medium, wherein a program is specifically implemented on the non-transitory computer-readable storage medium, executable by a processor to perform a method, the method comprising: Collect the first text encoding of the first text prompt in the latent space; Collect the second text encoding of the second text prompt in the potential space; Encode the image to generate the features of the image; The features of the image are segmented into feature blocks; Each feature block in the feature blocks is projected into the latent space using a projection operator, wherein the projection operator is trained to project normal feature blocks of the normal image closer to the first text code than the second text code, while simultaneously projecting noisy feature blocks of the normal image closer to the second text code than the first text code; and The projection of each feature block in the feature block is compared with the first text code and the second text code to detect an anomaly when the projection of the feature block in the feature block is closer to the second text code than the first text code.

Images