A zero-shot anomaly detection method and system based on triple perception learning enhanced visual language model

Abstract

The application discloses a kind of based on triple perception learning enhanced visual language model's zero sample exception detection method and system, it is related to computer vision field, method includes: extracting global and local visual features from input image;Visual coding process is corrected local feature in deep network by spatial perception attention enhancement module, and fine-grained attribute text description is generated for abnormal visual feature;Through attribute perception guide module, attribute text description and general text prompt are deeply semantically aligned;The similarity of enhanced visual feature and optimized text feature is calculated, and pixel-level exception segmentation map is generated;Inference stage converts segmentation map into spatial attention weight by exception perception reconstruction module, and feedback is generated to visual encoder final global feature representation and calculates exception score.The method of the application significantly improves the accuracy and generalization ability of model in industrial defect detection and other scenarios without target domain training samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and artificial intelligence, and particularly relates to a zero-shot anomaly detection method and system based on triple perception learning to enhance visual language models. BACKGROUND

[0002] Anomaly detection is a fundamental task in computer vision, with wide applications in industrial defect detection and medical image diagnosis. Traditional anomaly detection methods usually rely on a large number of normal samples to learn a standard distribution model, and data deviating from the distribution is identified as abnormal. Such methods will encounter significant limitations when facing data-scarce scenarios. In this context, new paradigms such as zero-shot anomaly detection (ZSAD) have emerged, aiming to reduce the dependence on training data in the target domain. Vision-language models (VLMs) represented by CLIP have attracted much attention due to their strong semantic transfer ability. CLIP can align image features with text prompts based on semantic similarity scores, enabling effective zero-shot anomaly detection without training data of target classes.

[0003] Thanks to the generalization ability of CLIP in zero-shot classification tasks, its potential application in anomaly detection has attracted a lot of research. Early attempts such as WinCLIP achieve anomaly localization by sliding different size windows on images. Subsequent research such as MVFA-CLIP introduces a lightweight adapter module to enhance the model's ability to represent local block-level features of images, aiming to bridge the domain gap between natural images and anomaly detection tasks. Recent research has shifted towards improving the model's anomaly semantic description ability through more refined prompt design. For example, AnomalyCLIP introduces a learnable text vector independent of specific objects, but it uses the word "damage" to describe all anomalies, ignoring the diversity of anomaly types in the real world. To address this issue, methods such as AdaCLIP and VCP-CLIP introduce dynamic prompts that adapt to image content, using visual features to guide text feature generation to produce more fine-grained and discriminative anomaly semantics.

[0004] However, despite the progress made by existing methods, research and experiments show that directly applying CLIP to zero-shot anomaly detection still faces inherent architectural limitations that restrict its performance: 1) Disorganized local feature space modeling. A proxy token phenomenon exists in the deep network of CLIP, where the model tends to unreasonably focus its attention on certain background areas rather than the key objects themselves when analyzing deep image features, leading to misjudgments of spatial relationships within the image and limiting the ability to accurately locate fine-grained local anomalies. 2) Limited semantic description of textual anomalies. The model has limited ability to distinguish between normal and abnormal states when understanding textual semantics. There is confusion and overlap in the feature representations of these two concepts within the model, making it difficult to form clear decision boundaries. Furthermore, existing methods rely on general or hand-designed textual cues, making it difficult to capture the complex and diverse anomaly types in the real world, further restricting detection accuracy. 3) Dilution of local anomaly features. CLIP was originally designed for image-level classification tasks, and its generated global feature representation lacks sensitivity to local, small-scale anomaly patterns. Therefore, when performing similarity calculations, local abnormal features are easily diluted or masked by global background information, significantly reducing the accuracy of detection.

[0005] Therefore, how to overcome the inherent architectural limitations of the CLIP model and comprehensively improve the accuracy, robustness, and generalization ability of zero-shot anomaly detection methods through systematic solutions has become a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the core limitations of the existing visual language model CLIP when directly applied to anomaly detection tasks, this invention provides a zero-shot anomaly detection method and system based on triple-perception learning to enhance the visual language model. Through the triple-perception learning strategy, the generalization ability and detection accuracy of the model are significantly improved. Accurate anomaly detection and localization can be performed without a target domain training dataset, effectively solving the application problem in data-scarce scenarios.

[0007] This invention adopts the following technical solution: a zero-shot anomaly detection method based on a triple-perception learning-enhanced visual language model, comprising the following steps:

[0008] S1. Preprocess the sample image to be processed, including size normalization and data standardization, and extract image-level global features representing global semantics and multi-level local features representing local details through the visual encoder of the visual language model.

[0009] Preferably, the pre-processed image is passed through the visual encoder of the visual language model CLIP to extract image-level global features from its final output layer to represent the global semantic information of the image, and to extract multi-level local features from its internal intermediate layers (such as layers 6, 12, 18, and 24) to represent the local structure and detail information of the image.

[0010] S2, set the learnable general normal and abnormal text prompts, and use a large language model (LLM) to generate fine-grained attribute text descriptions for different abnormal visual features.

[0011] Unlike the method of defining a single and general description for each specific abnormal category (such as "scratch" and "damage") in the prior art, the generation process of the fine-grained attribute text description is based on the induction of 22 core abnormal visual features from the visual manifestations of the abnormalities, and a large language model is used to generate multiple sets of accurate text descriptions for these 22 visual features. This method describes the abnormalities from their fundamental visual forms, achieving a more fine-grained and more generalizable coverage of the diversity of abnormalities in the real world.

[0012] Preferably, the fine-grained attribute text description generation process specifically includes the following steps:

[0013] First, the various abnormalities in the open-source industrial dataset (such as the VisA dataset) are summarized into a set of visual feature-based categories.

[0014] Next, according to the annotation mask of the abnormality, the boundary of the abnormality is outlined with a red outline on the original image for model recognition.

[0015] Then, for each summarized abnormal category, a number of sample images are randomly selected.

[0016] Subsequently, a large language model (LLM, such as GPT-4o) is used to generate descriptions, and a pre-set prompt template (Prompt Template) is used to guide it.

[0017] The template first sets the system role of the model as "an abnormality detection expert focusing on analyzing image abnormality types", and then through user prompt words, requires the model to give multiple specific, concise, and only focus on the appearance and shape of the abnormality descriptions for the provided image with marked abnormal areas, and explicitly instructs it to ignore the red markings and other background information.

[0018] Finally, the large number of descriptions generated by the model for all categories are integrated to form a comprehensive fine-grained abnormal attribute text library for subsequent training processes.

[0019] S3, introduce a spatial-aware attention augmentation module (SAAM) in the visual encoder deep network in step S1 to generate enhanced multi-level local visual features.

[0020] The purpose of the SAAM module is to solve the "proxy token phenomenon" in the CLIP visual encoder deep network, that is, attention is unreasonably focused on the background area, causing confusion in local feature space modeling.

[0021] Preferably, the module first replaces the traditional Query-Key (QK) attention mechanism of the last several Transformer layers of the visual encoder with a Value-Value (VV) autocorrelation attention mechanism.

[0022] Secondly, in order to improve the spatial awareness information of the model, the module uses the visual base model (DINOv2) to extract the local features (patch-level features) of the input image. Based on the features extracted by DINOv2, a self-similarity matrix S is calculated.

[0023] Then, the self-similarity matrix is thresholded and masked to suppress the correlation of areas with a similarity lower than a predetermined threshold to negative infinity, thereby enhancing positive inter-block associations and suppressing negative associations.

[0024] Finally, the matrix after thresholding and masking is fused with the aforementioned VV autocorrelation attention matrix to generate the final spatial-aware attention distribution, which is injected back into the last k layers of the CLIP visual encoder to dynamically guide the attention of the model and ensure that the output visual features have stronger spatial position relationship consistency.

[0025] S4, input the attribute text description generated in step S2 and the general text prompt into the attribute-aware guidance module (AAGM), perform semantic alignment through attribute-guided prompt learning, and generate visual representations associated with specific abnormal attributes through attribute-guided image aggregation; use semantically similar non-correct abnormal attribute descriptions as negative samples for training to obtain optimized text features.

[0026] The purpose of the attribute-aware guidance module is to solve the problem of limited abnormal text semantic description and overly generalized abnormal description. The attribute-guided prompt learning part aims to optimize the semantic representation of the text prompt.

[0027] The attribute-guided prompt learning is aligned by using a loss function aiming to minimize the Euclidean distance between the prompt and the attribute description, and suppresses the semantic similarity between different abnormal attributes, so as to enhance the discrimination of the model to diversified abnormal texts; secondly, the attribute-guided image aggregation is deep fused by using a cross-attention mechanism fusing visual and text information, so as to generate visual representation closely related to a specific abnormal type.

[0028] Preferably, the module uses an attribute-guided loss function (L) ) to optimize the learnable normal and abnormal general prompts. The loss function is aligned by minimizing the Euclidean distance between the abnormal prompt and the corresponding abnormal attribute description, and maximizing the distance between them and the normal prompt.

[0029] In addition, the discrimination of the model to abnormal text features is further enhanced by introducing attribute descriptions that are similar in semantics but wrong in category as hard negative samples, and suppressing the similarity between different abnormal attributes.

[0030] The attribute-guided image aggregation part aims to enhance the ability of the model to recognize and locate complex abnormalities. Since the abnormal area often spans multiple image blocks and the location and size are variable, the neighborhood of the extracted multi-level local features is adaptively averaged and pooled to obtain the aggregated features.

[0031] Subsequently, the aggregated visual features are fused with the aforementioned fine-grained abnormal attribute text description by using a visual-text cross-attention mechanism. This process can align the fine text attributes with the corresponding visual features, and generate customized visual representation for specific abnormal attributes, so as to improve the recognition and positioning accuracy of the model to complex abnormalities.

[0032] S5, similarity calculation is performed between the multi-level local visual features generated in step S3 and the text features optimized in step S4 in association with specific abnormal attributes, to generate a pixel-level abnormal segmentation map as an abnormal positioning map; meanwhile, similarity calculation is performed between the image-level global features in step S1 and the general text prompts, to generate a global feature score.

[0033] Preferably, the specific generation steps are as follows:

[0034] First, the local visual features of the output of each level of the visual encoder are obtained respectively, and cosine similarity calculation is performed between the local visual features and the optimized “normal” text prompts and “abnormal” text prompts in step S4, to obtain the normal score map and the abnormal score map of the level.

[0035] Subsequently, to integrate information of all levels to generate the final segmentation map, the model reverses all normal score maps of all levels and averages them with the abnormal score maps, and then fuses and averages the results of all levels.

[0036] Finally, the fused score map is upsampled and processed by applying a Gaussian smoothing filter. The upsampling operation can use bilinear interpolation, nearest neighbor interpolation or other appropriate interpolation methods to ensure that the result after upsampling has the same spatial resolution as the input image. Based on this, the final, smooth and fine-grained pixel-level anomaly segmentation map is obtained.

[0037] S6, input the pixel-level anomaly segmentation map obtained in step S5 into the anomaly-aware reconstruction module (AARM) to convert it into a spatial attention weight, dynamically guide the model to focus on the potential abnormal area, and generate a final visual feature representation.

[0038] The core purpose of the AARM module is to solve the problem of dilution of local abnormal information by the background due to the model's dependence on global features. The AARM module is only activated in the inference stage of the model.

[0039] Preferably, the AARM module specifically processes as follows:

[0040] First, the anomaly segmentation map generated in S5 is received as input, and a threshold suppression operation is performed to filter out low-confidence background noise and retain significant abnormal area signals.

[0041] Then, through a series of transformations such as size reshaping and Gaussian smoothing, the significant abnormal area information is converted into a standardized spatial attention weight map.

[0042] Finally, the weight map is fused into the attention matrix of the last layer of the CLIP visual encoder for weighted fusion, thereby dynamically redistributing the model's computing resources to potential abnormal areas in the image and generating a final visual feature that is more sensitive to local abnormalities.

[0043] S7, based on the final visual feature generated in step S6 and the optimized text feature in S4, calculate the final similarity, and combine the global feature score in S5, the maximum value of the anomaly segmentation map to obtain the final image-level anomaly score, and use it as the classification result of whether the image is abnormal.

[0044] In particular, the learnable general text prompt optimization method in step S2 is achieved by minimizing a combined overall loss function (L) ).

[0045] The combined loss function consists of three parts: global loss, local loss and attribute-guided loss, which are as follows:

[0046] The global loss is a cross-entropy loss based on the cosine similarity between image-level visual features and text embeddings, and the overall discrimination ability of the model at the image level is enhanced by introducing hard negative samples.

[0047] Secondly, to optimize the positioning accuracy at the pixel level, the local loss is composed of Focal loss and Dice loss, which acts on the normal and abnormal score maps generated by the visual encoder at multiple intermediate layers and compares them with the real segmentation mask.

[0048] Focal loss is used to solve the problem of serious imbalance between normal and abnormal pixels, and by reducing the weight of easily classified samples, the model can focus on the difficult-to-classify boundary area.

[0049] The Dice loss directly optimizes the spatial overlap between the predicted abnormal area and the real label, which helps to improve the accuracy of the segmentation contour.

[0050] Finally, the combined loss function also adds the attribute guidance loss mentioned above to specifically optimize the semantic representation of the text prompt.

[0051] The technical scheme of the present application also provides a zero-shot anomaly detection system based on triple perception learning to enhance the visual language model, which is used to implement the zero-shot anomaly detection method described above, comprising:

[0052] The image space perception module is used to preprocess the input sample image, including uniform size and standardization, and extract global image features and multi-level local image features through the visual encoder of the visual language model; and in the encoding stage of the visual encoder, a spatial perception attention enhancement module is introduced in the last several layers, replacing the traditional Query-Key attention mechanism of Transformer with a Value-Value autocorrelation attention mechanism; with the help of an independent visual base model, a self-similarity matrix is calculated, and the enhanced positive correlation features are suppressed through thresholding mask, and finally the enhanced visual features are generated.

[0053] The text anomaly semantic perception module is used to generate and optimize the text features of anomaly discrimination; a large language model is used to generate fine-grained abnormal attribute text descriptions for diversified abnormal visual features, and a learnable general normal and abnormal text prompt is defined; attribute guidance prompt learning in the attribute perception guidance module is performed, the general abnormal prompt and the fine-grained attribute description are semantically aligned in the training, and the description with similar semantics but class error is introduced as a negative sample; in the attribute-guided image aggregation, the local visual features and fine-grained attribute text are fused through the visual-text cross-attention mechanism to generate visual representations for specific abnormalities.

[0054] An abnormality perception feedback module is configured to perform final abnormality identification and positioning on the query image sent by the terminal device; similarity calculation is performed on the enhanced local visual features output by the image space perception module and the optimized text features output by the text abnormality semantic perception module to generate a pixel-level abnormality segmentation map, which is directly used as positioning basis for the abnormality region; an abnormality perception reconstruction module activated only in the inference stage converts the pixel-level abnormality segmentation map information into spatial attention weights, which are fed back to the model core calculation layer to generate final global visual features; a final image-level abnormality score is calculated through a comprehensive scoring mechanism, which is used as identification basis for judging whether the image as a whole has abnormality, and the positioning result is output together.

[0055] Compared with the prior art, the above technical scheme has the following technical effects:

[0056] The zero-shot abnormality detection system first pre-processes the input image through the image space perception module and extracts multi-level visual features. In the visual encoder coding stage, the spatial perception attention enhancement module (SAAM) helps the model output spatially consistent visual features with significantly enhanced attention guidance. The text abnormality semantic perception module generates fine-grained abnormality attribute descriptions using a large language model (LLM), and then, through the attribute-guided prompt learning in the attribute awareness guidance module (AAGM), aligns the general text prompts with these attribute description semantics, and through attribute-guided image aggregation, fuses the local visual features and attribute texts using the cross-attention mechanism to generate visual representations for specific abnormalities. The abnormality perception feedback module receives the visual and text features processed by the previous modules, calculates their similarity to generate a high-resolution pixel-level abnormality segmentation map, which is directly used as abnormality positioning basis. In the inference stage, the abnormality perception reconstruction module (AARM) further converts the pixel-level abnormality segmentation map information into attention weights and feeds them back to the model to dynamically focus on the potential abnormality region. Finally, through a comprehensive scoring mechanism, the global feature scores before and after focusing are combined with the segmentation map information to calculate an accurate image-level abnormality score as the final criterion for abnormality identification, and the complete detection result is fed back to the terminal. Through the above three perception collaborative work, the present application can efficiently and accurately perform zero-shot defect detection and positioning, significantly improving the detection performance and generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The figure is a schematic diagram of the three perception learning enhanced visual language model abnormality detection and positioning implementation method of the embodiment of the present application.

[0058] Figure 2 The figure is a schematic diagram of the spatial perception attention enhancement module implementation method of the embodiment of the present application.

[0059] Figure 3An implementation schematic diagram of image aggregation of attribute guidance in the attribute perception guidance module of the embodiment of the present application.

[0060] Figure 4 An abnormal visual feature definition schematic diagram of fine-grained attribute description of the embodiment of the present application.

[0061] Figure 5 A step flow schematic diagram of the zero-shot anomaly detection method of the present application.

[0062] Figure 6 A system architecture schematic diagram of the zero-shot anomaly detection system of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the application will be further described in detail below in combination with the drawings. The described embodiments are only a part of the embodiments involved in the present application. All non-innovative embodiments of other researchers in the field on the basis of the embodiments belong to the protection scope of the present application. At the same time, the step numbers in the embodiments are only set for the convenience of description and explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0064] In one embodiment of the present application, a zero-shot anomaly detection method based on triple perception learning enhanced visual language model is provided, which is a technical solution improvement for the visual language model CLIP in anomaly detection application. The main design idea is to introduce three core modules that work together, as shown in Figure 1 which can fundamentally solve the inherent limitations of the model in the three dimensions of image feature extraction, text semantic understanding and anomaly pattern perception.

[0065] First, to solve the problem that the attention of the model is easy to be dispersed when analyzing the deep features of the image, leading to misjudgment of local spatial relationship, the spatial perception attention enhancement module is introduced in the embodiment. Through the attention guidance mechanism, the feature regions associated with semantics in space are aggregated to ensure the accuracy and spatial position consistency of local feature expression.

[0066] Secondly, to overcome the bottleneck of the model's limited ability to distinguish between "normal" and "abnormal" text semantics, the attribute perception guidance module is designed in the embodiment. The large language model (LLM) is used to generate fine-grained abnormal attribute descriptions, and through the attribute-guided prompt learning process, the general abnormal prompt is deeply aligned with these accurate descriptions, thereby establishing a clear semantic boundary.

[0067] Finally, to improve the sensitivity of the model to small and local abnormal patterns, the embodiment proposes an abnormality perception remodeling module, which converts the pixel-level abnormality segmentation map into attention weights to guide the model to focus on potential abnormal areas.

[0068] The embodiment method cooperates the above three modules to enhance the visual language model in three dimensions of image features, text semantics and abnormality perception, thereby significantly improving the abnormality detection and positioning performance of the model in the data scarce scene.

[0069] The embodiment method flow is as shown in Figure 5 The embodiment method flow is as shown in

[0070] The sample image to be processed is preprocessed, including size unification and standardization operation of the image. Then, the image is input into the visual encoder of the visual language model CLIP to extract the multiple intermediate layer outputs as local visual features and the final layer output as global features.

[0071] In the embodiment, the size unification operation unifies all input samples to a specified size through a preset parameter. For example, if the specified size is 518, the input sample is uniformly scaled to a square image with a length and width of 518 pixels using nearest neighbor or bilinear interpolation. The standardization operation converts the pixel values of the image to a standard normal distribution by subtracting the mean value of each channel and dividing by the corresponding standard deviation.

[0072] In the embodiment, the model includes a fine-grained attribute text generation module, a spatial perception attention enhancement module (SAAM), an attribute perception guidance module (AAGM), and an abnormality perception remodeling module (AARM). The attribute perception guidance module includes attribute-guided prompt learning and attribute-guided image aggregation.

[0073] Further, the visual encoder is enhanced by the spatial perception attention enhancement module (SAAM) to solve the problem that the attention of the CLIP visual encoder is unreasonably concentrated in the background area when processing deep network features (i.e., the "proxy token phenomenon"), thereby causing confusion in local feature space modeling. Figure 2

[0074] Specifically, the traditional Query-Key (QK) attention mechanism of the deep network of the visual encoder is replaced by a Value-Value (VV) autocorrelation attention mechanism, and its mathematical expression is as follows:

[0075] ;

[0076] wherein, is a scaling factor, ​denotes the value vector matrix, and the superscript T denotes the transpose, The function is used for normalization and conversion to a probability distribution.

[0077] To further enhance the spatial relationship perception information, an independent visual base model (DINOv2) is used to extract the block-level features of the input image and calculate a self-similarity matrix representing the correlation between image blocks

[0078] ;

[0079] wherein, The function represents the visual encoder of DINOv2, which extracts image features.

[0080] Next, the self-similarity matrix is subjected to thresholding mask suppression processing.

[0081] In this process, a mask matrix is introduced to enhance positive inter-block correlation and suppress negative correlation:

[0082] ;

[0083] wherein, is a preset similarity threshold, denotes the similarity between the th image block and the th image block.

[0084] Finally, the mask thresholding processing result is fused with the aforementioned VV autocorrelation attention matrix to generate the final spatial perception attention distribution , which is injected into the last Transformer layer of the CLIP visual encoder to dynamically guide attention:

[0085] ;

[0086] ;

[0087] wherein, denotes the similarity matrix after mask thresholding processing of the self-similarity matrix, denotes the feature output by the th Transformer layer, denotes the linear projection of the output,​ Represents the Transformer Layered spatial perception of attention distribution This indicates the total number of Transformer layers. The enhanced local visual features generated in this way will be used for subsequent pixel-level anomaly map generation.

[0088] Furthermore, we set general normal and abnormal text prompts and use a large language model to generate fine-grained abnormal attribute descriptions.

[0089] The general templates for normal and abnormal text prompts are as follows:

[0090] ;

[0091] ;

[0092] in, and These represent the learnable text embedding vectors of length K in normal and abnormal prompts, respectively. State descriptions such as "normal" and "perfect" are used, and are independent of the specific object. A general marker indicating the category of detected anomalies.

[0093] It is particularly important to note that the fine-grained anomaly attribute description described in this embodiment differs from the traditional method of describing anomalies based on specific anomaly categories (such as "scratches" or "damage"). Instead, it summarizes the visual characteristics of the anomalies and generates fine-grained anomaly descriptions corresponding to these visual characteristics. The generation process specifically includes the following steps:

[0094] First, various anomalies in open-source industrial datasets (such as the VisA dataset) are analyzed and summarized into a set of categories based on visual representation (e.g., summarized into 23 core visual features, as shown in the appendix). Figure 4 (As shown).

[0095] Next, based on the labeled mask of the abnormal samples in the dataset, the boundaries of the abnormal regions are outlined on the original image using a striking red outline, so as to facilitate the recognition of subsequent large-scale language models.

[0096] Then, for each category of anomalous visual features, several sample images that have undergone the above preprocessing are randomly selected. A large language model (LLM, such as GPT-4o) is used, guided by a pre-defined prompt template to generate descriptions. The prompt template first defines the model's system role as an "anomaly detection expert focused on analyzing image anomaly types".

[0097] Subsequently, through the user prompt words, the model is required to give a plurality of (such as five) "specific, concise, and only focus on the appearance and shape of the abnormal itself" descriptions for the provided image with marked abnormal regions, and it is clearly instructed to ignore the red marks and other background information.

[0098] Finally, the large number of descriptions generated by the model for all categories are uniformly integrated to form a comprehensive fine-grained abnormal attribute text library for subsequent training process.

[0099] Further, the obtained general text description and fine-grained abnormal attribute description are input to the text encoder of the visual language model for encoding, and are input to the attribute awareness guidance module (AAGM). Through attribute-guided prompt learning and attribute-guided image aggregation, visual representations for specific abnormal attributes are generated.

[0100] Among them, the attribute awareness guidance module (AAGM) is used to solve the limitation of the CLIP text encoder that it is difficult to clearly describe the abnormal text semantics, including attribute-guided prompt learning and attribute-guided image aggregation.

[0101] The attribute-guided prompt learning adopts an attribute-guided loss function (L) ), which realizes semantic alignment by minimizing the Euclidean distance between the general abnormal prompt and the corresponding abnormal attribute description generated by the LLM, maximizing their distance from the normal prompt, and suppressing the semantic similarity between different abnormal attributes. In addition, this method also introduces semantically similar but class error attribute descriptions as hard negative samples for contrastive learning to further enhance the model's ability to distinguish abnormal text features. The mathematical expression of the loss function is:

[0102] ;

[0103] Among them, represents the Euclidean distance, and represent the feature vectors of the general normal and abnormal text prompts, represents the abnormal attribute feature vector, represents the set of abnormal attribute descriptions; represent the , th abnormal attribute feature, respectively.

[0104] The attribute-guided image aggregation is used to enhance the model's ability to identify and locate complex abnormalities, as shown in the accompanying Figure 3 Since the abnormal region often spans multiple image blocks and the location and size are variable, this step first aggregates the local features obtained above in Adaptive average pooling is performed within the neighborhood to obtain aggregated features that can represent the overall information of the local region. ,in, Its mathematical expression is:

[0105] ;

[0106] ;

[0107] in, Indicates the first Feature vectors of each patch Indicates the current number Centered on a patch Neighborhood, Indicates the first Layer aggregation features The function represents averaging the features of multi-level aggregation. This represents the visual features after averaging the features from multiple aggregation layers.

[0108] Subsequently, using a visual-text cross-attention mechanism, these aggregated average visual features are... This is integrated with the aforementioned fine-grained anomaly attribute text descriptions.

[0109] This fusion process can precisely align refined textual attributes with corresponding visual features, thereby generating customized visual representations for specific anomalous attributes, significantly improving the model's accuracy in identifying and locating complex and variable anomalies.

[0110] Furthermore, the enhanced local visual features and global features obtained are compared with the optimized general text features to calculate the similarity, generating pixel-level anomaly segmentation maps and image-level anomaly scores, and based on these, preliminary anomaly localization and identification are performed.

[0111] The pixel-level segmentation map is obtained by calculating the cosine similarity between multi-level enhanced local visual features and general normal text prompts and general abnormal text prompts, respectively. ) and anomaly score plot ( ).

[0112] Subsequently, the inverted normal score map and the abnormal score map are merged, and then upsampled and Gaussian smoothed to obtain the final segmentation map. Its mathematical expression is:

[0113] ;

[0114] in, Indicates the total number of feature layers selected. It is a matrix of all ones. denotes an up-sampling operation, denotes a Gaussian smoothing process with a standard deviation of , , denote the normal and abnormal score maps of the i-th level, respectively.

[0115] The image-level anomaly score is obtained by cosine similarity between the image global feature and the general anomaly text prompt. In the training phase, the pixel-level anomaly segmentation map is optimized by local loss calculation with the mask.

[0116] Specifically, the local loss (Llocal) aims to optimize the model's alignment ability at the pixel level, which is composed of two functions: Focal loss and Dice loss. Focal loss is mainly used to solve the problem of severe imbalance between normal and abnormal pixels. By reducing the weight of easy-to-classify samples (such as large-area normal background), the model can focus on the difficult-to-classify boundary region. Dice loss directly optimizes the spatial overlap between the predicted abnormal region and the true label, which helps to improve the accuracy of the segmentation contour.

[0117] The mathematical expression is:

[0118] ;

[0119] wherein, for each level i, , and denote the normal and abnormal score maps, respectively; is the real mask, is a full 1 matrix.

[0120] Meanwhile, the image-level anomaly score is optimized by global loss (Lglobal). This loss function is based on the cross-entropy loss of the cosine similarity between the image global visual feature and the "normal / abnormal" text prompt embedding.

[0121] To further enhance the model's ability to distinguish different abnormal text features, the global loss function also introduces hard negative samples (i.e., semantically similar but classically incorrect abnormal attribute descriptions) in training, forcing the model to learn more discriminative feature representations.

[0122] The mathematical expression is:

[0123] ;

[0124] wherein, denotes the cosine similarity, denotes the batch, denotes the set of abnormal attribute descriptions,​​​ a feature vector representing general abnormal text prompts, with representing global features of images in the input batch, representing abnormal attribute descriptions corresponding to image abnormalities, representing hard negative sample abnormal attribute descriptions.

[0125] Further, in the inference stage, the obtained pixel-level segmentation map is input into the abnormal perception reconstruction module to dynamically and feedback-based guide and optimize the final visual feature representation of the CLIP model.

[0126] Wherein, the abnormal perception reconstruction module explicitly guides the model to focus attention on the area related to the abnormality by converting the segmentation prediction result into a spatial attention weight. The specific implementation steps are as follows:

[0127] First, the module performs pixel-level abnormal segmentation on the input image performs threshold suppression operation to filter out low-confidence background noise signals and retain significant abnormal areas, thereby avoiding false detection of normal background areas:

[0128] ;

[0129] wherein, is the predicted abnormal score of the abnormal segmentation map at the spatial position , and τ is the abnormal confidence threshold.

[0130] Next, the segmentation map after threshold processing is reshaped into a block-level attention map aligned with the feature block size of the visual encoder, and the maximum abnormal score value in each block (patch) is taken as the abnormal attention value of the block:

[0131] ;

[0132] wherein, is a reshaping function, is a block set of the block-level attention map.

[0133] In order to further improve the spatial generalization ability of the model, the block-level attention map is applied with Gaussian smoothing processing to obtain the final abnormal perception attention weight :

[0134] ;

[0135] wherein, is an activation function, is a norm, is a standard one-dimensional Gaussian kernel function.

[0136] Finally, the obtained anomaly-aware attention weight is injected into the attention calculation process of the last layer of the CLIP model, and the original attention matrix is weighted and fused to generate the final global visual feature optimized by anomaly-aware guidance:

[0137] ;

[0138] wherein, and are hyperparameters for controlling the fusion ratio.

[0139] Further, the information of the initial global feature, the enhanced global feature, and the pixel-level segmentation map is comprehensively evaluated to calculate the final image-level anomaly score, which is used as the final criterion for the existence of anomalies.

[0140] wherein, the final image-level anomaly score is a composite score composed of three parts, aiming to integrate multi-dimensional anomaly information globally and locally to improve the robustness of discrimination.

[0141] Its composition includes:

[0142] 1) the similarity score of the initial global visual feature and the text feature ;

[0143] 2) the maximum value in the pixel-level anomaly segmentation map ;

[0144] 3) the similarity score of the global visual feature enhanced by the anomaly-aware reconstruction module (AARM) and the text feature .

[0145] Its mathematical expression is:

[0146] ;

[0147] Finally, the image-level anomaly score is compared with the preset threshold . If is greater than or equal to the threshold, it is determined that there is an anomaly in the image; otherwise, it is determined that there is no anomaly.

[0148] ;

[0149] wherein, is a binary variable, and when its value is 1, it indicates that there is an anomaly in the image, and when its value is 0, it indicates that there is no anomaly in the image.

[0150] The working mode of the embodiment is similar to that of a commonly used target detection engine. When an image of an industrial product appearance input by a user is received, the image is first processed by an inherent enhanced visual encoder (integrating a spatial perception attention enhancement module) to extract image-level global features and multi-level local features enhanced by spatial consistency. Subsequently, the visual features are compared with pre-optimized “normal” and “abnormal” text prompts in the text anomaly semantic perception module to generate an initial image-level anomaly score and a high-resolution pixel-level anomaly segmentation map. Then, in the inference stage, the anomaly perception reconstruction module is activated, and the system converts the generated pixel-level anomaly segmentation map information into spatial attention weights to dynamically guide and optimize the final global features of the model. Finally, the system obtains a final image-level anomaly score through a comprehensive scoring mechanism, combining the initial image-level anomaly score, the maximum response value of the pixel-level anomaly segmentation map, and the score calculated based on the optimized global features. The final score will be used as the recognition criterion for determining whether the image as a whole is abnormal, and the pixel-level anomaly segmentation map provides precise positioning information of the abnormal area.

[0151] For example, when a user inputs an image of a screw, the system will output a clear discrimination result (i.e., whether the screw has a production defect), and when it is determined to be abnormal, it will provide a positioning map highlighting the specific location of the defect.

[0152] The advantage of the embodiment is that through the triple perception learning framework, it systematically solves multiple inherent technical bottlenecks faced by visual language models in the zero-shot anomaly detection task. The triple perception information in this framework complements each other and collaboratively improves the comprehensive performance of the model in the three dimensions of image features, text semantics, and anomaly perception.

[0153] Specifically, the spatial-aware attention enhancement module (SAAM) addresses the spatial confusion problem of visual features in deep networks by injecting external spatial prior knowledge, effectively alleviating the "proxy token phenomenon" caused by improper attention dispersion. This ensures the semantic coherence of local features, providing a solid foundation for subsequent precise anomaly localization. Second, to overcome the difficulty of distinguishing between "normal" and "abnormal" semantics, the attribute-aware guidance module (AAGM) significantly enhances the model's ability to distinguish diverse abnormal semantics by introducing fine-grained attribute descriptions and guided learning, thereby establishing a clearer decision boundary in the feature space. Finally, the anomaly-aware reconstruction module (AARM) addresses the problem of the model's insensitivity to small, local anomalies by using a dynamic attention feedback mechanism to guide the model to focus on potential abnormal areas. This effectively prevents local abnormal signals from being diluted or overwhelmed by global background information. By addressing specific technical challenges in three dimensions, the method achieves higher-quality alignment of image-text features and anomaly identification, which directly determines the model's ability to achieve excellent final detection and localization performance without the need for target domain training data.

[0154] It should be noted that the present embodiment is not a simple calculation method, but can be applied to industrial production and assist in improving the production line of industrial products. For example, in actual application, the method of the present embodiment can be applied to a system as shown in Figure 6 The system includes:

[0155] The image space-aware module is configured to preprocess the input sample image, including uniform size and standardization, and extract global image features through a visual encoder of a visual language model CLIP and multi-level local image features from multiple intermediate layers (e.g., layers 6, 12, 18, and 24). In the visual encoder encoding stage, a spatial-aware attention enhancement module (SAAM) is introduced in the last few layers, replacing the traditional Query-Key (QK) attention mechanism of the Transformer with a Value-Value (VV) self-correlation attention mechanism. An independent visual base model (DINOv2) is used to calculate a self-similarity matrix, and a threshold mask is used to suppress and enhance positive correlation features, thereby generating enhanced visual features and effectively solving the feature space confusion problem caused by the "proxy token phenomenon" in deep networks.

[0156] The text anomaly semantic perception module is configured to generate and optimize text features for anomaly discrimination. A large language model (LLM) is used to generate a series of fine-grained anomaly attribute text descriptions for diversified anomaly visual features, and a learnable general normal and anomaly text prompt is defined. Then, attribute guidance prompt learning in the attribute awareness guidance module (AAGM) semantically aligns the general anomaly prompt with the fine-grained attribute description during training, and introduces semantically similar but class error descriptions as negative samples to solve the problem of insufficient anomaly text semantic description. Meanwhile, in the attribute-guided image aggregation, the local visual features and fine-grained attribute text are fused through a visual-text cross-attention mechanism to generate visual representations specific to the anomaly.

[0157] The anomaly perception feedback module is configured to perform final anomaly recognition and positioning on the query image sent by the terminal device. The core function of the module is to integrate the enhanced visual information and the optimized text information, and generate a final discrimination result through a dynamic feedback mechanism. The module calculates the similarity between the enhanced visual features output by the image space perception module and the optimized text features output by the text anomaly semantic perception module, and generates a high-resolution pixel-level anomaly segmentation map, which is directly used as the positioning basis for the anomaly region. Subsequently, the anomaly perception reconstruction module (AARM) activated only in the inference stage converts the segmentation map information into spatial attention weights and feeds them back to the core calculation layer of the model to dynamically guide the model to focus attention on the potential anomaly region, thereby generating the final global visual features optimized by focusing on the anomaly region. Finally, the module calculates the final image-level anomaly score through a comprehensive scoring mechanism: the mechanism integrates the similarity score of the original global features, the maximum value of the pixel-level anomaly segmentation map, and the score of the final global features optimized by focusing. The final image-level anomaly score calculated will be used as the basis for judging whether the image as a whole is abnormal, and will be output together with the positioning result.

[0158] Specifically, the embodiment is applicable to defect detection and positioning of industrial images, that is, product pictures are processed by the system, the system detects and positions the anomaly based on the similarity calculation of image features and text semantics, and then returns the corresponding detection and positioning result. For example, its working mode is similar to the commonly used image recognition: when an industrial part is produced on a production line, a machine on the production line takes a picture of the part, obtains an image and inputs it into the system. The system then calculates the anomaly score (for recognition) and the anomaly segmentation map (for positioning) of the image and returns the final result. This result can be used to decide whether the production line continues to produce, because when an industrial product with defects is produced, it means that the production line has a problem and may need to be stopped for maintenance, otherwise it will cause waste of resources.

[0159] The embodiment is suitable for defect detection and positioning of industrial images. Specifically, product pictures are enhanced with visual features by an image space perception module, and compared with optimized text features generated by a text anomaly semantic perception module. An anomaly perception feedback module further dynamically optimizes the features and calculates a final anomaly score using the comparison results to determine whether the image has a problem. Finally, the system returns the detection result to decide whether to continue production. If the product has defects, it indicates that the production line may have a fault and needs to be stopped and maintained to prevent resource waste. At the same time, the defect location and its shape information provided by the anomaly segmentation map can also provide accurate guidance for subsequent maintenance of the faulty machine.

[0160] The above merely describes the preferred embodiments of the present application, and it should be noted that those of ordinary skill in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should also be considered within the protection scope of the present application.

Claims

1. A zero-shot anomaly detection method for enhancing a visual language model based on triple perception learning, characterized in that, The method comprises the following steps: S1, preprocessing the sample image to be processed, including size normalization and data standardization, and extracting image-level global features representing global semantics and multi-level local features representing local details through the visual encoder of the visual language model; S2, setting a learnable general normal and abnormal text prompt, and using a large language model to generate fine-grained attribute text description for different abnormal visual features; S3, introducing a spatial perception attention enhancement module in the deep network of the visual encoder in step S1 to generate enhanced multi-level local visual features, which comprises: S3.1, replacing the traditional Query-Key attention mechanism of the deep network of the visual encoder with a Value-Value autocorrelation attention mechanism; S3.2, extracting local features of the input image using an independent visual base model and calculating a self-similarity matrix thereof; S3.3, thresholding the self-similarity matrix to suppress the correlation of regions with a similarity lower than a preset threshold to negative infinity to generate a mask matrix for enhancing positive correlation and suppressing weakly correlated regions; S3.4, fusing the mask matrix with the Value-Value autocorrelation attention to generate local features enhanced in spatial consistency, and injecting the local features into the last k layers of the visual encoder to dynamically guide the attention of the model; S4, inputting the attribute text description generated in step S2 and the general text prompt into an attribute perception guiding module, performing semantic alignment through attribute-guided prompt learning, and generating visual representations associated with specific abnormal attributes through attribute-guided image aggregation; using semantically similar non-correct abnormal attribute descriptions as negative samples for training to obtain optimized text features; S5, calculating the similarity of the multi-level local visual features generated in step S3 and the visual representations associated with specific abnormal attributes generated in step S4 with the text features optimized through semantic alignment in S4 to generate a pixel-level abnormal segmentation map as an abnormal positioning map; meanwhile, calculating the similarity of the image-level global features in step S1 and the general text prompt to generate a global feature score; S6, inputting the pixel-level abnormal segmentation map obtained in step S5 into an abnormal perception reconstruction module to convert it into spatial attention weights, dynamically guiding the model to focus on potential abnormal areas to generate a final visual feature representation; The abnormal perception reconstruction module is activated in the inference stage to convert the pixel-level abnormal segmentation map into spatial attention weights, and the specific method is as follows: S6.1, thresholding the pixel-level abnormal segmentation map generated in step S5 to retain significant abnormal areas and filter out background noise; S6.2, resizing the segmentation map after thresholding and applying Gaussian smoothing to generate a standardized spatial attention weight map; S6.3, weighting and fusing the generated spatial attention weight map with the original attention calculation process of the final layer of the visual encoder to dynamically direct the attention to potential abnormal areas to generate a final visual feature representation sensitive to local abnormalities; S7, calculate the final similarity based on the final visual features generated in step S6 and the optimized text features in S4, and combine the global feature score in S5 and the maximum value of the abnormal segmentation map to obtain the final image-level abnormal score, which is used as the classification judgment result of whether the image is abnormal.

2. The zero-sample anomaly detection method of claim 1, wherein, In step S2, the learnable general normal and abnormal text prompts are represented as follows: ； ； wherein, and respectively represent normal and abnormal cues in the length of learnable text embedding vectors; represents a state description word, represents a detection abnormal class generic label independent of specific objects; The fine-grained attribute text description is induced according to the abnormal visual form, and the core abnormal visual features are summarized. For the core abnormal visual features, a large language model is used to generate multiple sets of text descriptions.

3. The zero-sample anomaly detection method of claim 1, wherein, In step S4, the attribute perception guidance module performs semantic alignment on the general text prompt and the attribute text description through an attribute-guided loss function: The loss function minimizes the distance between the learnable abnormal prompt and the fine-grained abnormal attribute description generated by the large language model in the semantic space, while maximizing the distance from the learnable normal prompt. Hard negative samples of semantically similar but classically incorrect attribute descriptions are introduced to suppress the semantic similarity between different abnormal attributes. The formula is as follows: ； wherein, denotes the Euclidean distance, and denote the feature vectors of the generic normal and abnormal text cues, respectively, denotes the abnormal property feature vector, denotes the set of abnormal property descriptions; denote the first , abnormal property features, respectively.

4. The zero-sample anomaly detection method of claim 1, wherein, In step S4, the attribute perception guidance module performs attribute-guided image aggregation through the following steps: S4.1, perform self-adaptive average pooling on the multi-level local features extracted in step S1 in the neighborhood to obtain the aggregated visual features; S4.2, through the visual-text cross-attention mechanism, the aggregated visual features are fused with the fine-grained abnormal attribute text features obtained in step S2 to generate visual representations for specific abnormal attributes, enhancing the model's recognition and positioning accuracy for complex abnormalities.

5. The zero-sample anomaly detection method of claim 1, wherein, In step S5, the pixel-level abnormal segmentation map is obtained by calculating the cross-modal similarity between multi-level local visual features and text features. The generation steps are as follows: S5.1, the multi-level local visual features with spatial information enhancement generated in step S3 are calculated with the cosine similarity with the normal text cues and the abnormal text cues after the optimization alignment in S4, to obtain the normal score map of each level and the abnormal score map ; S5.2, invert all levels of normal score maps and average them with abnormal score maps, then fuse all levels of average results to integrate all levels of information to generate the final segmentation map; S5.3, upsample the fused score map and apply a Gaussian smoothing filter for processing. The upsampling operation includes but is not limited to bilinear interpolation, nearest neighbor interpolation; make the upsampled result have the same spatial resolution as the input image to obtain the final pixel-level abnormal segmentation map.

6. The zero-sample anomaly detection method of claim 5, wherein, In step S5.3, the pixel-level abnormal segmentation map is generated according to the following formula: ； wherein, is the total number of selected feature layers, is an all-one matrix, represents an up-sampling operation, represents a Gaussian smoothing process with a standard deviation of , , respectively represent the normal score map and the abnormal score map of the th level.

7. The zero-sample anomaly detection method of claim 1, wherein, The learnable general normal and abnormal text prompts in step S2 are optimized by minimizing a combined loss function, including global loss, local loss, and attribute-guided loss. The global loss uses a cross-entropy loss based on the similarity between image-level global features and text features for overall classification, as shown in the following formula: ； wherein, denotes the cosine similarity, denotes the batch, denotes the abnormal attribute description set, denotes the feature vector of the general abnormal text prompt, with denotes the image global feature in the input batch, denotes the abnormal attribute description corresponding to the image abnormality, denotes the hard negative sample abnormal attribute description; The local loss uses a segmentation loss combining Focal loss and Dice loss to optimize the pixel-level positioning accuracy, as shown in the following formula: ； wherein, , represent the first layer normal and abnormal score maps, respectively, is the ground truth mask, is an all-ones matrix, denotes an up-sampling operation; The attribute-guided loss uses the loss function used to optimize the text prompt in step S4. 8.A zero-shot anomaly detection system based on triple perception learning to enhance visual language model, for implementing the zero-shot anomaly detection method of any one of claims 1 to 7, characterized in that, Including: An image space perception module is configured to preprocess an input sample image, and extract global image features and multi-level local image features through a visual encoder of a visual language model; In a visual encoder coding stage, a spatial perception attention enhancement module is introduced in the last several layers, and a Query-Key attention mechanism of a traditional Transformer is replaced by a Value-Value self-correlation attention mechanism; a self-similarity matrix is calculated with the aid of an independent visual base model, and an enhanced positive correlation feature is suppressed through thresholding mask to generate an enhanced visual feature; A text abnormal semantic perception module is configured to generate and optimize text features for abnormality judgment; a large language model is used to generate fine-grained abnormal attribute text descriptions for abnormal visual features, and a learnable general normal and abnormal text prompt is defined; Attribute guidance prompts are learned in an attribute perception guidance module; in the training, the general abnormal prompt is semantically aligned with the fine-grained attribute description, and a description that is semantically similar but has a category error is introduced as a negative sample; in the attribute-guided image aggregation, local visual features and fine-grained attribute texts are fused through a visual-text cross-attention mechanism to generate visual representations for specific abnormalities; An abnormality perception feedback module is configured to perform final abnormality recognition and positioning on a query image sent by a terminal device; similarity calculation is performed on enhanced local visual features output by the image space perception module and optimized text features output by the text abnormal semantic perception module to generate a pixel-level abnormality segmentation map as a basis for abnormal region positioning; through an abnormality perception reconstruction module activated in the inference stage, the pixel-level abnormality segmentation map information is converted into spatial attention weights, which are fed back to the core calculation layer to generate a final global visual feature; An image-level abnormality score is calculated through a comprehensive scoring mechanism as a basis for judging whether the image as a whole has an abnormality, and the positioning result is output together.

Images