A method, apparatus, and electronic device for image anomaly detection based on a visual language model.

By employing a decompositional adaptation mechanism on the visual side and a dual-benchmark decoupling mechanism on the text side, the problems of feature heterogeneity conflict and multi-granularity task requirements in visual language models are resolved, achieving more efficient image anomaly detection, especially with significant results in industrial and medical image detection.

CN123118596APending Publication Date: 2026-11-17DALIAN NATIONALITIES UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611468539.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing anomaly detection methods based on visual language models neglect the heterogeneity of multi-layer features on the visual side, resulting in underfitting of deep features or overfitting of shallow features. Furthermore, they cannot meet the multi-granularity task requirements of image-level classification and pixel-level localization on the text side, leading to a failure to simultaneously improve detection accuracy and localization consistency.

Method used

A decompositional adaptation strategy on the visual side is adopted to perform cross-layer shared alignment and layer-specific residual compensation. A dual text reference benchmark is constructed to achieve task perception decoupling, and feature calibration is performed for image-level classification and pixel-level localization respectively.

Benefits of technology

By decomposing and adapting visual features and decoupling task awareness from text reference benchmarks, the problems of feature heterogeneity conflict and multi-granularity task requirements are solved, improving the overall performance of cross-domain anomaly detection, especially in industrial and medical image detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118596A_ABST
    Figure CN123118596A_ABST
Patent Text Reader

Abstract

This invention discloses an image anomaly detection method, apparatus, and electronic device based on a visual language model, relating to the field of computer vision technology. The method includes: acquiring multi-layer visual features extracted from an input image using a pre-trained visual language model; performing decompositional adaptation processing on the multi-layer visual features to obtain calibrated multi-layer visual features; constructing a text reference benchmark, which includes a first reference benchmark dynamically generated based on the multi-layer visual features and a second reference benchmark that does not dynamically change with the input image; performing image-level anomaly classification based on the calibrated multi-layer visual features and the first reference benchmark, and performing pixel-level anomaly localization based on the calibrated multi-layer visual features and the second reference benchmark. This invention enables task-aware decoupling between accurate visual feature calibration and text reference benchmark, thereby improving the overall performance of cross-domain anomaly detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer vision and industrial anomaly detection technology, specifically relating to an image anomaly detection method, device, and electronic device based on a visual language model. Background Technology

[0002] Visual anomaly detection plays a core role in industrial quality inspection and medical image analysis. In industrial scenarios, there are many types of surface defects (scratches, dents, contamination, etc.) on products, requiring models to be able to simultaneously perform image-level anomaly discrimination and pixel-level anomaly localization.

[0003] Most current mainstream anomaly detection methods (PaDiM, PatchCore, DRAEM, SimpleNet, etc.) assume that sufficient normal training samples in the target domain are available before deployment. In recent years, zero-shot anomaly detection methods based on CLIPs (WinCLIP, AnomalyCLIP, AdaCLIP, AA-CLIP, AF-CLIP, PromptMoE, etc.) are trained on an auxiliary source domain using image-level labels and pixel-level masks, and then directly deployed to the unseen target domain, requiring no target domain data. For example... Figure 1 As shown, existing methods use a uniform layer-independent adaptation to handle heterogeneous multi-layer features on the visual side, while on the text side, the same anchor points are shared for both image-level and pixel-level tasks, ignoring the differences in layer features and the task-specific anchor point requirements.

[0004] Therefore, it can be seen that there are two key adaptation problems in the existing CLIP base methods that have not yet been solved: On the visual side, a layer-independent unified adaptation strategy is adopted for the multi-layer visual features output by the pre-trained model. This ignores the heterogeneity between shallow features (emphasizing local texture) and deep features (emphasizing global semantics) in terms of semantic abstraction and cross-domain offset characteristics. This leads to the phenomenon of underfitting deep features or overfitting shallow features during unified adaptation, causing semantic drift.

[0005] On the text side, the same anchor points are shared for both image-level classification and pixel-level localization, ignoring the asymmetric requirements of the two tasks for anchor point characteristics: image-level classification benefits from instance-adaptive decision boundaries, while pixel-level localization relies on stable references consistent across patches. A single text benchmark cannot simultaneously address both, resulting in a failure to improve detection accuracy and localization consistency in sync. Summary of the Invention

[0006] This invention addresses the heterogeneous conflicts caused by the unified adaptation of visual multi-layer features and the inability of a single text anchor point to meet the needs of multi-granularity tasks in existing anomaly detection methods based on visual language models. It proposes an image anomaly detection method, device, and electronic device based on visual language models, which achieves task perception decoupling between accurate visual feature calibration and text reference benchmark, thereby improving the overall performance of cross-domain anomaly detection.

[0007] The technical means employed in this invention are as follows: An image anomaly detection method based on a visual language model includes the following steps: The multi-layer visual features of the input image are obtained by a pre-trained visual language model, and the multi-layer visual features include category lexical units and patch lexical units; The multi-layer visual features are subjected to decompositional adaptation processing to obtain calibrated multi-layer visual features. The decompositional adaptation processing includes cross-layer shared alignment and layer-specific residual compensation. A text reference benchmark is constructed, which includes a first reference benchmark dynamically generated based on the multi-layer visual features, and a second reference benchmark that does not dynamically change with the input image; Image-level anomaly classification is performed based on the calibrated multi-layer visual features and the first reference benchmark, and pixel-level anomaly localization is performed based on the calibrated multi-layer visual features and the second reference benchmark.

[0008] Further, the multi-layer visual features are subjected to decompositional adaptation processing to obtain calibrated multi-layer visual features, including: A fixed-depth separable Gaussian kernel is used to perform channel-level neighborhood aggregation on patch words in the multi-layer visual features to obtain layer-scale features. The unified alignment transformation is performed on the layer-scale features through a learnable adapter shared by all layers and scales and the projection layer of the pre-trained visual language model to obtain aligned features. The alignment features are residually calculated using a zero-initialized bottleneck projection layer, and the calculation results are superimposed onto the alignment features to obtain the calibrated multilayer visual features.

[0009] Furthermore, the layer-scale features are obtained according to the following formula:

[0010] in, Represents layer-scale features. Indicates category terms, This indicates the use of a separable Gaussian kernel at a fixed depth. patch morphemes Perform channel-level Gaussian neighborhood aggregation; The alignment feature is obtained according to the following formula:

[0011] in, Indicates alignment features, This represents a learnable adapter shared across all layers and scales. Presentation layer normalization operation, This represents the visual projection layer processing of a pre-trained visual language model. The calibrated multi-layer visual features are obtained according to the following formula:

[0012] in, This represents the alignment feature for applying bottleneck residual correction specific to layer l. and For layer-specific bottleneck projection matrix, For the reduction ratio, Representation layer parameters.

[0013] Furthermore, a text reference benchmark is constructed, including: The second reference benchmark is obtained by encoding based on a cue template containing learnable contextual lexical units. Based on the routing signal aggregated from the multi-layer visual features, an image conditional residual is generated through an expert hybrid network, and the image conditional residual is superimposed on the second reference benchmark to obtain the first reference benchmark. Image-level anomaly classification is performed based on the calibrated multi-layer visual features and the first reference benchmark, and pixel-level anomaly localization is performed based on the calibrated multi-layer visual features and the second reference benchmark.

[0014] Furthermore, the routing signal based on the aggregation of the multi-layer visual features generates image conditional residuals through an expert hybrid network, including: The routing signal is obtained by aggregating the patch words in the multi-layer visual features; The routing signal is processed by combining a two-layer multilayer perceptron with Top-K filtering, retaining the K largest logical values, and the gating weight is obtained according to the following formula:

[0015]

[0016] in, Indicates the gating weight, For routing temperature, This indicates that an exponential normalization operation is performed on the expert dimension k. To retain the maximum K' logit values ​​and set the remainder to 0. The operation, For gating networks, Indicates routing signal, Presentation layer normalization operation; The image conditional residual is generated by weighting and mixing the expert features from the normal expert database and the abnormal expert database based on the gating weights.

[0017] in, This represents the conditional residuals of normal basic text features. This represents the conditional residuals of the abnormal basic text features. These represent the normal expert database and the abnormal expert database, respectively. and These represent the k-th expert feature vectors in the normal expert database and the abnormal expert database, respectively. Update the image conditional residuals according to the following formula:

[0018] in, This represents the conditional residual update value of the normal basic text features. This represents the updated value of the conditional residuals for the abnormal basic text features. Indicates the scalar boundary.

[0019] Further, the step of performing image-level anomaly classification based on the calibrated multi-layer visual features and the first reference benchmark, and performing pixel-level anomaly localization based on the calibrated multi-layer visual features and the second reference benchmark, includes: After pooling the category units of each layer in the calibrated multi-layer visual features, the similarity is calculated with the first reference benchmark to obtain the image-level anomaly classification result. The similarity between the patch words of each layer in the calibrated multi-layer visual features and the second reference benchmark is calculated to generate anomaly maps for each layer. The anomaly maps of all layers are then fused to obtain pixel-level anomaly localization results.

[0020] Further, the step of pooling the category terms in each layer of the calibrated multi-layer visual features and then calculating the similarity with the first reference benchmark includes: The category words are averaged and pooled, and then the dot product operation is performed with the first reference benchmark. The classification probability is then output through the softmax function. The step of calculating the similarity between the patch words of each layer in the calibrated multi-layer visual features and the second reference benchmark to generate anomaly maps for each layer, and fusing the anomaly maps of all layers, includes: The patch terminology is multiplied by the second reference benchmark to calculate the dense anomaly graph, and average pooling is applied to the dense anomaly graphs of all layers to obtain the final anomaly graph.

[0021] Furthermore, the method also includes: Freeze the skeleton parameters of the pre-trained visual language model; Based on image-level classification loss, pixel-level segmentation loss, and pairwise patch alignment loss, the parameters of the learnable adapter, layer-specific bottleneck projection matrix, expert hybrid network, gating network, expert database, and learnable context lexical units are updated; the joint objective function is:

[0022] Where L represents the joint objective function; y represents the image-level classification probability vector; y represents the image-level ground truth label. Represents the focus loss function; This represents pixel-level segmentation loss. G represents the pixel-level anomaly map output by the model; G represents the pixel-level anomaly annotation map. This represents the L1 distance between the predicted anomaly map and the labeled map; This represents the pairwise patch alignment loss, used to constrain the local consistency between patch-level visual features and the text reference baseline; This represents the weighting coefficients for the pairwise patch alignment loss.

[0023] The present invention also discloses an image anomaly detection device based on a visual language model, used to implement the method described in any one of the above, comprising: The feature extraction module is used to obtain multi-layer visual features of the input image extracted by a pre-trained visual language model; The decomposition adaptation module is used to perform decomposition adaptation processing on the multi-layer visual features to obtain calibrated multi-layer visual features. The decomposition adaptation processing includes cross-layer shared alignment and layer-specific residual compensation. A reference benchmark construction module is used to construct a text reference benchmark, which includes a first reference benchmark dynamically generated based on the multi-layer visual features, and a second reference benchmark independent of the input image; The multi-granularity prediction module is used to perform image-level anomaly classification based on the calibrated multi-layer visual features and the first reference benchmark, and to perform pixel-level anomaly localization based on the calibrated multi-layer visual features and the second reference benchmark.

[0024] The present invention also discloses an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the image anomaly detection method based on a visual language model as described above.

[0025] Compared with the prior art, the present invention has the following advantages: The technical solution provided in this application effectively solves the problems of feature heterogeneity conflict and mismatch between multi-granularity task requirements in the prior art through a collaborative mechanism of visual-side decompositional adaptation and text-side dual-benchmark decoupling.

[0026] First, at the visual feature processing level, a decompositional adaptation strategy is adopted, which includes cross-layer shared alignment and layer-specific residual compensation. A unified alignment transformation is performed on features across all layers, ensuring the consistency of the baseline for features of different semantic depths in a unified space and preventing semantic drift caused by independent adaptation of each layer. Based on this, layer-specific bottleneck residual correction is applied, and a zero-initialization strategy is adopted, ensuring that the model maintains an identity mapping in the early stages of training, and subsequently performs minor compensation only for the heterogeneous offsets specific to each layer. This decomposition mechanism fundamentally eliminates the problems of deep underfitting and shallow overfitting caused by unified adaptation, significantly improving the cross-domain representation capability of multi-layer visual features.

[0027] Secondly, at the text reference benchmark construction level, a task-aware dual-benchmark decoupling assignment mechanism is proposed. For image-level classification tasks, a routing signal aggregated from multi-layer visual features is used to dynamically generate a first reference benchmark through an expert hybrid network. This allows the first benchmark to adaptively adjust the decision boundary based on the local patterns of the input image, enhancing the flexibility of global discrimination. For pixel-level localization tasks, a second reference benchmark, independent of the input image, is generated using a cue template encoding containing learnable contextual terms. This provides a stable semantic reference consistent across patches, avoiding the disruption of dense localization consistency caused by dynamic adjustments. By assigning the two complementary benchmarks to their respective prediction branches according to task granularity, an optimal balance between global adaptability and local stability is achieved. This ensures high-precision pixel-level localization while further improving the accuracy of image-level anomaly classification. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1The diagrams show a comparison of the execution architecture of existing methods and the method of the present invention, where (a) is a diagram of the execution architecture of existing methods and (b) is a diagram of the execution architecture of the method of the present invention.

[0030] Figure 2 This is a schematic diagram of the overall framework structure of the method of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1 like Figure 1 (b) and Figure 2 As shown, this invention provides an image anomaly detection method based on a visual language model. Under the premise of freezing the CLIP (Contrastive Language-Image Pre-training) skeleton, SSVA (Shared-to-Specific Visual Adaptation) is applied on the visual side by sharing the image with a specific visual adaptation model. This paper addresses the hierarchical adaptation problem of multi-layered heterogeneous features. On the text side, it uses DTAL (Dual Text Anchor Learning) to solve the different requirements of text anchors for image-level classification and pixel-level localization. SSVA calibrates multi-layered visual features by sharing specific pathways, while DTAL learns two complementary anchors and decouples and assigns them according to task granularity. SSVA extracts and calibrates multi-layered visual features, i.e., MSA (Multi-scale Spatial Aggregation) → SAA (Shared Alignment Adaptation / Adapter) → LSRA (Layer-Specific Residual Adaptation). DTAL learns static anchors and visually guided anchors and performs task decoupling and assignment. CLIP, as the pre-trained visual language model in this application, has its skeleton completely frozen and only lightweight components are trained. The method includes the following steps: S1. Obtain multi-layer visual features extracted from the input image by a pre-trained visual language model. The multi-layer visual features include category terms and patch terms.

[0034] First, input image Zoom to The resolution is then input into the frozen CLIP visual encoder (ViT-L / 14@336px), and then from the preset K intermediate Transformer layers (the first... Extracting visual features from multiple layers (layers). Features of each layer... ,in As category morphemes, There are N patch words, and C is the feature dimension.

[0035] S2. Perform decompositional adaptation processing on the multi-layer visual features to obtain calibrated multi-layer visual features. The decompositional adaptation processing includes cross-layer shared alignment and layer-specific residual compensation.

[0036] This embodiment performs multi-layer visual feature extraction based on the features of each layer and subsequent operations: S201. Channel-level neighborhood aggregation is performed on the patch words in the multi-layer visual features using a fixed-depth separable Gaussian kernel to obtain layer-scale features. This step is used for multi-scale spatial aggregation (MSA). Specifically, the patch words are reshaped into a two-dimensional grid, and a fixed-depth separable Gaussian kernel is used. (core size) Channel-level Gaussian neighborhood aggregation is performed using reflection filling, while category terms remain unchanged.

[0037] in, Represents layer-scale features. Indicates category terms, This indicates the use of a separable Gaussian kernel at a fixed depth. patch morphemes Perform channel-level Gaussian neighborhood aggregation. Preserve the original resolution for identity mapping. and Medium and large spatial contexts are provided respectively. Gaussian kernel weights are pre-computed and fixed during training, and MSA has no learnable parameters.

[0038] S202. The unified alignment transformation is performed on the layer-scale features using a learnable adapter shared across all layers and scales, and the projection layer of the pre-trained visual language model, to obtain aligned features. This step is used for shared alignment adaptation (SAA), specifically, each layer-scale feature... Learnable adapter shared across all layers and scales The subsequent normalized and frozen CLIP visual projection layer :

[0039] in, Indicates alignment features, This represents a learnable adapter shared across all layers and scales. Presentation layer normalization operation, This represents the visual projection layer processing of the pre-trained visual language model. Among them, It is implemented using a lightweight, trainable attention layer. The shared design forces features of different semantic depths to be adapted under a uniform alignment-preserving transformation, preventing independent drift of each layer.

[0040] S203. The alignment features are calculated using a zero-initialized bottleneck projection layer, and the calculation result is superimposed onto the alignment features to obtain the calibrated multilayer visual features. This step is used to achieve Layer-Specific Residual Adaptation (LSRA): the alignment features output by the SAA... Apply a specific bottleneck residual correction for the l-th layer above:

[0041] in, This represents the alignment feature for applying specific bottleneck residual correction at layer l. and For layer-specific bottleneck projection matrix, For the reduction ratio, Representation layer parameters. Zero initialization ensures that the residual term is zero at the start of training. Three scales are shared within the same layer. The parameters of different layers are independent.

[0042] S3. Construct a text reference benchmark, which includes a first reference benchmark dynamically generated based on the multi-layer visual features, and a second reference benchmark independent of the input image. This step specifically includes the following sub-steps: S301. Based on the routing signal aggregated from the multi-layer visual features, an image conditional residual is generated through an expert hybrid network, and the image conditional residual is superimposed on the basic text features to obtain the first reference benchmark. Simultaneously, a cue template containing learnable contextual terms is encoded to obtain the second reference benchmark.

[0043] First, construct lexical units containing learnable context. Normal text prompt template and abnormal text prompt template and will and The frozen CLIP text encoder is used for encoding to obtain static, normal basic text features. and static anomaly basic text features The specific implementation includes: constructing a pair of lexical units containing learnable context. Prompt template:

[0044] Where M is the context length, and the static normal basic text features are obtained by encoding with the frozen CLIP text encoder. and static anomaly basic text features This serves as a second reference benchmark. Although the contextual lexical units are optimized in the source domain, the resulting anchor points are independent of any specific test image and can serve as a stable reference for pixel-level localization.

[0045] Furthermore, the routing signal based on the aggregation of the multi-layer visual features generates image conditional residuals through an expert hybrid network, including: a. Aggregate the patch-level features in the multi-layer visual features to obtain the routing signal. This step aggregates the patch-level features of all 3K calibration branches output by SSVA to obtain the routing signal. Patch features are used instead of just category terms because patch features directly encode spatial information about local anomalous patterns.

[0046] b. The routing signal is processed using a two-layer multilayer perceptron combined with Top-K filtering, retaining the K largest logical values, and the gating weight is obtained according to the following formula:

[0047]

[0048] in, Indicates the gating weight, For routing temperature, This indicates that exponential normalization is performed on the expert dimension k to obtain the gating weights for each expert. To retain the maximum K' logit values ​​and set the remainder to 0. The operation, It is a two-layer multilayer perceptron (GELU activated) gated network. Indicates routing signal, Presentation layer normalization operation.

[0049] c. Based on the gating weights, the expert features in the normal expert database and the abnormal expert database are weighted and mixed to generate the image conditional residual. This residual is then scaled by a preset scalar bound and superimposed onto the basic text features. The image conditional residual is obtained according to the following formula:

[0050] in, This represents the conditional residuals of normal basic text features. This represents the conditional residuals of the abnormal basic text features. These represent the normal expert database and the abnormal expert database, respectively. and These represent the k-th expert feature vectors in the normal expert database and the abnormal expert database, respectively; expert database Initialize with a small standard deviation.

[0051] Before the training begins, establish regular expert databases. and anomaly expert database Both are The trainable parameter matrix is ​​given, where each row corresponds to an expert feature vector. During training, the gating network outputs the weights of each expert based on the routing signal, and then... and The expert vectors in the data are obtained by weighted summation. and Subsequently, the joint objective function was used to... and The gating network parameters and prompt template context words are updated end-to-end. In this embodiment, the gating network parameters and prompt template context words are updated end-to-end. and The elements of the feature vectors of each expert are distributed using a zero-mean Gaussian distribution. Random initialization, where For a small standard deviation, a value of 0.02 is preferred; after initialization and These parameters are used as learnable parameters in subsequent gradient updates. Initialization with a small standard deviation minimizes the initial image conditional residual magnitude, preventing damage to the semantic structure of the frozen CLIP text features during early training.

[0052] Update the image conditional residuals according to the following formula:

[0053] in, This represents the conditional residual update value of the normal basic text features. This represents the conditional residual update value of the abnormal basic text feature. Representing scalar boundaries. Relatives are represented by scalar boundaries. After scaling, it is directly superimposed onto the static anchor point without requiring additional normalization constraints. As the first reference base.

[0054] S302. Assign the first reference benchmark to the image-level anomaly classification task and the second reference benchmark to the pixel-level anomaly localization task.

[0055] S4. Based on the calibrated multi-layer visual features and the first reference benchmark, perform image-level anomaly classification, and based on the calibrated multi-layer visual features and the second reference benchmark, perform pixel-level anomaly localization. Specifically, this includes: a. After pooling the category terms of each layer in the calibrated multi-layer visual features, perform similarity calculation with the first reference benchmark to obtain the image-level anomaly classification result.

[0056] This step performs average pooling on the category terms, then performs a dot product operation with the first reference benchmark, and outputs the classification probability using the softmax function. Specifically, the first reference benchmark is used. The normalized category terms of each SSVA branch are averaged and then dot-producted with the visual guidance anchor points:

[0057] in, This represents the image-level visual features obtained by average pooling the normalized category terms output from each SSVA branch; and These represent the normal text reference features and the abnormal text reference features in the first reference benchmark, respectively. This represents a text reference matrix formed by stacking normal text reference features and abnormal text reference features according to the category dimension; Indicates the transpose operation; Obtain the similarity logit between the normal and abnormal categories; Convert the similarity logit to image-level classification probability. ,and It typically includes two components: normal probability and abnormal probability. The probability corresponding to the abnormal category can be used as the image-level anomaly score of the input image.

[0058] b. Calculate the similarity between the patch words of each layer in the calibrated multi-layer visual features and the second reference benchmark to generate anomaly maps for each layer, and fuse the anomaly maps of all layers to obtain pixel-level anomaly localization results.

[0059] This step involves performing a dot product operation between the patch terminology and the second reference benchmark to calculate a dense anomaly graph. Average pooling is then applied to the dense anomaly graphs across all layers to obtain the final anomaly graph. Specifically, the second reference benchmark is used. The anomaly graph is calculated by dot product of patch words in each branch and static anchor points. The final dense anomaly graph is obtained by average pooling of all branch anomaly graphs. .

[0060] Furthermore, the method also includes: Freeze all parameters of the CLIP skeleton of the pre-trained visual language model; optimize only the shared adapter in SAA. Hierarchical-specific bottleneck projection in LSRA Gated networks in VGRA (Visual-Guided Residual Adaptation) and expert database And learnable context lexical units in the prompt template. Joint objective function:

[0061] It includes image-level classification loss, pixel-level segmentation loss, and pairwise patch alignment loss. It is trained on a single NVIDIA RTX 4090 GPU using the AdamW optimizer. All parameters are fixed after training. It performs direct forward propagation on any unseen target domain image, simultaneously outputting image-level anomaly detection results and pixel-level anomaly heatmaps, without requiring any target domain data, labels, or test-time parameter updates.

[0062] The following specific application examples will further illustrate the scheme and effects of the method of the present invention.

[0063] This embodiment verifies the performance of the present invention in a zero-sample cross-domain scenario for industrial anomaly detection.

[0064] Protocol: A standard source-domain supervised, target-domain independent evaluation protocol is adopted. Training is performed on MVTec, with zero-shot testing on VisA, BTAD, DAGM, DTD, and six medical datasets; reverse training is performed on VisA and testing on MVTec. All comparison methods (AnomalyCLIP, AdaCLIP, AA-CLIP, AF-CLIP, PromptMoE) use the CLIP ViT-L / 14@336px skeleton.

[0065] Results: As shown in Table 1, in the MVTec→VisA direction, the image-level AUROC (Area Under the Receiver Operating Characteristic Curve) / AP (Average Precision) of this invention reached 88.9% / 90.8%, and the pixel-level AUROC reached 96.3%, maintaining its leading position even with the addition of the AF-CLIP baseline (88.5% / 90.0%, 96.2%). In the VisA→MVTec reverse evaluation, the pixel-level AUROC / PRO reached 92.2% / 85.1%. Across the six medical target domains, BrainMRI (I-AUROC 96.3%), Br35H (I-AUROC 98.4%), and ISIC (P-AUROC 94.6%) all achieved the best or second-best results.

[0066] In addition, this embodiment also verifies the independent contributions of each component of SSVA and DTAL through ablation experiments.

[0067] SSVA ablation: Using frozen CLIPs and fixed text anchors as the baseline (I-AUROC / P-AUROC = 88.1% / 88.2% on MVTec). Incremental addition of MSA improved the accuracy to 89.9% / 90.4%, validating the importance of multi-scale spatial context for dense localization. Further addition of SAA reached 90.9% / 91.3%, demonstrating that shared alignment-preserving pathways provide a stronger foundation for image-text matching. Adding LSRA on top of MSA+SAA achieved 91.7% / 91.5%, showing significant image-level gain while maintaining pixel-level stability, confirming that hierarchical residual adaptation primarily benefits global transfer capabilities.

[0068] Table 1: Pixel-level and image-level results in industrial and medical datasets.

[0069] Table 2 Results of SSVA ablation experiment

[0070] DTAL ablation: Fixed / fixed approach (87.7% / 95.9%) establishes a strong localization baseline but is limited at the image level. Decoupling the unrouting approach (global residual replacing visually guided routing) recovers to 88.6% / 96.1%, demonstrating that task-aware assignment is the core contribution. Adaptive / adaptive approach improves the image level (88.5%) but harms the pixel level (95.5%), confirming that directly applying adaptive anchors to dense branches disrupts localization consistency. Full DTAL achieves the best balance at 88.9% / 96.3%. Reverse assignment (fixed anchors for the image level, adaptive anchors for the pixel level) yields the worst results (87.3% / 95.3%), confirming the criticality of assignment direction.

[0071] Table 3: Results of DTAL ablation experiments

[0072] Example 2 This embodiment also discloses an image anomaly detection device based on a visual language model. This device is a functional modular encapsulation of the displacement sensor fault detection method in the foregoing embodiments. It is used to implement the method described in Embodiment 1, and includes: a feature extraction module, a decompositional adaptation module, a reference benchmark construction module, and a multi-granularity prediction module. Specifically, the feature extraction module is used to acquire multi-layer visual features extracted from the input image by a pre-trained visual language model; the decompositional adaptation module is used to perform decompositional adaptation processing on the multi-layer visual features to obtain calibrated multi-layer visual features, the decompositional adaptation processing including cross-layer shared alignment and layer-specific residual compensation; the reference benchmark construction module is used to construct a text reference benchmark, the text reference benchmark including a first reference benchmark dynamically generated based on the multi-layer visual features and a second reference benchmark independent of the input image; the multi-granularity prediction module is used to perform image-level anomaly classification based on the calibrated multi-layer visual features and the first reference benchmark, and to perform pixel-level anomaly localization based on the calibrated multi-layer visual features and the second reference benchmark.

[0073] Through the modular architecture described above, this embodiment transforms the image anomaly detection method based on a visual language model into a virtual product that can be independently distributed, integrated, or sold. This design not only covers the firmware implementation in embedded controllers but is also applicable to various hardware carriers such as host computer software, edge computing gateways, or dedicated detection instruments. It should be understood that the above module division is only a logical functional definition. In actual physical implementation, multiple modules can be combined into one processing unit, or one module can be split into multiple sub-units working collaboratively. As long as the whole can achieve the fault detection function described in this application, it should not be considered as departing from the protection scope of this application.

[0074] Example 3 This embodiment discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements an image anomaly detection method based on a visual language model as described in Embodiment 1.

[0075] Specifically, this embodiment maps the abstracted method flow and functional modules in Embodiment 1 onto a specific physical hardware carrier, thereby providing a physical implementation foundation for the technical solution. The electronic device here does not refer to a specific model or brand of controller, but rather encompasses all general-purpose or special-purpose computing devices with data storage and instruction execution capabilities. As long as the electronic device can read the program code in the memory through an internal bus or communication interface, and the processor executes the corresponding sampling, calculation, and judgment operations according to timing logic, it falls within the scope of this embodiment.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image anomaly detection method based on a visual language model, characterized in that, Includes the following steps: The multi-layer visual features of the input image are obtained by a pre-trained visual language model, and the multi-layer visual features include category lexical units and patch lexical units; The multi-layer visual features are subjected to decompositional adaptation processing to obtain calibrated multi-layer visual features. The decompositional adaptation processing includes cross-layer shared alignment and layer-specific residual compensation. A text reference benchmark is constructed, which includes a first reference benchmark dynamically generated based on the multi-layer visual features, and a second reference benchmark that does not dynamically change with the input image; Image-level anomaly classification is performed based on the calibrated multi-layer visual features and the first reference benchmark, and pixel-level anomaly localization is performed based on the calibrated multi-layer visual features and the second reference benchmark.

2. The image anomaly detection method based on a visual language model according to claim 1, characterized in that, The multi-layer visual features are subjected to decompositional adaptation processing to obtain calibrated multi-layer visual features, including: A fixed-depth separable Gaussian kernel is used to perform channel-level neighborhood aggregation on patch words in the multi-layer visual features to obtain layer-scale features; The unified alignment transformation is performed on the layer-scale features through a learnable adapter shared by all layers and scales and the projection layer of the pre-trained visual language model to obtain aligned features. The alignment features are residually calculated using a zero-initialized bottleneck projection layer, and the calculation results are superimposed onto the alignment features to obtain the calibrated multilayer visual features.

3. The image anomaly detection method based on a visual language model according to claim 2, characterized in that, The layer-scale features are obtained according to the following formula: in, Represents layer-scale features. Indicates category terms, This indicates the use of a separable Gaussian kernel at a fixed depth. patch morphemes Perform channel-level Gaussian neighborhood aggregation; The alignment feature is obtained according to the following formula: in, Indicates alignment features, This represents a learnable adapter shared across all layers and scales. Presentation layer normalization operation, This represents the visual projection layer processing of a pre-trained visual language model. The calibrated multi-layer visual features are obtained according to the following formula: in, This represents the alignment feature for applying bottleneck residual correction specific to layer l. and For layer-specific bottleneck projection matrix, For the reduction ratio, Representation layer parameters.

4. The image anomaly detection method based on a visual language model according to claim 1, characterized in that, Constructing a text reference benchmark includes: The second reference benchmark is obtained by encoding based on a cue template containing learnable contextual lexical units. Based on the routing signal aggregated from the multi-layer visual features, an image conditional residual is generated through an expert hybrid network, and the image conditional residual is superimposed on the second reference benchmark to obtain the first reference benchmark. Image-level anomaly classification is performed based on the calibrated multi-layer visual features and the first reference benchmark, and pixel-level anomaly localization is performed based on the calibrated multi-layer visual features and the second reference benchmark.

5. The image anomaly detection method based on a visual language model according to claim 4, characterized in that, The routing signal based on the aggregation of the multi-layer visual features is used to generate image conditional residuals through an expert hybrid network, including: The routing signal is obtained by aggregating the patch words in the multi-layer visual features; The routing signal is processed by combining a two-layer multilayer perceptron with Top-K filtering, retaining the K largest logical values, and the gating weight is obtained according to the following formula: in, Indicates the gating weight, For routing temperature, This indicates that an exponential normalization operation is performed on the expert dimension k. To retain the maximum K' logit values ​​and set the remainder to 0. The operation, For gating networks, Indicates routing signal, Presentation layer normalization operation; The image conditional residual is generated by weighting and mixing the expert features from the normal expert database and the abnormal expert database based on the gating weights. in, This represents the conditional residuals of normal basic text features. This represents the conditional residuals of the abnormal basic text features. These represent the normal expert database and the abnormal expert database, respectively. and These represent the k-th expert feature vectors in the normal expert database and the abnormal expert database, respectively. Update the image conditional residuals according to the following formula: in, This represents the conditional residual update value of the normal basic text features. This represents the updated value of the conditional residuals for the abnormal basic text features. Indicates the scalar boundary.

6. The image anomaly detection method based on a visual language model according to claim 4, characterized in that, The step of performing image-level anomaly classification based on the calibrated multi-layer visual features and the first reference benchmark, and performing pixel-level anomaly localization based on the calibrated multi-layer visual features and the second reference benchmark, includes: After pooling the category units of each layer in the calibrated multi-layer visual features, the similarity is calculated with the first reference benchmark to obtain the image-level anomaly classification result. The similarity between the patch words of each layer in the calibrated multi-layer visual features and the second reference benchmark is calculated to generate anomaly maps for each layer. The anomaly maps of all layers are then fused to obtain pixel-level anomaly localization results.

7. The image anomaly detection method based on a visual language model according to claim 6, characterized in that, The step of pooling the category terms in each layer of the calibrated multi-layer visual features and then calculating the similarity with the first reference benchmark includes: The category words are averaged and pooled, and then the dot product operation is performed with the first reference benchmark. The classification probability is then output through the softmax function. The step of calculating the similarity between the patch words of each layer in the calibrated multi-layer visual features and the second reference benchmark to generate anomaly maps for each layer, and fusing the anomaly maps of all layers, includes: The patch terminology is multiplied by the second reference benchmark to calculate the dense anomaly graph, and average pooling is applied to the dense anomaly graphs of all layers to obtain the final anomaly graph.

8. The image anomaly detection method based on a visual language model according to claim 1, characterized in that, The method further includes: Freeze the skeleton parameters of the pre-trained visual language model; Based on image-level classification loss, pixel-level segmentation loss, and pairwise patch alignment loss, the parameters of the learnable adapter, layer-specific bottleneck projection matrix, expert hybrid network, gating network, expert database, and learnable context lexical units are updated; the joint objective function is: Where L represents the joint objective function; y represents the image-level classification probability vector; y represents the image-level ground truth label. Represents the focus loss function; This represents pixel-level segmentation loss. G represents the pixel-level anomaly map output by the model; G represents the pixel-level anomaly annotation map. This represents the L1 distance between the predicted anomaly map and the labeled map; This represents the pairwise patch alignment loss, used to constrain the local consistency between patch-level visual features and the text reference baseline; This represents the weighting coefficients for the pairwise patch alignment loss.

9. An image anomaly detection device based on a visual language model, used to implement the method described in any one of claims 1-8, characterized in that, include: The feature extraction module is used to obtain multi-layer visual features of the input image extracted by a pre-trained visual language model; The decomposition adaptation module is used to perform decomposition adaptation processing on the multi-layer visual features to obtain calibrated multi-layer visual features. The decomposition adaptation processing includes cross-layer shared alignment and layer-specific residual compensation. A reference benchmark construction module is used to construct a text reference benchmark, which includes a first reference benchmark dynamically generated based on the multi-layer visual features, and a second reference benchmark independent of the input image; The multi-granularity prediction module is used to perform image-level anomaly classification based on the calibrated multi-layer visual features and the first reference benchmark, and to perform pixel-level anomaly localization based on the calibrated multi-layer visual features and the second reference benchmark.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements an image anomaly detection method based on a visual language model as described in any one of claims 1 to 8.