Foreground-background soft separation zero sample anomaly detection method based on CLIP model

Abstract

The invention discloses a foreground-background soft separation zero sample anomaly detection method based on a CLIP model, and aims to solve the technical problems of low stability and reliability of a detection result in an existing anomaly detection method. An enhanced text embedding vector is generated by constructing a multi-branch feature of a sample to be detected; soft separation is carried out on a foreground region and a background region of image features through a multi-layer perceptron, further feature enhancement is carried out on the foreground region and the background region through a multi-head self-attention mechanism, a foreground enhancement feature vector and a background enhancement feature vector are obtained, and on the basis that the background enhancement feature vector is suppressed, the foreground region and the background region are subjected to feature enhancement. Generating a visual feature vector; and according to the enhanced text embedding vector and the visual feature vector, completing anomaly detection of the to-be-detected sample.

Description

Technical Field

[0001] This invention relates to computer vision, and more specifically to a foreground-background soft separation zero-shot anomaly detection method based on the CLIP (Contrastive Language–Image Pre-training) model. Background Technology

[0002] Anomaly detection (AD) aims to identify data samples or regions that are significantly different from normal samples, and has important application value in fields such as industrial product inspection and medical imaging diagnosis. However, in practical applications, anomaly samples are usually characterized by low frequency of occurrence, diverse forms, and high annotation costs, making it difficult to obtain anomaly data and thus limiting the application of traditional supervised learning methods.

[0003] To address this, researchers have proposed unsupervised anomaly detection and zero-shot anomaly detection methods. Among these, the development of vision-language pre-trained models, especially the breakthrough of the CLIP model in aligning visual features with text semantics, has made anomaly detection possible without annotating abnormal samples.

[0004] Chinese patent application CN120635483A discloses a zero-sample anomaly detection method, apparatus, and electronic device. The method involves acquiring an image to be detected and extracting visual features using a CLIP model; visually enhancing the visual features to obtain enhanced visual features; injecting the enhanced visual features into a learnable text prompt template to generate adaptive text prompts; injecting the adaptive text prompts into a text encoder for encoding to obtain a text embedding representation; mapping the adaptive text prompts to a visual space to obtain visual prompts, and inputting them into local visual features to obtain scale visual features; calculating anomaly scores and anomaly localization maps at preset scales based on the text embedding representations and scale visual features, and fusing them to obtain the detection result.

[0005] Chinese patent application CN120672752A discloses a few-sample industrial anomaly detection method based on CLIP with dual-cue optimization. The method constructs learnable normal text cue templates and abnormal text cue templates using learnable normal word vectors and abnormal word vectors, respectively. Normal sample images used for model training undergo image enhancement and image anomaly synthesis processing to generate enhanced images and anomaly synthesized images. The learnable normal and abnormal text cue templates are input into the text encoder of the CLIP model to obtain text cue embeddings. The normal sample images, enhanced images, and anomaly synthesized images are input into the image encoder of the CLIP model to obtain visual embeddings. The similarity between the text cue embeddings and visual embeddings is measured to optimize the learnable text cue training, enabling the use of the trained CLIP model for industrial anomaly detection.

[0006] The shortcomings of the above-mentioned existing technologies are: (1) The text features are simple, which makes the model more biased towards common or significant normal areas during the alignment process, thereby weakening the ability to perceive fine-grained and local anomalies, resulting in low reliability of the detection results; (2) The visual features do not distinguish between foreground and background contributions. When there is a complex background or cross-domain distribution shift in the sample to be detected, the noise generated by the background area is easy to interfere with the anomaly localization results, resulting in low stability and reliability of the detection results. Summary of the Invention

[0007] The purpose of this invention is to address the technical problems of low stability and reliability of existing anomaly detection methods due to the limited expression of text features and the failure of visual features to distinguish between foreground and background contributions. This invention provides a zero-sample anomaly detection method based on the CLIP model with soft foreground-background separation.

[0008] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0009] A foreground-background soft separation zero-sample anomaly detection method based on the CLIP model, characterized by the following steps:

[0010] S1. A CLIP-based text encoder constructs multi-branch features of the sample to be detected and generates an enhanced text embedding vector based on the multi-branch features.

[0011] S2. Based on CLIP, an image encoder is constructed to build n layers of image features of the sample to be detected, where n≥1. For each layer of image features, feature enhancement is performed on the original feature dimension, semantic dimension and spatial dimension to obtain the basic semantic feature vector, semantic feature enhancement vector and spatial feature enhancement vector.

[0012] S3. Soft classification masks are generated by using a multilayer perceptron to generate the basic semantic feature vector, semantic feature enhancement vector, and spatial feature enhancement vector, respectively, to initially distinguish the foreground and background regions of the n-layer image features. Then, a multi-head self-attention mechanism is used to further enhance the features of the foreground and background regions, resulting in n foreground and background feature enhancement vectors. These vectors are accumulated and merged to obtain the final foreground and background feature enhancement vector, which includes a foreground enhancement feature vector and a background enhancement feature vector.

[0013] S4. Suppress the background enhancement feature vector in the foreground and background feature enhancement vector to obtain the foreground enhancement feature vector without background interference, and calculate the abnormal residual feature vector accordingly. Based on the abnormal residual feature vector, generate the visual feature vector.

[0014] S5. Based on the enhanced text embedding vector and visual feature vector, determine the abnormal feature vector of the sample to be detected, and output an abnormal detection report based on the abnormal feature vector, thereby completing the abnormal detection of the sample to be detected.

[0015] Furthermore, step S3 is as follows:

[0016] S3.1. Using a multilayer perceptron, soft classification masks are generated for the basic semantic feature vector, semantic feature enhancement vector, and spatial feature enhancement vector of each layer of image features to obtain the first foreground probability and the first background probability.

[0017] S3.2 Multiply the basic semantic feature vector, semantic feature enhancement vector, and spatial feature enhancement vector by the first foreground probability and the first background probability, respectively, to obtain the foreground and background regions of the n-layer image features. Then, further feature enhancement is performed on the foreground and background regions through a multi-head self-attention mechanism to obtain n foreground and background feature enhancement vectors. After accumulating and merging them, the final foreground and background feature enhancement vector is obtained.

[0018] Furthermore, step S4 is detailed as follows:

[0019] S4.1 Obtain the mean and maximum values ​​of each element in the foreground and background feature enhancement vector, and determine the global background feature vector of the sample to be detected based on the mean and maximum values.

[0020] S4.2. Compare the foreground and background feature enhancement vectors with the global background feature vectors using cosine similarity to obtain the second background probability. Based on the second background probability, determine the second foreground probability.

[0021] S4.3 Subtract the global background feature vector from the foreground and background feature enhancement vector to obtain the background-free foreground enhancement feature vector. Multiply the background-free foreground enhancement feature vector with the second foreground probability to obtain the abnormal residual feature vector.

[0022] S4.4 Add the abnormal residual feature vector and the foreground and background feature enhancement vector to obtain the visual feature vector.

[0023] Furthermore, in step S2, the semantic dimension of each layer of image features is enhanced to obtain the semantic feature enhancement vector, as follows:

[0024] a.1. Calculate and generate the foreground soft mask for each layer of image features based on the multi-dimensional anomaly score, and determine the background soft mask based on the foreground soft mask;

[0025] a.2. Adaptively threshold binarize the foreground soft mask and the background soft mask to obtain the hard mask of each layer of image features;

[0026] a.3. Based on the obtained hard mask, perform feature separation on the image features of each layer to obtain foreground features and background features;

[0027] a.4. Calculate the information richness weight of the foreground features based on the local diversity of the foreground features, and calculate the stability weight of the background features based on the correlation of the image features.

[0028] a.5. Based on the information richness weight and stability weight, the foreground features and background features are weighted and aggregated separately through a global self-attention mechanism to obtain the semantic feature enhancement vector.

[0029] Furthermore, in step S2, the spatial dimension feature enhancement is performed on each layer of image features to obtain the spatial feature enhancement vector, as follows:

[0030] b.1. Calculate and generate the foreground soft mask for each layer of image features based on the multi-dimensional anomaly score, and determine the background soft mask based on the foreground soft mask;

[0031] b.2. Adaptively threshold binarize the foreground soft mask and the background soft mask to obtain the hard mask of each layer of image features;

[0032] b.3. Based on the obtained hard mask, perform feature separation on the image features of each layer to obtain foreground features and background features;

[0033] b.4 Reshape the foreground and background features from a sequential form to a 2D spatial form, and determine the local window mask for the foreground features and the local window mask for the background features;

[0034] b.5. Calculate the information richness weight of the local window mask for the foreground features and perform weighted aggregation with the foreground features to obtain the foreground aggregated features;

[0035] b.6. Calculate the stability weight of the local window mask for the background features and perform weighted aggregation with the background features to obtain the background aggregated features;

[0036] b.7. Add the foreground aggregation features and the background aggregation features, and reshape the result back into a sequence to obtain the spatial feature enhancement vector.

[0037] Furthermore, step S1 is detailed as follows:

[0038] S1.1 Construct the basic text description of the sample to be detected based on the semantic category name and the preset positive and negative state template;

[0039] S1.2. Concatenate the predefined learnable context vector with the basic text description to generate the corresponding prompt text;

[0040] S1.3. Based on CLIP's text encoder, the prompt text is subjected to deep semantic encoding to obtain text features. Then, a three-path feature extraction strategy is used to extract multi-branch features of the text features. The first path is global average pooling to extract the overall semantic information of the text features and obtain global average features. The second path is sentence end token extraction to obtain sentence end token features. The third path is attention weighting, which calculates token weights through a learnable token selector network and aggregates key semantic information in a weighted manner to obtain attention weighted features.

[0041] S1.4. The global average features, sentence end token features, and attention-weighted features are weighted and fused to generate an enhanced text embedding vector.

[0042] Furthermore, the formulas for the multidimensional anomaly scores in steps a.1 and b.1 are as follows:

[0043]

[0044] In the formula, Indicates multi-dimensional anomaly scores, Representing image features Local significance, Weights representing local significance; Representing image features Distance to the feature center Weights representing distance; Representing image features semantic consistency Weights representing semantic consistency; Representing image features The temporal changes, Weights representing temporal variations;

[0045] The definition is as follows:

[0046]

[0047] In the formula, Indicates average pooling. Indicates the number of windows;

[0048] The definition is as follows:

[0049]

[0050] In the formula, Representing image features The feature center;

[0051] The definition is as follows:

[0052]

[0053] In the formula, Representing image features Global category tokens in the context of Tokens;

[0054] The definition is as follows:

[0055]

[0056] In the formula, This represents the image features of the current layer at the current time. This represents the image features of the previous layer corresponding to the previous time step.

[0057] Furthermore, the formulas for calculating and generating the foreground soft mask in steps a.1 and b.1 are as follows:

[0058]

[0059] In the formula, Indicates a foreground soft mask. This indicates an adaptive threshold.

[0060] Furthermore, step S5 is detailed as follows:

[0061] S5.1 Multiply the enhanced text embedding vector and the visual feature vector to obtain the preliminary anomaly feature vector;

[0062] S5.2. Add the preliminary abnormal feature vector and the foreground enhancement feature vector after removing background interference to obtain the final abnormal feature vector of the sample to be detected. Convert the final abnormal feature vector into a high-resolution pixel-level abnormal score map. After filtering and normalizing the abnormal score map, output an abnormal detection report to complete the abnormal detection of the sample to be detected. The abnormal detection report includes the sample to be detected with the abnormal region marked and the score value used to evaluate the accuracy of abnormal detection.

[0063] Compared with the prior art, the present invention has the following beneficial technical effects:

[0064] 1. This invention provides a zero-sample anomaly detection method based on CLIP model for soft separation of foreground and background. It uses a multilayer perceptron to perform soft separation of the foreground and background regions of image features, and then uses a multi-head self-attention mechanism to further enhance the features of the foreground and background regions. This effectively avoids the interference of noise generated in the background region on the anomaly localization results, thereby effectively improving the stability and reliability of the detection results.

[0065] 2. The present invention provides a foreground-background soft separation zero-sample anomaly detection method based on the CLIP model. By employing text feature representation with multi-branch features, it effectively distinguishes the semantic importance of image features, thereby improving the ability to perceive local anomalies during the detection process and enhancing the reliability of the detection results.

[0066] 3. The present invention provides a foreground-background soft separation zero-sample anomaly detection method based on the CLIP model. By performing feature suppression on the background region, it effectively reduces the impact of complex backgrounds and cross-domain scenes on the anomaly detection results, thereby improving the detection accuracy.

[0067] 4. The present invention provides a zero-sample anomaly detection method for foreground-background soft separation based on the CLIP model, which does not require a large number of samples to train the detection method and is suitable for practical application scenarios where it is difficult to obtain or label anomaly samples. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the computational process of an embodiment of a foreground-background soft separation zero-sample anomaly detection method based on the CLIP model of the present invention;

[0069] Figure 2 This is a schematic diagram of the calculation process of steps S2 and S3 in an embodiment of a foreground-background soft separation zero-sample anomaly detection method based on the CLIP model of the present invention.

[0070] Figure 3 This is an industrial image of a metal nut in an embodiment of a foreground-background soft separation zero-sample anomaly detection method based on the CLIP model of the present invention.

[0071] Figure 4 This is a diagram of the actual abnormal region of a metal nut in an embodiment of a foreground-background soft separation zero-sample anomaly detection method based on the CLIP model of the present invention.

[0072] Figure 5 This is a detection result diagram of the calibrated abnormal region in an embodiment of the foreground-background soft separation zero-sample anomaly detection method based on the CLIP model of the present invention. Detailed Implementation

[0073] To make the objectives, advantages, and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0074] like Figure 1 As shown in this embodiment, a foreground-background soft separation zero-shot anomaly detection method based on the CLIP model constructs a multi-branch text feature representation in the text modality and introduces a multi-dimensional foreground-background soft separation and background suppression mechanism in the visual modality, such as... Figure 3 As shown, this embodiment uses an industrial drawing containing a metal nut as the sample to be tested. Figure 4 As shown, there is a tiny scratch on the left edge of the metal nut, which is an abnormality. The detection method includes the following steps:

[0075] S1. Based on CLIP, a text encoder is used to construct multi-branch features of the sample to be detected, and an enhanced text embedding vector is generated based on the multi-branch features; the specific process is as follows:

[0076] S1.1 Construct the basic text description of the sample to be detected based on the semantic category name of the sample to be detected and the preset positive and negative state templates.

[0077] S1.2. Concatenate the predefined learnable context vector with the basic text description to generate the corresponding prompt text. The prompt text includes normal text prompts and abnormal text prompts, namely "a nut without defects" and "a nut with defects".

[0078] S1.3. Based on the CLIP text encoder, the prompt text is subjected to deep semantic encoding to obtain text features. Then, a three-path feature extraction strategy is used to extract multi-branch features of the text features. The first path is global average pooling, which extracts the overall semantic information of the text features to obtain global average features. The second path is sentence end token extraction, which extracts sentence end features and keeps them aligned with the original CLIP semantics to obtain sentence end token features. The third path is attention weighting, which calculates token weights through a learnable token selector network and aggregates key semantic information in a weighted manner to obtain attention weighted features. That is, the global average features, sentence end token features, and attention weighted features of the object "nut" are obtained, and contrast words such as "scratches" and "cracks" are locked to increase the proportion of these words in the multi-branch features.

[0079] S1.4. The global average feature, sentence-end token feature, and attention-weighted feature are weighted and fused to generate an enhanced text embedding vector. The global average feature is used to capture the overall semantic information of the sample to be detected, and the sentence-end token feature is used to enhance the stability of the text representation. In addition, the weights of the global average feature and sentence-end token feature are adjusted by a learnable vector. The attention-weighted feature is used to highlight the semantic components related to the anomaly. It is determined by a multilayer perceptron (MLP) and a normalization function (Softmax). The above three types of features are weighted and fused to generate an enhanced text embedding vector that is highly sensitive to the semantics of defects. That is, a vector representing normality and a vector representing anomaly. They not only focus on the nut itself, but also on the semantic details of the defect.

[0080] S2. Based on CLIP, an image encoder is used to construct n layers of image features for the sample to be detected, where n ≥ 1. In this embodiment, n is 3. Feature enhancement is performed on each layer of image features in the original feature dimension, semantic dimension, and spatial dimension to obtain the basic semantic feature vector, the semantic feature enhancement vector, and the spatial feature enhancement vector. The calculation process is as follows:

[0081] S2.1 Input the sample to be detected into the CLIP image encoder to obtain n layers of image features.

[0082] S2.2. Perform original feature mapping on each layer of image features to obtain the basic semantic feature vector. That is, to preserve the original feature mapping and prevent the loss of basic semantics.

[0083] S2.3. Perform semantic enhancement on each layer of image features to obtain the semantic feature enhancement vector. This approach leverages information richness for the foreground and stability metrics for the background to explicitly enhance image features of different properties through global self-attention. The semantic enhancement process includes: calculating a foreground soft mask for each layer of image features based on multi-dimensional anomaly scores, and determining a background soft mask based on the foreground soft mask; adaptively thresholding and binarizing the foreground and background soft masks to obtain a hard mask for each layer of image features; performing feature separation on each layer of image features based on the hard mask to obtain foreground and background features; calculating the information richness weight for the foreground features, where the information richness weight is the product of the difference and local diversity of the global classification tokens in the foreground features and image features; calculating the stability weight for the background features, where the stability weight is the product of the similarity and feature strength of the global classification tokens in the background features and image features; and weighting and aggregating the foreground and background features separately using a global self-attention mechanism based on the information richness weight and stability weight to obtain a semantic feature enhancement vector. .

[0084] S2.4. Perform spatial enhancement on the image features of each layer to obtain the spatial feature enhancement vector. The spatial enhancement process involves restoring the token sequence of image features to a 2D spatial arrangement, performing local patch aggregation, and enhancing the edges of the nut and local abnormal textures. At this point, the features at the scratch location are more prominent than the original features. The spatial enhancement process includes: calculating a foreground soft mask for each layer of image features based on multi-dimensional anomaly scores, determining a background soft mask based on the foreground soft mask, performing adaptive threshold binarization on the foreground and background soft masks to obtain a hard mask for each layer of image features, performing feature separation on each layer based on the hard mask to obtain foreground and background features, reshaping the foreground and background features from a sequence form to a 2D spatial form, and determining the local window masks for the foreground and background features using the unfold function; and calculating the information richness of the local window masks for the foreground features. The foreground features are weighted, normalized, and then weighted and aggregated with the foreground features to obtain the foreground aggregated features. The information richness weight is the product of the difference between the foreground features and the global classification tokens in the image features, and the local diversity weight is the product of the difference between the foreground features and the global classification tokens in the image features. For the local window mask of the background features, the stability weight is calculated, normalized, and then weighted and aggregated with the background features to obtain the background aggregated features. The stability weight is the product of the similarity between the background features and the global classification tokens in the image features, and the feature strength weight is the product of the similarity between the background features and the global classification tokens in the image features. The foreground aggregated features and the background aggregated features are added together and reconstructed back into a sequence form to obtain the spatial feature enhancement vector. The block effect generated by the spatial enhancement process achieves noise smoothing, signal enhancement and spatial clustering through local window aggregation, making the anomalous boundaries clearer and forming connected anomalous regions.

[0085] Among them, multidimensional anomaly scores The formula is as follows:

[0086]

[0087] In the formula, Representing image features Local significance, The weights representing local saliency, in this embodiment ; Representing image features Distance to the feature center The weights representing distance, in this embodiment ; Representing image features semantic consistency The weight representing semantic consistency, in this embodiment ; Representing image features The temporal changes, In this embodiment, the weights representing time-series changes are... .

[0088] in, The definition is as follows:

[0089]

[0090] In the formula, Indicates average pooling. In this embodiment, the number of windows is indicated. .

[0091] The definition is as follows:

[0092]

[0093] In the formula, Representing image features The feature center.

[0094] The definition is as follows:

[0095]

[0096] In the formula, Representing image features Global category tokens in the system.

[0097] The definition is as follows:

[0098]

[0099] In the formula, This represents the image features of the current layer at the current time. This represents the image features of the previous layer corresponding to the previous time step.

[0100] Foreground soft mask The calculation formula is as follows:

[0101]

[0102] In the formula, In this embodiment, the adaptive threshold is represented. .

[0103] S3. The basic semantic feature vectors are processed using a multilayer perceptron (MLP). Semantic feature enhancement vector and spatial feature enhancement vector A soft classification mask is generated to initially distinguish the high-confidence foreground and background regions of the n-layer image features. The high-confidence foreground region is the area where the scratch is located, and the background region is the solid-color background around the nut. Then, a multi-head self-attention mechanism is used to further enhance the features of the foreground and background regions, resulting in foreground and background feature enhancement vectors. These vectors include foreground enhancement feature vectors and background enhancement feature vectors. The calculation process is as follows:

[0104] S3.1, The basic semantic feature vectors are processed by a multilayer perceptron (MLP). Semantic feature enhancement vector and spatial feature enhancement vector A soft classification mask is generated to obtain the first foreground probability and the first background probability. The multilayer perceptron includes two linear layers and one activation layer. The first linear layer compresses the feature dimension to half. After activation by the ReLU function, the second linear layer maps the feature dimension to 1. Finally, the first foreground probability is output through the Sigmoid function, with a value range of [0, 1]. The first background probability is obtained by subtracting the first foreground probability from 1.

[0105] S3.2, Transfer the basic semantic feature vector Semantic feature enhancement vector Spatial feature enhancement vector The results are multiplied by the first foreground probability and the first background probability, respectively, and then further enhanced using a multi-head self-attention mechanism to obtain the foreground and background feature enhancement vector. Specifically, for the basic semantic feature vector... Semantic feature enhancement vector Spatial feature enhancement vector For each feature vector in the algorithm, its enhanced foreground feature vector and enhanced background feature vector are calculated. The enhanced foreground feature vector is then input into the foreground attention branch (Fg-Attention), and the enhanced background feature vector is input into the background attention branch (Bg-Attention). Both the foreground and background attention branches employ a multi-head self-attention mechanism with 8 attention heads, each with a dimension of 64. The foreground attention branch outputs the enhanced foreground feature vector through corresponding linear transformations and attention calculations; the background attention branch outputs the enhanced background feature vector through independent linear transformations and attention calculations. Finally, the enhanced foreground and enhanced background feature vectors are added together and normalized to obtain the foreground and background enhanced feature vectors. The foreground attention branch focuses on extracting salient features and anomalous signals, while the background attention branch focuses on consistent modeling of the global background.

[0106] Figure 2 This is a schematic diagram of the calculation process from step S2 to step S3. The high-confidence foreground region focuses on extracting salient features and anomalous signals, while the background region focuses on consistent modeling of the global background, and the basic semantic feature vectors are... Semantic feature enhancement vector Spatial feature enhancement vector By fusing these features, effective feature extraction can be achieved in complex contexts. Figure 2 This paper demonstrates the processing flow from image features to foreground and background enhancement features: First, the input image features are processed through multi-dimensional anomaly score calculation to obtain foreground and background soft masks. Then, adaptive threshold binarization is used to convert the foreground soft mask into a hard mask with values ​​of {0.5, 1.0}, where 1.0 represents a high-confidence foreground region and 0.5 represents a background region. Next, the image features are enhanced from three perspectives based on this hard mask. Finally, the enhanced features from the three perspectives are input into a multi-head self-attention mechanism, which includes a foreground attention branch and a background attention branch. The foreground branch focuses on extracting salient features and anomalous signals, while the background branch focuses on global background consistency modeling. The two branches process the enhanced foreground feature vector and enhanced background feature vector respectively through independent multi-head self-attention (8 heads × 64 dimensions), and finally fuse to output the foreground and background feature enhancement vector.

[0107] S4. Suppress the background enhancement feature vector to obtain a foreground enhancement feature vector free of background interference, and calculate the anomalous residual feature vector based on this. Then, generate a visual feature vector based on the anomalous residual feature vector. The calculation process is as follows:

[0108] S4.1 Obtain the mean and maximum values ​​of each element in the foreground and background feature enhancement vector, and determine the global background feature vector of the sample to be detected based on the mean and maximum values.

[0109] S4.2. Compare the foreground and background feature enhancement vectors with the global background feature vectors using cosine similarity to obtain the second background probability. Subtract the second background probability from 1 to obtain the second foreground probability.

[0110] S4.3 Subtract the global background feature vector from the foreground and background feature enhancement vector to obtain the background-free foreground enhancement feature vector. Multiply the background-free foreground enhancement feature vector with the second foreground probability to obtain the abnormal residual feature vector. That is, use the second foreground probability as a weight for gating suppression to maximize the signal-to-noise ratio and eliminate residual background interference.

[0111] S4.4 Add the abnormal residual feature vector and the foreground and background feature enhancement vector to obtain the visual feature vector.

[0112] S5. Based on the enhanced text embedding vector and visual feature vector, determine the abnormal feature vector of the sample to be detected, and output an anomaly detection report based on the abnormal feature vector, thereby completing the anomaly detection of the sample to be detected; the calculation process is as follows:

[0113] S5.1 Multiply the enhanced text embedding vector and the visual feature vector to obtain the preliminary abnormal feature vector.

[0114] S5.2. Add the initial anomaly feature vector and the foreground enhancement feature vector after removing background interference to obtain the final anomaly feature vector of the sample to be detected. Convert the final anomaly feature vector into a high-resolution pixel-level anomaly score map. After filtering and normalizing the anomaly score map, output an anomaly detection report to complete the anomaly detection of the sample to be detected. The anomaly detection report includes the sample to be detected with the anomaly region marked and the score value used to evaluate the anomaly detection accuracy, such as... Figure 5 As shown, at the location of the scratch, due to the high degree of semantic alignment, that is, it conforms to the anomaly description and has a large residual energy, a bright red spot will appear. The entire image is judged as an anomaly, and the score for evaluating the anomaly detection accuracy is 0.95.

[0115] Based on the above calculation process, this embodiment uses the MVTec industrial anomaly detection dataset, Visa industrial anomaly detection dataset, MPDD metal component defect dataset, BTAD industrial anomaly detection dataset, DAGM synthetic texture defect dataset, and DTD synthetic texture dataset from the industrial field as samples to be detected. It compares the anomaly detection methods AA-CLIP CVPR'2025, AF-CLIPMM'2025, and FAPrompt ICCV'2025 with the calculation method of this embodiment. The anomaly detection performance is evaluated from both image-level and pixel-level dimensions. The image-level evaluation metrics are AUROC and AP, and the pixel-level evaluation metrics are AUROC and PRO. The comparison results of each method under different evaluation metrics are shown in Table 1. In Table 1, FB-CLIP Ours is the name of the calculation method in this embodiment.

[0116] Table 1

[0117]

[0118] Based on the above calculation process, the BrainMRI dataset and Br35H brain medical imaging dataset from the medical field were used as the samples to be detected. The anomaly detection performance was evaluated from the image level, with AUROC and AP as the image level evaluation metrics. The ClinicDB colonoscopy image dataset and ColonDB colonoscopy image database from the medical field were used as the samples to be detected. The anomaly detection performance was evaluated from the pixel level, with AUROC and PRO as the pixel level evaluation metrics. The AA-CLIP CVPR'2025, AF-CLIP MM'2025, and FAPrompt ICCV'2025 anomaly detection methods were compared with the calculation method of this embodiment. The comparison results of each method under different evaluation metrics are shown in Table 2. In Table 2, FB-CLIPOurs is the name of the calculation method of this embodiment.

[0119] Table 2

[0120]

[0121] As can be seen from Tables 1 and 2, the anomaly detection method in this embodiment has high reliability and stability.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A foreground-background soft separation zero-sample anomaly detection method based on the CLIP model, characterized in that, Includes the following steps: S1. A CLIP-based text encoder constructs multi-branch features of the sample to be detected and generates an enhanced text embedding vector based on the multi-branch features. S2. Based on CLIP, an image encoder is constructed to build n layers of image features of the sample to be detected, where n≥1. For each layer of image features, feature enhancement is performed on the original feature dimension, semantic dimension and spatial dimension to obtain the basic semantic feature vector, semantic feature enhancement vector and spatial feature enhancement vector. S3. Soft classification masks are generated by using a multilayer perceptron to generate the basic semantic feature vector, semantic feature enhancement vector, and spatial feature enhancement vector, respectively, to initially distinguish the foreground and background regions of the n-layer image features. Then, a multi-head self-attention mechanism is used to further enhance the features of the foreground and background regions, resulting in n foreground and background feature enhancement vectors. These vectors are accumulated and merged to obtain the final foreground and background feature enhancement vector, which includes a foreground enhancement feature vector and a background enhancement feature vector. S4. Suppress the background enhancement feature vector in the foreground and background feature enhancement vector to obtain the foreground enhancement feature vector without background interference, and calculate the abnormal residual feature vector accordingly. Based on the abnormal residual feature vector, generate the visual feature vector. S5. Based on the enhanced text embedding vector and visual feature vector, determine the abnormal feature vector of the sample to be detected, and output an abnormal detection report based on the abnormal feature vector, thereby completing the abnormal detection of the sample to be detected.

2. The foreground-background soft separation zero-sample anomaly detection method based on the CLIP model according to claim 1, characterized in that, Step S3 is as follows: S3.

1. Using a multilayer perceptron, soft classification masks are generated for the basic semantic feature vector, semantic feature enhancement vector, and spatial feature enhancement vector of each layer of image features to obtain the first foreground probability and the first background probability. S3.2 Multiply the basic semantic feature vector, semantic feature enhancement vector, and spatial feature enhancement vector by the first foreground probability and the first background probability, respectively, to obtain the foreground and background regions of the n-layer image features. Then, further feature enhancement is performed on the foreground and background regions through a multi-head self-attention mechanism to obtain n foreground and background feature enhancement vectors. After accumulating and merging them, the final foreground and background feature enhancement vector is obtained.

3. The foreground-background soft separation zero-sample anomaly detection method based on the CLIP model according to claim 2, characterized in that, Step S4 is as follows: S4.1 Obtain the mean and maximum values ​​of each element in the foreground and background feature enhancement vector, and determine the global background feature vector of the sample to be detected based on the mean and maximum values. S4.

2. Compare the foreground and background feature enhancement vectors with the global background feature vectors using cosine similarity to obtain the second background probability. Based on the second background probability, determine the second foreground probability. S4.3 Subtract the global background feature vector from the foreground and background feature enhancement vector to obtain the background-free foreground enhancement feature vector. Multiply the background-free foreground enhancement feature vector with the second foreground probability to obtain the abnormal residual feature vector. S4.4 Add the abnormal residual feature vector and the foreground and background feature enhancement vector to obtain the visual feature vector.

4. The foreground-background soft separation zero-sample anomaly detection method based on the CLIP model according to claim 3, characterized in that: In step S2, the semantic dimension of each layer of image features is enhanced to obtain the semantic feature enhancement vector, as follows: a.

1. Calculate and generate the foreground soft mask for each layer of image features based on the multi-dimensional anomaly score, and determine the background soft mask based on the foreground soft mask; a.

2. Adaptively threshold binarize the foreground soft mask and the background soft mask to obtain the hard mask of each layer of image features; a.

3. Based on the obtained hard mask, perform feature separation on the image features of each layer to obtain foreground features and background features; a.

4. Calculate the information richness weight of the foreground features based on the local diversity of the foreground features, and calculate the stability weight of the background features based on the correlation of the image features. a.

5. Based on the information richness weight and stability weight, the foreground features and background features are weighted and aggregated separately through a global self-attention mechanism to obtain the semantic feature enhancement vector.

5. The foreground-background soft separation zero-sample anomaly detection method based on the CLIP model according to claim 4, characterized in that: In step S2, the spatial dimension feature enhancement is performed on each layer of image features to obtain the spatial feature enhancement vector, as follows: b.

1. Calculate and generate the foreground soft mask for each layer of image features based on the multi-dimensional anomaly score, and determine the background soft mask based on the foreground soft mask; b.

2. Adaptively threshold binarize the foreground soft mask and the background soft mask to obtain the hard mask of each layer of image features; b.

3. Based on the obtained hard mask, perform feature separation on the image features of each layer to obtain foreground features and background features; b.4 Reshape the foreground and background features from a sequential form to a 2D spatial form, and determine the local window mask for the foreground features and the local window mask for the background features; b.

5. Calculate the information richness weight of the local window mask for the foreground features and perform weighted aggregation with the foreground features to obtain the foreground aggregated features; b.

6. Calculate the stability weight of the local window mask for the background features and perform weighted aggregation with the background features to obtain the background aggregated features; b.

7. Add the foreground aggregation features and the background aggregation features, and reshape the result back into a sequence to obtain the spatial feature enhancement vector.

6. The foreground-background soft separation zero-sample anomaly detection method based on the CLIP model according to claim 5, characterized in that, Step S1 is as follows: S1.1 Construct the basic text description of the sample to be detected based on the semantic category name and the preset positive and negative state template; S1.

2. Concatenate the predefined learnable context vector with the basic text description to generate the corresponding prompt text; S1.

3. Based on CLIP's text encoder, the prompt text is subjected to deep semantic encoding to obtain text features. Then, a three-path feature extraction strategy is used to extract multi-branch features of the text features. The first path is global average pooling to extract the overall semantic information of the text features and obtain global average features. The second path is sentence end token extraction to obtain sentence end token features. The third path is attention weighting, which calculates token weights through a learnable token selector network and aggregates key semantic information in a weighted manner to obtain attention weighted features. S1.

4. The global average features, sentence end token features, and attention-weighted features are weighted and fused to generate an enhanced text embedding vector.

7. The foreground-background soft separation zero-sample anomaly detection method based on the CLIP model according to claim 6, characterized in that: The formulas for the multidimensional anomaly score in steps a.1 and b.1 are as follows: ； In the formula, Indicates multi-dimensional anomaly scores, Representing image features Local significance, Weights representing local significance; Representing image features Distance to the feature center Weights representing distance; Representing image features semantic consistency Weights representing semantic consistency; Representing image features The temporal changes, Weights representing temporal variations; The definition is as follows: ； In the formula, Indicates average pooling. Indicates the number of windows; The definition is as follows: ； In the formula, Representing image features The feature center; The definition is as follows: ； In the formula, Representing image features Global category tokens in the context of Tokens; The definition is as follows: ； In the formula, This represents the image features of the current layer at the current time. This represents the image features of the previous layer corresponding to the previous time step.

8. The foreground-background soft separation zero-sample anomaly detection method based on the CLIP model according to claim 7, characterized in that: The formulas for calculating and generating the foreground soft mask in steps a.1 and b.1 are as follows: ； In the formula, Indicates a foreground soft mask. This indicates an adaptive threshold.

9. The foreground-background soft separation zero-sample anomaly detection method based on the CLIP model according to claim 8, characterized in that, Step S5 is as follows: S5.1 Multiply the enhanced text embedding vector and the visual feature vector to obtain the preliminary anomaly feature vector; S5.

2. Add the preliminary abnormal feature vector and the foreground enhancement feature vector after removing background interference to obtain the final abnormal feature vector of the sample to be detected. Convert the final abnormal feature vector into a high-resolution pixel-level abnormal score map. After filtering and normalizing the abnormal score map, output an abnormal detection report to complete the abnormal detection of the sample to be detected. The abnormal detection report includes the sample to be detected with the abnormal region marked and the score value used to evaluate the accuracy of abnormal detection.

Images