Zero-sample industrial flaw detection method based on CLIP model

Abstract

The invention relates to a zero-sample industrial flaw detection method based on a CLIP model. According to the method, an optimized cue word template with a local feature guide prefix and an abnormal state condition suffix is constructed to guide a CLIP text encoder to focus an abnormal area, and a text description set with enhanced semantics is generated; based on the text description set and the industrial image, respectively extracting a visual block token feature and a fine-grained text embedding feature by using CLIP double encoders; performing residual mapping and weighted fusion processing on the text and the visual features; segmenting the fused visual features into a plurality of subspaces for parallel residual mapping and gating routing fusion, and decoupling abnormal features of different modes; and calculating an image-level abnormal probability and a pixel-level thermodynamic diagram based on the optimized cross-modal features. High-precision detection and positioning of tiny flaws of industrial products under the strict zero sample condition are achieved, the discrimination robustness of local flaws is remarkably improved, and meanwhile the advantage of light-weight deployment is maintained.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of industrial detection, and particularly relates to a zero-shot industrial defect detection method based on a CLIP model. BACKGROUND

[0002] In the field of industrial intelligent manufacturing, product quality control relies on efficient and accurate defect detection technology. Traditional detection methods are usually based on manually designed feature extraction rules (such as texture analysis or image enhancement), and need to customize algorithms for different products, which has the problems of poor universality and serious dependence on expert experience. With the development of deep learning, unsupervised anomaly detection methods based on reconstruction or feature embedding reduce the dependence on defect samples, but still need a large number of normal samples to train the model, and the robustness to small defects and complex backgrounds is insufficient. Especially in the new production line startup or rare defect scene, the cost of collecting sufficient samples is high, which restricts the flexible deployment of the detection system.

[0003] In recent years, the zero-shot capability of the visual-language pre-training model CLIP provides a new idea for industrial defect detection. This method realizes the semantic alignment of images and texts through contrastive learning, and can complete anomaly classification without target domain training data. However, the inherent architecture of CLIP has obvious limitations: first, its global feature alignment mechanism is difficult to capture subtle local defects (such as micron-level scratches or missing welds), resulting in insufficient fine-grained discrimination ability; second, the performance of the model is highly dependent on the design of the prompt, and the traditional fixed template cannot effectively guide the model to focus on the abnormal area; third, when working with segmentation models (such as SAM), the complexity of the multi-level architecture and the prompt bias will significantly reduce the reliability of the system. Existing improvement schemes such as test-time adapters can improve positioning accuracy, but they do not solve the core contradiction of cross-modal fine-grained alignment, and still have problems such as high false detection rate and fuzzy positioning in complex industrial scenes.

[0004] Therefore, there is an urgent need for an industrial defect detection method that operates under strict zero-shot conditions, while preserving the cross-domain generalization advantage of CLIP, breaking through the local perception bottleneck, and achieving high-precision anomaly localization without training data. SUMMARY

[0005] Therefore, it is necessary to provide a zero-shot industrial defect detection method based on a CLIP model in view of the above technical problems.

[0006] In a first aspect, the application provides a zero-shot industrial defect detection method based on a CLIP model, comprising:

[0007] S1, based on product category names, field labels and state labels, constructing an optimized prompt word template to generate a set of text descriptions with a local feature guided prefix and an abnormal state conditional suffix;

[0008] S2, generating visual block token features through a CLIP visual encoder and text embedding features through a CLIP text encoder based on the set of text descriptions and the industrial image to be detected; wherein the text embedding features include normal text embedding features and abnormal text embedding features; the local feature guide prefix is used to activate and guide the CLIP text encoder to focus on local abnormal features; and the abnormal state condition suffix is used to enhance the semantic separability of abnormal and normal features;

[0009] S3, performing residual mapping and weighted fusion on the text embedding features and the visual block token features to generate enhanced text features and enhanced visual features;

[0010] S4, segmenting the enhanced visual features into multiple subspaces for parallel residual mapping, fusing the outputs of each subspace through a gating routing mechanism, and generating fine-grained visual features;

[0011] S5, calculating image-level anomaly probability and pixel-level anomaly heat map based on the fine-grained visual features and the enhanced text features, and outputting the detection result.

[0012] In a second aspect, the present application also provides a zero-shot industrial defect detection system based on a CLIP model, which is used to implement the method of the first aspect. The system comprises:

[0013] A prompt word engineering module is configured to construct an optimized prompt word template based on product category names, field labels and state labels, and generate a set of text descriptions with a local feature guide prefix and an abnormal state condition suffix.

[0014] A multi-modal encoding unit is configured to generate visual block token features through a CLIP visual encoder and text embedding features through a CLIP text encoder based on the set of text descriptions and the industrial image to be detected; wherein the text embedding features include normal text embedding features and abnormal text embedding features; the local feature guide prefix is used to activate and guide the CLIP text encoder to focus on local abnormal features; and the abnormal state condition suffix is used to enhance the semantic separability of abnormal and normal features.

[0015] A feature enhancement module is configured to perform residual mapping and weighted fusion on the text embedding features and the visual block token features to generate enhanced text features and enhanced visual features.

[0016] A fine-grained visual processing module is configured to segment the enhanced visual features into multiple subspaces for parallel residual mapping, fuse the outputs of each subspace through a gating routing mechanism, and generate fine-grained visual features.

[0017] A detection result generation module is configured to calculate image-level anomaly probability and pixel-level anomaly heat map based on the fine-grained visual features and the enhanced text features, and output the detection result.

[0018] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the zero-shot industrial defect detection method based on the CLIP model in the first aspect when executing the computer program.

[0019] In a fourth aspect, the present application also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the zero-shot industrial defect detection method based on the CLIP model in the first aspect.

[0020] The zero-shot industrial defect detection method based on the CLIP model can guide the CLIP text encoder to focus on the abnormal area by constructing an optimized prompt word template with a local feature guided prefix and an abnormal state conditional suffix, and generate an enhanced semantic text description set; based on the text description set and the industrial image, the CLIP dual encoder is used to extract visual block token features and fine-grained text embedding features; residual mapping and weighted fusion processing are performed on the text and visual features to enhance the abnormal sensitivity while retaining the pre-training knowledge; then the fused visual features are segmented into multiple subspaces for parallel residual mapping and gated routing fusion to decouple the abnormal features of different modalities; finally, the image-level abnormal probability and the pixel-level heat map are calculated based on the optimized cross-modal features. Thus, high-precision detection and positioning of subtle defects of industrial products can be realized under the condition of strict zero-shot, the discrimination robustness of local defects such as scratches and pollution is significantly improved, and the lightweight deployment advantage is maintained. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flowchart of a zero-shot industrial defect detection method based on a CLIP model provided by the present application;

[0023] Figure 2 A flowchart of generating fine-grained visual features in an optional embodiment of the present application;

[0024] Figure 3 A structural diagram of a zero-shot industrial defect detection system based on a CLIP model provided by the present application. DETAILED DESCRIPTION

[0025] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0026] Reference Figure 1 It shows a flowchart of a zero-shot industrial defect detection method based on a CLIP model provided by the present application, which comprises the following steps:

[0027] S1, based on product category name, domain label and state label, constructing an optimized prompt word template to generate a text description set with a local feature guide prefix and an abnormal state condition suffix.

[0028] Specifically, the present application optimizes the prompt word template of the reference model AnoCLIP+ twice. On the basis of the original template "A[domain] photo of a[state][class].", the "Attention" prefix and "inthelocal." suffix are added to form a new template "Attention, this is a[domain] photo of a[state][class] (in thelocal)." Wherein, [domain] is a domain description (such as "industrial" / "medicine"), [state] contains a contrast state description ("normal" / "abnormal"), and [class] is a category description (such as "pcb1" / "candle"). This kind of optimization design activates the local attention ability of the CLIP text encoder through the "Attention" keyword. For abnormal categories, the abnormal state is embedded as a token by adding "in thelocal." at the end of the template, and the feature separability is enhanced by using the contrast learning prior of CLIP, which guides the CLIP model to pay more attention to the abnormal local features rather than the global image features when aligning vision and language, thereby improving the sensitivity of the model to fine-grained abnormalities.

[0029] S2, based on the text description set and the industrial image to be detected, generating visual block token features through a CLIP visual encoder and generating text embedding features through a CLIP text encoder; wherein the text embedding features include normal text embedding features and abnormal text embedding features; the local feature guide prefix is used to activate and guide the CLIP text encoder to focus on local abnormal features; the abnormal state condition suffix is used to enhance the semantic separability of abnormal and normal features.

[0030] Specifically, the industrial image to be detected is first processed by the visual encoder of CLIP (based on the VisionTransformer architecture), which divides the input image into multiple image blocks and extracts image local features at block level granularity to form a set of visual block token features. At the same time, the optimized text description set is processed by the text encoder of CLIP to generate normal state text embedding features and abnormal state text embedding features. Here, the local feature guide prefix "Attention" is used to activate and guide the CLIP text encoder to focus on local abnormal features, while the abnormal state condition suffix "in the local." is used to enhance the semantic separability of abnormal and normal features. These two designs can jointly improve the sensitivity of the model to the semantic expression of fine-grained abnormalities and the ability to accurately locate.

[0031] S3, residual mapping and weighted fusion are performed on the text embedding features and visual block token features to generate enhanced text features and enhanced visual features.

[0032] Specifically, the present application introduces a lightweight residual adapter, which is applied to the output end of the text encoder and the visual encoder. For text features, the residual adapter enhances the semantic expression ability of the text features by performing linear down-projection-nonlinear activation on the original text features, then performing residual fusion with the original features, and processing through layer normalization. For visual features, a similar residual adapter is applied to the features output by the visual encoder to refine the visual features of the specific query obtained from the visual encoder. This design not only retains the pre-training knowledge of CLIP, but also enables targeted adaptation to improve the performance and generalization ability of the model in the zero-shot anomaly detection task.

[0033] S4, the enhanced visual features are segmented into multiple subspaces for parallel residual mapping, and the outputs of each subspace are fused through a gating routing mechanism to generate fine-grained visual features.

[0034] Specifically, the present application innovatively introduces a multi-head adapter (MHA) and places it at the output connection part of the last Transformer layer of the visual encoder. The multi-head adapter is a lightweight extension of the residual adapter based on the idea of "parallel by sub-space", which equally divides each input feature (embedded token) into multiple sub-spaces (heads), applies a residual adapter to each sub-space independently, and performs a "dimension reduction-nonlinear-dimension increase" mapping with "bottleneck effect". In order to allow different tokens to dynamically select the most suitable head, the present application introduces a gate vector that can also be optimized, which weights and fuses the outputs of each sub-space through a gated routing mechanism, and finally concatenates the fused features of all sub-space outputs along the channel dimension to obtain fine-grained visual features. This design can effectively avoid the feature interference and expression bottleneck problems that the traditional single-channel adapter may encounter when processing polymorphic anomalies, enabling the model to simultaneously perceive multiple anomaly state features such as texture details (e.g., fine scratches), geometric shapes (e.g., notches, protrusions), and color anomalies (e.g., oil stains, pollution), significantly improving the model's comprehensive perception ability for complex and subtle defects.

[0035] S5, based on the fine-grained visual features and the enhanced text features, calculating image-level anomaly probability and pixel-level anomaly heat map, and outputting the detection result.

[0036] Specifically, for image-level anomaly detection, the present application calculates the global average of fine-grained visual features to obtain image-level global features, and then calculates image-level anomaly probability scores based on the features and enhanced text features. When the score is greater than a preset threshold, the image is determined to be abnormal, otherwise it is normal. For pixel-level anomaly positioning, the present application calculates local anomaly scores for each block token in the fine-grained visual features, and then upsamples these scores to the original image size to generate a pixel-level anomaly heat map, thereby achieving precise positioning of the abnormal area.

[0037] The above-mentioned zero-shot industrial defect detection method based on the CLIP model constructs an optimized prompt word template with a local feature guide prefix and an abnormal state condition suffix to guide the CLIP text encoder to focus on the abnormal area, and generates a set of enhanced semantic text descriptions; based on the set of text descriptions and industrial images, the CLIP dual encoder is used to extract visual block token features and fine-grained text embedding features; residual mapping and weighted fusion processing are performed on the text and visual features to enhance the abnormal sensitivity while retaining the pre-training knowledge; then the fused visual features are segmented into multiple subspaces for parallel residual mapping and gated routing fusion to decouple the abnormal features of different modalities; finally, the image-level abnormal probability and pixel-level heat map are calculated based on the optimized cross-modal features. Thus, high-precision detection and positioning of subtle defects of industrial products are realized under the condition of strict zero samples, the discrimination robustness of local defects such as scratches and pollution is significantly improved, and the lightweight deployment advantage is maintained.

[0038] In an alternative embodiment, S2 comprises the following steps:

[0039] S21, segmenting the industrial image to be detected into n image blocks by a ViT visual encoder to extract a set of block token features wherein p i represents the block-level granular image local feature of the i-th image block, and d represents the dimension number of the feature.

[0040] Specifically, a ViT-B-16-plus-240 visual encoder with consistent architecture as the benchmark model is used, and its visual feature extraction module directly inherits the CLIP model pre-trained by OpenAI based on the laion400m_e32 image-text multimodal dataset. In terms of visual encoder configuration, the input image size is uniformly adjusted to 314x240 pixels to adapt to the processing requirements of the VisionTransformer module, and no data enhancement strategy is used. The ViT visual encoder segments the input image into n image blocks, each image block corresponding to a block token, forming a set of block token features wherein, represents the block-level granular image local feature of the i-th image block, and d represents the dimension number of the feature. These block token features essentially encode rich local spatial semantic information, providing a basis for subsequent anomaly localization tasks.

[0041] S22, inputting the set of text descriptions into the CLIP text encoder to generate a set of normal state text embeddings and a set of abnormal state text embeddings

[0042] Specifically, prompt words obtained based on AnoVL by integrating 22 base templates, 7 sets of state contrast prompts, and 4 types of domain perception prompts are used, and the prefix "Attention" and the suffix "in the local." are further added. These optimized prompt word templates are input into the text encoder of CLIP to generate a normal state text embedding set and an abnormal state text embedding set where N is the number of prompt word templates.

[0043] S23, calculating a normal state average text embedding and an abnormal state average text embedding The normal state average text embedding is taken as a normal text embedding feature, and the abnormal state average text embedding is taken as an abnormal text embedding feature.

[0044] Specifically, all embedding vectors in the normal state text embedding set are averaged to obtain a normal state average text embedding Similarly, all embedding vectors in the abnormal state text embedding set are averaged to obtain an abnormal state average text embedding This averaging operation can reduce the noise influence that a single prompt word template may bring and improve the stability and representativeness of the text embedding feature.

[0045] Through the above steps S21 to S23, visual feature extraction of the industrial image to be detected and semantic embedding of the text description can be realized, providing basic data for subsequent residual mapping and weighted fusion. This feature extraction method fully utilizes the rich visual-linguistic alignment knowledge learned in the pre-training process of the CLIP model, and at the same time enhances the model's perception ability of abnormal features through optimized prompt word templates.

[0046] In an optional embodiment, S3 includes the following steps:

[0047] S31, performing a residual mapping operation on the text embedding feature Z t ={t + ,t -} to obtain an adaptive feature; the expression of the residual mapping operation is:

[0048]

[0049] where t + is the normal text embedding feature, t - is the abnormal text embedding feature, W i is a trainable linear weight, Act(·) is an activation function, Norm(·) is a normalization function, and z i ∈Zt .

[0050] Specifically, the activation function can be selected as GELU (Gaussian Error Linear Unit). The residual mapping operation adaptively adjusts the text embedding features through linear transformation and nonlinear activation, enhancing their expression ability. A residual adapter is introduced after the output features of the text encoder, which linearly projects the original text features, activates them nonlinearly, and then fuses them with the original features through residual fusion and normalization processing through layer normalization. This lightweight design can reduce computational overhead while improving text-visual alignment capabilities, helping to enhance the abnormal semantic expression ability of text features and make the model's understanding of abnormal descriptions more detailed and higher in differentiation.

[0051] S32, based on the adaptive features, enhanced text features are generated through weighted fusion, and the expression of weighted fusion is:

[0052]

[0053] wherein λ is a preset coefficient for controlling the proportion of original features and residual features, λ satisfies Z out = (1-λ) · z in + λ · Adpat(z in ); z in is the input feature, including the output features of the CLIP text encoder and the CLIP visual encoder; Adpat(·) is the residual adapter function.

[0054] Specifically, the preset coefficient λ can satisfy 0≤λ≤1, which is used to adjust the fusion degree of prior knowledge and specific knowledge to maintain the generalization ability of the original CLIP, improve its stability, and ensure correct information transmission. Through this weighted fusion method, the original text features Z and the residual features Adapt(Z) are combined in appropriate proportions to generate enhanced text features Z', which not only retains the basic semantic information of the original features, but also integrates the targeted enhancement provided by the residual adapter, thereby improving the expression ability of the text features.

[0055] S33, the same residual mapping and weighted fusion operations as steps S31 to S32 are performed on the visual block token features P to generate enhanced visual features P enhanced .

[0056] Specifically, after the visual encoder outputs the features, a lightweight residual adapter is inserted as a visual adapter to enhance the features, so as to refine the visual features of the specific query obtained from the visual encoder. The adapter improves the performance of zero-shot anomaly localization by jointly optimizing two discriminative tasks of pseudo labels of AnoVL and synthetic noise damage markers, and makes a proper trade-off between performance and cost. After the visual encoder extracts the visual features, the visual features are sent to the visual adapter (which internally uses a residual adapter), and then the enhanced visual features are output. Especially in the fine-grained anomaly detection scenario, the sensitivity and discriminability of the visual features to the abnormal regions can be improved.

[0057] Through the above steps S31 to S33, residual mapping and weighted fusion of the text embedding features and the visual block token features are realized to generate enhanced text features and enhanced visual features. This processing mode not only retains the rich semantic information learned in the pre-training process of the CLIP model, but also introduces specific enhancements for the anomaly detection task through the residual adapter, thereby improving the performance of the model in the zero-shot defect detection task.

[0058] Reference Figure 2 In an optional embodiment, S4 includes the following steps:

[0059] S41, the enhanced visual features P enhanced are uniformly divided into h subspaces in the channel dimension to obtain a set of subspace features {F 1 ,F 2 ,…,F h}; wherein, B is the batch size, L is the token number of the enhanced visual features P enhanced .

[0060] Specifically, this division method is based on an assumption: if different defect semantics are nearly independent of each other in the channel, there exists a linear projection P = diag(P (1) ,P (2) ,…,P (h) ), which can make the cross-semantic mutual information function I(F i ; F j ) ≈ 0, (i ≠ j) hold for all k ≠ j, so learning a residual mapping for each subspace can avoid different defect features occupying the same low-dimensional channel, thereby improving the expression specificity. In practice, the number of subspaces h can be set to 8, which achieves a good balance between expression ability and computational efficiency.

[0061] S42, for each subspace feature F kThe low-rank residual mapping in the form of Houlsby-Adapter is independently performed to obtain a mapping result F k′ The expression of the low-rank residual mapping is as follows:

[0062]

[0063] wherein σ(·) is an activation function, denotes a learnable weight matrix for feature dimension reduction, denotes a learnable weight matrix for feature dimension increase, r is a bottleneck scaling rate for controlling the degree of feature compression.

[0064] Specifically, the activation function σ(·) can be GELU. In practice, the scaling rate r can be set to 4, which can significantly reduce the model parameter quantity and the calculation complexity while maintaining sufficient expression capability. Through such a low-rank residual mapping, each subspace feature undergoes a processing procedure of "dimension reduction → nonlinear activation → dimension increase". Such a mapping with "bottleneck effect" can effectively extract key features in the subspace and filter out redundant information.

[0065] S43, calculating routing weights of each subspace through a learnable gating vector g; the calculation formula of the routing weights is as follows:

[0066] α k =softmax(g)M k ;

[0067] wherein, M k denotes a routing mask of the corresponding subspace, M k ∈{0,1} B×L×h .

[0068] Specifically, such a gating mechanism allows the model to dynamically determine which subspaces each token should pay more attention to, thereby achieving adaptive processing of different types of abnormal features. In practice, a sparse gating regularizer softmax(g) is sparsified, so that most tokens only activate a small number of heads, thereby reducing redundant calculations and forming a more distinct residual signal in the abnormal area.

[0069] S44, performing gated weighted fusion on the output of each subspace based on the routing weights to obtain fused features; the expression of the gated weighted fusion is as follows:

[0070]

[0071] wherein α is a preset coefficient for controlling the intensity of the residual signal, and ⊙ denotes an element-level multiplication operation.

[0072] Specifically, the intra-layer residual weight a can be set to 0.1, which can control the intensity of the residual signal and ensure that the model does not deviate too much from the original feature space while introducing new features. Through such a gated weighted fusion, the original features and the features processed by the adapter of each subspace are combined in appropriate proportions and adjusted according to the routing weight, thereby generating fused features.

[0073] S45, splicing the fused features output by all subspace along the channel dimension to obtain fine-grained visual features.

[0074] Specifically, such splicing operation recombines the features processed by each subspace to form complete fine-grained visual features, providing a basis for subsequent anomaly detection and positioning.

[0075] Through the above steps S41 to S45, the multi-head adapter processing of enhanced visual features is realized, and fine-grained visual features with stronger discriminative ability are generated. Such multi-head adapter design not only effectively avoids the feature interference and expression bottleneck that the traditional single-channel adapter may encounter when processing multi-state anomalies, but also significantly improves the comprehensive perception ability of the model to complex and subtle defects while achieving an optimized balance between parameter quantity and computational efficiency. Inspired by the idea of Houlsby's parameter-efficient adapter, the multi-head adapter only adds two layers of low-rank linear mapping in each subspace, with a total parameter quantity on the same order of magnitude as a single-head ResidualAdapter, but the expression ability is approximately linearly amplified; at the same time, the multi-head parallel shares the input, and the matrix multiplication can be completed with fewer calculation times, so the required computing power increases with the number of heads and tends to be flat. Such design not only preserves the global semantics of the original model, but also allows different heads to focus on different local anomaly patterns, providing more fine-grained and complementary feature representation for zero-shot defect detection, and is also beneficial to improve parameter and computational efficiency and reduce inference cost.

[0076] In an optional embodiment, the image-level anomaly probability is calculated, including the following steps:

[0077] S51, calculating the global average of the fine-grained visual features to obtain an image-level global feature.

[0078] S52, based on the image-level global feature and the enhanced text feature, calculating an image-level anomaly probability score; the calculation formula of the image-level anomaly probability score is:

[0079]

[0080] Where v is the image-level global feature.

[0081] Specifically, the calculation manner can be calculating the cosine similarity between the image-level global feature and the abnormal text embedding feature, divided by the sum of the cosine similarity between the image-level global feature and the normal and abnormal text embedding features. If the image contains an anomaly, its feature should be closer to the abnormal text embedding feature, thereby obtaining a higher anomaly probability score; conversely, if the image is normal, its feature should be closer to the normal text embedding feature, thereby obtaining a lower anomaly probability score.

[0082] S53, when the image-level anomaly probability score is greater than a preset threshold, determining that the image is abnormal, otherwise normal.

[0083] Specifically, a suitable threshold can be set according to specific application scenarios and requirements. The selection of the threshold can be a trade-off between detection accuracy and recall rate. A higher threshold can reduce false positives (misjudging normal samples as abnormal), but may increase false negatives (misjudging abnormal samples as normal); a lower threshold is the opposite. In actual application, the threshold can be adjusted on the validation set to find the best balance point.

[0084] Through the above steps S51 to S53, image-level anomaly detection based on fine-grained visual features and enhanced text features is realized. This detection method fully utilizes the cross-modal matching capability of the CLIP model, and realizes effective identification of abnormal images by comparing the similarity difference between image features and normal / abnormal text features.

[0085] In an optional embodiment, calculating the pixel-level anomaly heat map comprises the following steps:

[0086] S54, for each block token in the fine-grained visual feature, calculating a local anomaly score; the calculation formula of the local anomaly score is

[0087] Specifically, this calculation method is similar to the calculation of the image-level anomaly probability score, but is calculated for each block token separately, thereby obtaining a block-level anomaly score. If a certain block contains an anomaly, its feature should be closer to the abnormal text embedding feature, thereby obtaining a higher local anomaly score; conversely, if the block is normal, its feature should be closer to the normal text embedding feature, thereby obtaining a lower local anomaly score.

[0088] S55, upsampling the set of local anomaly scores to the original image size to generate a pixel-level anomaly heat map.

[0089] Specifically, first, the local anomaly score at the block level is reorganized into a two-dimensional heat map, and then the size of the heat map is adjusted to the size of the original image through upsampling methods such as bilinear interpolation to obtain the pixel-level anomaly heat map. The upsampling operation can map the block-level anomaly detection result to the pixel level, thereby realizing accurate positioning of the abnormal area.

[0090] Through the above steps S54 to S55, pixel-level anomaly positioning based on fine-grained visual features and enhanced text features is realized. This positioning method fully utilizes the block-level feature representation capability of the Vision Transformer and the cross-modal matching capability of the CLIP model, and realizes effective positioning of the abnormal area by comparing the similarity difference of the features of each block with the normal / abnormal text features.

[0091] The above-described zero-shot industrial defect detection method based on the CLIP model generates an enhanced semantic text description set by constructing an optimized prompt word template with a local feature guided prefix and an abnormal state conditional suffix to guide the CLIP text encoder to focus on the abnormal area; based on the text description set and the industrial image, the CLIP dual encoder is used to extract visual block token features and fine-grained text embedding features; residual mapping and weighted fusion processing are performed on the text and visual features to enhance the abnormal sensitivity while retaining the pre-training knowledge; then the fused visual features are segmented into multiple subspaces for parallel residual mapping and gated routing fusion to decouple the abnormal features of different modalities; finally, the image-level anomaly probability and the pixel-level heat map are calculated based on the optimized cross-modal features. Thus, high-precision detection and positioning of subtle defects of industrial products are realized under the condition of strict zero-shot, the discrimination robustness of local defects such as scratches and contamination is significantly improved, and the lightweight deployment advantage is maintained.

[0092] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0093] Based on the same inventive concept, the embodiments of the present application also provide a system for implementing the above-mentioned CLIP model-based zero-shot industrial defect detection method. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more embodiments of the CLIP model-based zero-shot industrial defect detection system provided below can be referred to the limitations of the CLIP model-based zero-shot industrial defect detection method described above, which will not be repeated here.

[0094] In one exemplary embodiment, as shown in Figure 3 A CLIP model-based zero-shot industrial defect detection system 30 is provided for implementing the methods in the above method embodiments, which includes:

[0095] A prompt word engineering module 31 is configured to construct an optimized prompt word template based on the product category name, the domain label and the state label, and generate a set of text descriptions with a local feature guided prefix and an abnormal state condition suffix.

[0096] A multi-modal encoding unit 32 is configured to generate visual block token features through a CLIP visual encoder and text embedding features through a CLIP text encoder based on the set of text descriptions and the industrial image to be detected; wherein the text embedding features include normal text embedding features and abnormal text embedding features; the local feature guided prefix is used to activate and guide the CLIP text encoder to focus on local abnormal features; and the abnormal state condition suffix is used to enhance the semantic separability of abnormal and normal features.

[0097] A feature enhancement module 33 is configured to perform residual mapping and weighted fusion on the text embedding features and the visual block token features to generate enhanced text features and enhanced visual features.

[0098] A fine-grained visual processing module 34 is configured to segment the enhanced visual features into multiple subspaces for parallel residual mapping, fuse the outputs of the subspaces through a gated routing mechanism, and generate fine-grained visual features.

[0099] A detection result generation module 35 is configured to calculate image-level abnormal probability and pixel-level abnormal heat maps based on the fine-grained visual features and the enhanced text features, and output the detection results.

[0100] The embodiments of the present application also provide a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0101] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned method embodiments.

[0102] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts are described in the part of the method embodiments. The above-described device embodiments are merely illustrative, wherein the components described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present disclosure according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0103] The above-described embodiments only express several implementation manners of the embodiments of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the embodiments of the present application. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the embodiments of the present application, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A zero-shot industrial defect detection method based on a CLIP model, characterized in that, The method comprises: S1, constructing an optimized prompt word template based on a product category name, a field label and a state label to generate a text description set with a local feature guide prefix and an abnormal state condition suffix; S2, generating visual block token features through a CLIP visual encoder and text embedding features through a CLIP text encoder based on the text description set and an industrial image to be detected; wherein the text embedding features include normal text embedding features and abnormal text embedding features; the local feature guide prefix is used to activate and guide the CLIP text encoder to focus on local abnormal features; and the abnormal state condition suffix is used to enhance the semantic separability of abnormal and normal features; S3, performing residual mapping and weighted fusion on the text embedding features and the visual block token features to generate enhanced text features and enhanced visual features; S4, segmenting the enhanced visual features into multiple subspaces for parallel residual mapping, fusing the outputs of the subspaces through a gating routing mechanism to generate fine-grained visual features; S5, calculating an image-level abnormal probability and a pixel-level abnormal heat map based on the fine-grained visual features and the enhanced text features to output a detection result.

2. The method of claim 1, wherein, The S2 comprises: S21, segmenting the industrial image to be detected into n image blocks through a ViT visual encoder, extracting a block token feature set wherein p i denotes the block-level granularity image local feature of the i-th image block, and d denotes the dimension number of the feature. S22, input the text description set into a CLIP text encoder to respectively generate a normal state text embedding set and an abnormal state text embedding set and an abnormal state text embedding set S23, compute normal state average text embedding and abnormal state average text embedding use the normal state average text embedding as the normal text embedding feature and the abnormal state average text embedding as the abnormal text embedding feature.

3. The method of claim 2, wherein, The S3 comprises: S31, obtaining a text embedding feature Z t = {t + ,t -}, performing a residual mapping operation to obtain an adaptive feature; an expression of the residual mapping operation is: where t + is the normal text embedding feature, t - is the abnormal text embedding feature, W i is a trainable linear weight, Act(·) is an activation function, Norm(·) is a normalization function, z i ∈ Z t ; S32, generating enhanced text features through weighted fusion based on the adaptive features; the expression of the weighted fusion is: wherein, λ is a preset coefficient for controlling the proportion of the original feature and the residual feature, λ satisfies Z out = (1-λ)·z in +λ·Adpat(z in ); z in is an input feature, including the output features of the CLIP text encoder and the CLIP visual encoder; and Adpat(·) is a residual adapter function. S33, performing the same residual mapping and weighted fusion operation as steps S31 to S32 on the visual block token feature P, to generate the enhanced visual feature P enhanced .

4. The method of claim 3, wherein, The S4 comprises: S41, enhance the visual feature P enhanced uniformly segmented into h subspaces in the channel dimension, to obtain a subspace feature set {F 1 , 2 , …, F h}; wherein, B is the batch size, L is the token number of the enhanced visual feature P enhanced . S42, for each subspace feature F in the subspace feature set k Independently perform the low-rank residual mapping in the form of Houlsby-Adapter to obtain the mapping result F k '; The expression of the low-rank residual mapping is: where σ(·) is an activation function, denotes a learnable weight matrix for feature dimension reduction, denotes a learnable weight matrix for feature dimension increase, r is a bottleneck scaling rate that controls the degree of feature compression. S43, calculating routing weights of each of the subspaces through a learnable gating vector g; the calculation formula of the routing weights is: a k = softmax(g)M k ; wherein, M k denotes a routing mask for the corresponding subspace, M k ∈{0,1} B×L×h ; S44, performing gated weighted fusion on the outputs of each of the subspaces based on the routing weights to obtain fused features; the expression of the gated weighted fusion is: wherein α is a preset coefficient for controlling the intensity of a residual signal, and represents an element-level multiplication operation; S45, concatenating the fused features of all the subspaces along the channel dimension to obtain the fine-grained visual features.

5. The method according to any one of claims 1 to 4, characterized in that, The calculation of the image-level abnormal probability comprises: S51, calculating a global average value of the fine-grained visual features to obtain an image-level global feature; S52, calculating an image-level abnormal probability score based on the image-level global feature and the enhanced text features; the calculation formula of the image-level abnormal probability score is: wherein v is the image-level global feature; S53, determining that an image is abnormal when the image-level abnormal probability score is greater than a preset threshold, and otherwise, determining that the image is normal.

6. The method of claim 5, wherein, The calculation of the pixel-level abnormal heat map comprises: S54, calculating a local abnormality score for each block token in the fine-grained visual features; the calculation formula of the local abnormality score is: S55, upsampling a set of the local abnormality scores to the original image size to generate the pixel-level abnormal heat map.

7. A CLIP model based zero-shot industrial defect detection system for implementing the method of any one of claims 1 to 6, characterized in that, The system comprises: A prompt word engineering module configured to construct an optimized prompt word template based on a product category name, a field label and a state label to generate a text description set with a local feature guide prefix and an abnormal state condition suffix; A multi-modal encoding unit is configured to generate visual patch token features by a CLIP visual encoder based on the set of text descriptions and an industrial image to be detected, and to generate text embedding features by a CLIP text encoder; wherein the text embedding features include normal text embedding features and abnormal text embedding features; the local feature guided prefix is configured to activate and guide the CLIP text encoder to focus on local abnormal features; and the abnormal state condition suffix is configured to enhance the semantic separability of abnormal and normal features; A feature enhancement module is configured to perform residual mapping and weighted fusion on the text embedding features and the visual patch token features to generate enhanced text features and enhanced visual features; A fine-grained visual processing module is configured to segment the enhanced visual features into multiple subspaces for parallel residual mapping, fuse the outputs of the subspaces by a gating routing mechanism, and generate fine-grained visual features. A detection result generation module is configured to calculate image-level abnormal probability and pixel-level abnormal heat maps based on the fine-grained visual features and the enhanced text features, and output a detection result.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images