A fine-tuning CLIP model-based zero-shot fabric defect detection method

Abstract

The application provides a zero-shot fabric defect detection method based on fine-tuning of a CLIP model, acquires a public data set and a private fabric data set and respectively performs pretreatment; a CoBi-CLIP anomaly detection network model is constructed, the CoBi-CLIP anomaly detection network model comprises parallel visual encoders and text encoders, and further comprises an anomaly classification module and an anomaly segmentation graph generation module between the visual encoders and the text encoders, a lightweight convolution adapter and a bidirectional alignment unit are respectively introduced in the visual encoders, the lightweight convolution adapter is used for extracting local context features of different scales in parallel by using a multi-branch bottleneck architecture, the bidirectional alignment unit is used for integrating macro semantic information and micro structure information by using a context injection and defect aggregation mechanism, the public data set is used for fine-tuning, and the private fabric data set is used for defect detection. The application improves the capture precision of the model on high-frequency structured defects and improves the positioning ability of tiny defects in a complex background.

Description

Technical Field

[0001] This invention relates to the technical field of anomaly detection, and more particularly to a method for detecting defects in fabrics using zero samples. Background Technology

[0002] Fabric defect detection is a crucial step in ensuring product quality in the textile industry. Traditional detection methods rely primarily on manual visual inspection, which is not only inefficient but also highly susceptible to factors such as fatigue and subjective experience, leading to high rates of missed and false detections. With the development of computer vision technology, deep learning-based automatic detection methods have gradually become mainstream. Supervised learning algorithms, represented by the YOLO series, can achieve high detection accuracy by training on a large number of labeled samples. However, in actual production scenarios, fabric defects are extremely rare and diverse. Existing technical solutions typically follow fully supervised or unsupervised paradigms, heavily relying on normal samples from the target domain (i.e., a specific production line) for model training or distribution modeling. This dependence causes the model to suffer a significant drop in detection accuracy when facing "cold start" production lines or new fabric types due to a lack of sufficient training data, limiting the algorithm's adaptability and generalization ability.

[0003] In recent years, visual-language pre-trained models, represented by CLIP (Contrastive Language-Image Pretraining), have demonstrated powerful zero-shot generalization capabilities through large-scale image-text alignment training. However, directly applying CLIP to fabric defect detection still has significant limitations: Granularity gap between global and local: CLIP pre-training focuses on aligning global class features with text descriptions, resulting in a lack of fine-grained semantic supervision for local patch features, making it difficult to identify subtle local defects. Lack of local perception: Visual encoders based on the standard ViT architecture mainly rely on global self-attention mechanisms, which are insufficient in capturing high-frequency structured defects (such as minor scratches or broken warp threads), easily causing defect signals to be smoothed or submerged in deep networks. Passive feature interaction: Existing improved methods mostly use simple linear adaptation, lacking dynamic explicit interaction between global semantics and local details, resulting in poor localization accuracy. Existing methods still have room for improvement in robustness, detection accuracy, and generalization ability. Summary of the Invention

[0004] To address the issue that existing CLIP-based anomaly detection methods overly rely on coarse-grained global feature alignment, neglecting the bias loss of the visual encoder when capturing local high-frequency defects, and that their implicit self-attention mechanism is prone to smoothing or masking weak anomalous signals by background noise, resulting in inaccurate localization and poor robustness when processing images with complex textures and small defects, this invention proposes a zero-shot fabric defect detection method, CoBi-CLIP (CLIP with Convolutional Adapters and Bidirectional Alignment), based on a fine-tuned CLIP model. Building upon the CLIP model, this method incorporates a lightweight convolutional adapter module and utilizes a multi-branch bottleneck architecture to extract local contextual features at different scales in parallel, injecting local spatial awareness into the visual encoder. This improves the model's accuracy in capturing high-frequency structured defects and solves the problem of perception loss when modeling small local defects in fabrics. Furthermore, a bidirectional alignment mechanism enhances multi-level feature interaction, integrating macroscopic semantic and microscopic structural information through explicit context injection and defect aggregation mechanisms. This addresses the issues of insufficient fine-grained semantic alignment and passive global-local feature interaction, improving the ability to locate small defects in complex backgrounds. Finally, by combining the focus loss and L1 loss into a comprehensive loss function to measure the deviation between the predicted results and the true labels, and with the help of a parameter-efficient bias fine-tuning strategy, the training efficiency and robustness of the model are significantly improved.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] A zero-sample fabric defect detection method based on a fine-tuned CLIP model includes the following steps:

[0007] S1: Obtain the public dataset and the private fabric dataset and preprocess them respectively;

[0008] S2: Construct a CoBi-CLIP anomaly detection network model, which includes a parallel visual encoder and a text encoder. Between the visual encoder and the text encoder, there is also an anomaly classification module and an anomaly segmentation map generation module. A lightweight convolutional adapter and a bidirectional alignment unit are introduced into the visual encoder. The lightweight convolutional adapter is used to extract local contextual features of different scales in parallel using a multi-branch bottleneck architecture. The bidirectional alignment unit is used to integrate macro-semantic and micro-structural information through context injection and defect aggregation mechanisms.

[0009] S3: Fine-tune the CoBi-CLIP anomaly detection network model using a public dataset to obtain a trained CoBi-CLIP anomaly detection network model.

[0010] S4: Use the trained CoBi-CLIP anomaly detection network model to perform anomaly detection on the private fabric dataset, and obtain anomaly classification results and anomaly segmentation maps.

[0011] Furthermore, the visual encoder adopts the ViT architecture, which employs stacked visual transformer coding layers, the stacked visual transformer coding layers including a 1st visual transformer coding layer to a 24th visual transformer coding layer; a lightweight convolutional adapter is introduced in the 1st to 7th visual transformer coding layers of the visual encoder.

[0012] The text encoder adopts a transformer architecture, which uses stacked text transformer encoding layers, which sequentially include a first text transformer encoding layer to a twelfth text transformer encoding layer.

[0013] Furthermore, a first bidirectional alignment unit is introduced between the 6th and 24th visual transformer coding layers of the visual encoder, a second bidirectional alignment unit is introduced between the 12th and 24th visual transformer coding layers, a third bidirectional alignment unit is introduced between the 18th and 24th visual transformer coding layers, and a fourth bidirectional alignment unit is introduced after the 24th visual transformer coding layer. The first, second, third, and fourth bidirectional alignment units are connected sequentially, and the parameters are shared among the four bidirectional alignment units.

[0014] Furthermore, each visual transformer coding layer in the first to seventh visual transformer coding layers sequentially includes a first layer normalization, a multi-head self-attention mechanism, a lightweight convolutional adapter, a second layer normalization, and a feedforward network. The input of the first layer normalization is connected to the output of the multi-head self-attention mechanism via a first residual connection, and the input of the second layer normalization is connected to the output of the feedforward network via a second residual connection.

[0015] Furthermore, the lightweight convolutional adapter utilizes a multi-branch bottleneck architecture to extract local contextual features at different scales in parallel, specifically including:

[0016] The patch feature output from the first residual join. Reconstructed into a spatial feature map And through a dimensionality reduction linear layer, the spatial feature map Dimensionality reduction is performed to obtain dimensionality-reduced feature maps; these feature maps are then fed in parallel into two depthwise separable convolutional branches with different kernels; the outputs of the two depthwise separable convolutions are then up-dimensionalized through linear up-dimensionality layers and concatenated, followed by... The fused feature map is obtained after convolutional fusion. Utilizing learnable layer scaling parameters Adjusting the fusion feature map The contribution of the spatial feature map Perform residual connections to obtain feature maps. and feature map Remodeling into patch features .

[0017] Furthermore, within each bidirectional alignment unit, macroscopic semantic and microscopic structural information are integrated through context injection and defect aggregation mechanisms, specifically including:

[0018] Context injection stage: Using the patch features input from the bidirectional alignment unit as the query and the global category features output from the last visual transformer encoding layer as the key and value, a cross-attention operation is performed to obtain the enhanced patch features;

[0019] Defect aggregation stage: Using the global category feature output by the last visual transformer encoding layer as the query and the enhanced patch feature as the key and value, perform cross attention operation to obtain the enhanced global category feature of the current bidirectional alignment unit.

[0020] Furthermore, the processing steps in the anomaly classification module include:

[0021] The final global category features enhanced by the fourth bidirectional alignment unit are mapped to a shared embedding space of the same dimension as the normal prompt embedding and defect / abnormal prompt embedding output by the text encoder through a pre-trained linear layer, resulting in a global feature embedding rich in multi-stage defect information. Subsequently, the global feature embedding rich in multi-stage defect information is aligned with the normal prompt embedding and defect / abnormal prompt embedding output by the text encoder through cosine similarity calculation to obtain a cosine similarity matrix. The probability score of the image belonging to an abnormal state is obtained by calculating the cosine similarity matrix through the Softmax normalization function.

[0022] Furthermore, the processing steps in the anomaly segmentation map generation module include:

[0023] Four enhanced patch features are extracted from the outputs of four bidirectional alignment units. These four enhanced patch features are then mapped through four independent, learnable linear layers to a subspace of the same dimension as the normal prompt embedding and defect / anomaly prompt embedding output by the text encoder, resulting in four levels of local feature embeddings. The cosine similarity between the patch features and the defect / anomaly prompt embedding at each spatial location corresponding to each level of local feature embedding is calculated, generating the original score distributions at four feature scales. The original scores at the four feature scales are then pixel-level averaged and aggregated. Finally, bilinear interpolation upsampling is performed on the averaged and aggregated feature map to restore the resolution of the averaged and aggregated feature map to the same size as the original input image, resulting in the final pixel-level predicted anomaly segmentation map.

[0024] Furthermore, during training, the pre-trained parameters of the visual encoder and text encoder are frozen, and only the parameters of the lightweight convolutional adapter, the bias term parameters of the second linear layer in the feedforward network of the text encoder, and the four learnable linear layer parameters of the anomaly segmentation map generation module are learned; the training loss adopts focus loss and L1 loss.

[0025] Furthermore, the normal and abnormal text features output by the text encoder are integrated using a cueing strategy: A photo of the [state][class] is used as a template, where [state] consists of various words describing normal and abnormal defect states, and [class] is the category name of the defect. The final text feature representation is obtained by averaging the text encoder outputs corresponding to all templates.

[0026] The beneficial effects of this invention are as follows:

[0027] This invention proposes a zero-shot visual anomaly detection method based on convolutional adapters and bidirectional alignment-enhanced CLIP. It introduces a lightweight convolutional adapter into the shallow network of the visual encoder to inject local spatial awareness, enhancing the model's sensitivity to high-frequency structured defects. A bidirectional alignment mechanism is proposed in the feature interaction logic, achieving context injection and defect aggregation through explicit global-local interaction units, improving the accurate localization of defects in complex backgrounds. A bias fine-tuning strategy is proposed to optimize text feature representation, improving the model's parameter efficiency and multi-domain generalization ability. Compared with existing technologies, the improved algorithm CoBi-CLIP exhibits superior detection accuracy in zero-shot scenarios. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a network structure diagram of the CoBi-CLIP fabric defect detection network model of the present invention, including the specific structure of the lightweight convolution adapter.

[0030] Figure 2 These are the fabric defect detection results of the present invention. From left to right, they are the original image, the original image with a truth mask, and the original image with a predicted heatmap. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] A zero-sample fabric defect detection method based on a fine-tuned CLIP model, such as Figure 1 As shown, the steps are as follows:

[0033] S1: Obtain the public dataset and the private fabric dataset and preprocess them respectively.

[0034] In this embodiment, the publicly available dataset uses the DAGM2007 dataset as fine-tuning training data. The DAGM2007 dataset has ten texture categories. The private dataset was collected from an actual production line. The private dataset contains a total of 9145 high-resolution images.

[0035] In this embodiment of the application, the method for preprocessing the dataset is as follows:

[0036] For the public dataset: Mosaic data augmentation was used to crop and stitch four different fabric images at random scales to simulate complex industrial background noise and enhance the model's robustness to multi-scale defects. All images were uniformly scaled to [size missing]. Pixel.

[0037] For the private fabric dataset: the LabelImg tool was used to accurately label the private dataset. For the final detection performance evaluation, and to adapt the model input, all images were uniformly scaled to [size missing]. Pixel.

[0038] S2: Construct a CoBi-CLIP anomaly detection network model, which includes a parallel visual encoder and a text encoder. Between the visual encoder and the text encoder, there is also an anomaly classification module and an anomaly segmentation map generation module. A lightweight convolutional adapter and a bidirectional alignment unit are introduced into the visual encoder. The lightweight convolutional adapter is used to extract local contextual features of different scales in parallel using a multi-branch bottleneck architecture. The bidirectional alignment unit is used to integrate macro-semantic and micro-structural information through context injection and defect aggregation mechanisms.

[0039] In this embodiment, the visual encoder adopts the ViT-L / 14 architecture, where 14 represents the image block size, meaning the input image is linearly projected into a series of blocks. Non-overlapping pixel patches. The visual encoder core consists of 24 repeatedly stacked visual transformer encoding layers, with global class features as input to the transformer layers. and patch features consisting of a series of patch features .

[0040] Each layer's internal structure includes, in sequence: first layer normalization, multi-head self-attention mechanism, second layer normalization, and feedforward network. The input of the first layer normalization is connected to the output of the multi-head self-attention mechanism via a first residual connection, and the input of the second layer normalization is connected to the output of the feedforward network via a second residual connection.

[0041] In this embodiment, to overcome the loss of detail signals caused by the standard ViT global self-attention mechanism, i.e., the tendency to mask high-frequency subtle defect signals, a lightweight convolutional adapter is embedded between the multi-head self-attention mechanism and the second layer normalization in the first 7 visual transformer coding layers of the visual encoder. Each visual transformer coding layer from the 1st to the 7th visual transformer coding layers sequentially includes a first layer normalization, a multi-head self-attention mechanism, a lightweight convolutional adapter, a second layer normalization, and a feedforward network. The input of the first layer normalization is connected to the output of the multi-head self-attention mechanism via a first residual connection, and the input of the second layer normalization is connected to the output of the feedforward network via a second residual connection.

[0042] like Figure 1 As shown, the feature extraction logic of the lightweight adapter is as follows:

[0043] Dimensional transformation: Transforming the patch features output from the first residual join. Reconstructed into a spatial feature map Furthermore, the feature dimension is compressed from 1024 dimensions to 128 dimensions through a dimensionality reduction linear layer to obtain a dimensionality-reduced feature map.

[0044] Multi-branch local capture: The dimensionality-reduced feature maps are fed in parallel into the convolution kernel with a kernel size of [missing value]. and The depth can separate the convolutional branches.

[0045] Fusion Output: The two branch outputs are each upscaled to 1024 dimensions using a linear upscaling layer and then concatenated. The fused feature map is obtained after convolutional fusion. Utilizing learnable layer scaling parameters Adjusting the fusion feature map The contribution of the spatial feature map Perform residual connections to obtain feature maps. and feature map Remodeling into patch features .

[0046] In this embodiment of the application, in order to overcome the technical problems of insufficient fine-grained semantic alignment and passive global-local feature interaction, the present invention introduces four bidirectional alignment units with shared parameters after layers 6, 12, 18 and 24 of the visual encoder.

[0047] Specifically, a first bidirectional alignment unit is introduced between the 6th and 24th visual transformer coding layers of the visual encoder, a second bidirectional alignment unit is introduced between the 12th and 24th visual transformer coding layers, a third bidirectional alignment unit is introduced between the 18th and 24th visual transformer coding layers, and a fourth bidirectional alignment unit is introduced after the 24th visual transformer coding layer. The first, second, third, and fourth bidirectional alignment units are connected sequentially.

[0048] The interaction process in each bidirectional alignment unit includes:

[0049] Context injection stage: The patch features (P1, P2, P3, P4 in the figure) input by the bidirectional alignment unit are used as queries, and the global category features output by the 24th layer visual transformer are used as queries. Using these as keys and values, a cross-attention operation is performed to inject high-level semantics into the local representation, resulting in the enhanced patch features, as shown in the formula:

[0050]

[0051] in, The enhanced patch features obtained for the current bidirectional alignment unit t. The patch feature is the input to the current bidirectional alignment unit t. The global category features are the input to the current bidirectional alignment unit t. For cross-attention operations, , , These represent the query matrix, key vector, and value vector during the context injection phase, respectively, where T denotes transpose. express Activation function Indicates the feature dimension.

[0052] Defect aggregation stage: Using the global category features output by the 24th layer visual transformer as the query and the enhanced patch features as the key and value, a cross-attention operation is performed to enable the global features to actively accumulate multi-level defect information, resulting in the enhanced global category features, as shown in the formula:

[0053]

[0054] in, This is the enhanced global category feature output from the previous bidirectional alignment unit. , , These are the query vector, key matrix, and value matrix for the defect aggregation stage, respectively.

[0055] Through four sequential iterations, global class features are gradually accumulated with multi-scale defect information.

[0056] In this embodiment of the application, the text encoder consists of 12 text transformer encoding layers. For example... Figure 1 As shown, this embodiment employs a bias fine-tuning strategy during training: freezing over 99% of the parameters of the text encoder and training only the bias terms of the second linear layer in the feedforward network. This design maximizes the preservation of the pre-trained semantic space and avoids catastrophic forgetting during fine-tuning in fabric-specific domains.

[0057] In this embodiment of the application, the text prompt adopts a prompt integration strategy: using "A photo of the [state] [class]" as a template, where [state] is composed of various words describing normal and abnormal states, and the average value of the text encoder output corresponding to all templates is taken as the final text feature representation.

[0058] In this embodiment of the application, the processing procedure in the anomaly classification module includes:

[0059] Specifically, the anomaly classification is achieved through visual and semantic similarity calculation. The final global category features enhanced by the bidirectional alignment unit (the fourth one) are mapped to a shared embedding space of the same dimension as the normal prompt embedding and defect / anomaly prompt embedding output by the text encoder using a CLIP pre-trained linear layer, resulting in a global feature embedding rich in multi-stage defect information. Subsequently, this global feature embedding is aligned with both the normal prompt embedding and defect / anomaly prompt embedding output by the text encoder, and cosine similarity is used to measure their proximity in the feature space, resulting in a cosine similarity matrix. Finally, the probability score of whether the image belongs to an abnormal state is calculated using the Softmax normalization function on the cosine similarity matrix.

[0060] In this embodiment of the application, the processing procedure in the anomaly segmentation map generation module includes:

[0061] Specifically, patch features at four different stages are extracted from layers 6, 12, 18, and 24 of the visual encoder. These features are enhanced by four bidirectional alignment units and then mapped through four independent, learnable linear projection layers to a subspace of the same dimension as the normal cue embedding and defect / anomaly cue embedding output by the text encoder, resulting in four levels of local feature embeddings. The cosine similarity between the patch features and the defect / anomaly cue embedding at each spatial location corresponding to each level of local feature embedding is calculated, thereby generating the original score distributions at four feature scales. Next, the original scores at the four feature scales are pixel-level averaged and aggregated to fuse shallow high-frequency structural information with deep semantic information. Finally, bilinear interpolation upsampling is performed on the averaged and aggregated feature map to restore its resolution to the same size as the original input image, thus obtaining the final pixel-level predicted anomaly segmentation map. Each pixel value in this segmentation map reflects the probability of a fabric defect existing at the corresponding physical location, and high-value regions in the heatmap enable precise localization of subtle defects.

[0062] S3: Fine-tune the CoBi-CLIP anomaly detection network model using a public dataset to obtain a trained CoBi-CLIP anomaly detection network model.

[0063] The training process is as follows:

[0064] In each training round, the input consists of a validation set of fabric images from the public dataset, along with normal and abnormal text prompts;

[0065] Freeze the pre-trained parameters of the visual encoder and text encoder, retaining only the parameters of the lightweight convolutional adapter, the bias term parameters of the second linear layer in the feedforward network of the text encoder, and the parameters of the four learnable linear layers generated by the anomaly segmentation map.

[0066] By using the forward propagation of the CoBi-CLIP anomaly detection network model, the probability score of an image belonging to an abnormal state and the anomaly segmentation map are predicted respectively.

[0067] The total loss is calculated based on the probability score of the image belonging to an abnormal state and the abnormal segmentation map. The parameters of the lightweight convolutional adapter, the bias term parameters of the second linear layer in the feedforward network of the text encoder, and the parameters of the four learnable linear layers of the abnormal segmentation map generation module are updated through backpropagation.

[0068] After completing a fixed number of training rounds, the trained CoBi-CLIP anomaly detection network model is obtained.

[0069] In this embodiment, the focus loss and L1 loss are used to optimize the model, and the total loss is defined as follows:

[0070]

[0071] in, This represents the probability score that the entire image belongs to an abnormal state. Indicates the outlier truth value of the image. This represents the predicted anomaly segmentation map. Focal represents the true mask. Representing focus loss, it primarily addresses the severe sample imbalance problem in fabric images, where normal areas and defective areas (a very small number of pixels) are separated. This indicates that L1 loss improves the structural consistency of defect edges.

[0072] In this embodiment, the model was fine-tuned using the validation set data of the public dataset DAGM2007, and validated using a private dataset. All experiments were performed on a single NVIDIA GeForce RTX 4090 graphics card. The specific training configuration is as follows: the Adam optimizer was used, and the learning rate was set to... The batch size was set to 4. The entire fine-tuning process only performed two rounds of iterations.

[0073] S4: Use the trained model to perform zero-shot anomaly detection on the private fabric dataset, and compare the similarity between the input image and text features to obtain the anomaly classification results and anomaly segmentation map.

[0074] Furthermore, the effectiveness of the algorithm improvement proposed in this invention was verified through ablation experiments, and the experimental results are shown in Table 1. V1 represents the model after CLIP uses the bias fine-tuning strategy; V2 represents the model with a lightweight convolutional adapter added to V1; and V3 represents the model with a bidirectional alignment unit added to V2, which is also the final model. As can be seen from the table, after using the bias fine-tuning strategy, all indicators showed significant improvement. This fully demonstrates that by simply fine-tuning the bias term in the text encoder feedforward network, the semantic gap between visual features and text embeddings can be effectively bridged without destroying the pre-trained semantic knowledge of CLIP, significantly enhancing the model's accuracy in distinguishing between normal and abnormal fabric states. In the V2 stage, after introducing the lightweight convolutional adapter, AP further increased to 85.5%, and IoU increased from 69.9% to 75.3%. This result validates the necessity of injecting inductive bias into the shallow network of the visual encoder. The output of the convolutional adapter effectively compensates for the shortcomings of standard ViT in modeling local spatial relationships, enhancing the ability to capture high-frequency structural defects in fabrics (such as minor scratches and warp breaks), thereby improving the accuracy of defect localization. In the V3 stage, after adding the bidirectional alignment unit, the model achieved optimal performance, with AP reaching 89.8%, AUROC reaching 99.1%, and the mean absolute error (MAE) decreasing to a minimum of 0.012. This indicates that the explicit global-local interaction mechanism designed in this invention plays a crucial role: through the cyclical enhancement of context injection and defect aggregation, global class features can actively accumulate local defect details from different levels, successfully bridging the granular differences between macroscopic semantics and microscopic structural anomalies. Furthermore, the bidirectional alignment mechanism can significantly suppress false positive activations in non-defect regions, allowing the model's attention to focus more accurately on defect regions. In summary, the three core improvement components proposed in this invention have a strong synergistic effect. Through progressive feature enhancement and alignment optimization, the CoBi-CLIP model exhibits excellent robustness and extremely high detection accuracy in fabric defect detection tasks, fully demonstrating the correctness of the improvement scheme.

[0075] Model prediction results are as follows Figure 2 As shown, Figure 2 From left to right: an original image of the defective fabric from the private dataset, the original image after overlaying with a red-marked mask, and a heatmap of defect segmentation predicted by the model. Figure 2 As can be seen, the model's predictions almost perfectly match the ground truth mask values, completely covering the entire defect location. This demonstrates that, even without training on a private dataset, and only fine-tuning the model using a public dataset, the model can exhibit remarkably good zero-shot defect localization capabilities on a private dataset.

[0076] Table 1 Ablation Experiment Results of the Model

[0077]

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A zero-sample fabric defect detection method based on a fine-tuned CLIP model, characterized in that, Including the following steps: S1: Obtain the public dataset and the private fabric dataset and preprocess them respectively; S2: Construct a CoBi-CLIP anomaly detection network model, which includes a parallel visual encoder and a text encoder. Between the visual encoder and the text encoder, there is also an anomaly classification module and an anomaly segmentation map generation module. A lightweight convolutional adapter and a bidirectional alignment unit are introduced into the visual encoder. The lightweight convolutional adapter is used to extract local contextual features of different scales in parallel using a multi-branch bottleneck architecture. The bidirectional alignment unit is used to integrate macro-semantic and micro-structural information through context injection and defect aggregation mechanisms. S3: Fine-tune the CoBi-CLIP anomaly detection network model using the preprocessed public dataset to obtain the trained CoBi-CLIP anomaly detection network model. S4: Use the pre-processed and trained CoBi-CLIP anomaly detection network model to perform anomaly detection on the private fabric dataset, and obtain anomaly classification results and anomaly segmentation maps.

2. The zero-sample fabric defect detection method based on a fine-tuned CLIP model according to claim 1, characterized in that, The visual encoder adopts the ViT architecture, which uses stacked visual transformer coding layers, including a first visual transformer coding layer to a 24th visual transformer coding layer. A lightweight convolutional adapter is introduced in the first to the 7th visual transformer coding layers of the visual encoder. The text encoder adopts a transformer architecture, which uses stacked text transformer encoding layers, which sequentially include a first text transformer encoding layer to a twelfth text transformer encoding layer.

3. The zero-sample fabric defect detection method based on a fine-tuned CLIP model according to claim 2, characterized in that, A first bidirectional alignment unit is introduced between the 6th and 24th visual transformer coding layers of the visual encoder; a second bidirectional alignment unit is introduced between the 12th and 24th visual transformer coding layers; a third bidirectional alignment unit is introduced between the 18th and 24th visual transformer coding layers; and a fourth bidirectional alignment unit is introduced after the 24th visual transformer coding layer. The first, second, third, and fourth bidirectional alignment units are connected sequentially, and the parameters are shared among the four bidirectional alignment units.

4. The zero-sample fabric defect detection method based on the fine-tuned CLIP model according to claim 3, characterized in that, Each visual transformer coding layer in the first to seventh visual transformer coding layers includes, in sequence, a first layer normalization, a multi-head self-attention mechanism, a lightweight convolutional adapter, a second layer normalization, and a feedforward network. The input of the first layer normalization is connected to the output of the multi-head self-attention mechanism through a first residual connection, and the input of the second layer normalization is connected to the output of the feedforward network through a second residual connection.

5. The zero-sample fabric defect detection method based on the fine-tuned CLIP model according to claim 4, characterized in that, The lightweight convolutional adapter utilizes a multi-branch bottleneck architecture to extract local contextual features at different scales in parallel, specifically including: The patch feature output from the first residual join. Reconstructed into a spatial feature map And by using a linear dimensionality reduction layer to refine the spatial feature map Dimensionality reduction is performed to obtain dimensionality-reduced feature maps; these feature maps are then fed in parallel into two depthwise separable convolutional branches with different kernels; the outputs of the two depthwise separable convolutions are then up-dimensionalized through linear up-dimensionality layers and concatenated, followed by... The fused feature map is obtained after convolutional fusion. Utilizing learnable layer scaling parameters Adjusting the fusion feature map The contribution of the spatial feature map Perform residual connections to obtain feature maps. and feature map Remodeling into patch features .

6. The zero-sample fabric defect detection method based on a fine-tuned CLIP model according to claim 3 or 4, characterized in that, In each bidirectional alignment unit, macroscopic semantic and microscopic structural information are integrated through context injection and defect aggregation mechanisms, specifically including: Context injection stage: Using the patch features input from the bidirectional alignment unit as the query and the global category features output from the last visual transformer encoding layer as the key and value, a cross-attention operation is performed to obtain the enhanced patch features; Defect aggregation stage: Using the global category feature output by the last visual transformer encoding layer as the query and the enhanced patch feature as the key and value, perform cross attention operation to obtain the enhanced global category feature of the current bidirectional alignment unit.

7. The zero-sample fabric defect detection method based on a fine-tuned CLIP model according to claim 6, characterized in that, The processing steps in the anomaly classification module include: The final global category features enhanced by the fourth bidirectional alignment unit are mapped to a shared embedding space of the same dimension as the normal prompt embedding and defect / abnormal prompt embedding output by the text encoder through a pre-trained linear layer, resulting in a global feature embedding rich in multi-stage defect information. Subsequently, the global feature embedding rich in multi-stage defect information is aligned with the normal prompt embedding and defect / abnormal prompt embedding output by the text encoder through cosine similarity calculation to obtain a cosine similarity matrix. The probability score of the image belonging to an abnormal state is obtained by calculating the cosine similarity matrix through the Softmax normalization function.

8. The zero-sample fabric defect detection method based on the fine-tuned CLIP model according to claim 6, characterized in that, The processing steps in the anomaly segmentation map generation module include: Four enhanced patch features are extracted from the outputs of four bidirectional alignment units. These four enhanced patch features are then mapped through four independent, learnable linear layers to a subspace of the same dimension as the normal prompt embedding and defect / anomaly prompt embedding output by the text encoder, resulting in four levels of local feature embeddings. The cosine similarity between the patch features and the defect / anomaly prompt embedding at each spatial location corresponding to each level of local feature embedding is calculated, generating the original score distributions at four feature scales. The original scores at the four feature scales are then pixel-level averaged and aggregated. Finally, bilinear interpolation upsampling is performed on the averaged and aggregated feature map to restore the resolution of the averaged and aggregated feature map to the same size as the original input image, resulting in the final pixel-level predicted anomaly segmentation map.

9. The zero-sample fabric defect detection method based on a fine-tuned CLIP model according to any one of claims 3, 4, 7 or 8, characterized in that, During training, the pre-trained parameters of the visual encoder and text encoder are frozen, and only the parameters of the lightweight convolutional adapter, the bias term parameters of the second linear layer in the feedforward network of the text encoder, and the four learnable linear layer parameters of the anomaly segmentation map generation module are learned. The training loss uses focus loss and L1 loss.

10. The zero-sample fabric defect detection method based on a fine-tuned CLIP model according to claim 7 or 8, characterized in that, The normal and abnormal text features output by the text encoder are processed using a cue ensemble strategy: A photo of the [state][class] is used as a template, where [state] consists of various words describing normal and abnormal defect states, and [class] is the category name of the defect. The final text feature representation is obtained by averaging the text encoder outputs corresponding to all templates.

Images