A Text-Guided Zero-Sample Surface Defect Segmentation Method

Abstract

This invention belongs to the field of surface defect detection technology and discloses a text-guided zero-shot surface defect segmentation method. It introduces human-provided text information as auxiliary information to achieve more accurate zero-shot surface defect segmentation; it fuses multi-level image features through an attention-based multi-level feature fusion module to form a more comprehensive feature representation; it fuses aligned text and image features through a multi-modal feature adaptive balancing module to improve the model's generalization ability; and it obtains a priori image-text correlation map based on the aligned features to guide the decoder, achieving pixel-level segmentation of target defects even in the absence of labeled samples. This invention can effectively segment surface defects under zero-shot conditions and is closer to the ground truth label than existing benchmarks.

Description

Technical Field

[0001] This invention relates to the field of surface defect detection technology, and in particular to a text-guided zero-sample surface defect segmentation method. Background Technology

[0002] In industrial production, defects are inevitably produced on the surface of objects due to manufacturing processes and the quality of production materials. These defects not only affect the appearance and performance of products, but more seriously, they can easily cause serious production accidents and bring enormous harm. Machine vision, as an automatic inspection technology, can operate in complex environments and is not only safe but also highly efficient and accurate. Currently, vision-based surface defect detection technologies mostly employ segmentation algorithms, which can locate, identify, and classify defects, meeting the high-precision requirements in industry.

[0003] However, in real-world industrial applications, defects are rare, and obtaining large-scale labeled defect datasets is difficult and time-consuming. When data is scarce, traditional surface defect segmentation algorithms often struggle to fully learn the diverse and complex features, resulting in insufficient generalization ability and poor model performance. Furthermore, current computer vision algorithms rely solely on image information, conveying intuitive visual content through visual features. However, high-quality data acquisition is difficult; images may contain blur, occlusion, or ambiguity, preventing the comprehensive capture of defect details and significantly reducing model performance. Moreover, using only image information wastes human expertise, resulting in models with weak adaptability to environmental changes and an inability to identify diverse defects.

[0004] Inspired by the human learning process, researchers have explored few-shot and zero-shot learning methods, designing algorithmic models capable of rapid learning and generalization under limited data conditions. Existing few-shot algorithms can segment target defects using only one support image, while zero-shot algorithms can segment unseen defects without any support images; therefore, zero-shot defect segmentation networks have attracted considerable attention. The rise of visual language models offers new insights into current challenges, demonstrating significant advantages in zero-shot tasks.

[0005] Visual language models can leverage human-provided textual guidance to offer additional instruction when image quality is low. Pre-trained large-scale visual language models, requiring no additional fine-tuning, can learn to capture broad semantic knowledge, adapt to diverse defects and segmentation needs across different scenarios, and enhance the model's generalization ability. Furthermore, domain expert knowledge can be incorporated into the model, compensating for the shortcomings of purely data-driven models and improving its performance in real-world applications. Therefore, we introduce additional textual guidance, combining visual and linguistic information to achieve zero-shot surface defect segmentation, better adapting to complex situations in real-world applications and meeting the practical needs of industrial applications. Summary of the Invention

[0006] This invention provides a text-guided zero-shot surface defect segmentation method. By introducing textual guidance information, additional clues are obtained, thereby further improving the surface defect segmentation model's ability to identify unseen defect categories. The zero-shot learning method enables the model to learn and generalize rapidly even in the absence of labeled data, meeting the diverse defect detection needs in practical applications.

[0007] The technical solution of this invention is as follows: a text-guided zero-sample surface defect segmentation method, which obtains text prompts based on text templates by using target defect categories and background information provided by humans; and extracts text embeddings and image features by using the text encoder and image encoder of the visual language model CLIP, respectively.

[0008] An attention-based multi-level feature fusion module fuses image features obtained from the image encoder to obtain multi-level fused features.

[0009] The image features and text embeddings are input into the multimodal feature adaptive balancing module. Image feature representations and text feature representations are extracted according to self-attention. Cross-attention is used to calculate the interaction between text feature representations and image feature representations, thereby promoting alignment and fusion between different modalities.

[0010] Prior image-text correlation maps are obtained based on text feature representation and image feature representation;

[0011] The multi-level fusion features and the prior image-text correlation map are concatenated along the channel dimension and input into the decoder to obtain the pixel-level segmentation result of the target defect.

[0012] The text template is “a photo of the {class}defect on the {background}”, and {class} and {background} appear in pairs, replaced with the actual defect category and background information;

[0013] The image encoder of the visual language model CLIP is an improved ResNet50 network, which extracts features from the target image to obtain corresponding image features. Where n is the nth layer feature obtained in the image encoder;

[0014] The improved ResNet50 network adds an attention pooling layer to the last layer of the four residual blocks based on the original ResNet50 network; in the attention pooling layer, the features F4 corresponding to the last residual block of the image encoder are obtained by global average pooling to obtain global features. By splicing High-level image features F are obtained from the final output of the image encoder through multi-head attention. out :

[0015]

[0016] Wherein, MHSA(·) represents multi-head attention mechanism, represents the enhanced global feature, and z represents the semantically compatible feature.

[0017] The attention-based multi-level feature fusion module includes a self-attention mechanism and a cross-attention mechanism; the intermediate feature F3 corresponding to the third layer residual block of the image encoder is upsampled to obtain F3′, which has the same size as the intermediate feature F2 corresponding to the second layer residual block of the image encoder, and F3′ and F2 are fused:

[0018]

[0019] Where Concat(·) represents concatenation; adjust the fusion feature F m Size obtains two-dimensional fusion features Where C represents the number of channels of the feature, N represents H1×W1, and H1 and W1 represent the features F m Dimensions;

[0020] The fusion feature is enhanced by a self-attention mechanism. A weight matrix is ​​generated by calculating the similarity between each position of the input fusion feature and other positions, and the value vectors of all positions are summed according to their weights.

[0021]

[0022] Where, d k =C represents the vector dimension, used to avoid Q1K1 T If the value is too large, SelfAttn(·) represents the self-attention mechanism, Softmax(·) represents the normalization function, and Q1, K1, and V1 are determined by F. m 'Obtained by calculation using linear transformation;

[0023] The self-attention-enhanced features are fused with the two-dimensional feature F. m Enhanced intermediate features are obtained through residual connections:

[0024] F S =F m ′+A S (4)

[0025] By fusing high-level image features and enhanced mid-level features through a cross-attention mechanism, a fused feature A is obtained. CThis is then residually connected to the enhanced intermediate features, and the feature size is adjusted to obtain the multi-level fused feature F. C ′:

[0026]

[0027]

[0028] Among them, Q2 is composed of enhanced intermediate features F S The mapping is obtained, while K2 and V2 are derived from high-level image features F. out The mapping is obtained.

[0029] The multimodal feature adaptive balancing module is a dual-branch network including a text branch and an image branch; the text branch receives text embeddings, and the image branch receives image features. The text embeddings and image features are respectively enhanced using a self-attention mechanism; the interaction between the text embeddings and image features is established using a cross-attention mechanism to align the text and image modalities, and the multimodal feature adaptive fusion and alignment is achieved based on learnable parameters.

[0030] The feature enhancement process involves obtaining image features F out and text embedding F t Enhancement is performed based on self-attention, and the enhanced features are concatenated with the original features as residuals to obtain enhanced text embeddings and enhanced image features:

[0031] F St =SelfAttn(F t )+F t (7)

[0032] F Si =SelfAttn(F out )+F out (8)

[0033] in, Q, K, and V are each derived from the input text embedded by F. t and image features F out Obtained by calculation using linear transformation;

[0034] The process for obtaining the interaction between the text embedding and image features is as follows:

[0035] In the text branch, cross-attention is used to compute and enhance the text embedding F. St The corresponding enhanced image feature F Si By aligning semantic information with visual features, text relevance features are obtained from the regions:

[0036] F Ct=CrossAttn(F St ,F Si ,F Si )+F St (9)

[0037] in, Representing the cross-attention mechanism, enhanced text embedding F St Q is obtained by calculating linear transformation, and the enhanced image features F are obtained by... Si K and V are obtained by calculating linear changes;

[0038] In the image branch, cross-attention is used to calculate the enhanced image feature F. Si Corresponding enhanced text embedding F St By identifying regions and establishing stronger semantic connections, image relevance features can be obtained.

[0039] F Ci =CrossAttn(F Si ,F St ,F St )+F Si (10)

[0040] Where CrossAttn(·) represents the cross-attention mechanism, which has the same structure as CrossAttn(·) in equation (9), and enhances the image features F Si Q is obtained by calculating linear transformation, and F is the enhanced text embedding. St K and V are obtained by calculating linear changes;

[0041] The adaptive fusion alignment process involves using text relevance features and image relevance features as residual terms, and introducing learnable parameters into the text branch and image branch respectively. Where C represents F Ct and F Ci The number of channels is used to adaptively adjust the contribution of the residual term, resulting in text alignment features and image alignment features:

[0042] F t ′=F t +αF Ct (11)

[0043]

[0044] Among them, F out ′By global features of alignment It is obtained by concatenating the semantically compatible features z′ of the alignment.

[0045] The prior image-text correlation map consists of aligned semantically compatible features. Text alignment features Calculated, K represents the number of categories, H4 and W4 represent the size of feature z′, C4 represents the number of channels of feature z′, and the prior image-text correlation map P′ is obtained through interpolation:

[0046]

[0047]

[0048] in, φ(·) represents the bilinear interpolation operation.

[0049] The decoder concatenates multi-level fused features and prior image-text correlation maps along the channel dimension, and then decodes them using the ASPP decoder to obtain pixel-level segmentation results of defects.

[0050] The beneficial effects of this invention are as follows:

[0051] 1. This invention proposes a text-guided zero-sample surface defect segmentation method by introducing human-provided text information. It performs pixel-level matching between text features and image features to achieve pixel-level classification, overcoming the problem of unlabeled samples for unseen defect categories.

[0052] 2. This invention proposes an attention-based multi-level image feature fusion strategy, which includes intermediate and high-level image features. This strategy compensates for the shortcomings of a single feature level and improves generalization ability while preserving sufficient detailed information and semantic consistency with text features.

[0053] 3. This invention proposes a multimodal feature adaptive balancing module, which includes branches for text and image, respectively, to perform modal self-reinforcement and interaction, fully utilizing contextual information to balance the contributions of the two branches. This module can dynamically adjust and fuse multimodal features, solving the feature misalignment problem.

[0054] 4. This invention guides downstream defect segmentation tasks by generating prior image-text correlation maps, providing multimodal prior guidance to help the model understand image content more comprehensively. Attached Figure Description

[0055] Figure 1 A flowchart of a text-guided zero-sample surface defect segmentation method provided in this embodiment;

[0056] Figure 2 This is a block diagram of the overall structure of the method provided in this embodiment;

[0057] Figure 3 This is a schematic diagram of the attention-based multi-level image feature fusion module provided in this embodiment;

[0058] Figure 4 A schematic diagram of the multimodal feature adaptive balancing module provided in this embodiment;

[0059] Figure 5 This is a visualization of the surface defect segmentation results provided in this embodiment. (a) is the text guidance; (b) is the image; (c) is the ground truth; (d) is the segmentation result; and (e) is the baseline. Detailed Implementation

[0060] This example provides a text-guided zero-sample surface defect segmentation method. The invention will be described in detail below with reference to the accompanying drawings and embodiments. Figure 1 and Figure 2 As shown, the following steps are included:

[0061] Step 1: Based on the given text template and human-provided prompt text, including the target defect category and background information, obtain text prompts; use image encoders and text encoders respectively to extract image features and text embeddings.

[0062] In this embodiment, as Figure 2 As shown, the given text template is “a photo of the {class}defect on the {background}”, where {class} represents the defect category and {background} represents the background information of the defect. The two appear in pairs and are replaced by human-provided prompt text.

[0063] The image encoder and text encoder are derived from the pre-trained image encoder and text encoder of the visual language model CLIP. The image encoder is a ResNet50 network consisting of four residual blocks, with each residual block corresponding to image features. Furthermore, an attention layer was added to the original network. Specifically, global average pooling was performed on F4 to obtain global features. The two features are concatenated to obtain Multi-head attention layer (MHSA) is used to obtain the final output image features F. out :

[0064]

[0065] Wherein, MHSA(·) represents multi-head attention mechanism, denoted by , where z represents global features and z represents semantically compatible features. It is obtained by splicing two features.

[0066] The text cues are embedded into a continuous vector space using a text encoder derived from a visual language model, resulting in the text embedding F. t Furthermore, the text encoder here is frozen during training and its parameters are not updated.

[0067] Step 2: Input the multi-level image features obtained from the image encoder into a system such as... Figure 3 In the attention-based multi-level feature fusion module shown, two intermediate features are concatenated to obtain an enhanced intermediate feature based on self-attention and residual connections. The enhanced intermediate feature is then fused with a high-level feature that has semantic consistency based on cross-attention and residual connections to obtain a multi-level fused feature.

[0068] In this embodiment, in the attention-based multi-level feature fusion module, the intermediate feature F3 is upsampled to obtain F3′, which has the same size as the intermediate feature F2. The two intermediate features F3′ and F2 are then concatenated to obtain the fused intermediate feature F. m :

[0069]

[0070] Where Concat(·) represents concatenation; adjust the fusion feature F m Size obtains two-dimensional fusion features Where C represents the number of channels of the feature, N represents H1×W1, and H1 and W1 represent the features F m The size.

[0071] The self-attention mechanism calculates the input feature F. m The similarity between each position and other positions is used to generate a weight matrix, and the value vectors of all positions are summed according to their weights to obtain the enhanced feature A. S , and fuse it with the two-dimensional feature F m Through residual connections, enhanced intermediate features F are obtained. S :

[0072]

[0073] F S =F m ′+A S (18)

[0074] Where, d k =C represents the vector dimension, used to avoid Q1K1 T If the value is too large, SelfAttn(·) represents the self-attention mechanism, Softmax(·) represents the normalization function, and Q1, K1, and V1 are determined by F. m It was obtained by calculation using linear transformation.

[0075] The cross-attention mechanism is used to fuse high-level features and enhanced mid-level features to improve feature representation capabilities; the enhanced mid-level features F S Linear transformation yields Q2, and high-level features F out Linear transformation yields K2 and V2, and cross-attention is used to obtain the fused feature A. C This is then residually connected to the enhanced intermediate features, and the feature size is adjusted to obtain the multi-level fused feature F. C ′:

[0076]

[0077]

[0078] Step 3: As Figure 2 As shown, the image features and text embeddings extracted by the image encoder and text encoder are input into the multimodal feature adaptive balancing module; as... Figure 4 As shown, the multimodal feature adaptive balancing module is a dual-branch network, including a text branch and an image branch, which are used for feature enhancement based on self-attention and residual connections for visual and language modalities, and feature fusion based on cross-attention and residual connections, respectively, to obtain text-related features and image-related features. At the same time, learnable parameters are introduced to dynamically adjust the contribution of the related features, enhance the expressive power of text and image features, and obtain text-aligned features and image-aligned features.

[0079] In this embodiment, the multimodal feature adaptive balancing module utilizes a self-attention mechanism to adjust image features F respectively. out and text embedding F t Feature enhancement is performed, and the enhanced features are concatenated with the original features as residuals to obtain enhanced text embeddings and enhanced image features:

[0080] F St =SelfAttn(F t )+F t (twenty one)

[0081] F Si =SelfAttn(F out )+F out (twenty two)

[0082] in, The Q, K, and V required in equations (21) and (22) are respectively derived from the input text embedded in F. t and image features F out Obtained by calculation using linear transformation;

[0083] An interaction between text and image is established using a cross-attention mechanism, aligning text and image modalities; this interaction, within the text branch, is based on cross-attention to obtain and enhance the text embedding F. St The corresponding enhanced image feature F Si Regions allow the model to focus on the most relevant image regions and obtain more accurate semantic information. By aligning semantic information with visual features, text relevance features are obtained.

[0084] F Ct =CrossAttn(F St ,F Si ,F Si )+F St (twenty three)

[0085] in, Representing the cross-attention mechanism, enhanced text embedding F St Q is obtained by using linear transformation calculation, and the enhanced image features F are obtained. Si K and V are obtained by calculating linear variations;

[0086] In the image branch, image features F are obtained and enhanced based on cross-attention. Si Corresponding enhanced text embedding F St Regions help extract semantic information that best matches image features, thereby establishing stronger semantic connections and obtaining image relevance features:

[0087] F Ci =CrossAttn(F Si ,F St ,F St )+F Si (twenty four)

[0088] Where CrossAttn(·) represents the cross-attention mechanism, which has the same structure as CrossAttn(·) in equation (23), and enhances the image features F Si Q is obtained by calculating linear transformation, and F is the enhanced text embedding. St K and V are obtained by linear transformation.

[0089] In the two branches, text relevance features and image relevance features are used as residual terms, respectively, through learnable parameters. and To dynamically adjust the contribution of the residual term in each branch, where C represents the correlation feature F. Ct and F Ci The number of channels is used to obtain text alignment features and image alignment features:

[0090] F t ′=Ft +αF Ct (25)

[0091]

[0092] Where α and β are learnable parameters that are continuously updated during training, F out ′By global features of alignment It is obtained by concatenating the semantically compatible features z′ of the alignment.

[0093] Step 4: Generate a prior image-text correlation map based on text features and image features to guide the decoder and improve segmentation accuracy.

[0094] The prior image-text correlation map is used to calculate aligned semantic compatibility features through dot product. Text alignment features The similarity is calculated as follows: K represents the number of categories, H4 and W4 represent the size of feature z′, and C4 represents the number of channels of feature z′. The prior image-text correlation map P′ is obtained through interpolation.

[0095]

[0096] in, φ(·) represents the bilinear interpolation operation.

[0097] Step 5: As Figure 2 As shown, the obtained multi-level fusion features and the prior image-text correlation map channel dimensions are concatenated and input into the decoder for decoding to obtain the final pixel-level defect segmentation result. A partial visualization of the surface defect segmentation results is shown in the image below. Figure 5 As shown.

[0098] This method, relying solely on text guidance, achieved a mean cross-union ratio (CUN) of 33.40% for surface defect categories on the Defect-4i dataset and 53.29% for surface defect categories on the MVTec-ZSS dataset. Figure 5 As shown, the present invention can effectively segment surface defects under zero-sample conditions, and is closer to the true label than existing benchmarks.

Claims

1. A text-guided zero-sample surface defect segmentation method, characterized in that, Based on the target defect category and background information provided by humans, text prompts are obtained according to the text template; text embeddings and image features are extracted by the text encoder and image encoder of the visual language model CLIP, respectively. An attention-based multi-level feature fusion module fuses image features obtained from the image encoder to obtain multi-level fused features. The image features and text embeddings are input into the multimodal feature adaptive balancing module. Image feature representations and text feature representations are extracted according to self-attention. Cross-attention is used to calculate the interaction between text feature representations and image feature representations, thereby promoting alignment and fusion between different modalities. Prior image-text correlation maps are obtained based on text feature representation and image feature representation; The multi-level fusion features and the prior image-text correlation map are concatenated along the channel dimension and input into the decoder to obtain the pixel-level segmentation result of the target defect; the text template is "a photo of the {class}defect on the {background}", and {class} and {background} appear in pairs, which are replaced with the actual defect category and background information; The image encoder of the visual language model CLIP is an improved ResNet50 network, which extracts features from the target image to obtain corresponding image features. , where n is the nth layer feature obtained in the image encoder; The improved ResNet50 network adds an attention pooling layer to the last layer of the four residual blocks based on the original ResNet50 network; in the attention pooling layer, the features corresponding to the last residual block of the image encoder are... Global features are obtained through global average pooling. By splicing them together, we can obtain High-level image features are obtained from the final output of the image encoder through multi-head attention. : ； in, This indicates a multi-head attention mechanism. Indicates enhanced global features, Represents semantically compatible features; the attention-based multi-level feature fusion module includes self-attention and cross-attention mechanisms; it integrates the intermediate features corresponding to the third-layer residual block of the image encoder. Obtained through upsampling Intermediate features corresponding to the second-layer residual blocks of the image encoder Same size, fused and : ； in, Indicates splicing; adjusts fusion features Size obtains two-dimensional fusion features ,in, , The number of channels representing the feature. express , and Representation of features Dimensions; The fusion feature is enhanced by a self-attention mechanism. A weight matrix is ​​generated by calculating the similarity between each position of the input fusion feature and other positions, and the value vectors of all positions are summed according to their weights. ； in, Representing the vector dimension to avoid The value is too large. This indicates the self-attention mechanism. Represents the normalization function. , and Depend on Obtained by calculation using linear transformation; The self-attention enhanced features are fused with the 2D features. Enhanced intermediate features are obtained through residual connections: ； By fusing high-level image features and enhanced mid-level features through a cross-attention mechanism, a fused feature is obtained. This is then residually connected to the enhanced intermediate features, and the feature size is adjusted to obtain multi-level fused features. : ； ； in, Enhanced intermediate features The mapping yields, and and From high-level image features The mapping is obtained; the multimodal feature adaptive balancing module is a dual-branch network including a text branch and an image branch; the text branch receives text embeddings, and the image branch receives image features. The two branches respectively use self-attention mechanisms to enhance the text embeddings and image features; the interaction between text embeddings and image features is established using a cross-attention mechanism to align the text and image modalities, and at the same time, multimodal feature adaptive fusion alignment is achieved based on learnable parameters; The feature enhancement process involves obtaining image features... and text embedding Enhancement is performed based on self-attention, and the enhanced features are concatenated with the original features as residuals to obtain enhanced text embeddings and enhanced image features: ； ； in, Each group , and Embedded by the input text respectively and image features Obtained by calculation using linear transformation; The process for obtaining the interaction between the text embedding and image features is as follows: In the text branch, cross-attention computation and enhanced text embedding are utilized. Corresponding enhanced image features By aligning semantic information with visual features, text relevance features are obtained from the regions: ； in, This represents a cross-attention mechanism, enhancing text embedding. Obtained by linear transformation calculation Enhanced image features Obtained by linear transformation calculation and ; In the image branch, cross-attention is used to calculate enhanced image features. Corresponding enhanced text embedding By identifying regions and establishing stronger semantic connections, image relevance features can be obtained. ； in, This represents the cross-attention mechanism, as shown in equation (9). Same structure, enhanced image features Obtained by linear transformation calculation Enhanced text embedding Obtained by linear transformation calculation and ; The adaptive fusion alignment process involves using text relevance features and image relevance features as residual terms, and introducing learnable parameters into the text branch and image branch respectively. , ,in express Number of channels, The number of channels is used to adaptively adjust the contribution of the residual term, resulting in text alignment features and image alignment features: ； ； in, Global features of alignment semantic compatibility features of alignment It was pieced together.

2. The text-guided zero-sample surface defect segmentation method according to claim 1, characterized in that, The prior image-text correlation map consists of aligned semantically compatible features. Text alignment features Calculated Indicates the number of categories. and Representation of features Size Representation of features The number of channels is used to obtain the prior image-text correlation map through interpolation. : ； ； in, , This indicates a bilinear interpolation operation.

3. The text-guided zero-sample surface defect segmentation method according to claim 1 or 2, characterized in that, The decoder concatenates multi-level fused features and prior image-text correlation maps along the channel dimension, and then decodes them using the ASPP decoder to obtain pixel-level segmentation results of defects.

Images