Zero sample surface defect segmentation method based on text guidance

Abstract

The invention belongs to the technical field of surface defect detection, and discloses a zero sample surface defect segmentation method based on text guidance. Text information provided by human beings is introduced as auxiliary information, so that more accurate zero sample surface defect segmentation is realized; multi-level image features are fused through a multi-level feature fusion module based on attention, and more comprehensive feature representation is formed; text and image features are fused and aligned through a multi-modal feature adaptive balance module, and the generalization ability of the model is improved; a prior image-text correlation graph is obtained based on the aligned features to guide a decoder to decode, and pixel-level segmentation of target defects is realized under the condition of lack of marked samples. The method can effectively segment the surface defect under a zero sample condition, and is closer to a true value label than an existing reference.

Description

Technical Field

[0001] The present invention relates to the technical field of surface defect detection, and in particular to a zero-sample surface defect segmentation method based on text guidance. Background Art

[0002] In the process of industrial production, defects are inevitable on the surface of objects due to the manufacturing process and the quality of production materials. These defects not only affect the appearance and performance of the product, but more seriously, surface defects can easily cause serious production accidents and bring great harm. As an automatic detection technology, machine vision can operate in complex environments. It is not only safe, but also has the characteristics of high efficiency and high precision. At present, vision-based surface defect detection technology mostly uses segmentation algorithms, which can locate, identify and classify defects to meet the high-precision requirements in industry.

[0003] However, in actual industrial applications, defects are rare, and obtaining large-scale annotated defect datasets is difficult and time-consuming. When data is lacking, traditional surface defect segmentation algorithms often find it difficult to fully learn the features of diversity and complexity, which leads to insufficient generalization of the model and poor model performance. In addition, current computer vision algorithms only use image information to convey intuitive visual content through visual features. However, high-quality data collection is difficult, and images may have problems such as blur, occlusion, or ambiguity, making it impossible to fully capture defect details, resulting in greatly reduced model performance. In addition, using only image information wastes human expertise, the model has weak ability to adapt to environmental changes, and cannot identify diverse defects.

[0004] Inspired by the human learning process, researchers have studied small-sample and zero-shot learning methods, and designed algorithm models that can quickly learn and generalize with a small amount of data. Existing small-sample algorithms can segment target defects with only one supporting image, while zero-shot algorithms can segment unseen defects without any supporting images, so zero-shot defect segmentation networks have attracted attention. The rise of visual language models provides new ideas for the current challenges, and they have shown significant advantages in zero-shot tasks.

[0005] The visual language model can use the textual guidance information provided by humans to provide additional guidance to the model when the image quality is low. The pre-trained large visual language model can learn to capture a wide range of semantic knowledge without additional fine-tuning, adapt to the segmentation needs of various defects and different scenarios, and enhance the generalization ability of the model. In addition, the knowledge of domain experts can be applied to the model to make up for the shortcomings of pure data-driven models and improve the performance of the model in practical applications. Therefore, we introduced additional textual guidance and combined visual and language information to achieve zero-sample surface defect segmentation, which better adapts to the complex situations in practical applications and meets the actual needs of industrial applications. Summary of the invention

[0006] The present invention provides a zero-shot surface defect segmentation method based on text guidance, which introduces text guidance information to obtain additional clues, thereby further improving the ability of the surface defect segmentation model to identify unseen defect categories. The zero-shot learning method enables the model to quickly learn and generalize in the absence of labeled data to meet the detection needs of diverse defects in practical applications.

[0007] The technical solution of the present invention is: a zero-sample surface defect segmentation method based on text guidance, which obtains text prompts according to text templates through target defect categories and background information provided by humans; extracts text embedding and image features respectively through the text encoder and image encoder of the visual language model CLIP;

[0008] The attention-based multi-level feature fusion module fuses the image features obtained by the image encoder to obtain multi-level fusion features;

[0009] The image features and text embeddings are input into a multimodal feature adaptive balancing module, image feature representation and text feature representation are extracted respectively according to self-attention, and the interaction between the text feature representation and the image feature representation is calculated using cross attention, so as to promote alignment and fusion between different modalities;

[0010] Obtain a priori image-text correlation graph based on text feature representation and image feature representation;

[0011] The multi-level fusion features and the prior image-text correlation map are concatenated in the channel dimension and input into the decoder to obtain the pixel-level segmentation result of the target defect.

[0012] The text template is "a photo of the {class}defect on the {background}", and {class} and {background} appear in pairs and are replaced by the actual defect category and background information;

[0013] The image encoder of the visual language model CLIP is an improved ResNet50 network, which extracts features of the target image and obtains the corresponding image features. Where n is the nth layer feature obtained in the image encoder;

[0014] The improved ResNet50 network adds an attention pooling layer to the last layer of the four-layer residual block based on the original ResNet50 network; in the attention pooling layer, the feature F corresponding to the last layer of the residual block of the image encoder is 4 Get global features through global average pooling Splice to get The high-level image features F of the final output of the image encoder are obtained through multi-head attention out :

[0015]

[0016] Among them, MHSA(·) represents the multi-head attention mechanism, represents the enhanced global feature, and z represents the semantic compatible feature.

[0017] The attention-based multi-level feature fusion module includes a self-attention mechanism and a cross-attention mechanism; the intermediate feature F corresponding to the third-layer residual block of the image encoder 3 F is obtained by upsampling 3 ′, the intermediate feature F corresponding to the second layer residual block of the image encoder 2 Same size, fusion F 3 ′ and F 2 :

[0018]

[0019] Among them, Concat(·) represents concatenation; adjust the fusion feature F m Dimensions to get two-dimensional fusion features Among them, C represents the number of channels of the feature, and N represents H 1 ×W 1 , H 1 and W 1 Representation feature F m Size;

[0020] The fusion feature is enhanced through the self-attention mechanism. The weight matrix is ​​generated by calculating the similarity between each position of the input fusion feature and other positions, and the value vectors of all positions are weighted and summed according to the weights:

[0021]

[0022] Among them, d k =C represents the vector dimension, which is used to avoid Q 1 K 1 T The value is too large, SelfAttn(·) represents the self-attention mechanism, Softmax(·) represents the normalization function, Q 1 , K 1 With V 1 By F m ' is obtained by linear change calculation;

[0023] The self-attention enhanced features are combined with the two-dimensional fusion features F m'Through residual connection, the enhanced intermediate features are obtained:

[0024] F S =F m ′+A S (4)

[0025] The high-level image features and enhanced mid-level features are fused through the cross-attention mechanism to obtain the fused feature A C , perform residual connection with the enhanced intermediate features, and adjust the feature size to obtain the multi-level fusion feature F C ′:

[0026]

[0027]

[0028] Among them, Q 2 By enhancing the intermediate features F S Mapping is obtained, and K 2 With V 2 By high-level image features F out Mapping obtained.

[0029] The multimodal feature adaptive balancing module is a dual-branch network including a text branch and an image branch; the text branch receives text embedding, and the image branch receives image features, and the two use a self-attention mechanism to perform feature enhancement on the text embedding and image features respectively; a cross-attention mechanism is used to establish the interaction between the text embedding and the image features, align the text and image modalities, and realize multimodal feature adaptive fusion alignment according to learnable parameters;

[0030] The feature enhancement process is to obtain the image feature F out and text embedding F t The enhancement is performed based on self-attention respectively, and the obtained enhanced features are connected with the original features as residual terms to obtain enhanced text embedding and enhanced image features:

[0031] F St =SelfAttn(F t )+F t (7)

[0032] F Si =SelfAttn(F out )+F out (8)

[0033] in, Q, K and V are respectively embedded in F by the input text t and image features F out Obtained by linear variation calculation;

[0034] The interaction between text embedding and image features is obtained as follows:

[0035] In the text branch, we use cross-attention calculation and enhanced text embedding F St The corresponding enhanced image feature F Si Region, align semantic information with visual features to obtain text relevance features:

[0036] F Ct =CrossAttn(F St ,F Si ,F Si )+F St (9)

[0037] in, Represents the cross attention mechanism, enhanced text embedding F St The enhanced image feature F is obtained by linear change calculation. Si Use linear change calculation to obtain K and V;

[0038] In the image branch, the cross attention calculation is used to enhance the image feature F Si The corresponding enhanced text embedding F St Regions, establish stronger semantic connections, and obtain image correlation features:

[0039] F Ci =CrossAttn(F Si ,F St ,F St )+F Si (10)

[0040] Among them, CrossAttn(·) represents the cross attention mechanism, which has the same structure as CrossAttn(·) in formula (9). The enhanced image feature F Si Using linear transformation to calculate Q, the enhanced text embedding F St Use linear change calculation to obtain K and V;

[0041] The adaptive fusion alignment implementation process is to use the text correlation features and image correlation features as residual terms, and introduce learnable parameters in the text branch and the image branch respectively. Where C represents F Ct and F Ci The number of channels is used to adaptively adjust the contribution of the residual term to obtain text alignment features and image alignment features:

[0042] F t ′=F t +αF Ct (11)

[0043]

[0044] Among them, F out ′ is composed of the global features of the alignment It is concatenated with the aligned semantically compatible feature z′.

[0045] The prior image-text correlation map is composed of aligned semantically compatible features and text alignment features Calculated, K represents the number of categories, H 4 and W 4 represents the size of feature z′, C 4 Represents the number of channels of feature z′, and obtains the prior image-text correlation map P′ through interpolation:

[0046]

[0047]

[0048] in, φ(·) represents the bilinear interpolation operation.

[0049] The decoder concatenates the multi-level fusion features and the prior image-text correlation map in the channel dimension, and decodes them through the ASPP decoder to obtain a pixel-level segmentation result of the defect.

[0050] The beneficial effects of the present invention are:

[0051] 1. The present invention introduces text information provided by humans and proposes a text-guided zero-sample surface defect segmentation method, which matches text features with image features at the pixel level to achieve pixel-level classification and overcome the problem of unlabeled samples of unseen defect categories.

[0052] 2. The present invention proposes an attention-based multi-level image feature fusion strategy, including intermediate and high-level features of the image, which makes up for the shortcomings of a single feature level, while improving the generalization ability and retaining sufficient detail information and semantic consistency with text features.

[0053] 3. The present invention proposes a multimodal feature adaptive balancing module, which includes two branches of different modes, text and image, and performs self-reinforcement and interaction of the modes, making full use of context information and balancing the contributions of the two branches. This module can dynamically adjust and fuse multimodal features to solve the problem of feature misalignment.

[0054] 4. The present invention generates a priori image-text correlation graph to guide the downstream defect segmentation task, provides multimodal prior guidance, and helps the model understand the image content more comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a text-guided zero-sample surface defect segmentation method provided in this embodiment;

[0056] Figure 2 The overall structural diagram of the method provided in this embodiment;

[0057] Figure 3 A schematic diagram of an attention-based multi-level image feature fusion module provided in this embodiment;

[0058] Figure 4 A schematic diagram of a multi-modal feature adaptive balancing module provided in this embodiment;

[0059] Figure 5 Visualization diagram of the surface defect segmentation result provided in this embodiment. (a) is the text guidance; (b) is the image; (c) is the true value; (d) is the segmentation result; (e) is the benchmark. DETAILED DESCRIPTION

[0060] This example provides a zero-sample surface defect segmentation method based on text guidance. The present invention is described in detail below in conjunction with the accompanying drawings and embodiments. Figure 1 and Figure 2 As shown, the following steps are included:

[0061] Step 1: Based on the given text template and the prompt text provided by humans, including the target defect category and background information, obtain the text prompt; use the image encoder and text encoder to extract image features and text embedding respectively.

[0062] In this embodiment, Figure 2 As shown, the given text template is "a photo of the {class} defect on the {background}", where {class} represents the defect category, and {background} represents the background information of the defect, and the two appear in pairs and are replaced by the prompt text provided by humans.

[0063] The image encoder and text encoder are from the pre-trained image encoder and text encoder of the visual language model CLIP. The image encoder is a Resnet50 network with four residual blocks, each of which corresponds to the image feature And an attention layer is added on the basis of the original network. Specifically, 4Perform global average pooling to obtain global features Concatenate the two features to get Use the multi-head attention layer (MHSA) to obtain the final output image feature F out :

[0064]

[0065] Among them, MHSA(·) represents the multi-head attention mechanism, represents the global feature, z represents the semantic compatibility feature, Obtained by concatenating two features.

[0066] The text prompt is embedded into a continuous vector space through the text encoder from the visual language model to obtain the text embedding F t In addition, the text encoder here is frozen during the training process and no parameter updates are performed.

[0067] Step 2: Input the multi-level image features obtained from the image encoder into Figure 3 In the attention-based multi-level feature fusion module shown, two intermediate features are concatenated to obtain enhanced intermediate features based on self-attention and residual connection; the enhanced intermediate features are fused with high-level features with semantic consistency based on cross attention and residual connection to obtain multi-level fused features.

[0068] In this embodiment, in the attention-based multi-level feature fusion module, the intermediate feature F 3 F is obtained by upsampling 3 ′, and the intermediate feature F 2 Same size, splicing F 3 ′ and F 2 The two intermediate features are combined to obtain the intermediate feature F m :

[0069]

[0070] Among them, Concat(·) represents concatenation; adjust the fusion feature F m Dimensions to get two-dimensional fusion features Among them, C represents the number of channels of the feature, and N represents H 1 ×W 1 , H 1 and W 1 Representation feature F m size.

[0071] The self-attention mechanism calculates the input feature F mThe weight matrix is ​​generated by the similarity of each position of ′ with other positions, and the value vectors of all positions are weighted and summed according to the weights to obtain the enhanced feature A S , and fused it with the two-dimensional feature F m ′Through residual connection, the enhanced intermediate feature F is obtained S :

[0072]

[0073] F S =F m ′+A S (18)

[0074] Among them, d k =C represents the vector dimension, which is used to avoid Q 1 K 1 T The value is too large, SelfAttn(·) represents the self-attention mechanism, Softmax(·) represents the normalization function, Q 1 , K 1 With V 1 By F m ' is obtained by linear calculation.

[0075] The cross attention mechanism is used to fuse high-level features and enhanced intermediate features to enhance feature representation capabilities; the enhanced intermediate features F S Linear transformation gives Q 2 , high-level features F out Linear transformation gives K 2 With V 2 , based on cross attention to obtain fusion feature A C , perform residual connection with the enhanced intermediate features, and adjust the feature size to obtain the multi-level fusion feature F C ′:

[0076]

[0077]

[0078] Step 3: If Figure 2 As shown, the image features and text embedding extracted from the image encoder and the text encoder are input into the multimodal feature adaptive balancing module; Figure 4As shown, the multimodal feature adaptive balancing module is a dual-branch network, including a text branch and an image branch, which are respectively used for feature enhancement of visual and language modalities based on self-attention and residual connection, and feature fusion based on cross-attention and residual connection, to obtain text correlation features and image correlation features. At the same time, learnable parameters are introduced to dynamically adjust the contribution of correlation features and enhance the expressiveness of text and image features to obtain text alignment features and image alignment features.

[0079] In this embodiment, in the multimodal feature adaptive balancing module, the self-attention mechanism is used to respectively out and text embedding F t Perform feature enhancement, connect the obtained enhanced features as residual terms and the original features to obtain enhanced text embedding and enhanced image features:

[0080] F St =SelfAttn(F t )+F t (twenty one)

[0081] F Si =SelfAttn(F out )+F out (twenty two)

[0082] in, Q, K and V required in equations (21) and (22) are respectively embedded in the input text F t and image features F out Obtained by linear variation calculation;

[0083] The interaction between text and image is established by using the cross attention mechanism to align the text and image modalities; in the text branch, the interaction is based on the cross attention to obtain the enhanced text embedding F St The corresponding enhanced image feature F Si Region, so that the model focuses on the most relevant image region and obtains more accurate semantic information, aligns semantic information with visual features, and obtains text relevance features:

[0084] F Ct =CrossAttn(F St ,F Si ,F Si )+F St (twenty three)

[0085] in, Represents the cross attention mechanism, enhanced text embedding F St Use linear change calculation to obtain Q and enhanced image features F SiUse linear change calculation to obtain K and V;

[0086] In the image branch, the image feature F is obtained and enhanced based on cross attention. Si The corresponding enhanced text embedding F St Regions help extract the semantic information that best matches the image features, thereby establishing stronger semantic connections and obtaining image correlation features:

[0087] F Ci =CrossAttn(F Si ,F St ,F St )+F Si (twenty four)

[0088] Among them, CrossAttn(·) represents the cross attention mechanism, which has the same structure as CrossAttn(·) in formula (23). The enhanced image feature F Si Using linear transformation to calculate Q, the enhanced text embedding F St K and V were obtained by linear calculation.

[0089] In the two branches, the text correlation features and image correlation features are used as residual terms respectively, and the learnable parameters and To dynamically adjust the contribution of each branch residual term, where C represents the correlation feature F Ct and F Ci The number of channels is , and the text alignment features and image alignment features are obtained:

[0090] F t ′=F t +αF Ct (25)

[0091]

[0092] Among them, α and β are learnable parameters, which are continuously updated during the training process. out ′ is composed of the global features of the alignment It is concatenated with the aligned semantically compatible feature z′.

[0093] Step 4: Generate a priori image-text correlation graph based on text features and image features to guide the decoder decoding and improve segmentation accuracy.

[0094] The prior image-text correlation map is used to calculate the aligned semantic compatibility features through dot product and text alignment features The similarity of, where K represents the number of categories, H 4 and W 4represents the size of feature z′, C 4 Represents the number of channels of feature z′, and obtains the prior image-text correlation map P′ through interpolation:

[0095]

[0096] in, φ(·) represents the bilinear interpolation operation.

[0097] Step 5: Figure 2 As shown in the figure, the obtained multi-level fusion features and the channel dimension of the prior image-text correlation map are spliced ​​and input into the decoder for decoding to obtain the final defect pixel-level segmentation result. The partial visualization result of surface defect segmentation is shown in the figure Figure 5 shown.

[0098] Relying only on text guidance, this method achieves an average intersection-over-union ratio of 33.40% on the surface defect categories in the Defect-4i dataset and an average intersection-over-union ratio of 53.29% on the surface defect categories in the MVTec-ZSS dataset. Figure 5 As shown, the present invention can effectively segment surface defects under zero-sample conditions and is closer to the true value label than the existing benchmark.

Claims

1. A text-guided zero-shot surface defect segmentation method, characterized in that: Through the target defect category and background information provided by humans, text prompts are obtained according to the text template; text embedding and image features are extracted respectively through the text encoder and image encoder of the visual language model CLIP; The attention-based multi-level feature fusion module fuses the image features obtained by the image encoder to obtain multi-level fusion features; The image features and text embeddings are input into a multimodal feature adaptive balancing module, image feature representation and text feature representation are extracted respectively according to self-attention, and the interaction between the text feature representation and the image feature representation is calculated using cross attention, so as to promote alignment and fusion between different modalities; Obtain a priori image-text correlation graph based on text feature representation and image feature representation; The multi-level fusion features and the prior image-text correlation map are concatenated in the channel dimension and input into the decoder to obtain the pixel-level segmentation result of the target defect.

2. The text-guided zero-sample surface defect segmentation method according to claim 1, characterized in that: The text template is "a photo of the {class}defect on the {background}", and {class} and {background} appear in pairs and are replaced by the actual defect category and background information; The image encoder of the visual language model CLIP is an improved ResNet50 network, which extracts features of the target image and obtains the corresponding image features. Where n is the nth layer feature obtained in the image encoder; The improved ResNet50 network adds an attention pooling layer to the last layer of the four-layer residual block based on the original ResNet50 network; in the attention pooling layer, the feature F4 corresponding to the last layer of the residual block of the image encoder is pooled by global average to obtain the global feature Splice to get The high-level image features F of the final output of the image encoder are obtained through multi-head attention out : Among them, MHSA(·) represents the multi-head attention mechanism, represents the enhanced global feature, and z represents the semantic compatible feature.

3. The text-guided zero-sample surface defect segmentation method according to claim 2, characterized in that: The attention-based multi-level feature fusion module includes a self-attention mechanism and a cross-attention mechanism; the intermediate feature F3 corresponding to the third-layer residual block of the image encoder is upsampled to obtain F3′, which has the same size as the intermediate feature F2 corresponding to the second-layer residual block of the image encoder, and F3′ and F2 are fused: Among them, Concat(·) represents concatenation; adjust the fusion feature F m Dimensions to get two-dimensional fusion features Among them, C represents the number of channels of the feature, N represents H1×W1, H1 and W1 represent the feature F m Size; The fusion feature is enhanced through the self-attention mechanism. The weight matrix is ​​generated by calculating the similarity between each position of the input fusion feature and other positions, and the value vectors of all positions are weighted and summed according to the weights: Among them, d k =C represents the vector dimension, which is used to avoid Q1K1 T The value is too large, SelfAttn(·) represents the self-attention mechanism, Softmax(·) represents the normalization function, Q1, K1 and V1 are represented by F m ' is obtained by linear change calculation; The self-attention enhanced features are combined with the two-dimensional fusion features F m 'Through residual connection, the enhanced intermediate features are obtained: F S =F m ′+A S (4) The high-level image features and enhanced mid-level features are fused through the cross-attention mechanism to obtain the fused feature A C , perform residual connection with the enhanced intermediate features, and adjust the feature size to obtain the multi-level fusion feature F C ′: Among them, Q2 is composed of the enhanced mid-level features F S Mapping is obtained, while K2 and V2 are obtained by high-level image features F out Mapping obtained.

4. The text-guided zero-sample surface defect segmentation method according to claim 3, characterized in that: The multimodal feature adaptive balancing module is a dual-branch network including a text branch and an image branch; the text branch receives text embedding, and the image branch receives image features, and the two use a self-attention mechanism to perform feature enhancement on the text embedding and image features respectively; a cross-attention mechanism is used to establish the interaction between the text embedding and the image features, align the text and image modalities, and realize multimodal feature adaptive fusion alignment according to learnable parameters; The feature enhancement process is to obtain the image feature F out and text embedding F t The enhancement is performed based on self-attention respectively, and the obtained enhanced features are connected with the original features as residual terms to obtain enhanced text embedding and enhanced image features: F St =SelfAttn(F t )+F t (7) F Si =SelfAttn(F out )+F out (8) in, Q, K and V are respectively embedded in F by the input text t and image features F out Obtained by linear variation calculation; The interaction between text embedding and image features is obtained as follows: In the text branch, we use cross-attention calculation and enhanced text embedding F St The corresponding enhanced image feature F Si Region, align semantic information with visual features, and obtain text relevance features: F Ct =CrossAttn(F St ,F Si ,F Si )+F St (9) in, Represents the cross attention mechanism, enhanced text embedding F St The enhanced image feature F is obtained by linear change calculation. Si Use linear change calculation to obtain K and V; In the image branch, the cross attention calculation is used to enhance the image feature F Si The corresponding enhanced text embedding F St Regions, establish stronger semantic connections, and obtain image correlation features: F Ci =CrossAttn(F Si ,F St ,F St )+F Si (10) Among them, CrossAttn(·) represents the cross attention mechanism, which has the same structure as CrossAttn(·) in formula (9). The enhanced image feature F Si The enhanced text embedding F is obtained by linear transformation calculation. St Use linear change calculation to obtain K and V; The adaptive fusion alignment implementation process is to use the text correlation features and image correlation features as residual terms, and introduce learnable parameters in the text branch and the image branch respectively. Where C represents F Ct and F Ci The number of channels is used to adaptively adjust the contribution of the residual term to obtain text alignment features and image alignment features: F t ′=F t +αF Ct (11) Among them, F out ′ is composed of the global features of the alignment It is concatenated with the aligned semantically compatible feature z′.

5. The text-guided zero-sample surface defect segmentation method according to claim 4, characterized in that: The prior image-text correlation map is composed of aligned semantically compatible features and text alignment features Calculated, K represents the number of categories, H4 and W4 represent the size of feature z′, C4 represents the number of channels of feature z′, and the prior image-text correlation map P′ is obtained by interpolation processing: in, φ(·) represents the bilinear interpolation operation.

6. The text-guided zero-sample surface defect segmentation method according to any one of claims 1 to 5, characterized in that: The decoder concatenates the multi-level fusion features and the prior image-text correlation map in the channel dimension, and decodes them through the ASPP decoder to obtain a pixel-level segmentation result of the defect.

Images